Archives
- 2013.05.18: Outage in rack 05-11 - Luke's report
- 2013.05.18: unplanned downtime in santa clara
- 2013.05.16: /etc/resolv.conf
- 2013.05.16: dhcp has been changed to the renumbered ips, dns to change shortly
- 2013.05.14: Support
- 2013.05.13: Additional unplanned downtime
- 2013.05.12: wow... that was a lot more than an hour.
- 2013.05.11: so, uh, sorry about breaking networking on wilson
- 2013.05.11: dhcp currently (kindof) broken, will fix after sleep
- 2013.05.05: An error in judgement
- 2013.05.03: network outage at 4:00AM pst
- 2013.05.02: and council today
- 2013.05.02: rehnquist is back, on fresh hardware.
- 2013.05.01: serious issues on rehnquist
- 2013.04.20: lozenges is down
- 2013.04.15: Update: New (direct allocation) IP addresses
- 2013.03.19: Attention old customers: New (direct allocation) IP addresses.
- 2013.03.17: update on pillar
- 2013.03.17: pillar hang/crash
- 2013.03.06: cauldron crashed
- 2013.02.25: About the new data center setup
- 2013.02.25: About the new datacentere setup
- 2013.02.17: Rehnquist is back up
- 2013.02.17: further breakage on rehnquist
- 2013.02.14: Taft rebooted itself
- 2013.02.10: Mailing Lists
- 2013.02.10: aand I should probably take those Cisco classes nick said I should take
- 2013.01.04: Waite unplanned reboot/kernel upgrade
- 2013.01.01: looks like that point to point link is down again
- 2012.12.13: hang/unclean reboot of coat
- 2012.12.11: extended network outage for 71.19.154/24 due to me being a tired idiot
- 2012.12.11: Network outage due to convergence of poor traffic engineering and point to point link outage
- 2012.12.09: one month credit for users effected by cerberus outage applied
- 2012.12.07: the IPv6 crash. Again. boutros
- 2012.11.26: Final(?) status update for boutros
- 2012.11.24: Update on the very serious problems with guests on boutrous.prgmr.com
- 2012.11.22: hamper rebooted
- 2012.11.22: I chose... poorly
- 2012.11.18: Peth0: too many iterations (6) in nv nic irq rx.
- 2012.11.12: that was a little frightening
- 2012.11.11: some lessons need to be learned twice
- 2012.11.08: moving the last prgmr.com servers out of Fremont
- 2012.10.31: pillar, tooth, kvm are moving out of Fremont
- 2012.10.31: packet loss at our fremont location
- 2012.10.31: moving servers pillar, tooth, stables, birds, and kvm out of Fremont
- 2012.10.30: Unplanned downtime on horn
- 2012.10.29: So it Begins (downtime for ingot, coral, scepter, stone, and table, moving out of he.net fremont to 55 s. market)
- 2012.10.28: replaced bad disks in bowl and rehnquist
- 2012.10.27: moving servers out of Fremont on Monday
- 2012.10.24: as I warned, hamper is moving tonight
- 2012.10.15: replaced bad disk in ingot.prgmr.com
- 2012.10.13: recent packet loss.
- 2012.10.12: For most of you, network will go down hard for 10 minutes at 21:00 PST
- 2012.10.07: an experiment in support.
- 2012.10.01: I forgot to set the rtc on taney. ugh
- 2012.10.01: report on marshall (yesterday's crash)
- 2012.09.30: an inconvienent page
- 2012.09.28: for want of a cr2032 (taney down)
- 2012.09.27: Scheduled Network Downtime on Monday night at 2300 PST
- 2012.09.14: network maintenance affecting he.net connection
- 2012.09.10: cauldron is oom-killing itself
- 2012.08.22: another IPv6/multicast related crash; hughes.
- 2012.08.16: latency to hosts in sacramento
- 2012.08.02: our stuff at he.net fremont down again
- 2012.07.22: lozenges crash
- 2012.07.22: double disk failure on crock
- 2012.07.21: network problem at rippleweb
- 2012.07.20: network downtime
- 2012.07.12: Distros updated
- 2012.07.05: replacing disks on hydra, crock, and fuller NOTE: reboot and kernel upgrade of Crock on Friday at 20:00 PST
- 2012.07.03: mysterious taney hang.
- 2012.06.29: rebuilding disk on chase
- 2012.06.29: IPv6 problem on black solved, gladwynn crash
- 2012.06.21: downtime on table.
- 2012.06.07: crock crash.
- 2012.06.07: reboot crock to fix ipv6
- 2012.06.07: routing change for mpt users
- 2012.05.30: major network outage today.
- 2012.05.24: New distros
- 2012.05.07: hung disks on fuller
- 2012.05.07: crock crashing
- 2012.05.07: unclean reboot of crock
- 2012.05.01: rebuilding disks on fuller, marshall, and bowl
- 2012.04.28: New IPs for some dom0's and ipv6 for dom0's
- 2012.04.18: Disk slowness on whetstone
- 2012.04.09: update on rehnquist
- 2012.04.05: replacing the sata card in rhenquist now
- 2012.04.05: rhenquist rebooted again
- 2012.04.04: rehnquist crash.
- 2012.04.01: network maintainince Sunday evening
- 2012.03.26: error cancelling users
- 2012.03.25: soft lockup on knife
- 2012.02.29: coins crash last night
- 2012.02.12: New update on prgmr.com servers
- 2012.02.02: mares down again.
- 2012.01.30: mares crashed
- 2012.01.28: rebuilding disk on horn
- 2012.01.20: stone reboot again
- 2012.01.17: DDoS attack
- 2012.01.16: branch rebooted.
- 2012.01.08: another bad drive in hydra
- 2012.01.01: hydra.prgmr.com is down
- 2011.12.27: taney ethernet going up and down
- 2011.12.22: stone down hard
- 2011.12.17: stone hung this morning, looks similar to the other two hangs
- 2011.12.16: lozenges down, same problem as Rutalidge last night
- 2011.12.14: rutledge raid rebuild
- 2011.12.11: disk replacement on birds
- 2011.10.26: replacing disk on knife. likely will require a reboot. - no reboot required!
- 2011.10.24: bad disk on marshall
- 2011.10.22: coral reboot
- 2011.10.20: replaced disks on jewel and bowl. No reboot
- 2011.10.02: unplanned reboot of birds.prgmr.com
- 2011.10.02: crock soft lockup. again.
- 2011.09.29: network issues at he.net fremont 1
- 2011.09.24: debian mirror
- 2011.09.22: soft lockup on crock
- 2011.09.21: changing to the new router on Friday night
- 2011.09.15: crock soft lockup.
- 2011.09.14: bull is down again after the move
- 2011.09.12: Robe, Mares and Bull moving tonight
- 2011.09.11: moving halter, horn, chariot, seashell and sword.
- 2011.09.02: clarification to privacy policy
- 2011.08.31: BUG: soft lockup detected on CPU#0!
- 2011.08.21: Minor clarifications to the AUP
- 2011.08.07: 'power event' at he.net
- 2011.08.06: Jewel is down again.
- 2011.08.05: Hardware issues with jewel
- 2011.07.31: crock hung; rebooted
- 2011.07.30: replacing drive on coins, no reboot
- 2011.07.30: replacing drive on hydra, no reboot
- 2011.07.07: I replaced a drive in coins; expect degraded performance.
- 2011.06.25: crock hung approx. 12 hours ago
- 2011.06.25: crock hung approx. 12 hours ago
- 2011.06.22: server move tonight
- 2011.06.22: lozenges crash
- 2011.06.22: network outage this morning
- 2011.06.20: server move on Tuesday and Wednesday night
- 2011.06.14: rebooting halter to replace disks
- 2011.06.01: cauldron crash
- 2011.05.31: mantle crashed
- 2011.05.31: ipv6 routing move
- 2011.05.26: bug: soft lockup on cauldron
- 2011.05.26: network problems at svtix
- 2011.05.19: DHCP is back
- 2011.05.17: replacing a bad drive on coat; no downtime expected
- 2011.05.16: packet loss at SVTIX
- 2011.05.15: move to MPT, day 2
- 2011.05.14: It has begun
- 2011.05.13: I am replacing Table tonight
- 2011.05.08: table outage
- 2011.05.06: power outage at he.net, network outage at SVTIX
- 2011.04.14: moving servers to market post tower
- 2011.04.13: unplanned network outage at svtix
- 2011.04.11: power outage at herakles (sacramento)
- 2011.03.13: poorly-planed reboot of girdle and cattle
- 2011.03.09: crock reboot again
- 2011.02.27: crock reboot
- 2011.02.21: knife reboot
- 2011.02.18: network outage at SVTIX again
- 2011.02.16: hamper reboot
- 2011.02.14: rebuilding RAID on ingot, no downtime.
- 2011.02.11: knife hung hard.
- 2011.02.04: network outage at SVTIX last night
- 2011.01.29: replacing a bad disk in knife
- 2011.01.27: migration off boar
- 2011.01.24: nameserver 65.19.174.2 appears to be down (it's back now)
- 2011.01.18: hydra shutdown and boar reboot
- 2011.01.14: we are shutting down new orders to focus on fixing hydra
- 2011.01.05: council network outage
- 2011.01.02: mares ethernet driver
- 2010.12.15: rebooting hamper to add new disks
- 2010.12.13: replaced disk on Pearl, expect poor performance for the next 10-16 hours
- 2010.12.13: rebooting branch to add new disks
- 2010.11.24: support email has been down since the last outage.
- 2010.11.23: another power outage at he.net
- 2010.11.21: power outage at he.net
- 2010.11.17: ipv6 routing, bgp and quagga
- 2010.11.08: Billing system woes
- 2010.10.29: dish hung, is coming back up now
- 2010.09.28: 22 guests rebooted on Mantle
- 2010.09.15: a discussion on the SLA
- 2010.09.14: svtix upstream outage
- 2010.09.12: NetBSD with Xen 4.0 (and Gentoo!)
- 2010.09.08: knife reboot
- 2010.08.20: crock reboot again
- 2010.08.18: rebooting cattle and girdle
- 2010.08.11: unclean reboot of horn and chariot
- 2010.08.01: possible dish disk failure
- 2010.08.01: BUG: soft lockup on dish, rebooted
- 2010.07.26: whetstone reboot
- 2010.07.17: rebuilding a disk on branch, expect some slowness
- 2010.07.16: problem on stables... up domains are OK, down domains stay down.
- 2010.07.06: dish reboot
- 2010.07.04: knife froze up hard and was rebooted
- 2010.06.28: hamper to be rebooted shortly
- 2010.06.25: Please comment on the first draft of the prgmr.com privacy policy
- 2010.06.24: roadblock in getting an ARIN allocation
- 2010.06.21: I messed up hamper when I was at the co-lo
- 2010.06.21: DoS in the he.net fremont location
- 2010.06.21: robe network driver
- 2010.06.20: network problems at SVTIX
- 2010.06.17: sorry about the early past-due notices
- 2010.06.15: on logging serial consoles.
- 2010.05.29: crock reboot again
- 2010.05.23: Knife reboot
- 2010.05.20: ungraceful reboot of crock
- 2010.05.07: Hamper threw a disk, expect IO to suffer
- 2010.04.30: becoming 'business grade' - also, dealing with abuse
- 2010.04.11: Network interuption in the Freemont data center
- 2010.03.28: knife reboot
- 2010.03.16: ~1hr network outage on branch due to forcedeth funnyness
- 2010.03.14: forward looking statements
- 2010.03.09: Horn (and the VMs on it) will be rebooted shortly
- 2010.03.09: Horn (and the VMs on it) will be rebooted shortly
- 2010.02.24: the next server will be limited to 512M accounts and below
- 2010.02.22: bad disk on robe; expect perormance problems
- 2010.02.14: (short) network outage at SVTIX, midnight, 2010-02-13
- 2010.02.10: Chris and I will be speaking at the local ACCU meeting tonight.
- 2010.02.07: sloooow disk I/O on horn due to bad disk
- 2010.01.31: IPv6 router upgrade at SVTIX
- 2010.01.03: partial network outage last night
- 2009.12.30: stables and birds going down for update and move
- 2009.12.30: hydra rebooting shortly
- 2009.12.30: boar.prgmr.com going down shortly
- 2009.12.29: (short) downtime on hydra, stables, birds and boar - tomorow
- 2009.12.29: ~20 min network outage today
- 2009.12.29: lion rebooting for kernel refresh and a move
- 2009.12.09: prgmr.com network outage tonight
- 2009.12.02: rebooting cerberus to replace a bad drive. Ugh. never again sata_nv
- 2009.11.23: IPv6 RA blocked in Fremont
- 2009.11.05: do pci-express post diagnostic cards exist?
- 2009.10.18: Xen domains not starting up.
- 2009.10.14: emergency network maintainince (also, upgrading to gigabit)
- 2009.09.11: provisioning delay
- 2009.08.28: Birds will be rebooted on Monday at 4pm PST
- 2009.08.24: I/O hang and unplanned (unclean) reboot of girdle
- 2009.08.05: birds back up after unplanned downtime.
- 2009.08.04: problems with billing (or, this is why I send invoices rather than direct billing)
- 2009.08.04: IPv6 router advertisements are down at the old he.net location
- 2009.08.03: all down users have been assigned new IPs
- 2009.08.01: hm. the DoS looks like a Syn flood.
- 2009.07.31: outage for customers on the 74.113.30/24 block
- 2009.06.23: apples, the new server, is up.
- 2009.06.13: rebuilding disk on stables
- 2009.06.07: see, this is why I don't assume you are stupid.
- 2009.05.30: adding users on the new servers in the rippleweb location.
- 2009.05.23: 2 new servers will be up at the rippleweb location soon
- 2009.05.16: we will have new hardware up soon
- 2009.05.09: new (old) prgmr.com SysAdmin
- 2009.05.08: monitoring latency/availability of bull.prgmr.com
- 2009.05.07: problem logging in to bull.prgmr.com fixed
- 2009.05.07: New support person starting tomorow
- 2009.05.07: provisioning progress
- 2009.05.05: double billing
- 2009.05.05: Current servers full
- 2009.04.25: some sort of networking problem at the freemont location
- 2009.04.22: billing issues
- 2009.03.22: problems on stables- oom killer errors
- 2009.03.14: the necessity of Kool-aid
- 2009.03.14: Charity and advertising
- 2009.03.07: how to stop your firewall from timing out your ssh session
- 2009.03.01: Co-location in San Jose: My plan.
- 2009.02.23: I broke dhcpd
- 2009.02.01: how to manually authenticate a mediawiki user email (also authentication for book.xen.prgmr.com now works)
- 2008.12.16: network outage of 1 hour
- 2008.11.17: Be fearful when others are greedy and greedy when others are fearful
- 2008.11.15: never again
- 2008.11.10: downtime and reboot for users of Hydra this friday night
- 2008.11.09: Maintainince planned on Friday the 14th of November
- 2008.10.29: SSD vs. SAS vs. SATA
- 2008.10.13: how to reset a MandyLion brand password token to factory defaults
- 2008.10.03: GNU screen for collaboration
- 2008.09.30: issuing credits tonight for Sept. downtime
- 2008.09.30: pre-payment discounts
- 2008.09.30: scheduled downtime. 30 mins. starting 20081003 at 22:00 PST
- 2008.09.21: rdns for 216.218.210.65/27 fixed
- 2008.09.20: has anyone experience with SVTIX?
- 2008.09.13: the NetBSD images are no longer a lie, sortof.
- 2008.09.06: rebooting hydra and lion
- 2008.09.05: more unplanned downtime.
- 2008.08.17: Coloma reboot.
- 2008.07.31: IPv6 RDNS setup
- 2008.07.29: unplanned reboot of coloma
- 2008.07.08: IPv6 by accident.
- 2008.07.01: http.xen.prgmr.com and wiki.xen.prgmr.com moved off of tahoe
- 2008.06.28: drive performance problem solved
- 2008.06.15: new server status
- 2008.06.12: New server will be up Thursday evening
- 2008.06.11: traffic shaping: yet again.
- 2008.06.10: traffic shaping: round 2
- 2008.06.09: lvm mirroring is supported and widely used.
- 2008.06.06: traffic shaping with tc and xen.
- 2008.06.03: New rdns policy
- 2008.06.02: price cut- for customers willing to move to the new server
- 2008.06.02: Snort IDS installed.
- 2008.05.28: freeSide billing up- Accepting new signups now.
- 2008.05.17: NTP on Coloma (and in general)
- 2008.05.13: Debian users: change your OpenSSH public key files
- 2008.05.13: in which luke is very clever.
- 2008.05.05: intermitent network problems 2008-05-05 8:00-11:00 PST
- 2008.04.30: only insane people would want to fly.
- 2008.04.26: problems with paypal subscriptions
- 2008.04.26: mirroring LVM tested
- 2008.04.12: redundant array of crappy disks.
- 2008.04.11: the hardware of prgmr.com.
- 2008.04.11: Data recovery: frozen peas aren't only good for skinned knees
- 2008.04.02: bulk e-mail
- 2008.03.30: xenconsole: Could not read tty from store: No such file or directory
- 2008.03.22: payment methods
- 2008.03.22: debian kernels working
- 2008.03.22: Ubuntu images
- 2008.03.02: Use strong passwords
- 2008.03.02: movable type is up and working
Monthly Archives
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
- December 2008
- November 2008
- October 2008
- September 2008
- August 2008
- July 2008
- June 2008
- May 2008
- April 2008
- March 2008
Categories
Category Monthly Archives
- abuse: April 2010
- abuse: June 2008
- abuse: April 2008
- billing: August 2009
- billing: May 2009
- billing: April 2009
- billing: May 2008
- billing: April 2008
- Business: June 2010
- Business: April 2010
- Business: May 2009
- Business: March 2009
- hardware: January 2011
- hardware: August 2010
- hardware: December 2009
- hardware: November 2009
- hosting status: June 2011
- hosting status: May 2011
- hosting status: December 2009
- hosting status: October 2009
- hosting status: August 2009
- hosting status: June 2009
- hosting status: May 2009
- hosting status: April 2009
- hosting status: November 2008
- hosting status: September 2008
- hosting status: August 2008
- hosting status: July 2008
- hosting status: June 2008
- hosting status: May 2008
- hosting status: April 2008
- hosting status: March 2008
- ipv6: May 2011
- new features: July 2012
- new features: September 2011
- new features: June 2010
- new features: November 2008
- new features: September 2008
- new features: July 2008
- new features: June 2008
- outage: September 2011
- outage: June 2011
- outage: May 2011
- outage: April 2011
- outage: February 2011
- outage: January 2011
- outage: November 2010
- outage: September 2010
- outage: July 2010
- outage: June 2010
- outage: May 2010
- outage: March 2010
- outage: December 2009
- outage: November 2009
- outage: October 2009
- outage: August 2009
- outage: April 2009
- outage: March 2009
- outage: December 2008
- outage: November 2008
- outage: September 2008
- outage: August 2008
- outage: July 2008
- outage: June 2008
- outage: May 2008
- security: June 2010
- security: October 2008
- tech support: March 2009
- todo: June 2008
- troubleshooting: October 2009
- troubleshooting: October 2008
- troubleshooting: May 2008
- troubleshooting: March 2008
