Dallas Network Problem – 6 January

The Dallas data centre experienced two losses of network connectivity this morning.

Original report 4:36 UTC: Loss of network connection.

Update: 5:02 UTC: Received confirmation from data centre that they are investigating.

Update: 5:06 UTC: Network connectivity restored; the outage lasted 30 minutes. Report from the data centre is problem was caused by scheduled network maintenance (border router updates), but there was a slip-up with the communication about this beforehand.

Update 5:22 UTC: Down again. Received confirmation that work is ongoing.

Update 5:42 UTC: Network connectivity restored; the outage lasted 30 minutes. We have no further reports from the data centre, but hope that the work is not completed.

Post-mortem report from data centre: The maintenance window has concluded. We’ll be working with the Datacenter team to ensure work like this can be completed without affecting our connectivity. Apparently it should’ve worked that way but something was misconfigured somewhere which is how we ended up offline. We’re coordinating the networking teams to figure this out ASAP. Our apologies for the inconvenience.