Outage Postmortem Report

Incident Overview:
Today, our data center faced an unexpected issue when several upstream transit carriers failed simultaneously. As a result, there was a service disruption making the data center inaccessible from 7:32 AM to 7:47 AM, totaling approximately 15 minutes of downtime.

Root Cause:
Although our network design includes multiple redundant carriers to prevent such incidents, the simultaneous failure of these carriers led to this unusual outage. Preliminary investigations suggest there may be an issue with the Border Gateway Protocol (BGP) configuration, which did not behave as expected under these circumstances.

Current Actions:
We are conducting a thorough review of our BGP configurations and failover procedures. We aim to identify and rectify any vulnerabilities that contributed to this incident. This review will help ensure that similar failures of multiple carriers will not result in future outages.

Apology and Commitment:
We sincerely apologize for any inconvenience caused by this outage. We understand the critical importance of continuous service and are committed to enhancing our systems to uphold and improve the reliability you expect from us.

Services were restored around 7:50 AM. The root cause analysis is still in progress.

We are investigating a current outage with engineers onsite and will update this page shortly