Incidents | Priority 1 IT Incidents reported on status page for Priority 1 IT https://status.p1it.com.au/ https://d1lppblt9t2x15.cloudfront.net/logos/aa4fe6587d050164d66f78fdbccdc56a.png Incidents | Priority 1 IT https://status.p1it.com.au/ en Datacentre Connectivity Interruption https://status.p1it.com.au/incident/565340 Sat, 31 May 2025 23:00:08 +0000 https://status.p1it.com.au/incident/565340#9b7feb5bf0546c2a59ae748b6ee16bfcf8095a39bf4a1cdcecc502278f682e43 Maintenance completed Datacentre Connectivity Interruption https://status.p1it.com.au/incident/565340 Sat, 31 May 2025 23:00:08 -0000 https://status.p1it.com.au/incident/565340#4be401a56dbe87afa46410d1a4d6166cf0b60e2d2bac6f954411bc157e0e32ac Network upgrades will occur on our data centre infrastructure to increase stability and performance. During this maintenance window, users will be unable to connect to hosted services for up to 2 hours. Datacentre Connectivity Interruption https://status.p1it.com.au/incident/565340 Sat, 31 May 2025 23:00:08 +0000 https://status.p1it.com.au/incident/565340#9b7feb5bf0546c2a59ae748b6ee16bfcf8095a39bf4a1cdcecc502278f682e43 Maintenance completed Datacentre Connectivity Interruption https://status.p1it.com.au/incident/565340 Sat, 31 May 2025 23:00:08 -0000 https://status.p1it.com.au/incident/565340#4be401a56dbe87afa46410d1a4d6166cf0b60e2d2bac6f954411bc157e0e32ac Network upgrades will occur on our data centre infrastructure to increase stability and performance. During this maintenance window, users will be unable to connect to hosted services for up to 2 hours. Datacentre Connectivity Interruption https://status.p1it.com.au/incident/565340 Sat, 31 May 2025 21:00:00 -0000 https://status.p1it.com.au/incident/565340#4be401a56dbe87afa46410d1a4d6166cf0b60e2d2bac6f954411bc157e0e32ac Network upgrades will occur on our data centre infrastructure to increase stability and performance. During this maintenance window, users will be unable to connect to hosted services for up to 2 hours. Datacentre Connectivity Interruption https://status.p1it.com.au/incident/565340 Sat, 31 May 2025 21:00:00 -0000 https://status.p1it.com.au/incident/565340#4be401a56dbe87afa46410d1a4d6166cf0b60e2d2bac6f954411bc157e0e32ac Network upgrades will occur on our data centre infrastructure to increase stability and performance. During this maintenance window, users will be unable to connect to hosted services for up to 2 hours. Network Connectivity Maintenance https://status.p1it.com.au/incident/553506 Mon, 12 May 2025 20:00:47 +0000 https://status.p1it.com.au/incident/553506#ca3fd1ea97ef76c2e5b4aa91bc8aec63a3d9d557c5a2aa003e4db1a734385874 Maintenance completed Network Connectivity Maintenance https://status.p1it.com.au/incident/553506 Mon, 12 May 2025 15:00:47 -0000 https://status.p1it.com.au/incident/553506#24b382dead9b87de7da4c3b8a1615ad25388e999eb3f89fa892e96a6a7aaf667 Our data centre is undergoing scheduled network connectivity maintenance affecting one of our upstream peers. Maintenance is expected to last up to 1 hour. No downtime or impact to client services is anticipated during this period. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Thu, 01 May 2025 04:16:00 -0000 https://status.p1it.com.au/incident/552738#de17be78e022864b53d6a02154cff58f9c0718f9de9c2442121fc9bb929e2d05 Incident Post-Mortem: DDoS Attack Overview We recently experienced a significant outage caused by a sophisticated Distributed Denial-of-Service (DDoS) attack targeting our upstream phone infrastructure carrier. What is a DDoS Attack? A Distributed Denial-of-Service (DDoS) attack is a malicious attempt to disrupt the normal traffic of a targeted service, network, or server by overwhelming it with excessive internet traffic from multiple sources. Incident Details Our upstream provider had multiple layers of DDoS prevention in place; however, the scale of this attack was unprecedented at approximately 1 terabit per second, overwhelming existing defenses. Throughout the incident, our team maintained constant communication with our upstream carrier, coordinating closely to mitigate the disruption. Historical Context We have relied on this upstream carrier for 13 years with an exceptionally high uptime and virtually zero major disruptions, significantly outperforming typical carrier reliability, including major providers such as Telstra and Optus. Mitigation and Improvements In response to this incident, our upstream carrier has now implemented a more sophisticated and robust DDoS mitigation solution powered by Cloudflare. Cloudflare's advanced systems are capable of handling DDoS attacks up to 350 terabits per second, providing significant protection well beyond current known threat levels. Nature of the Attack The scale, sophistication, and coordination of the DDoS attack strongly suggest it was state-sponsored. This attack was highly complex and intended to cause significant disruption. Conclusion We appreciate our customers' patience and understanding during this unprecedented incident. Our infrastructure now benefits from industry-leading defenses, significantly enhancing our ability to prevent such disruptions in the future. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Thu, 01 May 2025 04:16:00 -0000 https://status.p1it.com.au/incident/552738#de17be78e022864b53d6a02154cff58f9c0718f9de9c2442121fc9bb929e2d05 Incident Post-Mortem: DDoS Attack Overview We recently experienced a significant outage caused by a sophisticated Distributed Denial-of-Service (DDoS) attack targeting our upstream phone infrastructure carrier. What is a DDoS Attack? A Distributed Denial-of-Service (DDoS) attack is a malicious attempt to disrupt the normal traffic of a targeted service, network, or server by overwhelming it with excessive internet traffic from multiple sources. Incident Details Our upstream provider had multiple layers of DDoS prevention in place; however, the scale of this attack was unprecedented at approximately 1 terabit per second, overwhelming existing defenses. Throughout the incident, our team maintained constant communication with our upstream carrier, coordinating closely to mitigate the disruption. Historical Context We have relied on this upstream carrier for 13 years with an exceptionally high uptime and virtually zero major disruptions, significantly outperforming typical carrier reliability, including major providers such as Telstra and Optus. Mitigation and Improvements In response to this incident, our upstream carrier has now implemented a more sophisticated and robust DDoS mitigation solution powered by Cloudflare. Cloudflare's advanced systems are capable of handling DDoS attacks up to 350 terabits per second, providing significant protection well beyond current known threat levels. Nature of the Attack The scale, sophistication, and coordination of the DDoS attack strongly suggest it was state-sponsored. This attack was highly complex and intended to cause significant disruption. Conclusion We appreciate our customers' patience and understanding during this unprecedented incident. Our infrastructure now benefits from industry-leading defenses, significantly enhancing our ability to prevent such disruptions in the future. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Thu, 01 May 2025 04:16:00 -0000 https://status.p1it.com.au/incident/552738#de17be78e022864b53d6a02154cff58f9c0718f9de9c2442121fc9bb929e2d05 Incident Post-Mortem: DDoS Attack Overview We recently experienced a significant outage caused by a sophisticated Distributed Denial-of-Service (DDoS) attack targeting our upstream phone infrastructure carrier. What is a DDoS Attack? A Distributed Denial-of-Service (DDoS) attack is a malicious attempt to disrupt the normal traffic of a targeted service, network, or server by overwhelming it with excessive internet traffic from multiple sources. Incident Details Our upstream provider had multiple layers of DDoS prevention in place; however, the scale of this attack was unprecedented at approximately 1 terabit per second, overwhelming existing defenses. Throughout the incident, our team maintained constant communication with our upstream carrier, coordinating closely to mitigate the disruption. Historical Context We have relied on this upstream carrier for 13 years with an exceptionally high uptime and virtually zero major disruptions, significantly outperforming typical carrier reliability, including major providers such as Telstra and Optus. Mitigation and Improvements In response to this incident, our upstream carrier has now implemented a more sophisticated and robust DDoS mitigation solution powered by Cloudflare. Cloudflare's advanced systems are capable of handling DDoS attacks up to 350 terabits per second, providing significant protection well beyond current known threat levels. Nature of the Attack The scale, sophistication, and coordination of the DDoS attack strongly suggest it was state-sponsored. This attack was highly complex and intended to cause significant disruption. Conclusion We appreciate our customers' patience and understanding during this unprecedented incident. Our infrastructure now benefits from industry-leading defenses, significantly enhancing our ability to prevent such disruptions in the future. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Wed, 30 Apr 2025 04:18:00 -0000 https://status.p1it.com.au/incident/552738#1c6e95e06a579a1b5c542e0875314f9ab5a9ed1c6c608c0a1fb7130e4ee9256e We are observing calls and portal access returning to normal. Our upstream carrier has confirmed that the DDoS attack has ceased. Some supplementary services, including phonebook and voicemail, remain offline but are expected to be restored shortly. We will continue monitoring the situation closely and provide further updates as necessary. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Wed, 30 Apr 2025 04:18:00 -0000 https://status.p1it.com.au/incident/552738#1c6e95e06a579a1b5c542e0875314f9ab5a9ed1c6c608c0a1fb7130e4ee9256e We are observing calls and portal access returning to normal. Our upstream carrier has confirmed that the DDoS attack has ceased. Some supplementary services, including phonebook and voicemail, remain offline but are expected to be restored shortly. We will continue monitoring the situation closely and provide further updates as necessary. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Wed, 30 Apr 2025 04:18:00 -0000 https://status.p1it.com.au/incident/552738#1c6e95e06a579a1b5c542e0875314f9ab5a9ed1c6c608c0a1fb7130e4ee9256e We are observing calls and portal access returning to normal. Our upstream carrier has confirmed that the DDoS attack has ceased. Some supplementary services, including phonebook and voicemail, remain offline but are expected to be restored shortly. We will continue monitoring the situation closely and provide further updates as necessary. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Tue, 29 Apr 2025 22:45:00 -0000 https://status.p1it.com.au/incident/552738#9dc0b98a87a96043893a4d41b176803373b6c4ab7032458935d21fbd55c094e9 The ongoing DDoS attack targeting our upstream carrier has intensified, resulting in wider disruption affecting calls from off-net carriers. Phones powered by Aussie Broadband and Vonex internet connections remain largely unaffected at this time. We continue to monitor this situation closely and will provide additional updates as they become available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Tue, 29 Apr 2025 22:45:00 -0000 https://status.p1it.com.au/incident/552738#9dc0b98a87a96043893a4d41b176803373b6c4ab7032458935d21fbd55c094e9 The ongoing DDoS attack targeting our upstream carrier has intensified, resulting in wider disruption affecting calls from off-net carriers. Phones powered by Aussie Broadband and Vonex internet connections remain largely unaffected at this time. We continue to monitor this situation closely and will provide additional updates as they become available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Tue, 29 Apr 2025 22:45:00 -0000 https://status.p1it.com.au/incident/552738#9dc0b98a87a96043893a4d41b176803373b6c4ab7032458935d21fbd55c094e9 The ongoing DDoS attack targeting our upstream carrier has intensified, resulting in wider disruption affecting calls from off-net carriers. Phones powered by Aussie Broadband and Vonex internet connections remain largely unaffected at this time. We continue to monitor this situation closely and will provide additional updates as they become available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Tue, 29 Apr 2025 03:33:00 -0000 https://status.p1it.com.au/incident/552738#a682c1784f6e9a1839f36f38eecc2022513bac819a4d4ab6769441fdff5249df The DDOS attack remains prevalent for our upstream carrier. Faxing, Address Books, SMS, Voicemail and web portal access continue to be affected. Calling continues to be un-affected. We will continue to update this entry as new information becomes available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Tue, 29 Apr 2025 03:33:00 -0000 https://status.p1it.com.au/incident/552738#a682c1784f6e9a1839f36f38eecc2022513bac819a4d4ab6769441fdff5249df The DDOS attack remains prevalent for our upstream carrier. Faxing, Address Books, SMS, Voicemail and web portal access continue to be affected. Calling continues to be un-affected. We will continue to update this entry as new information becomes available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Tue, 29 Apr 2025 03:33:00 -0000 https://status.p1it.com.au/incident/552738#a682c1784f6e9a1839f36f38eecc2022513bac819a4d4ab6769441fdff5249df The DDOS attack remains prevalent for our upstream carrier. Faxing, Address Books, SMS, Voicemail and web portal access continue to be affected. Calling continues to be un-affected. We will continue to update this entry as new information becomes available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Tue, 29 Apr 2025 03:33:00 -0000 https://status.p1it.com.au/incident/552738#a682c1784f6e9a1839f36f38eecc2022513bac819a4d4ab6769441fdff5249df The DDOS attack remains prevalent for our upstream carrier. Faxing, Address Books, SMS, Voicemail and web portal access continue to be affected. Calling continues to be un-affected. We will continue to update this entry as new information becomes available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Mon, 28 Apr 2025 03:24:00 -0000 https://status.p1it.com.au/incident/552738#155061b770bbf819a42d37a6bbe846ae48300bd93861edc77583b5cda0161d35 Call services have resumed to normal operation. Issues with faxing and voicemail retrieval persist. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Mon, 28 Apr 2025 03:24:00 -0000 https://status.p1it.com.au/incident/552738#155061b770bbf819a42d37a6bbe846ae48300bd93861edc77583b5cda0161d35 Call services have resumed to normal operation. Issues with faxing and voicemail retrieval persist. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Mon, 28 Apr 2025 03:24:00 -0000 https://status.p1it.com.au/incident/552738#155061b770bbf819a42d37a6bbe846ae48300bd93861edc77583b5cda0161d35 Call services have resumed to normal operation. Issues with faxing and voicemail retrieval persist. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Mon, 28 Apr 2025 03:24:00 -0000 https://status.p1it.com.au/incident/552738#155061b770bbf819a42d37a6bbe846ae48300bd93861edc77583b5cda0161d35 Call services have resumed to normal operation. Issues with faxing and voicemail retrieval persist. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Mon, 28 Apr 2025 01:34:00 -0000 https://status.p1it.com.au/incident/552738#df614abd13e9887d5cf0ab0209510ed41c3b688027dfcc4c90080c9e748858e3 Connectivity issues have been restored for phone calls. There are still ongoing issues with voicemails and some of the web portals. We are continuing to monitor the situation. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Mon, 28 Apr 2025 01:34:00 -0000 https://status.p1it.com.au/incident/552738#df614abd13e9887d5cf0ab0209510ed41c3b688027dfcc4c90080c9e748858e3 Connectivity issues have been restored for phone calls. There are still ongoing issues with voicemails and some of the web portals. We are continuing to monitor the situation. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Mon, 28 Apr 2025 01:34:00 -0000 https://status.p1it.com.au/incident/552738#df614abd13e9887d5cf0ab0209510ed41c3b688027dfcc4c90080c9e748858e3 Connectivity issues have been restored for phone calls. There are still ongoing issues with voicemails and some of the web portals. We are continuing to monitor the situation. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Sun, 27 Apr 2025 23:03:00 -0000 https://status.p1it.com.au/incident/552738#4c5e1a4739ea697bc703e0b80f33ac441bae2d96fee8e5e211ff1034b300effb Incident Update: The outage caused by a DDoS attack on our upstream provider is ongoing. We are actively working with our carrier, who has not yet provided an ETA for resolution. Currently, voicemail services and web portals for PriorityPBX may also be impacted. We will continue to monitor closely and provide further updates as soon as more information is available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Sun, 27 Apr 2025 23:03:00 -0000 https://status.p1it.com.au/incident/552738#4c5e1a4739ea697bc703e0b80f33ac441bae2d96fee8e5e211ff1034b300effb Incident Update: The outage caused by a DDoS attack on our upstream provider is ongoing. We are actively working with our carrier, who has not yet provided an ETA for resolution. Currently, voicemail services and web portals for PriorityPBX may also be impacted. We will continue to monitor closely and provide further updates as soon as more information is available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Sun, 27 Apr 2025 23:03:00 -0000 https://status.p1it.com.au/incident/552738#4c5e1a4739ea697bc703e0b80f33ac441bae2d96fee8e5e211ff1034b300effb Incident Update: The outage caused by a DDoS attack on our upstream provider is ongoing. We are actively working with our carrier, who has not yet provided an ETA for resolution. Currently, voicemail services and web portals for PriorityPBX may also be impacted. We will continue to monitor closely and provide further updates as soon as more information is available. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Sun, 27 Apr 2025 08:00:00 -0000 https://status.p1it.com.au/incident/552738#68cbeb4c858b4752649fdc6cf52b0b152308ec98eb8abcd5486f2cb145efdec2 We're currently experiencing a significant DDoS cyberattack targeting our phone infrastructure, causing connectivity issues for a small portion of customers. Customers not using Aussie Broadband may experience disruptions. Over 95% of customers remain unaffected. We're actively monitoring this incident and will provide updates as soon as possible. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Sun, 27 Apr 2025 08:00:00 -0000 https://status.p1it.com.au/incident/552738#68cbeb4c858b4752649fdc6cf52b0b152308ec98eb8abcd5486f2cb145efdec2 We're currently experiencing a significant DDoS cyberattack targeting our phone infrastructure, causing connectivity issues for a small portion of customers. Customers not using Aussie Broadband may experience disruptions. Over 95% of customers remain unaffected. We're actively monitoring this incident and will provide updates as soon as possible. PriorityPBX Degraded Service https://status.p1it.com.au/incident/552738 Sun, 27 Apr 2025 08:00:00 -0000 https://status.p1it.com.au/incident/552738#68cbeb4c858b4752649fdc6cf52b0b152308ec98eb8abcd5486f2cb145efdec2 We're currently experiencing a significant DDoS cyberattack targeting our phone infrastructure, causing connectivity issues for a small portion of customers. Customers not using Aussie Broadband may experience disruptions. Over 95% of customers remain unaffected. We're actively monitoring this incident and will provide updates as soon as possible. Degraded Outlook performance on Office 365 Services https://status.p1it.com.au/incident/442627 Thu, 10 Oct 2024 22:00:00 -0000 https://status.p1it.com.au/incident/442627#3fdac26ed567528ec2ae25a33934c8f52974d491764195d3ee7b8c151b83766f Following an outage in Microsoft services evening of 10/10/2024, Microsoft outlook email services are suffering degraded performance. We will continue to monitor the situation and report updates as they are released. This is being tracked by Microsoft under incident MO907654 Intermittent Phone System Issues https://status.p1it.com.au/incident/372275 Tue, 21 May 2024 07:00:00 -0000 https://status.p1it.com.au/incident/372275#01b61fd4db7b92cc3add8c24e7bd8dcca4d3f6651e8f54a493e1fbb48feeb8a8 After a root cause analysis, we discovered the issue was with our upstream provider. The upstream provider suffered a router failure in their datacentre, and connections were lost for 1 minute while the redundant router accepted connections. Following the reconnection, SIP registrations took 4-7 minutes to re-negotiate and re-connect. Thanks for your patience and understanding. Intermittent Phone System Issues https://status.p1it.com.au/incident/372275 Tue, 21 May 2024 07:00:00 -0000 https://status.p1it.com.au/incident/372275#01b61fd4db7b92cc3add8c24e7bd8dcca4d3f6651e8f54a493e1fbb48feeb8a8 After a root cause analysis, we discovered the issue was with our upstream provider. The upstream provider suffered a router failure in their datacentre, and connections were lost for 1 minute while the redundant router accepted connections. Following the reconnection, SIP registrations took 4-7 minutes to re-negotiate and re-connect. Thanks for your patience and understanding. Intermittent Phone System Issues https://status.p1it.com.au/incident/372275 Tue, 21 May 2024 07:00:00 -0000 https://status.p1it.com.au/incident/372275#01b61fd4db7b92cc3add8c24e7bd8dcca4d3f6651e8f54a493e1fbb48feeb8a8 After a root cause analysis, we discovered the issue was with our upstream provider. The upstream provider suffered a router failure in their datacentre, and connections were lost for 1 minute while the redundant router accepted connections. Following the reconnection, SIP registrations took 4-7 minutes to re-negotiate and re-connect. Thanks for your patience and understanding. Intermittent Phone System Issues https://status.p1it.com.au/incident/372275 Mon, 20 May 2024 22:45:00 -0000 https://status.p1it.com.au/incident/372275#27c9ea444d60200d902b891bf5bb0cf4cf384d50a05503fad1dd105aa46cebda We are seeing phone system issues, which began at around 8:35 am this morning. We are seeing that services are restoring now 8:46am, and we are still chasing an update from our upstream carrier as to the cause of the outage. Intermittent Phone System Issues https://status.p1it.com.au/incident/372275 Mon, 20 May 2024 22:45:00 -0000 https://status.p1it.com.au/incident/372275#27c9ea444d60200d902b891bf5bb0cf4cf384d50a05503fad1dd105aa46cebda We are seeing phone system issues, which began at around 8:35 am this morning. We are seeing that services are restoring now 8:46am, and we are still chasing an update from our upstream carrier as to the cause of the outage. Intermittent Phone System Issues https://status.p1it.com.au/incident/372275 Mon, 20 May 2024 22:45:00 -0000 https://status.p1it.com.au/incident/372275#27c9ea444d60200d902b891bf5bb0cf4cf384d50a05503fad1dd105aa46cebda We are seeing phone system issues, which began at around 8:35 am this morning. We are seeing that services are restoring now 8:46am, and we are still chasing an update from our upstream carrier as to the cause of the outage. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 22:24:00 -0000 https://status.p1it.com.au/incident/356993#eab0e3a6f996cfebc9ef114fa9b5b402f7e65d94d833c680f2fae2314d20f802 Outage Postmortem Report Incident Overview: Today, our data center faced an unexpected issue when several upstream transit carriers failed simultaneously. As a result, there was a service disruption making the data center inaccessible from 7:32 AM to 7:47 AM, totaling approximately 15 minutes of downtime. Root Cause: Although our network design includes multiple redundant carriers to prevent such incidents, the simultaneous failure of these carriers led to this unusual outage. Preliminary investigations suggest there may be an issue with the Border Gateway Protocol (BGP) configuration, which did not behave as expected under these circumstances. Current Actions: We are conducting a thorough review of our BGP configurations and failover procedures. We aim to identify and rectify any vulnerabilities that contributed to this incident. This review will help ensure that similar failures of multiple carriers will not result in future outages. Apology and Commitment: We sincerely apologize for any inconvenience caused by this outage. We understand the critical importance of continuous service and are committed to enhancing our systems to uphold and improve the reliability you expect from us. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 22:24:00 -0000 https://status.p1it.com.au/incident/356993#eab0e3a6f996cfebc9ef114fa9b5b402f7e65d94d833c680f2fae2314d20f802 Outage Postmortem Report Incident Overview: Today, our data center faced an unexpected issue when several upstream transit carriers failed simultaneously. As a result, there was a service disruption making the data center inaccessible from 7:32 AM to 7:47 AM, totaling approximately 15 minutes of downtime. Root Cause: Although our network design includes multiple redundant carriers to prevent such incidents, the simultaneous failure of these carriers led to this unusual outage. Preliminary investigations suggest there may be an issue with the Border Gateway Protocol (BGP) configuration, which did not behave as expected under these circumstances. Current Actions: We are conducting a thorough review of our BGP configurations and failover procedures. We aim to identify and rectify any vulnerabilities that contributed to this incident. This review will help ensure that similar failures of multiple carriers will not result in future outages. Apology and Commitment: We sincerely apologize for any inconvenience caused by this outage. We understand the critical importance of continuous service and are committed to enhancing our systems to uphold and improve the reliability you expect from us. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 22:24:00 -0000 https://status.p1it.com.au/incident/356993#eab0e3a6f996cfebc9ef114fa9b5b402f7e65d94d833c680f2fae2314d20f802 Outage Postmortem Report Incident Overview: Today, our data center faced an unexpected issue when several upstream transit carriers failed simultaneously. As a result, there was a service disruption making the data center inaccessible from 7:32 AM to 7:47 AM, totaling approximately 15 minutes of downtime. Root Cause: Although our network design includes multiple redundant carriers to prevent such incidents, the simultaneous failure of these carriers led to this unusual outage. Preliminary investigations suggest there may be an issue with the Border Gateway Protocol (BGP) configuration, which did not behave as expected under these circumstances. Current Actions: We are conducting a thorough review of our BGP configurations and failover procedures. We aim to identify and rectify any vulnerabilities that contributed to this incident. This review will help ensure that similar failures of multiple carriers will not result in future outages. Apology and Commitment: We sincerely apologize for any inconvenience caused by this outage. We understand the critical importance of continuous service and are committed to enhancing our systems to uphold and improve the reliability you expect from us. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 22:24:00 -0000 https://status.p1it.com.au/incident/356993#eab0e3a6f996cfebc9ef114fa9b5b402f7e65d94d833c680f2fae2314d20f802 Outage Postmortem Report Incident Overview: Today, our data center faced an unexpected issue when several upstream transit carriers failed simultaneously. As a result, there was a service disruption making the data center inaccessible from 7:32 AM to 7:47 AM, totaling approximately 15 minutes of downtime. Root Cause: Although our network design includes multiple redundant carriers to prevent such incidents, the simultaneous failure of these carriers led to this unusual outage. Preliminary investigations suggest there may be an issue with the Border Gateway Protocol (BGP) configuration, which did not behave as expected under these circumstances. Current Actions: We are conducting a thorough review of our BGP configurations and failover procedures. We aim to identify and rectify any vulnerabilities that contributed to this incident. This review will help ensure that similar failures of multiple carriers will not result in future outages. Apology and Commitment: We sincerely apologize for any inconvenience caused by this outage. We understand the critical importance of continuous service and are committed to enhancing our systems to uphold and improve the reliability you expect from us. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:50:00 -0000 https://status.p1it.com.au/incident/356993#0852325368c29bb2da05bbee27a371c8eb07a3e57e42abc1eea47d3742e9c227 Services were restored around 7:50 AM. The root cause analysis is still in progress. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:50:00 -0000 https://status.p1it.com.au/incident/356993#0852325368c29bb2da05bbee27a371c8eb07a3e57e42abc1eea47d3742e9c227 Services were restored around 7:50 AM. The root cause analysis is still in progress. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:50:00 -0000 https://status.p1it.com.au/incident/356993#0852325368c29bb2da05bbee27a371c8eb07a3e57e42abc1eea47d3742e9c227 Services were restored around 7:50 AM. The root cause analysis is still in progress. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:50:00 -0000 https://status.p1it.com.au/incident/356993#0852325368c29bb2da05bbee27a371c8eb07a3e57e42abc1eea47d3742e9c227 Services were restored around 7:50 AM. The root cause analysis is still in progress. Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:42:00 -0000 https://status.p1it.com.au/incident/356993#fe5fe73170b982f75964d804cf62aae973ab6ae19f833f3031efb689751f9e2e We are investigating a current outage with engineers onsite and will update this page shortly Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:42:00 -0000 https://status.p1it.com.au/incident/356993#fe5fe73170b982f75964d804cf62aae973ab6ae19f833f3031efb689751f9e2e We are investigating a current outage with engineers onsite and will update this page shortly Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:42:00 -0000 https://status.p1it.com.au/incident/356993#fe5fe73170b982f75964d804cf62aae973ab6ae19f833f3031efb689751f9e2e We are investigating a current outage with engineers onsite and will update this page shortly Current Datacentre Outage https://status.p1it.com.au/incident/356993 Wed, 17 Apr 2024 21:42:00 -0000 https://status.p1it.com.au/incident/356993#fe5fe73170b982f75964d804cf62aae973ab6ae19f833f3031efb689751f9e2e We are investigating a current outage with engineers onsite and will update this page shortly