10:30 AM
We are experiencing a Data Center Outage that is affecting our services. This is a systemic outage at the facility.
Bria Solo, Teams, and Enterprise are affected and may see intermittent issues with login and hosted services (messaging, etc).
We are actively monitoring and will restore service when the Data Center is back online.
11:00 AM
We have been in close contact with the ISP and have now received material updates. After dispatching field technicians to the site they have identified the issue with their networking equipment that is causing the network instability. They are replacing this equipment but this will take them some time. We do not have a firm time to restore but from what we are told we are 1-2 hours away from resolution.
In parallel are working to bring up a redundant site to restore stability to Stretto Provisioning for affected customers.
12:00 PM
We have signs of a recovery. One of our connections has resumed and has been stable for more than 10 minutes for a few periods of time. At this time all alarms are clear and all services are available.
Our provider has told us that we may experience 1-2 more cycling of the connection within the next 15 minutes as they work to complete their recovery for the first system affecting our first link.
We are continuing to monitor the network status and are working to complete end-to-end testing of all services while they are up and stable.
We continue to work to activate a redundant instance of Stretto Provisioning at an alternate server.
Final Update:
Our provider has now stabilized our redundant connection. Based on preliminary information, the delay in the restoration of this connection was related to an additional issue with replacement hardware. We understand that a replacement line card was also faulty.
All CounterPath systems continue to be operational. All services are up and reachable. All monitoring and alarms have remained clear.
We are continuing work related to automated and manual checks of services and infrastructure.
We will provide a Root Cause Analysis (RCA) document to customers early in the coming week. We can not yet offer clarity on the timeline for document availability because we cannot finalize our RCA document without receiving our provider's detailed reports.
We do not expect to provide another update today.