Our database supplier, Microsoft Azure, has experienced severe operational issues this morning and until now. These issues should now be resolved and Bex is back to working as normal.
The source of the error from Microsoft and information recived:
"Current Status: SQL team has identified a configuration change on the metadata drop operation which has caused the overall issue. We have currently several parallel workstreams to revert the change to mitigate the issue. A number of customers should already see recovery and the number of recovered instances will increase progressively until full recovery."
We apologise for the inconvenience and have requested a full post-mortem from Microsoft.
Posted Jul 21, 2022 - 13:44 CEST
Update
Microsoft Current Status: We are currently running several parallel workstreams to revert the change to mitigate the issue and we are continuing to monitor progress. Impacted customers should start to see recovery as the applied mitigation fully propagates. A further update will be provided in 60 minutes, or as events warrant.
Posted Jul 21, 2022 - 12:45 CEST
Update
SQL team has identified a configuration change on the metadata drop operation which has caused the overall issue. We have currently several parallel workstreams to revert the change to mitigate the issue. A number of customers should already see recovery and the number of recovered instances will increase progressively until full recovery. The next update will be provided in 60 minutes, or as events warrant.
Posted Jul 21, 2022 - 11:35 CEST
Identified
Current Status: We have initiated a mitigation workstream and some customers should start seeing recovery. The next update will be provided in 60 minutes, or as events warrant.
Posted Jul 21, 2022 - 08:57 CEST
Investigating
Impact Statement: Starting at 03:58 UTC on 21 Jul 2022, you have been identified as a customer using SQL Database in West Europe who may experience issues accessing services. New connections to databases in this region may result in an error or timeout. Existing connections remain available to accept new requests, however if those connections are terminated then re-established, they may fail. Current Status: We are continuing our investigation to get the preliminary root cause and exploring options for mitigation. The next update will be provided in 60 minutes, or as events warrant.
Impact Statement: Starting at 03:58 UTC on 21 Jul 2022, you have been identified as a customer using SQL Database in West Europe who may experience issues accessing services. New connections to databases in this region may result in an error or timeout. Existing connections remain available to accept new requests, however if those connections are terminated then re-established, they may fail. Current Status: We are aware of this issue and are actively investigating. The next update will be provided within 60 minutes.
Posted Jul 21, 2022 - 08:09 CEST
This incident affected: Bex (Api & Integrations, Bex Backoffice).