Dashboards unavailable
Affected components
- Application
Updates
RESOLVED
Resolved. The expired credential was rotated and the service restarted at 06:53 UTC on 30 August, and dashboard widgets began loading normally again immediately. We have confirmed no further authentication failures since that time. This incident is being recorded after the fact, so the timestamp on this update reflects publication rather than recovery.
INVESTIGATING
Between 29 August 16:25 UTC and 30 August 06:53 UTC, dashboard widgets failed to load and displayed "Something went wrong. Please refresh". Other areas of the product were unaffected.
The cause was an expired credential used to reach our analytics data store. Dashboard queries could not authenticate, so every widget returned an error. No customer data was lost, altered, or exposed at any point.
The credential was rotated and the service restarted at 06:53 UTC, and dashboards recovered immediately. We are adding expiry monitoring on these credentials so that this cannot recur without warning.