Skip to content

Latest commit

 

History

History
15 lines (11 loc) · 657 Bytes

2021-02-16-ci-and-device-manager.md

File metadata and controls

15 lines (11 loc) · 657 Bytes

CI and Device Management services offline

Timeline of Events

  • 16:30 UTC - Notified of container publishing failure
  • 17:00 UTC - Issue traced back to internal database
  • 17:40 UTC - TUF-repo service offline
  • 18:30 UTC - Internal database restored
  • 18:50 UTC - All services restored

What Happened

Internal database maintenance caused the cluster size to be reduced. Because of this the database ran out of storage space and started to refuse connections from customer facing applications. The database was restarted, checked for integrity, and brought back online. Other services were enabled and connected properly.