Performance Degradation

Subject: Configuration and Secrets Storage (CSS)  Downtime Incident – Resolved and Services Restored

Summary:Over the past few days, an unexpected increase in the number of authentication requests led to an overload of our internal CSS service. This resulted in a temporary service degradation that affected systems relying on dynamic secrets. After identifying the root cause, we performed a controlled restoration using our backup mechanisms.

Timeline:

• The issue began building up over several days due to high lease generation.
• CSS experienced failure during a backup procedure due to resource pressure.
• A new CSS instance was deployed and restored from backup.
• All affected services are now healthy and operational.
  

Resolution Status:

• Incident resolved
• CSS restored from the recent backup
• All affected services have recovered

Next Steps:

• Alerting improvements already in place.
• Preventative measures rolled out to limit overuse and safeguard core services.
  

We apologize for any inconvenience this may have caused and appreciate your patience.