Depot - Notice history

We are seeing full connectivity restored, builds are processing normally.

We are seeing database connectivity restore, builds are able to process now with degraded performance.

GitHub have released all jobs to runners, builds are processing normally.

The fix has deployed, we are monitoring to ensure a full recovery.

We have identified a failing service preventing builder machines from acquiring disks and are deploying a fix.

All jobs are processing normally.

The system continues to behave as expected and the incident is now resolved.

As of 21:47 UTC we disabled a feature flag that controlled the faulty provisioning system functionality. The system has now stabilized and jobs are processing normally.

We will continue to monitor for any further issues.

This incident has been resolved, all jobs are processing normally.

Container builds in us-east-1 are also processing again, we are monitoring for a full recovery.

We are seeing some recovery for GitHub jobs, but separately are seeing increased start times for container builds in us-east-1.

We have implemented a performance fix and monitoring recovery.

We have deployed a fix for container image builds in us-east-1 and are monitoring recovery. You can now switch back your projects to us-east in your project settings.

We are deploying a mitigation for AWS issues in us-east-1. GitHub Actions jobs continue to be able to process normally, as do container builds in the EU Central region.

We have isolated the outage to only impacting Docker image builders in us-east-1. As a workaround, you can change your project region in Depot to eu-central to bring builds back online.

We have identified an upstream API limit within AWS that has taken down the container image builders. We are seeing some builds that are able to connect and process but are still working on bringing full capacity back online.

This delays appear to be caused by an AWS EC2 API issue, we have raised a support case with AWS

We have deployed a fix and the job queue is processing again.

We now have a workaround in place to force GitHub webhooks to send to Depot.

We are seeing recovery and are going to continue to watch the webhook delivery times.

We are seeing very slow delivery of GitHub Action webhooks but are beginning to see GitHub recover.

We are no longer seeing GitHub networking issues.

We are seeing normal queue times now. We'll continue to monitor.

We have identified that this appears to be issues with GitHub's network ingress returning "HTTP Status: BadGateway."

We'll continue to investigate.