Depot - Docker image builders failing to start – Incident details

All systems operational

Docker image builders failing to start

Resolved
Partial outage
Started about 1 year agoLasted about 4 hours

Affected

Container builds

Degraded performance from 7:18 PM to 8:56 PM, Partial outage from 8:56 PM to 10:24 PM, Operational from 9:17 PM to 10:24 PM, Partial outage from 10:24 PM to 11:33 PM, Operational from 10:24 PM to 11:33 PM, Partial outage from 11:33 PM to 11:41 PM, Operational from 11:33 PM to 11:41 PM

us-east-1

Degraded performance from 7:18 PM to 8:56 PM, Partial outage from 8:56 PM to 11:41 PM

eu-central-1

Degraded performance from 7:18 PM to 8:56 PM, Partial outage from 8:56 PM to 9:17 PM, Operational from 9:17 PM to 11:41 PM

GitHub Actions

Degraded performance from 7:18 PM to 7:40 PM, Operational from 7:18 PM to 7:40 PM, Degraded performance from 7:40 PM to 8:56 PM, Operational from 7:40 PM to 11:41 PM

Depot-managed Actions Runners

Degraded performance from 7:18 PM to 8:56 PM, Operational from 8:56 PM to 11:41 PM

Github.com - Actions

Operational from 7:18 PM to 11:41 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    We have deployed a fix for container image builds in us-east-1 and are monitoring recovery. You can now switch back your projects to us-east in your project settings.

  • Monitoring
    Monitoring

    We are deploying a mitigation for AWS issues in us-east-1. GitHub Actions jobs continue to be able to process normally, as do container builds in the EU Central region.

  • Update
    Update

    We have isolated the outage to only impacting Docker image builders in us-east-1. As a workaround, you can change your project region in Depot to eu-central to bring builds back online.

  • Update
    Update

    We have identified an upstream API limit within AWS that has taken down the container image builders. We are seeing some builds that are able to connect and process but are still working on bringing full capacity back online.

  • Identified
    Identified

    This delays appear to be caused by an AWS EC2 API issue, we have raised a support case with AWS

  • Investigating
    Investigating
    We are currently investigating this incident.