Depot - Notice history

All systems operational

100% - uptime

us-east-1 - Operational

100% - uptime
Aug 2023 · 99.72%Sep · 99.89%Oct · 99.96%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

eu-central-1 - Operational

100% - uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

us-west-2 - Operational

100% - uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023
100% - uptime

Depot-managed Actions Runners - Operational

100% - uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Github.com - Actions - Operational

API - Operational

100% - uptime
Aug 2023 · 99.92%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Website - Operational

100% - uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 99.99%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Notice history

Oct 2023

Issue downloading Depot CLI
  • Resolved
    UTC
    Resolved

    We have deployed a mitigation to work around Cloudflare's KV outage, the download server is returning success responses.

  • Investigating
    UTC
    Investigating

    A Cloudflare outage is currently affecting downloads of the Depot CLI. We are investigating potential mitigations.

Degraded build performance in eu-central-1 region
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We've deployed an initial fix, builds in eu-central-1 are running on primary infrastructure again. We're monitoring the fix.

  • Investigating
    UTC
    Investigating

    We are currently investigating degraded build performance in eu-central-1 - a number of build jobs are automatically re-routing to backup infrastructure and may experience reduced caching performance during this time.

Builds machines failing to launch
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We have deployed our fix, and we are seeing builds recovering. We will continue to monitor builds coming back online.

  • Update
    UTC
    Update

    We are deploying a fix to our control plane to workaround the Quay outage.

  • Update
    UTC
    Update

    We have identified the issue as being at the Quay registry and are currently monitoring that outage while working on a fix: https://status.quay.io/.

  • Identified
    UTC
    Identified

    We have identified the issue as being at the Quay registry and are currently monitoring that outage while working on a fix: https://status.quay.io/.

Sep 2023

failed to read dockerfile: failed to load cache key: no handler for git
  • Resolved
    UTC
    Resolved

    The queue backlog is clear, build start time has returned to fast.

  • Monitoring
    UTC
    Monitoring

    We are monitoring the rollout of the fix, we are currently waiting on a queue backlog

  • Resolved
    UTC
    Resolved

    We have deployed a fix, the error has cleared

  • Investigating
    UTC
    Investigating

    We are investigating an issue causing some builds to fail with the message "failed to read dockerfile: failed to load cache key: no handler for git".

Builders failing to launch
  • Resolved
    UTC
    Resolved

    We've deployed a fix and will continue to monitor the results.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

Issues starting builds
  • Resolved
    UTC
    Resolved

    We've deployed a fix to address this and are continuing to monitor it.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue that is causing some builds to fail to start.

Aug 2023

Issue starting builds
  • Resolved
    UTC
    Resolved

    We have deployed a fix, build service is restored.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue preventing build machines from starting.

API Outage
  • Resolved
    UTC
    Resolved

    A database fix is deployed, API service is restored

  • Identified
    UTC
    Identified

    We have identified a database issue as the outage cause and are working to remediate.

  • Investigating
    UTC
    Investigating

    We are investigating an API outage.

Builders failing to launch in us-east
  • Resolved
    UTC
    Resolved

    This incident has been resolved, build service is restored

  • Update
    UTC
    Update

    This capacity issue only exists in the us-east region, and we have capacity available in the eu-central region. You can change your region via the project settings.

  • Identified
    UTC
    Identified

    We have identified the issue and are working with our hosting provider to get additional capacity.

  • Investigating
    UTC
    Investigating

    We are currently investigating an issue that is causing builders to fail to launch.

AWS us-east-1 outage
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    All builds are now using primary infrastructure, we are monitoring to ensure that all service is restored.

  • Update
    UTC
    Update

    AWS is reporting that the us-east-1 outage has cleared, we are routing builds back to primary infrastructure.

  • Update
    UTC
    Update

    AWS is experiencing an outage in us-east-1 (https://health.aws.amazon.com/health/status). We are attempting a failover to backup infrastructure in another region.

  • Update
    UTC
    Update

    AWS is still experiencing issues in us-east-1, we are prepping to fail over to backup infra.

  • Identified
    UTC
    Identified

    AWS us-east-1 is currently rate-limiting infrastructure updates, we are monitoring.

  • Investigating
    UTC
    Investigating

    We are investigating lag in acquiring build machines, causing some build requests to time out.

Aug 2023 to Oct 2023

Next