All systems operational

Builds

99.95% uptime
Feb 2024 · 99.95%Mar · 99.90%Apr · 100.0%
Feb 202499.95% uptime
Mar 202499.90% uptime
Apr 2024100.0% uptime
API
99.95% uptime
Feb 2024 · 99.98%Mar · 99.93%Apr · 99.93%
Feb 202499.98% uptime
Mar 202499.93% uptime
Apr 202499.93% uptime

Depot-managed GitHub Actions Runners

99.95% uptime
Feb 2024 · 100.0%Mar · 99.93%Apr · 99.93%
Feb 2024100.0% uptime
Mar 202499.93% uptime
Apr 202499.93% uptime
Website
99.96% uptime
Feb 2024 · 100.0%Mar · 99.95%Apr · 99.94%
Feb 2024100.0% uptime
Mar 202499.95% uptime
Apr 202499.94% uptime
Github - Actions
Operational

Notice history

Apr 2024

GitHub runner capacity
  • Resolved
    Resolved

    We have deployed a fix and are observing normal job pickup times now.

  • Identified
    Identified

    We have identified an AWS capacity issue in our primary region that is impacting job pickup time. We are deploying a fix now.

OIDC token exchange failures
  • Resolved
    Resolved

    We've deployed a fix that has brought public OIDC token exchanges back online. We will continue to monitor recovery.

  • Investigating
    Investigating

    We are investigating OIDC token exchange failures for pull requests from public forks.

Monitoring GitHub outage
  • Resolved
    Resolved

    GitHub has cleared their outage, but we are monitoring webhook delivery times from their side.

  • Monitoring
    Monitoring

    We are still monitoring webhook flakiness due to the ongoing GitHub outage: https://www.githubstatus.com/incidents/7txcg0j03kkg. This is surfacing as longer than normal queue times while we wait for GitHub to pass the job to an online runner.

  • Identified
    Identified

    We are actively monitoring the ongoing GitHub outage that is impacting GitHub Actions: https://www.githubstatus.com/incidents/7txcg0j03kkg

Increased GitHub Actions queue times
  • Resolved
    Resolved

    AWS has cleared the capacity issue, all jobs are running again

  • Identified
    Identified

    This appears to be caused by a widespread capacity shortage in multiple AWS availability zones. We're starting to route to backup capacity now.

  • Investigating
    Investigating

    We are investigating an incident that is causing GitHub Actions jobs to take several minutes to start

GitHub outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are monitoring the rollback GitHub has performed and are observing jobs being processed: https://www.githubstatus.com/

  • Investigating
    Investigating

    GitHub is currently experiencing a widespread outage that is impacting webhook delivery to our runners: https://www.githubstatus.com/

Mar 2024

Degraded performance when starting Docker and GitHub Actions builds
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing queue times return to normal, we're continuing to monitor

  • Identified
    Identified

    We have identified a database issue preventing builds from starting, we are working to resolve the issue.

  • Investigating
    Investigating

    We are investigating an issue causing longer queue times when starting Docker and GitHub Actions builds

Issue launching Docker builders
  • Resolved
    Resolved

    We've deployed a fix and are observing a full recovery. We will continue to monitor performance.

  • Investigating
    Update

    We're experiencing additional database load that is impacting Docker builds and GitHub Actions Runners.

  • Investigating
    Investigating

    We are currently investigating an issue preventing Docker builds from starting

Degraded performance
  • Resolved
    Resolved

    We have identified the problem and are monitoring recovery.

Issues starting builds in us-east-1
  • Resolved
    Resolved

    We are monitoring build recovery after a brief network disruption at our upstream database provider.

  • Investigating
    Investigating

    We are currently investigating some builds failing to start in us-east-1.

Feb 2024

Issues starting builds in us-east-1
  • Resolved
    Resolved

    We've rolled out a fix, and all builds are recovering. We will continue to monitor.

  • Monitoring
    Monitoring

    We are rolling out a fix and monitoring recovery.

  • Identified
    Identified

    We've identified an issue with API latency that is causing builds to take longer than usual to start.

  • Investigating
    Investigating

    We are currently investigating delays in machines launching for builds in us-east-1.

Issue starting builds in us-east-1
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Build service is restored, we are monitoring.

  • Investigating
    Investigating
    We are currently investigating this incident.
Issues pushing from the ephemeral registry
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are currently investigating errors with running depot push to push an image from the ephemeral registry.

Issues receiving sign-in emails
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Update

    We are working to fail over our email delivery to an internal provider.

  • Identified
    Identified

    We've identified that our upstream email delivery provider is having an outage which is currently preventing sign-in emails from being delivered.

Issues acquiring builder in us-east-1
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing build service restore - this incident appears to have been caused by a database provider outage that has since recovered. We're continuing to monitor.

  • Investigating
    Investigating

    We are investigating an incident in us-east-1

Feb 2024 to Apr 2024