Depot - Notice history

This incident has been resolved and queue times for -latest labels are in the normal ranges again.

We have identified the problem and implemented a fix for those runner labels. We are monitoring queue times to verify they are in the expected range.

We have identified additional latency in starting depot-ubuntu-latest and depot-ubuntu-24.04 GitHub Actions runners. We are investigating whether the same latency exists in container builds.

We have resolved the latency issue that was impacting jobs and container builds from starting. We will continue to monitor the system.

We are continuing to monitor build service. We are aware of container builds being impacted as well and have rolled out a fix for those as well.

Build service is restored, we are currently monitoring the result.

We are currently investigating this incident.

The fix is fully deployed across the container builder regions, and we're observing full recovery. We will continue to monitor.

We have deployed a fix and are monitoring the rollout across the container builder regions.

We've identified a potential fix for amd64 builds and are working on deploying it.

We've isolated the problem to Intel container builds (i.e. linux/amd64) in both regions. We are working to identify the root cause.

This incident has been resolved, CLI downloads are functional again.

We have deployed a workaround to bypass the GitHub outage / rate limit, we are monitoring.

All jobs are processing normally, we are no longer seeing GitHub errors.

We are seeing a reduction of errors from GitHub and an increase in successful jobs processing. We will continue to monitor.

While some jobs are processing, many are still failing. Upstream GitHub appears to still be returning errors from their API. We are continuing to monitor.

This appears to have been caused by a GitHub rate limit reached during the outage. We are beginning to see jobs processing again and are monitoring progress.

We are seeing queue times resolved and normal processing again.

We are seeing queue times decreasing to normal levels. We are monitoring.

The stuck jobs have all processed successfully.

We are observing queue times returning to normal. We will continue to monitor.

We are continuing to see recovery but are continuing to monitor the remaining outliers.

We have deployed a fix and are monitoring for queue time recovery.

We've discovered a failure in our task scheduler that is blocking operations. We're deploying a fix that should unblock jobs.