Depot - Notice history

The authentication failure was caused by an expired OIDC signing certificate that has been renewed. We have deployed a fix to prevent this in the future. All builds are operational.

We needed extra capacity and are now monitoring.

Container build service is restored.

GitHub Actions jobs are processing normally. We are continuing to monitor recovery of container builds in us-east-1.

We are seeing service begin to restore for both container builds and GitHub Actions jobs as capacity rebalances. We're continuing to monitor.

This incident has been resolved and queue times for -latest labels are in the normal ranges again.

We have identified the problem and implemented a fix for those runner labels. We are monitoring queue times to verify they are in the expected range.

We have identified additional latency in starting depot-ubuntu-latest and depot-ubuntu-24.04 GitHub Actions runners. We are investigating whether the same latency exists in container builds.

We have resolved the latency issue that was impacting jobs and container builds from starting. We will continue to monitor the system.

We are continuing to monitor build service. We are aware of container builds being impacted as well and have rolled out a fix for those as well.

Build service is restored, we are currently monitoring the result.

We are currently investigating this incident.

The fix is fully deployed across the container builder regions, and we're observing full recovery. We will continue to monitor.

We have deployed a fix and are monitoring the rollout across the container builder regions.

We've identified a potential fix for amd64 builds and are working on deploying it.

We've isolated the problem to Intel container builds (i.e. linux/amd64) in both regions. We are working to identify the root cause.

This incident has been resolved, CLI downloads are functional again.

We have deployed a workaround to bypass the GitHub outage / rate limit, we are monitoring.

All jobs are processing normally, we are no longer seeing GitHub errors.

We are seeing a reduction of errors from GitHub and an increase in successful jobs processing. We will continue to monitor.

While some jobs are processing, many are still failing. Upstream GitHub appears to still be returning errors from their API. We are continuing to monitor.

This appears to have been caused by a GitHub rate limit reached during the outage. We are beginning to see jobs processing again and are monitoring progress.

We are seeing queue times resolved and normal processing again.

We are seeing queue times decreasing to normal levels. We are monitoring.