GitHub header
All Systems Operational
Git Operations ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
API Requests ? Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Oct 18, 2024

No incidents reported today.

Oct 17, 2024

No incidents reported.

Oct 16, 2024

No incidents reported.

Oct 15, 2024

No incidents reported.

Oct 14, 2024

No incidents reported.

Oct 13, 2024

No incidents reported.

Oct 12, 2024
Resolved - On October 11, 2024, starting at 05:59 UTC, DNS infrastructure in one of our sites started to fail to resolve lookups following a database migration. Attempts to recover the database led to cascading failures that impacted the DNS systems for that site. The team worked to restore the infrastructure and there was no customer impact until 17:31 UTC.

During the incident, impact to the following services could be observed:

- Copilot: Degradation in IDE code completions for 4% of active users during the incident from 17:31 UTC to 21:45 UTC.
- Actions: Workflow runs delay (25% of runs delayed by over 5 minutes) and errors (1%) between 20:28 UTC and 21:30 UTC. Errors while creating Artifact Attestations.
- Customer migrations: From 18:16 UTC to 23:12 UTC running migrations stopped and new ones were not able to start.
- Support: support.github.com was unavailable from 19:28 UTC to 22:14 UTC.
- Code search: 100% of queries failed between 2024-10-11 20:16 UTC and 2024-10-12 00:46 UTC.

Starting at 18:05 UTC, engineering attempted to repoint the degraded site DNS to a different site to restore DNS functionality. At 18:26 UTC the test system had validated this approach and a progressive rollout to the affected hosts proceeded over the next hour. While this mitigation was effective at restoring connectivity within the site, it caused issues with connectivity from healthy sites back to the degraded site, and the team proceeded to plan out a different remediation effort.

At 20:52 UTC, the team finalized a remediation plan and began the next phase of mitigation by deploying temporary DNS resolution capabilities to the degraded site. At 21:46 UTC, DNS resolution in the degraded site began to recover and was fully healthy at 22:16 UTC. Lingering issues with code search were resolved at 01:11 UTC on October 12.

The team continued to restore the original functionality within the site after public service functionality was restored. GitHub is working to harden our resiliency and automation processes around this infrastructure to make diagnosing and resolving issues like this faster in the future.

Oct 12, 01:11 UTC
Update - We’re continuing to work towards recovery of code search service.
Oct 12, 00:46 UTC
Update - We’ve identified the issue with code search and are working towards recovery of service.
Oct 12, 00:14 UTC
Update - We’re continuing to investigate issues with code search.
Oct 11, 23:31 UTC
Update - We’re continuing to investigate issues with code search. Copilot and Actions services are recovered and operating normally.
Oct 11, 22:57 UTC
Update - Copilot is operating normally.
Oct 11, 22:16 UTC
Update - We are rolling out a fix to address the network connectivity issues. Copilot is seeing recovery. support.github.com is recovered.
Oct 11, 22:14 UTC
Update - Actions is operating normally.
Oct 11, 21:46 UTC
Update - We continue to work on mitigations. Actions is starting to see recovery.
Oct 11, 21:28 UTC
Update - The mitigation attempt did not resolve the issue and we are working on a different resolution path. In addition to the previously listed impacts, some Actions runs will see delays in starting.
Oct 11, 20:52 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Oct 11, 20:48 UTC
Update - We continue to work on mitigations. In addition to previously listed impact, code search is also unavailable.
Oct 11, 20:15 UTC
Update - A mitigation for the network connectivity issues is being tested.
Oct 11, 20:05 UTC
Update - We continue to work on mitigations to restore network connectivity. In addition to the previously listed impact, access to support.github.com is also impacted.
Oct 11, 19:28 UTC
Update - We have identified the problem and are working on mitigations. In addition to previously listed impact, new Artifact Attestations cannot be created.
Oct 11, 19:05 UTC
Update - We have identified the problem is related to maintenance performed in our networking infrastructure. We are working to bring back the connectivity.

Copilot users in organizations or enterprises that have opted into the Content Exclusions feature will experience disabled completions in their editors.

Customer migrations remain paused as well.

Oct 11, 18:41 UTC
Update - We are investigating network connectivity issues. Some Copilot customers will see errors on API calls and experiences. We have also paused the remaining customer migration queue while we investigate due to an increase in errors.
Oct 11, 18:25 UTC
Update - We are investigating reports of issues with service(s): Copilot. We will continue to keep users updated on progress towards mitigation.
Oct 11, 17:58 UTC
Update - Copilot is experiencing degraded availability. We are continuing to investigate.
Oct 11, 17:56 UTC
Investigating - We are currently investigating this issue.
Oct 11, 17:53 UTC
Oct 11, 2024
Oct 10, 2024

No incidents reported.

Oct 9, 2024

No incidents reported.

Oct 8, 2024
Resolved - This incident has been resolved.
Oct 8, 23:32 UTC
Update - Codespaces is operating normally.
Oct 8, 23:32 UTC
Update - Codespace creation has been remediated in this region.
Oct 8, 23:32 UTC
Update - We are once again seeing signs of increased latency for codespace creation in this region, but are at the same time recovering previously unavailable resources.
Oct 8, 22:54 UTC
Update - Recovery continues slowly, and we are investigating strategies to speed up the recovery process.
Oct 8, 22:10 UTC
Update - We are continuing to see gradual recovery in the region and continue to validate the persistent fix.
Oct 8, 21:39 UTC
Update - The persistent fix has been applied, and are beginning to see improvements in the region. We are still working on follow-on effects, however, and expect recovery to be gradual.
Oct 8, 21:06 UTC
Update - We are nearing full application of the persistent fix and will provide more updates soon.
Oct 8, 20:26 UTC
Update - Mitigations we have put in place are yielding improvements in Codespace creation success rates in the affected region. We expect full recovery once the persistent fix fully rolls out.
Oct 8, 19:51 UTC
Update - We are continuing to work on mitigations while the more persistent fix rolls out.
Oct 8, 19:17 UTC
Update - We are continuing to apply mitigations while we deploy the more persistent fix. Full recovery is expected in 2 hours or less, but more updates will be coming soon.
Oct 8, 18:44 UTC
Update - We have applied some mitigations that are improving creation success rates while we work on the more comprehensive fix.
Oct 8, 18:08 UTC
Update - We have identified a possible root cause and are working on the fix.
Oct 8, 17:43 UTC
Update - Some Codespaces are failing to create successfully in the Western EU region. Investigating is ongoing.
Oct 8, 17:11 UTC
Update - Codespaces is experiencing degraded performance. We are continuing to investigate.
Oct 8, 17:08 UTC
Investigating - We are currently investigating this issue.
Oct 8, 17:02 UTC
Oct 7, 2024

No incidents reported.

Oct 6, 2024

No incidents reported.

Oct 5, 2024

No incidents reported.

Oct 4, 2024

No incidents reported.