[API] Increased Error Rates and Latencies
Updates
The situation is stable, we are closing this status. A public postmortem will be published in the coming days.
Our systems are back to normal, we are monitoring the situation. Every services should be back online.
We confirmed some services are recovering, some flaps are to be expected in the coming minutes.
New mitigations are being applied, we will publish new updates shortly.
Mitigations are still been applied, they did not solve the ongoing incident. The impact continues on both API services (full impact) & NLB traffic (partial packet loss).
All mitigations have been applied and the team is monitoring the system. Investigations into the NLB packet loss is ongoing.
The team is confirming that all known mitigations have been applied. Another part of the team is investigating packet drops on NLB in Geneva.
We are still actively working on applying mitigations. NLB services are correctly processing traffic and were not impacted, we are removing them from the impacted scope.
We are still actively working on applying mitigations, services are still unreachable.
We are still actively working on mitigations, the impact is extended api, nlb, sks and ai services in ch-gva-2.
The source of the problem has been localized and we are actively working on applying mitigations.
The impact is confined to the ch-gva-2 zone, the API is working correctly in other zones.
We are investigating an increased error rates and latencies on the API. We’ll post an update as soon as we have more information.
← Back