Post-mortem

Summary

On October 1, 2026, during a scheduled maintenance window to migrate to a new AI router, a partial outage occurred affecting Dedicated Inference services. Customers initially received HTTP 404 errors, followed by HTTP 502 errors. The root cause was identified as missing TLS configurations in some customer deployments.

Impact

Requests to impacted Dedicated Inference deployments resulted in errors for approximately 1.5 hours. This impacted all zones where dedicated inference is available.

Timeline

17:00 CET: Migration to the new vLLM router started.
17:15 CET: DNS change that resulted in routing issues, causing requests to end up with 404 errors.
17:30 CET: The routing issue has been fixed, and HTTP 502 errors were being returned
18:00 CET: The incident was escalated as a partial outage
18:07 CET: Mitigations are being gradually applied to fix the TLS configurations of the deployments
18:21 CET: All deployments across all zones have been successfully patched.
18:30 CET: Service back to nominal

What Happened

The new AI Router proxies to Dedicated inference deployments through mTLS. Some of the mTLS resources were missing for some deployments. Because of this misconfiguration, impacted deployments returned HTTP 502 errors for all inference requests.

What We Learned

We didn’t have a proper way to validate that all deployments had their mTLS configuration set up before the maintenance, and we incorrectly assumed this was true for all of them.

What We’ve Changed

We took an inventory and created the missing resources.

Moving Forward

Future infrastructural rollouts will be performed more gradually, progressing deployment by deployment, staged per zone.

Also, we are adding more processes to ensure the invariants that should hold are true before performing this kind of maintenance.

We haven’t seen any more HTTP 500 errors since our last communication.
The situation is nominal; we are sorry for the inconvenience.
A public post-mortem will follow.

The situation has been resolved; we will monitor it until tomorrow morning.

All the deployments have been fixed, we are monitoring the situation.

We discovered the issue and have already fixed some deployments.
We are continuing to apply the mitigation.

Customers with dedicated deployments might encounter HTTP 500 errors when making inference requests.