Frankfurt datacenter rn
/s
It was not quite this bad, but air temperatures inside the facility reached 50-60C according to our monitoring. That's enough to start killing multi-million euro server racks with powerful CPUs, GPUs, etc, just like an actual fire would. On site…
Read all of itShow less
It was not quite this bad, but air temperatures inside the facility reached 50-60C according to our monitoring. That's enough to start killing multi-million euro server racks with powerful CPUs, GPUs, etc, just like an actual fire would. On site staff can't even enter without health risk.
I shared a few more details on X: https://x.com/andyyen/status/2092797856880975887
tl;dr, We knew about this potential failure mode, but not prioritized due to super low probability, but Murphy's law.... Servers are safe, but we couldn't do automated failover, so full recovery unfortunately took over an hour.
