Network outage impacting judging backend.

Incident Report for Kattis

Postmortem

The outage was caused by a failure of a Power Distribution Unit, or PDU, in the datacenter, causing one of our racks to partially lose power. We have rerouted the power to an alternative PDU in the rack, and escalated the issue to our Colocation provider.

Posted Dec 06, 2023 - 23:10 CET

Resolved

This incident has been resolved.

Posted Dec 06, 2023 - 23:05 CET

Monitoring

Full judging capacity has been restored.

Posted Dec 06, 2023 - 23:03 CET

Update

Partial restoration of the judging backend has been completed. However, we are running with reduced capacity.

Posted Dec 06, 2023 - 22:34 CET

Update

Staff is on site and troubleshooting the issue.

Posted Dec 06, 2023 - 22:33 CET

Update

Pending submissions will be stored, and judged as soon as the judging backend is back online.

Posted Dec 06, 2023 - 21:29 CET

Update

We are continuing to work on a fix for this issue.

Posted Dec 06, 2023 - 21:26 CET

Identified

We have lost contact with a switch in our core infrastructure. We are sending staff to the datacenter to investigate.

Posted Dec 06, 2023 - 21:25 CET

Investigating

We are currently investigating this issue.

Posted Dec 06, 2023 - 21:22 CET

This incident affected: Kattis Education (Web) and Kattis Recruiting (Web).