[Network Resolved] Firewall Hardware failure resolved

Added by Benjamin Baer about 3 years ago

Thank you for your patience. The network, mail, chat, etc. is working again.

Our firewall had a hardware problem and blocked all the traffic.

Please be warned that their may be a further short disruption this afternoon.

Background what happened

At 13.7.21 01:50 there was a power spike or short power outage or similar.
More than half of our cloud bare metal servers have been rebootet at that time.
Most of the running thinlinc sessions has been killed.
The institutes firewall broke completely: two disks died at the same time.
Wen needed some time to understand why our whole cloud had been down.
After restarting all the virtual machines (around 09:00) the IT health seems fine, but all traffic to the internet still stopped at the broken firewall.
Around 10:00 we bypassed the firewall and fixed some more smaller issues (homepage and chat server still had some trouble).
Starting at 11:00 all services went back online.
At 15:00 the reinstalled firewall (incl. restore of configuration backup) becomes productive and it seems we're fully online again.

Public