Project

General

Profile

[Network Resolved] Firewall Hardware failure resolved

Added by Benjamin Baer 2 months ago

Thank you for your patience. The network, mail, chat, etc. is working again.

Our firewall had a hardware problem and blocked all the traffic.

Please be warned that their may be a further short disruption this afternoon.


Comments

Added by Carsten Rose 2 months ago

Background what happened

  • At 13.7.21 01:50 there was a power spike or short power outage or similar.
  • More than half of our cloud bare metal servers have been rebootet at that time.
  • Most of the running thinlinc sessions has been killed.
  • The institutes firewall broke completely: two disks died at the same time.
  • Wen needed some time to understand why our whole cloud had been down.
  • After restarting all the virtual machines (around 09:00) the IT health seems fine, but all traffic to the internet still stopped at the broken firewall.
  • Around 10:00 we bypassed the firewall and fixed some more smaller issues (homepage and chat server still had some trouble).
  • Starting at 11:00 all services went back online.
  • At 15:00 the reinstalled firewall (incl. restore of configuration backup) becomes productive and it seems we're fully online again.