Network issues fixed
The cause of our issues was an external Denial Of Service attack based around NTP (Network Time Protocol). It was targeted at a server here that used to provide that service to the outside world. We haven’t run an externally-facing NTP server in years, certainly not since we moved from 221. Blocking all access to that host resolved the issue, but created new ones as we have hosts that are outside our firewall that rely on being able to talk to that host.
Eventually, the right combination of blocking and access was installed that accomplished a stop to the DOS while not overtaxing the CPUs on the routers to the point they were dropping packets.
Some services may need a kick, and we’re hitting them as our monitoring tells us, but if you see something out of the ordinary, please let us know at email@example.com.