Please keep me posted if anyone sees any problems at this point. I’m pretty confident we are back at 100%.
I’ve implemented some basic (really generous) http/https rate limiting due to script-kiddies scanning our web services.
We are having some issues with the discuss.haiku-os.org container not reliably starting… I plan on digging into this as time presents itself.
For networking issues, i’ve seen one git push be successful, but hang-up on the client… this could be a Gerrit issue though and not our actual fault.
I’m sorry for all of the downtime this year, we’re going through a lot of growing pains. We’re trying to balance reliable infrastructure we can do maintenance on while not spending a ton of cash. The solution historically was to “not touch it, it’s working”, but that really isn’t a healthy long-term strategy. While the new infrastructure isn’t perfect, it offers us a lot more flexibility in the event of a server getting trashed… we also have instant KVM/ILO access to the server which I feel will help us out a lot.
As we grow, we also have the option now to purchase a second server to do more of an active/passive configuration with the new iSCSI storage attachment… but we opted to not take on the extra cost (yet). At the moment on online.net we’re paying ~89eur/mo. This is a pretty big jump from the 39eur/mo we paid Hetzner, but it gives us more growth options with fewer technical hurdles. Network performance has also greatly improved to areas outside of the EU as a bonus.
On the bright side, through all of these major issues, we didn’t lose any data . Backups are important kids, stay in school.