September 15, 2023, 3:01 AM EST to 3:35 AM EST

At 3:01 AM EST, our monitoring system, Grafana, detected missing data, indicating a potential issue with the connection. An emergency alert was immediately sent to our team.

At 3:03 AM EST, a customer reported that their Virtual Private Server (VPS) was unreachable, confirming the issue from a user perspective.

Actions Taken:

• At 3:05 AM EST, we confirmed that the link was down and initiated contact with the datacenter technician on-site to investigate the issue.
• At 3:07 AM EST, we attempted a remote reboot using IPMI to resolve the issue. Unfortunately, this did not resolve the problem.
• At 3:10 AM EST, the on-site datacenter technician responded and conducted a physical inspection of our hardware.
• At 3:20 AM EST, the on-site technician identified the issue as a problem with the switch routing, which resulted in an unreachable state for our services.
• At 3:28 AM EST, the on-site technician successfully fixed all issues with the switch and reconfigured the uplink connections.
• At 3:30 AM EST, we performed a network configuration via IPMI and remounted all drives. A final reboot was initiated to ensure the system was stable.

Resolution:


At 3:35 AM EST, all network services were confirmed to be back online, and there were no further reported issues. We will continue to monitor the situation for any potential recurrence, but it is believed that the problem has been fully resolved.