Billing Panel Issues

Incident Report for GlassHost

Postmortem

GlassHost billing system incident report - January 17th, 2024.

On Wednesday, January 17th, 2024, at 11:12 UTC, our billing system experienced an outage with a 502 error.

This incident was first reported by Nathan F, escalated to Nathan W at 13:19 UTC. Our customers were promptly informed about the downtime at status.glasshost.net.

At approximately 17:30 UTC, I woke up to a notification alerting me to the inaccessibility of the billing panel.

At that time, I checked the error log and found that one of our WHMCS module (Lagom), was root cause of the error, I promptly disable the module and reverted to WHMCS default theme. This solves the issue, the billing panel is accessible to the public again at 17:53 UTC.

At roughly 20:00 UTC, Nathan F resolved the module issue, restoring the client theme and functionality to normal.

We sincerely apologize for any inconvenience this may have caused you.

‌

I = Darren Nathanael

WHMCS = Billing System Software

Lagom = Billing System Theme

Posted Jan 18, 2024 - 01:01 UTC

Resolved

Simple module conflict after a recent update. A solution has been applied and all services are back to being up and running. Apologies for any inconvenience.

Posted Jan 17, 2024 - 20:51 UTC

Update

It appears to effect only the front page of the billing panel and the client theme. Currently waiting for Nathan F to get online and debug the custom module with the developer. Other than that, everything should be working.

- Darren

Posted Jan 17, 2024 - 17:59 UTC

Identified

The issue has been identified, it is related to one of our WHMCS module. The module has been removed temporarily, further details will be added.

- Darren

Posted Jan 17, 2024 - 17:53 UTC

Investigating

We are currently investigating this issue.

Posted Jan 17, 2024 - 13:19 UTC

This incident affected: Core Services (Billing).