Service interruption

Incident summary and impact
At approximately 06:11 UTC, a technical issue caused a general API outage affecting all credited services, including the Omnichannel API and Dashboard messaging. The outage directly prevented real-time credit deductions, meaning some customers’ traffic could not be processed leaving affected messages in ‘rejected’ status. For others, dashboard credit balances displayed incorrectly during the incident.
The core issue was resolved and all services were fully operational by 07:45 UTC.

Root cause and technical analysis
The root cause was an integer overflow in the database. The primary key ID column for the transaction logging table reached its maximum sequence limit, preventing the system from appending new records. This database write failure caused a cascading bottleneck: failed transaction records diverted to a backup queue, which grew rapidly and caused the balance synchronization service to time out, ultimately blocking incoming API requests.

Resolution and remediation
To restore service, engineering bypassed the stalled table by creating a fresh transaction table structure to accept live traffic immediately. We then isolated the congested queue to eliminate the timeout loop, returning the API to normal operations.
The following follow-up actions are being executed:

• Upgrading the underlying transaction ID data type to prevent future sequence exhaustion.
• Merging the temporary data back into the primary table to fully restore accurate customer credit histories and account balances.
• Implementing additional monitoring and alerting to catch sequence limits and queue bottlenecks before they impact production.
  

For any questions or concerns, kindly reach out to support@messente.com