Cakemail - Our databases are unreachable – Incident details

We apologize for the recent downtime. Due to an issue with a cluster of routers caused by a low-level network problem, some services experienced partial and total outages for approximately one hour. As a result, some machines in the cluster could not communicate with each other, although they were occasionally visible from other parts of the network.

To address the issue, we have adjusted our configurations, particularly on our database load balancer and the procedure for failover in the edge router cluster. We are taking all necessary measures to prevent this from happening in the future.

We understand the inconvenience this may have caused, and we apologize for the inconvenience.

Thank you for your patience and understanding.

We are back to normal and our team is monitoring the system stability.
We sincerely apologize for the delay and inconvenience. Our team will make a complete evaluation of the issue for an official post-mortem.

The team is working on the issue. Some of the services are back to normal but we haven’t fully recovered. We’ll keep you informed on the progress.