[Announcement] We're back! About the maintenance...

2024-03-16 19:16:00

Dear customers,

I would like to begin this statement by thanking you for your immense patience and understanding during our longer than desired absence.

In events such as we have just traversed, it is my strong belief that the only correct course of action here is one of full transparency and accountability.
As such, it is essential that we go over the events of the last few days, and what has led to them. Before we delve into the technical points however, let me address a few recurring points of concern.

First and foremost, we want to re-emphasize that the website downtime was NOT the result of a cybersecurity incident. As previously stated, your personal information and data are safe and have not been compromised in any way.
In light of this downtime, we understand the importance of ensuring our customers have sufficient time to access their account, settle any outstanding invoices, and replenish account funds if necessary.
As of writing, a 72 hours grace period is in place for all payment related concerns. This grace period may be extended if deemed necessary.
No service, dedicated or virtual will be suspended during this period, regardless of your invoice's current payment status.

This should give you plenty of time to ensure that any due payment can be made, and that your account funds can be replenished. A notice will be sent out prior to ending the grace period.

Furthermore, all pending orders that have been delayed due to our downtime will benefit from an additional 72 hours renewal date (on top of the usual renewal date adjustment to the date of delivery).

Thirdly, due compensation will be provided on a case by case basis for all customers who were unable to obtain assistance from our support team. They are actively focusing on all pending and urgent support requests that we were unable to get to during this period, while our accounting and delivery teams have resumed and are expediting all pending deliveries and refunds.

Now for the post mortem.

-Following a planned performance upgrade of our platforms’ hardware on February 29th, incompatibilities between older software and more recent hardware have led to unexpected complications
-On March 13th, at 15:09 EST, an attempt at mitigating those incompatibilities resulted in what appeared to be corruption on some of our infrastructure's file system. A few database tables were also affected.
-As a security precaution, we went into emergency maintenance mode to prevent any further damage from potentially occurring.
-Early attempts to fallback on our primary backup servers unfortunately failed, as those had become corrupted by the same material upgrade.
-Secondary backups were then successfully rolled back to shortly after.
-Out of an abundance of caution, we have chosen to extend our down time in order to run complete integrity tests on every single file and database of our infrastructure, and have thoroughly eliminated the unlikely possibility of a malicious event.

I am happy to report that those integrity tests have all been 99.5% successful, and no permanent data loss has occurred. A few incongruities do remain and it is possible that some customers encounter “odd” but harmless bugs in the coming days such as missing log entries, or missing spec details on a given service. Rest assured that our team is actively ironing out those remaining quirks.
We have also taken this time to accelerate our roadmap on increasing protections so that such events do not occur again. Multiple preventative measures that were already planned have now been completed.

While the duration of the incident was certainly unusual, I am proud of the work that my team has accomplished in the last few days. This was indeed a true test of our resilience, one that every company faces sooner or later, and I could not be more satisfied of the resolve, dedication, and competence with which it was met. We will continue learning from this incident in the coming days, but it is one that we have surmounted, and will, in consequence, grow from.

On behalf of my team and I, we would like to sincerely apologize for any inconvenience this downtime may have caused you. As always, we appreciate your patience and understanding as we worked diligently to resolve the issue. We remain committed to continuously improving our systems and processes to prevent similar incidents in the future.

If you have any further questions or concerns, please do not hesitate to reach out to our support team.
Sincerely,
Charles-René Paquet - CEO