Announcement 16/03/2024
Dear customers,
I would like to begin this statement by thanking you for your immense patience and understanding during our longer than desired absence.
In events such as we have just traversed, it is my strong belief that the only correct course of action here is one of full transparency and accountability.
As such, it is essential that we go over the events of the last few days, and what has led to them. Before we delve into the technical points however, let me address a few recurring points of concern.
First and foremost, we want to re-emphasize that the website downtime was NOT the result of a cybersecurity incident. As previously stated, your personal information and data are safe and have not been compromised in any way.
In light of this downtime, we understand the importance of ensuring our customers have sufficient time to access their account, settle any outstanding invoices, and replenish account funds if necessary.
As of writing, a 72 hours grace period is in place for all payment related concerns. This grace period may be extended if deemed necessary.
No service, dedicated or virtual will be suspended during this period, regardless of your invoice's current payment status.
This should give you plenty of time to ensure that any due payment can be made, and that your account funds can be replenished. A notice will be sent out prior to ending the grace period.
Furthermore, all pending orders that have been delayed due to our downtime will benefit from an additional 72 hours renewal date (on top of the usual renewal date adjustment to the date of delivery).
Thirdly, due compensation will be provided on a case by case basis for all customers who were unable to obtain assistance from our support team. They are actively focusing on all pending and urgent support requests that we were unable to get to during this period, while our accounting and delivery teams have resumed and are expediting all pending deliveries and refunds.
Now for the post mortem.
-Following a planned performance upgrade of our platforms hardware on February 29th, incompatibilities between older software and more recent hardware have led to unexpected complications
-On March 13th, at 15:09 EST, an attempt at mitigating those incompatibilities resulted in what appeared to be corruption on some of our infrastructure's file system. A few database tables were also affected.
-As a security precaution, we went into emergency maintenance mode to prevent any further damage from potentially occurring.
-Early attempts to fallback on our primary backup servers unfortunately failed, as those had become corrupted by the same material upgrade.
-Secondary backups were then successfully rolled back to shortly after.
-Out of an abundance of caution, we have chosen to extend our down time in order to run complete integrity tests on every single file and database of our infrastructure, and have thoroughly eliminated the unlikely possibility of a malicious event.
I am happy to report that those integrity tests have all been 99.5% successful, and no permanent data loss has occurred. A few incongruities do remain and it is possible that some customers encounter odd but harmless bugs in the coming days such as missing log entries, or missing spec details on a given service. Rest assured that our team is actively ironing out those remaining quirks.
We have also taken this time to accelerate our roadmap on increasing protections so that such events do not occur again. Multiple preventative measures that were already planned have now been completed.
While the duration of the incident was certainly unusual, I am proud of the work that my team has accomplished in the last few days. This was indeed a true test of our resilience, one that every company faces sooner or later, and I could not be more satisfied of the resolve, dedication, and competence with which it was met. We will continue learning from this incident in the coming days, but it is one that we have surmounted, and will, in consequence, grow from.
On behalf of my team and I, we would like to sincerely apologize for any inconvenience this downtime may have caused you. As always, we appreciate your patience and understanding as we worked diligently to resolve the issue. We remain committed to continuously improving our systems and processes to prevent similar incidents in the future.
If you have any further questions or concerns, please do not hesitate to reach out to our support team.
Sincerely,
Charles-René Paquet - CEO
Announcement 14/12/2022
Dear customers,
We would like to inform you of maintenance works in Bruxelles between the below time slots:
From: Wednesday, December 14, 2022 03:00 UTC
To: Wednesday, December 14, 2022 07:00 UTC
Your services will experience small downtimes within the specified time slot outlined above. Downtime during the whole maintenance window is not expected.
We appreciate your understanding and we acknowledge that maintenance is disruptive.
Rest assured that our team will do everything in their power to ensure that the maintenance goes as smooth as possible in as short as time as permissible.
If you have any questions or concerns please dont hesitate to open a ticket with our team and well be happy to discuss the situation with you further.
OneProvider - Worldwide Hosting Provider.
Announcement 08/10/2022
Dear Customer,
We will be performing planned maintenance at our Bucharest Datacenter 2 between the times shown below:
From: Saturday the 08th of October 2022 08:00 (local time) UTC +3
To: Saturday the 08th of October 2022 20:00 (local time) UTC +3
In the mentioned interval, we will perform maintenance and electricity infrastructure improvement operations on the primary circuit.
The main circuit will be de-energized, and the consumers will be moved to a temporary circuit. The estimated interruption is 15 minutes. After carrying out the proposed operations, this temporary circuit will be de-energized and returned to the final version. The second estimated interruption is 15 minutes.
This operation is in accordance with the Tier 3 standard. The secondary circuit will work without interruption and if the consumption on the entire cabinet is within the agreed nominal range and the equipment is powered in a redundant mode, the operation of the services will not be affected.
Our engineering team will monitor the power supply systems while these works are carried out.
If you have any questions please don't hesitate to contact us.
OneProvider - Worldwide Hosting Provider