On November 11st at 12:25 pm UTC, customer data in the production database was deleted. This included only data regarding organization accounts and class templates.
At 4:52 pm UTC, the data was restored and the system became fully functional again.
Since all organization and class data was missing, the system had become practically unavailable, for existing customers and activity. New organization accounts or class templates could still be created.
We restored a backup of the production database in a different cluster.
The backup was taken on November 11st at 6:33 am UTC, so the gap in data between the backup and the time of the incident was about 6 hours.
The production database has remained as-is, and only the missing data (of the organization accounts and the class templates) was restored from the backup.
Once the missing data had been restored, existing customers’ activity could be done again.
Changes in organizations (invites, role changes) or in class templates, between November 11st 6:33 am to 12:25 pm, were practically dismissed. Such changes will have to be applied manually again by the customers.
Other than that, there are no implications - on live training activity or on billing.
The data was deleted due to a human error when troubleshooting a production problem in the database.
We got an alert from our monitoring system about a failed API health check.
This revealed that data about organization accounts had been deleted.
We are still in the midst of exploring how exactly it happened.
Our actions will generally revolve around:
We apologize for the effect it had on you.
We strive to improve our service and put the reliability and security of your data and activity at the highest priority.
We are very critical of ourselves and will make sure to bring such incidents to the minimum.