Failure to replace or provision labs

Incident Report for Strigo

Resolved

Between 5:11 and 5:15 PM UTC, some 200 requests to provision and replace lab machines failed due to AWS EC2 request limits. Most of the provision or replacement attempts succeeded during retries, but some failed after a timeout.

We received a RequestLimitExceeded error from the AWS API, indicating that AWS couldn't process additional provisioning requests.

To deal with this problem, we'll optimize our retry mechanism further, and make sure our support team knows to replace any labs that we failed to replace or provision if any issue arises.

Posted Jun 13, 2022 - 20:11 IDT