Service Disruption - Cloud Workers in a Blocked state

Incident Report for Cribl

Resolved

December 02, 2021 | 11:06PM
The Cribl Cloud Operations team has restored data ingestion services to all of the affected Cloud organizations. Service is operational and being monitored.

December 02, 2021 | 10:48PM
Moving the incident to degraded status instead of an outage

December 02, 2021 | 10:47PM
We have updated 99% of our production tenants and service has been restored for them. The team is still taking care of the last 1% before we clear the incident.

December 02, 2021 | 09:32PM
The Cribl Cloud Operations team is in the process of completing the fix rollout. We have verified service restoration for ~65% of our production tenants and are working to restore service to the remaining tenants. Another update will be published within 30 minutes.

December 02, 2021 | 08:39PM
The Cribl Cloud Operations team is in the process of pushing out a fix that will remediate the blocked data to cloud worker issue. It is anticipated that this may take up to 45-60 minutes to complete. We will publish another update within that timeframe with an updated status.

December 02, 2021 | 08:07PM
Data flow to cloud workers is still impacted at this time. We are currently validating a fix that will unblock the worker data flow issue that tenants are experiencing. Another update will be posted within the next 30 minutes.

December 02, 2021 | 06:59PM
Root cause has been identified and we are working on a fix to un-block data to cloud workers again for all tenants impacted
Posted Dec 02, 2021 - 14:00 PST