Keboola Status

Today since 12:00 UTC we suffer jobs as well as sync actions failing to start due to bug introduced in the last deploy. We are now reverting to the previous working version. 

Update 13:15 UTC (resolved) : We have successfully reverted backend to the previous working version and platform is back operational now.

Update 13:30 UTC: Upon further investigation and damage analysis, we found that Queue v2 jobs were delayed in starting because of this incident. That could result in delays of up to one hour.

2024-04-25 02:23 UTC - We experienced another incident on connection.eu-central-1.keboola.com between 01:30 UTC and 02:30 UTC. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

We noticed a slowdown in our AWS EU stack’s job queue from 12:00 to 14:00 UTC, due to a temporary service performance issue. We sincerely apologize for any inconvenience this may have caused.

Our team has resolved the problem and we are taking steps to prevent future occurrences. Thank you for your understanding and patience.

13:07 UTC: We're investigating an issue where import to Storage shows "Storage API: The new primary key cannot be created; duplicate values in primary key columns exist" error. 
Next update in 15 minutes. 

13:24 UTC: We've rolled back the last release. We're investigating impact of the issue. 

13:54 UTC: We're still investigating the impact. 

14:34 UTC: We're still investigating the impact. So far it seems to have only impacted a very limited number of customer jobs. We will reach out to the affected customers separately as soon as we finish the impact investigation.  

15:09 UTC: We've finished investigating the impact and we're reaching out to the affected customers. 

AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance will start today, April 13, 2024, at 12:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

Maintenance should take less than three hours.

We apologize for any inconvenience caused and thank you for your understanding.

Update 12:00 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance started.

Update 13:30 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance is finished and the stack is now fully operational.

AWS us-east-1 stack (connection.keboola.com) maintenance will start today, April 06, 2024, 07:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

We apologize for any inconvenience caused and thank you for your understanding.

Update 07:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance started.

7:20 UTC: We are investigating an error when installing a python package in the transformations.
You may see errors such as: Job "XXXXXXX" ended with a user error "Failed to install package: io".
Next update in 15 min.

UPDATE 7:38 UTC: We have rolled back a previous version and all operations are back normal. We're sorry for this inconvenience. 

Update 11:00 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) maintenance started.

Update 11:52 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) service disruption has been resolved and the stack is now fully operational. Thank you for your patience.

We would like to remind you of the upcoming scheduled maintenance for Keboola stacks hosted on AWS and Azure.

9:21 UTC: We are investigating jobs not starting on AWS EU this morning. Next update in 30 minutes.

9:57 UTC: We are still investigating the issue. Next update in 30 minutes.

10:24 UTC: We have identified stuck job that was causing other jobs not starting and proceed to unblock it. The platform is operational now and all stuck jobs have started running. We continue investigating the root cause.