Failed jobs on eu-central-1 stack (AWS EU)

2023-01-09 07:45 UTC - We have identified an issue on one of the servers running Queue jobs on the EU Central 1 (AWS EU) stack. Numerous jobs are stuck in a terminating state and we are currently investigating the cause of the issue.

2023-01-09 08:05 UTC - We have unblocked the stuck jobs, which were unexpectedly terminated. We are investigating the root cause of the node failure.


Some jobs running two hours longer in AWS EU

In very rare circumstances, some (less than 10 per day) jobs may be delayed by almost exactly two hours in the AWS EU stack (connection.eu-central-1.keboola.com). During this period, the job will be stuck doing nothing for two full hours, and unfortunately terminating the job will not help

We are currently trying our best to debug and fix an underlying network connectivity issue. If you have any questions or concerns, please reach out to our support.

We are sorry for this inconvenience and will provide an update on this post once we know more or have an ETA of the fix.

Update 2023/01/23 8:40 - We have implemented a fix for the issue and so far there are no occurrences of this issue in the past 12hours. We're continuing to monitor the issue thouroughly.


Job failures on eu-central-1 stack (AWS EU)

2022-12-30 08:15 UTC - We are investigating occasional job failures that started on December 29, 2022 at 11:00 PM UTC. We will provide an update with new information when it becomes available.

2022-12-30 09:12 UTC - The error rate is lower, but there are still some occurrences of errors. We are investigating the root cause and will provide an update with new information when it becomes available.

2022-12-30 10:38 UTC - We have identified and fixed the problem, which was caused by rate limiting on the container registry. The last error occurred at 10:08 AM UTC. We are monitoring all systems closely.

2022-12-30 11:23 UTC - We don't see any new occurrences of errors. Platform is fully operational and incident is resolved. 

Failed jobs on eu-central-1 stack (AWS EU)

We have discovered a problem on one of servers running Queue jobs on eu-central-1 stack (AWS EU). Jobs were terminated unexpectedly in between 08:20 UTC and 09:20 UTC. The problem has been removed and all jobs should be running OK now again. We are still looking for the root cause to prevent it happening again in the future. We apologize for any inconvenience this may have caused.

Failed jobs on eu-central-1 stack (AWS EU)

We have discovered a problem on one of servers running Queue jobs on eu-central-1 stack (AWS EU). Jobs were terminated unexpectedly from 12:00 AM CET. We are investigating the cause of the problem

Update 12:50 PM CET - The problem has been removed and all jobs should be running OK now again. We are still looking for the root cause to prevent it happening again in the future.

We apologize for any inconvenience this may have caused.

Failed jobs on eu-central-1 stack (AWS EU)

We have discovered a problem on one of servers running Queue jobs on eu-central-1 stack (AWS EU). Jobs were terminated unexpectedly in between 13:00 UTC and 14:45 UTC. The problem has been removed and all jobs should be running OK now again. We are still looking for the root cause to prevent it happening again in the future. We apologize for any inconvenience this may have caused.

Delayed jobs start in AWS EU

2022-12-21 14:05 UTC - We are investigating delayed jobs start on AWS EU Keboola Connection stack (https://connection.eu-central-1.keboola.com). Next update in 30 minutes.

2022-12-21 14:55 UTC - Services are stable now. There could be some delayed jobs between 13:55-14:30 UTC. We are monitoring situation and investigation root cause. We apologize for any inconvenience this may have caused.

Failures of Google Drive Data source

We have seen an increase in the number of errors in the Google Drive data source since 15:00 UTC. We are currently investigating the issue and rolling back the recent release. 

The error messages are identified by this error message

Unrecognized options "sheets, outputBucket" under "root.parameters". 

We will provide an update in 15 minutes.

UPDATE 15:27 UTC: The issue has been resolved, and the previous version is functioning as expected. If you have encountered this issue, please restart your jobs and flows. We apologize for any inconvenience this may have caused.

Delayed orchestrations on Azure North Europe stack

2022-11-29 17:55 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

Update 2022-11-29 18:40 UTC - We have deployed a fix and the orchestration schedules will gradually catch on. Next update in 1 hour.

Update 2022-11-29 19:27 UTC - Orchestration schedules are now on time. The incident is no resolved, but we'll keep monitoring the situation. We apologize for the inconvenience.




Stuck jobs and unable to start workspaces in AWS EU

Nov 29 07:08 UTC - We are investigating multiple stuck jobs on connection.eu-central-1.keboola.com stack. Affected jobs became stuck around 03:00 AM UTC, other jobs are processing and starting without issues. Next update in 30 minutes or when new information will be available.

Nov 29 08:02 UTC - We have unblocked stuck jobs, and we no longer see queueing of jobs. We are investigating the root cause and impact of the incident. Next update when new information will be available.

Nov 29 08:35 UTC - We're still seeing further symptoms of the outage and we're actively investigating. 

Affected services are: 

  • Workspaces - partial outage (workspaces may have difficulties starting)

Nov 28 10:22 UTC - Platform is now fully operational. We're monitoring all systems closely. 

We're sorry for the inconvenience. If you experienced any job failures please run them again.