Keboola Status

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 30 minutes, around 8:00 UTC for GCP europe-west3, and 8:30 UTC for GCP us-east4 respectively. We will update this post with the progress of the partial maintenance.

2024-11-02 8:00 UTC: The announced partial maintenance of europe-west3 stack is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

2024-11-02 8:13 UTC: The maintenance of europe-west3 stack has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

2024-11-02 8:30 UTC: The announced partial maintenance of us-east4 stack is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

2024-11-02 8:39 UTC: The maintenance of us-east4 stack has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly.

🎉 This concludes GCP maintenance! Thank you for your patience.

We are investigating job start delays on the https://connection.us-east4.gcp.keboola.com/ stack. We will provide updates as new information becomes available.

Update 06:45 UTC: The platform has stabilized, and the job backlog has been cleared. Thank you for your patience, and we apologize for any inconvenience caused.

We are experiencing problems on our GCP EU stack (https://connection.europe-west3.gcp.keboola.com/). We are deeply sorry for the inconvenience this may cause. In the user interface, you can issue an error alert or task slowdown processing jobs. Next update in 30 minutes.

Oct 28th 21:11 UTC: We're still investigating scheduling issues within our underlying infrastructure. Next update in 30 minutes.

Oct 28th 21:18 UTC: The issue has been resolved, and job listing should now work as expected. Thank you for your patience, and sorry for any inconvenience.

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 30 minutes, around 12:00 UTC. We will update this post with the progress of the partial maintenance.

Update 12:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 12:22 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

2024-10-19 10:26 UTC We have noticed a slowdown in the processing of jobs on the https://connection.eu-central-1.keboola.com stack, but the jobs shouldn't end with an error.

Update 2024-10-19 10:57 UTC The problem has been identified and solved, the platform should be stable again. 

We apologize for the inconvenience.

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS, Azure, and GCP.

This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedules and the stacks affected.

Maintenance of all Azure Stacks – October 26, 2024

During database upgrades there will be a short service disruption on all Azure stacks, including all single-tenant stacks and Azure North Europe multi-tenant stack (connection.north-europe.azure.keboola.com). This will take place on Saturday, October 26, 2024 between 11:30 and 12:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 12:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

• 11:30–12:00 UTC: processing of new jobs stops.

• 12:00–12:15 UTC: service disruption

• 12:15 UTC: processing of jobs starts.

GCP EU and US Stack Maintenance – November 2, 2024

During database upgrades there will be a short service disruption on both GCP multi-tenant stacks. Here is the schedule.

• GCP eu-west3 (connection.europe-west3.gcp.keboola.com): Saturday, November 2, 2024, at 08:00 UTC.

• GCP us-east4 (connection.us-east4.gcp.keboola.com): Saturday, November 2, 2024, at 08:30 UTC.

Effect of the Maintenance

During the above periods, services will be scaled down and the processing of jobs may be delayed. For very brief periods (at around 8:00 UTC and 8:30 UTC, respectively) the service will be unavailable for up to five minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule for eu-west3

• 7:30–8:00 UTC: processing of new jobs stops.

• 8:00–8:10 UTC: service disruption

• 8:15 UTC: processing of jobs starts.

Detailed Schedule for us-east4

• 8:00–8:30 UTC: processing of new jobs stops.

• 8:30–8:40 UTC: service disruption

• 8:45 UTC: processing of jobs starts.

AWS EU and US Stack Maintenance – November 16, 2024

During database upgrades there will be a limited service disruption on our AWS multi-tenant stacks. Here is the schedule.

• AWS us-east-1 (connection.keboola.com): Saturday, November 16, 2024, at 07:00 UTC.

• AWS eu-central-1 (connection.eu-central-1.keboola.com): Saturday, November 16, 2024, at 08:00 UTC.

Effect of the Maintenance

The maintenance is expected to last no longer than 15 minutes, during which jobs may be delayed. While you will be able to log into the Keboola platform, starting new jobs will not be possible during the maintenance. Jobs already running will not be canceled—only delayed. Running data apps or workspaces will not be affected. Scheduled jobs will automatically start after the maintenance is completed.

Read more »

2024-10-14 16:30 UTC  We are observing a small number of instances where errors occur during job creation, and you may encounter the error message: “Decryption failed: Deciphering failed.” As a result, orchestrations may become stuck in the terminate state. If you experience this issue, please contact our support team.

We are actively investigating the situation and will provide an update later this evening.

2024-10-14 21:40 UTC We have successfully identified the affected orchestrations and deployed a fix that automatically terminates them. We now consider this incident resolved. We sincerely apologize once again for the inconvenience caused.

2024-10-07 18:10 UTC - We are investigating issues on north-europe.azure.keboola.com Stack

2024-10-07 18:30 UTC - Issues are caused by Azure networking problems, we are monitoring situation.

We are experiencing problems on our AWS EU stack (https://connection.eu-central-1.keboola.com/). We are deeply sorry for the inconvenience this may cause. In the user interface, you can issue an error alert or task slowdown processing jobs. Next update 30 minut.

Sep 26 08:34 UTC: We identified and fixed an overload on one of our Kubernetes node. All systems are now running normally. We’ve implemented measures to prevent recurrence.

Thank you for your patience.

2024-09-23 13:48 UTC - We are currently investigating potential modifications to the primary key for the FTP and S3 extractors that  occurred around September 13th, 2024. The issue has already been reverted, and we are conducting an analysis. We will provide more information as soon as we have further details.

UPDATE 2024-09-23 16:36 UTC - Our analysis confirms that no projects on single-tenant stacks were affected by the issue. We are continuing with the analysis of multi-tenant stack projects and will provide more information as soon as we have further details.

UPDATE 2024-09-25 9:30 UTC - Our analysis has been completed, and we now have a list of affected configurations. The issue with potential primary key modifications may have impacted not only FTP and S3 extractors but also other components using the Processor Create Manifest. We would like to highlight that not all configurations with this processor were affected.

In cases where a configuration experienced a primary key modification, the key was automatically restored after the next run. However, a small number of configurations did not revert to their original primary key due to duplicate records in the table.

These cases are limited, and the clients affected by this issue will be contacted individually by our support team today with further steps and recommendations.

If you have any questions or concerns, please reach out to our support team.