Reminder: Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to remind you of the upcoming scheduled maintenance for Keboola stacks hosted on AWS and Azure.

The announced maintenance of the Keboola platform for both Azure and AWS stacks is scheduled for March and April 2024.

Read more in the original maintenance announcement.

2024-03-16 Jobs not starting in AWS eu-central-1

9:21 UTC: We are investigating jobs not starting on AWS EU this morning. Next update in 30 minutes.

9:57 UTC: We are still investigating the issue. Next update in 30 minutes.

10:24 UTC: We have identified stuck job that was causing other jobs not starting and proceed to unblock it. The platform is operational now and all stuck jobs have started running. We continue investigating the root cause. 

Limited service disruption of Queue V1 for AWS EU

A limited service disruption of Queue V1 on the AWS EU stack (connection.eu-central-1.keboola.com) will begin today, Saturday, March 16, at 10:00 CET, as previously announced. This will impact only a small number of projects that continue to use Queue V1. Projects on Queue V2 will not be affected.  

We apologize for any inconvenience this may cause and appreciate your understanding.

Update 2024-03-16 10:10 CET: The limited service disruption has begun.

Update 2024-03-16 10:25 CET: The limited service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Limited Service Disruption: AWS EU's Queue V1 Unavailable on Saturday, March 16

Please be advised that there will be a one-hour service disruption next week. Check carefully to see if it will affect your projects.

Affected service: Queue Version 1 on connection.eu-central-1.keboola.com

Here are the details. Due to a necessary database upgrade to our AWS EU stack (connection.eu-central-1.keboola.com), there will be a limited service disruption of Queue Version 1 (V1) on Saturday, March 16, from 10:00 to 11:00 CET.

This will affect only a minority of projects that still use Queue V1. Projects on Queue V2 will not be affected.

Does your project use Queue Version 1? To find out, go to your Project Settings page and look for the queuev2 feature. If you don't see it, that means that your project does indeed use the old Queue V1, and that this service disruption will affect you.

Effects of the disruption

  1. Queue V1 and Orchestrations (Queue V1) processing will be temporarily halted.
  2. New Queue V1 jobs will be delayed until the upgrade is completed.
  3. All running Queue V1 jobs will be paused, and then restarted after the upgrade is completed.

We apologize for this inconvenience. We will appreciate your understanding as we strive to maintain and improve our service quality. If you have any questions or need further assistance, please contact our support team.


Job processing delays in AWS eu-central-1

2024-03-08 08:20 UTC: We are currently investigating an increase in latency within our job listings and delays in job execution on connection.eu-central-1.keboola.com. We will provide an update as soon as more information becomes available.

2024-03-08 09:05 UTC: We have identified the root cause and are actively working on a resolution. Job processing has not been affected since 08:20. We are continuing to investigate increased latency in Jobs and Flows page listings.

2024-03-08 09:15 UTC: The incident has been resolved. Latencies have returned to normal, and the platform is fully operational. We will continue to monitor the situation.

Failing queue jobs (There is no active transaction)

Following the release deployed at 8:30 UTC March 1st 2024, we encountered a bug that led to some queue jobs failing with the error "There is no active transaction." We addressed this issue by reverting the release at 9:50 UTC, which resolved the problem.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding and patience.

Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS and Azure.

We will perform the maintenance of each stack separately during March and April 2024. This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:

Impact of the Maintenance and Recommended Mitigation:

  • Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
  • Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
  • Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for March 23, April 06, and April 13 to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.

Thank you for your patience and cooperation.

List of tables in buckets may not work correctly

Today at 15:30 UTC we noticed a problem with the listing tables in Storage. Tables fail to display only to new users in the project. There can be related issues such as not being able to load data to workspace. All of which apply only to new users. We are seeing this problem across all stacks and regions. 

We are working on a fix, next update in 30 min.

UPDATE 16:10 We have identified the root cause and are working on a fix. Next update in 2 hours.

UPDATE 19:00 The incident is now resolved, and tables are displayed correctly in storage for all users.

We apologize for the inconvenience.