Limited Service Disruption: AWS EU's Queue V1 Unavailable on Saturday, March 16

Please be advised that there will be a one-hour service disruption next week. Check carefully to see if it will affect your projects.

Affected service: Queue Version 1 on connection.eu-central-1.keboola.com

Here are the details. Due to a necessary database upgrade to our AWS EU stack (connection.eu-central-1.keboola.com), there will be a limited service disruption of Queue Version 1 (V1) on Saturday, March 16, from 10:00 to 11:00 CET.

This will affect only a minority of projects that still use Queue V1. Projects on Queue V2 will not be affected.

Does your project use Queue Version 1? To find out, go to your Project Settings page and look for the queuev2 feature. If you don't see it, that means that your project does indeed use the old Queue V1, and that this service disruption will affect you.

Effects of the disruption

  1. Queue V1 and Orchestrations (Queue V1) processing will be temporarily halted.
  2. New Queue V1 jobs will be delayed until the upgrade is completed.
  3. All running Queue V1 jobs will be paused, and then restarted after the upgrade is completed.

We apologize for this inconvenience. We will appreciate your understanding as we strive to maintain and improve our service quality. If you have any questions or need further assistance, please contact our support team.


Job processing delays in AWS eu-central-1

2024-03-08 08:20 UTC: We are currently investigating an increase in latency within our job listings and delays in job execution on connection.eu-central-1.keboola.com. We will provide an update as soon as more information becomes available.

2024-03-08 09:05 UTC: We have identified the root cause and are actively working on a resolution. Job processing has not been affected since 08:20. We are continuing to investigate increased latency in Jobs and Flows page listings.

2024-03-08 09:15 UTC: The incident has been resolved. Latencies have returned to normal, and the platform is fully operational. We will continue to monitor the situation.

Failing queue jobs (There is no active transaction)

Following the release deployed at 8:30 UTC March 1st 2024, we encountered a bug that led to some queue jobs failing with the error "There is no active transaction." We addressed this issue by reverting the release at 9:50 UTC, which resolved the problem.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding and patience.

Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS and Azure.

We will perform the maintenance of each stack separately during March and April 2024. This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:

Impact of the Maintenance and Recommended Mitigation:

  • Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
  • Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
  • Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for March 23, April 06, and April 13 to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.

Thank you for your patience and cooperation.

List of tables in buckets may not work correctly

Today at 15:30 UTC we noticed a problem with the listing tables in Storage. Tables fail to display only to new users in the project. There can be related issues such as not being able to load data to workspace. All of which apply only to new users. We are seeing this problem across all stacks and regions. 

We are working on a fix, next update in 30 min.

UPDATE 16:10 We have identified the root cause and are working on a fix. Next update in 2 hours.

UPDATE 19:00 The incident is now resolved, and tables are displayed correctly in storage for all users.

We apologize for the inconvenience.

Email Delivery Issues

2024-01-11 13:04 - We are currently investigating a problem related to email delivery from the Keboola platform. We will provide an update as soon as we have more information.

UPDATE 13:19 - We have fixed the underlying issue (invalid SPF record). If you still have issues receiving messages from any Keboola platform please let us know. 

Limited service disruption for AWS US

A limited service disruption on AWS EU stack will start at 15:00 UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends.

We apologize for any inconvenience caused and thank you for your understanding.

Update 15:00 UTC: The limited service disruption has begun.

Update 15:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.