Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS and Azure.

We will perform the maintenance of each stack separately during March and April 2024. This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:

Impact of the Maintenance and Recommended Mitigation:

  • Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
  • Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
  • Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for March 23, April 06, and April 13 to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.

Thank you for your patience and cooperation.

List of tables in buckets may not work correctly

Today at 15:30 UTC we noticed a problem with the listing tables in Storage. Tables fail to display only to new users in the project. There can be related issues such as not being able to load data to workspace. All of which apply only to new users. We are seeing this problem across all stacks and regions. 

We are working on a fix, next update in 30 min.

UPDATE 16:10 We have identified the root cause and are working on a fix. Next update in 2 hours.

UPDATE 19:00 The incident is now resolved, and tables are displayed correctly in storage for all users.

We apologize for the inconvenience.

Support Service Unavailability

Due to the inoperability of the Atlassian Service Desk In the morning of January 18, 2024 our support services are temporarily unavailable.

We will provide an update as soon as services have returned to normal.

Many apologies for this inconvenience.

<January 18, 2024 12:30 CET>. Support systems have resumed normal operations

Email Delivery Issues

2024-01-11 13:04 - We are currently investigating a problem related to email delivery from the Keboola platform. We will provide an update as soon as we have more information.

UPDATE 13:19 - We have fixed the underlying issue (invalid SPF record). If you still have issues receiving messages from any Keboola platform please let us know. 

Limited service disruption for AWS US

A limited service disruption on AWS EU stack will start at 15:00 UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends.

We apologize for any inconvenience caused and thank you for your understanding.

Update 15:00 UTC: The limited service disruption has begun.

Update 15:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Limited service disruption for AWS EU

A limited service disruption on AWS EU stack will start at 13:00 UTC today Friday, December 29, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 13:00 UTC: The limited service disruption has begun.

Update 13:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Limited service disruption for AWS US and EU stacks on Friday, December 29

Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on December 29:

  • 13:00–14:00 UTC (14:00–15:00 CET) for the AWS EU stack
  • 15:00–16:00 UTC (07:00–08:00 PDT) for the AWS US stack

We anticipate the duration of the disruption will be approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.

During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be paused, but will resume after the upgrade. The platform will be unavailable for approximately 5 minutes. The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs or Queue v1 jobs.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.

We apologize for any inconvenience caused and thank you for your understanding.

Some flows processing longer than usual

Between 9:18 and 10:42 UTC some flows were processing longer than usual with jobs "stuck" in processing or waiting state. This was caused be a recent update which was already reverted. The issue affected all stacks but only a small portion of jobs. We sincerely apologize for the inconvenience.