Azure North Europe stack - Billing API problems and job delays

We are investigating a brief outage of our Billing API on the Azure North Europe stack (https://connection.north-europe.azure.keboola.com/) between 23:03 and 23:19 UTC. Non–Pay-As-You-Go projects were not affected.

Update 23:35 UTC: The service disruption also caused job delays across all projects in this stack during the affected period. All operations have since returned to normal. We apologize for the inconvenience.


Planned partial maintenance on Saturday, March 1, 2025 for all Azure stacks

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 30 minutes, around 12:00 UTC. We will update this post with the progress of the partial maintenance.

Update 12:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 12:13 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

Scheduled Partial Maintenance of all Azure stacks – March 1, 2025

We would like to inform you about the planned maintenance of all Keboola stacks hosted on Azure.

During the database upgrades there will be a short service outage on all Azure stacks, including all single-tenant stacks and Azure North Europe multi-tenant stack (connection.north-europe.azure.keboola.com). This will take place on Saturday, March 1, 2025 between 11:30 and 12:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 12:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

  • 11:30–12:00 UTC: processing of new jobs stops.
  • 12:00–12:15 UTC: service enhancement period.
  • 12:15 UTC: processing of jobs resumes.


Scheduled Partial Maintenance for Keboola AWS, Azure, and GCP Stacks – October and November 2024

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS, Azure, and GCP.

This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedules and the stacks affected.

Maintenance of all Azure Stacks – October 26, 2024

During database upgrades there will be a short service disruption on all Azure stacks, including all single-tenant stacks and Azure North Europe multi-tenant stack (connection.north-europe.azure.keboola.com). This will take place on Saturday, October 26, 2024 between 11:30 and 12:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 12:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

  • 11:30–12:00 UTC: processing of new jobs stops.

  • 12:00–12:15 UTC: service disruption

  • 12:15 UTC: processing of jobs starts.

GCP EU and US Stack Maintenance – November 2, 2024

During database upgrades there will be a short service disruption on both GCP multi-tenant stacks. Here is the schedule.

Effect of the Maintenance

During the above periods, services will be scaled down and the processing of jobs may be delayed. For very brief periods (at around 8:00 UTC and 8:30 UTC, respectively) the service will be unavailable for up to five minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule for eu-west3

  • 7:30–8:00 UTC: processing of new jobs stops.

  • 8:00–8:10 UTC: service disruption

  • 8:15 UTC: processing of jobs starts.

Detailed Schedule for us-east4

  • 8:00–8:30 UTC: processing of new jobs stops.

  • 8:30–8:40 UTC: service disruption

  • 8:45 UTC: processing of jobs starts.

AWS EU and US Stack Maintenance – November 16, 2024

During database upgrades there will be a limited service disruption on our AWS multi-tenant stacks. Here is the schedule.

Effect of the Maintenance

The maintenance is expected to last no longer than 15 minutes, during which jobs may be delayed. While you will be able to log into the Keboola platform, starting new jobs will not be possible during the maintenance. Jobs already running will not be canceled—only delayed. Running data apps or workspaces will not be affected. Scheduled jobs will automatically start after the maintenance is completed.

Issues across all Azure stacks

We're investigating issues across all Azure stacks. Next update in 15 minutes or when new information is available.

UPDATE 14:35 CEST: The issue seems to be resolved, we're still evaluating the impact. Next update in 30 minutes or when new information is available.

UPDATE 14:55 CEST: The outage was caused by routine maintenace of databases in Auzure. All running jobs affected during this outage should restart automatically. Some of the affected Storage jobs may be executed twice and in very rare cases when using incremental load without a primary key this could lead to data duplication.

We're sorry for this inconvenience, we will be taking measures to decrease impact of future maintenance events. 

For any additional questions, please contact support.

Limited service disruption for AWS US and EU stacks on Friday, December 29

Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on December 29:

  • 13:00–14:00 UTC (14:00–15:00 CET) for the AWS EU stack
  • 15:00–16:00 UTC (07:00–08:00 PDT) for the AWS US stack

We anticipate the duration of the disruption will be approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.

During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be paused, but will resume after the upgrade. The platform will be unavailable for approximately 5 minutes. The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs or Queue v1 jobs.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.

We apologize for any inconvenience caused and thank you for your understanding.

Brief UI inaccessibility across all stacks

We experienced a brief outage in the application where the platform was inaccessible due to a UI bug. We promptly addressed the issue by rolling back the latest change, and this resolution has been deployed across all stacks.

It's important to note that no flows, running jobs, or workspaces were affected during this incident.

Planned service maintenance on April 15th in AWS US and AWS EU stacks

Regrettably, we were unable to upgrade all necessary databases during the previous planned service disruption. Due to a strict deadline imposed by our service provider (AWS), we must carry out another service disruption for maintenance purposes.

This maintenance will impact both the AWS US and AWS EU stacks.

It is scheduled for Saturday, April 15, 2023,

  • between 07:00 and 08:00 UTC (09:00 and 10:00 CEST) for the AWS EU stack, and
  • between 09:00 and 10:00 UTC (02:00 and 03:00 PDT) for the AWS US stack.

During this time, Storage jobs will be paused or delayed, and the platform will be unavailable for a brief period (approximately 5 minutes). The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs.

We sincerely apologize for any inconvenience caused and appreciate your understanding.