Job processing slowing down on Azure NE stack

2023-11-26 3:19 UTC - The processing of jobs in Azure NE stack (https://connection.north-europe.azure.keboola.com) has been slowed down, we're investigating the issue.

2023-11-26 4:15 UTC - Slowdown of starting new jobs was caused by heavy load. The backlog of jobs has been cleared, and new jobs are processing with no delays. We apologise for the inconvenience.

Some jobs are failing with OAuth Broker deprecation notice

2023-10-11 5:54 UTC - Some runs of component using OAuth Broker are failing with User errors "OAuth Broker v2 has been deprecated". We are investigating the issue.

2023-10-11 6:30 UTC - We have identified the cause of the issue. Yesterday at 13:20 UTC, a new UI version was deployed that contained a bug in configuration editor.

During the configuration editing of components that are using the OAuth Broker, the version paremeter was of the broker 3 was inadvertently deleted.


It affected only configrations that were created or modified between 2023-10-10 13:20 UTC and 2023-10-11 6:00 UTC. Jobs of these configurations are ending with User Error "OAuth Broker v2 has been deprecated on September 30, 2019."


UI changes were reverted, but your failing configurations need to be fixed manually.

Go to the all version history in your configuration and use "Restore" button to revert changes you made. It will also revert OAuth configuration back to use newer broker v3. New configuration modifications won't break the settings anymore.

Second option to fix the issue is reauthorization of OAuth in your configuration.

We apologize for any inconvenience caused.


Delayed orchestrations on Azure North Europe stack

2022-11-29 17:55 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

Update 2022-11-29 18:40 UTC - We have deployed a fix and the orchestration schedules will gradually catch on. Next update in 1 hour.

Update 2022-11-29 19:27 UTC - Orchestration schedules are now on time. The incident is no resolved, but we'll keep monitoring the situation. We apologize for the inconvenience.




Stuck jobs in US stack

We are investigating stucked jobs since Oct 8 1:13 UTC on https://connection.keboola.com/ stack.

Update Oct 8, 3:00 UtC - Stucked jobs were unblocked. We continue to monitor the situation.

We are very sorry for any inconvenience.

Delayed orchestrations on Azure North Europe stack

2022-09-07 11:35 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-09-07 12:25 UTC - After our investigation, problem starts at 10:00 UTC. It is caused by outage of our Billing Api. Orchestrations that should have be scheduled after this time, are delayed in the order of minutes or tens of minutes. This applies to standard projects.

Orchestrations of Pay As You Go projects are not scheduled at the moment.

Next update in 60 minutes.

2022-09-07 14:00 UTC - We implemented a hotfix and the situation is returning back to normal. Orchestrations should be scheduled properly now. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)

We are monitoring a situation, next update in two hours.

2022-09-07 17:10 UTC - All orchestrations are executed without delay. We apologize for any inconvenience.


Azure North Europe stack - Pay-as-you-go projects - Billing api problems

2022-09-07 10:55 UTC - We are investigating problems with our Billing Api on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/). This is caused by degraded performance of one of the Azure services. Non Pay As You Go projects should not be affected.

Next update in 30 minutes.

2022-09-07 12:10 UTC - Problems with one of the Azure services still persist and our Billing Api is still unavailable. We are waiting for service recovery by Azure Support.

All of Pay As You Go projects are affected:

  • Your credit balance is temporary on zero value
  • Orchestration jobs are not scheduled at the moment
  • You cannot manually execute any component

Next update in 60 minutes.

2022-09-07 13:15 UTC - After our investigation, problem starts at 10:00 UTC. Azure service is still not working properly, but we implemented fix to avoid this situation until Azure Support resolve the root cause.

We keep monitoring the situation closely. At the moment Billing Api service is available, jobs and orchestrations are running.

Next update in 120 minutes.

2022-09-07 17:10 UTC - All Keboola Connection services are running normally. The incident is resolved. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)

We apologize for any inconvenience.

Delayed orchestrations on Azure North Europe stack

2022-08-02 7:00 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-08-02 7:30 UTC  After our investigation, problem starts at 6:00 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:15 UTC and all orchestrations are scheduled properly.


We are sorry for the inconvenience.

Delayed orchestrations on Azure North Europe stack

2022-07-02 7:45 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-07-02 8:40 UTC  After our investigation, problem starts at 6:40 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:45 UTC and all orchestrations are scheduled properly.

We are sorry for the inconvenience.