Keboola Status

2025-02-08 23:55 UTC We are experiencing issues with the Storage API on our stack. This may cause slower Queue jobs and occasional downtime for the Storage API and Connection administration. The next update will be in 30 minutes.

2025-02-09 0:35 UTC We identified an issue where some parts of our system were overloaded due to a high number of jobs running during the nightly peak after 11 PM. This caused slowdowns and occasional errors in our UI and Storage API and some Queue jobs may have been delayed or failed.

We have increased system capacity to prevent this issue in the future.The incident is now resolved.

We apologize for any inconvenience caused.

2025-02-11 1:44 UTC: We are investigating job execution delays at https://connection.north-europe.azure.keboola.com/. We will provide updates as soon as we have more information.

2025-02-11 2:20 UTC: We have identified that the job execution delays were caused by an underlying node issue. We have replaced the node and are monitoring the situation.

2025-02-11 3:05 UTC: Due to issues with a node, the following problems occurred:

• New jobs created between 01:08 UTC and 01:24 UTC did not start.
• Between 01:08 UTC and 02:35 UTC, notifications from Flows were delayed.
  

All jobs are now running normally, notifications are no longer delayed, and the platform is fully operational.

The incident is resolved. We apologize for any inconvenience.

2024-07-10 12:45 UTC We are experiencing some storage jobs failing with application errors since 12:00 UTC. We are currently investigating the issue and will provide an update within one hour.

2024-07-10 13:45 UTC We are still investigating the issue that is occurring since 12:00 UTC. Our team is working to identify and resolve the problem. We will provide another update within one hour.

2024-07-10 15:00 UTC We have likely identified the root cause of the problem. Our team is preparing to revert the changes that were deployed on the platform earlier today. Once the previous version is deployed, the issue should be resolved. We will provide another update within one hour.

2024-07-10 15:30 UTC The issue has been resolved by deploying the previous version. Jobs created after 15:25 UTC should no longer fail with internal errors. If you observe any issues, please feel free to contact our support team. We apologize for the inconvenience.

2024-07-12 06:50 UTC  We are investigating additional potential impacts of the earlier issue. In very rare cases, it may cause table export mismatches between jobs within the same project. This is not confirmed yet. We will contact affected customers if any are identified.

2024-06-22 7:40 UTC Due to an outage in the Snowflake in one of the regions, Telemetry Data has not been updated since 4:30 UTC and will be delayed for the duration of the outage.

Other functions of our platform should operate normally and remain unaffected by this outage.
This issue affects all of our customers, regardless of the Storage backend they are using (Snowflake, BigQuery, etc.).

2024-06-22 8:40 UTC THe Snowfalke outage has been resolved. Telemetry data has been fully updated and is now up-to-date. Thank you for patience.

2023-11-26 3:19 UTC - The processing of jobs in Azure NE stack (https://connection.north-europe.azure.keboola.com) has been slowed down, we're investigating the issue.

2023-11-26 4:15 UTC - Slowdown of starting new jobs was caused by heavy load. The backlog of jobs has been cleared, and new jobs are processing with no delays. We apologise for the inconvenience.

2023-10-11 5:54 UTC - Some runs of component using OAuth Broker are failing with User errors "OAuth Broker v2 has been deprecated". We are investigating the issue.

2023-10-11 6:30 UTC - We have identified the cause of the issue. Yesterday at 13:20 UTC, a new UI version was deployed that contained a bug in configuration editor.

During the configuration editing of components that are using the OAuth Broker, the version paremeter was of the broker 3 was inadvertently deleted.

It affected only configrations that were created or modified between 2023-10-10 13:20 UTC and 2023-10-11 6:00 UTC. Jobs of these configurations are ending with User Error "OAuth Broker v2 has been deprecated on September 30, 2019."

UI changes were reverted, but your failing configurations need to be fixed manually.

Go to the all version history in your configuration and use "Restore" button to revert changes you made. It will also revert OAuth configuration back to use newer broker v3. New configuration modifications won't break the settings anymore.

Second option to fix the issue is reauthorization of OAuth in your configuration.

2023-03-14 8:30 UTC - We are investigating delayed orchestrations on AWS US Keboola Connection stack (https://connection.keboola.com/).

2023-03-14 9:25 UTC - The issue is resolved. Thank you for patience and understanding.

2022-11-29 17:55 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

Update 2022-11-29 18:40 UTC - We have deployed a fix and the orchestration schedules will gradually catch on. Next update in 1 hour.

Update 2022-11-29 19:27 UTC - Orchestration schedules are now on time. The incident is no resolved, but we'll keep monitoring the situation. We apologize for the inconvenience.

We are investigating stucked jobs since Oct 8 1:13 UTC on https://connection.keboola.com/ stack.

Update Oct 8, 3:00 UtC - Stucked jobs were unblocked. We continue to monitor the situation.

We are very sorry for any inconvenience.

2022-09-07 11:35 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.

2022-09-07 12:25 UTC - After our investigation, problem starts at 10:00 UTC. It is caused by outage of our Billing Api. Orchestrations that should have be scheduled after this time, are delayed in the order of minutes or tens of minutes. This applies to standard projects.

Orchestrations of Pay As You Go projects are not scheduled at the moment.

Next update in 60 minutes.

2022-09-07 14:00 UTC - We implemented a hotfix and the situation is returning back to normal. Orchestrations should be scheduled properly now. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)

We are monitoring a situation, next update in two hours.

2022-09-07 17:10 UTC - All orchestrations are executed without delay. We apologize for any inconvenience.