2023-03-14 8:30 UTC - We are investigating delayed orchestrations on AWS US Keboola Connection stack (https://connection.keboola.com/).
2023-03-14 9:25 UTC - The issue is resolved. Thank you for patience and understanding.
2022-11-29 17:55 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.
Update 2022-11-29 18:40 UTC - We have deployed a fix and the orchestration schedules will gradually catch on. Next update in 1 hour.
Update 2022-11-29 19:27 UTC - Orchestration schedules are now on time. The incident is no resolved, but we'll keep monitoring the situation. We apologize for the inconvenience.
We are investigating stucked jobs since Oct 8 1:13 UTC on https://connection.keboola.com/ stack.
Update Oct 8, 3:00 UtC - Stucked jobs were unblocked. We continue to monitor the situation.
We are very sorry for any inconvenience.
2022-09-07 11:35 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.
2022-09-07 12:25 UTC - After our investigation, problem starts at 10:00 UTC. It is caused by outage of our Billing Api. Orchestrations that should have be scheduled after this time, are delayed in the order of minutes or tens of minutes. This applies to standard projects.
Orchestrations of Pay As You Go projects are not scheduled at the moment.
Next update in 60 minutes.
2022-09-07 14:00 UTC - We implemented a hotfix and the situation is returning back to normal. Orchestrations should be scheduled properly now. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)
We are monitoring a situation, next update in two hours.
2022-09-07 17:10 UTC - All orchestrations are executed without delay. We apologize for any inconvenience.
2022-09-07 10:55 UTC - We are investigating problems with our Billing Api on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/). This is caused by degraded performance of one of the Azure services. Non Pay As You Go projects should not be affected.
Next update in 30 minutes.
2022-09-07 12:10 UTC - Problems with one of the Azure services still persist and our Billing Api is still unavailable. We are waiting for service recovery by Azure Support.
All of Pay As You Go projects are affected:
- Your credit balance is temporary on zero value
- Orchestration jobs are not scheduled at the moment
- You cannot manually execute any component
Next update in 60 minutes.
2022-09-07 13:15 UTC - After our investigation, problem starts at 10:00 UTC. Azure service is still not working properly, but we implemented fix to avoid this situation until Azure Support resolve the root cause.
We keep monitoring the situation closely. At the moment Billing Api service is available, jobs and orchestrations are running.
Next update in 120 minutes.
2022-09-07 17:10 UTC - All Keboola Connection services are running normally. The incident is resolved. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)
We apologize for any inconvenience.
2022-08-02 7:00 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.
2022-08-02 7:30 UTC After our investigation, problem starts at 6:00 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:15 UTC and all orchestrations are scheduled properly.
2022-07-02 7:45 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.
2022-07-02 8:40 UTC After our investigation, problem starts at 6:40 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:45 UTC and all orchestrations are scheduled properly.
We are sorry for the inconvenience.
May 12, 14:00 UTC We are investigating a problem with waiting jobs in AWS US stack https://connection.keboola.com/
We will update this post when we have more information.
May 12, 14:30 UTC All operations are back in normal. The problems were detected only with part of jobs that ran on new queue.
2022-02-02 17:55 UTC We are experiencing delayed indexing of Events in our Storage. We are investigating the issue.
2022-02-02 19:23 UTC Problems with Storage Events delays were caused by increased load to our APIs. At the moment all of delayed events should be already processed and available in events feed. We are monitoring this situation, whether it will not happen again.
Dec 9 2021, 11:38 UTC We are currently investigating an issue regarding corrupted data obtained via our Telemetry Data component (keboola.ex-telemetry-data).
Next update in 60 minutes.
Dec 9 2021, 13:07 UTC We have identified the issue in our telemetry data and fixed it. The issue might cause job with no existing configuration to not be assigned to its actual project telemetry data.
We have modified the component so that it now loads data using full loads only. To ensure that you have the correct telemetry data, all you need to do is run the extractor (or wait for your pipeline to run it). We will re-implement the incremental fetching in the following months.
We are very sorry for any inconvenience caused.