We are investigating stucked jobs since Oct 2 11:00 UTC on https://connection.eu-central-1.keboola.com/ stack.
18:47 UTC - Stucked jobs were unblocked. We continue to monitor the situation. Everything should be operational now and running as expected.
]]>14:08 UTC: We are investigating created jobs stuck to start processing in AWS US and EU stacks. The next update in 15 minutes.
14:23 UTC: We found the root cause, rolled back the previous working version and newly created job now should process as expected. We are still trying to push previously created jobs (created within 1 hour ago) to the processing state. The next update in 15 minutes.
14:50 UTC: We have released the stuck jobs in US stack, we continue to release the stuck jobs in EU stack. Next update in 30 minutes.
15:15 UTC: All stuck jobs in EU have been released. Everything is operational now and running as expected.
]]>10:10 UTC: We are investigating newly created transformations (in last 24 hours) that if run fails with an error `Unrecognized option "blocks" under "container"`. Next update in 15 minutes.
10:30 UTC [resolved]: We have identified the root cause and revert the latest changes of UI. From now on if a user creates a new transformation it will run without an error. However already created transformation are still affected, and manual update of the transformation is needed. In such case on the transformation detail page go the raw config editing page by appending "/raw" to the transformation detail page url. Then remove the topmost "blocks" array from the configuration and save it.
]]>07:36 UTC: We are experiencing higher error rate of Jobs API, we are investigating it. Next update in 15 minutes.
07:52 UTC [resolved]: We identified the root cause - the projects migration running in the background caused higher load to our metadata database. This led to slower api response and in some cases timeout of the api request. However this didn't affect any running jobs nor running new ones. Everything is running as expected and fully operational now.
]]>Orchestrations, flows, and running jobs will be generally delayed, but not interrupted.
We'll update this post when the maintenance is finished.
UPDATE 09:06 UTC: The maintenance is taking longer than expected. Next update in 30 minutes or when new information is available.
UPDATE 09:33 UTC: The maintenance has successfully finished and the stack is now fully operational.
UPDATE 10:30 UTC: Unfortunately the database needs another immediate upgrade. We'll be turning on maintenance back on shortly. Next update in 30 minutes.
We're sorry for any inconvenience and thanks for your understanding.
UPDATE 11:00 UTC: We are starting second round of maintenance on the AWS EU stack. Stack should be back up and running within 90 minutes.
UPDATE 12:33 UTC: The maintenance has successfully finished and the stack is now fully operational.2022-09-22 17:50 UTC - We are investigating issue with our Billing Api on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/). Non Pay As You Go projects should not be affected.
Next update in 30 minutes.
2022-09-22 18:20 UTC - The issue was caused by heavy load on the underlying database during parallel migration of some projects to new Queue. Services are now returning to normal operations. It is possible that some orchestrations will have their schedule delayed. We are very sorry for any inconvenience.
We’re sorry to let you know that the previously announced maintenance on September 24th will have more of impact on the platform operations than previously thought. Due to unforeseen circumstances, we’re unable to ensure the previously announced scope of the maintenance, and we need to turn off the platform completely for about 90 minutes.
The AWS EU maintenance will start at 9:30 AM CET, and we expect it to finish by 11 AM CET.
During the maintenance, you will not be able to access your data and projects. All network connections will be terminated by the "HTTP 503 – down for maintenance" status message.
All running tasks will be monitored by us and restarted in case of any interruption. Orchestrations, flows, and running jobs will be generally delayed, but not interrupted. However, feel free to re-schedule your Saturday orchestrations to avoid this maintenance window.
We apologize for any inconvenience and thank you for your understanding.
2022-09-07 11:35 UTC - We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.
2022-09-07 12:25 UTC - After our investigation, problem starts at 10:00 UTC. It is caused by outage of our Billing Api. Orchestrations that should have be scheduled after this time, are delayed in the order of minutes or tens of minutes. This applies to standard projects.
Orchestrations of Pay As You Go projects are not scheduled at the moment.
Next update in 60 minutes.
2022-09-07 14:00 UTC - We implemented a hotfix and the situation is returning back to normal. Orchestrations should be scheduled properly now. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)
We are monitoring a situation, next update in two hours.
2022-09-07 17:10 UTC - All orchestrations are executed without delay. We apologize for any inconvenience.
2022-09-07 10:55 UTC - We are investigating problems with our Billing Api on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/). This is caused by degraded performance of one of the Azure services. Non Pay As You Go projects should not be affected.
Next update in 30 minutes.
2022-09-07 12:10 UTC - Problems with one of the Azure services still persist and our Billing Api is still unavailable. We are waiting for service recovery by Azure Support.
All of Pay As You Go projects are affected:
Next update in 60 minutes.
2022-09-07 13:15 UTC - After our investigation, problem starts at 10:00 UTC. Azure service is still not working properly, but we implemented fix to avoid this situation until Azure Support resolve the root cause.
We keep monitoring the situation closely. At the moment Billing Api service is available, jobs and orchestrations are running.
Next update in 120 minutes.
2022-09-07 17:10 UTC - All Keboola Connection services are running normally. The incident is resolved. Problems were caused by the outage of Azure CosmosDB (more information on https://status.azure.com/en-gb/status)
We apologize for any inconvenience.
]]>Synchronous actions, a mechanism behind some UI features, like testing credentials or listing available databases on a distant server, were affected by a bug and were not working properly from 12:50 UTC until 13:25 UTC when the revert of defective release was finished and the functionality is back to normal since then.
]]>2022-08-20 14:57 UTC - we are investigating some jobs failing on internal error. Next update in 30 minutes.
2022-08-20 15:30 UTC - we see the internal error is caused by one of the internal components failing to call Azure API, however don't know the root cause. We restarted one of the instances and see no error for now. We continue monitoring the issue.
On 24th September 2022 between 10:00 CET and 10:30 CET, https://connection.eu-central-1.keboola.com will undergo planned maintenance during which one of our internal databases will be upgraded. As a result, any loads or unloads to storage will be paused during that time for up to 30 minutes. Any currently running jobs will continue to run. However, they may be delayed by up to 30 minutes as well.
2022-08-15 10:40 UTC - We are investigating component job delays in connection.keboola.com. Next update when new information will be available or in hour.
2022-08-15 11:14 UTC - We have identified the root cause of the problem and we are working on a solution. Next update when new information will be available or in hour.
2022-08-15 14:04 UTC - We've added another worker to help with the workload and the processing times has returned to normal
2022-08-15 15:32 UTC - We are investigating reoccurrence of the issue causing jobs stuck in Created state. Next update when new information will be available or in hour.
2022-08-15 15:59 UTC - The reoccurrence was resolved. We keep monitoring the situation closely, but at the moment job runtimes should be back to the normal.
2022-08-06 16:35 UTC - We are investigating higher error rates and latencies in connection.north-europe.azure.keboola.com stack. Next update in 30 minutes.
2022-08-06 17:20 UTC - Higher errors rates are resolved and latencies are back to normal. Schedules might be delayed, the backlog is processed.
2022-08-06 17:46 UTC - Backlogs is cleared. This incident has been resolved.
2022-08-05 00:40 UTC - We are investigating component job delays in connection.keboola.com. Next update when new information will be available or in hour.
UPDATE 2022-08-05 00:55 UTC - We have identified the root cause and replaced the faulty instances.
UPDATE 2022-08-05 01:00 UTC - All operations are back to normal. We're sorry for this inconvenience.
2022-08-04 12:05 UTC We are investigating internal job errors. Next update in 30 minutes.
2022-08-04 12:36 UTC We experienced two waves of internal errors, between 11:50-11:57 and 12:00-12:07 UTC. The root cause has been removed and I don't see any other errors in the reporting. We continue to monitoring the situation. Next update in 30 minutes.
2022-08-04 13:06 UTC All Keboola Connection services are running normally. The incident is resolved.
We apologize for any inconvenience.
2022-08-04 09:40 UTC We are investigating bug, which causing error pages rendered as html source code.
2022-08-04 09:55 UTC Issue is now resolved all systems are working normally.
We are sorry for the inconvenience.
We are experiencing an issue with possibility of tables with _tmp suffix might be dropped.
If you manipulate with a table <X> and there also exist a table with name <X>_tmp, the table <X>_tmp could be accidentally dropped.
These tables (<X> and <X>_tmp) have to be in the same bucket and your project has to have Snowflake backend. We found the root cause and fix will be deployed in 2 hours.
Affected tables will be restored.
2022-08-02 17:00 UTC - We have deployed the fix of this issue and fixed affected tables. The issue is now resolved. We contact affected customers directly.
We are sorry for the inconvenience.
]]>2022-08-02 7:00 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.
2022-08-02 7:30 UTC After our investigation, problem starts at 6:00 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:15 UTC and all orchestrations are scheduled properly.
2022-07-27 11:12 UTC - Snowflake transformations does not respect selected warehouse size during input mapping which might lead to slower processing times. We are working on a fix which might be deployed in 4 hours.
2022-07-27 14:59 UTC - The fix was not yet deployed due to unexpected complications. We are working on it and the fix should be deployed in 4 hours.
2022-07-27 17:34 UTC - The fix was deployed. All Snowflake transformations are now using correct Warehouse size for execution. The issue is now resolved.
]]>We are experiencing an issue with possibility of tables with _tmp suffix might be dropped. These tables have to be in the same bucket, and there have to exist a table with the same name, just without the _tmp suffix. We found the root cause and we are working on a fix which might be deployed in 4 hours.
2022-07-27 11:17 UTC - We have deployed the fix of this issue. Affected customers were contacted directly. We'll undrop affected tables.
2022-07-27 16:34 UTC - We have fixed affected tables. The issue is now resolved. We are sorry for the inconvenience.
Starting on August 8th we will resume migrations to the new Job Queue in our Azure North Europe stack.
This is a follow up to the previous round of migration, and all relevant information can be found in our previous post.
If you're not sure if your project is already on the new Job Queue, go to orchestrations in your project and if the url ends with "orchestrations-v2" then your project is already on the new Job Queue.
If you have any questions regarding the migration, please contact us via support@keboola.com
Today at 07:38 (UTC) a new version (1.9.1) of PostgreSQL data destination was released which introduced a bug causing some configurations to write no data. The component version was rolled back to 1.9.0 at 13:48 (UTC) and everything should work as expected.
If you still encounter similar symptoms please let us know.
We're sorry for this inconvenience.
]]>We apologize for any inconvenience.
]]>We are aware of a bug in the UI that prevents tables in the input mapping list from being visible. We are in the process of reverting the release causing it and will find a fix afterward.
UPDATE 18:15 CET: The release introducing the bug was reverted so it does not happen again. You may need to reload the browser. The bug was introduced today at 16:04 CET.
2022-07-02 7:45 UTC We are investigating delayed orchestrations on Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com). Next update in 30 minutes.
2022-07-02 8:40 UTC After our investigation, problem starts at 6:40 UTC. Orchestrations that should have be scheduled after this time, were delayed in the order of minutes or tens of minutes. Situation returned back to normal at 7:45 UTC and all orchestrations are scheduled properly.
We are sorry for the inconvenience.
2022-06-23 22:30 UTC: We're investigating cases of jobs not processing in the US stack (https://connection.keboola.com/). Next update in 30 minutes.
2022-06-23 07 23:00 UTC: The situation is now under control and our services are running normally. Processing jobs will start to gradually speed up.
We are sorry for the inconvenience.
]]>2022-06-23 16:15 CET We detected a problem with non-function use cases in the Flow Builder tool at https://connection.north-europe.azure.keboola.com between 13:00 CET and 15:30 CET.
The situation is now under control and our service is running normally.
There was an increased error rate for components using keboola.processor-create-manifest in configurations. This error affected only component configurations which were using empty enclosure in CSV Settings.
Affected component may include data sources like AWS S3, FTP, etc.
We are sorry for any inconvenience. Please feel free to restart your jobs manually.
2022-06-20 16:44 UTC - We are investigating increased error rate of some component using keboola.processor-create-manifest.
2022-06-21 07:22 UTC - We identified a root cause and continue working on a fix.
2022-06-21 12:30 UTC - Incident is resolved, last occurrence of the error was at 12:13 UTC.
]]>