2023-04-14 02:11 UTC Due to internal incident few jobs ended with incorrect user exception `Component terminated. Possibly due to out of memory error`.
We are very sorry for the inconvenience.
2023-04-13 07:40 UTC We are investigating failing workspace creation on EU stack (connection.eu-central-1.keboola.com). The issue is manifesting as Loading data to workspace failed: Client error:... when you try to create a workspace from python transformation. More information within the hour.
2023-04-13 08:40 UTC We are still investigating root cause. This issue happens only when creating new workspace from python transformation with empty input mapping on connection.eu-central-1.keboola.com. More information within the hour.
2023-04-13 09:15 UTC The service disruption has been resolved and the stack is now fully operational.
We are very sorry for the inconvenience.
2023-04-12 09:50 UTC Table dependency graph shows error message on projects with queue v1. More information within the hour.
Update 2023-04-12 10:43 UTC We have deployed the latest functional version, the problem should be solved by now.
We apologize for the inconvenience.
2023-04-06 12:45 UTC We are investigating failing workspace creation on all stacks. The issue is manifesting as na Application error when you try to create a workspace. The fix is already on the way, we expect the operations to resume in 20 minutes.
2023-04-06 13:05 UTC The service disruption has been resolved and the stack is now fully operational.
We're sorry for this inconvenience.
2023-04-03 10:15 UTC - We are investigating delayed telemetry data. More information within the hour.
2023-04-03 11:30 UTC - Delayed telemetry data on all Keboola Connection stacks have been recorded since approximately 20:00 UTC on March 29. We were able to determine the root cause and perform a backfill. Now all telemetry data tables are up-to-date.
We are very sorry for the inconvenience. If you encounter any discrepancies, please contact us immediately.
2023-03-31 09:50 We are investigating failing workspace (Python, R, SQL) loads on all stacks.
2023-03-31 09:58 Affected are all newly created user workspaces (Python, R, SQL,.. ). A fix will be available soon. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-31 10:24 Problem occurred on when new table was added into workspace. This issue was resolved, we are now working to fix already corrupted workspaces. Workaround now is to remove tables from input mapping and add them again. The next update will be provided in 60 minutes or as soon as new information becomes available.
2023-03-31 11:00 We fixed remaining workspaces and preparing fix to prevent this problem in the future. If you encounter this issue please contact our support and mention this status post.
We sincerely apologize for any inconvenience caused and appreciate your understanding.
2023-03-30 16:22 UTC - We are investigating the delays in job start-up within the https://connection.north-europe.azure.keboola.com stack. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-30 16:54 UTC - The investigation into the cause of the issue is still ongoing. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-30 17:56 UTC - The investigation into the cause of the issue is still ongoing. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-30 18:55 UTC - We have identified and fixed the root cause of the issue. The job backlog has now been cleared. We will continue to monitor the situation to ensure that everything remains stable.
2023-03-30 19:18 UTC - The service disruption has been resolved and the stack is now fully operational.
Regrettably, we were unable to upgrade all necessary databases during the previous planned service disruption. Due to a strict deadline imposed by our service provider (AWS), we must carry out another service disruption for maintenance purposes.
This maintenance will impact both the AWS US and AWS EU stacks.
It is scheduled for Saturday, April 15, 2023,
- between 07:00 and 08:00 UTC (09:00 and 10:00 CEST) for the AWS EU stack, and
- between 09:00 and 10:00 UTC (02:00 and 03:00 PDT) for the AWS US stack.
During this time, Storage jobs will be paused or delayed, and the platform will be unavailable for a brief period (approximately 5 minutes). The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs.
We sincerely apologize for any inconvenience caused and appreciate your understanding.
A limited service disruption on AWS US stack will start at 10:00 a.m. UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.
All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends.
We apologize for any inconvenience caused and thank you for your understanding.
Update 10:00 a.m. UTC: The limited service disruption has begun.
Update 10:10 a.m. UTC: The service disruption has been resolved and the stack is now fully operational.
Thank you for your patience.
We have encountered a brief metadata DB outage in AWS US at 15:07 UTC. Affected services are
- legacy Transformations
- legacy Orchestrations
- projects and jobs running on old Queue V1
This outage may cause some jobs being executed during the outage fail or run twice in parallel.
We're sorry for this inconvenience.
UPDATE 15:39 UTC: All affected jobs were restarted and any duplicate executions were terminated.