We experienced a brief outage in the application where the platform was inaccessible due to a UI bug. We promptly addressed the issue by rolling back the latest change, and this resolution has been deployed across all stacks.
It's important to note that no flows, running jobs, or workspaces were affected during this incident.
Regrettably, we were unable to upgrade all necessary databases during the previous planned service disruption. Due to a strict deadline imposed by our service provider (AWS), we must carry out another service disruption for maintenance purposes.
This maintenance will impact both the AWS US and AWS EU stacks.
It is scheduled for Saturday, April 15, 2023,
During this time, Storage jobs will be paused or delayed, and the platform will be unavailable for a brief period (approximately 5 minutes). The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs.
We sincerely apologize for any inconvenience caused and appreciate your understanding.
A limited service disruption on AWS US stack will start at 10:00 a.m. UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.
All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends.
We apologize for any inconvenience caused and thank you for your understanding.
Update 10:00 a.m. UTC: The limited service disruption has begun.
Update 10:10 a.m. UTC: The service disruption has been resolved and the stack is now fully operational.
Thank you for your patience.
We have encountered a brief metadata DB outage in AWS US at 15:07 UTC. Affected services are
This outage may cause some jobs being executed during the outage fail or run twice in parallel.
We're sorry for this inconvenience.
UPDATE 15:39 UTC: All affected jobs were restarted and any duplicate executions were terminated.
A limited service disruption on AWS EU stack will start at 12:00 p.m. UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.
All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends.
We apologize for any inconvenience caused and thank you for your understanding.
Update 12:00 p.m. UTC: The limited service disruption has begun.
Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on March 21st and 22nd.
We anticipate that the limited service disruption will take approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.
During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.
All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.
We apologize for any inconvenience caused and thank you for your understanding.
Azure and Snowflake in Azure regions are reporting general service disruptions. We are closely monitoring the situation and, so far, we have observed only a few symptoms of the issues and the platform operations have not been impacted. Please refer to the status updates of the affected services for more information.
We're sorry for this inconvenience.
We're investigating increased error rate in AWS US stack (connection.keboola.com). Next update in 15 minutes or when new information is available.
UPDATE 04:40 UTC: We have identified and replaced a number of corrupted nodes with healthy ones, and operations are now back to normal. We apologize for the inconvenience caused.
UPDATE 05:40 UTC: This issue appears to be ongoing, and a new symptom has been identified: jobs are taking longer to start than usual, or are getting stuck in a waiting state. The next update will be in 30 minutes.
UPDATE 06:25 UTC: We're still investigating the issue. Next update in 30 minutes.
UPDATE 07:15 UTC: We have found the root cause and we're fixing it.
UPDATE 08:44 UTC: The root cause was fixed and all operations are back to normal.
2023-01-21 13:05 UTC - We have identified delayed jobs processing the Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com) since 08:30 UTC. We have restarted the affected services and all operations are back to normal. We apologize for any inconvenience this may have caused.
In very rare circumstances, some (less than 10 per day) jobs may be delayed by almost exactly two hours in the AWS EU stack (connection.eu-central-1.keboola.com). During this period, the job will be stuck doing nothing for two full hours, and unfortunately terminating the job will not help.
We are currently trying our best to debug and fix an underlying network connectivity issue. If you have any questions or concerns, please reach out to our support.
We are sorry for this inconvenience and will provide an update on this post once we know more or have an ETA of the fix.
Update 2023/01/23 8:40 - We have implemented a fix for the issue and so far there are no occurrences of this issue in the past 12hours. We're continuing to monitor the issue thouroughly.