Keboola Status

Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on March 21st and 22nd.

• On Tuesday March 21st at 12:00 pm UTC, the disruption will begin for AWS EU, and
• on Wednesday March 22nd at 10:00 am UTC, it will begin for AWS US.

We anticipate that the limited service disruption will take approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.

During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.

We apologize for any inconvenience caused and thank you for your understanding.

Azure and Snowflake in Azure regions are reporting general service disruptions. We are closely monitoring the situation and, so far, we have observed only a few symptoms of the issues and the platform operations have not been impacted. Please refer to the status updates of the affected services for more information.

• https://status.snowflake.com/incidents/0c76p8pck0y7
  
• https://status.azure.com/en-us/status
  

We're sorry for this inconvenience. 

We're investigating increased error rate in AWS US stack (connection.keboola.com). Next update in 15 minutes or when new information is available. 

UPDATE 04:40 UTC: We have identified and replaced a number of corrupted nodes with healthy ones, and operations are now back to normal. We apologize for the inconvenience caused.

UPDATE 05:40 UTC: This issue appears to be ongoing, and a new symptom has been identified: jobs are taking longer to start than usual, or are getting stuck in a waiting state. The next update will be in 30 minutes.

UPDATE 06:25 UTC: We're still investigating the issue. Next update in 30 minutes.

UPDATE 07:15 UTC: We have found the root cause and we're fixing it. 

UPDATE 08:44 UTC: The root cause was fixed and all operations are back to normal.

2023-01-21 13:05 UTC - We have identified delayed jobs processing the Azure North Europe Keboola Connection stack (https://connection.north-europe.azure.keboola.com) since 08:30 UTC. We have restarted the affected services and all operations are back to normal. We apologize for any inconvenience this may have caused.

In very rare circumstances, some (less than 10 per day) jobs may be delayed by almost exactly two hours in the AWS EU stack (connection.eu-central-1.keboola.com). During this period, the job will be stuck doing nothing for two full hours, and unfortunately terminating the job will not help. 

We are currently trying our best to debug and fix an underlying network connectivity issue. If you have any questions or concerns, please reach out to our support.

We are sorry for this inconvenience and will provide an update on this post once we know more or have an ETA of the fix.

Update 2023/01/23 8:40 - We have implemented a fix for the issue and so far there are no occurrences of this issue in the past 12hours. We're continuing to monitor the issue thouroughly.

Orchestrations, flows, and running jobs will be generally delayed, but not interrupted. 

We'll update this post when the maintenance is finished. 

UPDATE 09:06 UTC: The maintenance is taking longer than expected. Next update in 30 minutes or when new information is available.

UPDATE 09:33 UTC: The maintenance has successfully finished and the stack is now fully operational.

UPDATE 10:30 UTC: Unfortunately the database needs another immediate upgrade. We'll be turning on maintenance back on shortly. Next update in 30 minutes.

We're sorry for any inconvenience and thanks for your understanding. 

UPDATE 11:00 UTC: We are starting second round of maintenance on the AWS EU stack. Stack should be back up and running within 90 minutes.

We’re sorry to let you know that the previously announced maintenance on September 24th will have more of impact on the platform operations than previously thought. Due to unforeseen circumstances, we’re unable to ensure the previously announced scope of the maintenance, and we need to turn off the platform completely for about 90 minutes. 

The AWS EU maintenance will start at 9:30 AM CET, and we expect it to finish by 11 AM CET. 

During the maintenance, you will not be able to access your data and projects. All network connections will be terminated by the "HTTP 503 – down for maintenance" status message.

All running tasks will be monitored by us and restarted in case of any interruption. Orchestrations, flows, and running jobs will be generally delayed, but not interrupted. However, feel free to re-schedule your Saturday orchestrations to avoid this maintenance window.

We apologize for any inconvenience and thank you for your understanding.

2022-08-05 00:40 UTC - We are investigating component job delays in connection.keboola.com.  Next update when new information will be available or in hour.

UPDATE 2022-08-05 00:55 UTC - We have identified the root cause and replaced the faulty instances.

UPDATE 2022-08-05 01:00 UTC - All operations are back to normal. We're sorry for this inconvenience. 

Today at 07:38 (UTC) a new version (1.9.1) of PostgreSQL data destination was released which introduced a bug causing some configurations to write no data. The component version was rolled back to 1.9.0 at 13:48 (UTC) and everything should work as expected.

If you still encounter similar symptoms please let us know.

We're sorry for this inconvenience. 

This applies only to Azure North Europe stack (https://connection.north-europe.azure.keboola.com).

Microsoft Azure will be performing maintenance on database services between 16:00 UTC on Feb 10 2022 and 08:00 UTC on 11 Feb 2022. This maintenance will last 60 to 120 seconds and may happen multiple times in this window. This may cause

• brief platform unavailability, 
• job retries, 
• job errors and
• very rarely also duplicate job execution

As we do not know when this will happen exactly we're unable to plan a maintenance window. If you encounter any issues due to this maintenance please contact our support.