Some jobs running two hours longer in AWS EU

In very rare circumstances, some (less than 10 per day) jobs may be delayed by almost exactly two hours in the AWS EU stack (connection.eu-central-1.keboola.com). During this period, the job will be stuck doing nothing for two full hours, and unfortunately terminating the job will not help

We are currently trying our best to debug and fix an underlying network connectivity issue. If you have any questions or concerns, please reach out to our support.

We are sorry for this inconvenience and will provide an update on this post once we know more or have an ETA of the fix.

Update 2023/01/23 8:40 - We have implemented a fix for the issue and so far there are no occurrences of this issue in the past 12hours. We're continuing to monitor the issue thouroughly.


Scheduled Maintenance of AWS EU Stack starting now

As announced previously, we’re starting a planned maintenance on the AWS EU stack. One of our internal databases will be upgraded.

Orchestrations, flows, and running jobs will be generally delayed, but not interrupted. 

We'll update this post when the maintenance is finished. 

UPDATE 09:06 UTC: The maintenance is taking longer than expected. Next update in 30 minutes or when new information is available.

UPDATE 09:33 UTC: The maintenance has successfully finished and the stack is now fully operational.

UPDATE 10:30 UTC: Unfortunately the database needs another immediate upgrade. We'll be turning on maintenance back on shortly. Next update in 30 minutes.

We're sorry for any inconvenience and thanks for your understanding. 

UPDATE 11:00 UTC: We are starting second round of maintenance on the AWS EU stack. Stack should be back up and running within 90 minutes.

UPDATE 12:33 UTC: The maintenance has successfully finished and the stack is now fully operational.

Update on Scheduled Maintenance of AWS EU Stack on 2022-09-24

We’re sorry to let you know that the previously announced maintenance on September 24th will have more of impact on the platform operations than previously thought. Due to unforeseen circumstances, we’re unable to ensure the previously announced scope of the maintenance, and we need to turn off the platform completely for about 90 minutes. 

The AWS EU maintenance will start at 9:30 AM CET, and we expect it to finish by 11 AM CET. 

During the maintenance, you will not be able to access your data and projects. All network connections will be terminated by the "HTTP 503 – down for maintenance" status message.

All running tasks will be monitored by us and restarted in case of any interruption. Orchestrations, flows, and running jobs will be generally delayed, but not interrupted. However, feel free to re-schedule your Saturday orchestrations to avoid this maintenance window.

We apologize for any inconvenience and thank you for your understanding.

PostgreSQL data destination writing no data

Today at 07:38 (UTC) a new version (1.9.1) of PostgreSQL data destination was released which introduced a bug causing some configurations to write no data. The component version was rolled back to 1.9.0 at 13:48 (UTC) and everything should work as expected.

If you still encounter similar symptoms please let us know.

We're sorry for this inconvenience. 

Microsoft Azure services planned maintenance

This applies only to Azure North Europe stack (https://connection.north-europe.azure.keboola.com).

Microsoft Azure will be performing maintenance on database services between 16:00 UTC on Feb 10 2022 and 08:00 UTC on 11 Feb 2022. This maintenance will last 60 to 120 seconds and may happen multiple times in this window. This may cause

  • brief platform unavailability, 
  • job retries, 
  • job errors and
  • very rarely also duplicate job execution

As we do not know when this will happen exactly we're unable to plan a maintenance window. If you encounter any issues due to this maintenance please contact our support. 

Linking buckets from Data catalog errors

Since Jan 20 14:00:00 UTC linking buckets from Data catalog fails with this error message:

Invalid data - async: This field was not expected.

We're currently investigating this issue, next update in 30 minutes.

Update 13:50 UTC: The issue has been fixed and everything is working as expected. 

We're sorry for this inconvenience.

High error rate in AWS US stack

We're investigating high error rate in AWS US (https://connection.keboola.com) stack. Next update in 60 minutes or when new information is available. 

We're sorry for this inconvenience.

UPDATE 15:55 UTC It seems that the root cause is a service disruption in AWS. We're waiting for official confirmation of this issue. You may see intermittent errors (404 or 500), refreshing the page can help. Next update in 60 minutes or when new information is available.

UPDATE 16:25 UTC Service disruption in AWS US region may also cause issues in other Keboola Connection stacks, e. g. when running components jobs. Next update in 60 minutes or when new information is available.

UPDATE 16:55 UTC AWS acknowledge the service disruption and are active working towards recover. See https://status.aws.amazon.com/ for more details. Once the AWS service disruption is over our services should start running smoothly again. Next update in 60 minutes or when new information is available.

UPDATE 18:27 UTC Service disruption on the AWS US region persists. The availability of our services has improved slightly but we are still experiencing errors regarding Workspaces. We continue to monitor the situation. Next update in 2 hours.

UPDATE 21:40 UTC Service disruption on the AWS US region are reducing. Our affected services are showing significant improvement. Next update in 12 hours or as new information is available.

UPDATE Dec 8th, 07:12 UTC Most services in the affected AWS region have already been repaired. Our services are operating normally. Next update in 4 hours or as new information is available.


UPDATE Dec 8th, 15:130 UTC We're sorry for the late update. AWS services have already recovered. Everything should be running without any issues now.


Slow event processing in AWS US

We're investigating possible intermittent slower event processing in AWS US stack (https://connection.keboola.com/). The API response can be delayed up to 2 seconds. This may cause 

  • poor UI response and
  • slowing down of jobs that write metadata (e. g. column datatypes).

This issue does not cause any job failures.

We're sorry for this inconvenience. Next update in 24 hours or as new information is available.

UPDATE Dec 4th, 15:00 UTC We have identified a few possible root causes of this issue and minimized the impact. The situation is now stable, but we're monitoring it closely. 

This is the last update and we'll reopen the communication here only if the situation escalates.