Stuck jobs in AWS us-east-1 [resolved]

2022-04-15 10:30 UTC - We are investigation stuck jobs on a new queue. Some jobs will probably end up with an internal error. Next update in 15 min.

2022-04-15 10:50 UTC - The situation was resolved, however 6 jobs were terminated by an internal error with no automatic restart. We apologize for the inconvenience and please feel free to restart your jobs manually.

Affected jobs:

  • job-839828541
  • job-839828546
  • job-839830974
  • job-839831721
  • job-839831773
  • job-839831782



AWS Stacks Maintenance Announcement

A maintenance of AWS stacks Keboola Connection will take place on Saturday, Jan 8th, 2022 and should take less than three hours.

During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

All running tasks will be monitored by us and restarted in case of any interruption. Orchestrations and running transformations will be generally delayed, but not interrupted. However, feel free to re-schedule your saturday's orchestrations to avoid this maintenance window.

High error rate in Developer Portal

Service disruption in AWS US are also causing problems in Devloper Porta (apps-api.keboola.com & components.keboola.com). You may see intermittent errors 5XX, refreshing the page can help. 

AWS acknowledge the service disruption and are active working towards recover. See https://status.aws.amazon.com/ for more details. Once the AWS service disruption is over our services should start running smoothly again. Next update in 60 minutes or when new information is available.

UPDATE 19:10 UTC Service disruption on the AWS US region persists. We continue to monitor the situation. Next update in 2 hours.

UPDATE 21:40 UTC Service disruption on the AWS US region are reducing. Our affected services are showing significant improvement. Next update in 12 hours or as new information is available.

UPDATE Dec 8th, 07:12 UTC Most services in the affected AWS region have already been repaired. Our services are operating normally. Next update in 4 hours or as new information is available.

UPDATE Dec 8th, 15:130 UTC We're sorry for the late update. AWS services have already recovered. Everything should be running without any issues now.

Azure EU Maintenance Announcement

A maintenance of Azure EU Keboola Connection (connection.north-europe.azure.keboola.com) will take place on Saturday, Oct 9th, 2021 from 10:00 CET and should take less than two hours.

During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

Orchestrations and running component jobs should be generally delayed, but not interrupted.  However, feel free to re-schedule your saturday orchestrations to avoid this maintenance window.

Component synchronization down in all regions

Since 2020-11-25 13:45 UTC we have noticed an unfunctional synchronization of components between Keboola Connection and the Developer portal. The problem concerns AWS in the US region. Working in Keboola Connection is not affected by this error. It has an impact on the release and synchronization of components. You also cannot log in to the Developer Portal.

We are going to monitor the situation and keep you posted.

Update Nov 25, 2020 16:11 UTC - We have identified a problem. The outage is on the AWS side.

Update Nov 25, 2020 16:19 UTC - We found that the problem affects both AWS and Azure in all regions. Next update in 30 minutes.

Update Nov 25, 2020 16:59 UTC - Outage on the AWS side persists, see the status page. We continue to monitor the situation. Next update in 1 hour.

Update Nov 25, 2020 18:10 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.

Update Nov 25, 2020 20:15 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.

Update Nov 25, 2020 22:17 UTC - Issues on the AWS side are reducing. Login to the Developer Portal is already working and we are monitoring successful component synchronizations across all regions. Next update on Nov 26, 2020 06:30 UTC 

Update Nov 26, 2020 07:05 UTC - Work on the AWS side to troubleshoot still continues, but the error rate is reduced. We are no longer experiencing any issues with component synchronizations or other Keboola Connection services. We continue to monitor the situation. Next update in 4 hours.

Update Nov 26, 2020 10:23 UTC - Issues on the AWS services are solved. The incident is now resolved.