Today since 12:00 UTC we suffer jobs as well as sync actions failing to start due to bug introduced in the last deploy. We are now reverting to the previous working version. 

Update 13:15 UTC (resolved) : We have successfully reverted backend to the previous working version and platform is back operational now.

Update 13:30 UTC: Upon further investigation and damage analysis, we found that Queue v2 jobs were delayed in starting because of this incident. That could result in delays of up to one hour.

2024-04-25 02:23 UTC - We experienced another incident on connection.eu-central-1.keboola.com between 01:30 UTC and 02:30 UTC. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

We noticed a slowdown in our AWS EU stack’s job queue from 12:00 to 14:00 UTC, due to a temporary service performance issue. We sincerely apologize for any inconvenience this may have caused.

Our team has resolved the problem and we are taking steps to prevent future occurrences. Thank you for your understanding and patience.

13:07 UTC: We're investigating an issue where import to Storage shows "Storage API: The new primary key cannot be created; duplicate values in primary key columns exist" error. 
Next update in 15 minutes. 

13:24 UTC: We've rolled back the last release. We're investigating impact of the issue. 

13:54 UTC: We're still investigating the impact. 

14:34 UTC: We're still investigating the impact. So far it seems to have only impacted a very limited number of customer jobs. We will reach out to the affected customers separately as soon as we finish the impact investigation.  

15:09 UTC: We've finished investigating the impact and we're reaching out to the affected customers. 

AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance will start today, April 13, 2024, at 12:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

Maintenance should take less than three hours.

We apologize for any inconvenience caused and thank you for your understanding.

Update 12:00 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance started.

Update 13:30 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance is finished and the stack is now fully operational.

AWS us-east-1 stack (connection.keboola.com) maintenance will start today, April 06, 2024, 07:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

We apologize for any inconvenience caused and thank you for your understanding.

Update 07:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance started.

7:20 UTC: We are investigating an error when installing a python package in the transformations.
You may see errors such as: Job "XXXXXXX" ended with a user error "Failed to install package: io".
Next update in 15 min.

UPDATE 7:38 UTC: We have rolled back a previous version and all operations are back normal. We're sorry for this inconvenience. 

Update 11:00 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) maintenance started.

Update 11:52 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) service disruption has been resolved and the stack is now fully operational. Thank you for your patience.

We would like to remind you of the upcoming scheduled maintenance for Keboola stacks hosted on AWS and Azure.

9:21 UTC: We are investigating jobs not starting on AWS EU this morning. Next update in 30 minutes.

9:57 UTC: We are still investigating the issue. Next update in 30 minutes.

10:24 UTC: We have identified stuck job that was causing other jobs not starting and proceed to unblock it. The platform is operational now and all stuck jobs have started running. We continue investigating the root cause. 

We apologize for any inconvenience this may cause and appreciate your understanding.

Update 2024-03-16 10:10 CET: The limited service disruption has begun.

Update 2024-03-16 10:25 CET: The limited service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Please be advised that there will be a one-hour service disruption next week. Check carefully to see if it will affect your projects.

2024-03-08 08:20 UTC: We are currently investigating an increase in latency within our job listings and delays in job execution on connection.eu-central-1.keboola.com. We will provide an update as soon as more information becomes available.

2024-03-08 09:05 UTC: We have identified the root cause and are actively working on a resolution. Job processing has not been affected since 08:20. We are continuing to investigate increased latency in Jobs and Flows page listings.

2024-03-08 09:15 UTC: The incident has been resolved. Latencies have returned to normal, and the platform is fully operational. We will continue to monitor the situation.

Following the release deployed at 8:30 UTC March 1st 2024, we encountered a bug that led to some queue jobs failing with the error "There is no active transaction." We addressed this issue by reverting the release at 9:50 UTC, which resolved the problem.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding and patience.

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS and Azure.

Today at 15:30 UTC we noticed a problem with the listing tables in Storage. Tables fail to display only to new users in the project. There can be related issues such as not being able to load data to workspace. All of which apply only to new users. We are seeing this problem across all stacks and regions. 

We are working on a fix, next update in 30 min.

UPDATE 16:10 We have identified the root cause and are working on a fix. Next update in 2 hours.

UPDATE 19:00 The incident is now resolved, and tables are displayed correctly in storage for all users.

Due to the inoperability of the Atlassian Service Desk In the morning of January 18, 2024 our support services are temporarily unavailable.

We will provide an update as soon as services have returned to normal.

Many apologies for this inconvenience.

<January 18, 2024 12:30 CET>. Support systems have resumed normal operations

2024-01-11 13:04 - We are currently investigating a problem related to email delivery from the Keboola platform. We will provide an update as soon as we have more information.

UPDATE 13:19 - We have fixed the underlying issue (invalid SPF record). If you still have issues receiving messages from any Keboola platform please let us know. 

2024-04-01 06:35 UTC - We experienced incident on https://connection.keboola.com. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

A limited service disruption on AWS EU stack will start at 15:00 UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends.

We apologize for any inconvenience caused and thank you for your understanding.

Update 15:00 UTC: The limited service disruption has begun.

Update 15:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

A limited service disruption on AWS EU stack will start at 13:00 UTC today Friday, December 29, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 13:00 UTC: The limited service disruption has begun.

Update 13:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on December 29:

• 13:00–14:00 UTC (14:00–15:00 CET) for the AWS EU stack
• 15:00–16:00 UTC (07:00–08:00 PDT) for the AWS US stack

We anticipate the duration of the disruption will be approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.

During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be paused, but will resume after the upgrade. The platform will be unavailable for approximately 5 minutes. The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs or Queue v1 jobs.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.

We apologize for any inconvenience caused and thank you for your understanding.

Between 9:18 and 10:42 UTC some flows were processing longer than usual with jobs "stuck" in processing or waiting state. This was caused be a recent update which was already reverted. The issue affected all stacks but only a small portion of jobs. We sincerely apologize for the inconvenience.

Since 14:00 UTC, we have been experiencing slower job processing in the Azure North Europe stack (https://connection.north-europe.azure.keboola.com). We are currently investigating the issue.

Update 16:20 UTC we have rolled back a previous version of one of our services that was causing the delays and all operations are back normal. We're sorry for this inconvenience. 

We are investigating event triggers not working and also not visible in UI in AWS US stack (https://connection.keboola.com). No flows based on event triggers are being executed at the moment. Time based schedules are working as expected as well as event triggers in other stacks.

Update 13:35 UTC: We identified the root cause and will now proceed to restore event triggers configurations.

(Resolved) Update 14:05 UTC: We restored event triggers configurations, all event triggers are now working as expected and we don't observe any failures regarding even triggered flows. However if you still experience issues please contact us on support to resolve it further.

• It might leads to errors in the UI
• Job processing is not affected
  

We experienced a brief outage in the application where the platform was inaccessible due to a UI bug. We promptly addressed the issue by rolling back the latest change, and this resolution has been deployed across all stacks.

It's important to note that no flows, running jobs, or workspaces were affected during this incident.

2023-11-26 3:19 UTC - The processing of jobs in Azure NE stack (https://connection.north-europe.azure.keboola.com) has been slowed down, we're investigating the issue.

2023-11-26 4:15 UTC - Slowdown of starting new jobs was caused by heavy load. The backlog of jobs has been cleared, and new jobs are processing with no delays. We apologise for the inconvenience.