Failing queue jobs (There is no active transaction)

Following the release deployed at 8:30 UTC March 1st 2024, we encountered a bug that led to some queue jobs failing with the error "There is no active transaction." We addressed this issue by reverting the release at 9:50 UTC, which resolved the problem.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding and patience.

Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS and Azure.

We will perform the maintenance of each stack separately during March and April 2024. This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:

Impact of the Maintenance and Recommended Mitigation:

  • Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
  • Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
  • Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for March 23, April 06, and April 13 to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.

Thank you for your patience and cooperation.

Email Delivery Issues

2024-01-11 13:04 - We are currently investigating a problem related to email delivery from the Keboola platform. We will provide an update as soon as we have more information.

UPDATE 13:19 - We have fixed the underlying issue (invalid SPF record). If you still have issues receiving messages from any Keboola platform please let us know. 

Limited service disruption for AWS EU

A limited service disruption on AWS EU stack will start at 13:00 UTC today Friday, December 29, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 13:00 UTC: The limited service disruption has begun.

Update 13:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Some flows processing longer than usual

Between 9:18 and 10:42 UTC some flows were processing longer than usual with jobs "stuck" in processing or waiting state. This was caused be a recent update which was already reverted. The issue affected all stacks but only a small portion of jobs. We sincerely apologize for the inconvenience.

Event triggers in AWS US not working

We are investigating event triggers not working and also not visible in UI in AWS US stack (https://connection.keboola.com). No flows based on event triggers are being executed at the moment. Time based schedules are working as expected as well as event triggers in other stacks.

Update 13:35 UTC: We identified the root cause and will now proceed to restore event triggers configurations.

(Resolved) Update 14:05 UTC: We restored event triggers configurations, all event triggers are now working as expected and we don't observe any failures regarding even triggered flows. However if you still experience issues please contact us on support to resolve it further.


Data Apps Security Incident: November 13, 2023

We'd like to share an update about a proactive measure we've taken to ensure system security. On Monday, November 13, our diligent testing processes helped us to identify a vulnerability in the Data Apps component of our system. As part of our continuous efforts to strengthen security, we have fortified the protection against unauthorized decryption and access to some parts of our infrastructure that could have occurred under certain conditions.
We paused the provisioning of Data Apps as a precautionary step. By the day after discovering the vulnerability, November 14, our dedicated engineering team had comprehensively examined and effectively addressed the issue, ensuring the highest standards of security and reliability for our services. Following this, we thoroughly reviewed the configurations and logs of all Data Apps and found no suspicious activity.

If you have any questions or need more information, we encourage you to contact us at our support center: https://help.keboola.com/management/support.

Job processing slowing down on Azure NE stack

2023-11-03 14:40 UTC We're experiencing job processing slowing down on Azure NE stack (https://connection.north-europe.azure.keboola.com) due to heavy load. We apologise for the inconvenience.

Update 2023-11-03 15:10 UTC We have managed to find the cause and are working on a fix.

Update 2023-11-03 15:40 UTC We managed to mitigate the main cause. It will take some time for the platform to return to normal. 

Update 2023-11-03 16:05 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.