Limited service disruption for AWS EU

A limited service disruption on AWS EU stack will start at 13:00 UTC today Friday, December 29, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 13:00 UTC: The limited service disruption has begun.

Update 13:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Some flows processing longer than usual

Between 9:18 and 10:42 UTC some flows were processing longer than usual with jobs "stuck" in processing or waiting state. This was caused be a recent update which was already reverted. The issue affected all stacks but only a small portion of jobs. We sincerely apologize for the inconvenience.

Event triggers in AWS US not working

We are investigating event triggers not working and also not visible in UI in AWS US stack (https://connection.keboola.com). No flows based on event triggers are being executed at the moment. Time based schedules are working as expected as well as event triggers in other stacks.

Update 13:35 UTC: We identified the root cause and will now proceed to restore event triggers configurations.

(Resolved) Update 14:05 UTC: We restored event triggers configurations, all event triggers are now working as expected and we don't observe any failures regarding even triggered flows. However if you still experience issues please contact us on support to resolve it further.


Data Apps Security Incident: November 13, 2023

We'd like to share an update about a proactive measure we've taken to ensure system security. On Monday, November 13, our diligent testing processes helped us to identify a vulnerability in the Data Apps component of our system. As part of our continuous efforts to strengthen security, we have fortified the protection against unauthorized decryption and access to some parts of our infrastructure that could have occurred under certain conditions.
We paused the provisioning of Data Apps as a precautionary step. By the day after discovering the vulnerability, November 14, our dedicated engineering team had comprehensively examined and effectively addressed the issue, ensuring the highest standards of security and reliability for our services. Following this, we thoroughly reviewed the configurations and logs of all Data Apps and found no suspicious activity.

If you have any questions or need more information, we encourage you to contact us at our support center: https://help.keboola.com/management/support.

Job processing slowing down on Azure NE stack

2023-11-03 14:40 UTC We're experiencing job processing slowing down on Azure NE stack (https://connection.north-europe.azure.keboola.com) due to heavy load. We apologise for the inconvenience.

Update 2023-11-03 15:10 UTC We have managed to find the cause and are working on a fix.

Update 2023-11-03 15:40 UTC We managed to mitigate the main cause. It will take some time for the platform to return to normal. 

Update 2023-11-03 16:05 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.

Delayed job logs on AWS US

2023-10-30 15:17 UTC We're experiencing log delays on running jobs on AWS US stack (https://connection.keboola.com) due to heavy load. 

Update 2023-10-30 17:34 UTC We have managed to find the cause and are working on a fix.

Update 2023-10-30 19:50 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.

Deprecated Twitter Data Source

Due to the changes at Twitter we are no longer able to provide it as a Data Source component. 
We will be introducing an X Data Source if demand calls for it.

If you have any questions, or would like to push for introducing an X Data Source then please let us know at support@keboola.com.

Jobs processing AWS EU central stack slowing down

2023-09-13 12:55 UTC - Processing of jobs in AWS EU stack has been slowed down, we're investigating the issue.

2023-09-13 13:43 UTC - We see minor improvements. We're still investigating the issue, next update when new information will be available.

2023-09-13 14:31 UTC - The backlog of jobs has been cleared, and job processing should be functioning properly once more. We had to temporarily disable the "Child Jobs" checkbox on the Jobs page to address the issue. We are continuing to monitor the situation.

Jobs processing AWS EU central stack slowing down

13:10 UTC the processing of jobs in AWS EU stack has been slowed down, we're investigating the issue

13:40 UTC the issue has been resolved, jobs are now being processed normally. We're still investigating the root cause of this issue.

14:30 UTC we're investigating the issue again, since jobs in AWS EU stack are slowed down

15:00 UTC we have found a root cause of incident and preparing fix to mitigate the issue. Next update in 1 hour.

16:00 UTC we are still working on issue mitigation. Next update in 1 hour.

16:55 UTC issue was fixed, all systems are stable now. There is several projects which still have problem to list jobs, we are working to fix this issue. Next update when new information's will be available.

Failing sync actions in all stacks

11:40 UTC - We are investigating failing synchronous actions (check database credentials and similar) in all our stacks since 11:00 UTC.

11:50 UTC - We deployed previous version of affected service, all systems are now operational. We apologize for any inconvenience caused.