Event triggers in AWS US not working

We are investigating event triggers not working and also not visible in UI in AWS US stack (https://connection.keboola.com). No flows based on event triggers are being executed at the moment. Time based schedules are working as expected as well as event triggers in other stacks.

Update 13:35 UTC: We identified the root cause and will now proceed to restore event triggers configurations.

(Resolved) Update 14:05 UTC: We restored event triggers configurations, all event triggers are now working as expected and we don't observe any failures regarding even triggered flows. However if you still experience issues please contact us on support to resolve it further.

Investigating higher latency through all stacks

As of 29 November 13:45 UTC, we are investigating higher latency for some requests in the through all stacks.
  • It might leads to errors in the UI
  • Job processing is not affected
We'll be doing a rollback to the previous version. Next update in 30 min.

UPDATE 2023-11-29 13:20 UTC - All operations are back to normal and everything is fully working.

Brief UI inaccessibility across all stacks

We experienced a brief outage in the application where the platform was inaccessible due to a UI bug. We promptly addressed the issue by rolling back the latest change, and this resolution has been deployed across all stacks.

It's important to note that no flows, running jobs, or workspaces were affected during this incident.

Job processing slowing down on Azure NE stack

2023-11-26 3:19 UTC - The processing of jobs in Azure NE stack (https://connection.north-europe.azure.keboola.com) has been slowed down, we're investigating the issue.

2023-11-26 4:15 UTC - Slowdown of starting new jobs was caused by heavy load. The backlog of jobs has been cleared, and new jobs are processing with no delays. We apologise for the inconvenience.

Data Apps Security Incident: November 13, 2023

We'd like to share an update about a proactive measure we've taken to ensure system security. On Monday, November 13, our diligent testing processes helped us to identify a vulnerability in the Data Apps component of our system. As part of our continuous efforts to strengthen security, we have fortified the protection against unauthorized decryption and access to some parts of our infrastructure that could have occurred under certain conditions.
We paused the provisioning of Data Apps as a precautionary step. By the day after discovering the vulnerability, November 14, our dedicated engineering team had comprehensively examined and effectively addressed the issue, ensuring the highest standards of security and reliability for our services. Following this, we thoroughly reviewed the configurations and logs of all Data Apps and found no suspicious activity.

If you have any questions or need more information, we encourage you to contact us at our support center: https://help.keboola.com/management/support.

Job processing slowing down on Azure NE stack

2023-11-03 14:40 UTC We're experiencing job processing slowing down on Azure NE stack (https://connection.north-europe.azure.keboola.com) due to heavy load. We apologise for the inconvenience.

Update 2023-11-03 15:10 UTC We have managed to find the cause and are working on a fix.

Update 2023-11-03 15:40 UTC We managed to mitigate the main cause. It will take some time for the platform to return to normal. 

Update 2023-11-03 16:05 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.

Delayed job logs on AWS US

2023-10-30 15:17 UTC We're experiencing log delays on running jobs on AWS US stack (https://connection.keboola.com) due to heavy load. 

Update 2023-10-30 17:34 UTC We have managed to find the cause and are working on a fix.

Update 2023-10-30 19:50 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.

Some jobs are failing with OAuth Broker deprecation notice

2023-10-11 5:54 UTC - Some runs of component using OAuth Broker are failing with User errors "OAuth Broker v2 has been deprecated". We are investigating the issue.

2023-10-11 6:30 UTC - We have identified the cause of the issue. Yesterday at 13:20 UTC, a new UI version was deployed that contained a bug in configuration editor.

During the configuration editing of components that are using the OAuth Broker, the version paremeter was of the broker 3 was inadvertently deleted.

It affected only configrations that were created or modified between 2023-10-10 13:20 UTC and 2023-10-11 6:00 UTC. Jobs of these configurations are ending with User Error "OAuth Broker v2 has been deprecated on September 30, 2019."

UI changes were reverted, but your failing configurations need to be fixed manually.

Go to the all version history in your configuration and use "Restore" button to revert changes you made. It will also revert OAuth configuration back to use newer broker v3. New configuration modifications won't break the settings anymore.

Second option to fix the issue is reauthorization of OAuth in your configuration.

We apologize for any inconvenience caused.

Deprecated Twitter Data Source

Due to the changes at Twitter we are no longer able to provide it as a Data Source component. 
We will be introducing an X Data Source if demand calls for it.

If you have any questions, or would like to push for introducing an X Data Source then please let us know at support@keboola.com.