Keboola Status

We are experiencing problem with jobs data replication on AWS EU stack (https://connection.eu-central-1.keboola.com). Jobs are being triggered and run properly but they don't appear in jobs list across whole UI.

Investigation is under way. We will post any updates available.

We are sorry for any inconvenience this may have caused and thank you for your patience.

2024-12-18 14:35 UTC: We have identified and fixed the issue with the component responsible for job data replication. New jobs are now displaying correctly in the lists, and all jobs that ran during the outage period have been successfully backfilled.

We are registering Buffer API outage on AWS US stack. The API is currently not receiving any requests which also affects other platform features, like AI suggestions.

Investigation is under way. We will post any updates available.

We are sorry for any inconvenience this may have caused and thank you for your patience.

2024-12-18 15:15 UTC: We have discovered that the data storage behind the Buffer API was corrupted. While the service is now fully operational, it had not been receiving any incoming requests since 2024-12-17 23:00 UTC. Additionally, Buffer endpoint configurations were lost due to the data corruption. All customers with configured Buffer API integrations will be contacted by their account executives to explain the situation in more detail.

We are investigating failing Storage jobs with the error: "Insufficient privileges to operate on account 'KEBOOLA'" on the AWS US East stack (connection.keboola.com).

Currently, it is not possible to create Snowflake workspaces.

We apologize for the inconvenience and will provide an update in 30 minutes.

UPDATE 2024-12-15 15:33 UTC

We have discovered the root cause and are working on a solution. The next update will be provided as soon as it is resolved or in 30 minutes.

UPDATE 2024-12-15 15:55 UTC

The issue has been identified and fixed. All projects are now operating normally.

We are sorry for any inconvenience this may have caused and thank you for your patience.

We noticed that from about 12:30 CET they stopped working Data Apps on GCP stacks:

• connection.europe-west3.gcp.keboola.com
• connection.us-east4.gcp.keboola.com

We apologize for the inconvenience; we are currently investigating the issue and will provide an update in 30 minutes.

UPDATE 2024-12-11 13:01 UTC: We have identified the root cause, which only affected configurations with authorization enabled or a secret value set. This caused failures during the decryption process. We have fixed the permissions required for decryption in the runtime job, and all systems are now operating normally. We sincerely apologize for the inconvenience.

2024-11-20 08:00 UTC: Since 04:00 AM, we have been experiencing a high error rate on connection.us-east4.gcp.keboola.com affecting Snowflake operations. This includes Snowflake transformations, storage import/export jobs, and other processes, resulting in intermittent application errors.

2024-11-20 09:40 UTC: We're still waiting for confirmation from Snowflake.

2024-11-20 12:00 UTC: We're still waiting for confirmation from Snowflake.

2024-11-20 14:00 UTC: We're still waiting for confirmation from Snowflake.

2024-11-20 15:00 UTC: We haven't seen an error since 11:20 UTC and Snowflake have confirmed a fix on their side. All operations are back to normal. This is the last update. 

We're sorry for the inconvenience and thanks for your patience.

The announced partial maintenance of connection.keboola.com and connection.eu-central-1.keboola.com will start in one hour at 07:00 UTC. We will update this post with the progress of the partial maintenance for each stack.

Update 07:00 UTC: The scheduled partial maintenance for connection.keboola.com has begun.

Update 07:15 UTC: The maintenance of connection.keboola.com has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience. Maintenance of connection.eu-central-1.keboola.com will start at 08:00 UTC.

Update 08:00 UTC: The scheduled partial maintenance for connection.eu-central-1.keboola.com has begun.

13.11.2024 2:16 UTC - Due to an incident on Snowflake, the jobs on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/) are executing very slowly. So far, there doesn't seem to be any errors. The symptoms include jobs running longer than usual, workspaces not starting, table preview not loading data.

The cause of the issue on Snowflake side is not yet known - feel free to view the details at https://status.snowflake.com/incidents/kjd4lpptzmkh.

Update 2:48 UTC - Snowflake incident is still in progress.

Update 3:10 UTC - Snowflake incident is still in progress.

Update 4:35 UTC - The cause of the incident has been identified as an outage of 3rd party provider. 

Update 5:59 UTC - The incident has been identified as Azure connectivity issue and published here https://azure.status.microsoft/en-us/status. There is no ETA available.

Update 7:33 UTC - While the Azure incident is still not resolved, the Snowflake performance is now back to normal. The processing of jobs and other operations of Keboola platform is also back to normal. We will continue to monitor the situation closely.

Update 8:16 UTC - There are still delays in processing jobs and finishing flows. We're working on a fix. 

Update 9:28 UTC - All jobs should now be processing normally. The Azure incident is still not resolved however, so errors may still re-appear

Update 10:50 UTC - All operations are back to normal. 

We are sorry for this inconvenience and thanks for your patience.

On November 12, between 8:00 and 19:00 UTC we have experienced flow failing with error "Cannot process orchestration configuration: Unrecognized option 'isFake'" . It was due to an recent update on the UI, we have reverted to the previous working version and it working as expected. The error affected only flows created or edited within the above timeframe. We have fixed the root cause and fixed most of the affected flow configurations.

However, if you still experience the problem (since the unrecognized option was stored in the configuration of the flow) the option may have to removed manually. To do so, please go to "debug mode" mode of the Flow configuration (on the flow detail page click on three dots -> debug mode) and remove any "isFake": true/false property from the flow configuration. If you are unsure about the process, please contact our support.

12.11.2024 10:30 UTC - Since approximately 10:00 UTC, there is an issue running Jobs on Azure North Europe stack https://connection.north-europe.azure.keboola.com/. Jobs cannot be run (including scheduled ones). We have identified the root cause and working on a fix.

Other stacks are not affected.

Update 10:57 UTC - The root cause is resolved now and new jobs are now starting properly. However there might be flows stuck in processing or terminating state not able to finish. We're working on a fix for this.

Update 11:27 UTC - The fix will be deployed within the next 30 minutes.

Update 12:10 UTC - We are sorry for the delay, the fix will be released any minute now. 

Update 12:21 UTC - All stuck jobs are now processed and all flows work as expected. 

We are sorry for this inconvenience and thank you for your patience.

On November 8th, 2024, between 8:17 AM and 8:28 AM UTC, we experienced a brief outage in our notification service on all GCP stacks due to a deployment misconfiguration. The issue was identified and promptly resolved, and the service has been fully operational since 8:28 AM UTC. During this period, no notifications were sent.

We apologize for any inconvenience this may have caused and appreciate your understanding.