Keboola Status

We would like to inform you about the planned maintenance of Keboola stacks hosted on GCP.

This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedules and the stacks affected.

GCP EU and US Stack Maintenance – February 8, 2025

During database upgrades there will be a short service disruption on both GCP multi-tenant stacks. Here is the schedule.

• GCP europe-west3 (connection.europe-west3.gcp.keboola.com): Saturday, February 8, 2025, at 08:00 UTC.

• GCP us-east4 (connection.us-east4.gcp.keboola.com): Saturday, February 8, 2025, at 08:00 UTC.

Effect of the Maintenance

During the above periods, services will be scaled down and the processing of jobs may be delayed. For very brief periods (at around 8:00 UTC) the service will be unavailable for up to five minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule for europe-west3 and us-east4

• 7:30–8:00 UTC: processing of new jobs stops.

• 8:00–8:10 UTC: service disruption

• 8:15 UTC: processing of jobs starts.

Effect of the Maintenance

The maintenance is expected to last no longer than 15 minutes, during which jobs may be delayed. While you will be able to log into the Keboola platform, starting new jobs will not be possible during the maintenance. Jobs already running will not be canceled—only delayed. Running data apps or workspaces will not be affected. Scheduled jobs will automatically start after the maintenance is completed.

Read more »

We would like to inform you that our Buffer API is being deprecated as part of our commitment to providing you with the best tools and services. Effective 1.3.2025, the Buffer API will no longer be available.

To ensure a seamless transition, we are introducing Data Streams, full replacement and powerful and improved service designed to enhance your data integration experience. You can learn more about Data Streams and its benefits HERE.

Why the change?

• Enhanced Performance: Faster and more reliable data streaming
• Improved Functionality: Supports more use cases with advanced features.
• Streamlined Experience: Fresh UI with more intuitive and flexible integration process.

How to migrate:

We’ve created a migration tool to help you move current Buffer API endpoints to Data Streams quickly and easily. Our support team is also here to provide guidance and support throughout the process.

What you need to do:

1. Manually migrate existing Buffer API endpoint to Data Streams - Create new Data Streams, copy the stream URL and replace the current Buffer API endpoint within your integration with it
2. Contact our support team - we will run a script to migrate old Buffer API endpoints to new Data Streams so you can just copy paste the Data Stream URL and replace the current Buffer API endpoint within your integration with it

You can read more about Data Streams and how to set it up HERE.

Need help?

Our support team is here to help you every step of the way. If you have any questions or concerns about the migration, feel free to reach out to us.

Thank you for your understanding and for being a valued part of the Keboola community. We’re excited for you to experience the benefits of Data Streams!

We are experiencing problem with jobs data replication on AWS EU stack (https://connection.eu-central-1.keboola.com). Jobs are being triggered and run properly but they don't appear in jobs list across whole UI.

Investigation is under way. We will post any updates available.

We are sorry for any inconvenience this may have caused and thank you for your patience.

2024-12-18 14:35 UTC: We have identified and fixed the issue with the component responsible for job data replication. New jobs are now displaying correctly in the lists, and all jobs that ran during the outage period have been successfully backfilled.

We are registering Buffer API outage on AWS US stack. The API is currently not receiving any requests which also affects other platform features, like AI suggestions.

Investigation is under way. We will post any updates available.

We are sorry for any inconvenience this may have caused and thank you for your patience.

2024-12-18 15:15 UTC: We have discovered that the data storage behind the Buffer API was corrupted. While the service is now fully operational, it had not been receiving any incoming requests since 2024-12-17 23:00 UTC. Additionally, Buffer endpoint configurations were lost due to the data corruption. All customers with configured Buffer API integrations will be contacted by their account executives to explain the situation in more detail.

We are investigating failing Storage jobs with the error: "Insufficient privileges to operate on account 'KEBOOLA'" on the AWS US East stack (connection.keboola.com).

Currently, it is not possible to create Snowflake workspaces.

We apologize for the inconvenience and will provide an update in 30 minutes.

UPDATE 2024-12-15 15:33 UTC

We have discovered the root cause and are working on a solution. The next update will be provided as soon as it is resolved or in 30 minutes.

UPDATE 2024-12-15 15:55 UTC

The issue has been identified and fixed. All projects are now operating normally.

We are sorry for any inconvenience this may have caused and thank you for your patience.

We noticed that from about 12:30 CET they stopped working Data Apps on GCP stacks:

• connection.europe-west3.gcp.keboola.com
• connection.us-east4.gcp.keboola.com

We apologize for the inconvenience; we are currently investigating the issue and will provide an update in 30 minutes.

UPDATE 2024-12-11 13:01 UTC: We have identified the root cause, which only affected configurations with authorization enabled or a secret value set. This caused failures during the decryption process. We have fixed the permissions required for decryption in the runtime job, and all systems are now operating normally. We sincerely apologize for the inconvenience.

2024-11-20 08:00 UTC: Since 04:00 AM, we have been experiencing a high error rate on connection.us-east4.gcp.keboola.com affecting Snowflake operations. This includes Snowflake transformations, storage import/export jobs, and other processes, resulting in intermittent application errors.

2024-11-20 09:40 UTC: We're still waiting for confirmation from Snowflake.

2024-11-20 12:00 UTC: We're still waiting for confirmation from Snowflake.

2024-11-20 14:00 UTC: We're still waiting for confirmation from Snowflake.

2024-11-20 15:00 UTC: We haven't seen an error since 11:20 UTC and Snowflake have confirmed a fix on their side. All operations are back to normal. This is the last update. 

We're sorry for the inconvenience and thanks for your patience.

The announced partial maintenance of connection.keboola.com and connection.eu-central-1.keboola.com will start in one hour at 07:00 UTC. We will update this post with the progress of the partial maintenance for each stack.

Update 07:00 UTC: The scheduled partial maintenance for connection.keboola.com has begun.

Update 07:15 UTC: The maintenance of connection.keboola.com has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience. Maintenance of connection.eu-central-1.keboola.com will start at 08:00 UTC.

Update 08:00 UTC: The scheduled partial maintenance for connection.eu-central-1.keboola.com has begun.

13.11.2024 2:16 UTC - Due to an incident on Snowflake, the jobs on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/) are executing very slowly. So far, there doesn't seem to be any errors. The symptoms include jobs running longer than usual, workspaces not starting, table preview not loading data.

The cause of the issue on Snowflake side is not yet known - feel free to view the details at https://status.snowflake.com/incidents/kjd4lpptzmkh.

Update 2:48 UTC - Snowflake incident is still in progress.

Update 3:10 UTC - Snowflake incident is still in progress.

Update 4:35 UTC - The cause of the incident has been identified as an outage of 3rd party provider. 

Update 5:59 UTC - The incident has been identified as Azure connectivity issue and published here https://azure.status.microsoft/en-us/status. There is no ETA available.

Update 7:33 UTC - While the Azure incident is still not resolved, the Snowflake performance is now back to normal. The processing of jobs and other operations of Keboola platform is also back to normal. We will continue to monitor the situation closely.

Update 8:16 UTC - There are still delays in processing jobs and finishing flows. We're working on a fix. 

Update 9:28 UTC - All jobs should now be processing normally. The Azure incident is still not resolved however, so errors may still re-appear

Update 10:50 UTC - All operations are back to normal. 

We are sorry for this inconvenience and thanks for your patience.

On November 12, between 8:00 and 19:00 UTC we have experienced flow failing with error "Cannot process orchestration configuration: Unrecognized option 'isFake'" . It was due to an recent update on the UI, we have reverted to the previous working version and it working as expected. The error affected only flows created or edited within the above timeframe. We have fixed the root cause and fixed most of the affected flow configurations.

However, if you still experience the problem (since the unrecognized option was stored in the configuration of the flow) the option may have to removed manually. To do so, please go to "debug mode" mode of the Flow configuration (on the flow detail page click on three dots -> debug mode) and remove any "isFake": true/false property from the flow configuration. If you are unsure about the process, please contact our support.