Degraded Performance on GCP EU Stack

Sep 20 15:15 UTC: We are experiencing degraded performance on our GCP EU stack (https://connection.europe-west3.gcp.keboola.com/). We are deeply sorry for the inconvenience this may cause and appreciate your patience as we work through it. We will provide further updates as soon as we have more information.

Sep 22 18:53 UTC update:We have gained additional understanding of the performance degradation. The root cause seems to be in intermittent slowdown of query execution on Snowflake. While the execution of single query is not delayed significantly, the delays do accumulate to noticeable slowdowns of minutes on transformations consisting of multiple queries and even more for entire flows. We're in touch with Snowflake support in the process of uncovering all the technical details.

The symptoms of the performance degradation include longer running times of jobs - especially on Snowflake transformations, but Data Source jobs and Data Destination jobs are also affected, because they load/unload data from a Snowflake database. The performance degradation occurrence is random and somewhat time dependent so not all flows are affected in the same manner. The performance degradation does not cause any errors.

Sep 24 07:10 UTC update: We have now confirmed the root cause to be in slower execution of Snowflake queries at certain times. We have implemented a temporary resource increase to improve the situation. This means that you should see improved job run times. We're still working with Snowflake support on the solution.

Oct 1 7:54 UTC update: We are still working on the resolution together with Snowflake support. The temporary resource increase is still in place. This means that that the situation is contained and overall stack performance should be acceptable, but not perfect. 

At this moment we don't have a solution ready, nor an ETA for one. Thank you for your understanding and patience. We will provide further updates as soon as we have more information.

Issues across all Azure stacks

We're investigating issues across all Azure stacks. Next update in 15 minutes or when new information is available.

UPDATE 14:35 CEST: The issue seems to be resolved, we're still evaluating the impact. Next update in 30 minutes or when new information is available.

UPDATE 14:55 CEST: The outage was caused by routine maintenace of databases in Auzure. All running jobs affected during this outage should restart automatically. Some of the affected Storage jobs may be executed twice and in very rare cases when using incremental load without a primary key this could lead to data duplication.

We're sorry for this inconvenience, we will be taking measures to decrease impact of future maintenance events. 

For any additional questions, please contact support.

New and improved Jobs API endpoint

We've introduced a new and improved Jobs API endpoint, /search/jobs, which replaces the current /jobs endpoint. 

The new endpoint should be near drop-in replacement, supporting the same parameters and the same results, but with improved performance, especially for larger projects and complex filters. 

The old endpoint is now considered deprecated and will be removed some time in the future (will be announced separately).  

A major difference is that the new endpoint returns results from a secondary database, which is synchronized with a slight delay. 

This means a new job may not be shown right away after it's created and job detail may contain slightly outdated data if it was updated recently. The delay should be couple seconds at max during normal operation. 

Other difference is the new endpoint will return maximum 500 items per page.

See the Job Queue API documentation for more details.

Sync actions on Azure stacks are failing

2024-08-19 08:05 - We are investigating an issue with synchronous action on connection.north-europe.azure.keboola.com stack

2024-08-19 08:30 - We are still investigating the issue, problem is on all Azure instances.

2024-08-19 09:00 - Issue is resolved, synchronous actions are operational on all azure stacks.

Unable to view the project consumption on AWS US stack

2024-08-13 11:02 UTC - We are experiencing a problem with the project consumption dashboard. Displaying the detail page ends with an error and the dashboard is not available. 

2024-08-13 11:32 UTC - Unfortunately, the problem still persists, and we have not been able to find the root cause. We are still actively solving the problem.

2024-08-13 11:51 UTC - The problem has been fixed and the consumption dashboard is available again. 

We are apologize for the inconvenience.

Stuck jobs in AWS EU stack

2024-08-13 09:06 UTC - We are investigating an issue where jobs are delayed, the problem has so far only affected the AWS EU stack https://connection.eu-central-1.keboola.com. 

2024-08-13 09:35 UTC - The problem still persists, we have not yet been able to determine the cause of the problem.

2024-08-13 09:48 UTC - We have managed to discover and fix the problem, the scheduled jobs are currently delayed, we are now stabilizing the problem and will give next update when the stack is stable.

2024-08-13 10:00 UTC - The problem is solved completely and the stack is fully stable. We are sorry for the inconvenience.

Issue with Restoring Typed Tables Using Time Travel

2024-08-09 12:05 UTC - We are currently investigating an issue affecting the "Restore Table Using Time Travel" feature in Snowflake projects with Typed Tables. When attempting to restore a table, it is created successfully but cannot be exported or previewed. We will provide further updates as more information becomes available.


2024-08-12 05:46 UTC - We have identified the root cause of the issue and have prepared a fix. The deployment is scheduled for today. We will provide an update once the fix has been released.

2024-08-13 13:54 UTC - The fix has been released. Now, when you create a new table using the Time Travel restore feature, it will function as expected.