We're experiencing degraded Snowflake performance affecting all operations in the EU region.
This problem can affect our users in the following ways:
- Job execution time may be longer than usual.
- Jobs are randomly failing with the error message "Result not found".
This is similar to the issue we observed last week, which was caused by the Snowflake Cloud Service Layer in EU region being overloaded.
We have reported it to Snowflake support and will keep you posted as soon as we have an update.
UPDATE Mar 3, 10:43 CET - The performance of Snowflake operations has currently improved. We're still working with Snowflake support on mitigating this issue permanently.
UPDATE Mar 4, 19:50 CET - We were hit again today with degraded performance of Snowflake's Cloud Service Layer. Between 3:30 PM - 5 PM CET we observed a large increase in compilation time of Snowflake queries. After 5 PM it returned back to normal. We updated Snowflake with our latest observations and are hopeful for a quick resolution. We will update here with any more information within the next 12 hours.
UPDATE Mar 5, 06:35 CET - We hit again Snowflake performance degradation in last six hours. Snowflake is working on issue mitigation with highest severity. We will provide an update in three hours.
UPDATE Mar 5, 09:49 CET - Performance degradation peaks still occurs. We are in touch with Snowflake on resolution. We will provide an update in three hours.
UPDATE Mar 5, 13:43 CET - There has been no change of performance for the past few hours and its degradation still occurs. We will provide an update in three hours.
UPDATE Mar 5, 16:18 CET - We've been working with Snowflake support on mitigating the performance degradation. From our monitoring it seems that Keboola platform in EU region is now stabilised (since around 14:00). We will provide next update in three hours.
UPDATE Mar 5, 16:43 CET - The issue is a highest priority case with Snowflake support. We're closely collaborating on finding a permanent solution to the performance degradation. Status update from Snowflake:
Snowflake acknowledges the performance degradation issue reported by Keboola Czech SRO since February 24, 2020 and tracks it under the case #00097987 as a critical incident.
Snowflake Support together with Snowflake Engineering are continuously collaborating and conducting an in-depth investigation to identify the root cause and provide a viable solution to you on priority.
At this time, additional resources have been allocated in the Snowflake Services Layer which should show improved performance values.
We will provide next update in three hours.
UPDATE Mar 5, 20:55 CET - The performance is now stabilised and jobs are running as expected. We are monitoring the situation and We will provide next update in 12 hours.
UPDATE Mar 6, 09:23 CET - The performance has worsened slightly during the night job peak. We are monitoring the situation and will provide next update in 3 hours.
UPDATE Mar 6, 12:55 CET - The Snowflake Engineering team made some optimizations of the Cloud Service Layer for our account. Since 7:40 AM CET there have been no "Result not found" errors and job performance should return to normal state.
We will continue monitoring the situating and will provide another update in 6 hours.
UPDATE Mar 6, 21:45 CET - All operations are back to normal. Thanks for your patience and understanding.