Degraded Snowflake Performance (EU region) - March 2, 2020

We're experiencing degraded Snowflake performance affecting all operations in the EU region.

This problem can affect our users in the following ways:

  • Job execution time may be longer than usual.
  • Jobs are randomly failing with the error message "Result not found".

This is similar to the issue we observed last week, which was caused by the Snowflake Cloud Service Layer in EU region being overloaded.

We have reported it to Snowflake support and will keep you posted as soon as we have an update.

UPDATE Mar 3, 10:43 CET - The performance of Snowflake operations has currently improved. We're still working with Snowflake support on mitigating this issue permanently.

UPDATE Mar 4, 19:50 CET - We were hit again today with degraded performance of Snowflake's Cloud Service Layer. Between 3:30 PM - 5 PM CET we observed a large increase in compilation time of Snowflake queries. After 5 PM it returned back to normal. We updated Snowflake with our latest observations and are hopeful for a quick resolution. We will update here with any more information within the next 12 hours.

UPDATE Mar 5, 06:35 CET  - We hit again Snowflake performance degradation in last six hours. Snowflake is working on issue mitigation with highest severity. We will provide an update in three hours.

UPDATE Mar 5, 09:49 CET  - Performance degradation peaks still occurs. We are in touch with Snowflake on resolution. We will provide an update in three hours.

UPDATE Mar 5, 13:43 CET  - There has been no change of performance for the past few hours and its degradation still occurs. We will provide an update in three hours.

UPDATE Mar 5, 16:18 CET - We've been working with Snowflake support on mitigating the performance degradation. From our monitoring it seems that Keboola platform in EU region is now stabilised (since around 14:00). We will provide next update in three hours. 

UPDATE Mar 5, 16:43 CET - The issue is a highest priority case with Snowflake support. We're closely collaborating on finding a permanent solution to the performance degradation. Status update from Snowflake: 

Snowflake acknowledges the performance degradation issue reported by Keboola Czech SRO since February 24, 2020 and tracks it under the case #00097987 as a critical incident.

Snowflake Support together with Snowflake Engineering are continuously collaborating and conducting an in-depth investigation to identify the root cause and provide a viable solution to you on priority.

At this time, additional resources have been allocated in the Snowflake Services Layer which should show improved performance values.

We will provide next update in three hours. 

UPDATE Mar 5, 20:55 CET - The performance is now stabilised and jobs are running as expected. We are monitoring the situation and We will provide next update in 12 hours.

UPDATE Mar 6, 09:23 CET - The performance has worsened slightly during the night job peak. We are monitoring the situation and  will provide next update in 3 hours.

UPDATE Mar 6, 12:55 CET - The Snowflake Engineering team made some optimizations of the Cloud Service Layer for our account.  Since 7:40 AM CET there have been no "Result not found" errors and job performance should return to normal state.

We will continue monitoring the situating and will provide another update in 6 hours.

UPDATE Mar 6, 21:45 CET - All operations are back to normal. Thanks for your patience and understanding.

GoodData writer failures (US region)

We have encountered a small number of failed jobs in US region between 4:46 AM - 9:17 AM UTC.

Affected jobs finished with Application Error caused by invalid SSL connection to GoodData and we're investigating the issue.

UPDATE Nov 24, 1:42 PM (UTC) - Only projects having custom GoodData domains was affected by this issue. Problem was with validation of a renewed SSL certificate. The fix is deploying right now.

Week in Review -- October 20 and 27, 2017

New Applications

Dark Sky Extractor

Dark Sky is the new name for the old Forecast.io service. So we have prepared a new extractor for it.  It allows you to look up the weather anywhere on the globe. You can use our migration tool to migrate your existing configurations from the deprecated Forecast.io extractor.

For more information, please see our documentation.

Currency Extractor in EU region

This extractor is now also available in the EU version of Keboola Connection.

Features

  • Dropbox Writer now allows you to specify a custom destination folder name for uploaded data.
  • We have added the ability to configure your own OAuth credentials.  We did this because we are still waiting for quota limits increase for the Google Analytics API for our Google Analytics Extractor.  So now your extractor can run under independent quotas.
  • Tableau Writer now uses the new version of Google Drive and Dropbox Writers to upload data to Tableau.
  • Removed 8 kB script size limit in Rstudio/Jupyter sandboxes
  • Improved handling errors in Rstudion/Jupyter sandboxes. Invalid package names will now throw an error with an explanation instead of an application error.
  • Read-only bucket credentials to your Redshift backend are now deprecated. See this more detailed post about accessing your data since April 18, 2018.

Fixes

  • Table import events in the Storage panel once again have a link to the imported file.

Failures

Unfortunately on Sunday we had some problems with our storage for KBC Events, which delayed execution of jobs.  We apologize for any inconvenience that may have caused.

Skipped Orchestrations at 0:00 UTC

A few Orchestrations which were scheduled to run at exactly midnight UTC may not have started due to an issue with the Orchestration Scheduler. The issue arose on the 15th of September and has now been resolved.

If you have orchestrations which are scheduled to run at 0:00 UTC, please check that these orchestrations have run correctly.


Sincere apologies for the inconvenience.

Job failures

On May 5, 2017 at 6:39:03 PM UTC+2 one of our AWS RDS server was restarted. This lead to loss of connection between our lock server and job processes. The outcome may be following

  • Orchestrations, that have all tasks successfully finished, but the orchestration itself failed
  • Failed tasks within orchestrator; the task execution finished successfully, only the job status was not saved correctly

Week in Review -- April 25, 2017

Developers Documentation

We completely rewrote documentation of our Generic Extractor. It was moved from github to developers.keboola.com. Follow this step-by-step tutorial to learn how to configure extractor for new api.

Transformations

Update of python for custom science and transformations to version 3.5.3.

GoodData Writer

Intermittent data load errors were fixed using zip for data loads to avoid bug in GoodData WebDav with gzip compression.


Week In Review -- February 13 and 20, 2017

Google BigQuery Writer in private beta

We've been working hard on a BigQuery writer for the Google Cloud Platform.  If you'd like to get early access for the beta release, please let us know at support@keboola.com.

Azor Extractor

There is a new third-party extractor for Azor.  Azor is a Czech service that allows you to monitor your competitor's products prices and availability.

Fixes

  • Trying to add a column with the same name as an existing column, but only different casing will result in an error message.  (ex: customerEmail and CustomerEmail)

  • Table ID was added to workspace loading errors to help determine the cause of the error.

  • Attempting to create an output table with column names beginning with "_" will result in an error.





Shared buckets

We're happy to announce the release of the Shared Buckets  feature.  It's an easy way to share data between projects in Keboola Connection.

    This will help you:

    • Have greater organizational control over your data
    • Speedup your data workflow
    • Reduce your project usage totals

    For more details about Shared Buckets, see our User documentation. Developers can find more info in our API documentation in Apiary.