GoodData Writer failures in EU region

There is some problem on GoodData API since about 08:00 UTC which causes failures on some model updates and data loads.

We are investigating the problem with GoodData support and will keep you updated.

Update 14:20 UTC - The GoodData Technical Support team is still investigating this issue.

Our GoodData writer component has not changed since January, so we are waiting for a clarification of the root cause from their support team.

Update 15:10 UTC - This issue is related to GoodData's release today. They are preparing a hotfix for it and expect it to be deployed in a few hours.

Update June 5, 08:20 UTC - The problem has been resolved. GoodData deployed a hotfix for their API last night. Since June 4 21:00 UTC we have not seen any new errors from the GoodData Writer.

Degraded Snowflake Performance (EU region) - March 2, 2020

We're experiencing degraded Snowflake performance affecting all operations in the EU region.

This problem can affect our users in the following ways:

  • Job execution time may be longer than usual.
  • Jobs are randomly failing with the error message "Result not found".

This is similar to the issue we observed last week, which was caused by the Snowflake Cloud Service Layer in EU region being overloaded.

We have reported it to Snowflake support and will keep you posted as soon as we have an update.

UPDATE Mar 3, 10:43 CET - The performance of Snowflake operations has currently improved. We're still working with Snowflake support on mitigating this issue permanently.

UPDATE Mar 4, 19:50 CET - We were hit again today with degraded performance of Snowflake's Cloud Service Layer. Between 3:30 PM - 5 PM CET we observed a large increase in compilation time of Snowflake queries. After 5 PM it returned back to normal. We updated Snowflake with our latest observations and are hopeful for a quick resolution. We will update here with any more information within the next 12 hours.

UPDATE Mar 5, 06:35 CET  - We hit again Snowflake performance degradation in last six hours. Snowflake is working on issue mitigation with highest severity. We will provide an update in three hours.

UPDATE Mar 5, 09:49 CET  - Performance degradation peaks still occurs. We are in touch with Snowflake on resolution. We will provide an update in three hours.

UPDATE Mar 5, 13:43 CET  - There has been no change of performance for the past few hours and its degradation still occurs. We will provide an update in three hours.

UPDATE Mar 5, 16:18 CET - We've been working with Snowflake support on mitigating the performance degradation. From our monitoring it seems that Keboola platform in EU region is now stabilised (since around 14:00). We will provide next update in three hours. 

UPDATE Mar 5, 16:43 CET - The issue is a highest priority case with Snowflake support. We're closely collaborating on finding a permanent solution to the performance degradation. Status update from Snowflake: 

Snowflake acknowledges the performance degradation issue reported by Keboola Czech SRO since February 24, 2020 and tracks it under the case #00097987 as a critical incident.

Snowflake Support together with Snowflake Engineering are continuously collaborating and conducting an in-depth investigation to identify the root cause and provide a viable solution to you on priority.

At this time, additional resources have been allocated in the Snowflake Services Layer which should show improved performance values.

We will provide next update in three hours. 

UPDATE Mar 5, 20:55 CET - The performance is now stabilised and jobs are running as expected. We are monitoring the situation and We will provide next update in 12 hours.

UPDATE Mar 6, 09:23 CET - The performance has worsened slightly during the night job peak. We are monitoring the situation and  will provide next update in 3 hours.

UPDATE Mar 6, 12:55 CET - The Snowflake Engineering team made some optimizations of the Cloud Service Layer for our account.  Since 7:40 AM CET there have been no "Result not found" errors and job performance should return to normal state.

We will continue monitoring the situating and will provide another update in 6 hours.

UPDATE Mar 6, 21:45 CET - All operations are back to normal. Thanks for your patience and understanding.

GoodData writer failures (US region)

We have encountered a small number of failed jobs in US region between 4:46 AM - 9:17 AM UTC.

Affected jobs finished with Application Error caused by invalid SSL connection to GoodData and we're investigating the issue.

UPDATE Nov 24, 1:42 PM (UTC) - Only projects having custom GoodData domains was affected by this issue. Problem was with validation of a renewed SSL certificate. The fix is deploying right now.

Week in Review -- October 20 and 27, 2017

New Applications

Dark Sky Extractor

Dark Sky is the new name for the old Forecast.io service. So we have prepared a new extractor for it.  It allows you to look up the weather anywhere on the globe. You can use our migration tool to migrate your existing configurations from the deprecated Forecast.io extractor.

For more information, please see our documentation.

Currency Extractor in EU region

This extractor is now also available in the EU version of Keboola Connection.

Features

  • Dropbox Writer now allows you to specify a custom destination folder name for uploaded data.
  • We have added the ability to configure your own OAuth credentials.  We did this because we are still waiting for quota limits increase for the Google Analytics API for our Google Analytics Extractor.  So now your extractor can run under independent quotas.
  • Tableau Writer now uses the new version of Google Drive and Dropbox Writers to upload data to Tableau.
  • Removed 8 kB script size limit in Rstudio/Jupyter sandboxes
  • Improved handling errors in Rstudion/Jupyter sandboxes. Invalid package names will now throw an error with an explanation instead of an application error.
  • Read-only bucket credentials to your Redshift backend are now deprecated. See this more detailed post about accessing your data since April 18, 2018.

Fixes

  • Table import events in the Storage panel once again have a link to the imported file.

Failures

Unfortunately on Sunday we had some problems with our storage for KBC Events, which delayed execution of jobs.  We apologize for any inconvenience that may have caused.

Skipped Orchestrations at 0:00 UTC

A few Orchestrations which were scheduled to run at exactly midnight UTC may not have started due to an issue with the Orchestration Scheduler. The issue arose on the 15th of September and has now been resolved.

If you have orchestrations which are scheduled to run at 0:00 UTC, please check that these orchestrations have run correctly.


Sincere apologies for the inconvenience.

Job failures

On May 5, 2017 at 6:39:03 PM UTC+2 one of our AWS RDS server was restarted. This lead to loss of connection between our lock server and job processes. The outcome may be following

  • Orchestrations, that have all tasks successfully finished, but the orchestration itself failed
  • Failed tasks within orchestrator; the task execution finished successfully, only the job status was not saved correctly

Week in Review -- April 25, 2017

Developers Documentation

We completely rewrote documentation of our Generic Extractor. It was moved from github to developers.keboola.com. Follow this step-by-step tutorial to learn how to configure extractor for new api.

Transformations

Update of python for custom science and transformations to version 3.5.3.

GoodData Writer

Intermittent data load errors were fixed using zip for data loads to avoid bug in GoodData WebDav with gzip compression.