Snowflake's Behavior Change Pending: Changes to String-to-Date Conversion

On Dec 1st, 2019, Snowflake will turn on strict string-to-date conversion for all Keboola Connection projects.

By introducing this change, Snowflake wants to address issues such as invalid date strings rolling over. For example, 07-32 (July 32nd) has been interpreted as August 1st. Rollover will no longer be supported for dates, times, or timestamps.

A full description of affected cases and tips on what to do are provided in the Snowflake 3.45 Release Notes.

We have a list of affected queries in the Keboola Connection projects ready and will be contacting all users that may be affected by this change. You'll have plenty of time to check everything and fix any potential issue. 

Thanks a lot for your cooperation. 


Transformation Overview and Table Graph errors

Since October 11, 2:00 PM UTC we are experiencing failures on the Transformation Overview page and on the Storage Table Graph page in all regions with the following error message:

Cannot load GoodData Writer: Client error response [status code] 410 [reason phrase] Gone

We're investigating the issue and will update this post with our findings.

We apologize for the inconvenience.

UPDATE October 15, 5:15 PM UTC: a new version was released and the bug was fixed.

Week in Review — October 11, 2019

New Components

  • Smartsupp extractor by Revolt BI downloads data from the live chat platform.

Updates

  • Input mapping filter can be set on an ignored column.
  • Sandboxes with duration longer than 5 days display the expiration date correctly.

Updated Components

  • Oracle and PostgreSQL extractors use optimized tables and columns loading.
  • R transformation and sandbox updated to version 3.6.1.
  • R transformation supports R Markdown.
  • GoodData Writer fixes the link to the GoodData project in the EU region.

Deprecations

Increased "Obsolete output mapping" transformation user errors

Since October 10, 3:00 am UTC we are experiencing higher rate of "Obsolete output mapping" user errors in transformation jobs. This issue is caused by a recent update of the transformation component. We'll be rolling back the release as soon as possible and keep you updated on further events.

UPDATE 8:45 am UTC: We have successfully deployed a previous version and all affected configurations should be working now. 

UPDATE October 13, 6:11 PM UTC: A new version with a fix has been released and deployed.

We apologize for the inconvenience.

Week in review -- March 27, 2019

New Features

  • Added fulltext search in API tokens.
  • Added fulltext search and sorting by name in Orchestrations.

Minor Updates

  • We have discontinued Productboard and all your new ideas come directly into our inbox. Please feel free to share with us anything on your mind.
  • Python and R transformations show the link to the transformation configuration.
  • After successfully validating a SQL transformation you can immediately run it. 


Power BI Desktop not showing Snowflake tables

We're investigating issues with Power BI not showing tables in your Snowflake workspace.

We're sorry for this inconvenience, we'll be updating this post with our findings.

UPDATE 17:26 CET

Snowflake confirmed they can reproduce the issue and are investigating further. 

The issue affects transient tables (CREATE TABLE statement), but does not affect views. As a temporary workaround we suggest to log in directly into the Snowflake workspace and create views from all tables in the schema. These views should be immediately visible in Power BI.


UPDATE 11:20 CET

Snowflake are preparing a new version of the ODBC Driver which allows Power BI to see the tables. Use the support button in your project to contact us to obtain this driver.


Speeding up transformation outputs in your projects

We're working hard every day to make Keboola Connection faster and minimize the time you spend waiting for your results. This effort includes a wide variety of components of the platform architecture. Some of the changes are straightforward and transparent to the end users, but others are unexpectedly complicated. 

This is the case of transformations. We rolled out an update earlier this year and were forced to immediately rollback to the previous version as it broke the data flow in a few projects. This time, we're more prepared. We have identified the source of the incompatibility, and we'll be rolling out the update silently, only for those projects that will not be adversely affected. The projects that would be affected will not be updated. Instead, they will be notified to take steps to fix the incompatibilities. Then they'll become eligible for the update as well. 

Parallel output processing

In the original system, when all transformations in a phase are executed, the output processing starts. It takes all the transformations sequentially and processes the outputs one by one. The order of the executions is not defined, but it is predictable and, most importantly, it doesn't change between runs. Some projects rely on a specific order of output processing to achieve certain goals.

To speed up the output, we have decided to queue all outputs at once and let Storage handle all jobs as fast as possible. But, as you already may have noticed, this mean the transformations can run in any order or even in parallel. This may affect the result of the output if you relied on a particular order.

Updating the project

The project overview and all affected transformations will notify you about multiple outputs being written to the same table in Storage. You will be easily navigated to the places that need to be fixed. Please contact our support if you need any help doing that. 

Once you have fixed all instances where multiple outputs are written to the same table in your project, you can immediately contact us using the support button. We will turn on the update in your project. Or, wait until we update your project automatically (we're watching all projects regularly).



Failing Jobs

Since 2018-12-19, 19:13:00 UTC we're experiencing a higher rate of application errors in all regions due to an outage in AWS ECR. 

We're investigating the issue and update this post once there.

We apologize for the inconvenience.

Update 19:30 UTC

This outage also affects the Developer Portal.

Update 20:05 UTC

As of 19:51 UTC the issue is resolved by Amazon

Stalled jobs in EU region

Around 1:30am CEST one of the job worker instances stopped processing assigned jobs. This could have lead to jobs being stuck in the processing state for a long time without any activity.

At 11:15am CEST the worker instance was terminated and all unfinished jobs started processing on other instances.

We're sorry for this inconvenience.