All Snowflake transformations now have SQLdep enabled. 

SQLdep allows you to visualize the data flow in your SQL. You can now easily see the impact of your changes or track the lineage of each column. 

SQLdep Snowflake support is still in beta, so do not hesitate to contact support@keboola.com with any questions or bugs.

We are investigating Snowflake partial outage. Queries are now queued.

We will inform about the progress.

UPDATE 11:21 AM PDT Snowflake warehouses are up and running. Queued queries are being processed. We will update the post when we'll have more information about incident cause from Snowflake engineering.

New Features

• Generic Extractor introduces some backward incompatible changes, but your current running configurations will remain unaffected. Learn more about Generic Extractor compatibility levels.
• Generic Extractor supports stopping pagination using an attribute in the response, see more on the Limit Stop feature.

Minor Improvements

• Processors configuration is now editable where eligible.
• Transformation scrips and queries editors link to the documentation. 

Fixes

• Transformations UI refreshes properly on transformation or bucket delete or revert/rollback.
• Transformation copy or restore does not delete transformation name and description.

We've spent a lot of time on squashing bugs in GoodData Writer, Google Drive Writer, Docker Sandbox and Gelf logger and on several new features which are not yet ready. Besides, there are two nice things we can tell you about.

UI of the KBC is being revised for a few weeks now and you can see the first results in the unification of buttons for creating new resources. 

Database Writers now allow triggering `nullable` values for all columns at once.

New Features

• List of deprecated components in the Project Overview:

In the Project Overview you may now see a warning that you are using deprecated components in your project. 

To prevent any inconveniences, please keep your components up-to-date. 

Clicking on a component from the list will take you to a component configuration, from where you can start a migration process.

We're encountering an increased number of application errors in MySQL transformations. We're looking into this issue and will update this post once we know more. This may also affect GoodData writer jobs.

UPDATE Aug 21, 9:10pm PST

The error seems to happens randomly on all Storage API async exports from tables stored on Snowflake. We have contacted Snowflake support to help us identify the root cause of the issue.

You may encounter the error in any component that uses async exports from Snowflake or in any external integration exporting data from a Snowflake bucket. The error does not appear in Snowflake or Redshift transformations.

UPDATE Aug 22, 9:45pm PST

The error was not yet mitigated, Snowflake engineering team is looking into it. If you're encountering this problem, please contact us at support@keboola.com for a possible workaround.

We're sorry for this inconvenience.

UPDATE Aug 23, 5:24am PST

Snowflake engineers has identified a potential fix for this issue but it is not deployed yet. Meanwhile we have found the workaround for the issue and we are updating the components. GoodData writer was fixed now, transformation and other components will be updated within few hours. 

This issue is happening randomly and less than one percent of exports were affected.

New Features

• Database writers UI  - Disabled tables no longer load the whole content into the container when run and only first row of the table is loaded instead. This saves storage credits and decrease time of jobs run process. It works only for newly disabled tables, so old tables have to be redisabled (enable export -> disable export).
• Python for transformations and custom science applications has been updated to version 3.6.2
  

New Features

• Ability to test component's image against live configurations (Read more in our documentation)
• Markdown descriptions for tables and buckets are now available (see the images bellow)

Fig. 1 Click "Describe the ..." to add description

Fig. 2 Markdown is supported when describing buckets or tables.

Description of buckets and tables can be added/edited in various places:

• In bucket detail (Storage section),
• in table detail (Storage section)
• and anywhere in application where table preview is available

Deprecation

• We finally turned off Legacy Routing (Original status here)
• EOL of the synchronous exports has been announced and we'll turn it off on August 21st, 2017

Tutorials

• Our colleague prepared a small tutorial how to connect to the Keboola provided Snowflake database from Mode Analytics

Fixes

• Due to recent outages of Quay we started migration of our components to AWS ECR
• We updated the Pigeon component to use the newer Storage API client and send files to File Uploads encrypted

We were notified from Snowflake support about problem that could potentially affect Keboola Connection Snowflake Backed projects.

The issue happened between 9:19pm UTC August 8th and 6:45pm August 9th UTC.

There is a full alert from Snowflake below. We are investingating with them if we can find out if there are some affected KBC projects.

We will keep you informed of any progress.

UPDATE 2017-08-12 We have received list of projects which might be affected by this issue. In these projects one or more queries which might return wrong results were executed. We have notified these projects in KBC. If you need any assistance please contact support@keboola.com.

Dear Snowflake Customer,

We have identified a problem that could produce incorrect results in some queries that perform memory intensive join operations. It is unlikely that this issue impacts you; however, we wanted to make sure you were aware of the problem. The issue was isolated to the US-West region between 2:19pm Pacific time on Tuesday, August 8th and 11:45am Pacific time on Wednesday, August 9th. Queries performed in other regions or outside that timeframe were not impacted.

Because of the nature of the issue, it is difficult for us to determine if any of your queries experienced the problem. However, we are able to perform further analysis on a query-by-query basis. If you have a question about whether a specific query was impacted, please submit a support request to Snowflake support via the Support Portal or via email to support@snowflake.net .

We are very sorry about this issue and will do everything possible to help you resolve any problems this may have caused you.

We observe components failures hosted on quay.io from 08:25  AM CET. Python Transformations, Facebook Extractor, Google Sheets Writer are between affected components.

Quay.io outage is not reported yet. To avoid this issue in future we are moving all components to Keboola provided AWS repository.

We will inform about the progress.

9:09 AM CET RESOLVED Last error was seen at 08:53 AM CET. Quay.io has confirmed and fixed the issue.