New Features

• Generic Extractor introduces some backward incompatible changes, but your current running configurations will remain unaffected. Learn more about Generic Extractor compatibility levels.
• Generic Extractor supports stopping pagination using an attribute in the response, see more on the Limit Stop feature.

Minor Improvements

• Processors configuration is now editable where eligible.
• Transformation scrips and queries editors link to the documentation. 

Fixes

• Transformations UI refreshes properly on transformation or bucket delete or revert/rollback.
• Transformation copy or restore does not delete transformation name and description.

We're encountering an increased number of application errors in MySQL transformations. We're looking into this issue and will update this post once we know more. This may also affect GoodData writer jobs.

UPDATE Aug 21, 9:10pm PST

The error seems to happens randomly on all Storage API async exports from tables stored on Snowflake. We have contacted Snowflake support to help us identify the root cause of the issue.

You may encounter the error in any component that uses async exports from Snowflake or in any external integration exporting data from a Snowflake bucket. The error does not appear in Snowflake or Redshift transformations.

UPDATE Aug 22, 9:45pm PST

The error was not yet mitigated, Snowflake engineering team is looking into it. If you're encountering this problem, please contact us at support@keboola.com for a possible workaround.

We're sorry for this inconvenience.

UPDATE Aug 23, 5:24am PST

Snowflake engineers has identified a potential fix for this issue but it is not deployed yet. Meanwhile we have found the workaround for the issue and we are updating the components. GoodData writer was fixed now, transformation and other components will be updated within few hours. 

This issue is happening randomly and less than one percent of exports were affected.

Due to a change in the Intecom API authorization, the current extractor stopped working. It took us a while to go through the Intercom OAuth registration, but the result is here - the new Intercom Extractor. The current version is immediately deprecated, please configure and authorize the new extractor. You can start downloading your Intercom data in seconds.

The full documentation is available here.

Updated Components

• GoodData Writer supports login parameter in the Create User API call to specify different mailing address
• GoodData Writer supports full parameter in the Synchronize Dataset API call to perform full sync without data preservation
• DB2 Extractor supports SSH tunneling

New Features

• Documentation to generate Google OAuth 2.0 tokens for Generic Extractor including token refreshing
  

Community News

• ZOOT developed extractor to download data from CJAffiliate’s Commission detail service API
  

  

Minor Improvements

• Storage API Client for R supports incremental writer to Storage

Fixes

• Snowflake Extractor supports large files
  

We're updating datatype handling in the Transformations UI and backend. The update features datatype validation and new datatype configuration format. 

In rare cases errors may occur. If it happens in your project, please get in touch with us at support@keboola.com. 

Thanks for your understanding.

Since Jun 02 21:56:15 CEST we're experiencing a high rate of Google Analytics Extractor application failures. They are most probably related to an exceeded account quota. 

We're investigating this issue and will update this post with the outcome.

New Components

Snowflake DB Extractor 

Salesforce Analytics Cloud Writer

Martin Humpolec, a Salesforce specialist, developed and published a Salesforce Analytics Cloud writer.

Updated Components

• Facebook and Facebook Ads extractors retry when receive one of the following errors: Unkwnon error / Unexpected error / Please reduce the amount of data error

New Features

• Snowflake and Redshift transformation input mapping can convert empty values to null

Community News

• David Ešner's FTP extractor supports compressed files

Minor Improvements

• All components can now handle files/tables larger than 5 GB
  
• Under Your Account / Sessions (https://connection.keboola.com/admin/account/sessions) you can see all your active sessions and revoke them
• Keboola Connection session ids are encrypted using CRYPT_BLOWFISH

Fixes

• Fixed privileges for deleting linked buckets 

No project should be affected in their daily operations and no action is required, this is a purely informational message about an upcoming change.

From May 3rd we will begin gradually turn on strict primary key checking in all applications. This will cause all components writing to Storage to fail if the primary key on the table does not match the one set in the component's configuration. 

Previously the primary key was set when the table was created and was not checked any further. 

We recently started monitoring primary key mismatches and tried to fix them automatically - the component executor tried to modify the primary key on the table so that it would match the configuration. A few situations arose where it couldn't be fixed automatically:

• New primary key contained columns not present in the table
• New primary key could not be set as there were duplicate values in the new primary key columns
• Several configurations in the project were changing the primary key back and forth

We have contacted all projects affected by these issues and resolved them. We will continue to monitor this situation closely until we switch on the strict checking. 

We have slightly updated some of the input fields in the Transformations UI. 

• Packages, Stored Files and Requires are editable straight away and save immediately after each change.
• Queries and Scripts are editable straight away too, but you can reset (to the previous state) or save the change with a button.

No more clicking to edit a field. If you delete or add something by accident, you can always revert to the previous version.

Let us know what you think!

On April 11 between 4:15pm and 4:31pm CEST our connection to quay.io was down. All Docker images stored there were inaccessible. That caused failures in running some components. If you encountered an application error during that period, this outage was likely the culprit. 

Unfortunately status.quay.io remains silent about this outage. 

We are sorry for this inconvenience and to mitigate similar issues in the future, we will move all used Docker images from public repositories (DockerHub and Quay.io) to our private repository in AWS ECR.