During last days we have turned on strict input/output mapping validation. Each input/output mapping is checked against the table in Storage if

• all columns exist
• the primary key is equal in both cases 
• datatype/indexes/distkey/sortkey or filter column names have the same letter case

Although we tried to detect all breaches of this ruleset beforehand and contact project owners some have unfortunately slipped through. We're closely monitoring all errors and fixing/restarting all failed orchestrations. 

In case your project is subject to this issue on a larger scale than a single failure, we're able to remove the validation temporarily. Please contact us at support@keboola.com with any further questions/requests.

We're deeply sorry for any inconvenience. 

We're currently investigating issues with Snowflake workspaces (sandboxes, transformations). We'll keep this post updated.

Update 7:36pm CEST: We have passed information to Snowflake support team and they're investigating the issue.

Update 22:36pm CEST: Snowflake team has identified the issue and is working on fix which should be deployed later tonight.

Update 05:55am CEST: The issue has been resolved and all operations are back to normal. Due to high number of affected transformations/orchestrations we won't be restarting them to prevent system overload. Please restart your orchestrations manually if needed.

Thanks for your patience and understanding.

We have introduced maximum query execution time limit for Snowflake transformations. 

If the query execution time exceeds 15 minutes, it will be terminated. This limit should not affect any of current transformations. 

This limit helps us to prevent accidental warehouse overloading by inefficient user queries (cartesian product). This was one of the causes of this week failures.

Between Oct 07 02:00 (GMT+2) and Oct 07 02:24 (GMT+2) some transformation jobs were failing. We have identified the root cause of this issue and fixed it. All affected orchestrations will be restarted shortly.

We're sorry for this inconvenience.

Snowflake backend is down since 11:02am GMT+2. We're trying to fix it, post mortem will follow once we recover platform.

UPDATE:

Dead DWH was switched, everything is running on new backend. All running jobs, employing Snowflake backend, crashed. You have to run it again - everything will be OK for now. In case you're suffering under some other issues, contact us at support@keboola.com, please. 

We'll also publish official post-mortem as soon as Snowflake guys resolve our Issue. Stay tuned and thanks for patience!

UPDATE II:

If you're experiencing errors in your personal sandboxes, reset your credentials under the "Sandbox" link on the right-top of Transformations (https://connection.keboola.com/admin/projects/$pid$/transformations/sandbox). 

Some jobs have failed between 2016-09-26 17:30 CEST and 2016-09-26 17:40 CEST due to problem with one of our DB servers.

All affected orchestrations have been checked and restarted.

We are very sorry for any inconvenience.

If you have any concerns about this, please contact us at support@keboola.com.

As some of you might know, Google Analytics API doesn't always return precise data. Under certain circumstances, the data returned are sampled. Read more about sampling here.

Anti-Sampling

You can choose from one of two anti-sampling algorithms - DailyWalk or Adaptive. Both are based on the same principle, to divide the wanted date range into smaller chunks.

DailyWalk as the name suggest divides the date range by days. So the extractor needs to make as many request as there are days in the date range.

Adaptive algorithm is using more sophisticated approach, and divides the date range into few smaller date ranges. Read in-depth explanation of the algorithm here, if you are interested. 

DailyWalk algorithm might be more precise in some cases, but usually you will get the same results faster with the Adaptive algorithm.

Experiment with them and use what suits you best.

We have encountered a bug in the Google Analytics Extractor. It was downloading data only for one profile, even if the query was set to "All profiles".

This bug was introduced on 16th of September and it is now fixed.

We are very sorry for any inconvenience. Please adjust your date ranges and download the missing data if needed. 

Call for testers: OpenRefine Transformations BETA

Our new  OpenRefine transformations need some testers. Do you want early access and want to play with OpenRefine in Keboola Connection? Please contact us at support@keboola.com.

Google Analytics Extractor

The Google Analytics Extractor can now parse urls for queries.  So you can create your query with the very convenient  Google Analytics Query Explorer and simply copy and paste the url.

Versions management

Simplified access to latest version diff.

Announced Redshift maintenance canceled

We are cancelling previously announced maintenance of Redshift projects 

Other improvements and bugfixes

• New Hubspot extractor published
  
• Adwords extractor - more verbose logging
• PostgreSQL extractor - fixed retry mechanism which previously caused invalid CSVs with duplicated header
• Storage - fixed table size and rows count for Snowflake backed projects
• Storage configurations - numeric ids are generated, fixed bug in duplicate ids creation
• GoodData Writer - fixed grain settings for tables with custom identifiers 
• Elasticsearch Writer - SSH tunnel support added

We've created new HubSpot extractor which enables you to extract basic data from your CRM.

It is based on our Generic Extractor with predefined templates to helps download your data easily. That also brings an option to adjust the configuration to your needs by switching to JSON editor.

If you want to learn more about the configuration of the extractor, how to get your HubSpot API Token and details of the extractor output tables, look into our new documentation.

Feel free to try the new extractor and in if you have any questions or something is missing, please contact us at support@keboola.com.