Week in Review -- May 30, 2017

New version of Dropbox extractor

Our third-party partner Radek Tomášek published the new version of Dropbox extractor which uses Dropbox API v2. Old extractor uses API v1 which will be deprecated soon so you are recommended to migrate to the new extractor soon. See http://status.keboola.com/dropbox-extractor-v2

Pipedrive extractor

An extractor to Pipedrive: Sales CRM & Pipeline Management Software was published to all users.

Transformations

Detail of transformation job was slightly improved for better intelligibility.

Connection

We fixed email validation of invitation form which prevented you from inviting users with emails with e.g. geographic top-level domains like '.london'.

Week in Review -- May 22, 2017

We're sorry for the delayed delivery of our newsletter. So here's what's new since April 25th. 

New Components

Snowflake DB Extractor 

Salesforce Analytics Cloud Writer

Martin Humpolec, a Salesforce specialist, developed and published a Salesforce Analytics Cloud writer.

Updated Components

  • Facebook and Facebook Ads extractors retry when receive one of the following errors: Unkwnon error / Unexpected error / Please reduce the amount of data error

New Features

  • Snowflake and Redshift transformation input mapping can convert empty values to null

Community News

  • David Ešner's FTP extractor supports compressed files

Minor Improvements

  • All components can now handle files/tables larger than 5 GB
  • Under Your Account / Sessions (https://connection.keboola.com/admin/account/sessions) you can see all your active sessions and revoke them
  • Keboola Connection session ids are encrypted using CRYPT_BLOWFISH

Fixes

  • Fixed privileges for deleting linked buckets 


Temporarily Out of Order Component Synchronous Actions

A release yesterday has broken some synchronous actions for some Keboola Connection components.

No jobs are affected by this, the components run as before, but creating and updating components may be affected.

We are currently working on a patch and will update here with more information.

Thank you for your patience.

Update: 2017-05-17 11:38

All synchronous actions should now be working as expected.  

Dropbox Extractor (v2)

It is my tremendously big pleasure to announce an important update - Dropbox Extractor component was updated in Keboola Connection (KBC) and this new version is officially released and available for everybody.   

It's been quite a while since the original version was developed. So happened, it was one of my first components written in Node.js and also my first-ever KBC component where I didn't use a generic UI, but developed a custom one (I also used React for the first time). I loved that, but (on the other hand) I was also aware the first version was anything, but perfect.

The new version is here and the main reason for this update is the end of the support of the Dropbox API v1 (used in the original extractor). It became deprecated and (by the end of the June 2017) this old API stops working completely. 

Good news is that this new version of the Dropbox extractor brings a lot of important and valuable improvements. The main change is usage of the official Dropbox Chooser component. You can easily authorize and select csv files from your Dropbox. The Chooser component generates a link (you can paste already existing one) and all these links are encrypted in the KBC configuration, so the whole process is very fast (even with large files), convenient and secure.

Migration note: There are breaking changes between the previous (v1) and new (v2) versions and if you use the original Dropbox Extractor (v1) and want to keep using Dropbox extractor in KBC in the future (after June 28, 2017), you have to create a new configuration in this updated version (v2) and select all your required files again. Unfortunately, no automated migration tool between v1 and v2 is available as the architecture is completely different.

I would love to say big thanks to Keboola, Tomáš Kačur in particular. He provided very helpful feedback as well as helped me to fine-tune the UI part. 

This extractor is developed independently by Radek Tomášek. For more information on how to use this extractor, please refer to the documentation. In case you run into some issues or you have more questions about the usage of the component (e.g. migration between the old and new version), please don't hesitate to contact me directly.

Job failures

On May 5, 2017 at 6:39:03 PM UTC+2 one of our AWS RDS server was restarted. This lead to loss of connection between our lock server and job processes. The outcome may be following

  • Orchestrations, that have all tasks successfully finished, but the orchestration itself failed
  • Failed tasks within orchestrator; the task execution finished successfully, only the job status was not saved correctly

Week in Review -- April 25, 2017

Developers Documentation

We completely rewrote documentation of our Generic Extractor. It was moved from github to developers.keboola.com. Follow this step-by-step tutorial to learn how to configure extractor for new api.

Transformations

Update of python for custom science and transformations to version 3.5.3.

GoodData Writer

Intermittent data load errors were fixed using zip for data loads to avoid bug in GoodData WebDav with gzip compression.


Job failures

There were few job failures and the UI encountered a brief outage in some API calls on May 3 between 16:07 and 16:14 GMT+2/CEST. 

Affected jobs and API calls returned an Application error. 

The outage was caused by an automatic upgrade of one of our metadata DB servers. We will be turning off automatic upgrades to prevent this in the future.

We're sorry for this inconvenience.

Job failures

There were few jobs failures between 16:00 - 17:00 CET and 21:00 - 22:30 CET. We have identified the root causes of the problem and rollbacked to previous version.

Affected jobs were returning error:  Authentication token has expired. The user must authenticate again

We're sorry for this inconvenience.

New Generic Extractor Documentation

Long neglected, long awaited, it's here!

We are done with a major overhaul of our Generic Extractor documentation. It is now completely moved to developers.keboola.com. The new documentation contains over 100 runnable examples. We have also created a tutorial which guides you through how to configure the Generic Extractor for a new API.

We have also a configuration map with links to documentation of all parts of the configuration.

In case you haven't heard of the Generic Extractor, it is our universal REST API client. It reads data (in JSON format) from an API according to how you configure it, then converts the results to CSV files and imports them into KBC Storage. This way you can extract data from new APIs in tens of minutes. You can use the Generic Extractor to create a new extractor or perform ad-hoc extractions from an API. 

If you are interested, give a try to the Tutorial.