Week in Review -- November 30, 2017

Database extractors

After recent announcement of new database extractors we are bringing other improvements to these extractors.

  • By default the new bucket is created for each extractor configuration. Previously extractor configurations were sharing one bucket which led to collisions.
  • You can reload the list of tables fetched from database. It is useful when you are tuning credentials permissions or you have just added some tables to the database. 
  • Primary key is now validated against table created in storage. Warning is thrown if the configured key is different than the key defined on the table.

Generic extractor

The new ignoreErrors option was introduced. This option allows you to force Generic Extractor to ignore certain extraction errors.  Read more in documentation.

Fixes

  • Google Drive Writer in append mode was overwriting rows instead of appending new rows under some circumstances
  • Google Big Query extractor was ignoring files exported to multiple slices
  • Tableau Writer is now available in the EU region


New version of AdWords Extractor

We have just released a new version of AdWords Extractor. It works with AdWords API v201710 (see the Release notes) and you no longer need your own developer token to use it.

The previous version of the extractor is deprecated and you can use our migration tool which will migrate your AWQL queries. However, you have to reauthorize the extractor and give it access to your AdWords data again. The previous version uses AdWords API v201705 which will be switched off on 28 March 2018.

Week in Review -- November 21, 2017

New Features

  • There is a new extractor for what3words service (https://what3words.com/)

Improvements

  • Configuration of MySQL Extractor allows parameter database to be optional
  • Loading data to Storage Workspace allows renaming of columns
  • Configuration errors from DarkSky Extractor are properly marked as user errors
  • Unified incremental load settings for all database writers

Job Failures Tuesday, November 21, 2017

We have been experiencing temporary technical difficulties today around 9:15 AM CET and 9:20 AM CET.

Some component jobs may have failed as a result. We're investigating the issue and post an update when the root cause is found.

UPDATE 11:35 AM CET Jobs storage was temporarily unavailable for about two minutes. Jobs scheduling wasn't affected and running jobs were waiting until storage came up again so these jobs weren't affected. Unfortunately few orchestrations have failed, we'll do step to prevent this in the future.


Job Failures Saturday, November 18, 2017

We have been experiencing temporary technical difficulties today around 3:00 AM CET and 4:00 PM CET.

Some component jobs may have failed as a result.  We have identified and fixed the issue. All systems have returned to normal operations and all jobs are now being processed normally.

Google Analytics Extractor - Custom Authorization

Recently we have been hitting quota limits of the Google Analytics API with our extractor. As these limits are shared between all users of the extractor, we decided to add a new authorization option - Custom Authorization.

With Custom Authorization, you can provide your own API keys for the Google Analytics extractor, so you will have your own quota limits. This way you can monitor these limits as you need, and other users won't be affected.

Please check out our step-by-step guide on how to set up Custom Authorization in the Google Analytics Extactor.

In case of any problems or questions, please contact us on support@keboola.com.

Developer Portal

As part of our effort to make Keboola Connection an open platform we would like to announce the availability of the Keboola Developer Portal - https://apps.keboola.com/. Some of you may already be aware that it is possible to modify applications in KBC through the Developer Portal API. Now we are adding an application to the API so that you can get things done more easily.

The Keboola Developer Portal is a completely separate application from Keboola Connection. It is region-less because it has no access to customer projects. The Developer Portal provides an authoritative list of KBC applications for each region. Each Keboola Connection stack reads this list regularly. As usual in Keboola, the Developer Portal is done in an API first approach, so everything that the application does can be done programmatically too.

What does the Developer Portal do?

  • It allows users registered to vendors to create and modify applications in KBC without Keboola Tech support.
  • It provides service accounts for automating deployment and testing.
  • It provides Docker AWS ECR repositories for applications (thus avoiding Quay and Dockerhub issues).
  • It provides an authoritative list of publicly available approved applications.

Anyone can sign up for the Developer Portal. However until you join a vendor or create a new one, you will not be able to do anything. Both creating and joining a vendor must be approved. Joining a vendor must be approved by the vendor administrator, and creating a vendor must be approved by Keboola Tech. Once a member of a vendor, you can modify and add new applications to that vendor. There's no need to fill a checklist and go through our support. When you create an application, it becomes immediately available for you to use. Using the Developer Portal, you can also set up automated deployment for your application.

Keboola Tech still has some things to do:

  • approve new vendors
  • change some controlled properties of an application (e.g. memory limit)
  • approve a new application before it can become public

Next steps:

  • We still have a long list of things to be done on the Developer Portal, so we consider it as beta quality at the moment.
  • We are soon going to hold a developer meet-up in the Czech Republic where we’d like to hear about your experience developing applications for KBC (in January probably).
  • If you have previously used the Developer Portal API, we encourage you to try the application. If you are having trouble joining your vendor, please contact us on support (there may be some friction depending on how your account was created).
  • If you have created complex applications using Custom Science, go ahead and check the Developer Portal, it may now be more convenient to turn them into full applications.
  • If you have never created an application for Keboola Connection, we highly encourage you to try it - it's much easier now. In fact a component can be created from scratch in an hour.

 

Deprecating MySQL Storage and Transformations

Support for MySQL in Keboola Connection is coming to its end. Here's what will happen.

Effective immediately

  • New projects and projects without existing MySQL transformations will not be able to create new MySQL transformations.

MySQL Storage Backend (supported until January 2018)

  • The default storage backend for all projects is immediately switched from MySQL to Snowflake.
  • All MySQL buckets in all projects will be migrated to Snowflake in January 2018. This will not affect any operations, only a short maintenance on the project will be required.
  • No changes in the project are required.
  • You can apply for a sooner migration at support@keboola.com.

MySQL Transformations (supported until April 2018)

  • Your existing MySQL transformations will need to be migrated to Snowflake by the end of April 2018.
  • If you need any help migrating your MySQL transformations, contact support@keboola.com.

These steps will allow us to deprecate a piece of the legacy infrastructure and focus on the state of the art technologies. The Snowflake storage backend and transformations have significant performance and scaling benefits, so your projects will run faster than on MySQL without any extra charge.

Week in Review -- November 13, 2017

New Features

  • We are pleased to announce a significant speed boost to Snowflake load to workspace (also speeding up Input Mapping inTransformations). For tables with primary keys the speed difference is up to 80%.
  • We have also simplified our database extractors, as we mentioned here last week.
  • Fixed CSV Import. The component generates meta-data now.

Minor Imporvements

  • We have updated our Terms and Conditions.
    Per section 14.3 of our Master Software Subscription Agreement, we would like to notify you of a change to the Agreement and its new version released November 10th. The purpose of this change was to provide further protection to Client Data and to comply with requirements of some third party providers (namely Google). We do not consider this change to be a material change, however we are obliged to provide you with this notification. There is no action required from you in relation to this change if you accept it. We are setting out the original and revised language for your review below. Please do not hesitate to contact us with any questions or concerns at legal@keboola.com.

    Original (version from March 16th, 2016):
    6.7.1. Permitted Use of Client Data. Keboola will only use and will cause all Affiliates and Keboola employees, agents, subcontractors and representatives to only use Client Data for purposes of performing the Services. Keboola shall follow Client instructions when processing Client Data and will not use Client Data other than as authorized by this Agreement. 

    Updated (version dated November 10th, 2017):
    6.7.1. Permitted Use of Client Data. Keboola will only use and will cause all Affiliates and Keboola employees, agents, subcontractors and representatives to only use Client Data for purposes of performing the Services. Keboola shall follow Client instructions when processing Client Data and will not use Client Data other than as authorized by this Agreement. Client Data which Keboola downloads (with permission granted by Client during component configuration) from Google, Facebook, Twitter or other third party APIs ("Third Party Data") are stored and handled the same as Client Data under this Agreement. Third Party Data will not be shared anywhere (it remains stored securely on Keboola systems) until Client decides to share that data (for example, by configuring Data writer to Client or third party systems). Client can delete all Third Party Data and data source configurations from Keboola storage anytime. This section shall be read along side with and be subject to the provisions of sections 6.9 (Retention and Deletion) and 6.10 (Compliance with Law) of this Agreement.

New Database Extractors

We're very pleased to announce the arrival of simplified database extractors.  (Available for MySQL, MSSQL, PGSQL, Redshift, Snowflake, Oracle and IBM DB2)

Configuring the new extractors can not be any easier.  All you need to do is select the tables you want to import and ... that's it! 

You can, of course, choose which columns you want to import, whether it will be incremental or not, and decide where the output should be, but the defaults are usually sufficient.   
Also, have no fear, you can still write advanced queries for your extractor if that's what you need.

If you run into any problems with the new extractors or require any assistance, please don't hesitate to ask at support@keboola.com.