Gmail Extractor

We are proud to announce a new member of our group of Docker extractors - Gmail Extractor.

This extractor helps you fetch messages from your Inbox by specifying simple queries. These queries are identical to those you're using in Gmail's web interface. If you're more into advanced search, check this Gmail's Help site.

Main features of the Gmail Extractor:

  • you can specify multiple queries
  • output is divided to multiple tables, which gives you more freedom when selecting/joining data
  • multiple runs of the same query won't cause downloading the same data
  • you can specify the headers you want to download

The process for adding the Gmail Extractor is very similar to the other extractors (e.g. Adwords Extractor v2)

1. Find Gmail Extractor in the list of extractors

2. Create a new configuration for this extractor

3. Authorize the extractor to access your inbox

4. Configure the extractor and specify the queries you want to run

For more detailed information about the extractor configuration options please see the documentation.


AdWords Exractor API Update

Extractor will update to API v201601 of AdWords API in the beginning of April. Please be sure to review your configuration so that it does not use no-longer supported metric names till the end of March. Also do not use new metrics until the update, we will let you know about it.

GoodData Writer configuration status update

By now, all writers are reading configurations of datasets, date dimensions and filters from Components Configurations API only. Corresponding tables in Storage API still exist and are updated even when they are not used for reading. 

Table users is not used for reading nor updating anymore and information about users created by writers are stored in Writer's backend exclusively. You can access the data using API: http://docs.keboolagooddatawriterv2.apiary.io/#reference/projects-and-users-provisioning/users.

Table filters_projects is not used anymore, GoodData URIs of the filters were moved to Component configuration to section filters.

Table filters_users is not used anymore, information about assigned filters to users is obtained directly from GoodData API. Notice that it brings implicit obsoletion of sync-filters API call. The API call still rebuilds filters-users relations according to the filters_users table but the table isn't updated.

Last tables actively used by Writers are projects and project_users and they will be migrated soon, probably this week. They will be moved to Writer's backend similarly to users table.

Please don't delete configuration tables yourself, whole buckets will be deleted automatically when they won't be used anymore, probably within two weeks.

Project Limits

Today we’re introducing limits to all Keboola Connection projects.  You can find them in the “Users & Settings” section.

It will let you know what your limits are for storage, user licenses, orchestrations, etc. 


If your project or component is over a limit, the metrics will be shown in red brick


Keep in mind that these are just soft quotas which can be easily exceeded.  So If you go over a limit, you don’t need to be afraid of anything happening to your project usage (we all are in the cloud after all, lots of room up here :) Our goal is just to keep the red metrics at a minimum, so you may be hearing from us if too many red boxes hang around for too long.  The end result should be that you get your desired performance, and we get our profit :)

The project settings will also list your monthly cost, project type and days remaining until project expiration (typically proof-of-concept or demo projects will have expiration conditions).

Project expiration will also be announced on the project homepage (Overview):


Since we’re moving to this new system from our old filing cabinet and fax machine solution, there might be some glitches in the numbers displayed. If you find any discrepancies with the numbers there, please let us know. 

Week in Review

SSL for Transformation Sandboxes

Connection through SSL is now available when creating sandboxes for transformations.


Intercom.io Extractor

Extractor for https://www.intercom.io/ has been added to Keboola Connection.


"Load more" button added in Notifications

You can now browse older notifictionas with "Load more" button on Keboola Connection Notifictions page.


Changes to R transformations

We have made some internal changes to R transformation backend which resolves a couple of edge-case issues and bring few new features:

  • We have unified R Transformations and R Custom Science to use exactly the same environment - this solves issues when using code from transformations in Custom Science.
  • R Transformation scripts now have properly set encoding to UTF8 - manually setting encoding with Sys.setlocale("LC_CTYPE", "cs_CZ.UTF-8") is no longer necessary
  • R packages are now installed from multiple backup CRAN repositories - this solves an issue where a package installation would randomly fail in case of CRAN outage
  • When installing R packages we now automatically load them - this solves an issue where a package was successfully installed, but failed to load. This also has the nice side effect that, you no longer need to call library() explicitly to load packages in your R code.
  • R script output is now available in Transformation events, so you can do some basic logging with print('message') or a bit nicer write('message', stdout())

You don't need to make any changes to your R scripts. If you run into any incompatibilities, let us know.

Other posts this week

New version of AdWords Extractor

AdWords Extractor has been rewritten as Docker component and the old version will be switched off by the end of March. Till then its users are kindly requested to migrate their configuration to the new version. Here is how to do it:

  1. Choose AdWords v2 from list of extractors

  2. Create new configuration and give it some name as usual

  3. Click on Authorize Account button which will redirect you to Google and ask for authorization to download your AdWords data.
  4. Add extraction configuration to the Parameters text area. It is in json format and must contain AdWords developer token, customer id, bucket where you want the data to be saved and configuration of queries. Optionally it may contain parameters since and until to specify dates interval for stats (it is "-1 day" i.e. "yesterday" by default). See documentation for more information.

Please note that the new Extractor saves data incrementally which means that you have to add primary keys if you want to fill data to existing tables.

Upcoming configuration changes in GoodData Writer

Our GoodData Writer is going to begin saving configurations to our Components Configurations in the Storage API, rather than the sys stage bucket and tables it uses now. This will bring a performance boost, reduce the API response time (i.e. improve KBC UI performance), improve the clarity of the configuration and bring better versioning and rollback possibilities. This change will apply for bucket attributes (containing GoodData credentials and project identifiers), date dimensions, datasets and filters, see the API docs for more information: http://docs.keboolagooddatawriterv2.apiary.io/#introduction/configuration-in-storage-api/components-api

We will proceed in several steps. First, configurations of all writers will be automatically migrated to the Configuration storage in several waves over the upcoming days. You will be notified by the notifications system in KBC UI once the migration is complete. Once completed, every configuration change performed using the API (including KBC UI) will be written to both places. Please keep in mind however, that if you alter your writer’s configuration directly in sys bucket, these changes won't be synchronised to the new configuration and you will need to manage the synchronisation yourself.

In the second step, we will switch to reading configurations from the Configuration storage rather than the sys bucket. And in the third step, writing to the sys bucket will be stopped and sys bucket configuration tables (data_sets, date_dimensions and filters) will be deleted. We will inform you about each of these steps when they occur.

The remaining configuration tables (projects, users, project_users, filters_projects and filters_users) will not be changing for now, but in the long term they will be completely removed and their data will be only stored in the Writer's backend. So by the end of this journey, we will say goodbye to the whole configuration bucket in sys stage.

Transformation failures

Feb 25th 10:00 GMT+0100

We're investigating issues when working with transformation configurations (creating, saving, deleting). We're sorry for this inconvenience, it will be fixed shortly.

UPDATE Feb 25th 10:10 GMT+0100

The issue has been resolved. Thank you for your patience.