Extractor will update to API v201601 of AdWords API in the beginning of April. Please be sure to review your configuration so that it does not use no-longer supported metric names till the end of March. Also do not use new metrics until the update, we will let you know about it.
By now, all writers are reading configurations of datasets, date dimensions and filters from Components Configurations API only. Corresponding tables in Storage API still exist and are updated even when they are not used for reading.
Table users is not used for reading nor updating anymore and information about users created by writers are stored in Writer's backend exclusively. You can access the data using API: http://docs.keboolagooddatawriterv2.apiary.io/#reference/projects-and-users-provisioning/users.
Table filters_projects is not used anymore, GoodData URIs of the filters were moved to Component configuration to section filters.
Table filters_users is not used anymore, information about assigned filters to users is obtained directly from GoodData API. Notice that it brings implicit obsoletion of sync-filters API call. The API call still rebuilds filters-users relations according to the filters_users table but the table isn't updated.
Last tables actively used by Writers are projects and project_users and they will be migrated soon, probably this week. They will be moved to Writer's backend similarly to users table.
Please don't delete configuration tables yourself, whole buckets will be deleted automatically when they won't be used anymore, probably within two weeks.
Today we’re introducing limits to all Keboola Connection projects. You can find them in the “Users & Settings” section.
UI Improvements
When creating new Transformation, the transformation backend is set to default project backend:
The data sample filter in Storage console is now displayed in a more logical order:
Other posts this week
SSL for Transformation Sandboxes
Connection through SSL is now available when creating sandboxes for transformations.
Intercom.io Extractor
Extractor for https://www.intercom.io/ has been added to Keboola Connection.
"Load more" button added in Notifications
You can now browse older notifictionas with "Load more" button on Keboola Connection Notifictions page.
Changes to R transformations
We have made some internal changes to R transformation backend which resolves a couple of edge-case issues and bring few new features:
- We have unified R Transformations and R Custom Science to use exactly the same environment - this solves issues when using code from transformations in Custom Science.
- R Transformation scripts now have properly set encoding to UTF8 - manually setting encoding with Sys.setlocale("LC_CTYPE", "cs_CZ.UTF-8") is no longer necessary
- R packages are now installed from multiple backup CRAN repositories - this solves an issue where a package installation would randomly fail in case of CRAN outage
- When installing R packages we now automatically load them - this solves an issue where a package was successfully installed, but failed to load. This also has the nice side effect that, you no longer need to call library() explicitly to load packages in your R code.
- R script output is now available in Transformation events, so you can do some basic logging with print('message') or a bit nicer write('message', stdout())
You don't need to make any changes to your R scripts. If you run into any incompatibilities, let us know.
Other posts this week
AdWords Extractor has been rewritten as Docker component and the old version will be switched off by the end of March. Till then its users are kindly requested to migrate their configuration to the new version. Here is how to do it:
- Choose AdWords v2 from list of extractors
- Create new configuration and give it some name as usual
- Click on Authorize Account button which will redirect you to Google and ask for authorization to download your AdWords data.
- Add extraction configuration to the Parameters text area. It is in json format and must contain AdWords developer token, customer id, bucket where you want the data to be saved and configuration of queries. Optionally it may contain parameters since and until to specify dates interval for stats (it is "-1 day" i.e. "yesterday" by default). See documentation for more information.
Please note that the new Extractor saves data incrementally which means that you have to add primary keys if you want to fill data to existing tables.
Our GoodData Writer is going to begin saving configurations to our Components Configurations in the Storage API, rather than the sys stage bucket and tables it uses now. This will bring a performance boost, reduce the API response time (i.e. improve KBC UI performance), improve the clarity of the configuration and bring better versioning and rollback possibilities. This change will apply for bucket attributes (containing GoodData credentials and project identifiers), date dimensions, datasets and filters, see the API docs for more information: http://docs.keboolagooddatawriterv2.apiary.io/#introduction/configuration-in-storage-api/components-api
We will proceed in several steps. First, configurations of all writers will be automatically migrated to the Configuration storage in several waves over the upcoming days. You will be notified by the notifications system in KBC UI once the migration is complete. Once completed, every configuration change performed using the API (including KBC UI) will be written to both places. Please keep in mind however, that if you alter your writer’s configuration directly in sys bucket, these changes won't be synchronised to the new configuration and you will need to manage the synchronisation yourself.
In the second step, we will switch to reading configurations from the Configuration storage rather than the sys bucket. And in the third step, writing to the sys bucket will be stopped and sys bucket configuration tables (data_sets, date_dimensions and filters) will be deleted. We will inform you about each of these steps when they occur.
The remaining configuration tables (projects, users, project_users, filters_projects and filters_users) will not be changing for now, but in the long term they will be completely removed and their data will be only stored in the Writer's backend. So by the end of this journey, we will say goodbye to the whole configuration bucket in sys stage.
Feb 25th 10:00 GMT+0100
We're investigating issues when working with transformation configurations (creating, saving, deleting). We're sorry for this inconvenience, it will be fixed shortly.
UPDATE Feb 25th 10:10 GMT+0100
The issue has been resolved. Thank you for your patience.
Latest addition to our growing family of database extractors is Firebird Extractor.
It is now available in our component list with generic UI. Check out sample configuration to help you get started.
Today we are pleased to introduce a new component that will help you index your KBC data in Elasticsearch.
It is now available in our component list with our generic UI. Please check out the documentation on Github to help you get started.