Keboola Status

By now, all writers are reading configurations of datasets, date dimensions and filters from Components Configurations API only. Corresponding tables in Storage API still exist and are updated even when they are not used for reading. 

Table users is not used for reading nor updating anymore and information about users created by writers are stored in Writer's backend exclusively. You can access the data using API: http://docs.keboolagooddatawriterv2.apiary.io/#reference/projects-and-users-provisioning/users.

Table filters_projects is not used anymore, GoodData URIs of the filters were moved to Component configuration to section filters.

Table filters_users is not used anymore, information about assigned filters to users is obtained directly from GoodData API. Notice that it brings implicit obsoletion of sync-filters API call. The API call still rebuilds filters-users relations according to the filters_users table but the table isn't updated.

Last tables actively used by Writers are projects and project_users and they will be migrated soon, probably this week. They will be moved to Writer's backend similarly to users table.

Please don't delete configuration tables yourself, whole buckets will be deleted automatically when they won't be used anymore, probably within two weeks.

AdWords Extractor has been rewritten as Docker component and the old version will be switched off by the end of March. Till then its users are kindly requested to migrate their configuration to the new version. Here is how to do it:

1. Choose AdWords v2 from list of extractors
   

   
2. Create new configuration and give it some name as usual
   

   
3. Click on Authorize Account button which will redirect you to Google and ask for authorization to download your AdWords data.
   
4. Add extraction configuration to the Parameters text area. It is in json format and must contain AdWords developer token, customer id, bucket where you want the data to be saved and configuration of queries. Optionally it may contain parameters since and until to specify dates interval for stats (it is "-1 day" i.e. "yesterday" by default). See documentation for more information.
   

   

Please note that the new Extractor saves data incrementally which means that you have to add primary keys if you want to fill data to existing tables.

Our GoodData Writer is going to begin saving configurations to our Components Configurations in the Storage API, rather than the sys stage bucket and tables it uses now. This will bring a performance boost, reduce the API response time (i.e. improve KBC UI performance), improve the clarity of the configuration and bring better versioning and rollback possibilities. This change will apply for bucket attributes (containing GoodData credentials and project identifiers), date dimensions, datasets and filters, see the API docs for more information: http://docs.keboolagooddatawriterv2.apiary.io/#introduction/configuration-in-storage-api/components-api

We will proceed in several steps. First, configurations of all writers will be automatically migrated to the Configuration storage in several waves over the upcoming days. You will be notified by the notifications system in KBC UI once the migration is complete. Once completed, every configuration change performed using the API (including KBC UI) will be written to both places. Please keep in mind however, that if you alter your writer’s configuration directly in sys bucket, these changes won't be synchronised to the new configuration and you will need to manage the synchronisation yourself.

In the second step, we will switch to reading configurations from the Configuration storage rather than the sys bucket. And in the third step, writing to the sys bucket will be stopped and sys bucket configuration tables (data_sets, date_dimensions and filters) will be deleted. We will inform you about each of these steps when they occur.

The remaining configuration tables (projects, users, project_users, filters_projects and filters_users) will not be changing for now, but in the long term they will be completely removed and their data will be only stored in the Writer's backend. So by the end of this journey, we will say goodbye to the whole configuration bucket in sys stage.

DoubleClick Extractor Update

DoubleClick Extractor has been updated to support version 2.3 of DoubleClick API.

Creating tables with large amount of columns bugfix

Creating tables with large amount of columns caused internal error in Storage API. Now it produces a user error with proper error message. Current limits for number of columns are 4096 on MySQL backend and 1600 on Redshift.

GoodData Writer API news

We are progressively working on new version of GoodData Writer API. You can see it's current status in Apiary: http://docs.keboolagooddatawriterv2.apiary.io. This week we finished resource for handling user filters and SSO access to projects.

Other posts this week

• Twitter extractor issues
• SAS Writer
  
• Automatic configuration of GoodData Writer from existing project
  
• Jobs outage
  

If you create new Writer's configuration from existing GoodData project, our app will read it and attempt to set it up automatically. It will also prepare data bucket in Storage API out stage with tables. So you just need to fill them with data and you can start using the writer instantly.

GoodData platform is experiencing some technical difficulties, details can be found here: https://support.gooddata.com/hc/en-us/articles/215573708

The problem affects access to some projects according to the statement but it looks that login to the frontend isn't working too. Some jobs of GoodData Writer may be also affected.

In last days several errors with message like "Could not export table out.c-main.products from Storage API: Table Activities not found in bucket out.c-main." appeared unexpectedly. It is direct consequence of this change from December. When you call upload project or load-data and load-data-multi API calls without parameter tables, Writer takes all configured tables and tries to upload them. But when some table is missing from Storage API and still has configuration in the writer, this failure happens. And it didn't happen earlier because Writer automatically removed configuration for deleted SAPI tables.

But because this problem confused several of our clients, we decided to make this behaviour more comfortable. Now if you call upload project or load-data and load-data-multi without parameter tables, Writer will ignore configurations of non-existing tables and won't fail. However if you call load-data or load-data-multi with explicitly listed tables (in parameter tables) and some of these tables doesn't exist, the job will still fail.

We apologize for the confusion.

Tables are not automatically added to GoodData Writer UI any more and you have to add them there explicitly. Naturally you can also delete no longer needed tables. This improvement should benefit clarity and unites behaviour across other writers. Non-configured tables (i.e. those with all columns ignored) will be removed from your existing configurations soon by automated script.

Sklik Extractor has been rewritten as Docker component and the old version will be switched off by the end of December. Till then its users are kindly requested to migrate their configuration to the new version. Here is how to do it:

1. Choose Sklik v2 from list of extractors
   

   
2. Give it some name as usual

   
   
   
3. Add extraction configuration to the text area. It is in json format and must contain Sklik account's username and password and bucket where you want the data to be saved. Optionally it may contain parameters since and until to specify dates interval for stats (it is "-1 day" i.e. "yesterday" by default)
   

   

Please note that the new Extractor saves data incrementally which means that you have to add primary keys if you want to fill data to existing tables. You can find the required format of data tables in documentation.

Some data loads are failing due to a bug in WebDav storage on GoodData's site introduced in their Saturday release. They are working on fix, be patient please. Jobs affected by this problem fail with message "Csv file has not been uploaded to 'https://secure-di.gooddata.com/uploads/....csv'". We apologize for all the inconveniences.

Update 16:15 UTC: According to GoodData support this problem unfortunately won't be fixed earlier than on November 26 9:00 UTC. See their maintenance announcement