On May 5, 2017 at 6:39:03 PM UTC+2 one of our AWS RDS server was restarted. This lead to loss of connection between our lock server and job processes. The outcome may be following

• Orchestrations, that have all tasks successfully finished, but the orchestration itself failed

• Failed tasks within orchestrator; the task execution finished successfully, only the job status was not saved correctly

Developers Documentation

We completely rewrote documentation of our Generic Extractor. It was moved from github to developers.keboola.com. Follow this step-by-step tutorial to learn how to configure extractor for new api.

Transformations

Update of python for custom science and transformations to version 3.5.3.

GoodData Writer

Intermittent data load errors were fixed using zip for data loads to avoid bug in GoodData WebDav with gzip compression.

There were few job failures and the UI encountered a brief outage in some API calls on May 3 between 16:07 and 16:14 GMT+2/CEST. 

Affected jobs and API calls returned an Application error. 

The outage was caused by an automatic upgrade of one of our metadata DB servers. We will be turning off automatic upgrades to prevent this in the future.

We're sorry for this inconvenience.

There were few jobs failures between 16:00 - 17:00 CET and 21:00 - 22:30 CET. We have identified the root causes of the problem and rollbacked to previous version.

Affected jobs were returning error:  Authentication token has expired. The user must authenticate again

We're sorry for this inconvenience.

Long neglected, long awaited, it's here!

We are done with a major overhaul of our Generic Extractor documentation. It is now completely moved to developers.keboola.com. The new documentation contains over 100 runnable examples. We have also created a tutorial which guides you through how to configure the Generic Extractor for a new API.

We have also a configuration map with links to documentation of all parts of the configuration.

In case you haven't heard of the Generic Extractor, it is our universal REST API client. It reads data (in JSON format) from an API according to how you configure it, then converts the results to CSV files and imports them into KBC Storage. This way you can extract data from new APIs in tens of minutes. You can use the Generic Extractor to create a new extractor or perform ad-hoc extractions from an API. 

If you are interested, give a try to the Tutorial.

UI Improvements

•  Filter configurations in each section(transformation/extractors/writers/orchestrations/applications)

• Move configuration to trash without confirm dialog - you can revert deleted configuration right after if you did delete the configuration by mistake.

No project should be affected in their daily operations and no action is required, this is a purely informational message about an upcoming change.

From May 3rd we will begin gradually turn on strict primary key checking in all applications. This will cause all components writing to Storage to fail if the primary key on the table does not match the one set in the component's configuration. 

Previously the primary key was set when the table was created and was not checked any further. 

We recently started monitoring primary key mismatches and tried to fix them automatically - the component executor tried to modify the primary key on the table so that it would match the configuration. A few situations arose where it couldn't be fixed automatically:

• New primary key contained columns not present in the table
• New primary key could not be set as there were duplicate values in the new primary key columns
• Several configurations in the project were changing the primary key back and forth

We have contacted all projects affected by these issues and resolved them. We will continue to monitor this situation closely until we switch on the strict checking. 

Some data loads may end with similar error message: {"upload_status.json":"SLI CSV \"dataset.outcmaincalled_numbers.csv\" does not contain header line"}. It is caused by an error in GoodData WebDav and GoodData support is working on a fix. The occurence is rather sporadic and appears in one or two client projects a day and usually vanishes after few hours.

We have slightly updated some of the input fields in the Transformations UI. 

• Packages, Stored Files and Requires are editable straight away and save immediately after each change.
• Queries and Scripts are editable straight away too, but you can reset (to the previous state) or save the change with a button.

No more clicking to edit a field. If you delete or add something by accident, you can always revert to the previous version.

Let us know what you think!

On April 11 between 4:15pm and 4:31pm CEST our connection to quay.io was down. All Docker images stored there were inaccessible. That caused failures in running some components. If you encountered an application error during that period, this outage was likely the culprit. 

Unfortunately status.quay.io remains silent about this outage. 

We are sorry for this inconvenience and to mitigate similar issues in the future, we will move all used Docker images from public repositories (DockerHub and Quay.io) to our private repository in AWS ECR.