Updated Components

• Dark Sky extractor supports future dates (and gets a forecast for up to 7 days ahead).

Minor Improvements

• Setting up MFA in Keboola Connection can display the code in plain text where QR is not available.
• We have changed Docker storage driver for Docker Runner job workers. We hope this will stabilize docker response times and minimize startup and shutdown overhead. This will be most significant when running short jobs.
• Editing components in Keboola Developer Portal got a facelift.

New Features

• You can describe your Orchestrations now. This can be useful when you collaborate with your colleagues or want to quickly find what you need among dozens of Orchestrations.

• Also, Orchestrations just got their internal versioning system (joining Extractors, Writers and Applications). At this stage, the versioning is read-only, so you can compare two versions. Rollbacks and revision copies will be added soon.

Minor Improvements

• Improved performance of incremental load. Also, the writer now checks if the primary keys in its configuration match with those in the target database table.
  

Fixes

• Google Calendar Extractor is now available also in the EU region.

• Gmail Extractor is now renamed to Gmail Messages.

Unfortunately tonight there were a few more unexpected Application errors and delayed or longer running jobs between 1:25am–4:55am CEST (4:25pm–7:55pm) in the US region.

We have experimented with different storage drives (swapping from SSD to throughput optimized HDD) which lead to initial issues with building Custom Science apps. Attempts to provision further resources lead to too many running jobs at once (you could see "SQLSTATE[HY000] [1040] Too many connections" in the failed app events) and removing some of the additional resources could have yielded some other Application errors too. 

Currently we're running SSD drives again with enough resources to process all workloads. Please restart your failed jobs.

We hope we'll be able to stabilize this whole unfortunate situation as soon as possible and we're very sorry for inconvenience.

Unfortunately we we're unable to find a fix for yesterday's failures, so on Thursday June 7th between 3:49am CEST and 7:38am CEST (1:49am–5:38am UTC, 6:49pm–10:38pm PT) there was an increased application error rate on our Docker host instances in the US region.

The servers are now stabilized and it is safe to restart the failed jobs.

We're looking into this issue. We have started additional instances to help with the load and we'll be looking into the HW architecture of the instances to help us figure out what causes the issue. Meanwhile we'll try to implement a retry on such failed jobs.

We're sorry for this inconvenience.

On June 6th between 2:15am CEST and 2:35am CEST (5:15pm PT and 5:35pm PT, 12:15am UTC and 12:35am UTC) there was an increased rate of application errors on one of our Docker host instances in the US region. The instance is now fully operational and the jobs are safe to restart.

Furthermore one of our EU region Docker host instances went down at 6:56am CEST and caused a few unexpected application errors. There is a new one in place, we recommend restarting any failed jobs.

We're sorry for this inconvenience, we're working on preventing these errors in the future.

New Components

• You can now download your documents from OneDrive. The component was developed by Jakub Bartel.

Google Big Query Writer (Beta)

• Write data from KBC to Google Big Query with our new writer. The component was developed by us, Keboola s.r.o. 

Updated Components

• Now offers exchange rates of GBP.

Oracle Extractor

• Now supports exporting columns of the LOB datatype family.

Gmail Attachments Extractor

• Added support for processors, see example.

Create Manifest Processor 

• Will test, if all the slices of a CSV file have the same header columns.

Flatten Folders Processor 

• Throws a user exception when the flattened filename is too long (longer than 255 characters).

• Has now configurable flatten strategies. Added `hash-sha256` strategy, which solves an issue with 255 characters filename limit in default `concat` strategy.

Today between 2:30 and 7:00 CEST we experienced issues with GoodData Writer. Ironically, it failed to connect to a third-party service for utilization monitoring. The problem was fixed so there should be no other job failure. We are going to inspect the extent of the damage.

Core

• Removed 1000 slices limit for load into storage
• All components now support configuration parameters encryption by default
• Deprecated MySQL transformations can no longer be edited
• Configuration version is displayed on job detail if available

Components

• New Magento 2 Extractor  developed by Leo Chan
• AWS S3 extractor default limit for 1000 files in one run was removed
• Tree Depth Application supports incomplete trees (items with non-existing root).

Bugfixes

• Component configuration state is not updated in case of attached processor failure. e.g in case of AWS S3 extractor and New Files Only option in leaves files as unprocessed in case of processor failure so the files can be processed again until the whole pipeline of processors is executed successfully.
• MSSQL Writer - fixed support of unicode characters
• Google Sheets Writer - fixed writing of large tables without performance issues
• When specifying transformation output mapping, the bucket name is automatically webalized as being typed

Developers

We are happy to introduce the first version of Keboola Storage API Javascript client.

Between 17:00 May 15, 2018 and 8:00 on May 16, 2018 CET we experienced issues with the SQL Server database extractor. We recommend to review your orchestrations and take appropriate actions if needed.

If you were affected by this, please accept our sincere apologies.

Between 9 May 2018, 10:53 CEST and 11 May 2018, 09:30 CEST there were job failures for SQL Server Writer configurations that had nullable data types. The issue was created by a new version of the writer so we have rolled it back to the  previous version while we investigate the root cause.

We're sorry for any inconvenience.