Week in Review -- May 28, 2018

New Components

OneDrive Extractor (Beta)
  • You can now download your documents from OneDrive. The component was developed by Jakub Bartel.

Google Big Query Writer (Beta)

  • Write data from KBC to Google Big Query with our new writer. The component was developed by us, Keboola s.r.o. 


Updated Components

Currency Extractor

  • Now offers exchange rates of GBP.

Oracle Extractor

  • Now supports exporting columns of the LOB datatype family.

Gmail Attachments Extractor

  • Added support for processors, see example.

Create Manifest Processor 

  • Will test, if all the slices of a CSV file have the same header columns.

Flatten Folders Processor 

  • Throws a user exception when the flattened filename is too long (longer than 255 characters).
  • Has now configurable flatten strategies. Added `hash-sha256` strategy, which solves an issue with 255 characters filename limit in default `concat` strategy.


New Features

  • Component badges.
  • Validate your SQL in transformations with new Validate feature, provided by SQLDep.

  • MySQL extractor now supports incremental fetching. You can extract just the most recent records from a database table and write them incrementally into Storage.


    Minor Improvements

    • Show more / show less button added to the list of inputs and outputs of a job. It now shows all the tables and the view is more compact.

    GoodData Writer Issues

    Today between 2:30 and 7:00 CEST we experienced issues with GoodData Writer. Ironically, it failed to connect to a third-party service for utilization monitoring. The problem was fixed so there should be no other job failure. We are going to inspect the extent of the damage.

    Week in Review -- May 16, 2018

    Core

    Components

    Bugfixes

    • Component configuration state is not updated in case of attached processor failure. e.g in case of AWS S3 extractor and New Files Only option in leaves files as unprocessed in case of processor failure so the files can be processed again until the whole pipeline of processors is executed successfully.
    • MSSQL Writer - fixed support of unicode characters
    • Google Sheets Writer - fixed writing of large tables without performance issues
    • When specifying transformation output mapping, the bucket name is automatically webalized as being typed

    Developers

    We are happy to introduce the first version of Keboola Storage API Javascript client.


    SQL Server Extractor Connection Issues

    Between 17:00 May 15, 2018 and 8:00 on May 16, 2018 CET we experienced issues with the SQL Server database extractor. We recommend to review your orchestrations and take appropriate actions if needed.

    If you were affected by this, please accept our sincere apologies.

    SQL Server writer failures

    Between 9 May 2018, 10:53 CEST and 11 May 2018, 09:30 CEST there were job failures for SQL Server Writer configurations that had nullable data types. The issue was created by a new version of the writer so we have rolled it back to the  previous version while we investigate the root cause.

    We're sorry for any inconvenience. 

    Week in Review -- April 30, 2018

    Core

    • Improved generated configuration changes descriptions
    • Added configuration version to jobs results of Docker-based components (it is not yet available for legacy components like transformation and gooddata-writer)
    • Refreshed Manage API docs with working examples
    • Fixed loading of large tables for R-studio and Jupyter sandboxes
    • Fixed random CSV Import upload errors in EU region

    Components

    • Improved "show details" experience for input and output mappings
    • Added visibility of columns non-existing in Storage to writers
    • Increased query timeout for all Keboola Provisioned Snowflake writers from 15 seconds to 15 minutes
    • Added support of unconventional column names to MySQL extractor
    • Removed static state from MongoDB extractor

    Processors

    • Added support of snappy format to processor-decompress
    • Added processor filter-files
    • Added support for sanitization of invalid utf-8 in processor-iconv

    Developers

    New Debug API call is available (replaces very rarely used sandbox, dry-run and input-data calls). It creates a snapshot of the data directory used for running the component and stores it in your KBC project. To learn more, feel free to go through the API Docs or through the tutorial. In short the API call:

    • uses the same calling convention as the Run API,
    • filters encrypted values from the data directory,
    • works with all components (previously only those without encryption were supported),
    • works with Processors,
    • works with Configuration Rows,
    • works also with broken components and configurations (even if the run fails, you'll still get a snapshot of the data directory).

    Python transformations

    Pip version 10 was released recently which removes the pip.main method (more reading). The recommended way to install packages from within python is:

    import subprocess
    import sys
    subprocess.call([sys.executable, '-m', 'pip', 'install', '--disable-pip-version-check', 'PACKAGE_NAME'])

    Currently there are 70 transformations using the removed pip methods. If your projects are using them we'll be contacting you with a list of affected transformations. This breaking change introduced in pip is currently blocking us from upgrading python to 3.6.5 where pip 10 is used by default.

    Unexpected Job Failures

    Between April 28 2:30 and 3:15 UTC there was a high rate of application errors on one of our instances processing component jobs. 

    The instance was under heavy load and we're investigating the root cause. Instance is now back to normal and is safe to restart the jobs.

    We're sorry for any inconvenience. 

    Degraded performance of Google Sheets Writer

    On March 23, 2018 we released a new version of Google Sheets Writer to remove workaround which resized sheet's grid. Unfortunately this version caused a significant performance degradation for tables with larger number of rows.

    We decided to revert this version to bring back original performance.

    We are working on proper fix and it'll be released soon.

    Orchestration Notification Updates Resulted in Deleted Tasks

    There was an update to the orchestrator this week that had an unfortunate side-effect.  If you updated your orchestrations' notifications it would delete the orchestration's tasks.

    Thankfully, the orchestrations are versioned, so if this happened to you, we will restore the tasks from the last version.
    If you have any concerns about this please contact us at support@keboola.com.  

    For what it's worth, updating notifications will no longer delete orchestration tasks, please accept our humble apologies if you were affected.

    Week in Review -- April 09, 2018

    Updated Components

    Google AdWords Reports

    • This extractor is finally enabled also for customers using EU instance

    Snowflake Writer

    • Added support of VARIANT data type

    Google Drive Extractor/Writer, Google Sheets Writer

    • We added support for Team Drives

    Impala Extractor

    • Added support for internal tables

    Generic Components

    We continue with removing so called "static state" from components. Few weeks ago we removed static state from Transformations, and there was a time for additional components. JSON configurations are also editable straight away. This includes configurations from templates (e.g. Youtube Extractor) and configurations for Custom Science Apps (e.g. Custom Science Python).

    Fixes

    • Python/R transformation sandboxes correctly apply filters in input mappings, so input data will be loaded correctly
    • CSV Import uses server side encryption in S3 stage (before uploading to our storage) by default
    • Gmail Extractor supports "message parts" in more sections and there should no longer be messages without parts
    • ThoughtSpot writer correctly handles the "Test Credentials" action

    Deprecations

    We are deprecating direct import from URL into Storage. Please use the new Http Extractor instead which gives you much more flexibility.