Week in review — January 20, 2020

New Features and Updates

Incremental Fetching

The following extractors now support Incremental Fetching:

  • Snowflake Extractor
  • Oracle Extractor
  • Redshift Extractor
  • Firebird Extractor
  • Cloudera Impala Extractor 

Job Performance

When a job is finished, we show you how long it took. If a job takes 10% longer (or less) than the average, we'll show you the difference.

Transformation Buckets

The list of Transformation Buckets is no longer collapsible and looks like other lists of configurations in an application.

Bucket details in Data Catalog

In a source project (a project which shares a bucket and "owns" data), you are able to see the bucket detail.

  • You can see how the bucket is shared (to organization members, projects members, selected projects, etc.).
  • You can edit the description of the bucket — it will be available for those users who want to link the bucket to their project.
  • You can see a list of tables and bucket properties (size, row count, etc.).
  • Also, if someone uses the bucket, you can see which project it is (Used by section).

In a destination project (where you can see and use a bucket):

  • You can see the description of the bucket and decide if you want to use it.
  • You can see a list of tables and bucket properties (size, row count, etc.).
  • If you decide to use the bucket, you will also see the link to Storage.

Show tables searched by metadata in Writers

In case you use our new feature Scaffolds, related writers (e.g., the Snowflake database writer) will show you all input tables that are searched by metadata.

Environment updates

  • R updated to 3.6.2
  • Python updated to 3.7.5


New Components


VPN connection errors

We are investigating VPN connection issues for extractors and writers in both regions. 

UPDATE Jan 01, 10:03 AM (UTC) - Our VPN provider is investigating the issue

UPDATE Jan 01, 11:54 AM (UTC) - Our VPN provider has found the root cause and fixed the issue. All VPN connections should be working again.

Week in Review — December 12, 2019

New Components

TimeDoctor extractor 

  • Time Doctor is a time tracking software which allows employees to track time per project and task. Team managers are then able to monitor their computer activity and evaluate effectiveness.
  • The Time Doctor extractor for Keboola Connection allows you to download Time Doctor data about users and their activity. 
  • This component was developed by KDS Team.

Snapchat Marketing extractor

  • The Snapchat marketing extractor allows users to download data about marketing campaigns and statistics via the Snapchat Marketing API.
  • It allows to download statistics for campaigns, adsquads, and ads with all supported metrics from Snapchat with either hourly or daily granularity.
  • This component was developed by KDS Team.

Updated Components

  • All database extractors now retry on all the SQL queries they perform.
  • Firebird and Impala extractors now support simple table & column select mode and incremental fetching.

    New Features

    New type of sharing buckets

    • From now on, you can share buckets to specified users. Any listed project users in the organization can use the shared data bucket.

    Minor Improvements

    • Increased speed of name, description and metadata editing in components.
    • Latest Activity shown on the Dashboard also for configurations. We also added grouping by events.
    • Added button for creating a bucket and file in the Storage console.
    • Tables can no longer be created as direct text input. 
    • Redesigned error display for jobs; now errors have their own box and details can be shown in the modal window.
    • Orchestration detail now shows the trendline for previous jobs.
    • Already seen last time was added to Dashboard Latest Activity.

    Slow jobs in EU region

    We are experiencing snowflake jobs being processed too slow in EU region. We keep on investigating the issue.


    UPDATE Dec 11, 10:57 AM (UTC) - We increased computing power of our Snowflake warehouse and there should be no slow jobs for now, furthermore we keep on monitoring the jobs and investigating the root cause.

    Gooddata Writer/Extractor failures in US region

    We are experiencing failures of GoodData writer and extractor configurations that use Keboola provisioned projects in US region. We have identified the root cause and are working on fix it asap.

    UPDATE Dec 11, 10:30 AM (UTC): We have fixed the problem and everything is now working properly. Moreover we found out that regarding GoodData writer, the failure happened only for the Keboola provisioned projects created between Dec 10, 20:00 and Dec 11, 10:15 (UTC).

    Week in Review — December 3, 2019

    New Components

    AppsFlyer Extractor

    • We’re happy to welcome the AppsFlyer Extractor to the family!

    • AppsFlyer is the market leader in mobile advertising attribution and analytics, helping marketers to pinpoint their targeting, optimize their ad spend, and boost ROI.

    • This component was developed by KDS Team.

    DynamoDB Writer

    • Another great addition to our component collection is DynamoDB Writer!

    • This component lets you write data from our Storage into AWS' DynamoDB, which is a key-value, document-based database. It's fully managed, multiregion, multimaster, durable with built-in security, backup and restore, and in-memory caching for internet-scale applications.

    • This component was developed by KDS Team.

    CSV to XLSX Processor

    • Highly demanded utility to convert data in CSV format to XLSX is here!

    • This component was also developed by KDS Team.


    Updated Components

    AWS S3 Extractor

    • Uses parallel download and it's more than 20 times faster!

    Snowflake Extractor

    • Supports incremental load.

    • Supports config rows.

    Oracle Extractor

    • Supports incremental load.

    • Supports configuration rows.

    • Implemented retry mechanism for all types of queries

    Redshift Extractor

    • Supports incremental load.

    MySQL Extractor

    • Implemented retry mechanism for all types of queries

    PostgreSQL Extractor

    • Added support for PostgreSQL Server v12


    Improvements

    Transformations

    • Transformations are collapsed by default.

    Data Catalog

    • Improved "empty" page

    • Type of sharing is now shown and will be improved further. 

    • Improved “initial" modal

    • Bucket description is shown where applicable. 

    Storage

    • When creating an alias table, you can now filter data by the EMPTY_STRING or SPACE characters.


    Fixes

    • Fixed a bug in the Darksky extractor date conversion. The bug caused dates in the date column for places in UTC+ timezones to be one day less than the actual forecast date.


    Developers


    GoodData writer failures (US region)

    We have encountered a small number of failed jobs in US region between 4:46 AM - 9:17 AM UTC.

    Affected jobs finished with Application Error caused by invalid SSL connection to GoodData and we're investigating the issue.

    UPDATE Nov 24, 1:42 PM (UTC) - Only projects having custom GoodData domains was affected by this issue. Problem was with validation of a renewed SSL certificate. The fix is deploying right now.

    Job termination issue

    We are investigating job termination issues for some components. After termination request is accepted jobs remain in terminating state until finished. 

    UPDATE Nov 22, 9:28 AM (UTC) - We have identified the root cause and prepared the fix which should be released in few hours. Only minority of jobs termination was affected by this issue.

    UPDATE Nov 22, 10:49 AM (UTC) - The fix was deployed and job termination is fully functional again for all jobs.

    AWS performance issues in EU region

    Amazon Elastic Compute Cloud is facing increased network connectivity errors to instances, hence the performance of Keboola Connection in EU region is impacted. As a consequence connection to Snowflake and creation of Jupyter/R sanboxes fails, most of the jobs are slow or stalled.

    UPDATE Nov 12, 8:39 AM (UTC) Amazon have identified the root cause and are working to resolve the issue.

    UPDATE Nov 12, 9:10 AM (UTC) Amazon has restored most of the network connectivity to impacted instances and is completing recovery for the remaining connectivity errors. However the performance of Keboola Connection remains impacted.

    UPDATE Nov 12, 9:36 AM (UTC) Keboola Connection is back to operational state, however some jobs might not start immediately due to queuing with the many recently triggered jobs.