Refined Storage Console

We're happy to announce a small technology update of our Storage Console. Several months in the making, this will allow us to bring new features in the near future.

Even though the primary purpose of this update is to bring the code up to date and align the design with the rest of the UI, we have already made some small improvements:

  • You can no longer see an additional loading page when navigating to Storage from other pages.
  • Search in buckets (or tables) highlights the matched parts of your search query in yellow.

  • An active bucket is highlighted on the left side when its detail or a detail of its table is active.

  • Files and Jobs sections are automatically reloaded every 20 seconds.
  • Event sections have predefined searches, so you can filter events faster.

  • Buttons Create Bucket, Link Bucket and Reload are now bigger.

  • There's an option to create Table Alias directly from a table detail (in Actions).

  • Other minor cosmetic things like navigation, buttons, etc.

New Year's Deprecations

It is always a good idea to start the New Year with something new. We decided to do it differently and start this New Year with deprecations. Cleaning up deprecated or obsolete parts of our system also has a place in our TODO lists.

So we'd like to announce the following deprecations of a few components in the US region, MySQL transformations and storage bucket/tables attributes. 

Components

Google Drive Writer (wr-google-drive)

  • We announced the new version of Google Drive Writer on June 28th, 2017.
  • This is the last call, the old component will be shut down by the end of this month - January 31st, 2019.

Lucky Guess (rt-lucky-guess)

  • Enhanced Analysis for Redshift backend is no longer available.
  • This component will be shut down on January 31st, 2019.

SalesForce Extractor (ex-salesforce)

  • This component is already deprecated.
  • This component will be shutdown on January 31st, 2019.

YouTube Extractor (ex-youtube)

  • This component is already deprecated and no jobs were run in last months.
  • This component will be shut down immediately.

Zendesk Extractor (ex-zendesk)

  • A new version of the Zendesk extractor was announced on July 27, 2016.
  • Like the YouTube Extractor, no jobs were run in last months and we will shut it down immediately.

Restbox (restbox)

  • This component was deprecated on July 16, 2018 with multiple replacements.
  • Same here, no jobs were run in last months and it will be shut down immediately.

MySQL Transformations

MySQL Transformations were deprecated on November 13, 2017. As we promised, MySQL Transformations were supported in 2018 and will be finally shut down on January 31st, 2019.

Deprecated Bucket and Table attributes

We are deprecating Bucket and Table attributes. These attributes were used as configuration storage for legacy components. If you need to store additional information with buckets or tables, please use the Metadata API.


Outage Post-Mortem: December 13th, 2018

There was an outage of the Keboola Connection platform in the US region from Dec 13, 2018, 23:19 CET to Dec 14, 2018, 02:28 CET. It was caused by an update of the Elasticsearch Service.

It was a self-service update of the managed Amazon Elasticsearch Service, and we decided to do it because updates were tested and all other clusters we're using were updated successfully - without any issues.

Updates like this usually go smoothly, but not this time. Unfortunately, the cluster froze and refused to accept any requests.

After detecting this, we decided to start a new service and restore from backup.

All services were fully restored to their normal state by Dec 14, 2018, at 02:28 CET.

We have already taken action to prevent this kind of failure from happening again, and we'll be testing all future updates on testing clusters (using snapshots from production clusters).

Any orchestrations in the US region that were scheduled to start during the outage were not started. Therefore you will need to run them manually or wait until their next scheduled run.

We want to sincerely apologize for the inconvenience caused by this outage.

Weeks in review -- December 10, 2018

New Features

Partial Label

We show a PARTIAL label for jobs which didn't run the whole configuration, but only part of it. Typically these are jobs when only one transformation is run from a bucket or when one file is exported using AWS S3 extractor, HTTP extractor, etc.

Transformation description, Last Runs, and Updates

  • There's a new option to save transformation with description.

  • The Last Runs section and Updates section have been added also to the Transformation Detail page and all components which support "single runs" (e.g. when HTTP extractor or AWS S3 extractor is extracting one file)

Input Mapping data types

In transformations with Snowflake backend, data types should be populated automatically for tables created with the components which set data types - MySQL extractor, Oracle extractor, MSSQL Server extractor, PostgreSQL extractor, DB2 extractor, and Snowflake extractor.

Input Mapping load type

For Snowflake backend, you can set Clone Table load type which leads to an ultra-fast load to a workspace. Most of your tables should be loaded under 10 seconds.

You can find more about this feature in our documentation.


Enhancements

  • You can see a component type in Orchestration Tasks, thus having both MySQL writer and MySQL extractor in task won't be confusing anymore
  • API Token can be Refreshed, Sent or Deleted directly from the token detail page

  • We improved how data is loaded to Storage from our components - compression is used by default. Loads to Storage are about 40% faster for sliced files. So it impacts Credit Consumption too.
  • MSSQL Server Extractor has new option WITH(NOLOCK) and supports incremental fetching with smalldatetime column
  • Orchestration Notifications page has been improved, and notifications can be set easier

New components and component updates

New Orchestration Detail page

The orchestration section is one of the last parts of Keboola Connection which does not yet have a unified user interface. We believe that users should feel comfortable using the different parts of our UI, so unifying the interface elements is very important to us.

We have launched the new Orchestration Detail page

What has changed?

  • There is no longer a sidebar on the left side with a list of orchestrations
  • Tasks, Schedule and Notifications now have their own place in the detail page - they're no longer combined in that table
  • We added new sidebar on the right side that you should feel familiar with because it is the same as other components
  • Orchestration Action buttons (Run, Enable/Disable and Delete) have been moved to the right sidebar
  • Also, information about creation date and updates now also have their own place in the right sidebar

This is the beginning of the Orchestration Interface tuning and there are more things to come.

---

Just for completeness, this is the previous version:

Week in Review -- September 19, 2018

Updated Components

Minor Improvements

  • Link to a configuration from the Job Detail page has been improved for the Transformation component and also for some components which support configuration using Configuration Rows
  • From now you can switch to another Keboola Connection region directly from the Login Page

  • We unified search inputs across application, so searching will work in more unified way for you

Client libraries

Week in Review -- June 26, 2018

Updated components

  • FlexiBee extractor has configurable hostname
  • MySQL Writer uses temporary tables for incremental loading, so DROP privilege can be replaced by CREATE TEMPORARY TABLES privilege for incremental loading

Table and column deletion improvements

This applies to table aliases and also to shared buckets (even if tables are already linked to project).

  • Table can be deleted also with its aliases

  • Same applies for columns - column deletion will be propagated also to aliased tables - with two exceptions:
    • not if column is used as filter in alias
    • not if column is in alias with aliasColumnsAutosync turned off

Other changes

  • There's new information on API Tokens page - if token has permission to manage other tokens
  • Output Mapping settings are no longer collapsed and you can edit them with few UI interactions

Snowflake Outage - US Region

We are experiencing Snowflake partial outage which started at 18:11 CEST.

UPDATE, 19:33 CEST: It looks like the problem is solved. We also replaced one of our servers where we detected more connection issues.

UPDATE, 05:00 CEST: The errors began to appear again. We are working with Snowflake support on fix.

UPDATE, 08:46 CEST: Issue was resolved. The root cause was invalid cache of Online Certificate Status Protocol (OCSP)  utilised by Snowflake drivers. We are further investigating the issue with Snowflake support.


Degraded performance of Google Sheets Writer

On March 23, 2018 we released a new version of Google Sheets Writer to remove workaround which resized sheet's grid. Unfortunately this version caused a significant performance degradation for tables with larger number of rows.

We decided to revert this version to bring back original performance.

We are working on proper fix and it'll be released soon.

Week in Review -- April 09, 2018

Updated Components

Google AdWords Reports

  • This extractor is finally enabled also for customers using EU instance

Snowflake Writer

  • Added support of VARIANT data type

Google Drive Extractor/Writer, Google Sheets Writer

  • We added support for Team Drives

Impala Extractor

  • Added support for internal tables

Generic Components

We continue with removing so called "static state" from components. Few weeks ago we removed static state from Transformations, and there was a time for additional components. JSON configurations are also editable straight away. This includes configurations from templates (e.g. Youtube Extractor) and configurations for Custom Science Apps (e.g. Custom Science Python).

Fixes

  • Python/R transformation sandboxes correctly apply filters in input mappings, so input data will be loaded correctly
  • CSV Import uses server side encryption in S3 stage (before uploading to our storage) by default
  • Gmail Extractor supports "message parts" in more sections and there should no longer be messages without parts
  • ThoughtSpot writer correctly handles the "Test Credentials" action

Deprecations

We are deprecating direct import from URL into Storage. Please use the new Http Extractor instead which gives you much more flexibility.