Keboola Status

There was an outage of the Keboola Connection platform in the US region from Dec 13, 2018, 23:19 CET to Dec 14, 2018, 02:28 CET. It was caused by an update of the Elasticsearch Service.

It was a self-service update of the managed Amazon Elasticsearch Service, and we decided to do it because updates were tested and all other clusters we're using were updated successfully - without any issues.

Updates like this usually go smoothly, but not this time. Unfortunately, the cluster froze and refused to accept any requests.

After detecting this, we decided to start a new service and restore from backup.

All services were fully restored to their normal state by Dec 14, 2018, at 02:28 CET.

We have already taken action to prevent this kind of failure from happening again, and we'll be testing all future updates on testing clusters (using snapshots from production clusters).

Any orchestrations in the US region that were scheduled to start during the outage were not started. Therefore you will need to run them manually or wait until their next scheduled run.

We want to sincerely apologize for the inconvenience caused by this outage.

New Features

Partial Label

Transformation description, Last Runs, and Updates

• There's a new option to save transformation with description.
  

• The Last Runs section and Updates section have been added also to the Transformation Detail page and all components which support "single runs" (e.g. when HTTP extractor or AWS S3 extractor is extracting one file)

Input Mapping data types

In transformations with Snowflake backend, data types should be populated automatically for tables created with the components which set data types - MySQL extractor, Oracle extractor, MSSQL Server extractor, PostgreSQL extractor, DB2 extractor, and Snowflake extractor.

Input Mapping load type

For Snowflake backend, you can set Clone Table load type which leads to an ultra-fast load to a workspace. Most of your tables should be loaded under 10 seconds.

You can find more about this feature in our documentation.

Enhancements

• You can see a component type in Orchestration Tasks, thus having both MySQL writer and MySQL extractor in task won't be confusing anymore
• API Token can be Refreshed, Sent or Deleted directly from the token detail page

• We improved how data is loaded to Storage from our components - compression is used by default. Loads to Storage are about 40% faster for sliced files. So it impacts Credit Consumption too.
• MSSQL Server Extractor has new option WITH(NOLOCK) and supports incremental fetching with smalldatetime column
• Orchestration Notifications page has been improved, and notifications can be set easier

New components and component updates

• In RStudio/Jupyter sandboxes there's a hard memory limit set to 8GB. This limit is equal to memory limit in Transformations. With this change, you can be sure that what worked in Sandbox will work in Transformation too.
• Rename Files processor to rename files using regular expressions: https://components.keboola.com/components/jakub-bartel.processor-rename-files
• XLS processor to convert .xls(x) files to .csv files: https://components.keboola.com/components/jakub-bartel.processor-xls2csv
• Snowflake writer uses atomic swap of tables during full load (prevents empty table issue)

The orchestration section is one of the last parts of Keboola Connection which does not yet have a unified user interface. We believe that users should feel comfortable using the different parts of our UI, so unifying the interface elements is very important to us.

We have launched the new Orchestration Detail page

What has changed?

• There is no longer a sidebar on the left side with a list of orchestrations
• Tasks, Schedule and Notifications now have their own place in the detail page - they're no longer combined in that table
• We added new sidebar on the right side that you should feel familiar with because it is the same as other components
• Orchestration Action buttons (Run, Enable/Disable and Delete) have been moved to the right sidebar
• Also, information about creation date and updates now also have their own place in the right sidebar

This is the beginning of the Orchestration Interface tuning and there are more things to come.

---

Just for completeness, this is the previous version:

Updated Components

• Salesforce Extractor has new UI and an option to migrate configuration to a new format
• Teradata Extractor has reached a public beta stage
• and MySQL extractor has improved handling of the "server has gone away" error

Minor Improvements

• Link to a configuration from the Job Detail page has been improved for the Transformation component and also for some components which support configuration using Configuration Rows
• From now you can switch to another Keboola Connection region directly from the Login Page

• We unified search inputs across application, so searching will work in more unified way for you

Client libraries

• New version of Storage API client for R has been released

Updated components

• FlexiBee extractor has configurable hostname
• MySQL Writer uses temporary tables for incremental loading, so DROP privilege can be replaced by CREATE TEMPORARY TABLES privilege for incremental loading

Table and column deletion improvements

This applies to table aliases and also to shared buckets (even if tables are already linked to project).

• Table can be deleted also with its aliases

• Same applies for columns - column deletion will be propagated also to aliased tables - with two exceptions:
• not if column is used as filter in alias
• not if column is in alias with aliasColumnsAutosync turned off

Other changes

• There's new information on API Tokens page - if token has permission to manage other tokens
• Output Mapping settings are no longer collapsed and you can edit them with few UI interactions

We are experiencing Snowflake partial outage which started at 18:11 CEST.

UPDATE, 19:33 CEST: It looks like the problem is solved. We also replaced one of our servers where we detected more connection issues.

UPDATE, 05:00 CEST: The errors began to appear again. We are working with Snowflake support on fix.

UPDATE, 08:46 CEST: Issue was resolved. The root cause was invalid cache of Online Certificate Status Protocol (OCSP)  utilised by Snowflake drivers. We are further investigating the issue with Snowflake support.

On March 23, 2018 we released a new version of Google Sheets Writer to remove workaround which resized sheet's grid. Unfortunately this version caused a significant performance degradation for tables with larger number of rows.

We decided to revert this version to bring back original performance.

We are working on proper fix and it'll be released soon.

Updated Components

Google AdWords Reports

• This extractor is finally enabled also for customers using EU instance

Snowflake Writer

• Added support of VARIANT data type

Google Drive Extractor/Writer, Google Sheets Writer

• We added support for Team Drives

Impala Extractor

• Added support for internal tables

Generic Components

We continue with removing so called "static state" from components. Few weeks ago we removed static state from Transformations, and there was a time for additional components. JSON configurations are also editable straight away. This includes configurations from templates (e.g. Youtube Extractor) and configurations for Custom Science Apps (e.g. Custom Science Python).

Fixes

• Python/R transformation sandboxes correctly apply filters in input mappings, so input data will be loaded correctly
• CSV Import uses server side encryption in S3 stage (before uploading to our storage) by default
• Gmail Extractor supports "message parts" in more sections and there should no longer be messages without parts
• ThoughtSpot writer correctly handles the "Test Credentials" action

Deprecations

We are deprecating direct import from URL into Storage. Please use the new Http Extractor instead which gives you much more flexibility.

Linked/Source Buckets

From now, you'll be able to find source/linked buckets information in Storage section in Keboola Connection. This is very helpful when you need to find out which projects are using (linking) your shared bucket. Also, vice versa, which bucket is the source for your bucket.

MFA required also for Google Login

If you have MFA (Multi-factor Authentication) enabled, a confirmation code is required if you use "Login with Google" functionality. Please contact us if you have problems with login.

Facebok Extractor uses page access token for page/posts insights retrieval

Due to breaking changes recently introduced by Facebok API our facebook extractor is updated so it uses page access token for page or posts insights retrieval instead of user access token. This leads to slower extraction if more pages are included in a query. User access token is still used for other data retrieval such as feed, likes, comments. Moreover, this change does not affect facebook ads extractor. 

Improvements

• We slightly updated UI for the recently published Tokens page

Fixes

• Project Power consumption is shown only for the current + 2 previous months. This is only a temporary limitation.  It will soon be solved and return to showing you more data

In November 2014, we were really happy to announce a new cool feature -- Read-only bucket credentials to your Redshift backend. Time has passed and we have decided to deprecate this feature.

Though accessing data in a bucket is very useful, with the current scenario it can lead to an assumption how data is stored and how it'll be stored in future. The latter is much worse, because any change in our backend can lead to breakage of the system, which was programmed to access the data in a bucket.

Thus, we recommend you to access your data in a more robust way:

1. Create a new Redshift Writer
2. Choose a database provided by Keboola or your own Redshift database
3. Configure which tables you need to synchronize
4. Run the newly configured writer
5. Copy the credentials created in step 2 to the software which needs to access your data

The existing Bucket credentials continue to work until April 18, 2018. However, we strongly recommend to migrate to the Redshift Writer as soon as possible.

The creation of new credentials and credentials listing will be disabled in the Storage section with the publication of this post.

Also, the Bucket credentials have been marked as deprecated in our Storage API.