New UI for Keboola Connection

We started redesigning Keboola Connection in June, 2019. The first pages we changed were the pages related to authentication, followed by moving and redesigning the navigation. Today, we're happy to announce that the 3rd part of Keboola Connection redesign is ready and deployed. The content itself hasn't changed that much, but we invested a lot of time and effort into making all parts look similar to bring you better user experience.

Let's summarize the biggest changes:

  • New project dashboard - shows what's happening in the project.
  • Scaffolds - predefined blocks of configurations for specific use cases
  • Data Catalog - a new way of sharing data in the organization
  • Option to share buckets for a specific project in the organization
  • Option to share individual tables from different buckets (buckets are created automatically)

Brand New Dashboard

The new dashboard is the entry point to your project and you should find the most important information on the project there. You can learn what users are active in your project, who created a configuration recently, which jobs are currently running, etc.

Extractors, Writers and Applications are Components

We grouped all components together in the main navigation.

Each section has two parts:

  • Directory - list of all components available for the selected type (extractor, writer, and application)
  • My components - list of all components you have configured (also for the selected type)

Scaffolds (New in Keboola Connection)

Scaffolds allow you to use a group of prepared configurations for a specific use case by filling in only a few details (e.g., credentials). No additional configuration is required.

Data Catalog (New in Keboola Connection)

This is a new way you can now see the data shared from/to your project:

  • Data which you can use in your project (the section Shared with you)
  • Data you shared from your project (the section Shared from this project)

Bucket Sharing

You can share a bucket directly from the Data Catalog section:

Simple sharing - when you already have a prepared bucket you want to share.

Advanced sharing - when you want to create a new bucket with specific tables.

Also, a new type of sharing has been added. From now on, you can share a bucket to a specific project in your organization.

Storage, Transformations, Orchestrations

These sections have been slightly redesigned, but you should be able to find everything in the same place.

There are a few things we want to mention:

  • In the orchestration detail, the configured token is more visible.
  • Jobs in the orchestration detail can be filtered by their status (successful vs. failed).
  • In the transformation detail, you can find the information about the backend you used and the phase on the right side.

Jobs

The Job detail page has also been redesigned.

Again, as with Transformations or Orchestrations, the content stays the same. We only reorganized it a bit. The event detail is shown in modal. Job parameters and Mapping (information about input/output tables) are shown in modal as well. Just click "Show all" in the particular section.

User and Project menus

These menus haven't changed much either. They were moved to the same line (as was Trash).

Anyway, we're going to continue to improve Keboola Connection for you, there are definitely more things to come. We hope you like our new UI and would love to hear your feedback!

Week in Review -- September 9, 2019

New Features, Improvements and Minor Fixes

  • When checking events in the job detail, the Load More button tries to load up to 1,000 events at once.
  • You can search in component configurations when adding a new task to an orchestration phase.

  • Refreshing your token no longer breaks access to your project.
  • Remove Empty Files and Folders processor has a new option available to remove files with whitespace characters only: remove_files_with_whitespace


New Components

Error when saving configuration Post-Mortem

Some of our users experienced errors in Keboola Connection (in both regions) yesterday when saving configurations of several components.

An error was displayed to users who tried to save their configuration between Jul 9, 2019, 12:35 CET and Jul 10, 2019, 14:18 CET.

Mainly users who have an older version of Chrome Browser or any version of the Firefox browser were affected.

The issue was caused by wrong response headers (unsupported by the mentioned browsers) being sent to a preflight request on one of our API services.

We sincerely apologize for the inconvenience caused.

Navigation Moved and Redesigned

Three weeks ago we announced the start of Keboola Connection redesign.

We're glad we can share another important update with you.

We moved the navigation from the left side to the top

This allows you to more quickly navigate within a project and should improve user experience. All sections remain the same and the content hasn't changed.

  • Project Settings can be found on the right side (by clicking on the "cogs")

  • The Project Selector is almost the same as before

  • The User Settings menu has a new home on the top right side

  • Also, we added a shortcut to the Trash, so you no longer need to navigate there through User & Settings


We hope you'll like this update as much as we do.

Week in Review -- June 27, 2019

New Features and Improvements

Require MFA for organization

There's a new option in organization-level settings which allows you to require Multi-Factor Authentication for each member of your organization and project.

Users not having the Multi-Factor Authentication enabled will be prompted to enable it.

Set same data type to all columns at once

When setting data types in transformation's Input Mapping you can set same data type to all columns at once.

GoodData Writer data types

When configuring tables in GoodData writer, we try to look for metadata for columns and set types for you automatically.


New Components

Google Translate

Translates text into any supported language.

https://components.keboola.com/components/kds-team.ex-google-translation

Google NLP

New component which uses Google Cloud Natural Language API to extract text features using pre-trained machine learning models.

https://components.keboola.com/components/kds-team.ex-google-nlp


Starting Keboola Connection redesign

Many of you have noticed that our website was redesigned a few months ago. We have been continuing to prepare and apply a new design to all remaining parts of our system:

Four months ago, we introduced the Refined Storage Console, even though the main point of the changes was to remove old code. Four weeks ago, we started using the Roboto font in Keboola Connection and Developer Portal.

We're now happy to announce that the first part of the redesign has been applied to all pages you see during the authentication process.

The authentication process itself wasn't affected by the changes and you should be able to go through it with ease.

There are many more things to come from us. We're already working on the second phase of the redesign. Stay tuned.

Refined Storage Console

We're happy to announce a small technology update of our Storage Console. Several months in the making, this will allow us to bring new features in the near future.

Even though the primary purpose of this update is to bring the code up to date and align the design with the rest of the UI, we have already made some small improvements:

  • You can no longer see an additional loading page when navigating to Storage from other pages.
  • Search in buckets (or tables) highlights the matched parts of your search query in yellow.

  • An active bucket is highlighted on the left side when its detail or a detail of its table is active.

  • Files and Jobs sections are automatically reloaded every 20 seconds.
  • Event sections have predefined searches, so you can filter events faster.

  • Buttons Create Bucket, Link Bucket and Reload are now bigger.

  • There's an option to create Table Alias directly from a table detail (in Actions).

  • Other minor cosmetic things like navigation, buttons, etc.

New Year's Deprecations

It is always a good idea to start the New Year with something new. We decided to do it differently and start this New Year with deprecations. Cleaning up deprecated or obsolete parts of our system also has a place in our TODO lists.

So we'd like to announce the following deprecations of a few components in the US region, MySQL transformations and storage bucket/tables attributes. 

Components

Google Drive Writer (wr-google-drive)

  • We announced the new version of Google Drive Writer on June 28th, 2017.
  • This is the last call, the old component will be shut down by the end of this month - January 31st, 2019.

Lucky Guess (rt-lucky-guess)

  • Enhanced Analysis for Redshift backend is no longer available.
  • This component will be shut down on January 31st, 2019.

SalesForce Extractor (ex-salesforce)

  • This component is already deprecated.
  • This component will be shutdown on January 31st, 2019.

YouTube Extractor (ex-youtube)

  • This component is already deprecated and no jobs were run in last months.
  • This component will be shut down immediately.

Zendesk Extractor (ex-zendesk)

  • A new version of the Zendesk extractor was announced on July 27, 2016.
  • Like the YouTube Extractor, no jobs were run in last months and we will shut it down immediately.

Restbox (restbox)

  • This component was deprecated on July 16, 2018 with multiple replacements.
  • Same here, no jobs were run in last months and it will be shut down immediately.

MySQL Transformations

MySQL Transformations were deprecated on November 13, 2017. As we promised, MySQL Transformations were supported in 2018 and will be finally shut down on January 31st, 2019.

Deprecated Bucket and Table attributes

We are deprecating Bucket and Table attributes. These attributes were used as configuration storage for legacy components. If you need to store additional information with buckets or tables, please use the Metadata API.


Outage Post-Mortem: December 13th, 2018

There was an outage of the Keboola Connection platform in the US region from Dec 13, 2018, 23:19 CET to Dec 14, 2018, 02:28 CET. It was caused by an update of the Elasticsearch Service.

It was a self-service update of the managed Amazon Elasticsearch Service, and we decided to do it because updates were tested and all other clusters we're using were updated successfully - without any issues.

Updates like this usually go smoothly, but not this time. Unfortunately, the cluster froze and refused to accept any requests.

After detecting this, we decided to start a new service and restore from backup.

All services were fully restored to their normal state by Dec 14, 2018, at 02:28 CET.

We have already taken action to prevent this kind of failure from happening again, and we'll be testing all future updates on testing clusters (using snapshots from production clusters).

Any orchestrations in the US region that were scheduled to start during the outage were not started. Therefore you will need to run them manually or wait until their next scheduled run.

We want to sincerely apologize for the inconvenience caused by this outage.

Weeks in review -- December 10, 2018

New Features

Partial Label

We show a PARTIAL label for jobs which didn't run the whole configuration, but only part of it. Typically these are jobs when only one transformation is run from a bucket or when one file is exported using AWS S3 extractor, HTTP extractor, etc.

Transformation description, Last Runs, and Updates

  • There's a new option to save transformation with description.

  • The Last Runs section and Updates section have been added also to the Transformation Detail page and all components which support "single runs" (e.g. when HTTP extractor or AWS S3 extractor is extracting one file)

Input Mapping data types

In transformations with Snowflake backend, data types should be populated automatically for tables created with the components which set data types - MySQL extractor, Oracle extractor, MSSQL Server extractor, PostgreSQL extractor, DB2 extractor, and Snowflake extractor.

Input Mapping load type

For Snowflake backend, you can set Clone Table load type which leads to an ultra-fast load to a workspace. Most of your tables should be loaded under 10 seconds.

You can find more about this feature in our documentation.


Enhancements

  • You can see a component type in Orchestration Tasks, thus having both MySQL writer and MySQL extractor in task won't be confusing anymore
  • API Token can be Refreshed, Sent or Deleted directly from the token detail page

  • We improved how data is loaded to Storage from our components - compression is used by default. Loads to Storage are about 40% faster for sliced files. So it impacts Credit Consumption too.
  • MSSQL Server Extractor has new option WITH(NOLOCK) and supports incremental fetching with smalldatetime column
  • Orchestration Notifications page has been improved, and notifications can be set easier

New components and component updates