Keboola-provided credentials for Snowflake and Redshift database writers

When configuring a Snowflake or Redshift database writer, you can use a Keboola-provided database.

In the past, when you selected this option, the credentials were stored in a configuration in a plain-text format. Storing the credentials this way allowed you to copy the password and use it in your favorite database client (or another system) even if you didn't copy it right after the creation.

To improve the overall security, we decided to show you your password only once and store it encrypted. From now on, when you create a new Keboola-provided database (Snowflake or Redshift), you will see the password only once, right after its creation.

Backward compatibility

The existing credentials will remain untouched. But if you delete them, there's no option to create them the old way.

Week in review -- June 29th, 2020

New Features and Updates

Project Description

Project description is no longer in a read-only mode; you can modify it to fit your needs.

Looker Writer Connection Name

Deprecation of Storage API .NET Client

We decided to deprecate the old and no longer maintained .NET version of the Storage API client. As a replacement we recommend you one of the supported Storage API clients.

Renaming Storage Buckets and Tables

There's a separate post explaining this new feature.

Selecting Bucket in Input Mapping

You can select a whole bucket when adding new tables to Input Mapping. This was originally enabled only for transformations; now you can use this feature for all remaining components.

Bug Fixes

  • Generic Extractor no longer stops after the 2nd page when downloading data in child jobs (only configurations with Limit Stop setting were affected)
  • CSV import component supports full load again (due to a bug, all imports were performed incrementally).
  • MySQL writer no longer writes an "empty string" instead of a null for columns with DATE and DATETIME data types.

New Components

  • CSOB CEB extractor for downloading bank statements from the CSOB CEB Business Connector service
  • Azure Blob Storage writer for exporting any input CSV files into designated Blob containers
  • Sisense writer for sending tables from Keboola Connection to a Sisense database platform
  • Zendesk writer for creating and updating Zendesk Support properties with the flexibility of defining their own parameters

Renaming Storage Buckets and Tables

An option to rename buckets and tables was one of the most requested features on our wishlist. It is very useful when you want to name your bucket by its contents (e.g., "email-orders") rather than "in.c-keboola-ex-gmail-587163382".

From now on, you'll be able to change the names of buckets and tables.

Rename Bucket

To rename a bucket, navigate to the bucket detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (there are some limitations though).

Rename Table

To rename a table, navigate to the table detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (the same limitations apply).

Consequent Changes

Despite the fact that adding the option to rename a bucket or a table does not look like a very big deal, we had to make some substantial changes under the hood. Some of the consequences are worth mentioning here:

Hidden "c-" prefix

We no longer show the "c-" prefix in the names of buckets and tables. It is still a part of the bucket and table ID, but the ID is no longer displayed in most cases. If you need to access the ID for some reason, it is still available on the detail page of each bucket and table.

This is an example of how buckets and tables are displayed without the "c-" prefix:

Stage Selector

When searching for a specific bucket or table, just select a stage and the buckets will be filtered by the selected stage.

Database Writers with Configuration Rows support

We're happy to announce the arrival of Configuration Rows, our new powerful configuration format, to database writers.

From now on, you'll see a migration button in the configuration detail of each database writer (Snowflake, MySQL, SQL Server, Oracle, PostgreSQL, Impala, Hive, and Redshift).

Just click Migrate Configuration and the configuration will be migrated to the new format.

After the migration, you'll see more information about each table. All tables can be easily reordered, so you can move more important tables to the top and they will be uploaded first.

Also, you will be able to see information about each table on a new table detail page, with Last runs and Versions in a sidebar.

Underlying Important Changes

While there were certain limitations in the old configuration format, this is no longer true in the new "rows format".

The following features are worth mentioning:

  • Disabled tables will no longer be exported from Storage (previously, they were exported with limit=1 and not used in the database writer).
  • Each table has its own state with information about the date/time of the last import (previously, an upload of a single table cleared the state for other tables).


Deprecation of public File Uploads

If you are uploading a file to Storage (manually or automatically), there's an option to upload it with the Public flag. The file can then be accessed publicly outside of Keboola Connection.

Only a minority of Keboola Connection users take advantage of this feature, and they do so in a very non-standard way (e.g., for HTML files). That's why we decided to deprecate it. Also, the new File Storage we have prepared (Azure Blob Storage) doesn't support public File Uploads, and we want to make this behavior consistent across all supported File Storage Backends.

The option to create Public Files from the UI has been removed (effective with the publication of this post).

The option to create Public Files via an API will be removed in about three months, by the end of June, 2020.

An alternative solution could be the AWS S3 Writer component, but we don't recommend relying on Public Files at all. Not even outside of Keboola Connection.

Week in review — January 20, 2020

New Features and Updates

Incremental Fetching

The following extractors now support Incremental Fetching:

  • Snowflake Extractor
  • Oracle Extractor
  • Redshift Extractor
  • Firebird Extractor
  • Cloudera Impala Extractor 

Job Performance

When a job is finished, we show you how long it took. If a job takes 10% longer (or less) than the average, we'll show you the difference.

Transformation Buckets

The list of Transformation Buckets is no longer collapsible and looks like other lists of configurations in an application.

Bucket details in Data Catalog

In a source project (a project which shares a bucket and "owns" data), you are able to see the bucket detail.

  • You can see how the bucket is shared (to organization members, projects members, selected projects, etc.).
  • You can edit the description of the bucket — it will be available for those users who want to link the bucket to their project.
  • You can see a list of tables and bucket properties (size, row count, etc.).
  • Also, if someone uses the bucket, you can see which project it is (Used by section).

In a destination project (where you can see and use a bucket):

  • You can see the description of the bucket and decide if you want to use it.
  • You can see a list of tables and bucket properties (size, row count, etc.).
  • If you decide to use the bucket, you will also see the link to Storage.

Show tables searched by metadata in Writers

In case you use our new feature Scaffolds, related writers (e.g., the Snowflake database writer) will show you all input tables that are searched by metadata.

Environment updates

  • R updated to 3.6.2
  • Python updated to 3.7.5


New Components


New UI for Keboola Connection

We started redesigning Keboola Connection in June, 2019. The first pages we changed were the pages related to authentication, followed by moving and redesigning the navigation. Today, we're happy to announce that the 3rd part of Keboola Connection redesign is ready and deployed. The content itself hasn't changed that much, but we invested a lot of time and effort into making all parts look similar to bring you better user experience.

Let's summarize the biggest changes:

  • New project dashboard - shows what's happening in the project.
  • Scaffolds - predefined blocks of configurations for specific use cases
  • Data Catalog - a new way of sharing data in the organization
  • Option to share buckets for a specific project in the organization
  • Option to share individual tables from different buckets (buckets are created automatically)

Brand New Dashboard

The new dashboard is the entry point to your project and you should find the most important information on the project there. You can learn what users are active in your project, who created a configuration recently, which jobs are currently running, etc.

Extractors, Writers and Applications are Components

We grouped all components together in the main navigation.

Each section has two parts:

  • Directory - list of all components available for the selected type (extractor, writer, and application)
  • My components - list of all components you have configured (also for the selected type)

Scaffolds (New in Keboola Connection)

Scaffolds allow you to use a group of prepared configurations for a specific use case by filling in only a few details (e.g., credentials). No additional configuration is required.

Data Catalog (New in Keboola Connection)

This is a new way you can now see the data shared from/to your project:

  • Data which you can use in your project (the section Shared with you)
  • Data you shared from your project (the section Shared from this project)

Bucket Sharing

You can share a bucket directly from the Data Catalog section:

Simple sharing - when you already have a prepared bucket you want to share.

Advanced sharing - when you want to create a new bucket with specific tables.

Also, a new type of sharing has been added. From now on, you can share a bucket to a specific project in your organization.

Storage, Transformations, Orchestrations

These sections have been slightly redesigned, but you should be able to find everything in the same place.

There are a few things we want to mention:

  • In the orchestration detail, the configured token is more visible.
  • Jobs in the orchestration detail can be filtered by their status (successful vs. failed).
  • In the transformation detail, you can find the information about the backend you used and the phase on the right side.

Jobs

The Job detail page has also been redesigned.

Again, as with Transformations or Orchestrations, the content stays the same. We only reorganized it a bit. The event detail is shown in modal. Job parameters and Mapping (information about input/output tables) are shown in modal as well. Just click "Show all" in the particular section.

User and Project menus

These menus haven't changed much either. They were moved to the same line (as was Trash).

Anyway, we're going to continue to improve Keboola Connection for you, there are definitely more things to come. We hope you like our new UI and would love to hear your feedback!

Week in Review -- September 9, 2019

New Features, Improvements and Minor Fixes

  • When checking events in the job detail, the Load More button tries to load up to 1,000 events at once.
  • You can search in component configurations when adding a new task to an orchestration phase.

  • Refreshing your token no longer breaks access to your project.
  • Remove Empty Files and Folders processor has a new option available to remove files with whitespace characters only: remove_files_with_whitespace


New Components

Error when saving configuration Post-Mortem

Some of our users experienced errors in Keboola Connection (in both regions) yesterday when saving configurations of several components.

An error was displayed to users who tried to save their configuration between Jul 9, 2019, 12:35 CET and Jul 10, 2019, 14:18 CET.

Mainly users who have an older version of Chrome Browser or any version of the Firefox browser were affected.

The issue was caused by wrong response headers (unsupported by the mentioned browsers) being sent to a preflight request on one of our API services.

We sincerely apologize for the inconvenience caused.

Navigation Moved and Redesigned

Three weeks ago we announced the start of Keboola Connection redesign.

We're glad we can share another important update with you.

We moved the navigation from the left side to the top

This allows you to more quickly navigate within a project and should improve user experience. All sections remain the same and the content hasn't changed.

  • Project Settings can be found on the right side (by clicking on the "cogs")

  • The Project Selector is almost the same as before

  • The User Settings menu has a new home on the top right side

  • Also, we added a shortcut to the Trash, so you no longer need to navigate there through User & Settings


We hope you'll like this update as much as we do.