New Features and Improvements

Require MFA for organization

There's a new option in organization-level settings which allows you to require Multi-Factor Authentication for each member of your organization and project.

Users not having the Multi-Factor Authentication enabled will be prompted to enable it.

Set same data type to all columns at once

When setting data types in transformation's Input Mapping you can set same data type to all columns at once.

GoodData Writer data types

When configuring tables in GoodData writer, we try to look for metadata for columns and set types for you automatically.

New Components

Google Translate

Translates text into any supported language.

https://components.keboola.com/components/kds-team.ex-google-translation

Google NLP

New component which uses Google Cloud Natural Language API to extract text features using pre-trained machine learning models.

https://components.keboola.com/components/kds-team.ex-google-nlp

The old version of GoodData Writer (v2) was already unpublished a few months ago and does not allow you to create new configurations since then. Now the time has come to deprecate the component entirely and set a date for its final shut down. 

The date is September 31, 2019, so you are encouraged to migrate old configurations until then. The ui shows a deprecation notice and offers you to run the migration process.

New Features

• You can now schedule orchestrations by table events.  Do you want to run an orchestration as soon as a key table gets updated?  You can do that now!  Please refer to the documentation to help guide you through it.
  

  
  
  
  
• When creating a new token you can now restrict its access to the configuration trash bin
  

  

Improvements

Transformations

• Transformations can now be copied to the same bucket or an other transformation bucket
  
  
• Simplified version diffs for scripts
  

  
  
  
  
• You will now be given a warning if you cannot extend your sandbox because of required system upgrades
  

  
  
  
  

Orchestrations

• For notifications, you can now choose from any email of any user that has previously worked on the orchestration or of course enter one of your choosing

  
  
  
  
• On the orchestrations listing page it will show if the orchestrations were manually triggered, event triggered, or scheduled
  

  
  
  
  

SQL Server extractor

• Allows incremental fetching by timestamp data type columns
  
  
• A new option to disable use of the fallback driver on failure

Developers

• We are now always displaying the documentation link for components, so please make sure that your components have it filled in the Developer Portal.  If you have the documentation link then you can remove excess documentation from the long description.

Today, Thursday 20, 2019 from 02:02 CET, one of our job workers became unresponsive.

This resulted in some jobs taking longer to process as they had to wait for one of the other available workers whose workload had increased as a result.

The affected instance was replaced at 15:50 CET and we are working on improving the automated detection of such failures.

Thank you for your understanding.

Many of you have noticed that our website was redesigned a few months ago. We have been continuing to prepare and apply a new design to all remaining parts of our system:

Four months ago, we introduced the Refined Storage Console, even though the main point of the changes was to remove old code. Four weeks ago, we started using the Roboto font in Keboola Connection and Developer Portal.

We're now happy to announce that the first part of the redesign has been applied to all pages you see during the authentication process.

The authentication process itself wasn't affected by the changes and you should be able to go through it with ease.

There are many more things to come from us. We're already working on the second phase of the redesign. Stay tuned.

• Fixed error when adding metadata to columns with special characters in name.

• Column metadata are now displayed even when they cannot be edited (due to insufficient permissions, table being an alias or table being hosted on Redshift).

Developers

• Removed support for deprecated private Quay.io and private Docker Hub repositories (no active components were using it).

Update 12:36 UTC:

Everything should be back to normal. We'll keep monitoring our systems.

Update 9:15 UTC:

You should not experience any more errors, but things are a bit overloaded so longer execution and waiting times are expected. Thank you for patience.

Update 7:52 UTC:

Snowflake services restored. We're resuming processing of jobs.

Update 6:23 UTC:

Snowflake services are still not restored. We're slowing down job processing, so you'll see a much larger amount of queued jobs. EU region is unaffected.

Original post:

Some Snowflake queries started failing around 03:14 UTC which is causing errors in job processing. Failures are caused by incident which is currently investigated by Snowflake https://status.snowflake.com/incidents/0sjfn3d5jq2q .

We will provide an update as soon as the issue is resolved.

Column descriptions and user defined data types

Description and custom data type can be provided for each table column in Storage. Custom data type allows you to override data type provided by the system. These data types are then used as defaults for transformation and writer table inputs.

You can explore and edit these values on Storage table detail.

Chained aliases

An alias can be created from another alias. Also aliases created in Shared Buckets are propagated to linked buckets and can be further aliased. This simplifies data preparation and sharing, tables which don't require additional processing can be directly aliased to shared buckets.

Chaining is supported only for aliases with automatically synchronized columns and without filter.

Automatic Incremental Processing

With automatic incremental processing, the component will receive only data modified since the last successful run of that component.

Code Templates for  Jupyter and RStudio sandboxes

For Jupyter and RStudio sandboxes, code templates can be defined. Code templates can be set for a given user or for the entire project. A Jupyter template is a notebook file (.ipynb). An RStudio template is a simple text file. If a sandbox is loaded from transformation, the transformation code will be appended after the template code.

Google BigQuery

• New extractor with Google Service Account service authentication was published
• New writer with Google Service Account service authentication was published
• Previous version of the writer is deprecated and will be shut down on 1. 8. 2019.The migration to new version of extractor is available.

Other Updates

• Create a single task orchestration from component configuration
• New version of Zboží.cz Extractor by Medio - Get your daily impressions, clicks, cost and conversion stats for preset time range or previous day.
  
• Python sandboxes and transformations were upgraded to Python version 3.7.3
• R sandboxes and transformation were upgraded to R version 3.5.3

One of our core databases in US region has been restarted due to some failures by AWS. All running jobs failed or will fail due to this reason.

UPDATE: The problem affected ca 26 jobs. There was some unspecified low-level failure on the database or its underlying hardware and AWS automatically switched to a backup instance. We are going to investigate if our platform can handle the failure and restart the jobs automatically next time. Thanks for your understanding, we are sorry for any inconvenience.

Today, on April 13 2019 from 14:14 UTC to 14:15 there were some orchestration failures in the US region due to an internal system upgrade. Very few orchestrations were affected, please review your orchestration and restart if needed. This issue is not related to the issue with failing orchestrations happened previously today.