Weeks in review -- May 20, 2019

Improvements

  • Transformation buckets can now run in parallel. That means when you put multiple transformation buckets in one orchestration phase, they will execute in parallel - the same way extractors or writers work. Previously the transformation jobs would be queued to execute sequentially.
  • The editor of transformation queries now uses the official Snowflake syntax highlighter, so the syntax highlighting should now be consistent with the Snowflake Sandbox.

  • The organization settings now offer a new feature to require MFA (Multi-Factor Authentication) for all organization members.

  • A new GoodData User Management Application was released. It replaces part of the functionality of the old GoodData writer and allows to manage users in the GoodData project, invite new users to the project, change roles and assign data permissions for each user.

Fixes

  • A more readable Roboto font is now used in Keboola Connection and Developer Portal UI.
  • Fixed error when adding metadata to columns with special characters in name.
  • Column metadata are now displayed even when they cannot be edited (due to insufficient permissions, table being an alias or table being hosted on Redshift).

Developers

  • Removed support for deprecated private Quay.io and private Docker Hub repositories (no active components were using it).


Snowflake issues in US region

Update 12:36 UTC:

Everything should be back to normal. We'll keep monitoring our systems.


Update 9:15 UTC:

You should not experience any more errors, but things are a bit overloaded so longer execution and waiting times are expected. Thank you for patience.


Update 7:52 UTC:

Snowflake services restored. We're resuming processing of jobs.


Update 6:23 UTC:

Snowflake services are still not restored. We're slowing down job processing, so you'll see a much larger amount of queued jobs. EU region is unaffected.


Original post:

Some Snowflake queries started failing around 03:14 UTC which is causing errors in job processing. Failures are caused by incident which is currently investigated by Snowflake https://status.snowflake.com/incidents/0sjfn3d5jq2q .

We will provide an update as soon as the issue is resolved.


Week in review -- April 26, 2019

Column descriptions and user defined data types

Description and custom data type can be provided for each table column in Storage. Custom data type allows you to override data type provided by the system. These data types are then used as defaults for transformation and writer table inputs.

You can explore and edit these values on Storage table detail.

Chained aliases

An alias can be created from another alias. Also aliases created in Shared Buckets are propagated to linked buckets and can be further aliased. This simplifies data preparation and sharing, tables which don't require additional processing can be directly aliased to shared buckets.

Chaining is supported only for aliases with automatically synchronized columns and without filter.

Automatic Incremental Processing

With automatic incremental processing, the component will receive only data modified since the last successful run of that component.

Code Templates for  Jupyter and RStudio sandboxes

For Jupyter and RStudio sandboxes, code templates can be defined. Code templates can be set for a given user or for the entire project. A Jupyter template is a notebook file (.ipynb). An RStudio template is a simple text file. If a sandbox is loaded from transformation, the transformation code will be appended after the template code.

Google BigQuery

  • New extractor with Google Service Account service authentication was published
  • New writer with Google Service Account service authentication was published
  • Previous version of the writer is deprecated and will be shut down on 1. 8. 2019.The migration to new version of extractor is available.

Other Updates

  • Create a single task orchestration from component configuration
  • New version of Zboží.cz Extractor by Medio - Get your daily impressions, clicks, cost and conversion stats for preset time range or previous day.
  • Python sandboxes and transformations were upgraded to Python version 3.7.3
  • R sandboxes and transformation were upgraded to R version 3.5.3

Jobs failures

One of our core databases in US region has been restarted due to some failures by AWS. All running jobs failed or will fail due to this reason.

UPDATE: The problem affected ca 26 jobs. There was some unspecified low-level failure on the database or its underlying hardware and AWS automatically switched to a backup instance. We are going to investigate if our platform can handle the failure and restart the jobs automatically next time. Thanks for your understanding, we are sorry for any inconvenience.

Some Orchestration Failures in the US Region

Today, on April 13 2019 from 14:14 UTC to 14:15 there were some orchestration failures in the US region due to an internal system upgrade. Very few orchestrations were affected, please review your orchestration and restart if needed. This issue is not related to the issue with failing orchestrations happened previously today.

Failing Orchestrations in the US Region

[2019-04-13 8:20 CET]

We are registering increased amount of failing orchestrations since 2:27 CET. We are investigating the issue now and will keep you updated.


[2019-04-13 8:45 CET]

We traced down the problem and fix it. Everything seems to be working now, we continue monitoring it. Please review your orchestrations and restart them if necessary



Week in review -- April 15, 2019

Updated Components

DB extractors:

  • Source shows schema as well as table name in tables list. Also it's visible from first glance which tables are created from a storage table and which are from SQL. 

  • Table schema is shown also in config rows detail sidebar

  • When adding tables you can click the schema name and all tables in schema at once


Delayed jobs in EU region

Execution of some table import jobs scheduled after 07:42 UTC was delayed up to 30 minutes. The delay was caused by new platform release which was immediately rolled back. All systems are now operational.

Week in review -- March 27, 2019

New Features

  • Added fulltext search in API tokens.
  • Added fulltext search and sorting by name in Orchestrations.

Minor Updates

  • We have discontinued Productboard and all your new ideas come directly into our inbox. Please feel free to share with us anything on your mind.
  • Python and R transformations show the link to the transformation configuration.
  • After successfully validating a SQL transformation you can immediately run it. 


New version of GoodData Writer

We have released a new version of GoodData writer. It runs fully in our Docker infrastructure and so uses its full potential. 

The UI was rewritten and the UX was brought closer to other writers. The component itself handles only updates of data model and data loads. Provisioning of projects and users is a standalone app with backend in AWS Lambda and is used by the ui seamlessly. (You can look to API documentation in Apiary.)

Other custom features of the old Writer like mandatory user filters and cloned projects have to be handled in a custom way outside of the Writer.

The old Writer was unpublished but you can still use existing configurations. Its depreciation and a way of configuration migration will be announced in the near future. But you are strongly encouraged not to create new configurations anymore.