Updated Components

Event triggered orchestrations

Following writers now set data types based on metadata:

• Tableau TDE
• Impala
• Mssql
• Mysql
• Oracle
• PostgreSql
• Redshift
• Thoughspot
• BigQuery v2

You can read more about Storage data types in the docs.

Gooddata Writer v3

• Fact grain is no longer reset when switching from full to incremental load

Other components

• Our new FTP Extractor is out of beta
• AdForm Extractor is deprecated

New Features

Notification of concurrent editing

A warning message is displayed when you try to edit a configuration that's being edited by someone else at the same time. It also warns you if you are editing the same configuration in a different browser tab. 

CAPSLOCK notification for password field

When you focus the password field and your capslock is enabled, you'll see a notification. 

Keboola is hosting our inaugural Q3/2019 customer webinar and we want you there! If you haven’t done so already, please register now.  Learn about new features, functionalities, and components that we have built during the first half of the year.  Additionally, see the exciting plans that we have for Keboola over the next 3-12 months.

We hope you can join us!  The webinar will be available on video by request afterward as well.

Register here.

Some of our users experienced errors in Keboola Connection (in both regions) yesterday when saving configurations of several components.

An error was displayed to users who tried to save their configuration between Jul 9, 2019, 12:35 CET and Jul 10, 2019, 14:18 CET.

Mainly users who have an older version of Chrome Browser or any version of the Firefox browser were affected.

The issue was caused by wrong response headers (unsupported by the mentioned browsers) being sent to a preflight request on one of our API services.

We sincerely apologize for the inconvenience caused.

Three weeks ago we announced the start of Keboola Connection redesign.

We're glad we can share another important update with you.

We moved the navigation from the left side to the top

This allows you to more quickly navigate within a project and should improve user experience. All sections remain the same and the content hasn't changed.

• Project Settings can be found on the right side (by clicking on the "cogs")
  

• The Project Selector is almost the same as before

• The User Settings menu has a new home on the top right side

• Also, we added a shortcut to the Trash, so you no longer need to navigate there through User & Settings

We hope you'll like this update as much as we do.

Zendesk is down, so our support system is inaccessible. Keboola Connection is unaffected, but if you run into anything critical, please reach us via email.

We'll update this post when we obtain more information.

Update 14:36 UTC - Looks like things are back to normal now.

New Features and Improvements

Require MFA for organization

There's a new option in organization-level settings which allows you to require Multi-Factor Authentication for each member of your organization and project.

Users not having the Multi-Factor Authentication enabled will be prompted to enable it.

Set same data type to all columns at once

When setting data types in transformation's Input Mapping you can set same data type to all columns at once.

GoodData Writer data types

When configuring tables in GoodData writer, we try to look for metadata for columns and set types for you automatically.

New Components

Google Translate

Translates text into any supported language.

https://components.keboola.com/components/kds-team.ex-google-translation

Google NLP

New component which uses Google Cloud Natural Language API to extract text features using pre-trained machine learning models.

https://components.keboola.com/components/kds-team.ex-google-nlp

The old version of GoodData Writer (v2) was already unpublished a few months ago and does not allow you to create new configurations since then. Now the time has come to deprecate the component entirely and set a date for its final shut down. 

The date is September 31, 2019, so you are encouraged to migrate old configurations until then. The ui shows a deprecation notice and offers you to run the migration process.

New Features

• You can now schedule orchestrations by table events.  Do you want to run an orchestration as soon as a key table gets updated?  You can do that now!  Please refer to the documentation to help guide you through it.
  

  

  
  
  
• When creating a new token you can now restrict its access to the configuration trash bin
  

  

Improvements

Transformations

• Transformations can now be copied to the same bucket or an other transformation bucket
  
  
• Simplified version diffs for scripts
  

  

  
  
  
• You will now be given a warning if you cannot extend your sandbox because of required system upgrades
  

  

  
  
  

Orchestrations

• For notifications, you can now choose from any email of any user that has previously worked on the orchestration or of course enter one of your choosing

  

  
  
  
• On the orchestrations listing page it will show if the orchestrations were manually triggered, event triggered, or scheduled
  

  

  
  
  

SQL Server extractor

• Allows incremental fetching by timestamp data type columns
  
  
• A new option to disable use of the fallback driver on failure

Developers

• We are now always displaying the documentation link for components, so please make sure that your components have it filled in the Developer Portal.  If you have the documentation link then you can remove excess documentation from the long description.

Today, Thursday 20, 2019 from 02:02 CET, one of our job workers became unresponsive.

This resulted in some jobs taking longer to process as they had to wait for one of the other available workers whose workload had increased as a result.

The affected instance was replaced at 15:50 CET and we are working on improving the automated detection of such failures.

Thank you for your understanding.

Many of you have noticed that our website was redesigned a few months ago. We have been continuing to prepare and apply a new design to all remaining parts of our system:

Four months ago, we introduced the Refined Storage Console, even though the main point of the changes was to remove old code. Four weeks ago, we started using the Roboto font in Keboola Connection and Developer Portal.

We're now happy to announce that the first part of the redesign has been applied to all pages you see during the authentication process.

The authentication process itself wasn't affected by the changes and you should be able to go through it with ease.

There are many more things to come from us. We're already working on the second phase of the redesign. Stay tuned.

• Fixed error when adding metadata to columns with special characters in name.

• Column metadata are now displayed even when they cannot be edited (due to insufficient permissions, table being an alias or table being hosted on Redshift).

Developers

• Removed support for deprecated private Quay.io and private Docker Hub repositories (no active components were using it).

Update 12:36 UTC:

Everything should be back to normal. We'll keep monitoring our systems.

Update 9:15 UTC:

You should not experience any more errors, but things are a bit overloaded so longer execution and waiting times are expected. Thank you for patience.

Update 7:52 UTC:

Snowflake services restored. We're resuming processing of jobs.

Update 6:23 UTC:

Snowflake services are still not restored. We're slowing down job processing, so you'll see a much larger amount of queued jobs. EU region is unaffected.

Original post:

Some Snowflake queries started failing around 03:14 UTC which is causing errors in job processing. Failures are caused by incident which is currently investigated by Snowflake https://status.snowflake.com/incidents/0sjfn3d5jq2q .

We will provide an update as soon as the issue is resolved.

Column descriptions and user defined data types

Description and custom data type can be provided for each table column in Storage. Custom data type allows you to override data type provided by the system. These data types are then used as defaults for transformation and writer table inputs.

You can explore and edit these values on Storage table detail.

Chained aliases

An alias can be created from another alias. Also aliases created in Shared Buckets are propagated to linked buckets and can be further aliased. This simplifies data preparation and sharing, tables which don't require additional processing can be directly aliased to shared buckets.

Chaining is supported only for aliases with automatically synchronized columns and without filter.

Automatic Incremental Processing

With automatic incremental processing, the component will receive only data modified since the last successful run of that component.

Code Templates for  Jupyter and RStudio sandboxes

For Jupyter and RStudio sandboxes, code templates can be defined. Code templates can be set for a given user or for the entire project. A Jupyter template is a notebook file (.ipynb). An RStudio template is a simple text file. If a sandbox is loaded from transformation, the transformation code will be appended after the template code.

Google BigQuery

• New extractor with Google Service Account service authentication was published
• New writer with Google Service Account service authentication was published
• Previous version of the writer is deprecated and will be shut down on 1. 8. 2019.The migration to new version of extractor is available.

Other Updates

• Create a single task orchestration from component configuration
• New version of Zboží.cz Extractor by Medio - Get your daily impressions, clicks, cost and conversion stats for preset time range or previous day.
  
• Python sandboxes and transformations were upgraded to Python version 3.7.3
• R sandboxes and transformation were upgraded to R version 3.5.3

One of our core databases in US region has been restarted due to some failures by AWS. All running jobs failed or will fail due to this reason.

UPDATE: The problem affected ca 26 jobs. There was some unspecified low-level failure on the database or its underlying hardware and AWS automatically switched to a backup instance. We are going to investigate if our platform can handle the failure and restart the jobs automatically next time. Thanks for your understanding, we are sorry for any inconvenience.

Today, on April 13 2019 from 14:14 UTC to 14:15 there were some orchestration failures in the US region due to an internal system upgrade. Very few orchestrations were affected, please review your orchestration and restart if needed. This issue is not related to the issue with failing orchestrations happened previously today.

[2019-04-13 8:20 CET]

We are registering increased amount of failing orchestrations since 2:27 CET. We are investigating the issue now and will keep you updated.

[2019-04-13 8:45 CET]

We traced down the problem and fix it. Everything seems to be working now, we continue monitoring it. Please review your orchestrations and restart them if necessary

Updated Components

DB extractors:

• Source shows schema as well as table name in tables list. Also it's visible from first glance which tables are created from a storage table and which are from SQL. 

  

• Table schema is shown also in config rows detail sidebar

  

• When adding tables you can click the schema name and all tables in schema at once

  

  

Execution of some table import jobs scheduled after 07:42 UTC was delayed up to 30 minutes. The delay was caused by new platform release which was immediately rolled back. All systems are now operational.

New Features

• Added fulltext search in API tokens.
• Added fulltext search and sorting by name in Orchestrations.

Minor Updates

• We have discontinued Productboard and all your new ideas come directly into our inbox. Please feel free to share with us anything on your mind.
• Python and R transformations show the link to the transformation configuration.
• After successfully validating a SQL transformation you can immediately run it. 

We have released a new version of GoodData writer. It runs fully in our Docker infrastructure and so uses its full potential. 

The UI was rewritten and the UX was brought closer to other writers. The component itself handles only updates of data model and data loads. Provisioning of projects and users is a standalone app with backend in AWS Lambda and is used by the ui seamlessly. (You can look to API documentation in Apiary.)

Other custom features of the old Writer like mandatory user filters and cloned projects have to be handled in a custom way outside of the Writer.

The old Writer was unpublished but you can still use existing configurations. Its depreciation and a way of configuration migration will be announced in the near future. But you are strongly encouraged not to create new configurations anymore.

We're investigating issues with Power BI not showing tables in your Snowflake workspace.

We're sorry for this inconvenience, we'll be updating this post with our findings.

UPDATE 17:26 CET

Snowflake confirmed they can reproduce the issue and are investigating further. 

The issue affects transient tables (CREATE TABLE statement), but does not affect views. As a temporary workaround we suggest to log in directly into the Snowflake workspace and create views from all tables in the schema. These views should be immediately visible in Power BI.

UPDATE 11:20 CET

Snowflake are preparing a new version of the ODBC Driver which allows Power BI to see the tables. Use the support button in your project to contact us to obtain this driver.

Today, on March 15 2019 from 16:34:15 UTC to 16:35:12 UTC there were some orchestration failures in the US region due to an internal system upgrade.

There were not many failures (around 20, so very few are affected), but if you had an orchestration running at that time, please check to make sure that you were unaffected.

We are working on making sure that this will not happen again for any future upgrades.

We were affected by a brief outage of the EU region Snowflake database on Mar 07 between 17:45:00 UTC and 18:25:00 UTC in the EU region. The problem affected extractors and transformations. Please check your orchestrations and re-run them if necessary. Projects in the US region were unaffected. We apologise for the inconvenience caused.

New Features

• Oracle Writer - supports setting custom schema in credentials configuration
• GitHub Extractor - adds organization to downloaded commits and issues
• Storage API - supports multiple where filters and order by statement inside datapreview and asynchronous table export. You can quickly search for your data in datapreview with multiple conditions. We are going to add this feature to UI soon.
• BigQuery Extractor - supports extracting data from the EU region

UI

• full page table preview for Storage tables

Bug fixes

• OAuth Broker API - fixed missing component credentials
• Oracle Extractor - fixed manifest for exported tables if there were more than one table with the same name in different schemas
• Transformations 
• unpaired closing comment tag ( */ ) in SQL query is now properly identified as a user error
  
• added additional retries when creating a workspace

GoodData SSO login using GET links will stop working on March 18. It means that property link in the response of SSO login using GoodData Writer will stop working too. 

If you use any custom solution built upon our Writer, you need to migrate it to the new POST login, i.e. take property encryptedClaims from our resource and call this GoodData API call: https://help.gooddata.com/display/doc/API+Reference#/reference/authentication/sso-pgp-login which will login your user. SSO links to GoodData in our Connection UI are already migrated to the new method.

We're happy to announce a small technology update of our Storage Console. Several months in the making, this will allow us to bring new features in the near future.

Even though the primary purpose of this update is to bring the code up to date and align the design with the rest of the UI, we have already made some small improvements:

• You can no longer see an additional loading page when navigating to Storage from other pages.
• Search in buckets (or tables) highlights the matched parts of your search query in yellow.

• An active bucket is highlighted on the left side when its detail or a detail of its table is active.

• Files and Jobs sections are automatically reloaded every 20 seconds.
• Event sections have predefined searches, so you can filter events faster.

• Buttons Create Bucket, Link Bucket and Reload are now bigger.

• There's an option to create Table Alias directly from a table detail (in Actions).

• Other minor cosmetic things like navigation, buttons, etc.

UPDATE 2019-02-25 9:00 AM UTC
A fix has beed deployed and the problem now longer occurs. We suppose, that around 50 jobs suffered from this issue.
We do sincerely apologize for the trouble this may have caused to you. Don't hesitate to contact our support for help.

2019-02-24 10:00 PM UTC
We're experiencing issues with new OAuth Broker API. 

In some cases it might not return the authorised credentials for a component's job. Re-running the job might be successful.

The fix will be deployed very soon.

If you haven't migrated to the new version, please wait until the fix is deployed.

We're terribly sorry for any inconvenience.

We have just released a new version of our OAuth Broker API.

OAuth Broker is a KBC service, which handles the authorisation flow for all KBC components (extractors, writers, ...) using OAuth authorisation and also stores the credentials (tokens) for them.

The new version was needed to simplify integration with KBC and allows us to implement new features into this API more easily.
The features we are preparing are for example automatic refreshing of OAuth tokens if needed, using more than one OAuth client id for better quota limits handling and so on.

The old OAuth Broker is now deprecated, and we ask you to migrate affected configurations credentials before May 1, 2019. We can't migrate these credentials automatically because we cannot modify configurations in your project without your consent.

In the project Overview, you can see whether your project contains any configurations needing migration:

We need to apply an important OS-level security update to R/Python sandboxes environment. Because of that, the existing sandboxes cannot be extended. This means the following:

• R/Python sandboxes created prior 2019/02/12 will be terminated no later than 2019/02/17 14:00 UTC even if you try to extend them.
• If you wish to keep the contents of sandbox created prior 2019/02/12 14:00 UTC, please save them manually and recreate the sandbox
• R/Python sandboxes created after 2019/02/12 14:00 UTC are unaffected
• SQL sandboxes are unaffected

Component Updates

• Python Transformations - now uses the same Python version 3.7.2 as in the transformation sandbox.
• R transformations have a new backend (v 3.5.2), and we added docs about how to do opt-in in the new version.
• Storage Writer - now supports the `recreate` mode that will drop and create the target table.
• Processor Decompress - supports graceful decompression, will skip the file that failed to decompress.
• Mysql/Mssql/ extractors - allow any numeric or datetime type for incremental fetching.
• PostgreSQL -  has automatic increment fetching. UI has to be migrated to the new version (by the green button in the config overview).
• Generic Extractor now supports usage of deeply nested functions.
• Zendesk Extractor - fixed extracting of custom ticket values fields, existing configurations need to be resaved (switch to template->scroll to the bottom-> select a template again and save).
• New component for Mailgun (sending emails).

UI Updates

• Generic Snowflake sandbox - now uses CLONE TABLE load type. It's way faster and it only loads complete tables (no rows sampling).

  

• You can choose a backend version of R/Python transformations.

  

• Snowflake writer - adding a new table now autoloads column datatypes if present (usual for tables originated from db extractors).
• Transformations Output  - shows warning when there are 2 output mappings with the same destination table within one phase.

  

• PostgreSQL Extractor - query editor now supports PostreSQL specific syntax.

Storage and Project Management Updates

• All newly created tables in Storage have 16MB cell size instead of 1MB.
• Limit 110 columns in data preview were removed, contents of wider tables are displayed normally.
• Organization invitations are now working similarly to project invitations - an invited user has to accept the invitation.