Week in review -- June 29th, 2020

New Features and Updates

Project Description

Project description is no longer in a read-only mode; you can modify it to fit your needs.

Looker Writer Connection Name

Deprecation of Storage API .NET Client

We decided to deprecate the old and no longer maintained .NET version of the Storage API client. As a replacement we recommend you one of the supported Storage API clients.

Renaming Storage Buckets and Tables

There's a separate post explaining this new feature.

Selecting Bucket in Input Mapping

You can select a whole bucket when adding new tables to Input Mapping. This was originally enabled only for transformations; now you can use this feature for all remaining components.

Bug Fixes

  • Generic Extractor no longer stops after the 2nd page when downloading data in child jobs (only configurations with Limit Stop setting were affected)
  • CSV import component supports full load again (due to a bug, all imports were performed incrementally).
  • MySQL writer no longer writes an "empty string" instead of a null for columns with DATE and DATETIME data types.

New Components

  • CSOB CEB extractor for downloading bank statements from the CSOB CEB Business Connector service
  • Azure Blob Storage writer for exporting any input CSV files into designated Blob containers
  • Sisense writer for sending tables from Keboola Connection to a Sisense database platform
  • Zendesk writer for creating and updating Zendesk Support properties with the flexibility of defining their own parameters

OAuth Component Authorization errors

We’re currently experiencing OAuth authorization errors returning error:

Docker encryption error: Contact support@keboola.com and attach this exception id

Only authorization of new configuration is affected, running jobs aren't affected. First occurrence of the error was at 14:02 UTC.

We are performing rollback. Next update in 60 minutes or as new information becomes available.

UPDATE 14:42 UTC - Rollback was successfully performed on EU region. Rollback for US region is in progress. Next update in 60 minutes or as new information becomes available.

UPDATE 15:01 UTC - Rollback was performed also on US region. All system are operational.

"Something went wrong" - UI error in OAuth relying components

June 17, 10:26 UTC - After releasing a new version of OAuth broker we have identified that UI of components relying on OAuth authorization is broken, displaying only message: "Something went wrong".

UPDATE 11:01 UTC - We have reverted the release to previous version, which restored the functionality. We are still investigating this issue. It's possible that the problem also affected jobs of these components.

UPDATE 11:31 UTC - We have confirmed that the jobs of the components were unaffected by this bug

Failed attempts to run jobs in US region

Some API calls to run jobs are ending with an application error in the US region. We're investigating the causes.

Update 0:40 UTC: The problem is resolved now. One of the API worker freeze. Running jobs were not affected. We will post a detailed analysis within a week.


Snowflake query incidents

We are investigating Snowflake query failures. Affected queries ends up with similar error message:

Processing aborted due to error 300005:3495968563; incident 9229003

Only minority of projects and queries are affected. We noticed first occurrences on July 8th. We are in touch with Snowflake Support, issue is related to new Snowflake releases and they are investigating it. Next update in 120 minutes or as new information becomes available.

UPDATE 8:20 UTC - Snowflake engineering is working on this issue.

UPDATE 9:35 UTC - Snowflake engineering has already identified the issue, they are testing the changes and also working on rolling out the change. Next update in 120 minutes or as new information becomes available.

UPDATE 10:50 UTC - Fixed version will be released within 24 hours by Snowflake. Next update in 6 hours or as new information becomes available.

UPDATE 17:24 UTC - Snowflake engineering is still working on releasing the patch. The estimate release in 24 hours still holds. Next update in 4 hours or as new information becomes available.

UPDATE 22:10 UTC - According to Snowflake engineering, the the issue is fixed. We're monitoring the situation.

UPDATE June 11, 05:28 UTC - Previously affected queries were executed successfully. Unfortunately few other queries ended up with incident between  Jun 11 02:20:59 - Jun 11 02:25:50 in EU region and one query at Jun 11 01:10:00 UTC in US region. Snowflake engineering is investigating the issue. Next update in 4 hours or as new information becomes available.

UPDATE June 11, 8:39 UTC - Snowflake engineering confirmed that queries which failed tonight were still running on affected release while clusters were still migrating to newer release. At the moment we don't register any failures of queries running on new release. We're monitoring the situation. Next update in 12 hours or as new information becomes available.

UPDATE June 11, 20:52 UTC - There were no query failures since last update. We'll continue to monitor the situation.

UPDATE June 12, 6:30 UTC - There were no query failures since last update. The issue is now resolved. We apologize for the inconvenience. If you have any questions or see any related issues, please contact Keboola Support.

Oracle extractor higher error rate

We are investigating higher error rate of Oracle extractor jobs. Affected jobs ends with error:

DB query failed: Export process failed: Connection error: IO Error: The Network Adapter could not establish the connection Tried 5 times.

Next update in 60 minutes or as new information becomes available.

UPDATE 10:14 UTC: We have preventively rollbacked to previous version of extractor and we are monitoring presence of the failures. Next update in 60 minutes or as new information becomes available.

UPDATE 10:40 UTC: After the rollback all previously affected configurations are running ok. We are investigating what caused the regression in new release and we'll provide details in next three days.

UPDATE June 9, 07:14 UTC: Failures were caused by incorrect connection parameters handling in component. Only configurations using SSH Tunnel were affected. We are working on better test coverage for these cases to avoid the similar issues. We sincerely apologize for the errors.  

MySQL extractor errors

We are investigating MySQL extractor errors for some configurations ending with error message: 

The "incrementalFetchingColumn" must be configured, if incremental fetching is enabled.

The issue is probably caused by new extractor release on Jun 08 05:55 UTC, we are doing rollback.

UPDATE 07:07 UTC: We have rollbacked the previous version. All affected configurations should start working in five minutes.

UPDATE 07:18 UTC: All affected configurations are working again, last error at Jun 08 07:07:45

GoodData Writer failures in EU region

There is some problem on GoodData API since about 08:00 UTC which causes failures on some model updates and data loads.

We are investigating the problem with GoodData support and will keep you updated.

Update 14:20 UTC - The GoodData Technical Support team is still investigating this issue.

Our GoodData writer component has not changed since January, so we are waiting for a clarification of the root cause from their support team.

Update 15:10 UTC - This issue is related to GoodData's release today. They are preparing a hotfix for it and expect it to be deployed in a few hours.

Update June 5, 08:20 UTC - The problem has been resolved. GoodData deployed a hotfix for their API last night. Since June 4 21:00 UTC we have not seen any new errors from the GoodData Writer.

Python/R Sandboxes failures

6:28pm UTC: We are experiencing Python/R Sandbox failing to create in EU and US region and keep on investigating the problem. We'll keep you updated.

6:38pm UTC: We have identified the root cause being expired CA intermediate certificate. We proceed to replace the expired certificate for the sandbox instances. Next update within an hour.

7:38pm UTC: We have successfully replaced CA intermediate certificate for the US region and Python/R sandboxes successfully create now. EU region CA intermediate certificate replacement  is on the way. Next update within an hour.

7:43pm UTC: The Python/R Sandboxes failing to create is resolved now. We have replaced the expired CA intermediate certificate and new Python/R sandboxes create successfully now in both US and EU region. We are sorry for the inconvenience.


Renaming Storage Buckets and Tables

An option to rename buckets and tables was one of the most requested features on our wishlist. It is very useful when you want to name your bucket by its contents (e.g., "email-orders") rather than "in.c-keboola-ex-gmail-587163382".

From now on, you'll be able to change the names of buckets and tables.

Rename Bucket

To rename a bucket, navigate to the bucket detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (there are some limitations though).

Rename Table

To rename a table, navigate to the table detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (the same limitations apply).

Consequent Changes

Despite the fact that adding the option to rename a bucket or a table does not look like a very big deal, we had to make some substantial changes under the hood. Some of the consequences are worth mentioning here:

Hidden "c-" prefix

We no longer show the "c-" prefix in the names of buckets and tables. It is still a part of the bucket and table ID, but the ID is no longer displayed in most cases. If you need to access the ID for some reason, it is still available on the detail page of each bucket and table.

This is an example of how buckets and tables are displayed without the "c-" prefix:

Stage Selector

When searching for a specific bucket or table, just select a stage and the buckets will be filtered by the selected stage.