New Components

• POP3 Attachment extractor – allows you to download email attachments from a POP3 mailbox. There is no restriction on attachment type, so you can also download XLS files or images.

Updated Components

• MongoDB extractor – added an option to “Include parent in Primary Key” in Mapping mode. This will allow better PK generation and the behavior will be much more consistent with Generic Extractor.

• Looker writer – added support for Google BigQuery as the underlying database.

UI Improvements

• If there's a name collision, a numeric suffix is added (for example, if there is an account table from another bucket and you add a new bucket which has a table with the same name, it will be named account_2).
• If there are already tables from the bucket, only the remaining tables will be added.

Force Unlink

There’s a new option to “force unlink” a bucket from projects that have it linked. When you disable sharing, it will “force unlink” the bucket automatically.

Edit Bucket Sharing

From now on, if there are projects that have the bucket linked, you can only add more projects/users. Edit sharing is available from “shared from this projects” list and from “bucket detail in data catalog” pages (if you have sufficient privileges). Until the bucket is linked, sharing can be changed without limitations.

Since 2020-10-15 10:02 UTC Start investigating overload in EU. Next update in 1 hour or as new information

UPDATE 2020-10-15 11:14 UTC: All operations are back to normal and everything is fully working. No jobs should be affected.

We experienced overload in EU

Since 2020-10-12 11:40 UTC Start investigating overload in EU. Next update in 1 hour or as new information becomes available.

UPDATE 2020-10-12 12:56 UTC:  Everything should be back to normal. No jobs should be affected. We'll keep monitoring our platform closely.

UPDATE 2020-10-12 13:57 UTC: All operations are back to normal and everything is fully working.

Dear customers,

We are in the final phase of shifting and finalizing the calculation of time credits. We thank you for your patience and partnership in working with us and transitioning your contracts over to the new credit system.

As a part of finalizing, we are implementing some changes that weren’t applied previously and will begin to calculate starting now.

• Transformations with multiple transformation backends - If you have a transformation that runs an SQL transformation with a R/Python transformation following it, in the same bucket, the SQL transformation is being calculated until the R/Python transformations are finished. As a result, these SQL jobs will have its child jobs run time deducted from their own, resulting in lower time credits consumption of affected clients. The change will cover jobs since the beginning of 2020.
• SQL sandboxes time credits usage - Most of our clients don’t have SQL sandboxes consumption added to their overall consumption. In our analysis, this is a fraction of total consumption, so the impact is not very material for the majority of clients. For those clients that have a material impact, your CSM will be reaching out to you to let you know the impact, otherwise you can assume your usage is not going to impact you.
• COVID-19 Error jobs - We haven’t been calculating credits for error jobs since April 2020 due to Covid and a higher than normal rate of Snowflake related issues. We will begin adding the consumption from job errors starting October.

Telemetry Component

We’re planning to release a new Keboola Connection component, which will allow the users to get telemetry data about their project or organization. That will eventually replace our temporary solutions like GoodData telemetry dashboard or direct writing of telemetry data to some projects. In the end, all of the Keboola Connection users will have easy access to documented telemetry data on demand.

We experienced Connection API higher error rate and issue with starting new Python sandboxes between 17:45 - 17:55 UTC. The issue is now resolved, jobs processing wasn't affected. 

We're investigating an issue with Snowflake in US region which causes some Storage operations with tables to be stuck in processing state. This can cause jobs to be executing longer than expected or seemingly "forever". Terminating and restarting the job does not help in such a case. Only certain projects are affected.

Next update in 1 hour or as new information becomes available.

Update 10:50 UTC: The stuck jobs are unblocked now and should be finishing, we're monitoring the situation if the issue reappears. A post mortem will be published once we get an RCA from Snowflake.

When configuring a Snowflake or Redshift database writer, you can use a Keboola-provided database.

In the past, when you selected this option, the credentials were stored in a configuration in a plain-text format. Storing the credentials this way allowed you to copy the password and use it in your favorite database client (or another system) even if you didn't copy it right after the creation.

To improve the overall security, we decided to show you your password only once and store it encrypted. From now on, when you create a new Keboola-provided database (Snowflake or Redshift), you will see the password only once, right after its creation.

Backward compatibility

The existing credentials will remain untouched. But if you delete them, there's no option to create them the old way.

New Components

• LiveRamp Identity Resolution application - solving some of the main challenges with customer and prospect data by returning people-based identifiers and metadata for your consumer records
  

• KBC Project Metadata extractor - Keboola metadata extractor downloads metadata about all objects in your Keboola project.
  

• Avro2CSV processor - Avro is a row-oriented remote procedure call and data serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.
  

Updated Components

• MySQL extractor - the "Transaction Isolation Level" setting is now configurable (documentation)

• Oracle extractor / Oracle writer - the "Tnsnames" can be now used to provide login credentials to Oracle database (documentation)

• Generic extractor - added option "caCertificate" which allows you configure custom certificate authority bundle in crt/pem format. (documentation)

Minor Improvements

• Google BigQuery - updated google-cloud-bigquery package

• Python updated to 3.8.5
  

• Julia updated to 1.5.0

We are investigating slight performance degradation of Snowflake in US region, there are no job failures or increased queue backlog but everything seems to run slightly slower. Degradation started around 00:00 AM UTC. Next update in 120 minutes or as new information becomes available.

UPDATE 2020-09-02 14:59 UTC: We still see slight performance degradation of some queries. We are in touch with Snowflake support. Next update tomorrow or as new information becomes available. 

UPDATE 2020-09-03 06:31 UTC: The issue is now resolved, performance went back to normal around 2020-09-03 00:00 UTC. We are waiting for more details about the issue from Snowflake.

On September 1st between 8:25 PM UTC and 9:48 PM UTC there was an incident on Snowflake service which led to Storage job failures.

The issue is now resolved and all systems are operational. We apologize for the inconvenience caused by this incident.

Since 2020-08-25 8:35 UTC we are experiencing Storage errors in the US region due to a reported incident in Snowflake. We are going to monitor the situation and keep you posted within 90 minutes.

UPDATE 2020-08-25 9:20 UTC: Snowflake incident update:

We have identified the problem with the Snowflake Service that is interrupting the following services:
1. Access to Snowflake UI
2. Cannot execute queries

Incident Start Time: 01:20 PT Aug 25, 2020
We will provide an update within 30 minutes or as soon as we have more details on the status of the issue.

Updated Components

• AWS S3 Extractor supports Authentication with an AWS role (documentation)

•  Twitter Extractor supports Direct Messages

To extract direct messages, you must reauthorize the account since an additional permission (DMs) is needed.

•  MongoDB Extractor supports custom URI connection

•  MSSQL Extractor supports encrypted (SSL) connection

UI Improvements

•  Storage job detail now has a permalink

To get a permalink for the job detail, click on the job ID. A popup with the job detail will appear, and the URL in your browser will change.

Minor Improvements

• Storage API List files endpoint is now returning only valid (not expired) files. Expired files can be retrieved with showExpired option, more in our documentation https://keboola.docs.apiary.io/#reference/files/list-files/list-files

There is some problem on GoodData API since about 01:00 CEST (23:00 UTC) which causes job timeouts. We are investigating the problem.

Next update in 60 minutes or as new information becomes available.

13 Aug 2020 10:30 UTC We are investigating the problem with GoodData support. So far it looks like problem is in WebDav integration on GoodData side. Next update in 60 minutes or as new information becomes available.

13 Aug 2020 11:30 UTC GoodData support is still investigating issue. Next update in 60 minutes or as new information becomes available.

13 Aug 2020 12:05 UTC According to our investigation it seems that all GoodData jobs are working now normally. We will continue to monitor situation. If you have GoodData job which is running unusually long, try to restart it, if it doesn't help please contact out support, we will hand over your pid to GoodData for further investigation. We are still waiting for GoodData support for detailed information's about incident. Next update in 2 hours or as new information becomes available.

13 Aug 2020 14:07 UTC We are still waiting for resolution from GoodData. Next update will be when new information becomes available. If you want detailed information for you project contact GoodData support with you pid and import time of last csv before failure.

13 Aug 2020 16:40 UTC Job failures were caused by slower GoodData DHWs, GoodData take action and fix problem, but several  jobs timeout during this period. All services are stable now. 

04 Aug 2020 04:28 UTC We're seeing a higher load and longer execution time in EU and US Snowflake warehouse queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

04 Aug 2020 04:53 UTC We have added some processing power to the Snowflake warehouse in both regions but the backlog is still present. Snowflake identified the problem with the interruption which may cause the processing slowdown. Next update in 60 minutes or as new information becomes available.

04 Aug 2020 05:20 UTC Backlogs are cleared in both regions and the situation seems normal, but we're monitoring it closely for next couple hours. Next update in two hours or as new information becomes available.

04 Aug 2020 06:25 UTC Snowflake marked the incident as resolved, everything should be back to normal. We'll keep monitoring our platform closely.

04 Aug 2020 07:27 UTC Query times are back to normal and all the backlogs are cleared for some time already. The incident is completely resolved. 

Updated Components

• Elasticsearch Writer supports collumns mapping

  

• PostgreSQL Extractor supports Encrypted SSL Connection

  

  

• Zendesk Extractor no longer has custom tickets fields values set as a primary key in the mapping. 
  

Snowflake announced a planned change of behavior to the the SQL RIGHT function which returns returns the rightmost substring of its input. The change is planned for July 27-31, 2020.

With the current behavior, the RIGHT function returns an empty string if the input string is shorter than the requested length.

With the new behavior the RIGHT function will return the whole input string if it is shorter than the requested length. 

See the example below:

SELECT RIGHT('hello', 2) AS "RIGHT", LEFT('hello', 2) AS "LEFT"
UNION SELECT RIGHT('hello', 5), LEFT('hello', 5)
UNION SELECT RIGHT('hello', 50), LEFT('hello', 50);

Current behavior

New behavior

While the current behavior is obviously incorrect (and inconsistent with the LEFT function) we can't rule out the option that results of your transformations might be affected or that some code is inadvertently relying on the current buggy behavior. Please feel free to contact our support (use the support button in your project) where we can provide you with further assistance (e.g. a list of possibly affected transformations or queries).

10 Jul 2020 08:55 UTC We're seeing a higher load and longer execution time in EU Snowflake queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

10 Jul 2020 09:55 UTC Unfortunately we have no update at the moment. We have added some processing power to the Snowflake warehouse and we are monitoring the situation closely to see if that helps. Job processing should be fine, you may only see slight delays. Next update in 60 minutes or as new information becomes available.

10 Jul 2020 11:10 UTC No update at the moment. Next update in 60 minutes or as new information becomes available.

10 Jul 2020 12:30 UTC No update at the moment. Next update in 90 minutes or as new information becomes available.

10 Jul 2020 14:00 UTC We're in touch with Snowflake support and trying to identify the root cause. Next update in 3 hours or as new information becomes available.

10 Jul 2020 18:00 UTC We're changing certain scaling parameters of the Snowflake warehouse to see if it can help resolve the issue. Next update in 24 hours or as new information becomes available.

11 Jul 2020 18:50 UTC The configuration change helped and we're fully operational. This is the last update of the incident. Thanks for your patience!

9 Jul 2020 08:24 UTC We're seeing a higher load and longer execution time in EU Snowflake queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 8:54 UTC We have added additional power to the warehouse to help process the queued queries. Currently the situation seems normal, but we're monitoring it closely for next couple hours. Next update in 90 minutes or as new information becomes available.

UPDATE 9 Jul 2020 9:18 UTC After all the additional workload has been processed we have scaled down the cluster, but we're seeing queuing jobs again. We have again scaled up the cluster to help with the load. We're monitoring the situation closely. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 10:15 UTC We're in touch with Snowflake support to resolve this issue. Meanwhile we have decreased the worker capacity so the Storage jobs may be queued on our end. This should take some load off the Snowflake warehouse to maximize it's performance. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 11:45 UTC Snowflake engineering team is resolving the underlying issue. Thanks to the throttling on our end there are currently no delays in jobs processing. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 12:30 UTC Snowflake informed us that the issue was fixed. We're restoring platform parameters to the original values and will continue monitoring the situation. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 13:20 UTC All operations are back to normal and everything is fully working.

We're sorry for the inconvenience and appreciate your patience.

New Components

• Lytics writer - writes data to custom Lytics streams.
• Lytics extractor - downloads Segments and Users from the Lytics platform.
• ARES Opendata extractor - extracts a full snapshot of the CZ ARES registry opendata dataset.
• CSV to JSON processor - converts CSV files to JSON files.

Updated Components

• FTP extractor - when an FTP extractor is configured via the UI, the Decompress option automatically adds the keboola.processor-flatten-folders processor to the configuration.

UI Improvements

From now on, you'll be able to see invited users on the dashboard.

We are introducing shorter expiration times for files in Storage.

From July 13, newly created files will have the following expiration settings:

- 15 days for table import files and manual uploads
- 48 hours for table export files

Expiration of existing files will not be affected by this change.

New Features and Updates

Project Description

Looker Writer Connection Name

Deprecation of Storage API .NET Client

Renaming Storage Buckets and Tables

Selecting Bucket in Input Mapping

Bug Fixes

• Generic Extractor no longer stops after the 2nd page when downloading data in child jobs (only configurations with Limit Stop setting were affected)
• CSV import component supports full load again (due to a bug, all imports were performed incrementally).
• MySQL writer no longer writes an "empty string" instead of a null for columns with DATE and DATETIME data types.

New Components

• CSOB CEB extractor for downloading bank statements from the CSOB CEB Business Connector service
• Azure Blob Storage writer for exporting any input CSV files into designated Blob containers
  
• Sisense writer for sending tables from Keboola Connection to a Sisense database platform
• Zendesk writer for creating and updating Zendesk Support properties with the flexibility of defining their own parameters

We’re currently experiencing OAuth authorization errors returning error:

Docker encryption error: Contact support@keboola.com and attach this exception id

Only authorization of new configuration is affected, running jobs aren't affected. First occurrence of the error was at 14:02 UTC.

We are performing rollback. Next update in 60 minutes or as new information becomes available.

June 17, 10:26 UTC - After releasing a new version of OAuth broker we have identified that UI of components relying on OAuth authorization is broken, displaying only message: "Something went wrong".

UPDATE 11:01 UTC - We have reverted the release to previous version, which restored the functionality. We are still investigating this issue. It's possible that the problem also affected jobs of these components.

UPDATE 11:31 UTC - We have confirmed that the jobs of the components were unaffected by this bug

Some API calls to run jobs are ending with an application error in the US region. We're investigating the causes.

Update 0:40 UTC: The problem is resolved now. One of the API worker freeze. Running jobs were not affected. We will post a detailed analysis within a week.

We are investigating Snowflake query failures. Affected queries ends up with similar error message:

Processing aborted due to error 300005:3495968563; incident 9229003

Only minority of projects and queries are affected. We noticed first occurrences on July 8th. We are in touch with Snowflake Support, issue is related to new Snowflake releases and they are investigating it. Next update in 120 minutes or as new information becomes available.

UPDATE 8:20 UTC - Snowflake engineering is working on this issue.

UPDATE 9:35 UTC - Snowflake engineering has already identified the issue, they are testing the changes and also working on rolling out the change. Next update in 120 minutes or as new information becomes available.

UPDATE 10:50 UTC - Fixed version will be released within 24 hours by Snowflake. Next update in 6 hours or as new information becomes available.

UPDATE 17:24 UTC - Snowflake engineering is still working on releasing the patch. The estimate release in 24 hours still holds. Next update in 4 hours or as new information becomes available.

UPDATE 22:10 UTC - According to Snowflake engineering, the the issue is fixed. We're monitoring the situation.

UPDATE June 11, 05:28 UTC - Previously affected queries were executed successfully. Unfortunately few other queries ended up with incident between  Jun 11 02:20:59 - Jun 11 02:25:50 in EU region and one query at Jun 11 01:10:00 UTC in US region. Snowflake engineering is investigating the issue. Next update in 4 hours or as new information becomes available.

UPDATE June 11, 8:39 UTC - Snowflake engineering confirmed that queries which failed tonight were still running on affected release while clusters were still migrating to newer release. At the moment we don't register any failures of queries running on new release. We're monitoring the situation. Next update in 12 hours or as new information becomes available.

UPDATE June 11, 20:52 UTC - There were no query failures since last update. We'll continue to monitor the situation.

UPDATE June 12, 6:30 UTC - There were no query failures since last update. The issue is now resolved. We apologize for the inconvenience. If you have any questions or see any related issues, please contact Keboola Support.

We are investigating higher error rate of Oracle extractor jobs. Affected jobs ends with error:

DB query failed: Export process failed: Connection error: IO Error: The Network Adapter could not establish the connection Tried 5 times.

Next update in 60 minutes or as new information becomes available.

UPDATE 10:14 UTC: We have preventively rollbacked to previous version of extractor and we are monitoring presence of the failures. Next update in 60 minutes or as new information becomes available.

We are investigating MySQL extractor errors for some configurations ending with error message: 

The "incrementalFetchingColumn" must be configured, if incremental fetching is enabled.

The issue is probably caused by new extractor release on Jun 08 05:55 UTC, we are doing rollback.

There is some problem on GoodData API since about 08:00 UTC which causes failures on some model updates and data loads.

We are investigating the problem with GoodData support and will keep you updated.

Update 14:20 UTC - The GoodData Technical Support team is still investigating this issue.

Our GoodData writer component has not changed since January, so we are waiting for a clarification of the root cause from their support team.

Update 15:10 UTC - This issue is related to GoodData's release today. They are preparing a hotfix for it and expect it to be deployed in a few hours.

Update June 5, 08:20 UTC - The problem has been resolved. GoodData deployed a hotfix for their API last night. Since June 4 21:00 UTC we have not seen any new errors from the GoodData Writer.

6:28pm UTC: We are experiencing Python/R Sandbox failing to create in EU and US region and keep on investigating the problem. We'll keep you updated.

6:38pm UTC: We have identified the root cause being expired CA intermediate certificate. We proceed to replace the expired certificate for the sandbox instances. Next update within an hour.

7:38pm UTC: We have successfully replaced CA intermediate certificate for the US region and Python/R sandboxes successfully create now. EU region CA intermediate certificate replacement  is on the way. Next update within an hour.

7:43pm UTC: The Python/R Sandboxes failing to create is resolved now. We have replaced the expired CA intermediate certificate and new Python/R sandboxes create successfully now in both US and EU region. We are sorry for the inconvenience.

An option to rename buckets and tables was one of the most requested features on our wishlist. It is very useful when you want to name your bucket by its contents (e.g., "email-orders") rather than "in.c-keboola-ex-gmail-587163382".

From now on, you'll be able to change the names of buckets and tables.

Rename Bucket

To rename a bucket, navigate to the bucket detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (there are some limitations though).

Rename Table

To rename a table, navigate to the table detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (the same limitations apply).

Consequent Changes

Despite the fact that adding the option to rename a bucket or a table does not look like a very big deal, we had to make some substantial changes under the hood. Some of the consequences are worth mentioning here:

Hidden "c-" prefix

We no longer show the "c-" prefix in the names of buckets and tables. It is still a part of the bucket and table ID, but the ID is no longer displayed in most cases. If you need to access the ID for some reason, it is still available on the detail page of each bucket and table.

This is an example of how buckets and tables are displayed without the "c-" prefix:

Stage Selector

When searching for a specific bucket or table, just select a stage and the buckets will be filtered by the selected stage.