Keboola Status

We are investigating Snowflake query failures. Affected queries ends up with similar error message:

Processing aborted due to error 300005:3495968563; incident 9229003

Only minority of projects and queries are affected. We noticed first occurrences on July 8th. We are in touch with Snowflake Support, issue is related to new Snowflake releases and they are investigating it. Next update in 120 minutes or as new information becomes available.

UPDATE 8:20 UTC - Snowflake engineering is working on this issue.

UPDATE 9:35 UTC - Snowflake engineering has already identified the issue, they are testing the changes and also working on rolling out the change. Next update in 120 minutes or as new information becomes available.

UPDATE 10:50 UTC - Fixed version will be released within 24 hours by Snowflake. Next update in 6 hours or as new information becomes available.

UPDATE 17:24 UTC - Snowflake engineering is still working on releasing the patch. The estimate release in 24 hours still holds. Next update in 4 hours or as new information becomes available.

UPDATE 22:10 UTC - According to Snowflake engineering, the the issue is fixed. We're monitoring the situation.

UPDATE June 11, 05:28 UTC - Previously affected queries were executed successfully. Unfortunately few other queries ended up with incident between  Jun 11 02:20:59 - Jun 11 02:25:50 in EU region and one query at Jun 11 01:10:00 UTC in US region. Snowflake engineering is investigating the issue. Next update in 4 hours or as new information becomes available.

UPDATE June 11, 8:39 UTC - Snowflake engineering confirmed that queries which failed tonight were still running on affected release while clusters were still migrating to newer release. At the moment we don't register any failures of queries running on new release. We're monitoring the situation. Next update in 12 hours or as new information becomes available.

UPDATE June 11, 20:52 UTC - There were no query failures since last update. We'll continue to monitor the situation.

UPDATE June 12, 6:30 UTC - There were no query failures since last update. The issue is now resolved. We apologize for the inconvenience. If you have any questions or see any related issues, please contact Keboola Support.

We are investigating higher error rate of Oracle extractor jobs. Affected jobs ends with error:

DB query failed: Export process failed: Connection error: IO Error: The Network Adapter could not establish the connection Tried 5 times.

Next update in 60 minutes or as new information becomes available.

UPDATE 10:14 UTC: We have preventively rollbacked to previous version of extractor and we are monitoring presence of the failures. Next update in 60 minutes or as new information becomes available.

We are investigating MySQL extractor errors for some configurations ending with error message: 

The "incrementalFetchingColumn" must be configured, if incremental fetching is enabled.

The issue is probably caused by new extractor release on Jun 08 05:55 UTC, we are doing rollback.

There is some problem on GoodData API since about 08:00 UTC which causes failures on some model updates and data loads.

We are investigating the problem with GoodData support and will keep you updated.

Update 14:20 UTC - The GoodData Technical Support team is still investigating this issue.

Our GoodData writer component has not changed since January, so we are waiting for a clarification of the root cause from their support team.

Update 15:10 UTC - This issue is related to GoodData's release today. They are preparing a hotfix for it and expect it to be deployed in a few hours.

Update June 5, 08:20 UTC - The problem has been resolved. GoodData deployed a hotfix for their API last night. Since June 4 21:00 UTC we have not seen any new errors from the GoodData Writer.

6:28pm UTC: We are experiencing Python/R Sandbox failing to create in EU and US region and keep on investigating the problem. We'll keep you updated.

6:38pm UTC: We have identified the root cause being expired CA intermediate certificate. We proceed to replace the expired certificate for the sandbox instances. Next update within an hour.

7:38pm UTC: We have successfully replaced CA intermediate certificate for the US region and Python/R sandboxes successfully create now. EU region CA intermediate certificate replacement  is on the way. Next update within an hour.

7:43pm UTC: The Python/R Sandboxes failing to create is resolved now. We have replaced the expired CA intermediate certificate and new Python/R sandboxes create successfully now in both US and EU region. We are sorry for the inconvenience.

An option to rename buckets and tables was one of the most requested features on our wishlist. It is very useful when you want to name your bucket by its contents (e.g., "email-orders") rather than "in.c-keboola-ex-gmail-587163382".

From now on, you'll be able to change the names of buckets and tables.

Rename Bucket

To rename a bucket, navigate to the bucket detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (there are some limitations though).

Rename Table

To rename a table, navigate to the table detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (the same limitations apply).

Consequent Changes

Despite the fact that adding the option to rename a bucket or a table does not look like a very big deal, we had to make some substantial changes under the hood. Some of the consequences are worth mentioning here:

Hidden "c-" prefix

We no longer show the "c-" prefix in the names of buckets and tables. It is still a part of the bucket and table ID, but the ID is no longer displayed in most cases. If you need to access the ID for some reason, it is still available on the detail page of each bucket and table.

This is an example of how buckets and tables are displayed without the "c-" prefix:

Stage Selector

When searching for a specific bucket or table, just select a stage and the buckets will be filtered by the selected stage.

Today since 7:00 UTC until 7:35 UTC we encountered AWS S3 extractor failing jobs with the error:

Invalid cipher text for key #KEBOOLA_USER_AWS_SECRET_KEY Value is not an encrypted value.

We found the root cause of the issue and immediately fixed it. There was no leaked key or secret, only wrong naming in the environment setup. We sincerely apologize for the error.

Transformations

• Python/R and Julia transformations have their default RAM limit increased to 16 GB. This applies to sandboxes as well.
  

Snowflake Platform Update

• Distinct keyword will be disallowed in an ordered window function or a window frame. The full post can be found here.

New Components

• OneDrive Excel Sheets extractor - extracts from a OneDrive account or from a SharePoint account.
• OneDrive Excel Sheets writer - writes to a OneDrive account or to a SharePoint account.
  
• Zoom Webinar Registrator - obtains a list of people to be registered for a specific Zoom Webinar and processes the registration.

Updated Components

• Database writer - many databases writers now support config row configurations. The full post can be found here.
• AWS S3 extractor now uses the ListObjectV2 method for listing objects in a bucket. This improves the performance for versioned buckets.
• MongoDB extractor - added support for incremental fetching

Security Improvements

• TLS security update: as of May 12, 2020, Transport Layer Security (TLS) 1.0 and 1.1 are no longer supported for securing connections to Keboola Connection endpoints. More information can be found here.

UI Improvements

• In "generic UI" components, the documentation and configuration parts were split into separate boxes
  

• MongoDB has a new detail layout to match other database extractors.

Developer Portal UI Improvements

• You are able to preview and validate the form created by the configuration schema.

We're happy to announce the arrival of Configuration Rows, our new powerful configuration format, to database writers.

From now on, you'll see a migration button in the configuration detail of each database writer (Snowflake, MySQL, SQL Server, Oracle, PostgreSQL, Impala, Hive, and Redshift).

Just click Migrate Configuration and the configuration will be migrated to the new format.

After the migration, you'll see more information about each table. All tables can be easily reordered, so you can move more important tables to the top and they will be uploaded first.

Also, you will be able to see information about each table on a new table detail page, with Last runs and Versions in a sidebar.

Underlying Important Changes

While there were certain limitations in the old configuration format, this is no longer true in the new "rows format".

The following features are worth mentioning:

• Disabled tables will no longer be exported from Storage (previously, they were exported with limit=1 and not used in the database writer).
• Each table has its own state with information about the date/time of the last import (previously, an upload of a single table cleared the state for other tables).

Summary

Original post https://status.keboola.com/mysql-extractor-errors

On May 11th 2020 10:12 UTC we have released a new version (5.5.1) of MySQL extractor in which a bug was present.
It caused errors in the UI:

Decoding JSON response from component failed: Syntax error

It also affected jobs of this extractor. Although the jobs seemed to finish successfully, they didn't process any data.
The flawed version was released at 10:12 UTC and reverted at 13:09 UTC.

DB query failed: Trying to access array offset on value of type null

The cause of the first problem was missing command in Dockerfile. This was fixed in release 5.5.2.

The second error was introduced with an update of PHP version. It was fixed in latest release (5.5.3).

What Are We Doing About This?

We have added tests to this component to cover these cases.