Keboola Status

9 Jul 2020 08:24 UTC We're seeing a higher load and longer execution time in EU Snowflake queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 8:54 UTC We have added additional power to the warehouse to help process the queued queries. Currently the situation seems normal, but we're monitoring it closely for next couple hours. Next update in 90 minutes or as new information becomes available.

UPDATE 9 Jul 2020 9:18 UTC After all the additional workload has been processed we have scaled down the cluster, but we're seeing queuing jobs again. We have again scaled up the cluster to help with the load. We're monitoring the situation closely. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 10:15 UTC We're in touch with Snowflake support to resolve this issue. Meanwhile we have decreased the worker capacity so the Storage jobs may be queued on our end. This should take some load off the Snowflake warehouse to maximize it's performance. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 11:45 UTC Snowflake engineering team is resolving the underlying issue. Thanks to the throttling on our end there are currently no delays in jobs processing. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 12:30 UTC Snowflake informed us that the issue was fixed. We're restoring platform parameters to the original values and will continue monitoring the situation. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 13:20 UTC All operations are back to normal and everything is fully working.

We're sorry for the inconvenience and appreciate your patience.

New Components

• Lytics writer - writes data to custom Lytics streams.
• Lytics extractor - downloads Segments and Users from the Lytics platform.
• ARES Opendata extractor - extracts a full snapshot of the CZ ARES registry opendata dataset.
• CSV to JSON processor - converts CSV files to JSON files.

Updated Components

• FTP extractor - when an FTP extractor is configured via the UI, the Decompress option automatically adds the keboola.processor-flatten-folders processor to the configuration.

UI Improvements

From now on, you'll be able to see invited users on the dashboard.

We are introducing shorter expiration times for files in Storage.

From July 13, newly created files will have the following expiration settings:

- 15 days for table import files and manual uploads
- 48 hours for table export files

Expiration of existing files will not be affected by this change.

New Features and Updates

Project Description

Looker Writer Connection Name

Deprecation of Storage API .NET Client

Renaming Storage Buckets and Tables

Selecting Bucket in Input Mapping

Bug Fixes

• Generic Extractor no longer stops after the 2nd page when downloading data in child jobs (only configurations with Limit Stop setting were affected)
• CSV import component supports full load again (due to a bug, all imports were performed incrementally).
• MySQL writer no longer writes an "empty string" instead of a null for columns with DATE and DATETIME data types.

New Components

• CSOB CEB extractor for downloading bank statements from the CSOB CEB Business Connector service
• Azure Blob Storage writer for exporting any input CSV files into designated Blob containers
  
• Sisense writer for sending tables from Keboola Connection to a Sisense database platform
• Zendesk writer for creating and updating Zendesk Support properties with the flexibility of defining their own parameters

We’re currently experiencing OAuth authorization errors returning error:

Docker encryption error: Contact support@keboola.com and attach this exception id

Only authorization of new configuration is affected, running jobs aren't affected. First occurrence of the error was at 14:02 UTC.

We are performing rollback. Next update in 60 minutes or as new information becomes available.

June 17, 10:26 UTC - After releasing a new version of OAuth broker we have identified that UI of components relying on OAuth authorization is broken, displaying only message: "Something went wrong".

UPDATE 11:01 UTC - We have reverted the release to previous version, which restored the functionality. We are still investigating this issue. It's possible that the problem also affected jobs of these components.

UPDATE 11:31 UTC - We have confirmed that the jobs of the components were unaffected by this bug

Some API calls to run jobs are ending with an application error in the US region. We're investigating the causes.

Update 0:40 UTC: The problem is resolved now. One of the API worker freeze. Running jobs were not affected. We will post a detailed analysis within a week.

We are investigating Snowflake query failures. Affected queries ends up with similar error message:

Processing aborted due to error 300005:3495968563; incident 9229003

Only minority of projects and queries are affected. We noticed first occurrences on July 8th. We are in touch with Snowflake Support, issue is related to new Snowflake releases and they are investigating it. Next update in 120 minutes or as new information becomes available.

UPDATE 8:20 UTC - Snowflake engineering is working on this issue.

UPDATE 9:35 UTC - Snowflake engineering has already identified the issue, they are testing the changes and also working on rolling out the change. Next update in 120 minutes or as new information becomes available.

UPDATE 10:50 UTC - Fixed version will be released within 24 hours by Snowflake. Next update in 6 hours or as new information becomes available.

UPDATE 17:24 UTC - Snowflake engineering is still working on releasing the patch. The estimate release in 24 hours still holds. Next update in 4 hours or as new information becomes available.

UPDATE 22:10 UTC - According to Snowflake engineering, the the issue is fixed. We're monitoring the situation.

UPDATE June 11, 05:28 UTC - Previously affected queries were executed successfully. Unfortunately few other queries ended up with incident between  Jun 11 02:20:59 - Jun 11 02:25:50 in EU region and one query at Jun 11 01:10:00 UTC in US region. Snowflake engineering is investigating the issue. Next update in 4 hours or as new information becomes available.

UPDATE June 11, 8:39 UTC - Snowflake engineering confirmed that queries which failed tonight were still running on affected release while clusters were still migrating to newer release. At the moment we don't register any failures of queries running on new release. We're monitoring the situation. Next update in 12 hours or as new information becomes available.

UPDATE June 11, 20:52 UTC - There were no query failures since last update. We'll continue to monitor the situation.

UPDATE June 12, 6:30 UTC - There were no query failures since last update. The issue is now resolved. We apologize for the inconvenience. If you have any questions or see any related issues, please contact Keboola Support.

We are investigating higher error rate of Oracle extractor jobs. Affected jobs ends with error:

DB query failed: Export process failed: Connection error: IO Error: The Network Adapter could not establish the connection Tried 5 times.

Next update in 60 minutes or as new information becomes available.

UPDATE 10:14 UTC: We have preventively rollbacked to previous version of extractor and we are monitoring presence of the failures. Next update in 60 minutes or as new information becomes available.

We are investigating MySQL extractor errors for some configurations ending with error message: 

The "incrementalFetchingColumn" must be configured, if incremental fetching is enabled.

The issue is probably caused by new extractor release on Jun 08 05:55 UTC, we are doing rollback.