Snowflake Incident in US region

Since 2020-08-25 8:35 UTC we are experiencing Storage errors in the US region due to a reported incident in Snowflake. We are going to monitor the situation and keep you posted within 90 minutes.

UPDATE 2020-08-25 9:20 UTC: Snowflake incident update:

We have identified the problem with the Snowflake Service that is interrupting the following services:
1. Access to Snowflake UI
2. Cannot execute queries

Incident Start Time: 01:20 PT Aug 25, 2020
We will provide an update within 30 minutes or as soon as we have more details on the status of the issue.
UPDATE 2020-08-25 9:50 UTC: The problem seems to disappear, we don't get any more errors since 9:30 UTC. But Snowflake hasn't updated the incident yet so we are still monitoring the situation.

UPDATE 2020-08-25 10:15 UTC: A fix to resolve the issue is applied and the situation is still being monitored.

Week in Review - August 14th, 2020

Updated Components

  • AWS S3 Extractor supports Authentication with an AWS role (documentation)

  •  Twitter Extractor supports Direct Messages

To extract direct messages, you must reauthorize the account since an additional permission (DMs) is needed.

  •  MongoDB Extractor supports custom URI connection

  •  MSSQL Extractor supports encrypted (SSL) connection


UI Improvements

  •  Storage job detail now has a permalink

To get a permalink for the job detail, click on the job ID. A popup with the job detail will appear, and the URL in your browser will change.


Minor Improvements

GoodData Writer Failures

There is some problem on GoodData API since about 01:00 CEST (23:00 UTC) which causes job timeouts. We are investigating the problem.

Next update in 60 minutes or as new information becomes available.


13 Aug 2020 10:30 UTC We are investigating the problem with GoodData support. So far it looks like problem is in WebDav integration on GoodData side. Next update in 60 minutes or as new information becomes available.

13 Aug 2020 11:30 UTC GoodData support is still investigating issue. Next update in 60 minutes or as new information becomes available.

13 Aug 2020 12:05 UTC According to our investigation it seems that all GoodData jobs are working now normally. We will continue to monitor situation. If you have GoodData job which is running unusually long, try to restart it, if it doesn't help please contact out support, we will hand over your pid to GoodData for further investigation. We are still waiting for GoodData support for detailed information's about incident. Next update in 2 hours or as new information becomes available.

13 Aug 2020 14:07 UTC We are still waiting for resolution from GoodData. Next update will be when new information becomes available. If you want detailed information for you project contact GoodData support with you pid and import time of last csv before failure.

13 Aug 2020 16:40 UTC Job failures were caused by slower GoodData DHWs, GoodData take action and fix problem, but several  jobs timeout during this period. All services are stable now. 

Snowflake Slowdown in the EU and US Region

04 Aug 2020 04:28 UTC We're seeing a higher load and longer execution time in EU and US Snowflake warehouse queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

04 Aug 2020 04:53 UTC We have added some processing power to the Snowflake warehouse in both regions but the backlog is still present. Snowflake identified the problem with the interruption which may cause the processing slowdown. Next update in 60 minutes or as new information becomes available.

04 Aug 2020 05:20 UTC Backlogs are cleared in both regions and the situation seems normal, but we're monitoring it closely for next couple hours. Next update in two hours or as new information becomes available.

04 Aug 2020 06:25 UTC Snowflake marked the incident as resolved, everything should be back to normal. We'll keep monitoring our platform closely.

04 Aug 2020 07:27 UTC Query times are back to normal and all the backlogs are cleared for some time already. The incident is completely resolved. 


Upcoming Snowflake Behaviour Change to the RIGHT Function

Snowflake announced a planned change of behavior to the the SQL RIGHT function which returns returns the rightmost substring of its input. The change is planned for July 27-31, 2020.

With the current behavior, the RIGHT function returns an empty string if the input string is shorter than the requested length.

With the new behavior the RIGHT function will return the whole input string if it is shorter than the requested length. 

See the example below:

SELECT RIGHT('hello', 2) AS "RIGHT", LEFT('hello', 2) AS "LEFT"
UNION SELECT RIGHT('hello', 5), LEFT('hello', 5)
UNION SELECT RIGHT('hello', 50), LEFT('hello', 50);

Current behavior

RIGHT LEFT
lo he
hello hello
  hello

New behavior

RIGHT LEFT
lo he
hello hello
hello  hello


While the current behavior is obviously incorrect (and inconsistent with the LEFT function) we can't rule out the option that results of your transformations might be affected or that some code is inadvertently relying on the current buggy behavior. Please feel free to contact our support (use the support button in your project) where we can provide you with further assistance (e.g. a list of possibly affected transformations or queries).


Snowflake Slowdown in the EU Region

10 Jul 2020 08:55 UTC We're seeing a higher load and longer execution time in EU Snowflake queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

10 Jul 2020 09:55 UTC Unfortunately we have no update at the moment. We have added some processing power to the Snowflake warehouse and we are monitoring the situation closely to see if that helps. Job processing should be fine, you may only see slight delays. Next update in 60 minutes or as new information becomes available.

10 Jul 2020 11:10 UTC No update at the moment. Next update in 60 minutes or as new information becomes available.

10 Jul 2020 12:30 UTC No update at the moment. Next update in 90 minutes or as new information becomes available.

10 Jul 2020 14:00 UTC We're in touch with Snowflake support and trying to identify the root cause. Next update in 3 hours or as new information becomes available.

10 Jul 2020 18:00 UTC We're changing certain scaling parameters of the Snowflake warehouse to see if it can help resolve the issue. Next update in 24 hours or as new information becomes available.

11 Jul 2020 18:50 UTC The configuration change helped and we're fully operational. This is the last update of the incident. Thanks for your patience!


Snowflake Slowdown in the EU Region

9 Jul 2020 08:24 UTC We're seeing a higher load and longer execution time in EU Snowflake queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 8:54 UTC We have added additional power to the warehouse to help process the queued queries. Currently the situation seems normal, but we're monitoring it closely for next couple hours. Next update in 90 minutes or as new information becomes available.

UPDATE 9 Jul 2020 9:18 UTC After all the additional workload has been processed we have scaled down the cluster, but we're seeing queuing jobs again. We have again scaled up the cluster to help with the load. We're monitoring the situation closely. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 10:15 UTC We're in touch with Snowflake support to resolve this issue. Meanwhile we have decreased the worker capacity so the Storage jobs may be queued on our end. This should take some load off the Snowflake warehouse to maximize it's performance. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 11:45 UTC Snowflake engineering team is resolving the underlying issue. Thanks to the throttling on our end there are currently no delays in jobs processing. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 12:30 UTC Snowflake informed us that the issue was fixed. We're restoring platform parameters to the original values and will continue monitoring the situation. Next update in 60 minutes or as new information becomes available.

UPDATE 9 Jul 2020 13:20 UTC All operations are back to normal and everything is fully working.

We're sorry for the inconvenience and appreciate your patience.

Week in Review - July 7th, 2020

New Components

Updated Components

  • FTP extractor - when an FTP extractor is configured via the UI, the Decompress option automatically adds the keboola.processor-flatten-folders processor to the configuration.

UI Improvements

From now on, you'll be able to see invited users on the dashboard.