Job Failures Saturday, November 18, 2017

We have been experiencing temporary technical difficulties today around 3:00 AM CET and 4:00 PM CET.

Some component jobs may have failed as a result.  We have identified and fixed the issue. All systems have returned to normal operations and all jobs are now being processed normally.

Snowflake Incident

We are investigating Snowflake partial outage. Queries are now queued.

We will inform about the progress.

UPDATE 11:21 AM PDT Snowflake warehouses are up and running. Queued queries are being processed. We will update the post when we'll have more information about incident cause from Snowflake engineering.

Snowflake Incident Investigation

We were notified from Snowflake support about problem that could potentially affect Keboola Connection Snowflake Backed projects.

The issue happened between 9:19pm UTC August 8th and 6:45pm August 9th UTC.

There is a full alert from Snowflake below. We are investingating with them if we can find out if there are some affected KBC projects.

We will keep you informed of any progress.

UPDATE 2017-08-12 We have received list of projects which might be affected by this issue. In these projects one or more queries which might return wrong results were executed. We have notified these projects in KBC. If you need any assistance please contact support@keboola.com.


Dear Snowflake Customer,


We have identified a problem that could produce incorrect results in some queries that perform memory intensive join operations. It is unlikely that this issue impacts you; however, we wanted to make sure you were aware of the problem. The issue was isolated to the US-West region between 2:19pm Pacific time on Tuesday, August 8th and 11:45am Pacific time on Wednesday, August 9th. Queries performed in other regions or outside that timeframe were not impacted.


Because of the nature of the issue, it is difficult for us to determine if any of your queries experienced the problem. However, we are able to perform further analysis on a query-by-query basis. If you have a question about whether a specific query was impacted, please submit a support request to Snowflake support via the Support Portal or via email to support@snowflake.net .


We are very sorry about this issue and will do everything possible to help you resolve any problems this may have caused you.


Components errors

We observe components failures hosted on quay.io from 08:25  AM CET. Python Transformations, Facebook Extractor, Google Sheets Writer are between affected components.

Quay.io outage is not reported yet. To avoid this issue in future we are moving all components to Keboola provided AWS repository.

We will inform about the progress.

9:09 AM CET RESOLVED Last error was seen at 08:53 AM CET. Quay.io has confirmed and fixed the issue.

Job failures

There were jobs failures between 4:30 AM - 5:30 AM CET. Failures were caused by low disk space of one of worker servers.

We are sorry for this inconvenience and we're taking steps to mitigate this problem in the future.

Fixed IP Address Ranges - temporarily reverted

Some database extractor timeout errors have occurred in a few projects due to yesterday's switch to static IPs. 

We have temporarily reverted back from from new Fixed IPs as we attempt to resolve these issues.

We will post an update when the fixed IPs return and the timeout issues resolved.
 

Job failures

There were few jobs failures between 16:00 - 17:00 CET and 21:00 - 22:30 CET. We have identified the root causes of the problem and rollbacked to previous version.

Affected jobs were returning error:  Authentication token has expired. The user must authenticate again

We're sorry for this inconvenience.

Trouble Signing In

Signing into KBC with your Google account is not working presently.
We're investigating and hope to have it resolved shortly.
In the meantime, please sign in using the standard email and password method.

Sorry for this inconvenience,  this post will be updated when further information becomes available.

UPDATE 08:21 UTC We have identified the issue. The fix will be deployed in ASAP.

UPDATE 08:55 UTC Signing in with your Google account is again fully operational.