Keboola Status

We're encountering difficulties with our MySQL transformation server, some jobs are ending up with an application exception. We're closely monitoring this issue and restarting all failed orchestrations until the situation gets stable.

We're sorry for any inconvenience.

We're affected by DockerHub and Quay.io Service Disruption. Some components are failing. 

We'll update this post once the disruption is over and will restart all affected orchestrations.

You can also check the official DockerHub status page and Quay.IO status page for more information.

We're sorry for this inconvenience.

UPDATE: The Quay.IO disruption is over.

Between Oct 26 10:40am UTC and Oct 26 12:50am UTC we encountered issues connecting to Snowflake. 

Some jobs might have ended with Internal error or one of these exceptions:

• The error message SFExecuteQueryFailed could not be found in the en-US locale. Check that /en-US/SFMessages.xml exists., SQL state S1000 in SQLPrepare
• Failed to execute query. Error code=-1, GS error code=601, GS error message=Service temporarily unavailable. Retry the request later.
  

We're investigating the root cause with Snowflake support and identifying and restarting affected orchestrations. 

We're encountering a higher number of "Output mapping does not match destination table" errors in applications today since 10:30am. This was caused by deploying strict validation in output mapping. Our detection script did not detect and prevent all cases, so we're rolling back this release and restarting all affected orchestrations.

We're sorry for this inconvenience.

On October 21, 2016 at 11:43:58 PM CEST/UTC+2 one of our database servers was restarted due to a hardware failure. Jobs running during that time failed when trying to reconnect to the database later in the processing.

Update October 21, 2016 at 01:20 AM CEST/UTC+2: Jobs/orchestrations affected with this restart didn't stop processing and finished successfully, they may show incorrect job result as error instead of success.

We're sorry for this inconvenience and we're restarting all failed orchestrations.

Our transformation MySQL server was under heavy load between October 20th 12:00am and 3:00am UTC. Transformation processes were slowed down or halted. 

We have identified the blocking processes and all operations returned to normal. In a few cases we have terminated stalled transformations and restarted the orchestrations. 

We're sorry for this inconvenience.

We're opening OpenRefine transformations to public. You can now use OpenRefine in your transformation pipeline. 

No further need to write long string replaces in SQL or study how to open CSV files in Python or R. OpenRefine excels in data cleanup and many other data wrangling tasks. 

To create an OpenRefine transformation choose OpenRefine (beta) when creating a new transformation.

Learn more about OpenRefine and its functions and about OpenRefine integration in Keboola Connection.

During last days we have turned on strict input/output mapping validation. Each input/output mapping is checked against the table in Storage if

• all columns exist
• the primary key is equal in both cases 
• datatype/indexes/distkey/sortkey or filter column names have the same letter case

Although we tried to detect all breaches of this ruleset beforehand and contact project owners some have unfortunately slipped through. We're closely monitoring all errors and fixing/restarting all failed orchestrations. 

In case your project is subject to this issue on a larger scale than a single failure, we're able to remove the validation temporarily. Please contact us at support@keboola.com with any further questions/requests.

We're deeply sorry for any inconvenience. 

We're currently investigating issues with Snowflake workspaces (sandboxes, transformations). We'll keep this post updated.

Update 7:36pm CEST: We have passed information to Snowflake support team and they're investigating the issue.

Update 22:36pm CEST: Snowflake team has identified the issue and is working on fix which should be deployed later tonight.

Update 05:55am CEST: The issue has been resolved and all operations are back to normal. Due to high number of affected transformations/orchestrations we won't be restarting them to prevent system overload. Please restart your orchestrations manually if needed.

Thanks for your patience and understanding.

Between Oct 07 02:00 (GMT+2) and Oct 07 02:24 (GMT+2) some transformation jobs were failing. We have identified the root cause of this issue and fixed it. All affected orchestrations will be restarted shortly.

We're sorry for this inconvenience.