Keboola Status

We have investigated issues with Amazon DNS service in us-east region.

It affects jobs processing and API availability of our components. Thanks for patience.

UPDATE: 

AWS confirmed the problem - 5:15 AM PDT We are investigating elevated errors resolving the DNS hostname used to access the EC2 APIs.

AWS status update - 5:51 AM PDT We have identified the root cause of the issue causing errors resolving the DNS hostname used to access the EC2 APIs and are currently working to resolve.

Our transformation MySQL server was under heavy load between October 20th 12:00am and 3:00am UTC. Transformation processes were slowed down or halted. 

We have identified the blocking processes and all operations returned to normal. In a few cases we have terminated stalled transformations and restarted the orchestrations. 

We're sorry for this inconvenience.

Even though the weekly status has been a little weak recently, it does not mean that we're dawdling. We are working on some quite big internal things, which take much more than a week. For example (in no particular order):

• Developer Portal, which will enable 3rd party developers to manage their applications.
• Keboola Connection in EU Region.
• OpenRefine transformations.
• Validation and simplification of input and output mapping.
• Internal changes to transformations and sandboxes using so called Workspaces.
• RStudio and Jupyter Sandboxes.
• Collecting more statistics about running jobs:

  
• New Redshift, Oracle and MSSQL Writers and DynamoDB, Facebook and S3 Extractors.
• Support of LuckyGuess applications (e.g. Anomaly Detection) on Snowflake.
• Shared buckets between projects (to replace copying data via Restbox).
• Resolving problem with ever growing tables on Redshift.
• Plus we fixed the annoying problem that yelled "Table already exists" when loading data into sandbox.

All of the above are in various state of completeness (except the last item, which is complete). So stay tuned for more announcements when these features are finished.

We're opening OpenRefine transformations to public. You can now use OpenRefine in your transformation pipeline. 

No further need to write long string replaces in SQL or study how to open CSV files in Python or R. OpenRefine excels in data cleanup and many other data wrangling tasks. 

To create an OpenRefine transformation choose OpenRefine (beta) when creating a new transformation.

Learn more about OpenRefine and its functions and about OpenRefine integration in Keboola Connection.

During last days we have turned on strict input/output mapping validation. Each input/output mapping is checked against the table in Storage if

• all columns exist
• the primary key is equal in both cases 
• datatype/indexes/distkey/sortkey or filter column names have the same letter case

Although we tried to detect all breaches of this ruleset beforehand and contact project owners some have unfortunately slipped through. We're closely monitoring all errors and fixing/restarting all failed orchestrations. 

In case your project is subject to this issue on a larger scale than a single failure, we're able to remove the validation temporarily. Please contact us at support@keboola.com with any further questions/requests.

We're deeply sorry for any inconvenience. 

We're currently investigating issues with Snowflake workspaces (sandboxes, transformations). We'll keep this post updated.

Update 7:36pm CEST: We have passed information to Snowflake support team and they're investigating the issue.

Update 22:36pm CEST: Snowflake team has identified the issue and is working on fix which should be deployed later tonight.

Update 05:55am CEST: The issue has been resolved and all operations are back to normal. Due to high number of affected transformations/orchestrations we won't be restarting them to prevent system overload. Please restart your orchestrations manually if needed.

Thanks for your patience and understanding.

We have introduced maximum query execution time limit for Snowflake transformations. 

If the query execution time exceeds 15 minutes, it will be terminated. This limit should not affect any of current transformations. 

This limit helps us to prevent accidental warehouse overloading by inefficient user queries (cartesian product). This was one of the causes of this week failures.

Between Oct 07 02:00 (GMT+2) and Oct 07 02:24 (GMT+2) some transformation jobs were failing. We have identified the root cause of this issue and fixed it. All affected orchestrations will be restarted shortly.

We're sorry for this inconvenience.

Snowflake backend is down since 11:02am GMT+2. We're trying to fix it, post mortem will follow once we recover platform.

UPDATE:

Dead DWH was switched, everything is running on new backend. All running jobs, employing Snowflake backend, crashed. You have to run it again - everything will be OK for now. In case you're suffering under some other issues, contact us at support@keboola.com, please. 

We'll also publish official post-mortem as soon as Snowflake guys resolve our Issue. Stay tuned and thanks for patience!

UPDATE II:

If you're experiencing errors in your personal sandboxes, reset your credentials under the "Sandbox" link on the right-top of Transformations (https://connection.keboola.com/admin/projects/$pid$/transformations/sandbox). 

Some jobs have failed between 2016-09-26 17:30 CEST and 2016-09-26 17:40 CEST due to problem with one of our DB servers.

All affected orchestrations have been checked and restarted.

We are very sorry for any inconvenience.

If you have any concerns about this, please contact us at support@keboola.com.