Keboola Status

Last week was shorter for us due to Easter, but we managed to do more things in a shorter time.

Versions Management

We introduced simple management for transformation buckets. You are now able to rollback to the previous version of your transformation. Or simply copy one version to another.

Read more about Versions Management in previous post.

Documentation

We're still working on our documentation to provide you with best support. Whether for easier components configurations or diving more into Keboola Connection in general.

Check our post about Documentation Changes.

Patches

Thanks to you, our users, we can detect anomalies (or bugs) in components faster. Thus after applying patches, components will be better for all, not only for issue reporters.

• Small fixes have been applied to Keboola Connection UI
• Bings Ads extractor, provided by David Ešner, has been updated

Wait... what is R?

We've developed the backend and UI for transformations in R. All R transformations run in our public Docker image and have 1 GB memory allowance (will increase in near future). 

There's a guide how to develop and test your R scripts locally and a list of best practices and limitations.

Start typing you R scripts now!

From now on you can use all OUT tables in all transformations. Previous restrictions on using OUT tables in Redshift transformations were removed.

Here's a list of things that have changed in last couple weeks, in case you didn't notice in the UI directly.

No Custom Credentials

Support for custom transformation or sandbox credentials was dropped in the UI. Everything in the UI is provisioned. You can still use your own credentials using the API.

Input Mapping Within Redshift

When adding an input mapping in a Redshift transformation from a Redshift (Storage) backend, the engine chooses the fastest data transfer route. As the transformations and storage share the same cluster, the table never leaves the database and is transferred directly within the cluster. You have an additional option to set, if the input mapping is performed as a CREATE VIEW or CREATE TABLE from the original table. View is faster (there is no data transfer at all), but might cause memory issues, if you start piling up views on top of each other, table is slower (the data is duplicated), but may perform better on larger set of queries.

Also COPY Options pane disappeared from input mappings within Redshift, they didn't make any sense there.

Note: Be careful with mismatching datatypes.

Sandbox Credentials

Sandbox Credentials page now shows both MySQL and Redshift credentials with all their details. 

Running Processes

You can now terminate both MySQL and Redshift processes. 

Note: Terminating a database process is not immediate, it might take some time - there are some rollbacks on the backend.

Note: Running a Redshift transformation uses both MySQL and Redshift credentials. Terminating any of them will end the transformation.

You can define Distribution Style for Redshift input mapping.

Here are some hints, how to design tables: http://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-best-dist-key.html

All Redshift transformations can now be easily integrated into SQLdep. A single click uploads your current transformation (and all dependecies) to SQLdep and shows you a big picture of what’s happening in the transformation. You can look at it from the table, column or query perspective. 

Note: this is only available for Redshift transformations. For more information about our Redshift backend please contact support@keboola.com.

On the new sandbox credentials page you can now share your sandbox database to another user in the same project. The user, you’re sharing your sandbox with, needs to have active sandbox credentials, and your sandbox database will show up in the databases list in his MySQL client.

You can now define which rows are deleted in the output table. This only makes sense with incremental turned on though. 

Our main transformation server (TAPI-A) was down between 8:21pm and 8:27pm PST. We had to restart this server to solve connectivity issues. We tried to find all crashed orchestrations and restart them. 

EDIT 8:51pm PST: We’re still having issues with TAPI-A server. Switching all transformations to OVH server.

EDIT 9:45pm PST: The issues persisted on the OVH server. We found a bug in the transformation engine and did a rollback to a previous version. Everything should be operating normally now.

We apologize for this inconvenience, if you find any failed orchestrations, that were not restarted manually, please feel free start them again.