For all columns without a defined datatype in a Redshift transformation we added a default LZO compression. This will lead to a marginally slower transformation times but significantly less memory and disk space usage.
You can override this behavior by defining a datatype without a compression.
Wait... what is R?
We've developed the backend and UI for transformations in R. All R transformations run in our public Docker image and have 1 GB memory allowance (will increase in near future).
There's a guide how to develop and test your R scripts locally and a list of best practices and limitations.
Start typing you R scripts now!
AWS recently issued some information about connectivity issue in US-EAST-1 Region, where majority of our infrastructure is located. This may result in 500, 503 and 504 application errors within the infrastructure (our components) or when reaching out to other APIs (extractors).
We're sorry for any inconvenience. we'll keep this post updated with current status. You can also check the current status at http://status.aws.amazon.com/, row Amazon Elastic Compute Cloud (N. Virginia).
---
9:23 AM PST We are investigating possible Internet connectivity issues in the US-EAST-1 Region.
10:09 AM PST We are continuing to investigate Internet connectivity issues in the US-EAST-1 Region.
From now on you can use all OUT tables in all transformations. Previous restrictions on using OUT tables in Redshift transformations were removed.
We had some issues in one of our components (job handling queue) between 10pm and 10:15pm PST on Sunday night (03:00–03:15 UTC Monday). That had resulted in some failed orchestrations (scheduled at that time or starting it's tasks at that time) . We're planning to upgrade the component in the coming weeks, but if the error occurs again, we'll upgrade it immediately. The upgrade will be accompanied with a short maintenance downtime.
We're sorry for any inconvenience.
After the maintenance window we've started experiencing some technical difficulties with our Elasticsearch cluster resulting in orchestration processing glitches. We're working on it and monitoring all processes closely. Please let us know at support@keboola.com if you encounter any unexpected behavior.
We have slightly changed the layout of the transformation detail (sections moved to tabs) and added an Overview graph.
The graph shows the data flow around the current transformation, including all required transformations, input/output tables and transformations from the same bucket, that work with any of the tables as the current transformation. This allows you to get a bird's eye view on what you're working with.
The difference between this and Visual SQL (or SQLdep) is that Visual SQL goes deeper in the transformation and analyses the queries, whereas Overview shows you the environment around the transformation.
There's a direct link to GoodData LDM Visualizer in the model page in GoodData Writer. So in a single click you can compare what is defined in Keboola Connection and what has been already uploaded to GoodData and how the model is interpreted.
We're now showing bucket and transformation descriptions in the UI. Currently there's no way to change the bucket description - it is defined setup when the bucket is created, but we're working on a way to make them editable. Transformation descriptions can be changed easily in the transformation detail
Due to a possible significant inconsistency in real and estimated number of rows (and table sizes) on the MySQL Storage backend which could lead to data loss when editing the data sample, we turned off the data sample editing for all IN and OUT MySQL buckets. SYS buckets are fully editable and will always show and edit all available rows.
Redshift works as expected - editing is allowed for all tables if they contain less than 800 cells (excluding headers).
