We have increased memory limit for all R transformations to 8GB. Please note that there's still a 10GB storage limit on the Docker worker and that R might consume some of that space for swapping.
Have a nice day!
Our internal logging system was struck with a failure on some nodes in our infrastructure. This could lead to one of following
- stalled jobs
- failing jobs without any message
- untraceable application errors
This outage lasted between 6:20am-3:00pm CEST (4:20am-1:00pm UTC / 9:20pm-7:00am PST).
We're sorry for any inconvenience, you can restart affected jobs, all systems are fully functional now.
Redshift writer is the new addition to the growing list of writers. You can now write your data to any Redshift warehouse, or if your project includes Redshift, we can provision you with it's own database - that's useful when you need to perform read-only operations in an external application, such as chart.io.
We have introduced query limits for Redshift clusters to prevent deadlocks and keep the clusters in good shape:
- 5 queries in parallel (further queries will be waiting in a queue for 60 minutes)
- 60 minutes execution time per query
If the query time exceeds 60 minutes, it will be terminated with a user error "Query cancelled on user's request" and/or "An exception occurred while executing".
These limits will take place during the next maintenance window of each cluster (after it reboots).
If you need to change the limits, contact us at support@keboola.com.
EDIT September 22nd: Due to a high number of requests for a higher number of concurrent queries we've increased the limit from 2 to 5 concurrent queries on each cluster.
A new version of Transformation API was released today with a new feature - we're blocking all SELECT queries.
These queries do not perform any real operation to your data (if not accompanied with CREATE TABLE or CREATE VIEW) and caused our servers to load the result in memory. If your transformation contains a pure SELECT query, it will fail with a Query not valid error message.
The fix is easy - delete or comment the SELECT query, it won't effect your transformation.
Thanks for your understanding.
We have launched a new MySQL server for DB Writer. All current credentials (both for reading and writing) are now obsolete, if you have any applications connecting to the MySQL database provided by DB Writer please update the credentials from the writer's page.
If you want to load some data in your R transformation but it can't be stored in a table, you can now use Saved Files feature. This allows you to specify multiple tags and for each tag the engine will download the latest stored file in File uploads with this tag. If no file is found, the transformation will fail. The files are then stored in /data/in/user/{tag} and the manifest files in in /data/in/user/{tag}.manifest.
This comes extremely handy when you externally pre-generate binary data for a transformation (model, bucketing criteria). You just upload the file to the File upload and assign a certain tag, that you will then use in a transformation.
One of our Redshift clusters was getting stuck on certain operations. It caused a bottleneck on Provisioning API that delivers database credentials to transformations and other components.
Affected orchestration have been restarted. We're sorry for this inconvenience.
Amazon Simple Queue Service is reporting increased error rate in our main region. This affects majority of our applications and APIs. We hope for a quick fix, please bear with us, we'll post any updates here.
UPDATE 11:41pm PST / 08:41 CEST: We've migrated Storage API to SQS in a different region, so it's fully functional now. We're working on migrating all other APIs and apps to a different region as well.
UPDATE 00:05am PST / 09:05 CEST: All components should be working now, we're migrating process terminating queues (this bit is not working yet). You can restart your failed jobs.
UPDATE 00:52am PST / 09:52 CEST: Everything is back up and working normally. We're sorry for any inconvenience, you can now restart failed orchestrations and terminate all stuck or waiting processes.
On Wednesday at 9.15am PST We incorrectly configured sql_mode on the MySQL transformation server to value STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION.
This has caused many errors in you transformations and orchestrations - mostly with messages like
- Invalid datetime format: 1292 Truncated incorrect datetime value: ...
-
String data, right truncated: 1406 Data too long for column ...
-
Incorrect decimal value: '' for column ...
- Incorrect integer value: '' for column ...
We identified the error and fixed it on Thursday at 12.20am PST.
We are sorry for this inconvenience, we're working on setting up the environment so this error can't happen again. We tried to rerun all failed orchestrations, but if there's anything left, please restart the orchestration, everything should run correctly now.