CSV Import

Are you tired of loading CSV files into a table over and over again? Do you sometimes forget to set the incremental flag or specify a wrong delimiter?... Well those days are over!  Now there's a simple new way to load CSV files.

The new CSV Import allows you to create and save a configuration and just select the file next time. You'll find it under Extractors in your project and you can read a bit more in our documentation.

Job Failures

Today, July 12th 2016, between 10:03 and 10:07 UTC+2 our Elastic cluster was unavailable. 

Some jobs had failed, we're restarting all affected orchestrations. We're sorry for this inconvenience. 

Week in Review -- June 6th, 2016

Let's start the week with a resume of features/improvements introduced in the last week.

New/Updated Components

  • MongoDB extractor
  • Prague's stand up comedian and node.js developer Radek Tomasek created FTP/FTPS Writer and added visual configuration for his SFTP/WebDAV writer

UI Improvements

  • Primary key in input mappings is automatically populated when the destination table already exists
  • Performance improvements in generic extractor (and in extractors based on generic extractor, eg. all templated extractors) - parsing large JSON responses is significantly faster

Redshift Incremental Load Issues (duplicate rows)

On Jun 2, 2016 3:40pm UTC+2 a new version of Storage API was released containing the following bug.

Incremental loads into Redshift tables with primary keys do not correctly deduplicate data - rows with duplicate primary key may exist in the table.

We'll be deploying the original version shortly and then we'll dedup all affected tables.

We're sorry for this inconvenience, we'll keep you updated in this post. 

UPDATE Jun 3, 2016, 2:30pm UTC+2

Original version deployed. We're starting recovery process, no service outage will be required.

UPDATE Jun 4, 2016, 9:15am UTC+2

Recovery process for all affected tables is finished, all duplicate records should be mitigated. We're now investigating the root cause of the issue to prevent similar incidents in the future.

OAuth issues (Dropbox Writer, TDE Writer)

We're experiencing errors in our legacy OAuth component. Affected components are

  • Dropbox Writer
  • TDE Writer (using Dropbox Writer)

Some configurations may not run. We're sorry for this inconvenience and investigating this issue. 

UPDATE 10pm UTC+2: The issue was resolved, everything should be running smoothly again. Please let us know if you still have errors.

Incidents on June 1st 2016

Around 9pm UTC+2 we encountered 2 system wide issues.

1) One of our API servers ran out of disk space. Requests running on this server might have finished with an error. This influenced UI, orchestrations, listing jobs or worker jobs using our APIs.

2) AWS encountered increased API error rates. This might have influenced all components of Keboola Connection, from UI to orchestrations.

Both issues are now resolved and all operations are resumed. We're sorry for any inconvenience and thank you for your patience. We're currently going through failed orchestrations and restarting them. 

Failed jobs

One of our metadata servers was restarted by AWS at 1:55pm UTC+2. 

This may have caused some jobs end with an application error. 

We're sorry for this inconvenience and we'll restart all affected orchestrations. 

Transformation Running Failures (Transformation X not found)

We've encountered an issue running newly created transformations. Please stay with us, the fix currently being tested and will be deployed within the next hour. 

The symptom is a failing job with error Transformation 'x' not found. Only newly created transformations are affected by this bug, all old transformations should work.

We're sorry for this inconvenience!

EDIT 4.00 pm CEST: The bugfix has been deployed to production. 

Redshift Transformation Input Mapping Update

As of now we're removing type input mapping attribute in Redshift transformations. 

All tables which were set to be created using CREATE VIEW from another schema will now be automatically created using CREATE TABLE and the data will be copied.

There are many reasons behind this update, the most important being a deadlock when deleting tables in Storage that had a view created in a transformation or in an orphaned sandbox.

You may experience slightly slower transformation times and heavier disk usage caused by physically copying data.

Should you experience any unexpected behaviour or a major slowdown, please let us know.

PS: No action required, we're glad you made it here safely.

Transformation jobs forking

Between 11:08am CEST and 2:03pm CEST under certain circumstances transformation jobs were forking from existing processing transformation jobs. We have found the root cause, fixed it, terminated all forks and restarted all failed orchestrations. If you're still seeing this issue, please let us know.

UPDATE: This issue occurred again between 4:49pm CEST and 8:05pm CEST.

We're sincerely sorry for this inconvenience.