Keboola Status

We're changing the way Redshift transformations and sandboxes are provisioned.

During the week from September 26th to September 30th all projects will be migrated. What does that mean for you?

Faster with less errors

In certain situations (eg. table locks) creating of a sandbox would take a long time. After the migration provisioning will no longer depend on any locks in the whole Redshift cluster.

Provisioning and data loading

We're offloading the input mapping work to Storage API. Storage API is now in charge of creating the sandbox or transformation workspace for you as well as loading the data into it. Storage API will decide the fastest way to load required data into your workspace.

Credentials change

Username, password and schema for your Redshift sandbox will change and new sandboxes will be created. Your current sandboxes will be deleted 7 days after the migration. UI will no longer serve credentials for your current sandbox, only for the newly generated one. You will not be able to load data into your current sandbox.

No direct access to bucket data

If your transformation uses data directly from a Redshift schema, this won't be supported. Queries like

SELECT * FROM "in.c-mybucket"."table" 

will no longer have access to in.c-mybucket Redshift schema. If your transformation contains such queries, please adjust the query so that it uses a table specified in the input mapping.

Transparent and hassle free migration

There will be no service interruption and no expected delay in your transformations. Unless you're using direct access to bucket data there is no action required. In case of any problems the whole migration is reversible. 

If you are concerned about your operations please get in touch with us at support@keboola.com. We can try out the migration or change the date of migration. 

One of yesterday's updates has introduced a bug in transformation engine. Output mappings with boolean columns are failing with the following error:

Only a few projects are affected by this bug and we're working on a fix that will be ready within next couple hours. We will update this post when the fix is deployed. 

We're sorry for this inconvenience.

UPDATE 1:36 CEST: The fix was deployed into production. Everything should work as expected.

Due to a bug in AWS Redshift that we're unable to resolve without service interruption we're announcing system wide Redshift backend maintenance on September 24th.

The bug shows increased table sizes. If you're experiencing unexpected project size growth you might be affected by this bug.

During the maintenance all data on Redshift backend in affected projects will be unavailable and all Redshift transformations will not be able to execute. The maintenance will last around 3 hours. 

The schedule of the maintenance in different timezones is following:

PDT: Saturday, 24 September 2016, 05:00:00-08:00:00
UTC: Saturday, 24 September 2016, 12:00:00-15:00:00
CEST: Saturday, 24 September 2016, 14:00:00-17:00:00

If you're concerned about the maintenance or you need to schedule a different time please contact us at support@keboola.com.

Update September 16th: The maintenance is cancelled. We're working on an on the fly detect-and-fix solution. Release date unknown yet, so if your project shows any growing table size symptoms, contact us at support@keboola.com to help you fix the problem.

Since our last update here's what happened in Keboola Connection

• David Ešner updated the Mailkit Extractor to retrieve raw campaign data from the API
• We have reworked the Google Analytics Extractor and we'll start offering migration tool this week

Bugfixes, minor changes

• The UI loads configuration changes in the background, so it should stay up to date. This does not work when the configuration is open for editing.

After last GoodData maintenance some uploads to GoodData using our writer are started failing with the following error:

Running the LoadData task again manually will be successful. If you're running more tasks in one job, you can disable the failing job to get all the other tasks through. We have discussed the error with GoodData support and this is what they have to say:

I have already confirmed by our R&D engineers, that we have introduced the issue during the R119. We have prepared the fix for the issue and the internal discussion about the fix delivery has already been triggered. The fix unfortunately requires short time (up to 10 minutes) GoodData upload subsystems outage, therefore the our responsible managers are trying to find the best window for its delivery.

I can also confirm, that temporal switching OFF of the gZip in the command would serve as workaround, but I do as well understand the concerns you are mentioning that keeping you away of using it.

I'll share further information w/ you as soon as I have them available.

Currently we cannot turn off gzipped transfers for all projects as it will slow down all uploads. Let's wait if GoodData can release the fix during the weekend and if not, we'll implement temporary gzip disabling for affected projects early next week. Please bear with us, feel free to drop us a line at support@keboola.com if you're concerned about your project.

UPDATE 4:08pm CEST: GoodData announced maintenance for August 20th 2016. During this maintenance this bug should be resolved.

UPDATE Monday August 22 7:30am CEST: We're still rarely experiencing this issue.

UPDATE Monday August 22 2:40pm CEST: We have deployed a workaround fix for the bug so GoodData data loads now should work as expected.

We're sorry for this inconvenience.

Since our last update here's what happened in Keboola Connection

• Radek Tomasek from BlueSky Digital introduced his new IBM dashDB writer
• Sun Marketing introduced extractors for Heureka.sk, Pricemania.sk and Srovname.cz 
• We have introduced versioning and diffing   to the majority of our components
• We've introduced a replacement Storage API Importer
• Custom Science applications are finally public - it's your GitHub/Bitbucket code in R, Python or PHP executed in our infrastructure
• We have announced switching from VPNs to SSH tunnels
• There's a new Redshift DB extractor
• The JSON Parser public service now runs over HTTPS and has better error messages

Bugfixes, minor changes

• Facebook Ads extractor was updated to API v2.7

Since 2016-07-28 there were no updates in our currency tables. This was fixed today and all records were pushed into all projects that have Currency Extractor set up. 

We're sorry for this inconvenience, let us know, if you have any concerns about this at support@keboola.com.

Are you tired of loading CSV files into a table over and over again? Do you sometimes forget to set the incremental flag or specify a wrong delimiter?... Well those days are over!  Now there's a simple new way to load CSV files.

The new CSV Import allows you to create and save a configuration and just select the file next time. You'll find it under Extractors in your project and you can read a bit more in our documentation.

Today, July 12th 2016, between 10:03 and 10:07 UTC+2 our Elastic cluster was unavailable. 

Some jobs had failed, we're restarting all affected orchestrations. We're sorry for this inconvenience. 

Let's start the week with a resume of features/improvements introduced in the last week.

New/Updated Components

• MongoDB extractor
• Prague's stand up comedian and node.js developer Radek Tomasek created FTP/FTPS Writer and added visual configuration for his SFTP/WebDAV writer

UI Improvements

• Primary key in input mappings is automatically populated when the destination table already exists
• Performance improvements in generic extractor (and in extractors based on generic extractor, eg. all templated extractors) - parsing large JSON responses is significantly faster