We're changing the way Redshift transformations and sandboxes are provisioned.

During the week from September 26th to September 30th all projects will be migrated. What does that mean for you?

Faster with less errors

In certain situations (eg. table locks) creating of a sandbox would take a long time. After the migration provisioning will no longer depend on any locks in the whole Redshift cluster.

Provisioning and data loading

We're offloading the input mapping work to Storage API. Storage API is now in charge of creating the sandbox or transformation workspace for you as well as loading the data into it. Storage API will decide the fastest way to load required data into your workspace.

Credentials change

Username, password and schema for your Redshift sandbox will change and new sandboxes will be created. Your current sandboxes will be deleted 7 days after the migration. UI will no longer serve credentials for your current sandbox, only for the newly generated one. You will not be able to load data into your current sandbox.

No direct access to bucket data

If your transformation uses data directly from a Redshift schema, this won't be supported. Queries like

SELECT * FROM "in.c-mybucket"."table" 

will no longer have access to in.c-mybucket Redshift schema. If your transformation contains such queries, please adjust the query so that it uses a table specified in the input mapping.

Transparent and hassle free migration

There will be no service interruption and no expected delay in your transformations. Unless you're using direct access to bucket data there is no action required. In case of any problems the whole migration is reversible. 

If you are concerned about your operations please get in touch with us at support@keboola.com. We can try out the migration or change the date of migration. 

One of yesterday's updates has introduced a bug in transformation engine. Output mappings with boolean columns are failing with the following error:

Only a few projects are affected by this bug and we're working on a fix that will be ready within next couple hours. We will update this post when the fix is deployed. 

We're sorry for this inconvenience.

UPDATE 1:36 CEST: The fix was deployed into production. Everything should work as expected.

As you may have noticed, we have launched a new version of Google Analytics Extractor.

To help you migrate your existing configurations from the old extractor, we prepared a Migration Tool.

The migration process will migrate all your existing configurations, which are properly authorised and also replace the old extractor in all orchestration tasks in the project.

How to

1. Click "Proceed to Migration" button in any configuration of the old Google Analytics Extractor
2. A dialog window will open with a list of all configurations in the project
3. Click "Migrate" button and a migration job will be created
4. You can watch progress of the migration job either in the migration dialog window or in the Jobs list
5. After job is finished, you can see the result of the migration for every configuration and also affected orchestrations on the "Affected Orchestrations" tab

Backward compatibility

The only thing that is not migrated automatically are the Date Range settings.

In the old extractor, the date ranges were specified in parameters of a orchestration task by "since" and "until" attributes.

In the new extractor, the date ranges are specified in the extractors configuration and you can have multiple date ranges per query.

After migration, the orchestration task parameters will remain as they were. If present, the orchestration task with new extractor will fail. Simply move your date range settings from the orchestration task parameters to the extractors query configuration.

Orchestration error:

Move "since" and "until" parameters from orchestration task to query configuration: 

For more information about new Google Analytics Extractor see documentation.

If you have any troubles or questions, please contact support@keboola.com.

New version of Google Drive Extractor, running on container based architecture, is now available. The old one has been marked deprecated.

Features

Not much has changed in terms of functionality, but the configuration UI is now much faster :) Also the extractor now uses the newest version of the Google Drive and Google Sheets API.

Migration

To help you migrate your configuration to the new version of this extractor, we have prepared a migration tool, which will be available very soon.

Tomáš Trnka created writer which creates xlsx files from selected Storage API tables and send them to your Amazon S3.

Add user to project API call of GoodData Writer allows assigning filters to user invited to project by email, see api documentation.

Due to a bug in AWS Redshift that we're unable to resolve without service interruption we're announcing system wide Redshift backend maintenance on September 24th.

The bug shows increased table sizes. If you're experiencing unexpected project size growth you might be affected by this bug.

During the maintenance all data on Redshift backend in affected projects will be unavailable and all Redshift transformations will not be able to execute. The maintenance will last around 3 hours. 

The schedule of the maintenance in different timezones is following:

PDT: Saturday, 24 September 2016, 05:00:00-08:00:00
UTC: Saturday, 24 September 2016, 12:00:00-15:00:00
CEST: Saturday, 24 September 2016, 14:00:00-17:00:00

If you're concerned about the maintenance or you need to schedule a different time please contact us at support@keboola.com.

Update September 16th: The maintenance is cancelled. We're working on an on the fly detect-and-fix solution. Release date unknown yet, so if your project shows any growing table size symptoms, contact us at support@keboola.com to help you fix the problem.

Our AdWords Extractor will be updated to use new version v201607 of AdWords API on Monday September 12. Please be sure to update your configuration to not use no longer supported metric names till then. Also do not use new metrics until the update, we will let you know about it. See release notes and migration guide for more information.

Snowflake is experiencing service outage which impacts projects with Snowflake backend.

Snowflake team is working to restore services as quickly as possible. 

https://support.snowflake.net/hc/en-us/articles/213508163-August-23-2016-Service-Disruption-Update

We are sorry for any inconvenience.

UPDATE 11:12am PST

Snowflake service is back up and running.

Since our last update here's what happened in Keboola Connection

• David Ešner updated the Mailkit Extractor to retrieve raw campaign data from the API
• We have reworked the Google Analytics Extractor and we'll start offering migration tool this week

Bugfixes, minor changes

• The UI loads configuration changes in the background, so it should stay up to date. This does not work when the configuration is open for editing.

Mailkit has extended its API with new functions allowing retrieval of RAW data about your campaigns. A new functionality has been added to the Mailkit Extractor as well to match the new API modifications. Now you can retrieve all data using the new three functions. They are designed to pull large volumes of data and the retrieval is much faster than with the original recursive functions. 

More information about the modification can be found in the documentation.