Transformation Events

As a part of the ongoing Transformation API overhaul we've changed transformation events. We tried to keep it simple, so there's one event for each:

  • Engine startup
  • Start of a phase
  • Database cleanup (happens at the beginning and end of a phase)
  • Input mapping
  • Input mapping that takes longer than 120s (configurable)
  • Transformation (as a whole, not a query in the transformation)
  • Query that takes longer than 120s (configurable)
  • Output mapping
  • Engine shutdown (success or error)
There are no more any START or END events and the engine produces less events. Further activity can be found in related storage events or jobs (eg. table imports and exports).

New Extractor (and Migration Guide)

We developed a new version of the extractor and renamed the current version to (Deprecated).

As some of the changes introduced in the new version are backwards incompatible, these two versions will be running aside for a period of time and we kindly ask you to migrate your configurations.


  • Extractor now runs asynchronously - less CURL errors, more scalability and durability, monitoring via Jobs app
  • Configuration is stored in sys.c-ex-salesforce, was in sys.c-SFDC previously
  • Data is stored in in.c-ex-salesforce-config, was in in.c-config previously (where config is configuration name)
  • Some changes in the UI (mostly the menu on the right)

Migration Guide

As there is an OAuth authorization in the process, we can't automate and test the process. Follow this guide for each extractor configuration in your project. Migration can be performed in 4 easy steps:

  1. Copy configuration
  2. Reauthorize extractor
  3. Run extraction
  4. Integrate 

1. Copy Configuration

Using the UI create a new configuration in extractor and copy & paste all queries and credentials you have in your (Deprecated) configuration.

2. Reauthorize Extractor

As the extractor runs on a different worker, you need to get new OAuth tokens. Do it simply by clicking on Authorize SalesForce in the right menu.

3. Run extraction

You can now run all queries by clicking on Run all queries in the UI. You can monitor the progress in the Jobs application.

If you have any incremental query in your configuration you need to migrate the data extracted by these queries first. Repeat this for every incremental query: 

  • Use Storage application and make a snapshot of the two original incremental tables (eg. in.c-SFDC01.User and in.c-SFDC01.User_deleted). 
  • Use the Create new table from snapshot function to copy the tables to the new bucket (eg. in.c-ex-salesforce-SFDC01.User and in.c-ex-salesforce-SFDC01.User_deleted). If the destination bucket does not exist, simply create the bucket manually or run any non-incremental query.

4. Integrate

Once you have downloaded the initial set of data you may need to alter some transformations and orchestrations to integrate the new extractor in the whole pipeline.


Create a new orchestration task with the new extractor with the same parameters and then delete the old Deprecated extractor. 


There are two options how to migrate the transformations. You can change the input mappings from the old tables to the new tables (the structure and column names remain the same), or you can keep the old names and simply delete the old tables and make an alias for each deleted table (eg. delete in.c-SFDC01.User and make an alias, eg. in.c-ex-salesforce-SFDC01.User->in.c-SFDC01.User). 

End Of Life Announcement

The (Deprecated) extractor will be terminated on January 15th. If you have any trouble migratings your configuration, please contact

Transformation Input Mapping: Views and Tables

We just introduced an icon in input mapping to show, whether the input mapping is created as a view or a table

Running Redshift transformations and reading data from Redshift Storage (which is the current recommended fastest option) you can choose between creating a table or a view in the input mapping. Whats the difference?

Views are lightning fast to create. Input mapping just aliases a table to your working schema within the cluster and that's it (including all filters). You can then layer another view on that and another... until you're done and you can set the final view as a source table for an output mapping. All the work is then done when processing the output mapping. That is the snatch - it is easier to reach the cluster's limits (memory, disk) with one large query (multiple nested views). And because the cluster is out of memory, it will also terminate all other queries running on the cluster at the same time. 

So please be careful when using views. If you're not sure, feel free to reach out to for more assistance. 

Orchestration UI Update

We released a new version of Orchestration API and UI. The UI is completely rebuildm is lightning fast and everything can be now configured within a single UI. 

Here's a short list of the most exciting features:

  • orchestrations dashboard with the most important details about every orchestration
  • new orchestrations get a token and configuration table automatically (no more Storage Console)
  • orchestration tasks have a configuration UI, components are identified by names and icons instead of URLs

Happy orchestrating!

UPDATE: A short outage occurred while migrating Orchestrator configurations causing some of the orchestrations to fail to start on schedule. We restarted all failed orchestrations manually. We're sorry for this inconvenience.

SSL issues

We're experiencing some issues in our internal infrastructure. 

You can see following errors on affected orchestrations:

  • Error response from component
  • [curl] (#35)

We're working on a fix and restarting all failed orchestrations. We're sorry for this inconvenience.

Transformation MySQL server down

Our MySQL transformation server TAPI-A is down. We're offloading all work to the backup server, rerunning all failed orchestrations and restarting the TAPI-A server. 

We're sorry for this inconvenience, all operations should be back to normal soon and your transformations will switch to the restarted server automatically.

Provisioning API Update

We'll be updating Provisioning API today at 1:15 pm PST. This shouldn't take longer than 15 minutes. Transformations will be resumed, your credentials will still be valid, only creating of new sandboxes in the Transformation UI will be disabled during this period. 

1:30 pm PST: Migration completed. Thanks for your patience.

Randomly Failing Orchestrations

We're experiencing an AWS-wide DNS problem. Every 01 minute of each hour the DNS does not respond, which can cause failing attempts to connect between components in Keboola Connection. This will cause the running orchestration to fail.

It is reported all across AWS:

If your orchestration fails with an application error around 01 minute of an hour (it can propagate to the job a bit later, eg. at XX:02), you're most likely affected by this issue. Just re-running the orchestration will solve the problem. 

Please bear with us while we're trying to solve this issue with AWS Support. Here are some examples of failed orchestrations that might help you identify this issue.

Redshift Transformations: Input Mapping Enhancements

These options were turned off for VIEWs (input mappings from Redshift Storage to Redshift Transformation created as a VIEW):

  • Data Types (will inherit Storage data types in the future)
  • Sort Key
  • Dist Key

These options are still present for TABLE input mappings and for MySQL Storage or Transformation.

And on the other hand we turned on Days filter for all Input mapping (was previously turned off for all Redshift transformations).