tag:status.keboola.com,2013:/posts Keboola Status 2019-01-22T01:25:03Z Keboola Connection "Data Framework" tag:status.keboola.com,2013:Post/1365625 2019-01-21T22:58:40Z 2019-01-22T01:25:03Z KBC is not accessible in all regions

[2019-01-22 1:21 UTC]

Snowflake just announced that disabling OCSP check is able circumvent the error. KBC is fully working, you shouldn't have any issue for now!

[2019-01-22 00:51 UTC]

We were removing all OCSP validation and KBC platform is working OK in both (US/EU) regions for now

At this time, we have no more updates from Snowflake support team yet. You shouldn't have any issue with Keboola Connection. In case of any hiccups, please open ticket directly from your KBC project. Once we have RCA report from Snowflake, this post will be updated.

We're very sorry for this inconvenience and thank you so much for your patience with us and Snowflake engineers.

[2019-01-22 00:34 UTC]

SQL Sandboxes are fully working. Take care that all existing credentials were discarded - use new combination of username and password.

[2019-01-22 00:16 UTC]

Just a few components are still having an issue. 

To make up for this outage we're going to add additional resources and run your jobs in Keboola Connection for next few hours on Warp Drive.

[2019-01-22 00:08 UTC]

Almost everything is working now. Last issues are in Transformation Sandboxes.

[2019-01-22 00:02 UTC]

We're very close to fully working platform. Bear with us! 

[2019-01-21 23:50 UTC]

Component jobs are still serving errors from Snowflake DWH. We're disabling OCSP checks on other places in our infrastructure.


[2019-01-21 23:37 UTC]

Snowflake just confirmed its SSL validation issue in their ODBC driver (https://community.snowflake.com/s/group/0F90Z000000U8d9/alerts-awsus-west).

[2019-01-21 23:35 UTC]

US region is working.

[2019-01-21 23:34 UTC]

EU region is working.

[2019-01-21 23:32 UTC]

We're building app version with temporarily modified OSCP checks.

[2019-01-21 23:17 UTC]

This issue seem to be connected with OCSP cert validation on Snowflake side. We're still working on it.

[2019-01-21 22:50 UTC]

Starting 2019-01-21 22:33 UTC, all customers are seeing error messages throughout their account. We’re aware of the issue and are working on it urgently.

We’re really sorry to be holding you up today! Please know our engineering and operations teams are working hard to get everything up and running and we will update you right here in 30 minutes with the latest information.

tag:status.keboola.com,2013:Post/1364775 2019-01-21T16:06:27Z 2019-01-21T16:06:27Z December Failed Jobs Postmortem

In December 2018 we had two incidents (2018-12-14 and 2018-12-19) which resulted in a number of failed jobs. The first one caused 0.8% of jobs to fail (in a 24h window) and the second one caused 1.2% of jobs to fail (in a 24h window). 

Both incidents were caused by unavailability of the Docker container registry (Amazon ECR). In the first incident we were receiving exceeded quota errors and we initially thought that these were related to higher infrastructure load. A thorough investigation showed that we were nowhere near the limits and now we finally got a confirmation from Amazon that this was an error on their side. The second incident was caused by complete unavailability of the ECR for approximately 30 minutes.

Technical background:

The Docker container registry is used to store the executable code for each component running in Keboola Connection. It is accessed on every job run to make sure that a job is run with the most recent version of the component code. During 2017 we moved most of our components to the Amazon ECR which proved to be very reliable. The outage mentioned above is the first one since 2016 when we began using it. 

Most of the Keboola's infrastructure is duplicated with automatic fail-safe mechanisms in place. That means that minor outages in the underlying services are not noticeable by the end-users. Duplicating the Docker container registry, however, is not an easy task because Docker is not really ready for that yet. So this remains a single point of failure.

Measures already taken and yet to be taken:

  • We have immediately implemented a retry mechanism in our code which will handle short outages, the retry mechanism will also be further improved.
  • We have already started (prior to the incident) reworking the component code validation tooling so that the number of queries to the ECR is reduced by several orders. This will help reduce the impact, should a similar incident happen again.
  • We'll use a dedicated ECR for each Keboola Connection region which will reduce the affected scope for any similar incident in the future. 

tag:status.keboola.com,2013:Post/1363828 2019-01-16T09:22:03Z 2019-01-16T09:22:04Z Job errors

Between 2019-01-15 15:58 and 2019-01-16 8:25 UTC we had a bug in our platform which caused some jobs to fail with user error "Some columns are missing in the csv file". The bug affected jobs where data was imported to Storage with non-default delimiter (default is colon). It is also possible that in some cases an extra column was created in the table. The column contains no data. This column needs to be deleted manually otherwise any subsequent jobs will fail.

We do sincerely apologize for the trouble this may have caused to you. Don't hesitate to contact our support for help.

tag:status.keboola.com,2013:Post/1362225 2019-01-11T10:59:33Z 2019-01-12T19:31:24Z Snowflake issues

Some Snowflake transformations fail with such error: odbc_exec(): SQL error: [unixODBC]SQL execution internal error: Processing aborted due to error 300010:1087106694; incident 5370475., SQL state XX000 in SQLExecDirect. It seems that there is some issue in the Snowflake warehouse. We will keep you informed.

Update 19:30 UTC: The problem should be fixed in the US region. (Last occurrence in our infrastructure is from 12:08 UTC.) It seems there was an issue specific to some CREATE TABLE AS SELECT and INSERTs that use window functions.

Jakub Matějka
tag:status.keboola.com,2013:Post/1360115 2019-01-11T10:38:11Z 2019-01-11T10:38:11Z New Year's Deprecations
It is always a good idea to start the New Year with something new. We decided to do it differently and start this New Year with deprecations. Cleaning up deprecated or obsolete parts of our system also has a place in our TODO lists.

So we'd like to announce the following deprecations of a few components in the US region, MySQL transformations and storage bucket/tables attributes. 


Google Drive Writer (wr-google-drive)

  • We announced the new version of Google Drive Writer on June 28th, 2017.
  • This is the last call, the old component will be shut down by the end of this month - January 31st, 2019.

Lucky Guess (rt-lucky-guess)

  • Enhanced Analysis for Redshift backend is no longer available.
  • This component will be shut down on January 31st, 2019.

SalesForce Extractor (ex-salesforce)

  • This component is already deprecated.
  • This component will be shutdown on January 31st, 2019.

YouTube Extractor (ex-youtube)

  • This component is already deprecated and no jobs were run in last months.
  • This component will be shut down immediately.

Zendesk Extractor (ex-zendesk)

  • A new version of the Zendesk extractor was announced on July 27, 2016.
  • Like the YouTube Extractor, no jobs were run in last months and we will shut it down immediately.

Restbox (restbox)

  • This component was deprecated on July 16, 2018 with multiple replacements.
  • Same here, no jobs were run in last months and it will be shut down immediately.

MySQL Transformations

MySQL Transformations were deprecated on November 13, 2017. As we promised, MySQL Transformations were supported in 2018 and will be finally shut down on January 31st, 2019.

Deprecated Bucket and Table attributes

We are deprecating Bucket and Table attributes. These attributes were used as configuration storage for legacy components. If you need to store additional information with buckets or tables, please use the Metadata API.

Vladimír Kriška
tag:status.keboola.com,2013:Post/1361573 2019-01-09T07:53:06Z 2019-01-09T07:53:07Z Security enhancements to GoodData projects access

Direct single sign-on access to GoodData projects from Keboola Connection has been improved and its security enhanced. Due to it, access has been disabled to all existing users and you have to enable it again in Writer's configuration. 

Also, you are no longer able to switch between projects, once you are in GoodData. You have to return back to Connection and access the other project from there. Although it can be a little less comfortable, it brings improved security and we believe it is a better solution for you.

Jakub Matějka
tag:status.keboola.com,2013:Post/1360109 2019-01-04T08:08:08Z 2019-01-04T11:53:33Z Facebook Ads Extractor Failed Jobs

Between 1:00 - 8:20 CET January 4th, 2019 some Facebook extractor jobs failed after yesterday update of the extractor. We rolled back to the previous working version and continue investigating the issue. Please revise your facebook ads extractor latest jobs and restart if needed. 

We are sorry for this inconvenience.

Update 12:50 CET

The issue has been resolved. The problem was in configurations that used own its authorization token, i.e., not authorized under Keboola Facebook App.

tag:status.keboola.com,2013:Post/1359278 2019-01-02T18:59:31Z 2019-01-07T11:07:01Z Weeks in review -- January 2, 2019

New components and component updates

  • New FTP extractor is now in beta. It is packed with the same features as AWS S3 extractor and it is extensible with processors. We aim to gradually replace the current FTP extractor which is developed by 3rd party.
  • New fit-into-storage processor that allows to import non-csv files (TXT, JSON) into storage by wrapping them into CSV tables.
  • AWS S3 Extractor fixes bug with file matching. When the Subfolders option was off and Wildcard option was on, the extractor would erroneously download files contained in subfolders exactly matching the key.
  • MongoDB extractor export mode is now set to "raw" by default on new configurations.


Input and output mapping for components is now parallelized. Configurations with large number of tables in input and output mapping (except transformations) should now run considerably faster. This affects both extractors and writers. Most recently updated extractors (e.g. AWS S3, FTP, HTTP, MySQL, MSSQL, Storage) also load tables into storage during the extraction of other tables. Together, these make up to 40% run time reduction for some configurations.

Input mapping load type is now shown in input mapping details:

The Run Orchestration dialog now has the option to select/deselect all tasks:


The option to have gzipped CSV file on component output mapping (/data/out/tables/) was removed. Only plain CSV files are accepted now. To our best knowledge this was never used.

tag:status.keboola.com,2013:Post/1359407 2019-01-02T06:18:45Z 2019-01-02T06:18:45Z New version of AdWords Extractor

We have just released a new version of AdWords Extractor. It works with AdWords API v201809 (see the Release notes).

The previous version of the extractor is deprecated and you can use our migration tool which will migrate your AWQL queries. As usual, you have to reauthorize the extractor and give it access to your AdWords data again. The previous version uses AdWords API v201802 which will be switched off on 30 January 2019.

Jakub Matějka
tag:status.keboola.com,2013:Post/1355137 2018-12-19T19:26:25Z 2018-12-19T20:07:04Z Failing Jobs

Since 2018-12-19, 19:13:00 UTC we're experiencing a higher rate of application errors in all regions due to an outage in AWS ECR. 

We're investigating the issue and update this post once there.

We apologize for the inconvenience.

Update 19:30 UTC

This outage also affects the Developer Portal.

Update 20:05 UTC

As of 19:51 UTC the issue is resolved by Amazon

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1353644 2018-12-15T00:09:44Z 2018-12-15T11:46:49Z Failed Jobs

Between 23:16 - 23:22 UTC December 14th, 2018 some jobs failed with Application error due to spikes in infrastructure load. 

We are investigating the root cause and taking measures so that this does not repeat. 

We deeply apologize for the inconvenience caused.

Update [Dec 15, 2018, 08:19 CET]: This still happens occasionally.

Update [Dec 15, 2018, 10:57 CET]: We're still working on mitigating the issue. Comparing to historical hourly avg, actual job error rate is 6% bigger.

Update [Dec 15, 2018, 11:47 CET]: We're going to deploy patch into production. It will take about 2 hours. Expected resolution: 14:00 CET

Update [Dec 15, 2018, 12:45 CET]: Patch has been deployed.

tag:status.keboola.com,2013:Post/1353435 2018-12-14T14:26:14Z 2018-12-14T14:48:59Z Outage Post-Mortem: December 13th, 2018

There was an outage of the Keboola Connection platform in the US region from Dec 13, 2018, 23:19 CET to Dec 14, 2018, 02:28 CET. It was caused by an update of the Elasticsearch Service.

It was a self-service update of the managed Amazon Elasticsearch Service, and we decided to do it because updates were tested and all other clusters we're using were updated successfully - without any issues.

Updates like this usually go smoothly, but not this time. Unfortunately, the cluster froze and refused to accept any requests.

After detecting this, we decided to start a new service and restore from backup.

All services were fully restored to their normal state by Dec 14, 2018, at 02:28 CET.

We have already taken action to prevent this kind of failure from happening again, and we'll be testing all future updates on testing clusters (using snapshots from production clusters).

Any orchestrations in the US region that were scheduled to start during the outage were not started. Therefore you will need to run them manually or wait until their next scheduled run.

We want to sincerely apologize for the inconvenience caused by this outage.

Vladimír Kriška
tag:status.keboola.com,2013:Post/1353194 2018-12-13T22:37:38Z 2018-12-14T07:55:41Z We are experiencing momentary technical difficulties in US
As of around 23:30 CET Dec 13, 2018 we observed technical issues related to out job servers, we are continuing to investigate and will update as soon as we know more.

  • Update: [Dec 13, 2018 23:55 CET] - We're hot on the trail and expect to have the issue resolved shortly.
  • Update: [Dec 14, 2018 02:43 CET] - Service is now fully restored in US region

We're very sorry for any inconvenience.

To make recovery faster we boosted Snowflake performance for next 24 hours.

And you'll hear from us with "post mortem" soon.

tag:status.keboola.com,2013:Post/1352369 2018-12-12T12:09:35Z 2018-12-12T12:22:49Z Weeks in review -- December 10, 2018

New Features

Partial Label

We show a PARTIAL label for jobs which didn't run the whole configuration, but only part of it. Typically these are jobs when only one transformation is run from a bucket or when one file is exported using AWS S3 extractor, HTTP extractor, etc.

Transformation description, Last Runs, and Updates

  • There's a new option to save transformation with description.

  • The Last Runs section and Updates section have been added also to the Transformation Detail page and all components which support "single runs" (e.g. when HTTP extractor or AWS S3 extractor is extracting one file)

Input Mapping data types

In transformations with Snowflake backend, data types should be populated automatically for tables created with the components which set data types - MySQL extractor, Oracle extractor, MSSQL Server extractor, PostgreSQL extractor, DB2 extractor, and Snowflake extractor.

Input Mapping load type

For Snowflake backend, you can set Clone Table load type which leads to an ultra-fast load to a workspace. Most of your tables should be loaded under 10 seconds.

You can find more about this feature in our documentation.


  • You can see a component type in Orchestration Tasks, thus having both MySQL writer and MySQL extractor in task won't be confusing anymore
  • API Token can be Refreshed, Sent or Deleted directly from the token detail page

  • We improved how data is loaded to Storage from our components - compression is used by default. Loads to Storage are about 40% faster for sliced files. So it impacts Credit Consumption too.
  • MSSQL Server Extractor has new option WITH(NOLOCK) and supports incremental fetching with smalldatetime column
  • Orchestration Notifications page has been improved, and notifications can be set easier

New components and component updates

Vladimír Kriška
tag:status.keboola.com,2013:Post/1348408 2018-11-28T20:25:17Z 2018-11-28T20:46:54Z 28 November, 2018 -- Snowflake availability in the EU

From around 21:10 CET today, Snowflake began having availability issues in the EU region.


[Update 21:33 CET -- They have discovered the cause and are working on a remedy] 

[Update 21:42 CET -- Service is now fully restored in the EU region]

tag:status.keboola.com,2013:Post/1347993 2018-11-28T15:02:27Z 2018-11-28T15:02:27Z New Orchestration Detail page

The orchestration section is one of the last parts of Keboola Connection which does not yet have a unified user interface. We believe that users should feel comfortable using the different parts of our UI, so unifying the interface elements is very important to us.

We have launched the new Orchestration Detail page

What has changed?

  • There is no longer a sidebar on the left side with a list of orchestrations
  • Tasks, Schedule and Notifications now have their own place in the detail page - they're no longer combined in that table
  • We added new sidebar on the right side that you should feel familiar with because it is the same as other components
  • Orchestration Action buttons (Run, Enable/Disable and Delete) have been moved to the right sidebar
  • Also, information about creation date and updates now also have their own place in the right sidebar

This is the beginning of the Orchestration Interface tuning and there are more things to come.


Just for completeness, this is the previous version:

Vladimír Kriška
tag:status.keboola.com,2013:Post/1348311 2018-11-28T13:53:58Z 2018-11-28T13:53:58Z 28 November, 2018 -- Storage Job Failures in EU

During deployment of Keboola Connection in the EU region at around 13:30 CET today there was a timing issue that resulted in some failed storage jobs.

The problem occurred because of the asynchronous timing between the rollover of a worker and the API server.  A small number of jobs that were created with the previous version then failed when executed against the new API.  These jobs failed with one of the following error messages "`Invalid source type" or "Workspace not found".

There was no corruption of data by these failures as no jobs that were in progress were affected.


tag:status.keboola.com,2013:Post/1348262 2018-11-28T09:09:08Z 2018-11-28T09:09:08Z 28 November, 2018 -- Stalled Jobs in the EU region
Around 5:00am CEST one of the job worker instances stopped processing assigned jobs. This could have lead to jobs being stuck in the processing state for a long time without any activity.

At 9:30am CEST the worker instance was terminated and all unfinished jobs started processing on other instances.

We're sorry for this inconvenience.

tag:status.keboola.com,2013:Post/1347904 2018-11-27T10:05:49Z 2018-11-27T10:05:49Z Week in review -- November 27, 2018

New Features

You can now provide your project with a markdown description on your project's Overview page

Updated Components

SQL Server Extractor now supports incremental fetching on DateTime and Identity columns

Unexpected Events

Many of you have noticed that we added validation of SQL queries to the Transformation Detail page last week.

Due to some problems in availability of this feature we decided to revert this change.

For now you can still use the "standard" validation button on the right.

We love this feature too and the plan is to bring it back after an additional round of improvements.

tag:status.keboola.com,2013:Post/1345437 2018-11-19T15:01:40Z 2018-11-19T15:01:41Z SQL Validation Outage

Today between 10 and 11 a.m. (UTC) SQL validations and SQLdep functionality in Transformations were returning errors.

The errors were caused by temporary outage of external SQLdep service (https://status.sqldep.com/779807590). Transformation jobs were not affected.

Erik Žigo
tag:status.keboola.com,2013:Post/1344382 2018-11-16T00:00:20Z 2018-11-16T08:53:59Z Storage Jobs Errors - US region

We're experiencing a high number of errors in Storage Jobs in the US region. The errors are caused by invalid SSL connection to Snowflake and we're investigating the issue.

Update 01:02 am CET: Snowflake issues an official statement.

Update 07:00 am CET: Snowflake updated an issue. We are working on a workaround.

Update 07:56 am CET: Issue is resolved. Snowflake moved the Keboola account to a new certificate provider. Now we are processing backlog of jobs.

Update 09:20 am CET: Backlog is cleared. All systems are operating normally.

tag:status.keboola.com,2013:Post/1341189 2018-11-14T14:59:36Z 2018-11-15T15:08:38Z Week in Review -- November 14, 2018

Revised Project Invitations

Inviting users to KBC projects has been reworked. Previously, users were directly added to the project via Users & Settings -> Users menu. Now, when adding a new user to the project, an invitation email is sent to the user.

After clicking on the link in the email, the user is redirected to the login screen, where pending invitations to projects are displayed.

The invited user can accept or decline the pending invitations in the Account Settings.

See the docs for more details.

New Features

  • We have added some predefined searches to the job list. See the docs for a more detailed description of the search queries.

Minor Improvements

  • Jobs related to Python/R sandboxes now show the proper link to a configuration:

  • Improved description of components in events

  • Configurations in trash are now sorted by the date of deletion:

  • Component configuration state is reset on copy:
If you're making a copy of a component configuration (e.g. Zendesk Extractor), the state of the configuration is cleared for the new copy. For example, Zendesk Extractor downloads all tickets from Zendesk on the first run and stores the date and time into its state. On the subsequent runs the Extractor only downloads the tickets created after that time. Now, the new copy of this configuration will also download all tickets on its first run as there is nothing in its state.

tag:status.keboola.com,2013:Post/1337843 2018-11-02T11:24:49Z 2018-11-02T11:24:50Z Week in Review -- November 1, 2018

UI Improvements and fixes

  • Dialog for adding a new task to orchestration now contains configuration description 
  • Every job detail events log now contains component info
  • Improved orchestration notifications ui
  • Improved brief summary of orchestration tasks
  • Unified view of data mappings in the job detail page with respect to the running component as follows: 
    • Input - represents all the data flowing from the storage into the component - typically this would apply for transformation or writer component type. 
    • Output - represents all the data flowing from the component into the storage, typically this would apply for extractor component type.

  • When creating a new extractor in Developer portal  the default bucket is automatically turned on
  • Improved naming of new phase and phase names capitalisation

tag:status.keboola.com,2013:Post/1339184 2018-11-02T08:13:27Z 2018-11-02T08:13:27Z Orchestrations slow down in EU

November 2, between 2:43am CET and 3:14 CET we experienced Snowflake service degradation in EU region that led to slow down of the orchestrations. The issue has been resolved. We apologize for the inconvenience caused.

tag:status.keboola.com,2013:Post/1335558 2018-10-24T18:13:34Z 2018-10-24T18:13:34Z Week in Review -- October 24, 2018

Complexity Badges

There are components in KBC which you can setup in just a few clicks with only basic knowledge. But there are also components which require you to have deep knowledge about how it works. For this reason we have introduced Complexity Badges to be able distinguish between these types of components even before you start setting them up.

Not all components have assigned Complexity Badges yet, but we are continuously working on updating the components catalog.

Minor Improvements

  • When you are retrying failed orchestrations it will now continue from where it failed by default.
  • Transformation backends have new labels with icons.


Restbox shutdown

We would like to remind you that the Restbox component was deprecated in July 2018 and will be completely shut down on November 1, 2018.

If you still have some Restbox configurations in your projects, please migrate them to one of following replacements:

tag:status.keboola.com,2013:Post/1333758 2018-10-19T10:47:06Z 2018-10-19T10:47:07Z Table primary key consistency incident

We have identified a bug in the primary key implementation in Storage which could lead to improper data deduplication. Only a very limited number of tables is affected by this bug – 7 tables in all KBC projects in all stacks. We'll be contacting owners of the affected projects soon to help fixing the affected tables.

Root Cause

The deduplication stopped working when a column used in a compound primary key was deleted.

During this operation, the information about the whole primary key was unknowingly dropped in the Snowflake backend and this was not propagated correctly to our Storage metadata that still contained the primary key (minus the deleted column). In Snowflake, commands such as ALTER TABLE ... DROP COLUMN ... immediately drop the whole primary key if it’s dropping a column of a compound primary key. The deduplication process retrieves primary key information from the DESCRIBE TABLE ... Snowflake command which shows no primary key in the affected tables, but our metadata still incorrectly shows that a primary key is set.

We are implementing a fix that will store and retrieve primary keys from a single source.

Operating with Primary Key Columns

  • Deleting columns which are part of the table primary key is no longer supported in Storage.
  • To delete a primary key column please drop the primary key first.
  • To change primary key of a table you will need to first remove the primary key and then set it again.
tag:status.keboola.com,2013:Post/1328456 2018-10-08T11:41:34Z 2018-10-08T11:41:34Z Week in Review -- October 08, 2018

Updated Components

  • MS SQL Server extractor has had a driver update.  Advanced queries now have slightly different behaviour so it is recommended to switch to simple table and column selected configurations.  Where that is not possible, please see the documentation for further help.
  • Snowflake writer - minor bugfixes and enhancements
    • primary key is added when creating a table
    • nonexistent schema/warehouse validation
    • warehouse detected correctly for users with different login name and user name
  • Database extractors - column datatypes are now shown in UI

Minor Improvements

    • You can now change the sharing type of a shared bucket in the storage console.

    tag:status.keboola.com,2013:Post/1330188 2018-10-08T11:09:25Z 2018-10-08T11:09:25Z Stalled jobs in EU region

    Around 1:30am CEST one of the job worker instances stopped processing assigned jobs. This could have lead to jobs being stuck in the processing state for a long time without any activity.

    At 11:15am CEST the worker instance was terminated and all unfinished jobs started processing on other instances.

    We're sorry for this inconvenience.

    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1329850 2018-10-07T06:43:17Z 2018-10-07T06:43:17Z Failed and delayed jobs in EU region

    Database storing locks was restarted at 03:49 UTC which caused the job failures. Also some of the jobs were queued after this failure.

    The backlog of all jobs was cleared at 06:15 UTC. The system is fully operational now. We're working on infrastructure changes which should prevent similar issues.

    tag:status.keboola.com,2013:Post/1328422 2018-10-06T10:25:29Z 2018-10-06T10:25:30Z US Region Maintenance Finished

    The maintenance period in the US region has just finished. All systems are back online and working properly.

    Thank you for your patience.

    Ondrej Hlavacek