tag:status.keboola.com,2013:/posts Keboola Status 2017-10-10T18:21:41Z Keboola Connection "Data Framework" tag:status.keboola.com,2013:Post/1197138 2017-10-10T10:31:36Z 2017-10-10T18:21:41Z Week in Review -- October 9th, 2017
The longstanding issue with slow SHOW DATABASES query executions on Snowflake has finally been resolved. We will now gradually enforce snowflake query execution limits where applicable by contract. Before turning the limits back on, we will check the query history and notify you if you may be affected.
(Query times in milliseconds)

New Features

Minor Improvements

We continue with cleaning up the KBC UI. So far, we have fixed many little annoying issues:

  • List of configurations now shows when the configuration was created (with exact date on hover) 
  • Cleaned up Job list (removed unused stuff).
  • Backend in Transformations is now preselected to default project backend.
  • CSV importer now supports .TSV and .GZ files too.
  • "Test Credentials" button and result polished.
  • Removed 'sticky' buttons when editing configurations.


  • Fixed files input mapping not showing in certain configurations.
  • Fixed a couple of errors with the Save button when editing transformation queries.
  • Fixed bug in Redshift writer with mixed case table names.


  • The encryption API is simplified to a single endpoint. If you manually encrypt values for KBC, please follow the docs. We will be contacting you if you are using the deprecated API calls.

Ondřej Popelka
tag:status.keboola.com,2013:Post/1195929 2017-10-04T12:49:43Z 2017-10-04T18:59:51Z Job Failures [Not Resolved]

Today, October 4, 2017, we experienced several jobs failures between 12:07--14:40 PM CEST. One of our worker servers became unresponsive which led to the jobs failures. We have shut down the instance and launched a new one. Everything is working properly now and we will further investigate the root cause of the issue. 

We're still encountering rare job failures. We're monitoring the issue closely and trying to find the root cause.

We are sorry for any inconvenience.

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1194027 2017-09-27T20:07:45Z 2017-09-27T20:07:45Z Week in Review -- September 26, 2017

New Features and Improvements

  • Geneea NLP app now supports relations - e.g. My pizza was completely burned. > burn(OBJECT:pizza)
  • Redshift transformations input mapping now supports ZSTD compression.
  • Improved handling of events search errors

New Applications


GeoIP is latest transformation component power by MaxMind. It helps you to obtain location data from your IP address inputs.

Screen Shot 2017-09-18 at 23221 PMpng

Please check out the BitBucket Documentation to get you started.
Screen Shot 2017-09-18 at 24213 PMpng

Data Health Application

Data Health Application is an app designed to aid users to produce a clean data file. For more details, please check out Keboola Blog.

Screen Shot 2017-09-20 at 120626 PMpng

tag:status.keboola.com,2013:Post/1193772 2017-09-25T14:46:38Z 2017-09-25T14:46:38Z Job Failures [Resolved]

Today, September 25, 2017, we experienced jobs failures between 15:11 - 16:15 PM CET. One of our worker servers went down and while launching a new one automatic job restart mechanism didn't seem to work properly and this led to the jobs failures. Everything is working properly now and we will further investigate the cause of the issue. However we recommend to review your orchestrations and take appropriate action if needed. 

We are sorry for any inconvenience.

tag:status.keboola.com,2013:Post/1192325 2017-09-19T15:36:01Z 2017-09-19T15:36:01Z Restbox S3 Wildcard Download Bug

On Tuesday September 19th around 7am CEST a new version of Restbox was deployed. This release contained a bug affecting all wildcard downloads from S3. The older version was restored at 5:20pm CEST.

We're sorry for this inconvenience.

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1192268 2017-09-19T12:37:53Z 2017-09-19T12:37:53Z Skipped Orchestrations at 0:00 UTC

A few Orchestrations which were scheduled to run at exactly midnight UTC may not have started due to an issue with the Orchestration Scheduler. The issue arose on the 15th of September and has now been resolved.

If you have orchestrations which are scheduled to run at 0:00 UTC, please check that these orchestrations have run correctly.

Sincere apologies for the inconvenience.

Erik Žigo
tag:status.keboola.com,2013:Post/1190940 2017-09-13T14:18:44Z 2017-09-13T14:18:44Z SQLdep for Snowflake (BETA)

All Snowflake transformations now have SQLdep enabled. 

SQLdep allows you to visualize the data flow in your SQL. You can now easily see the impact of your changes or track the lineage of each column. 

SQLdep Snowflake support is still in beta, so do not hesitate to contact support@keboola.com with any questions or bugs.

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1190754 2017-09-12T18:11:00Z 2017-09-12T18:24:07Z Snowflake Incident

We are investigating Snowflake partial outage. Queries are now queued.

We will inform about the progress.

UPDATE 11:21 AM PDT Snowflake warehouses are up and running. Queued queries are being processed. We will update the post when we'll have more information about incident cause from Snowflake engineering.

tag:status.keboola.com,2013:Post/1190370 2017-09-12T13:44:24Z 2017-09-12T13:44:24Z Week in Review -- September 11, 2017

New Features

  • Generic Extractor introduces some backward incompatible changes, but your current running configurations will remain unaffected. Learn more about Generic Extractor compatibility levels.
  • Generic Extractor supports stopping pagination using an attribute in the response, see more on the Limit Stop feature.

Minor Improvements

  • Processors configuration is now editable where eligible.
  • Transformation scrips and queries editors link to the documentation. 


  • Transformations UI refreshes properly on transformation or bucket delete or revert/rollback.
  • Transformation copy or restore does not delete transformation name and description.
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1189047 2017-09-05T19:49:04Z 2017-09-05T19:49:05Z Week in Review -- September 5, 2017

We've spent a lot of time on squashing bugs in GoodData Writer, Google Drive Writer, Docker Sandbox and Gelf logger and on several new features which are not yet ready. Besides, there are two nice things we can tell you about.

UI of the KBC is being revised for a few weeks now and you can see the first results in the unification of buttons for creating new resources. 

Database Writers now allow triggering `nullable` values for all columns at once.

Jakub Matějka
tag:status.keboola.com,2013:Post/1187287 2017-08-30T09:26:18Z 2017-08-30T09:26:18Z Week in Review -- August 28, 2017

New Features

  • List of deprecated components in the Project Overview:

    In the Project Overview you may now see a warning that you are using deprecated components in your project. 

    To prevent any inconveniences, please keep your components up-to-date. 

    Clicking on a component from the list will take you to a component configuration, from where you can start a migration process.


    • Fixed issue in Orchestrator, when you changed notification settings during orchestrators run, the settings will return to previous values after orchestrator has finished. (github issue in Czech language)

    Miroslav Čillík
    tag:status.keboola.com,2013:Post/1185565 2017-08-22T02:49:30Z 2017-08-23T12:28:00Z MySQL Transformations Failures

    We're encountering an increased number of application errors in MySQL transformations. We're looking into this issue and will update this post once we know more. This may also affect GoodData writer jobs.

    UPDATE Aug 21, 9:10pm PST

    The error seems to happens randomly on all Storage API async exports from tables stored on Snowflake. We have contacted Snowflake support to help us identify the root cause of the issue.

    You may encounter the error in any component that uses async exports from Snowflake or in any external integration exporting data from a Snowflake bucket. The error does not appear in Snowflake or Redshift transformations.

    UPDATE Aug 22, 9:45pm PST

    The error was not yet mitigated, Snowflake engineering team is looking into it. If you're encountering this problem, please contact us at support@keboola.com for a possible workaround.

    We're sorry for this inconvenience.

    UPDATE Aug 23, 5:24am PST

    Snowflake engineers has identified a potential fix for this issue but it is not deployed yet. Meanwhile we have found the workaround for the issue and we are updating the components. GoodData writer was fixed now, transformation and other components will be updated within few hours. 

    This issue is happening randomly and less than one percent of exports were affected.

    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1185367 2017-08-21T14:45:41Z 2017-08-21T14:45:42Z Week in Review -- August 21, 2017

    New Features

    • Database writers UI  - Disabled tables no longer load the whole content into the container when run and only first row of the table is loaded instead. This saves storage credits and decrease time of jobs run process. It works only for newly disabled tables, so old tables have to be redisabled (enable export -> disable export).
    • Python for transformations and custom science applications has been updated to version 3.6.2
    tag:status.keboola.com,2013:Post/1183436 2017-08-14T14:23:18Z 2017-08-14T14:23:18Z Week in Review -- August 14, 2017

    New Features

    • Ability to test component's image against live configurations (Read more in our documentation)
    • Markdown descriptions for tables and buckets are now available (see the images bellow)

    Fig. 1 Click "Describe the ..." to add description

    Fig. 2 Markdown is supported when describing buckets or tables.

    Description of buckets and tables can be added/edited in various places:

    • In bucket detail (Storage section),
    • in table detail (Storage section)
    • and anywhere in application where table preview is available


    • We finally turned off Legacy Routing (Original status here)
    • EOL of the synchronous exports has been announced and we'll turn it off on August 21st, 2017



    • Due to recent outages of Quay we started migration of our components to AWS ECR
    • We updated the Pigeon component to use the newer Storage API client and send files to File Uploads encrypted
    tag:status.keboola.com,2013:Post/1182343 2017-08-10T10:36:03Z 2017-08-12T16:02:03Z Snowflake Incident Investigation

    We were notified from Snowflake support about problem that could potentially affect Keboola Connection Snowflake Backed projects.

    The issue happened between 9:19pm UTC August 8th and 6:45pm August 9th UTC.

    There is a full alert from Snowflake below. We are investingating with them if we can find out if there are some affected KBC projects.

    We will keep you informed of any progress.

    UPDATE 2017-08-12 We have received list of projects which might be affected by this issue. In these projects one or more queries which might return wrong results were executed. We have notified these projects in KBC. If you need any assistance please contact support@keboola.com.

    Dear Snowflake Customer,

    We have identified a problem that could produce incorrect results in some queries that perform memory intensive join operations. It is unlikely that this issue impacts you; however, we wanted to make sure you were aware of the problem. The issue was isolated to the US-West region between 2:19pm Pacific time on Tuesday, August 8th and 11:45am Pacific time on Wednesday, August 9th. Queries performed in other regions or outside that timeframe were not impacted.

    Because of the nature of the issue, it is difficult for us to determine if any of your queries experienced the problem. However, we are able to perform further analysis on a query-by-query basis. If you have a question about whether a specific query was impacted, please submit a support request to Snowflake support via the Support Portal or via email to support@snowflake.net .

    We are very sorry about this issue and will do everything possible to help you resolve any problems this may have caused you.

    tag:status.keboola.com,2013:Post/1181990 2017-08-09T06:57:07Z 2017-08-09T07:11:18Z Components errors

    We observe components failures hosted on quay.io from 08:25  AM CET. Python Transformations, Facebook Extractor, Google Sheets Writer are between affected components.

    Quay.io outage is not reported yet. To avoid this issue in future we are moving all components to Keboola provided AWS repository.

    We will inform about the progress.

    9:09 AM CET RESOLVED Last error was seen at 08:53 AM CET. Quay.io has confirmed and fixed the issue.

    tag:status.keboola.com,2013:Post/1172622 2017-08-08T16:54:25Z 2017-08-08T16:54:25Z New Intercom Extractor

    Due to a change in the Intecom API authorization, the current extractor stopped working. It took us a while to go through the Intercom OAuth registration, but the result is here - the new Intercom Extractor. The current version is immediately deprecated, please configure and authorize the new extractor. You can start downloading your Intercom data in seconds.

    The full documentation is available here.

    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1181296 2017-08-07T11:55:54Z 2017-08-11T16:14:12Z End of life announcement for synchronous table exports

    The deprecated table synchronous exports will be shut down on 21st Aug 2017. 

    This API call was replaced by Asynchronous table export for full table unloads and Data Preview for small samples of table data.

    All Keboola Connection components use the new methods. As there are only a few projects currently using the deprecated method directly, these projects will be notified individually. For all other projects there is no action required.

    tag:status.keboola.com,2013:Post/1178856 2017-08-01T11:57:51Z 2017-08-01T11:57:51Z Week in Review - July, 31, 2017

    New Features

    • RStudio and Jupyter Sandboxes now support (and enforce) HTTPS connections.

    Updated Components

    Qlik and Looker writers were updated:

    • Improved and unified UI

    • Optional provisioned Redshift database


    • Fixed freezing of columns in Google Sheets Writer

    tag:status.keboola.com,2013:Post/1176830 2017-07-26T08:43:36Z 2017-07-26T08:43:37Z Week in Review -- July 25, 2017

    Last week's developments:

    Storage API

    • The API no longer trims leading or trailing white spaces from configuration descriptions or row names

    tag:status.keboola.com,2013:Post/1177171 2017-07-26T05:38:01Z 2017-07-26T05:47:18Z Job failures

    There were jobs failures between 4:30 AM - 5:30 AM CET. Failures were caused by low disk space of one of worker servers.

    We are sorry for this inconvenience and we're taking steps to mitigate this problem in the future.

    tag:status.keboola.com,2013:Post/1175096 2017-07-20T14:11:20Z 2017-07-20T14:11:20Z Deprecated Facebook Ads API v2.8 and below

    Facebook is deprecating all Marketing API versions prior to v2.9 on Wednesday, July 26, 2017 and recommends upgrading to version v2.9 immediately. Facebook Ads Extractor uses v2.8 by default for all newly created configurations but after July 26 all newly created configurations will have default api-version set to v2.9.

    The migration should only require changing the api-version parameter in all existing configurations of our Facebook Ads Extractor. However, since there may be some breaking changes, we strongly recommend to change the api-version in your configurations manually, review any possible changes and take the appropriate actions. For more details on the new version, please read Marketing API Changes in v2.9 in the Facebook API changlelog. Of those changes we'd like to highlight the deprecation of date_preset values which are replaced with new ones, e.g. last_3_days, replaced by last_3d .

    In other Facebook news, they have announced version v2.10 of Facebook API and Facebook Marketing API.

    tag:status.keboola.com,2013:Post/1175014 2017-07-19T10:33:54Z 2017-07-19T10:33:54Z Week in review -- July 19, 2017

    GoodData Writer

    • Upload Project job isn't executing reports anymore. This caused troubles on Vertica backend. You can add report execution API call into orchestrator if needed

    Python sandboxes

    • Updated to Jupyter 5.0.0 
    • Current directory is now */data/*

      Transformations (SQLdep)

      • Fixed issue with lineage on aliased cloumns

      Generic Extractor

      • Delimiter character can now be configured in Data Field, see documentation
      • Functions can now be used in Login Authentication in API Request, see documentation

      Storage API CLI

      • Added create-bucket command
      • It is now possible to run this tool as Docker container
      • Latest version is available on GitHub

      Storage API Client for Python

      • Alpha version of the client is available on GitHub.  
      • Call for contributions: any pythonistas out there want to contribute, please do so:) 
      Miroslav Čillík
      tag:status.keboola.com,2013:Post/1171109 2017-07-06T20:05:23Z 2017-07-06T20:05:24Z Storage Jobs Stuck in Processing State

      We have noticed an increased number of Storage jobs stuck in the processing state, in rare cases causing a complete halt and queuing of all new Storage jobs in a project.

      We have identified the root cause. Due to the Snowflake incident last night it seems some of the database transactions were still open and they blocked queries from consecutive jobs. We have terminated all orphaned transactions and jobs started processing and restarting. 

      Please note, that queued jobs are processed in random order. 

      We are sorry for this inconvenience.

      tag:status.keboola.com,2013:Post/1170920 2017-07-06T08:06:53Z 2017-07-06T08:06:54Z Sudden Jobs Failures on July 6th, 2017

      On July 6th, 2017, between 9:16-9:18am CET one of our internal databases was forced to update and restart by AWS. Result of this action was Application error failure of all running components jobs. The error may show up significantly later than the restart.  We recommend to review your orchestrations jobs and take action if needed. We are sorry for this inconvenience and we're taking steps to mitigate this problem in the future.

      tag:status.keboola.com,2013:Post/1170861 2017-07-06T03:04:52Z 2017-07-06T07:06:35Z Jobs queues overload (resolved)

      since 3:50am CET we are experiencing server queues overload. We are still investigating the issue and will inform about the progress.

      UPDATE 5:45am CET- We found the possible root of the cause, Snowflake queries are being unusually queued, furthermore we are unable to raise power or number of cluster of our Snowflake warehouses and we waiting for Snowflake Support findings.

      UPDATE  6:10am CET from Snowflake support: Engineering has started mitigation steps to address this issue, we will provide another update shortly.

      UPDATE 7:00am CET: Snowflake queries queues has got empty and everything looks to be back to normal.

      UPDATE RESOLVED 9:00am CET:  We confirm the issue has been resolved, the cause was the unexpected queueing of Snowflake queries. Snowflake confirmed and rolled back their latest release. However, as a consequence our waiting jobs reached backoff threshold and timeout, which resulted in orchestration failures. We recommend to review your project orchestrations jobs and take action if needed.

      tag:status.keboola.com,2013:Post/1168386 2017-06-30T09:19:50Z 2017-06-30T09:19:50Z Apifier extractor (Public Beta)

      We are pleased to announce the public beta release of the Apifier extractor. The Apifier extractor runs a web crawler and stores the results in Keboola storage. The crawler is created and configured in Apifier and can also be adjusted in extractor configuration. This way you can extract structured data from any website and import it directly do Keboola.

      Apifier already contains many community web crawlers that can be reused, so you can seamlessly download restaurant reviews from Yelp, hotels from TripAdvisor, or request the creation of a custom crawler for an additional fee.

      tag:status.keboola.com,2013:Post/1168113 2017-06-29T12:53:24Z 2017-06-29T12:53:24Z Fixed IP Address Ranges

      We have mitigated all issues which forced us to rollback fixed IP ranges announced few weeks ago.

      It is our pleasure to announce that as of today fixed IP ranges are back in production, all our outgoing network connections in the US region are using a pool of four static IP addresses. 

      This can help you meet your company security standards. To find out more, please visit the IP Addresses page in our documentation.


      Some of you were employing a very old concept where we performed an on-demand network source routing which allowed us to force the source IP under syrup-out.keboola.com. This was deprecated almost a year ago. 

      Today, we are also announcing that old source routing is deprecated and will be turned off at the end of this month. If you rely on source IP, please move all your existing firewall rules to our new addresses before July 30, 2017. Then you can remove our legacy (syrup-out.keboola.com) from your firewall. 

      After July 30, 2017 we'll still hold that IP address but it won't be used anymore.

      Martin Halamíček
      tag:status.keboola.com,2013:Post/1168459 2017-06-28T13:58:02Z 2017-06-28T13:58:02Z New Google Drive Writer

      As we announced earlier, we have decided to split the current Google Drive Writer into two separate components to simplify their usage:

      • Google Sheets Writer - as its name implies, is designed to upload tabular data from Storage to Google Sheets.
      • (New) Google Drive Writer - will handle uploading general files to Google Drive.

      Google Sheets Writer has already been available for some time and now we are proud to release the new version of Google Drive Writer.

      The Google Drive Writer is best for uploading general files into Drive. It allows you to create a new file on each run and therefore is ideal for backup purposes.


      • Advanced Input Mapping
        • choose which columns will be uploaded
        • filter data by date or by column values

      • Update existing files or create new files


      The existing Google Drive Writer will be deprecated soon and a migration tool will be made available to help you transfer your existing configurations to new Google Drive or Google Sheets Writer respectively.

      Don't hesitate to give us feedback or ask a question, write to support@keboola.com.

      Miroslav Čillík
      tag:status.keboola.com,2013:Post/1168092 2017-06-28T08:50:45Z 2017-06-28T08:50:45Z Week in Review -- June 28, 2017

      RStudio and Jupyter Sandboxes

      RStudio and Jupyter sandboxes are now in public beta, you can try them right now in your projects.
      You can now also see and extend your sandbox expiration.

      We are working on:

      • HTTPS Support.
      • Configurable (pay as you go) memory and disk limits.
      • Creating better UI and integration with transformations.
      • Allowing tables and files to be added to a running sandbox.

      Keboola Connection in EU

      We have launched Keboola Connection in a new region outside of the USA. You can now create your projects in the EU-Frankfurt region, please contact support@keboola.com if you are interested. This new region is now in beta.

      Minor improvements

      • Improved performance of PostgreSQL extractor.