Job failures

There were jobs failures between 4:30 AM - 5:30 AM CET. Failures were caused by low disk space of one of worker servers.

We are sorry for this inconvenience and we're taking steps to mitigate this problem in the future.

Deprecated Facebook Ads API v2.8 and below

Facebook is deprecating all Marketing API versions prior to v2.9 on Wednesday, July 26, 2017 and recommends upgrading to version v2.9 immediately. Facebook Ads Extractor uses v2.8 by default for all newly created configurations but after July 26 all newly created configurations will have default api-version set to v2.9.

The migration should only require changing the api-version parameter in all existing configurations of our Facebook Ads Extractor. However, since there may be some breaking changes, we strongly recommend to change the api-version in your configurations manually, review any possible changes and take the appropriate actions. For more details on the new version, please read Marketing API Changes in v2.9 in the Facebook API changlelog. Of those changes we'd like to highlight the deprecation of date_preset values which are replaced with new ones, e.g. last_3_days, replaced by last_3d .

In other Facebook news, they have announced version v2.10 of Facebook API and Facebook Marketing API.

Week in review -- July 19, 2017

GoodData Writer

  • Upload Project job isn't executing reports anymore. This caused troubles on Vertica backend. You can add report execution API call into orchestrator if needed

Python sandboxes

  • Updated to Jupyter 5.0.0 
  • Current directory is now */data/*

    Transformations (SQLdep)

    • Fixed issue with lineage on aliased cloumns

    Generic Extractor

    • Delimiter character can now be configured in Data Field, see documentation
    • Functions can now be used in Login Authentication in API Request, see documentation

    Storage API CLI

    • Added create-bucket command
    • It is now possible to run this tool as Docker container
    • Latest version is available on GitHub

    Storage API Client for Python

    • Alpha version of the client is available on GitHub.  
    • Call for contributions: any pythonistas out there want to contribute, please do so:) 

    Storage Jobs Stuck in Processing State

    We have noticed an increased number of Storage jobs stuck in the processing state, in rare cases causing a complete halt and queuing of all new Storage jobs in a project.

    We have identified the root cause. Due to the Snowflake incident last night it seems some of the database transactions were still open and they blocked queries from consecutive jobs. We have terminated all orphaned transactions and jobs started processing and restarting. 

    Please note, that queued jobs are processed in random order. 

    We are sorry for this inconvenience.

    Sudden Jobs Failures on July 6th, 2017

    On July 6th, 2017, between 9:16-9:18am CET one of our internal databases was forced to update and restart by AWS. Result of this action was Application error failure of all running components jobs. The error may show up significantly later than the restart.  We recommend to review your orchestrations jobs and take action if needed. We are sorry for this inconvenience and we're taking steps to mitigate this problem in the future.

    Jobs queues overload (resolved)

    since 3:50am CET we are experiencing server queues overload. We are still investigating the issue and will inform about the progress.

    UPDATE 5:45am CET- We found the possible root of the cause, Snowflake queries are being unusually queued, furthermore we are unable to raise power or number of cluster of our Snowflake warehouses and we waiting for Snowflake Support findings.

    UPDATE  6:10am CET from Snowflake support: Engineering has started mitigation steps to address this issue, we will provide another update shortly.

    UPDATE 7:00am CET: Snowflake queries queues has got empty and everything looks to be back to normal.

    UPDATE RESOLVED 9:00am CET:  We confirm the issue has been resolved, the cause was the unexpected queueing of Snowflake queries. Snowflake confirmed and rolled back their latest release. However, as a consequence our waiting jobs reached backoff threshold and timeout, which resulted in orchestration failures. We recommend to review your project orchestrations jobs and take action if needed.

    Apifier extractor (Public Beta)

    We are pleased to announce the public beta release of the Apifier extractor. The Apifier extractor runs a web crawler and stores the results in Keboola storage. The crawler is created and configured in Apifier and can also be adjusted in extractor configuration. This way you can extract structured data from any website and import it directly do Keboola.

    Apifier already contains many community web crawlers that can be reused, so you can seamlessly download restaurant reviews from Yelp, hotels from TripAdvisor, or request the creation of a custom crawler for an additional fee.

    Fixed IP Address Ranges

    We have mitigated all issues which forced us to rollback fixed IP ranges announced few weeks ago.

    It is our pleasure to announce that as of today fixed IP ranges are back in production, all our outgoing network connections in the US region are using a pool of four static IP addresses. 

    This can help you meet your company security standards. To find out more, please visit the IP Addresses page in our documentation.

    PLEASE NOTE

    Some of you were employing a very old concept where we performed an on-demand network source routing which allowed us to force the source IP under syrup-out.keboola.com. This was deprecated almost a year ago. 

    Today, we are also announcing that old source routing is deprecated and will be turned off at the end of this month. If you rely on source IP, please move all your existing firewall rules to our new addresses before July 30, 2017. Then you can remove our legacy 54.85.151.211 (syrup-out.keboola.com) from your firewall. 

    After July 30, 2017 we'll still hold that IP address but it won't be used anymore.

    New Google Drive Writer

    As we announced earlier, we have decided to split the current Google Drive Writer into two separate components to simplify their usage:

    • Google Sheets Writer - as its name implies, is designed to upload tabular data from Storage to Google Sheets.
    • (New) Google Drive Writer - will handle uploading general files to Google Drive.

    Google Sheets Writer has already been available for some time and now we are proud to release the new version of Google Drive Writer.

    The Google Drive Writer is best for uploading general files into Drive. It allows you to create a new file on each run and therefore is ideal for backup purposes.

    Features

    • Advanced Input Mapping
      • choose which columns will be uploaded
      • filter data by date or by column values

    • Update existing files or create new files


    Migration

    The existing Google Drive Writer will be deprecated soon and a migration tool will be made available to help you transfer your existing configurations to new Google Drive or Google Sheets Writer respectively.

    Don't hesitate to give us feedback or ask a question, write to support@keboola.com.

    Week in Review -- June 28, 2017

    RStudio and Jupyter Sandboxes

    RStudio and Jupyter sandboxes are now in public beta, you can try them right now in your projects.
    You can now also see and extend your sandbox expiration.

    We are working on:

    • HTTPS Support.
    • Configurable (pay as you go) memory and disk limits.
    • Creating better UI and integration with transformations.
    • Allowing tables and files to be added to a running sandbox.

    Keboola Connection in EU

    We have launched Keboola Connection in a new region outside of the USA. You can now create your projects in the EU-Frankfurt region, please contact support@keboola.com if you are interested. This new region is now in beta.

    Minor improvements

    • Improved performance of PostgreSQL extractor.