Skipped Orchestrations at 0:00 UTC

A few Orchestrations which were scheduled to run at exactly midnight UTC may not have started due to an issue with the Orchestration Scheduler. The issue arose on the 15th of September and has now been resolved.

If you have orchestrations which are scheduled to run at 0:00 UTC, please check that these orchestrations have run correctly.


Sincere apologies for the inconvenience.

Snowflake Incident

We are investigating Snowflake partial outage. Queries are now queued.

We will inform about the progress.

UPDATE 11:21 AM PDT Snowflake warehouses are up and running. Queued queries are being processed. We will update the post when we'll have more information about incident cause from Snowflake engineering.

Week in Review -- September 11, 2017

New Features

  • Generic Extractor introduces some backward incompatible changes, but your current running configurations will remain unaffected. Learn more about Generic Extractor compatibility levels.
  • Generic Extractor supports stopping pagination using an attribute in the response, see more on the Limit Stop feature.

Minor Improvements

  • Processors configuration is now editable where eligible.
  • Transformation scrips and queries editors link to the documentation. 

Fixes

  • Transformations UI refreshes properly on transformation or bucket delete or revert/rollback.
  • Transformation copy or restore does not delete transformation name and description.

Week in Review -- September 5, 2017

We've spent a lot of time on squashing bugs in GoodData Writer, Google Drive Writer, Docker Sandbox and Gelf logger and on several new features which are not yet ready. Besides, there are two nice things we can tell you about.

UI of the KBC is being revised for a few weeks now and you can see the first results in the unification of buttons for creating new resources. 

Database Writers now allow triggering `nullable` values for all columns at once.

Week in Review -- August 28, 2017

New Features

  • List of deprecated components in the Project Overview:

    In the Project Overview you may now see a warning that you are using deprecated components in your project. 

    To prevent any inconveniences, please keep your components up-to-date. 

    Clicking on a component from the list will take you to a component configuration, from where you can start a migration process.

    Fixes

    • Fixed issue in Orchestrator, when you changed notification settings during orchestrators run, the settings will return to previous values after orchestrator has finished. (github issue in Czech language)

    MySQL Transformations Failures

    We're encountering an increased number of application errors in MySQL transformations. We're looking into this issue and will update this post once we know more. This may also affect GoodData writer jobs.

    UPDATE Aug 21, 9:10pm PST

    The error seems to happens randomly on all Storage API async exports from tables stored on Snowflake. We have contacted Snowflake support to help us identify the root cause of the issue.

    You may encounter the error in any component that uses async exports from Snowflake or in any external integration exporting data from a Snowflake bucket. The error does not appear in Snowflake or Redshift transformations.

    UPDATE Aug 22, 9:45pm PST

    The error was not yet mitigated, Snowflake engineering team is looking into it. If you're encountering this problem, please contact us at support@keboola.com for a possible workaround.

    We're sorry for this inconvenience.

    UPDATE Aug 23, 5:24am PST

    Snowflake engineers has identified a potential fix for this issue but it is not deployed yet. Meanwhile we have found the workaround for the issue and we are updating the components. GoodData writer was fixed now, transformation and other components will be updated within few hours. 

    This issue is happening randomly and less than one percent of exports were affected.

    Week in Review -- August 21, 2017

    New Features

    • Database writers UI  - Disabled tables no longer load the whole content into the container when run and only first row of the table is loaded instead. This saves storage credits and decrease time of jobs run process. It works only for newly disabled tables, so old tables have to be redisabled (enable export -> disable export).
    • Python for transformations and custom science applications has been updated to version 3.6.2

    Week in Review -- August 14, 2017

    New Features

    • Ability to test component's image against live configurations (Read more in our documentation)
    • Markdown descriptions for tables and buckets are now available (see the images bellow)

    Fig. 1 Click "Describe the ..." to add description

    Fig. 2 Markdown is supported when describing buckets or tables.

    Description of buckets and tables can be added/edited in various places:

    • In bucket detail (Storage section),
    • in table detail (Storage section)
    • and anywhere in application where table preview is available

    Deprecation

    • We finally turned off Legacy Routing (Original status here)
    • EOL of the synchronous exports has been announced and we'll turn it off on August 21st, 2017

    Tutorials

    Fixes

    • Due to recent outages of Quay we started migration of our components to AWS ECR
    • We updated the Pigeon component to use the newer Storage API client and send files to File Uploads encrypted