Snowflake Outage (US Region)

We have encountered an increased number of Snowflake connection errors between 04:11 UTC and 04:21 UTC. This may have caused failed storage and component jobs.

Furthermore Snowflake announced possible SQL query failures between 02:24 UTC and 03:30 UTC. 

We're sorry for this inconvenience, all systems are operational now. 

Snowflake connection issues (US region)

We're experiencing random connection errors to Snowflake started on Sep 12, 22:08 UTC.

These errors caused a few transformations failures. We're investigating the issue and we'll update this post with our findings.

Update, Sep 14, 07:15 UTC

These rare errors are still occuring.

Update, Sep 14, 13:07 UTC

Snowflake confirms the issue is on their side and are investigating it.

Investigating incident (US region)

We're currently investigating an incident in the US region that may have caused component job and orchestration failures today, Tuesday, Sep 11, between 02:30 UTC and 06:30 UTC.

Update 08:30 UTC

The DB storing job locks (preventing jobs from running multiple times on multiple workers) was restarted at 02:33 UTC. All connections were terminated and all component jobs and transformations running at that time were disconnected. This could lead to any or both of the following situations:

  • Failure at any time later during the job execution
  • A new parallel job execution

Any jobs started after the DB restart were not affected by this issue.

We apologize for this inconvenience. We're planning an infrastructure change to prevent such huge impact during similar situations. 

If you have any further questions, please use the support button in your project.

Week in Review -- August 22, 2018

Updated Components

Storage

  • Added Configuration Row detail API resource

Slack Extractor

  • Fixed pagination. The extractor was only downloading 1000 records from a channel history.
    Existing configuration of the extractor needs to be re-saved to apply this fix.


New Features

  • When creating new API tokens, you can now choose Orchestrator from the Custom Access components dropdown



Minor Improvements

  • Added Incremental label to Output Mapping

  • Storage API and Manage API tokens are now generated from alphanumeric characters instead of hexadecimal

  • Added Documentation badge to component detail. Additional external documentation is available for component marked with this badge. 

  • Filter expression operators were renamed to be more intuitive

    EQ (equals) -> IN
    NE (not equal) -> NOT IN

  • It's no longer necessary to click the <-Back button when adding tasks to Orchestration. The New Task modal will reset its state and after reopening it shows the list of components. Previously, after reopening, it showed a list of configurations of the previously selected component

September, October 2018 Maintenance Schedule

US Region

Sat, September 22, 2018

A maintenance will start at 6:00 am UTC. We'll be migrating metadata storage of Storage Files during the maintenance.
The maintenance won't be system wide. It will be performed for each project in connection.keboola.com separately and it should take few minutes for each project depending on the number of project's files. It should not affect project's running jobs, these will be queued during the short project maintenance. Only projects created before August 16, 2018 will be affected by this migration.

Sat, October 6, 2018

A maintenance will start at 6:00 am UTC and should take less than two hours. We'll be migrating metadata of Jobs during the maintenance. All jobs will be stopped during the maintenance and should be restarted after the maintenance. Orchestrations scheduled during the maintenance period won't be started.

All projects in connection.keboola.com will be inaccessible during the maintenance.

EU Region

Sat, October 6, 2018

A maintenance will start at 6:00 am UTC and should take less than two hours. We'll be migrating metadata of Jobs during the maintenance. All jobs will be stopped during the maintenance and should be restarted after the maintenance. Orchestrations scheduled during the maintenance period won't be started.

All projects in connection.eu-central-1.keboola.com will be inaccessible during the maintenance.

Week in Review -- August 2, 2018

Improvements

  • Jobs and Storage Files search syntax hints
  • Configurations quick search on component detail page

Updated components

Developers

Storage Form Uploads Deprecation

We are deprecating rarely used Storage HTTP Form Upload. This type of upload is supported only in US KBC region and is using legacy AWS Signature Version 2. The upload can be replaced by direct S3 upload using AWS SDK or by Importer Service for POCs or smaller files. We’ll notify projects using this type of upload to help them migrate.