Storage Jobs Errors - US region

We're experiencing a high number of errors in Storage Jobs in the US region. The errors are caused by invalid SSL connection to Snowflake and we're investigating the issue.

Update 01:02 am CET: Snowflake issues an official statement.

Update 07:00 am CET: Snowflake updated an issue. We are working on a workaround.

Update 07:56 am CET: Issue is resolved. Snowflake moved the Keboola account to a new certificate provider. Now we are processing backlog of jobs.

Update 09:20 am CET: Backlog is cleared. All systems are operating normally.

Week in Review -- November 14, 2018

Revised Project Invitations

Inviting users to KBC projects has been reworked. Previously, users were directly added to the project via Users & Settings -> Users menu. Now, when adding a new user to the project, an invitation email is sent to the user.

After clicking on the link in the email, the user is redirected to the login screen, where pending invitations to projects are displayed.

The invited user can accept or decline the pending invitations in the Account Settings.

See the docs for more details.

New Features

  • We have added some predefined searches to the job list. See the docs for a more detailed description of the search queries.


Minor Improvements

  • Jobs related to Python/R sandboxes now show the proper link to a configuration:

  • Improved description of components in events

  • Configurations in trash are now sorted by the date of deletion:

  • Component configuration state is reset on copy:
If you're making a copy of a component configuration (e.g. Zendesk Extractor), the state of the configuration is cleared for the new copy. For example, Zendesk Extractor downloads all tickets from Zendesk on the first run and stores the date and time into its state. On the subsequent runs the Extractor only downloads the tickets created after that time. Now, the new copy of this configuration will also download all tickets on its first run as there is nothing in its state.


Week in Review -- November 1, 2018

UI Improvements and fixes

  • Dialog for adding a new task to orchestration now contains configuration description 
  • Every job detail events log now contains component info
  • Improved orchestration notifications ui
  • Improved brief summary of orchestration tasks
  • Unified view of data mappings in the job detail page with respect to the running component as follows: 
    • Input - represents all the data flowing from the storage into the component - typically this would apply for transformation or writer component type. 
    • Output - represents all the data flowing from the component into the storage, typically this would apply for extractor component type.

  • When creating a new extractor in Developer portal  the default bucket is automatically turned on
  • Improved naming of new phase and phase names capitalisation




Orchestrations slow down in EU

November 2, between 2:43am CET and 3:14 CET we experienced Snowflake service degradation in EU region that led to slow down of the orchestrations. The issue has been resolved. We apologize for the inconvenience caused.

Week in Review -- October 24, 2018

Complexity Badges

There are components in KBC which you can setup in just a few clicks with only basic knowledge. But there are also components which require you to have deep knowledge about how it works. For this reason we have introduced Complexity Badges to be able distinguish between these types of components even before you start setting them up.

Not all components have assigned Complexity Badges yet, but we are continuously working on updating the components catalog.

Minor Improvements

  • When you are retrying failed orchestrations it will now continue from where it failed by default.
  • Transformation backends have new labels with icons.

Components

Restbox shutdown

We would like to remind you that the Restbox component was deprecated in July 2018 and will be completely shut down on November 1, 2018.

If you still have some Restbox configurations in your projects, please migrate them to one of following replacements:


Table primary key consistency incident

We have identified a bug in the primary key implementation in Storage which could lead to improper data deduplication. Only a very limited number of tables is affected by this bug – 7 tables in all KBC projects in all stacks. We'll be contacting owners of the affected projects soon to help fixing the affected tables.

Root Cause

The deduplication stopped working when a column used in a compound primary key was deleted.

During this operation, the information about the whole primary key was unknowingly dropped in the Snowflake backend and this was not propagated correctly to our Storage metadata that still contained the primary key (minus the deleted column). In Snowflake, commands such as ALTER TABLE ... DROP COLUMN ... immediately drop the whole primary key if it’s dropping a column of a compound primary key. The deduplication process retrieves primary key information from the DESCRIBE TABLE ... Snowflake command which shows no primary key in the affected tables, but our metadata still incorrectly shows that a primary key is set.

We are implementing a fix that will store and retrieve primary keys from a single source.

Operating with Primary Key Columns

  • Deleting columns which are part of the table primary key is no longer supported in Storage.
  • To delete a primary key column please drop the primary key first.
  • To change primary key of a table you will need to first remove the primary key and then set it again.

Week in Review -- October 08, 2018

Updated Components

  • MS SQL Server extractor has had a driver update.  Advanced queries now have slightly different behaviour so it is recommended to switch to simple table and column selected configurations.  Where that is not possible, please see the documentation for further help.
  • Snowflake writer - minor bugfixes and enhancements
    • primary key is added when creating a table
    • nonexistent schema/warehouse validation
    • warehouse detected correctly for users with different login name and user name
  • Database extractors - column datatypes are now shown in UI

Minor Improvements

    • You can now change the sharing type of a shared bucket in the storage console.

    Stalled jobs in EU region

    Around 1:30am CEST one of the job worker instances stopped processing assigned jobs. This could have lead to jobs being stuck in the processing state for a long time without any activity.

    At 11:15am CEST the worker instance was terminated and all unfinished jobs started processing on other instances.

    We're sorry for this inconvenience.

    Failed and delayed jobs in EU region

    Database storing locks was restarted at 03:49 UTC which caused the job failures. Also some of the jobs were queued after this failure.

    The backlog of all jobs was cleared at 06:15 UTC. The system is fully operational now. We're working on infrastructure changes which should prevent similar issues.