Delayed orchestrator in EU region

[13.07.2018 10:45 CET]

We're currently investigating delayed orchestrator launches in EU region. 

[13.07.2018 11:13 CET]

There was a problem with one of our nodes that delayed and in some cases even made it impossible to launch orchestrations. All affected customers are now moved to a new node and orchestrator operates normally. We're monitoring the performance of other nodes to make sure everything works. 

We also notices somewhat similar behavior in one of the nodes in US region. We retired the node as well as a preventive measure. So far we have no information of any customer impacted by this issue in US region. 

New version of Sklik Extractor

Our Sklik Extractor gets a new version updated to use Sklik API version Drak. It allows you to download data about any resource supporting methods createReport and readReport (see Seznam's blog post for more information). 

The new API uses different logic for structuring the data so unfortunately, we can't offer automatic migration. See extractor's documentation for more information about its configuration.

The old extractor is deprecated and will be shut down by the end of August.

Week In Review -- July 9, 2018

New Components

Updated Components

  • The GoodData Reports extractor now accepts an optional host parameter so it will work in the EU or any custom domain.
  • The Email Attachments extractor has been updated with the following improvements
    • It now strictly enforces the one text file attachment per email rule
    • It ignores any non-textual attachments (signature images etc...)
    • It can process multiple emails simultaneously
  • The Gmail Extractor no longer has a limit of 65k for fields (backend storage limits still apply)
  • The HTTP Extractor now accepts a "maxRedirects" parameter to tell it how many times to retry when it receives an http redirect status code. 

Minor Improvements

  • R has been updated to version 3.5.0 (for transformations and RStudio sandboxes)



Processing delays for GoodData Writer jobs in the EU region

[02.07.2018 8:06 CET]

We're experiencing some delays in the processing of GoodData Writer jobs in the EU region, and are currently investigating the cause of the issue.

UPDATE [02.07.2018 10:06 CET] 

The tasks are queued in GoodData EU platform. We are investigation this issue with GoodData support.

UPDATE [02.07.2018 15:27 CET] 

The issue has been resolved, and all GoodData Writer tasks in the EU region are working normally again.


Week in Review -- June 26, 2018

Updated components

  • FlexiBee extractor has configurable hostname
  • MySQL Writer uses temporary tables for incremental loading, so DROP privilege can be replaced by CREATE TEMPORARY TABLES privilege for incremental loading

Table and column deletion improvements

This applies to table aliases and also to shared buckets (even if tables are already linked to project).

  • Table can be deleted also with its aliases

  • Same applies for columns - column deletion will be propagated also to aliased tables - with two exceptions:
    • not if column is used as filter in alias
    • not if column is in alias with aliasColumnsAutosync turned off

Other changes

  • There's new information on API Tokens page - if token has permission to manage other tokens
  • Output Mapping settings are no longer collapsed and you can edit them with few UI interactions

Snowflake Outage - US Region

We are experiencing Snowflake partial outage which started at 18:11 CEST.

UPDATE, 19:33 CEST: It looks like the problem is solved. We also replaced one of our servers where we detected more connection issues.

UPDATE, 05:00 CEST: The errors began to appear again. We are working with Snowflake support on fix.

UPDATE, 08:46 CEST: Issue was resolved. The root cause was invalid cache of Online Certificate Status Protocol (OCSP)  utilised by Snowflake drivers. We are further investigating the issue with Snowflake support.


Week in Review -- June 19, 2018

Updated Components

Minor Improvements

  • Setting up MFA in Keboola Connection can display the code in plain text where QR is not available.
  • We have changed Docker storage driver for Docker Runner job workers. We hope this will stabilize docker response times and minimize startup and shutdown overhead. This will be most significant when running short jobs.
  • Editing components in Keboola Developer Portal got a facelift.

Week in Review -- June 6, 2018

New Features

Orchestration Description
  • You can describe your Orchestrations now. This can be useful when you collaborate with your colleagues or want to quickly find what you need among dozens of Orchestrations.


Orchestration Versioning
  • Also, Orchestrations just got their internal versioning system (joining Extractors, Writers and Applications). At this stage, the versioning is read-only, so you can compare two versions. Rollbacks and revision copies will be added soon.


Minor Improvements

MySQL Writer
  • Improved performance of incremental load. Also, the writer now checks if the primary keys in its configuration match with those in the target database table.


Fixes

  • Google Calendar Extractor is now available also in the EU region.

Docker Jobs Application Errors

Unfortunately tonight there were a few more unexpected Application errors and delayed or longer running jobs between 1:25am–4:55am CEST (4:25pm–7:55pm) in the US region.

We have experimented with different storage drives (swapping from SSD to throughput optimized HDD) which lead to initial issues with building Custom Science apps. Attempts to provision further resources lead to too many running jobs at once (you could see "SQLSTATE[HY000] [1040] Too many connections" in the failed app events) and removing some of the additional resources could have yielded some other Application errors too. 

Currently we're running SSD drives again with enough resources to process all workloads. Please restart your failed jobs.

We hope we'll be able to stabilize this whole unfortunate situation as soon as possible and we're very sorry for inconvenience.