tag:status.keboola.com,2013:/posts Keboola Status 2018-08-07T16:47:04Z Keboola Connection "Data Framework" tag:status.keboola.com,2013:Post/1309897 2018-08-07T12:28:19Z 2018-08-07T16:47:04Z Jupyter and RStudio Sandboxes are not starting in US Region

2:27pm CEST: We're investigating the issue.

5:34pm CEST: New sandboxes are now running correctly. If you can't connect to an existing sandbox, please recreate it. We are deeply sorry for the trouble and frustration.

]]>
Erik Žigo
tag:status.keboola.com,2013:Post/1308578 2018-08-02T09:44:29Z 2018-08-02T09:44:29Z Week in Review -- August 2, 2018

Improvements

  • Jobs and Storage Files search syntax hints
  • Configurations quick search on component detail page

Updated components

Developers

Storage Form Uploads Deprecation

We are deprecating rarely used Storage HTTP Form Upload. This type of upload is supported only in US KBC region and is using legacy AWS Signature Version 2. The upload can be replaced by direct S3 upload using AWS SDK or by Importer Service for POCs or smaller files. We’ll notify projects using this type of upload to help them migrate.

]]>
tag:status.keboola.com,2013:Post/1306482 2018-07-27T22:15:26Z 2018-07-27T22:15:27Z Jupyter and R Studio sandboxes start timeouts

We are investigating issues with expired certificates for Jupyter and R Studio (beta) sandboxes.

UPDATE 11:30 CEST  - New certificates were deployed and issue is resolved. Existing sandboxes have to be created again.

]]>
tag:status.keboola.com,2013:Post/1306004 2018-07-24T17:35:45Z 2018-07-24T17:35:45Z Failing Create Manifest Processor

Between 4:00 pm CEST and 7:30 pm CEST the new version of Create Manifest processor was failing with an application exception. This change was rolled back and the processor is functional now. 

We're sorry for this inconvenience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1305913 2018-07-24T12:01:23Z 2018-07-24T12:01:23Z Week In Review -- July 24, 2018

New Components

Updated Components

  • Deprecated components cannot create new versions by copying from older versions.
  • Improved autocomplete in output mapping across components.
  • New release of Oracle extractor with massive performance increase.
]]>
tag:status.keboola.com,2013:Post/1304679 2018-07-20T07:45:29Z 2018-07-21T08:35:04Z EU Region Maintenance Announcement

A maintenance will take place on Saturday, July 21st, 2018 from 8:00 am CEST and should take less than two hours.

We will be upgrading metadata databases.

All projects in connection.eu-central-1.keboola.com will be inaccessible during the maintenance.

- UPDATE 08:00 CEST - Scheduled maintenance is currently in progress.

- UPDATE 10:10 CEST - The maintenance has been performed. We are starting job workers and verifying that all systems are functioning normally.

- UPDATE 10:21 CEST - All workers are running and jobs backlog is processed.

- UPDATE 10:31 CEST - The maintenance has been completed. Thank you for your patience.

]]>
tag:status.keboola.com,2013:Post/1304300 2018-07-19T07:07:27Z 2018-07-19T07:07:27Z Errors in EU region

Few jobs have failed between 2:30 am and 04:51 am in EU region. We are investigating the issue.

]]>
tag:status.keboola.com,2013:Post/1301335 2018-07-16T07:04:05Z 2018-07-16T07:04:05Z Restbox Component Deprecation Announcement

The time has come and the Restbox component as we know it is now being deprecated. But do not worry, its functionality will not disappear.

The Restbox component was once a great tool to collect data from various sources that were not complex enough to deserve a dedicated extractor (or writer) and shared similar processing aspects, such as decompression, audit tools, and CSV formatting. This has now been broken up into multiple separate components and processors to achieve or exceed the required functionality.

Today we have the following components to replace the Restbox component:

Detailed documentation for most of them can be found at help.keboola.com.

The Restbox component is now available in the US region only, and, as of this announcement, no new configurations can be created. The component will be supported until November 1, 2018 when it will be finally shut down. Please migrate your configurations to one of the above-mentioned components. 

If you have any trouble migrating your configurations, please do not hesitate to contact us using the support button in your project.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1302722 2018-07-13T08:47:42Z 2018-07-13T09:28:25Z Delayed orchestrator in EU region

[13.07.2018 10:45 CET]

We're currently investigating delayed orchestrator launches in EU region. 

[13.07.2018 11:13 CET]

There was a problem with one of our nodes that delayed and in some cases even made it impossible to launch orchestrations. All affected customers are now moved to a new node and orchestrator operates normally. We're monitoring the performance of other nodes to make sure everything works. 

We also notices somewhat similar behavior in one of the nodes in US region. We retired the node as well as a preventive measure. So far we have no information of any customer impacted by this issue in US region. 

]]>
tag:status.keboola.com,2013:Post/1302077 2018-07-11T15:24:53Z 2018-07-11T15:24:53Z New version of Sklik Extractor

Our Sklik Extractor gets a new version updated to use Sklik API version Drak. It allows you to download data about any resource supporting methods createReport and readReport (see Seznam's blog post for more information). 

The new API uses different logic for structuring the data so unfortunately, we can't offer automatic migration. See extractor's documentation for more information about its configuration.

The old extractor is deprecated and will be shut down by the end of August.

]]>
tag:status.keboola.com,2013:Post/1301241 2018-07-09T09:29:55Z 2018-07-09T09:29:55Z Week In Review -- July 9, 2018

New Components

Updated Components

  • The GoodData Reports extractor now accepts an optional host parameter so it will work in the EU or any custom domain.
  • The Email Attachments extractor has been updated with the following improvements
    • It now strictly enforces the one text file attachment per email rule
    • It ignores any non-textual attachments (signature images etc...)
    • It can process multiple emails simultaneously
  • The Gmail Extractor no longer has a limit of 65k for fields (backend storage limits still apply)
  • The HTTP Extractor now accepts a "maxRedirects" parameter to tell it how many times to retry when it receives an http redirect status code. 

Minor Improvements

  • R has been updated to version 3.5.0 (for transformations and RStudio sandboxes)



]]>
tag:status.keboola.com,2013:Post/1299289 2018-07-03T05:26:10Z 2018-07-03T05:33:22Z Snowflake Transformations Errors in US Region

Between 5:00 am and 7:21 am CEST we experienced Snowflake connection errors for some transformations in the US region. We are investigating the issue and will work on a remedy with Snowflake team.

]]>
tag:status.keboola.com,2013:Post/1299028 2018-07-02T07:23:26Z 2018-07-02T13:37:10Z Processing delays for GoodData Writer jobs in the EU region

[02.07.2018 8:06 CET]

We're experiencing some delays in the processing of GoodData Writer jobs in the EU region, and are currently investigating the cause of the issue.

UPDATE [02.07.2018 10:06 CET] 

The tasks are queued in GoodData EU platform. We are investigation this issue with GoodData support.

UPDATE [02.07.2018 15:27 CET] 

The issue has been resolved, and all GoodData Writer tasks in the EU region are working normally again.


]]>
tag:status.keboola.com,2013:Post/1297637 2018-06-28T20:37:36Z 2018-06-28T20:37:36Z Week in Review -- June 26, 2018

Updated components

  • FlexiBee extractor has configurable hostname
  • MySQL Writer uses temporary tables for incremental loading, so DROP privilege can be replaced by CREATE TEMPORARY TABLES privilege for incremental loading

Table and column deletion improvements

This applies to table aliases and also to shared buckets (even if tables are already linked to project).

  • Table can be deleted also with its aliases

  • Same applies for columns - column deletion will be propagated also to aliased tables - with two exceptions:
    • not if column is used as filter in alias
    • not if column is in alias with aliasColumnsAutosync turned off

Other changes

  • There's new information on API Tokens page - if token has permission to manage other tokens
  • Output Mapping settings are no longer collapsed and you can edit them with few UI interactions
]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1297081 2018-06-25T16:38:06Z 2018-06-26T07:52:16Z Snowflake Outage - US Region

We are experiencing Snowflake partial outage which started at 18:11 CEST.

UPDATE, 19:33 CEST: It looks like the problem is solved. We also replaced one of our servers where we detected more connection issues.

UPDATE, 05:00 CEST: The errors began to appear again. We are working with Snowflake support on fix.

UPDATE, 08:46 CEST: Issue was resolved. The root cause was invalid cache of Online Certificate Status Protocol (OCSP)  utilised by Snowflake drivers. We are further investigating the issue with Snowflake support.


]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1295326 2018-06-19T11:41:53Z 2018-06-19T11:41:53Z Week in Review -- June 19, 2018

Updated Components

Minor Improvements

  • Setting up MFA in Keboola Connection can display the code in plain text where QR is not available.
  • We have changed Docker storage driver for Docker Runner job workers. We hope this will stabilize docker response times and minimize startup and shutdown overhead. This will be most significant when running short jobs.
  • Editing components in Keboola Developer Portal got a facelift.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1293304 2018-06-13T08:04:18Z 2018-06-13T08:21:48Z Week in Review -- June 6, 2018

New Features

Orchestration Description
  • You can describe your Orchestrations now. This can be useful when you collaborate with your colleagues or want to quickly find what you need among dozens of Orchestrations.


Orchestration Versioning
  • Also, Orchestrations just got their internal versioning system (joining Extractors, Writers and Applications). At this stage, the versioning is read-only, so you can compare two versions. Rollbacks and revision copies will be added soon.


Minor Improvements

MySQL Writer
  • Improved performance of incremental load. Also, the writer now checks if the primary keys in its configuration match with those in the target database table.


Fixes

  • Google Calendar Extractor is now available also in the EU region.
]]>
tag:status.keboola.com,2013:Post/1291856 2018-06-08T03:04:21Z 2018-06-08T07:29:27Z Docker Jobs Application Errors

Unfortunately tonight there were a few more unexpected Application errors and delayed or longer running jobs between 1:25am–4:55am CEST (4:25pm–7:55pm) in the US region.

We have experimented with different storage drives (swapping from SSD to throughput optimized HDD) which lead to initial issues with building Custom Science apps. Attempts to provision further resources lead to too many running jobs at once (you could see "SQLSTATE[HY000] [1040] Too many connections" in the failed app events) and removing some of the additional resources could have yielded some other Application errors too. 

Currently we're running SSD drives again with enough resources to process all workloads. Please restart your failed jobs.

We hope we'll be able to stabilize this whole unfortunate situation as soon as possible and we're very sorry for inconvenience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1291561 2018-06-07T06:16:23Z 2018-06-08T07:29:37Z Docker Jobs Application Errors

Unfortunately we we're unable to find a fix for yesterday's failures, so on Thursday June 7th between 3:49am CEST and 7:38am CEST (1:49am–5:38am UTC, 6:49pm–10:38pm PT) there was an increased application error rate on our Docker host instances in the US region.

The servers are now stabilized and it is safe to restart the failed jobs.

We're looking into this issue. We have started additional instances to help with the load and we'll be looking into the HW architecture of the instances to help us figure out what causes the issue. Meanwhile we'll try to implement a retry on such failed jobs.

We're sorry for this inconvenience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1291256 2018-06-06T07:19:46Z 2018-06-06T07:19:46Z Docker Jobs Application Errors

On June 6th between 2:15am CEST and 2:35am CEST (5:15pm PT and 5:35pm PT, 12:15am UTC and 12:35am UTC) there was an increased rate of application errors on one of our Docker host instances in the US region. The instance is now fully operational and the jobs are safe to restart.

Furthermore one of our EU region Docker host instances went down at 6:56am CEST and caused a few unexpected application errors. There is a new one in place, we recommend restarting any failed jobs.

We're sorry for this inconvenience, we're working on preventing these errors in the future.


]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1288393 2018-06-01T06:44:57Z 2018-06-01T06:44:58Z Week in Review -- May 28, 2018

New Components

OneDrive Extractor (Beta)
  • You can now download your documents from OneDrive. The component was developed by Jakub Bartel.

Google Big Query Writer (Beta)

  • Write data from KBC to Google Big Query with our new writer. The component was developed by us, Keboola s.r.o. 


Updated Components

Currency Extractor

  • Now offers exchange rates of GBP.

Oracle Extractor

  • Now supports exporting columns of the LOB datatype family.

Gmail Attachments Extractor

  • Added support for processors, see example.

Create Manifest Processor 

  • Will test, if all the slices of a CSV file have the same header columns.

Flatten Folders Processor 

  • Throws a user exception when the flattened filename is too long (longer than 255 characters).
  • Has now configurable flatten strategies. Added `hash-sha256` strategy, which solves an issue with 255 characters filename limit in default `concat` strategy.


New Features

  • Component badges.
  • Validate your SQL in transformations with new Validate feature, provided by SQLDep.

  • MySQL extractor now supports incremental fetching. You can extract just the most recent records from a database table and write them incrementally into Storage.


    Minor Improvements

    • Show more / show less button added to the list of inputs and outputs of a job. It now shows all the tables and the view is more compact.

    ]]>
    Miroslav Čillík
    tag:status.keboola.com,2013:Post/1289721 2018-06-01T04:57:25Z 2018-06-01T06:34:21Z GoodData Writer Issues

    Today between 2:30 and 7:00 CEST we experienced issues with GoodData Writer. Ironically, it failed to connect to a third-party service for utilization monitoring. The problem was fixed so there should be no other job failure. We are going to inspect the extent of the damage.

    ]]>
    tag:status.keboola.com,2013:Post/1282404 2018-05-16T17:11:17Z 2018-05-16T17:29:13Z Week in Review -- May 16, 2018

    Core

    Components

    Bugfixes

    • Component configuration state is not updated in case of attached processor failure. e.g in case of AWS S3 extractor and New Files Only option in leaves files as unprocessed in case of processor failure so the files can be processed again until the whole pipeline of processors is executed successfully.
    • MSSQL Writer - fixed support of unicode characters
    • Google Sheets Writer - fixed writing of large tables without performance issues
    • When specifying transformation output mapping, the bucket name is automatically webalized as being typed

    Developers

    We are happy to introduce the first version of Keboola Storage API Javascript client.


    ]]>
    tag:status.keboola.com,2013:Post/1283926 2018-05-15T21:41:54Z 2018-05-16T06:44:16Z SQL Server Extractor Connection Issues

    Between 17:00 May 15, 2018 and 8:00 on May 16, 2018 CET we experienced issues with the SQL Server database extractor. We recommend to review your orchestrations and take appropriate actions if needed.

    If you were affected by this, please accept our sincere apologies.

    ]]>
    tag:status.keboola.com,2013:Post/1282357 2018-05-11T07:53:53Z 2018-05-11T07:53:53Z SQL Server writer failures

    Between 9 May 2018, 10:53 CEST and 11 May 2018, 09:30 CEST there were job failures for SQL Server Writer configurations that had nullable data types. The issue was created by a new version of the writer so we have rolled it back to the  previous version while we investigate the root cause.

    We're sorry for any inconvenience. 

    ]]>
    tag:status.keboola.com,2013:Post/1278617 2018-04-30T15:57:56Z 2018-05-01T08:44:04Z Week in Review -- April 30, 2018

    Core

    • Improved generated configuration changes descriptions
    • Added configuration version to jobs results of Docker-based components (it is not yet available for legacy components like transformation and gooddata-writer)
    • Refreshed Manage API docs with working examples
    • Fixed loading of large tables for R-studio and Jupyter sandboxes
    • Fixed random CSV Import upload errors in EU region

    Components

    • Improved "show details" experience for input and output mappings
    • Added visibility of columns non-existing in Storage to writers
    • Increased query timeout for all Keboola Provisioned Snowflake writers from 15 seconds to 15 minutes
    • Added support of unconventional column names to MySQL extractor
    • Removed static state from MongoDB extractor

    Processors

    • Added support of snappy format to processor-decompress
    • Added processor filter-files
    • Added support for sanitization of invalid utf-8 in processor-iconv

    Developers

    New Debug API call is available (replaces very rarely used sandbox, dry-run and input-data calls). It creates a snapshot of the data directory used for running the component and stores it in your KBC project. To learn more, feel free to go through the API Docs or through the tutorial. In short the API call:

    • uses the same calling convention as the Run API,
    • filters encrypted values from the data directory,
    • works with all components (previously only those without encryption were supported),
    • works with Processors,
    • works with Configuration Rows,
    • works also with broken components and configurations (even if the run fails, you'll still get a snapshot of the data directory).

    Python transformations

    Pip version 10 was released recently which removes the pip.main method (more reading). The recommended way to install packages from within python is:

    import subprocess
    import sys
    subprocess.call([sys.executable, '-m', 'pip', 'install', '--disable-pip-version-check', 'PACKAGE_NAME'])

    Currently there are 70 transformations using the removed pip methods. If your projects are using them we'll be contacting you with a list of affected transformations. This breaking change introduced in pip is currently blocking us from upgrading python to 3.6.5 where pip 10 is used by default.

    ]]>
    tag:status.keboola.com,2013:Post/1277785 2018-04-28T08:00:41Z 2018-04-28T08:00:41Z Unexpected Job Failures

    Between April 28 2:30 and 3:15 UTC there was a high rate of application errors on one of our instances processing component jobs. 

    The instance was under heavy load and we're investigating the root cause. Instance is now back to normal and is safe to restart the jobs.

    We're sorry for any inconvenience. 

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1273307 2018-04-16T07:40:40Z 2018-04-16T07:40:41Z Degraded performance of Google Sheets Writer

    On March 23, 2018 we released a new version of Google Sheets Writer to remove workaround which resized sheet's grid. Unfortunately this version caused a significant performance degradation for tables with larger number of rows.

    We decided to revert this version to bring back original performance.

    We are working on proper fix and it'll be released soon.

    ]]>
    Vladimír Kriška
    tag:status.keboola.com,2013:Post/1272211 2018-04-13T15:00:08Z 2018-04-13T15:00:08Z Orchestration Notification Updates Resulted in Deleted Tasks

    There was an update to the orchestrator this week that had an unfortunate side-effect.  If you updated your orchestrations' notifications it would delete the orchestration's tasks.

    Thankfully, the orchestrations are versioned, so if this happened to you, we will restore the tasks from the last version.
    If you have any concerns about this please contact us at support@keboola.com.  

    For what it's worth, updating notifications will no longer delete orchestration tasks, please accept our humble apologies if you were affected.

    ]]>
    tag:status.keboola.com,2013:Post/1270520 2018-04-09T15:32:52Z 2018-04-09T15:32:52Z Week in Review -- April 09, 2018

    Updated Components

    Google AdWords Reports

    • This extractor is finally enabled also for customers using EU instance

    Snowflake Writer

    • Added support of VARIANT data type

    Google Drive Extractor/Writer, Google Sheets Writer

    • We added support for Team Drives

    Impala Extractor

    • Added support for internal tables

    Generic Components

    We continue with removing so called "static state" from components. Few weeks ago we removed static state from Transformations, and there was a time for additional components. JSON configurations are also editable straight away. This includes configurations from templates (e.g. Youtube Extractor) and configurations for Custom Science Apps (e.g. Custom Science Python).

    Fixes

    • Python/R transformation sandboxes correctly apply filters in input mappings, so input data will be loaded correctly
    • CSV Import uses server side encryption in S3 stage (before uploading to our storage) by default
    • Gmail Extractor supports "message parts" in more sections and there should no longer be messages without parts
    • ThoughtSpot writer correctly handles the "Test Credentials" action

    Deprecations

    We are deprecating direct import from URL into Storage. Please use the new Http Extractor instead which gives you much more flexibility.

    ]]>
    Vladimír Kriška