tag:status.keboola.com,2013:/posts Keboola Status 2018-10-08T11:41:34Z Keboola Connection "Data Framework" tag:status.keboola.com,2013:Post/1328456 2018-10-08T11:41:34Z 2018-10-08T11:41:34Z Week in Review -- October 08, 2018

Updated Components

  • MS SQL Server extractor has had a driver update.  Advanced queries now have slightly different behaviour so it is recommended to switch to simple table and column selected configurations.  Where that is not possible, please see the documentation for further help.
  • Snowflake writer - minor bugfixes and enhancements
    • primary key is added when creating a table
    • nonexistent schema/warehouse validation
    • warehouse detected correctly for users with different login name and user name
  • Database extractors - column datatypes are now shown in UI

Minor Improvements

    • You can now change the sharing type of a shared bucket in the storage console.

    ]]>
    tag:status.keboola.com,2013:Post/1330188 2018-10-08T11:09:25Z 2018-10-08T11:09:25Z Stalled jobs in EU region

    Around 1:30am CEST one of the job worker instances stopped processing assigned jobs. This could have lead to jobs being stuck in the processing state for a long time without any activity.

    At 11:15am CEST the worker instance was terminated and all unfinished jobs started processing on other instances.

    We're sorry for this inconvenience.

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1329850 2018-10-07T06:43:17Z 2018-10-07T06:43:17Z Failed and delayed jobs in EU region

    Database storing locks was restarted at 03:49 UTC which caused the job failures. Also some of the jobs were queued after this failure.

    The backlog of all jobs was cleared at 06:15 UTC. The system is fully operational now. We're working on infrastructure changes which should prevent similar issues.

    ]]>
    tag:status.keboola.com,2013:Post/1328422 2018-10-06T10:25:29Z 2018-10-06T10:25:30Z US Region Maintenance Finished

    The maintenance period in the US region has just finished. All systems are back online and working properly.

    Thank you for your patience.

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1328423 2018-10-06T08:17:23Z 2018-10-06T08:17:23Z EU Region Maintenance Finished

    The maintenance period in the EU region has just finished. All systems are back online and working properly.

    Thank you for your patience. 

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1328421 2018-10-06T06:08:19Z 2018-10-06T06:08:19Z US and EU Region Maintenance

    The pre-announced maintenance period for the US and EU regions has just started. The estimated downtime is 5 hours. 

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1328420 2018-10-05T10:00:00Z 2018-10-05T16:51:27Z Scheduled US and EU Maintenance

    There will be a maintenance period on Saturday, October 6th, 2018 from 8:00am CEST and should take less than 5 hours.

    We will be upgrading component job indices and metadata databases.

    All projects in connection.keboola.com and connection.eu-central-1.keboola.com will be inaccessible during the maintenance.

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1328764 2018-10-04T09:21:42Z 2018-10-08T11:19:21Z Degraded Snowflake Performance (US region)

    Since September 25 we're experiencing degraded Snowflake performance affecting all Snowflake operations. 

    We're sorry for this inconvenience, we're working with Snowflake to fix this issue.

    Update, October 8

    Snowflake Engineering team has discovered and fixed the issue (waiting for an official statement from Snowflake). We're seeing operation times going back to normal.

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1328096 2018-10-02T06:40:27Z 2018-10-02T16:34:38Z Jobs delay issues in US region

    Storage jobs processing was delayed between 04:30 - 06:04 UTC. Worker nodes didn't started as expected to process the workload. 

    We are investigating the issue.

    ]]>
    tag:status.keboola.com,2013:Post/1325796 2018-09-26T08:33:20Z 2018-09-28T17:31:10Z Jobs delay issues in US region

    We are experiencing minor job queue issues in US region which may result in delaying orchestrations. We are investigating this issue at the moment. We are sorry for inconvenience.

    UPDATE, Sep 28, 06:30 UTC - We have confirmed from Snowflake support that they are tracking the issue in the increase in the compilation time of the queries and their engineering is actively investigating the issue.

    UPDATE, Sep 28, 10:45 UTC - Snowflake support reported that they are experiencing a degradation of service with elevated query failures in our US region. https://support.snowflake.net/s/feed/0D50Z00008QI7AqSAL

    UPDATE, Sep 28, 15:00 UTC - Snowflake support reported that they have resolved the issue and the US west region is again operating at 100%

    ]]>
    tag:status.keboola.com,2013:Post/1325386 2018-09-25T11:12:12Z 2018-09-25T11:12:13Z Week in Review -- September 25, 2018

    Keboola Connection in AUS region

    We have launched Keboola Connection in AUS region. You can now create your projects in the AUS region. Please contact info@keboola.com if you are interested.


    Feature Wishlist

    According to our open-source philosophy, we always want to hear new ideas from you. To make communication easier, we have added a Feature Wishlist link to Keboola Connection sidebar which leads to productboard site where you can request new features.



    ]]>
    tag:status.keboola.com,2013:Post/1325357 2018-09-25T07:12:52Z 2018-09-25T09:56:01Z Google Sign In issues

    Some users are not able authenticate to KBC using Google Sign In. We are investigating the issue.

    In the meantime, please sign in using the standard email and password method.

    UPDATE, Sep 22, 07:47 UTC - We have identified the issue. Google users with some special characters in name were affected. We have made changes in interaction with Google. It should resolve this issue. Fix should be deployed in two hours.

    UPDATE, Sep 22, 09:55 UTC - Fix was deployed. You should be able to authenticate to KBC using Google Sign In. 


    ]]>
    tag:status.keboola.com,2013:Post/1324307 2018-09-22T05:20:47Z 2018-09-22T10:12:01Z Scheduled Maintenance (US Region)

    Scheduled maintenance will start at 06:00 am UTC. 

    The maintenance won't be system wide. It will be performed for each project in connection.keboola.com separately and it should take few minutes for each project depending on the number of project's files. It should not affect project's running jobs, these will be queued during the short project maintenance. Only projects created before August 16, 2018 will be affected by this migration.

    We'll update this status about the progress.

    Update, Sep 22, 06:00 UTC - Migrations have started.

    Update, Sep 22, 10:11 UTC - All projects have been migrated.


    ]]>
    tag:status.keboola.com,2013:Post/1323899 2018-09-21T04:29:45Z 2018-09-21T05:31:24Z Failing Oracle Extractor

    Friday, September 26, 2018 04:30 UTC

    Due to a configuration error the Oracle extractor was failing to process jobs from 22:00 UTC Thursday, September 20 to around 04:20 UTC on Friday, September 21, 2018.

    The error has now been corrected and all new jobs will process correctly.

    Sincere apologies for the error and inconvenience.

    ]]>
    tag:status.keboola.com,2013:Post/1323163 2018-09-19T16:44:25Z 2018-09-19T16:44:25Z Week in Review -- September 19, 2018

    Updated Components

    Minor Improvements

    • Link to a configuration from the Job Detail page has been improved for the Transformation component and also for some components which support configuration using Configuration Rows
    • From now you can switch to another Keboola Connection region directly from the Login Page

    • We unified search inputs across application, so searching will work in more unified way for you

    Client libraries

    ]]>
    Vladimír Kriška
    tag:status.keboola.com,2013:Post/1320309 2018-09-17T10:08:19Z 2018-09-17T10:08:19Z SQL Server Extractor Driver Update

    The Microsoft SQL Server Extractor will be updated to use a new driver on October 1, 2018.

    This update may effect the format of datetime columns if you are using the advanced query option in your configuration.
    Standard configurations (not advanced queries) will not have any change to their output.

    The old driver exports datetime columns in the format `yyyy-mm-dd hh:mm:ss` however the new driver includes milliseconds when outputting datetime values `yyyy-mm-dd hh:mm:ss.ms`. 

    There are 2 options to mitigate this change:

    1) Recommended: Switch the configuration to select the tables and columns that you need and not use an advanced query.

    2) Convert the datetime output column in the SQL query, ex: CONVERT(DATETIME2(0), my_column)

    If you have any concerns or questions regarding this update please contact us using the support button in your Keboola Connection project.

    Thank you

    ]]>
    tag:status.keboola.com,2013:Post/1322157 2018-09-16T07:08:56Z 2018-09-16T08:33:59Z Increased Error Rate in MySQL Extractor

    We have encountered an increased error rate in MySQL Extractor after a new version release on Friday, Sept 14, 16:00 UTC.

    These errors happened only when the job had a connection issue and had to reconnect to the MySQL server.

    The previous version is now rolled back and we're working on a fix.

    We're sorry for this inconvenience.


    Update Sunday, Sept 16, 2018, 08:15 UTC 

    The MySQL Extractor has been updated to resolve this issue. 

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1321564 2018-09-14T07:15:07Z 2018-09-14T07:15:07Z Snowflake Outage (US Region)

    We have encountered an increased number of Snowflake connection errors between 04:11 UTC and 04:21 UTC. This may have caused failed storage and component jobs.

    Furthermore Snowflake announced possible SQL query failures between 02:24 UTC and 03:30 UTC. 

    We're sorry for this inconvenience, all systems are operational now. 

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1321263 2018-09-13T06:41:59Z 2018-09-14T13:08:43Z Snowflake connection issues (US region)

    We're experiencing random connection errors to Snowflake started on Sep 12, 22:08 UTC.

    These errors caused a few transformations failures. We're investigating the issue and we'll update this post with our findings.

    Update, Sep 14, 07:15 UTC

    These rare errors are still occuring.

    Update, Sep 14, 13:07 UTC

    Snowflake confirms the issue is on their side and are investigating it.

    ]]>
    tag:status.keboola.com,2013:Post/1320608 2018-09-11T08:03:19Z 2018-09-11T08:55:43Z Investigating incident (US region)

    We're currently investigating an incident in the US region that may have caused component job and orchestration failures today, Tuesday, Sep 11, between 02:30 UTC and 06:30 UTC.

    Update 08:30 UTC

    The DB storing job locks (preventing jobs from running multiple times on multiple workers) was restarted at 02:33 UTC. All connections were terminated and all component jobs and transformations running at that time were disconnected. This could lead to any or both of the following situations:

    • Failure at any time later during the job execution
    • A new parallel job execution

    Any jobs started after the DB restart were not affected by this issue.

    We apologize for this inconvenience. We're planning an infrastructure change to prevent such huge impact during similar situations. 

    If you have any further questions, please use the support button in your project.

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1314156 2018-08-22T12:45:01Z 2018-08-22T12:46:42Z Week in Review -- August 22, 2018

    Updated Components

    Storage

    • Added Configuration Row detail API resource

    Slack Extractor

    • Fixed pagination. The extractor was only downloading 1000 records from a channel history.
      Existing configuration of the extractor needs to be re-saved to apply this fix.


    New Features

    • When creating new API tokens, you can now choose Orchestrator from the Custom Access components dropdown

    Minor Improvements

    • Added Incremental label to Output Mapping
    • Storage API and Manage API tokens are now generated from alphanumeric characters instead of hexadecimal
    • Added Documentation badge to component detail. Additional external documentation is available for component marked with this badge. 
    • Filter expression operators were renamed to be more intuitive

      EQ (equals) -> IN
      NE (not equal) -> NOT IN
    • It's no longer necessary to click the <-Back button when adding tasks to Orchestration. The New Task modal will reset its state and after reopening it shows the list of components. Previously, after reopening, it showed a list of configurations of the previously selected component
    ]]>
    Miroslav Čillík
    tag:status.keboola.com,2013:Post/1313799 2018-08-22T06:49:51Z 2018-08-22T06:49:51Z September, October 2018 Maintenance Schedule

    US Region

    Sat, September 22, 2018

    A maintenance will start at 6:00 am UTC. We'll be migrating metadata storage of Storage Files during the maintenance.
    The maintenance won't be system wide. It will be performed for each project in connection.keboola.com separately and it should take few minutes for each project depending on the number of project's files. It should not affect project's running jobs, these will be queued during the short project maintenance. Only projects created before August 16, 2018 will be affected by this migration.

    Sat, October 6, 2018

    A maintenance will start at 6:00 am UTC and should take less than two hours. We'll be migrating metadata of Jobs during the maintenance. All jobs will be stopped during the maintenance and should be restarted after the maintenance. Orchestrations scheduled during the maintenance period won't be started.

    All projects in connection.keboola.com will be inaccessible during the maintenance.

    EU Region

    Sat, October 6, 2018

    A maintenance will start at 6:00 am UTC and should take less than two hours. We'll be migrating metadata of Jobs during the maintenance. All jobs will be stopped during the maintenance and should be restarted after the maintenance. Orchestrations scheduled during the maintenance period won't be started.

    All projects in connection.eu-central-1.keboola.com will be inaccessible during the maintenance.

    ]]>
    tag:status.keboola.com,2013:Post/1309897 2018-08-07T12:28:19Z 2018-08-07T16:47:04Z Jupyter and RStudio Sandboxes are not starting in US Region

    2:27pm CEST: We're investigating the issue.

    5:34pm CEST: New sandboxes are now running correctly. If you can't connect to an existing sandbox, please recreate it. We are deeply sorry for the trouble and frustration.

    ]]>
    Erik Žigo
    tag:status.keboola.com,2013:Post/1308578 2018-08-02T09:44:29Z 2018-08-02T09:44:29Z Week in Review -- August 2, 2018

    Improvements

    • Jobs and Storage Files search syntax hints
    • Configurations quick search on component detail page

    Updated components

    Developers

    Storage Form Uploads Deprecation

    We are deprecating rarely used Storage HTTP Form Upload. This type of upload is supported only in US KBC region and is using legacy AWS Signature Version 2. The upload can be replaced by direct S3 upload using AWS SDK or by Importer Service for POCs or smaller files. We’ll notify projects using this type of upload to help them migrate.

    ]]>
    tag:status.keboola.com,2013:Post/1306482 2018-07-27T22:15:26Z 2018-07-27T22:15:27Z Jupyter and R Studio sandboxes start timeouts

    We are investigating issues with expired certificates for Jupyter and R Studio (beta) sandboxes.

    UPDATE 11:30 CEST  - New certificates were deployed and issue is resolved. Existing sandboxes have to be created again.

    ]]>
    tag:status.keboola.com,2013:Post/1306004 2018-07-24T17:35:45Z 2018-07-24T17:35:45Z Failing Create Manifest Processor

    Between 4:00 pm CEST and 7:30 pm CEST the new version of Create Manifest processor was failing with an application exception. This change was rolled back and the processor is functional now. 

    We're sorry for this inconvenience.

    ]]>
    Ondrej Hlavacek
    tag:status.keboola.com,2013:Post/1305913 2018-07-24T12:01:23Z 2018-07-24T12:01:23Z Week In Review -- July 24, 2018

    New Components

    Updated Components

    • Deprecated components cannot create new versions by copying from older versions.
    • Improved autocomplete in output mapping across components.
    • New release of Oracle extractor with massive performance increase.
    ]]>
    tag:status.keboola.com,2013:Post/1304679 2018-07-20T07:45:29Z 2018-07-21T08:35:04Z EU Region Maintenance Announcement

    A maintenance will take place on Saturday, July 21st, 2018 from 8:00 am CEST and should take less than two hours.

    We will be upgrading metadata databases.

    All projects in connection.eu-central-1.keboola.com will be inaccessible during the maintenance.

    - UPDATE 08:00 CEST - Scheduled maintenance is currently in progress.

    - UPDATE 10:10 CEST - The maintenance has been performed. We are starting job workers and verifying that all systems are functioning normally.

    - UPDATE 10:21 CEST - All workers are running and jobs backlog is processed.

    - UPDATE 10:31 CEST - The maintenance has been completed. Thank you for your patience.

    ]]>
    tag:status.keboola.com,2013:Post/1304300 2018-07-19T07:07:27Z 2018-07-19T07:07:27Z Errors in EU region

    Few jobs have failed between 2:30 am and 04:51 am in EU region. We are investigating the issue.

    ]]>
    tag:status.keboola.com,2013:Post/1301335 2018-07-16T07:04:05Z 2018-07-16T07:04:05Z Restbox Component Deprecation Announcement

    The time has come and the Restbox component as we know it is now being deprecated. But do not worry, its functionality will not disappear.

    The Restbox component was once a great tool to collect data from various sources that were not complex enough to deserve a dedicated extractor (or writer) and shared similar processing aspects, such as decompression, audit tools, and CSV formatting. This has now been broken up into multiple separate components and processors to achieve or exceed the required functionality.

    Today we have the following components to replace the Restbox component:

    Detailed documentation for most of them can be found at help.keboola.com.

    The Restbox component is now available in the US region only, and, as of this announcement, no new configurations can be created. The component will be supported until November 1, 2018 when it will be finally shut down. Please migrate your configurations to one of the above-mentioned components. 

    If you have any trouble migrating your configurations, please do not hesitate to contact us using the support button in your project.

    ]]>
    Ondrej Hlavacek