tag:status.keboola.com,2013:/posts Keboola Status 2019-09-18T07:02:00Z Keboola Connection "Data Framework" tag:status.keboola.com,2013:Post/1456219 2019-09-18T07:01:59Z 2019-09-18T07:02:00Z Job errors in the EU region [post-mortem]

Summary

On Sunday, September 15th at 01:32 UTC, orchestrator and other component jobs started failing in the EU region. In the following hours, our worker servers weren't able to handle the workload, and the job backlog started to increase. We manually resolved the incident, and the platform was in full operation with a clean backlog at 08:26 UTC.

What happened?

One of the MySQL instances was automatically restarted and patched on September 15th at 01:32 UTC.
The instance is required for the lock mechanism for job processing, and it also stores information about queues for the worker servers. The 2-minute downtime of the database instance caused a failure of the jobs that were running at the moment. Additionally, the running workers weren't able to fetch the information about the queues, and some of them gave up restarts and stopped. With only half of the processing capacity left, the workload could not be processed.

Once we discovered the incident, we replaced all our worker servers and added more capacity to clean up the backlog faster.

What are we doing about this?

We have implemented notifications about upcoming instance patches and are going to perform updates during scheduled and announced maintenance windows.

We are also working on a completely new job processing and scheduling mechanism that will prevent similar issues from occurring down the road. We sincerely apologize for the inconvenience caused.


]]>
tag:status.keboola.com,2013:Post/1454829 2019-09-16T11:00:43Z 2019-09-16T11:00:44Z Week in Review -- September 16, 2019

New Features, Improvements and Minor Fixes


New Components

]]>
tag:status.keboola.com,2013:Post/1455838 2019-09-16T07:14:32Z 2019-09-16T07:14:32Z Job restarts in EU region

Few orchestrator and transformation jobs were restarted on September 16, 06:36 UTC

This was caused by issue in zone rebalancing. We apologize for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/1455438 2019-09-15T05:45:25Z 2019-09-15T08:29:39Z Job errors in EU region

We are investigating job failures in EU region started at 1:32 UTC.

We will provide an update when we'll have more information. 

UPDATE 06:06 UTC - We have identified the issue and fixed the cause. Backlog is processing now.

UPDATE 07:54 UTC - There is still backlog of orchestration jobs. We have increased the processing capacity. It should be cleared in half an hour.

UPDATE 08:26 UTC - The backlog was cleared. All services are running.

We apologize for the inconvenience, we'll share more details in a post-mortem.

]]>
tag:status.keboola.com,2013:Post/1452381 2019-09-09T11:24:34Z 2019-09-09T11:24:34Z Week in Review -- September 9, 2019

New Features, Improvements and Minor Fixes

  • When checking events in the job detail, the Load More button tries to load up to 1,000 events at once.
  • You can search in component configurations when adding a new task to an orchestration phase.

  • Refreshing your token no longer breaks access to your project.
  • Remove Empty Files and Folders processor has a new option available to remove files with whitespace characters only: remove_files_with_whitespace


New Components

]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1451344 2019-09-05T04:35:54Z 2019-09-05T04:35:54Z End of life GoodData Writer v2

Support for deprecated GoodData Writer v2  is coming to its end, the writer will be shut down on September 30, 2019 .

If you are still using the old Writer you should see a deprecation notice in Keboola Connection. Please proceed to migration.

If you need any help migrating the writer, contact Keboola support.


]]>
tag:status.keboola.com,2013:Post/1451362 2019-09-04T12:13:10Z 2019-09-04T12:27:07Z End of life of old OAuth Broker

This is a reminder that the Deprecated  OAuth broker will be shut down on September 30, 2019

Please start the migration if your projects aren't migrated yet. If you have any questions please contact Keboola support.

]]>
tag:status.keboola.com,2013:Post/1450385 2019-08-31T13:44:28Z 2019-08-31T16:17:25Z Jobs failures and timeouts due to AWS shortage in US

We are experiencing some jobs failures and timeouts in the US region due to a shortage in one availability zone in the Amazon Elastic Compute Cloud service. We are going to monitor the situation and keep you posted.

UPDATE 15:48 CEST: Apparently it has some overlap to other AWS services too because login to Developer Portal (which uses AWS Lambda and Cognito) timeouts intermittently.

UPDATE 15:54 CEST: AWS confirms that some EC2 instances are impaired and some EBS volumes are experiencing degraded performance within a single Availability Zone in the US-EAST-1 Region. Some EC2 APIs are also experiencing increased error rates and latencies. They are working to resolve the issue.

UPDATE 16:37 CEST: The works on resolving the issue are still in progress.

UPDATE 17:06 CEST: The impaired instances and EC2 APIs are being recovered. AWS support continues to work towards recovery for all affected EC2 instances.

UPDATE 18:04 CEST: Recovery is in progress for instance impairments and degraded EBS volume performance. On our side, it looks that the problems more or less disappeared an hour ago and the platform is back to normal.

]]>
tag:status.keboola.com,2013:Post/1445390 2019-08-16T17:25:06Z 2019-08-16T17:25:06Z Broken Login [Post-Mortem]

Summary

On 2019-08-15 12:23 UTC, we deployed a broken version of Keboola Connection. It prevented some users from accessing their projects. The problem was fixed at 13:01 when we rolled back to a previous version. We sincerely apologize for interrupting your work and wasting your time.

What Happened?

There was an error in a permission check, and only users with the permission to create a project were allowed to enter a project. Such a scenario is not covered in the functional tests, and the situation was overlooked during peer review of the change. As soon as we identified the problem, we immediately deployed a previous version of Keboola Connection. That itself took about 15 minutes.

What Are We Doing About It?

We're extending the software tests to include more scenarios.

We're also updating monitoring alarms to make sure that we know about a problem before you tell us through our support channel.


]]>
tag:status.keboola.com,2013:Post/1444605 2019-08-16T16:59:18Z 2019-08-16T16:59:19Z Week in Review - August, 16th, 2019

New Features

  • Tables in shared buckets and table aliases now contain metadata from the source tables. In practice, this means that when you create an input mapping for transformations using tables in shared buckets (or aliases), you see the source table data types.

  • In the API response, the source table metadata are contained in the `sourceTable` node – so both the table and alias metadata are available.

New Components

A number of components by Revolt BI:

Updated Components

GoodData Writer

  • Supports reading the Logical Data Model (LDM) from a project.

Minor Improvements & Fixes

  • Google Drive verification issue in the EU region has been resolved.

  • In the input mapping of Snowflake transformations, the TIMESTAMP data type now defaults to TIMESTAMP_NTZ.

  • Terminated job is colored in the same way as a terminated label.



]]>
tag:status.keboola.com,2013:Post/1445085 2019-08-15T12:52:43Z 2019-08-15T13:09:26Z Broken login [resolved]

There is an issue which causes a 404 error (Page not found) to be displayed after login. We're working on a fix, which should be ready shortly.

The issue affects only log-in of certain users on the UI. Jobs running inside projects are unaffected.

Update 13:01 UTC: The problem is fixed. We apologize for the inconvenience, we'll share more details in a post-mortem.


]]>
tag:status.keboola.com,2013:Post/1444642 2019-08-14T11:54:06Z 2019-08-14T11:54:07Z Google components authorisation issue [Post-Mortem]

Since July 26 2019 there was an issue with authorisation of our Google related components using OAuth in the EU region.

We needed to undergo a verification process of our apps by Google. This process has been recently made more strict and we weren't able to finish it before the deadline.

We are very sorry about this, we should have paid more attention to this process and the consequences related to not complying with it.

We have now successfully verified most of our Google components. We are still waiting for the verification of GMail Extractor and GMail Attachments Extractor.

]]>
Miroslav Čillík
tag:status.keboola.com,2013:Post/1439142 2019-07-30T17:20:08Z 2019-07-30T17:20:08Z Creating projects from try.keboola.com in US region not working [resolved]

Today, 30.7.2019, 18:30 CET we noticed errors when creating projects using promocodes via https://try.keboola.com. We rollbacked app to the previous version and everything is working now. Moreover we continue investigating the cause of the error.

]]>
tag:status.keboola.com,2013:Post/1437356 2019-07-26T12:12:41Z 2019-08-14T10:03:09Z Google Drive extractor authorization in EU [resolved]

We are investigating a Google Drive Extractor authorization verification issue in Keboola Connection EU.

Only creation of new configurations is affected.

We will provide an update as soon as the issue is resolved.

Update 2019-08-14: The verification issue is resolved and new configuration are working again.


]]>
tag:status.keboola.com,2013:Post/1426793 2019-07-19T12:04:17Z 2019-07-19T12:04:18Z Week in Review -- July 19, 2019

Updated Components

Event triggered orchestrations

  • Disabled orchestrations are not triggered anymore
  • Settings (like emails for error notifications) are taken from the orchestration configuration
Writers

Following writers now set data types based on metadata:

  • Tableau TDE
  • Impala
  • Mssql
  • Mysql
  • Oracle
  • PostgreSql
  • Redshift
  • Thoughspot
  • BigQuery v2

You can read more about Storage data types in the docs.

Gooddata Writer v3

  • Fact grain is no longer reset when switching from full to incremental load

Other components

New Features

Notification of concurrent editing

A warning message is displayed when you try to edit a configuration that's being edited by someone else at the same time. It also warns you if you are editing the same configuration in a different browser tab. 

CAPSLOCK notification for password field

When you focus the password field and your capslock is enabled, you'll see a notification. 

]]>
tag:status.keboola.com,2013:Post/1433286 2019-07-17T05:30:12Z 2019-07-17T05:30:13Z Q3 Customer Webinar

Keboola is hosting our inaugural Q3/2019 customer webinar and we want you there! If you haven’t done so already, please register now.  Learn about new features, functionalities, and components that we have built during the first half of the year.  Additionally, see the exciting plans that we have for Keboola over the next 3-12 months.

We hope you can join us!  The webinar will be available on video by request afterward as well.

Register here.

]]>
tag:status.keboola.com,2013:Post/1431255 2019-07-11T10:03:11Z 2019-07-11T10:03:11Z Error when saving configuration Post-Mortem

Some of our users experienced errors in Keboola Connection (in both regions) yesterday when saving configurations of several components.

An error was displayed to users who tried to save their configuration between Jul 9, 2019, 12:35 CET and Jul 10, 2019, 14:18 CET.

Mainly users who have an older version of Chrome Browser or any version of the Firefox browser were affected.

The issue was caused by wrong response headers (unsupported by the mentioned browsers) being sent to a preflight request on one of our API services.

We sincerely apologize for the inconvenience caused.

]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1429157 2019-07-09T17:15:10Z 2019-07-09T17:15:11Z Navigation Moved and Redesigned

Three weeks ago we announced the start of Keboola Connection redesign.

We're glad we can share another important update with you.

We moved the navigation from the left side to the top

This allows you to more quickly navigate within a project and should improve user experience. All sections remain the same and the content hasn't changed.

  • Project Settings can be found on the right side (by clicking on the "cogs")

  • The Project Selector is almost the same as before

  • The User Settings menu has a new home on the top right side

  • Also, we added a shortcut to the Trash, so you no longer need to navigate there through User & Settings


We hope you'll like this update as much as we do.

]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1426864 2019-07-02T14:02:54Z 2019-07-02T14:37:00Z Support system unavailable

Zendesk is down, so our support system is inaccessible. Keboola Connection is unaffected, but if you run into anything critical, please reach us via email.

We'll update this post when we obtain more information.

Update 14:36 UTC - Looks like things are back to normal now.

]]>
tag:status.keboola.com,2013:Post/1424646 2019-06-27T12:25:27Z 2019-06-27T12:25:28Z Week in Review -- June 27, 2019

New Features and Improvements

Require MFA for organization

There's a new option in organization-level settings which allows you to require Multi-Factor Authentication for each member of your organization and project.

Users not having the Multi-Factor Authentication enabled will be prompted to enable it.

Set same data type to all columns at once

When setting data types in transformation's Input Mapping you can set same data type to all columns at once.

GoodData Writer data types

When configuring tables in GoodData writer, we try to look for metadata for columns and set types for you automatically.


New Components

Google Translate

Translates text into any supported language.

https://components.keboola.com/components/kds-team.ex-google-translation

Google NLP

New component which uses Google Cloud Natural Language API to extract text features using pre-trained machine learning models.

https://components.keboola.com/components/kds-team.ex-google-nlp


]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1422208 2019-06-24T16:53:58Z 2019-06-24T16:53:58Z Deprecation of GoodData Writer v2

The old version of GoodData Writer (v2) was already unpublished a few months ago and does not allow you to create new configurations since then. Now the time has come to deprecate the component entirely and set a date for its final shut down. 

The date is September 31, 2019, so you are encouraged to migrate old configurations until then. The ui shows a deprecation notice and offers you to run the migration process.

]]>
Jakub Matějka
tag:status.keboola.com,2013:Post/1422237 2019-06-21T08:58:40Z 2019-06-21T08:58:40Z Week in Review -- June 21, 2019

New Features

  • You can now schedule orchestrations by table events.  Do you want to run an orchestration as soon as a key table gets updated?  You can do that now!  Please refer to the documentation to help guide you through it.


  • When creating a new token you can now restrict its access to the configuration trash bin

Improvements

Transformations

  • Transformations can now be copied to the same bucket or an other transformation bucket

  • Simplified version diffs for scripts


  • You will now be given a warning if you cannot extend your sandbox because of required system upgrades


Orchestrations

  • For notifications, you can now choose from any email of any user that has previously worked on the orchestration or of course enter one of your choosing

  • On the orchestrations listing page it will show if the orchestrations were manually triggered, event triggered, or scheduled


SQL Server extractor

  • Allows incremental fetching by timestamp data type columns

  • A new option to disable use of the fallback driver on failure

Developers

  • We are now always displaying the documentation link for components, so please make sure that your components have it filled in the Developer Portal.  If you have the documentation link then you can remove excess documentation from the long description.


]]>
tag:status.keboola.com,2013:Post/1422150 2019-06-20T14:19:29Z 2019-06-20T14:19:29Z Jobs Delays in the US Region

Today, Thursday 20, 2019 from 02:02 CET, one of our job workers became unresponsive.

This resulted in some jobs taking longer to process as they had to wait for one of the other available workers whose workload had increased as a result.

The affected instance was replaced at 15:50 CET and we are working on improving the automated detection of such failures.

Thank you for your understanding.

]]>
tag:status.keboola.com,2013:Post/1420176 2019-06-14T15:24:51Z 2019-06-14T15:24:51Z Starting Keboola Connection redesign

Many of you have noticed that our website was redesigned a few months ago. We have been continuing to prepare and apply a new design to all remaining parts of our system:

Four months ago, we introduced the Refined Storage Console, even though the main point of the changes was to remove old code. Four weeks ago, we started using the Roboto font in Keboola Connection and Developer Portal.

We're now happy to announce that the first part of the redesign has been applied to all pages you see during the authentication process.

The authentication process itself wasn't affected by the changes and you should be able to go through it with ease.

There are many more things to come from us. We're already working on the second phase of the redesign. Stay tuned.

]]>
Vladimír Kriška
tag:status.keboola.com,2013:Post/1409028 2019-05-20T16:53:24Z 2019-05-20T16:53:40Z Weeks in review -- May 20, 2019

Improvements

  • Transformation buckets can now run in parallel. That means when you put multiple transformation buckets in one orchestration phase, they will execute in parallel - the same way extractors or writers work. Previously the transformation jobs would be queued to execute sequentially.
  • The editor of transformation queries now uses the official Snowflake syntax highlighter, so the syntax highlighting should now be consistent with the Snowflake Sandbox.

  • The organization settings now offer a new feature to require MFA (Multi-Factor Authentication) for all organization members.

  • A new GoodData User Management Application was released. It replaces part of the functionality of the old GoodData writer and allows to manage users in the GoodData project, invite new users to the project, change roles and assign data permissions for each user.

Fixes

  • A more readable Roboto font is now used in Keboola Connection and Developer Portal UI.
  • Fixed error when adding metadata to columns with special characters in name.
  • Column metadata are now displayed even when they cannot be edited (due to insufficient permissions, table being an alias or table being hosted on Redshift).

Developers

  • Removed support for deprecated private Quay.io and private Docker Hub repositories (no active components were using it).


]]>
Ondřej Popelka
tag:status.keboola.com,2013:Post/1409623 2019-05-17T05:05:29Z 2019-05-17T12:37:33Z Snowflake issues in US region

Update 12:36 UTC:

Everything should be back to normal. We'll keep monitoring our systems.


Update 9:15 UTC:

You should not experience any more errors, but things are a bit overloaded so longer execution and waiting times are expected. Thank you for patience.


Update 7:52 UTC:

Snowflake services restored. We're resuming processing of jobs.


Update 6:23 UTC:

Snowflake services are still not restored. We're slowing down job processing, so you'll see a much larger amount of queued jobs. EU region is unaffected.


Original post:

Some Snowflake queries started failing around 03:14 UTC which is causing errors in job processing. Failures are caused by incident which is currently investigated by Snowflake https://status.snowflake.com/incidents/0sjfn3d5jq2q .

We will provide an update as soon as the issue is resolved.


]]>
tag:status.keboola.com,2013:Post/1401484 2019-04-26T10:22:25Z 2019-04-26T10:22:25Z Week in review -- April 26, 2019

Column descriptions and user defined data types

Description and custom data type can be provided for each table column in Storage. Custom data type allows you to override data type provided by the system. These data types are then used as defaults for transformation and writer table inputs.

You can explore and edit these values on Storage table detail.

Chained aliases

An alias can be created from another alias. Also aliases created in Shared Buckets are propagated to linked buckets and can be further aliased. This simplifies data preparation and sharing, tables which don't require additional processing can be directly aliased to shared buckets.

Chaining is supported only for aliases with automatically synchronized columns and without filter.

Automatic Incremental Processing

With automatic incremental processing, the component will receive only data modified since the last successful run of that component.

Code Templates for  Jupyter and RStudio sandboxes

For Jupyter and RStudio sandboxes, code templates can be defined. Code templates can be set for a given user or for the entire project. A Jupyter template is a notebook file (.ipynb). An RStudio template is a simple text file. If a sandbox is loaded from transformation, the transformation code will be appended after the template code.

Google BigQuery

  • New extractor with Google Service Account service authentication was published
  • New writer with Google Service Account service authentication was published
  • Previous version of the writer is deprecated and will be shut down on 1. 8. 2019.The migration to new version of extractor is available.

Other Updates

  • Create a single task orchestration from component configuration
  • New version of Zboží.cz Extractor by Medio - Get your daily impressions, clicks, cost and conversion stats for preset time range or previous day.
  • Python sandboxes and transformations were upgraded to Python version 3.7.3
  • R sandboxes and transformation were upgraded to R version 3.5.3
]]>
Martin Halamíček
tag:status.keboola.com,2013:Post/1399282 2019-04-18T12:40:36Z 2019-04-18T13:53:41Z Jobs failures

One of our core databases in US region has been restarted due to some failures by AWS. All running jobs failed or will fail due to this reason.

UPDATE: The problem affected ca 26 jobs. There was some unspecified low-level failure on the database or its underlying hardware and AWS automatically switched to a backup instance. We are going to investigate if our platform can handle the failure and restart the jobs automatically next time. Thanks for your understanding, we are sorry for any inconvenience.

]]>
Jakub Matějka
tag:status.keboola.com,2013:Post/1397546 2019-04-13T14:49:29Z 2019-04-13T14:49:29Z Some Orchestration Failures in the US Region

Today, on April 13 2019 from 14:14 UTC to 14:15 there were some orchestration failures in the US region due to an internal system upgrade. Very few orchestrations were affected, please review your orchestration and restart if needed. This issue is not related to the issue with failing orchestrations happened previously today.

]]>
tag:status.keboola.com,2013:Post/1397481 2019-04-13T06:19:46Z 2019-04-13T07:19:33Z Failing Orchestrations in the US Region

[2019-04-13 8:20 CET]

We are registering increased amount of failing orchestrations since 2:27 CET. We are investigating the issue now and will keep you updated.


[2019-04-13 8:45 CET]

We traced down the problem and fix it. Everything seems to be working now, we continue monitoring it. Please review your orchestrations and restart them if necessary



]]>