Jobs failures and timeouts due to AWS shortage in US

We are experiencing some jobs failures and timeouts in the US region due to a shortage in one availability zone in the Amazon Elastic Compute Cloud service. We are going to monitor the situation and keep you posted.

UPDATE 15:48 CEST: Apparently it has some overlap to other AWS services too because login to Developer Portal (which uses AWS Lambda and Cognito) timeouts intermittently.

UPDATE 15:54 CEST: AWS confirms that some EC2 instances are impaired and some EBS volumes are experiencing degraded performance within a single Availability Zone in the US-EAST-1 Region. Some EC2 APIs are also experiencing increased error rates and latencies. They are working to resolve the issue.

UPDATE 16:37 CEST: The works on resolving the issue are still in progress.

UPDATE 17:06 CEST: The impaired instances and EC2 APIs are being recovered. AWS support continues to work towards recovery for all affected EC2 instances.

UPDATE 18:04 CEST: Recovery is in progress for instance impairments and degraded EBS volume performance. On our side, it looks that the problems more or less disappeared an hour ago and the platform is back to normal.

Broken Login [Post-Mortem]

Summary

On 2019-08-15 12:23 UTC, we deployed a broken version of Keboola Connection. It prevented some users from accessing their projects. The problem was fixed at 13:01 when we rolled back to a previous version. We sincerely apologize for interrupting your work and wasting your time.

What Happened?

There was an error in a permission check, and only users with the permission to create a project were allowed to enter a project. Such a scenario is not covered in the functional tests, and the situation was overlooked during peer review of the change. As soon as we identified the problem, we immediately deployed a previous version of Keboola Connection. That itself took about 15 minutes.

What Are We Doing About It?

We're extending the software tests to include more scenarios.

We're also updating monitoring alarms to make sure that we know about a problem before you tell us through our support channel.


Week in Review - August, 16th, 2019

New Features

  • Tables in shared buckets and table aliases now contain metadata from the source tables. In practice, this means that when you create an input mapping for transformations using tables in shared buckets (or aliases), you see the source table data types.

  • In the API response, the source table metadata are contained in the `sourceTable` node – so both the table and alias metadata are available.

New Components

A number of components by Revolt BI:

Updated Components

GoodData Writer

  • Supports reading the Logical Data Model (LDM) from a project.

Minor Improvements & Fixes

  • Google Drive verification issue in the EU region has been resolved.

  • In the input mapping of Snowflake transformations, the TIMESTAMP data type now defaults to TIMESTAMP_NTZ.

  • Terminated job is colored in the same way as a terminated label.



Broken login [resolved]

There is an issue which causes a 404 error (Page not found) to be displayed after login. We're working on a fix, which should be ready shortly.

The issue affects only log-in of certain users on the UI. Jobs running inside projects are unaffected.

Update 13:01 UTC: The problem is fixed. We apologize for the inconvenience, we'll share more details in a post-mortem.


Google components authorisation issue [Post-Mortem]

Since July 26 2019 there was an issue with authorisation of our Google related components using OAuth in the EU region.

We needed to undergo a verification process of our apps by Google. This process has been recently made more strict and we weren't able to finish it before the deadline.

We are very sorry about this, we should have paid more attention to this process and the consequences related to not complying with it.

We have now successfully verified most of our Google components. We are still waiting for the verification of GMail Extractor and GMail Attachments Extractor.

Google Drive extractor authorization in EU [resolved]

We are investigating a Google Drive Extractor authorization verification issue in Keboola Connection EU.

Only creation of new configurations is affected.

We will provide an update as soon as the issue is resolved.

Update 2019-08-14: The verification issue is resolved and new configuration are working again.


Week in Review -- July 19, 2019

Updated Components

Event triggered orchestrations

  • Disabled orchestrations are not triggered anymore
  • Settings (like emails for error notifications) are taken from the orchestration configuration
Writers

Following writers now set data types based on metadata:

  • Tableau TDE
  • Impala
  • Mssql
  • Mysql
  • Oracle
  • PostgreSql
  • Redshift
  • Thoughspot
  • BigQuery v2

You can read more about Storage data types in the docs.

Gooddata Writer v3

  • Fact grain is no longer reset when switching from full to incremental load

Other components

New Features

Notification of concurrent editing

A warning message is displayed when you try to edit a configuration that's being edited by someone else at the same time. It also warns you if you are editing the same configuration in a different browser tab. 

CAPSLOCK notification for password field

When you focus the password field and your capslock is enabled, you'll see a notification. 

Q3 Customer Webinar

Keboola is hosting our inaugural Q3/2019 customer webinar and we want you there! If you haven’t done so already, please register now.  Learn about new features, functionalities, and components that we have built during the first half of the year.  Additionally, see the exciting plans that we have for Keboola over the next 3-12 months.

We hope you can join us!  The webinar will be available on video by request afterward as well.

Register here.

Error when saving configuration Post-Mortem

Some of our users experienced errors in Keboola Connection (in both regions) yesterday when saving configurations of several components.

An error was displayed to users who tried to save their configuration between Jul 9, 2019, 12:35 CET and Jul 10, 2019, 14:18 CET.

Mainly users who have an older version of Chrome Browser or any version of the Firefox browser were affected.

The issue was caused by wrong response headers (unsupported by the mentioned browsers) being sent to a preflight request on one of our API services.

We sincerely apologize for the inconvenience caused.