Broken Login [Post-Mortem]

Summary

On 2019-08-15 12:23 UTC, we deployed a broken version of Keboola Connection. It prevented some users from accessing their projects. The problem was fixed at 13:01 when we rolled back to a previous version. We sincerely apologize for interrupting your work and wasting your time.

What Happened?

There was an error in a permission check, and only users with the permission to create a project were allowed to enter a project. Such a scenario is not covered in the functional tests, and the situation was overlooked during peer review of the change. As soon as we identified the problem, we immediately deployed a previous version of Keboola Connection. That itself took about 15 minutes.

What Are We Doing About It?

We're extending the software tests to include more scenarios.

We're also updating monitoring alarms to make sure that we know about a problem before you tell us through our support channel.