Investigating: Legacy Sandbox throws Internal error on create from transformation

2022-01-19 07:45 UTC We're investigating issue in Legacy Sandboxes, internal error is being thrown when user tries to create sandbox from transformation

2022-01-19 10:57 UTC We've identified the issue and prepared a fix. We will let you know as soon as the fix is deployed.

2022-01-19 15:52 UTC The fix has been deployed and Sandboxes creation works as expected. We are sorry for any inconvenience.

Investigating: Configuration row of copied configurations can not be deleted

2022-01-18 18:15 UTC We're investigating issue, where rows of configuration, which was copied from another configuration cannot be deleted nor updated. 

2022-01-18 18:55 UTC We've identified the issue is related to internal database migration. Only configurations of components with rows, such as DB writers and extractors are affected.

2022-01-18 19:36 UTC We know what causes the problem and we are working on a fix. We will let you know as soon as the fix is deployed.

2022-01-19 9:30 UTC The fix has been deployed. We are sorry for any inconvenience.

2022-01-19 11:32 UTC We're preparing a report with the list of affected configurations and next steps. At the moment, we advise users to delete affected (copied) configurations and create new copy.

Investigating: Buckets can't be created from UI

2022-01-18 15:04 UTC We have identified a bug in latest release of Keboola Connection. Because of it buckets cannot be created from the UI. They can still be created via API. We are reverting to previous version. We will inform you once the previous working version is deployed.

UPDATE 2022-01-18 15:53 UTC The issue has been resolved, everything is working as expected.

Investigating issue with inaccessible workspaces

2022-01-18 10:30 UTC We are investigating an issue when a created workspace is inaccessible from UI and the request for workspace detail ends with error: 

2022-01-18 10:50 UTC We have identified the problem and we're working on a fix. Only workspaces created from transformations were affected.

2022-01-18 17:00 UTC The fix has been deployed, the workspaces now work as expected.

Failing creation of projects for pay-as-you-go customers

There was an error in the platform causing failure during the creation of projects for pay-as-you-go customers since yesterday, 16 January 2022, 14:30 UTC. The problem was fixed a moment ago and we are investigating its cause.


January 19 2022, 10:00 UTC - all users which registered in pay-as-you-go (between Jan 15 22:00:00 UTC and January 16 2022, 14:30 UTC), should be now able to access their projects. 

In case of redirect to last step of pay-as-you-go, cookies for domain https://connection.north-europe.azure.keboola.com/ must be cleared (closing browser window should be sufficient) 

Clear cookies help:

Chrome https://support.google.com/chrome/answer/95647?hl=en&co=GENIE.Platform%3DDesktop

Edge https://support.microsoft.com/en-us/microsoft-edge/delete-cookies-in-microsoft-edge-63947406-40ac-c3b8-57b9-2a946a29ae09

Firefox https://support.mozilla.org/en-US/kb/clear-cookies-and-site-data-firefox

Safari https://support.apple.com/guide/safari/manage-cookies-sfri11471/mac


Planned maintenance due to table migration

In order to keep our codebase clean, we occasionally have to make large changes to it. This time we have to migrate all projects on all stacks to use the new code.
This is an internal change only. Afterwards, you will not notice anything different.

The migration will take place between January 17 and January 21, 2022. During the migration, we will disable all projects for a few seconds to convert the data.
All your configurations will remain intact, and your usage of production and development branches will be unchanged.


Delayed processing of job in AWS EU stack

2022-01-12 8:40 UTC We are experiencing number of jobs in waiting state more than usual. We are investigating the issue.

2022-01-12 9:05 UTC There is sudden increased traffic on our EU Snowflake warehouse so that we upgraded it to larger instance and the queued jobs were immediately processed. The delay should be fixed soon. We are still monitoring the warehouse until the traffic is settled down.

connection.eu-central-1.keboola.com maintenance

UPDATE 12:05 UTC - Previously announced maintenance of connection.eu-central-1.keboola.com will start in one hour at 13:00 UTC. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

UPDATE 13:00 UTC - EU stack (connection.eu-central-1.keboola.com) maintenance started.

UPDATE 14:53 UTC - EU stack (connection.eu-central-1.keboola.com) is finished. Platform should be stable, we continue to monitor it.

connection.keboola.com maintenance

UPDATE 06:54 UTC - Previously announced maintenance of connection.keboola.com will start in one hour. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

UPDATE 08:00 UTC - US stack (connection.keboola.com) maintenance started.

UPDATE 10:43 UTC - US stack (connection.keboola.com) is finished. Platform should be stable we continue to monitor it.

High API errors rate in AWS service

The AWS service in the US-EAST-1 region where we operate the AWS US stack is disrupted by networking connectivity issues of some instances in one availability zone. So far our service does not seem to be directly affected but it may reach some of its parts eventually. We are monitoring the situation and let you know in an hour about its progress.

UPDATE: The Keboola Academy site is down too most probably due to this outage.

UPDATE 14:30 CET: The problem was identified by the AWS team (a power outage in one data center in USE1-AZ4 availability zone) and they are already restoring the power and recovering from the problem. It seems that some other services like Slack and SolarWinds Papertrail were affected too but the Connection seems to be unaffected except for some short job processing delays. We are still monitoring the situation and let you know about the situation in an hour.

UPDATE 15:30 CET: Power to all affected instances and network devices was restored and recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone is being seen. The impact on the Connection stack should be almost zero by now.