2022-01-18 10:30 UTC We are investigating an issue when a created workspace is inaccessible from UI and the request for workspace detail ends with error:
There was an error in the platform causing failure during the creation of projects for pay-as-you-go customers since yesterday, 16 January 2022, 14:30 UTC. The problem was fixed a moment ago and we are investigating its cause.
January 19 2022, 10:00 UTC - all users which registered in pay-as-you-go (between Jan 15 22:00:00 UTC and January 16 2022, 14:30 UTC), should be now able to access their projects.
In case of redirect to last step of pay-as-you-go, cookies for domain https://connection.north-europe.azure.keboola.com/ must be cleared (closing browser application should be sufficient)
If closing browser did not help, clear cookies according instruction for you browser:
Chrome https://support.google.com/chrome/answer/95647?hl=en&co=GENIE.Platform%3DDesktop
Firefox https://support.mozilla.org/en-US/kb/clear-cookies-and-site-data-firefox
Safari https://support.apple.com/guide/safari/manage-cookies-sfri11471/mac
2022-01-12 9:05 UTC There is sudden increased traffic on our EU Snowflake warehouse so that we upgraded it to larger instance and the queued jobs were immediately processed. The delay should be fixed soon. We are still monitoring the warehouse until the traffic is settled down.
UPDATE 12:05 UTC - Previously announced maintenance of connection.eu-central-1.keboola.com will start in one hour at 13:00 UTC. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.
UPDATE 13:00 UTC - EU stack (connection.eu-central-1.keboola.com) maintenance started.
UPDATE 14:53 UTC - EU stack (connection.eu-central-1.keboola.com) is finished. Platform should be stable, we continue to monitor it.
UPDATE 06:54 UTC - Previously announced maintenance of connection.keboola.com will start in one hour. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.
UPDATE 08:00 UTC - US stack (connection.keboola.com) maintenance started.
UPDATE 10:43 UTC - US stack (connection.keboola.com) is finished. Platform should be stable we continue to monitor it.
The AWS service in the US-EAST-1 region where we operate the AWS US stack is disrupted by networking connectivity issues of some instances in one availability zone. So far our service does not seem to be directly affected but it may reach some of its parts eventually. We are monitoring the situation and let you know in an hour about its progress.
UPDATE: The Keboola Academy site is down too most probably due to this outage.
UPDATE 14:30 CET: The problem was identified by the AWS team (a power outage in one data center in USE1-AZ4 availability zone) and they are already restoring the power and recovering from the problem. It seems that some other services like Slack and SolarWinds Papertrail were affected too but the Connection seems to be unaffected except for some short job processing delays. We are still monitoring the situation and let you know about the situation in an hour.
UPDATE 15:30 CET: Power to all affected instances and network devices was restored and recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone is being seen. The impact on the Connection stack should be almost zero by now.
A maintenance of AWS stacks Keboola Connection will take place on Saturday, Jan 8th, 2022 and should take less than three hours.
connection.keboola.com - 09:00 CET
connection.eu-central-1.keboola.com - 12:00 CET
During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.
All running tasks will be monitored by us and restarted in case of any interruption. Orchestrations and running transformations will be generally delayed, but not interrupted. However, feel free to re-schedule your saturday's orchestrations to avoid this maintenance window.
2021-12-16 17:40 UTC We are experiencing number of jobs in waiting state more than usual. We continue investigating the issue.
2021-12-16 18:45 UTC The issue has been resolved, everything is working as expected.
2021-12-16 19:10 UTC Further investigation revealed the parallel config rows execution might have been affected leaving some jobs stuck. Please review your jobs run as a configuration in parallel, terminate such jobs if they seem to be stuck and run them again.
Regarding the security issue (CVE-2021-44228) with the Log4j zero-day vulnerability, we have completed all necessary steps to investigate if our system had been compromised.
Please reach out if you have any questions.