Delayed processing of jobs in AWS EU stack

2022-04-01 07:36 UTC We are experiencing higher number of jobs in waiting state more than usual. We are investigating the issue. Next update in one hour or when new information will be available.

2022-04-01 08:36 UTC Backlog is cleared, delays were caused by increased traffic.

Scheduled Maintenance (Azure North Europe Region)

Scheduled maintenance of Azure North Keboola Connection (https://connection.north-europe.azure.keboola.com) will take place on Thursday, Mar 31th 2022 at 05:00 pm UTC and should take less than one hour. 

It should not affect project's running jobs, these will be queued during the short project maintenance. Orchestrations and running transformations will be generally delayed, but not interrupted. 

During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

We'll update this status about the progress.


UpdateMar 31th, 16:57 UTC - All changes done, in the end not maintenance was necessary.

Some Python Transformations Failing on Date Parsing Error

2022-03-16 12:45 UTC
We have discovered some failing python transformations that throw an exception with the message "bad escape \d at position".  This error was caused by a breaking change to the underlying regular expression library.

If your transformation is failing in this way you can fix it by specifying "regex==2022.03.02" in the packages input.

2022-03-18 15:00 UTC regex package is not part of our base images, so there will be no further actions. Users have set working version of package in their dependencies


High error rate from Sklik API

We are seeing high error rate response from Sklik API

Since 2022-03-15 00:00 UTC we are experiencing jobs failures due to a shortage in Sklik API. We are going to monitor the situation and keep you posted.


Since 2022-03-15 09:30 UTC last error from Sklik API was 8:34 UTC, for now it looks like Sklik API operates normally.

Errors When loading Tables in AWS US region

We're seeing failed table loads, it seems that the cause is a problem with Amazon AWS services.

Update 21:22 UTC (Resolved): AWS confirmed recovery of the services experiencing errors hence we consider the issue to be resolved as well.

Update 21:10 UTC: AWS informed about issues in us-east-1 region and started investigation however we are not seeing anymore issues on our side and US multitenant stack seem to be fully operational now. We continue monitoring the situation.

Update 20:56 UTC: The issues started occurring at approximately at 20:44 UTC. The issue is isolated to the US multitenant stack https://connection.keboola.com/. 


Joining project does not work

2022-03-02 19:03 UTC We're investigating reports that project join screen does not work. Next update in 30 minutes. 

2022-03-02 19:33 UTC We've found the problematic change and initiated a rollback. Next update in 30 minutes.

2022-03-02 19:55 UTC The problem has been resolved on all stacks. Sorry for inconvenience. 

Migration to new Job Queue in North Europe

In the following weeks, starting tomorrow 2.3.2022, we will be continuously migrating projects in our North Europe stack to the new version of the Job Queue.

At first glance, these are mostly cosmetic changes, but under the hood we have totally reworked how jobs are processed in order to make them more reliable, faster, and makes it easier for us to bring more features to you:

  • Dynamic backends
  • Parallel execution of configuration rows
  • Long running jobs
  • Versioning and trash for Orchestrator configurations
  • More concise API of the individual services

Please read more about new Job Queue and the migration here.

We will start the migration with our internal projects and in the following weeks we will migrate projects which don't have any incompatible components. (The list of incompatible components can also be found here). 
If you are using one of the components from the list in you project and want your project to be migrated to the new Job Queue, please contact our support.

The migration of all projects should be completed within a month or two.

Please contact us on support@keboola.com if you have any questions about the upcoming migration or the new Job Queue in general.

We will keep you posted about the progress of the migration.

Multi-factor authentication devices registered via Safari do not work

There is a problem with MFA registration in Safari for both, hardware keys like Yubikey and TouchID. The registration succeeds but it's not possible to verify the MFA upon login.

If you registered a hardware key in Safari, use other browser to log in (it will work). Then remove the registered device and re-register it using the other browser.

If you registered TouchID in Safari, please contact support to have your MFA disabled to allow you to login. After that please reregister your TouchID the using different browser.