We are seeing high error rate response from Sklik API
Since 2022-03-15 00:00 UTC we are experiencing jobs failures due to a shortage in Sklik API. We are going to monitor the situation and keep you posted.
Since 2022-03-15 09:30 UTC last error from Sklik API was 8:34 UTC, for now it looks like Sklik API operates normally.
We're seeing failed table loads, it seems that the cause is a problem with Amazon AWS services.
Update 21:22 UTC (Resolved): AWS confirmed recovery of the services experiencing errors hence we consider the issue to be resolved as well.
Update 21:10 UTC: AWS informed about issues in us-east-1 region and started investigation however we are not seeing anymore issues on our side and US multitenant stack seem to be fully operational now. We continue monitoring the situation.
Update 20:56 UTC: The issues started occurring at approximately at 20:44 UTC. The issue is isolated to the US multitenant stack https://connection.keboola.com/.
2022-03-08 14:20 UTC - We are investigating delayed telemetry data for projects migrated to new queue in https://connection.eu-central-1.keboola.com/ stack. There is significant drop of consumed credits in telemetry since March 4. Next update in 24 hours.
2022-03-09 08:19 UTC - Telemetry data were backfilled and telemetry reports should be updated. Please contact support if you still see incorrect data.
2022-03-02 19:03 UTC We're investigating reports that project join screen does not work. Next update in 30 minutes.
2022-03-02 19:33 UTC We've found the problematic change and initiated a rollback. Next update in 30 minutes.
2022-03-02 19:55 UTC The problem has been resolved on all stacks. Sorry for inconvenience.
In the following weeks, starting tomorrow 2.3.2022, we will be continuously migrating projects in our North Europe stack to the new version of the Job Queue.
At first glance, these are mostly cosmetic changes, but under the hood we have totally reworked how jobs are processed in order to make them more reliable, faster, and makes it easier for us to bring more features to you:
Please read more about new Job Queue and the migration here.
We will start the migration with our internal projects and in the following weeks we will migrate projects which don't have any incompatible components. (The list of incompatible components can also be found here).
If you are using one of the components from the list in you project and want your project to be migrated to the new Job Queue, please contact our support.
The migration of all projects should be completed within a month or two.
Please contact us on support@keboola.com if you have any questions about the upcoming migration or the new Job Queue in general.
We will keep you posted about the progress of the migration.
There is a problem with MFA registration in Safari for both, hardware keys like Yubikey and TouchID. The registration succeeds but it's not possible to verify the MFA upon login.
If you registered a hardware key in Safari, use other browser to log in (it will work). Then remove the registered device and re-register it using the other browser.
If you registered TouchID in Safari, please contact support to have your MFA disabled to allow you to login. After that please reregister your TouchID the using different browser.
2022-02-24 15:47 UTC - We are investigating failures of some jobs on new queue in https://connection.eu-central-1.keboola.com/ stack. Affected jobs are throwing error: Job "XYZ" terminated unexpectedly. We're currently working on mitigating the issue and will update when it gets resolved or in 60 minutes.
2022-02-24 16:05 UTC - The errors were occurring between 15:00 - 15:30. The issue is now resolved, we are monitoring the situation and investigating root cause of error.
2022-02-24 16:10 UTC - We have seen few reoccurrences of the issue. We're currently working on mitigating the issue and will update when it gets resolved or in 60 minutes.
2022-02-24 17:19 UTC - Issue is resolved, platform is stable. Last error at 16:30. Issue was caused by overload of one of the nodes, we've upgraded the nodes and we are working on better resources management to prevent similar issues. We apologize for the inconvenience.
2022-02-21 13:30 UTC We discovered an issue that may cause it not possible to create new projects on Azure North Europe. So it is not possible to create new Pay as you go projects.
We're currently working on mitigating the issue and will update when it gets resolved or in 60 minutes.
Update 14:30 UTC The issue is now fixed.
Due to a misconfiguration, there was a brief error present in job processing in https://connection.keboola.com/ https://connection.eu-central-1.keboola.com/ stacks which caused some jobs to fail (14 jobs was affected). The error was present from 12:10:01 to 12:14:07 CET. We apologize for the inconvenience.
We see cases of orchestrations skipping their tasks in phases. The symptom is that orchestration finishes with success but in very short time with no tasks executed. We do not know yet what are the exact properties of an orchestration to trigger the issue.
We are rolling back todays changes in orchestration processing which should fix this issue.
Update 16:15 UTC The issue is now fixed.