2023-04-12 09:50 UTC Table dependency graph shows error message on projects with queue v1. More information within the hour.
Update 2023-04-12 10:43 UTC We have deployed the latest functional version, the problem should be solved by now.
We apologize for the inconvenience.
2023-04-06 12:45 UTC We are investigating failing workspace creation on all stacks. The issue is manifesting as na Application error when you try to create a workspace. The fix is already on the way, we expect the operations to resume in 20 minutes.
2023-04-06 13:05 UTC The service disruption has been resolved and the stack is now fully operational.
We're sorry for this inconvenience.
2023-03-31 09:50 We are investigating failing workspace (Python, R, SQL) loads on all stacks.
2023-03-31 09:58 Affected are all newly created user workspaces (Python, R, SQL,.. ). A fix will be available soon. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-31 10:24 Problem occurred on when new table was added into workspace. This issue was resolved, we are now working to fix already corrupted workspaces. Workaround now is to remove tables from input mapping and add them again. The next update will be provided in 60 minutes or as soon as new information becomes available.
2023-03-31 11:00 We fixed remaining workspaces and preparing fix to prevent this problem in the future. If you encounter this issue please contact our support and mention this status post.
We sincerely apologize for any inconvenience caused and appreciate your understanding.
2023-03-30 16:22 UTC - We are investigating the delays in job start-up within the https://connection.north-europe.azure.keboola.com stack. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-30 16:54 UTC - The investigation into the cause of the issue is still ongoing. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-30 17:56 UTC - The investigation into the cause of the issue is still ongoing. The next update will be provided in 30 minutes or as soon as new information becomes available.
2023-03-30 18:55 UTC - We have identified and fixed the root cause of the issue. The job backlog has now been cleared. We will continue to monitor the situation to ensure that everything remains stable.
2023-03-30 19:18 UTC - The service disruption has been resolved and the stack is now fully operational.
2023-03-14 14:15 UTC - Since Sunday, we have been experiencing component failures when communicating with the Facebook Graph API on our Azure North Europe stack https://connection.north-europe.azure.keboola.com
Following components failing with application error:
keboola.ex-facebook-adsWe apologize for any inconvenience caused and thank you for your understanding.
2023-03-01 16:00 CET - We are investigating hidden "Transformation v2" configurations on the UI. The next update in 15 minutes or when more info is available.
2023-03-01 16:10 CET - We have identified the root cause and prepared a fix which will be deployed within 10 minutes
(Resolved) 2023-03-01 16:24 CET - The fix has been deployed and transformations v2 are no more hidden in the UI. We advise users to reload their browsers as this was an UI issue.
2023-02-21 07:41 UTC - We experienced another incident on connection.eu-central-1.keboola.com between Feb 20 18:00 UTC and Feb 21 06:40 UTC. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.
2023-02-20 15:20 UTC - A small number of jobs on the connection.eu-central-1.keboola.com stack either ended by timeout or with a "Component terminated. Possibly due to out of memory error" error message during a recent incident between Feb 19 15:10 UTC and Feb 20 14:00 UTC due to an underlying node failure. We're actively investigating the cause and taking measures to prevent this from happening again.
2023-02-20 15:56 UTC - The incident has been resolved, with the last occurrence of the error happening at Feb 20 14:35 UTC. We are continuing to monitor the situation closely to prevent any reoccurrence.
2023-02-10 09:20 UTC - We are currently investigating the problem of failing jobs on all stacks that occurred on 2023-02-09 08:48 UTC. The error is manifested by the error message "K8S request has failed: events is forbidden: User "system:serviceaccount:job-queue-jobs:daemon-service-account" cannot list resource "events" in API group "" in the namespace "job-queue-jobs"".
UPDATE 09:41 UTC: We have identified the problem and rolled back previous version of our service. All services are now operating normally.
UPDATE 10:35 UTC After a deeper research we found that this problem affected only a small fraction of the jobs.
We're sorry for this inconvenience.