2024-05-15 12:14 UTC We are experiencing Facebook ads extractor component to fail due to Facebook outage. Facebook reports the outage is already solved, however we are still sill seeing components failing on "Service temporarily unavailable", which is error message directly from Facebook api. We continue monitoring the issue and wait for Facebook to resolve the outage correctly. More info on https://metastatus.com/ or https://developers.facebook.com/community/threads/3832997983612514/
2024-05-12 We have experienced some jobs stuck to continue and failed between 03:00 - 03:45 UTC. The root cause was invalid job node, we had to shut it down to prevent further issues.
Since 2024-05-06 21:30 UTC, the dbt Cloud Job Trigger component has been failing due to the error 'Missing columns: scribe_enabled.' We are currently investigating the root cause and will provide more information as soon as it becomes available.
UPDATE 2024-05-07 08:43 UTC - Since 08:18 UTC, we have not observed any further errors. The DBT Cloud has resumed returning the scribe_enabled field, which has resolved the failures. We are currently investigating the cause of this change on their side.
UPDATE 2024-05-07 11:08 UTC - We have continued to observe a few occurrences of errors. A new version of the component was deployed at 10:05 UTC, designed not to fail if the scribe_enabled field is missing in the response. In such cases, scribe_enabled will be set as empty.
We have not observed any further errors since the deployment of the new version. The incident is now resolved.
Due to change how we allow projects manage their tokens, projects using KBC Project Metadata component (kds-team.ex-kbc-project-metadata-v2) are failing while requesting token for a given project.
This component is however experimental and not longer maintained, so we're investigating best scenarios to mitigate failing jobs in affected projects. For now, we're reverting the change. It should propagate within ~90 minutes from this post.
Thanks for understanding.
All DBT and No-code transformations have been affected by a bug and no tables were imported from these transformations to Storage (in process of output mapping). This issue has been detected, root cause identified and version with this issue was rollbacked.
DBT and no-code transforamtions running in interval 9AM CEST 2024-05-02 - 2pm CEST 2024-05-03 might be affected, and re-run is recommended.
This issue affected https://connection.north-europe.azure.keboola.com/ only.
Today since 12:00 UTC we suffer jobs as well as sync actions failing to start due to bug introduced in the last deploy. We are now reverting to the previous working version.
Update 13:15 UTC (resolved) : We have successfully reverted backend to the previous working version and platform is back operational now.
Update 13:30 UTC: Upon further investigation and damage analysis, we found that Queue v2 jobs were delayed in starting because of this incident. That could result in delays of up to one hour.
13:07 UTC: We're investigating an issue where import to Storage shows "Storage API: The new primary key cannot be created; duplicate values in primary key columns exist" error.
Next update in 15 minutes.
13:24 UTC: We've rolled back the last release. We're investigating impact of the issue.
13:54 UTC: We're still investigating the impact.
14:34 UTC: We're still investigating the impact. So far it seems to have only impacted a very limited number of customer jobs. We will reach out to the affected customers separately as soon as we finish the impact investigation.
15:09 UTC: We've finished investigating the impact and we're reaching out to the affected customers.
AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance will start today, April 13, 2024, at 12:00 UTC, as announced earlier.
You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
Maintenance should take less than three hours.
We apologize for any inconvenience caused and thank you for your understanding.
Update 12:00 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance started.
Update 13:30 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance is finished and the stack is now fully operational.
AWS us-east-1 stack (connection.keboola.com) maintenance will start today, April 06, 2024, 07:00 UTC, as announced earlier.
You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
We apologize for any inconvenience caused and thank you for your understanding.
Update 07:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance started.
Update 08:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance is finished and the stack is now fully operational.
Thank you for your patience.
A limited service disruption on Azure EU stack will start at 11:00 a.m. UTC today, as announced earlier. Storage jobs processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.
We apologize for any inconvenience caused and thank you for your understanding.
Update 11:00 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) maintenance started.
Update 11:52 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) service disruption has been resolved and the stack is now fully operational. Thank you for your patience.