Delay in Telemetry Data Updates

2024-06-22 7:40 UTC Due to an outage in the Snowflake in one of the regions, Telemetry Data has not been updated since 4:30 UTC and will be delayed for the duration of the outage.

Other functions of our platform should operate normally and remain unaffected by this outage.
This issue affects all of our customers, regardless of the Storage backend they are using (Snowflake, BigQuery, etc.).

2024-06-22 8:40 UTC THe Snowfalke outage has been resolved. Telemetry data has been fully updated and is now up-to-date. Thank you for patience.

Snowflake Security Notice

We want to inform our users of a recent security concern proactively announced by Snowflake. Snowflake has released important proactive measures that our Security team is currently following. Please find instructions here - https://community.snowflake.com/s/article/Communication-ID-0108977-Additional-Information

The Keboola Security team are performing queries in both Keboola managed and unmanaged Snowflake accounts to identify activity against the published list of suspected IP addresses - following the advice provided by Snowflake. If our investigations return any activity from the identified IP addresses, we will be in touch with your Keboola Admin immediately. Furthermore we will notify users once Snowflake provide an update from their ongoing inquiry.

Slow jobs listing in AWS eu-central-1 stack

2024-05-23 12:30 UTC - We are investigation slow responses in jobs list requests that affects the UI responsiveness. 

2024-05-23 13:48 UTC -  We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.

Job delays in AWS us-east-1 stack

2024-05-20 05:53 UTC - We are investigating job delays in AWS us-east-1 stack (connection.keboola.com). The next update will be provided when we have more information.

2024-05-20 06:38 UTC - We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.

Job delays in AWS us-east-1 stack

2024-05-19 16:24 UTC - We are investigating the higher error rate and job delays in AWS us-east-1 stack (connection.keboola.com). The next update will be provided when we have more information.

2024-05-19 16:49 UTC - The platform has stabilized, and the job backlog is being cleared. Jobs were delayed but not errored as a result of this incident.

 

Failing Facebook ads component due to Facebook outage

2024-05-15 12:14 UTC We are experiencing Facebook ads extractor component to fail due to Facebook outage. Facebook reports the outage is already solved, however we are still sill seeing components failing on "Service temporarily unavailable", which is error message directly from Facebook api. We continue monitoring the issue and wait for Facebook to resolve the outage correctly. More info on https://metastatus.com/ or https://developers.facebook.com/community/threads/3832997983612514/

Failures of dbt Cloud Job Trigger component

Since 2024-05-06 21:30 UTC, the dbt Cloud Job Trigger component has been failing due to the error 'Missing columns: scribe_enabled.' We are currently investigating the root cause and will provide more information as soon as it becomes available.

UPDATE 2024-05-07 08:43 UTC - Since 08:18 UTC, we have not observed any further errors. The DBT Cloud has resumed returning the scribe_enabled field, which has resolved the failures. We are currently investigating the cause of this change on their side.

UPDATE 2024-05-07 11:08 UTC - We have continued to observe a few occurrences of errors. A new version of the component was deployed at 10:05 UTC, designed not to fail if the scribe_enabled field is missing in the response. In such cases, scribe_enabled will be set as empty.
We have not observed any further errors since the deployment of the new version. The incident is now resolved.