Since 2021-07-20 17:00 UTC some jobs processing may be delayed and job termination requests may be unsuccessful in all Azure stacks. The total number of affected jobs or requests is very small.
This bug was introduced due to a network settings change. The change has been reverted and is currently being deployed to all Azure stacks. If you experience any of the mentioned symptoms please get in touch with our support so we can mitigate the issue faster.
We're very sorry for this inconvenience.
We're seeing increased user error rate in Python transformation jobs since May 25 16:08 UTC. Most common error message is "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed." but also other errors may appear.
We're sorry for this inconvenience, we're actively investigating the issue. Next update in 30 minutes
Update 07:28 UTC: We have rolled back Python transformations from version 1.4.1 to version 1.4.0 and everything seems to be working again. If you encounter further issues please get in touch with us. We'll be monitoring this issue and post an update in one hour.
Update 08:36 UTC: The situation is stable, we're not seeing any further errors since the rollback. Please accept our apology for the inconvenience.
From 2021-05-18 21:13 UTC to 2021-05-19 05:06 UTC we experienced increased jobs failures in the Azure North Europe region.
These errors were caused by a faulty deploy on a single node. The deploy has been fixed and the situation is back to normal and you can restart the failed jobs.
We're sorry for this inconvenience.
Since 2021-04-21 10:00 UTC we're seeing increased error rate in Snowflake connections. Users may experience failed Snowflake transformation jobs. We're investigating the root cause.
UPDATE 18:00 UTC: The last error occurred on 2021-04-22 11:24 UTC and we haven't seen any further failures since then. All operations back to normal. We're in touch with Snowflake support to find the root cause and prevent this from happening in the future.
We're sorry for this inconvenience and thanks for your understanding.
March 15, 2021, 20:16 UTC We're seeing an authentication errors of multiple Microsoft or Azure related services. This affects all stacks and all regions. In AWS regions only some component jobs are affected (mainly PowerBI writer, OneDrive Excel Sheets writer, OneDrive Excel Sheets extractor). in Azure regions more services may be affected.
You may see application errors or increased job running times.
See https://status.azure.com/en-us/status for more details.
Next update in 60 minutes.
UPDATE March 15, 2021, 21:44 UTC - Engineers at Azure are currently rolling out mitigation worldwide. Full mitigation expected within 60 minutes.
UPDATE March 16, 2021, 0:16 UTC - Engineers at Azure have rolled out a fix to all affected regions for Azure AD. Internal telemetry and customer reports suggests that the error rate for dependent services is rapidly decreasing. Microsoft services are in the process of recovery. Some services may have varying times of recovery following the underlying fix. The next update will be provided at 8:00 UTC.
UPDATE March 16, 2021, 7:52 UTC - The issue related to Azure Active Directory has been resolved. Azure reports problems with authentication to the Storage Service now, but we don't see any impact on Keboola Connection. The next update will be provided at 12:00 UTC.
UPDATE March 16, 2021, 11:23 UTC - Issues on the Azure services are solved. The incident is now resolved.
We have identified a system wide failure of the deprecated RTB House extractor by Revolt BI (revolt-bi.ex-rtbhouse). The API version used in this extractor was just deprecated on the service side.
Please upgrade your configurations to the new RTB House extractor - https://components.keboola.com/components/revolt-bi.ex-rtb-house-v5.
A scaling script running at 12:00 AM CEST failed to scale up the Snowflake warehouse in EU region. All storage and transformation jobs in the EU were affected by this issue and were significantly slower than usual.
To help process the queued load we have scaled up the warehouse at 9:45 AM CEST and will keep it running until all load is processed.
We're sorry for this inconvenience and we'll be implementing safeguards to prevent this from happening again.
We are investigating decreased performance of Snowflake in EU region which unfortunately reoccured after previous resolutions. We are in touch in with Snowflake support. Job performance and sandbox loading times may be affected. Next update at 12:30pm UTC.
Update 12:30 UTC: We are handling the performance issue with Snowflake support, we've offset the slowdown by scaling up the cluster. We'll have more information in about an hour. We caught the issue early on, so we hope it will have minimal impact on jobs, apart from a small slowdown. So far we've seen 3 job failures because of this across the whole EU region. We'll post another update at 14:30 UTC or sooner if there are any new information or situation changes.
Update 14:30 UTC: We are still working with Snowflake on resolving the issue. The situation is currently stable and we did not see any jobs failing since the last update. Our main goal is currently to mitigate the issue before the midnight job surge. Next update in 18:30 UTC or sooner if there are any new information or situation changes.
Update 18:15 UTC: We are still working on mitigating the slowdown. We've seen only 3 related job failures since last update, so we still consider the situation stable. We think that the issue will be resolved in the following hour.
Update 19:30 UTC: We're monitoring the situation and the performance is improved and close to previous values. We should have fresh aggregated monitoring data an aprroximately 15 minutes and we expect them to show complete recovery to the standard performance.
Update 20:01 UTC: The issue has been resolved.
On Dec 1st, 2019, Snowflake will turn on strict string-to-date conversion for all Keboola Connection projects.
By introducing this change, Snowflake wants to address issues such as invalid date strings rolling over. For example, 07-32 (July 32nd) has been interpreted as August 1st. Rollover will no longer be supported for dates, times, or timestamps.
A full description of affected cases and tips on what to do are provided in the Snowflake 3.45 Release Notes.
We have a list of affected queries in the Keboola Connection projects ready and will be contacting all users that may be affected by this change. You'll have plenty of time to check everything and fix any potential issue.
Thanks a lot for your cooperation.
Since October 11, 2:00 PM UTC we are experiencing failures on the Transformation Overview page and on the Storage Table Graph page in all regions with the following error message:
Cannot load GoodData Writer: Client error response [status code] 410 [reason phrase] Gone
We're investigating the issue and will update this post with our findings.
We apologize for the inconvenience.
UPDATE October 15, 5:15 PM UTC: a new version was released and the bug was fixed.