A scaling script running at 12:00 AM CEST failed to scale up the Snowflake warehouse in EU region. All storage and transformation jobs in the EU were affected by this issue and were significantly slower than usual.
To help process the queued load we have scaled up the warehouse at 9:45 AM CEST and will keep it running until all load is processed.
We're sorry for this inconvenience and we'll be implementing safeguards to prevent this from happening again.
We are investigating decreased performance of Snowflake in EU region which unfortunately reoccured after previous resolutions. We are in touch in with Snowflake support. Job performance and sandbox loading times may be affected. Next update at 12:30pm UTC.
Update 12:30 UTC: We are handling the performance issue with Snowflake support, we've offset the slowdown by scaling up the cluster. We'll have more information in about an hour. We caught the issue early on, so we hope it will have minimal impact on jobs, apart from a small slowdown. So far we've seen 3 job failures because of this across the whole EU region. We'll post another update at 14:30 UTC or sooner if there are any new information or situation changes.
Update 14:30 UTC: We are still working with Snowflake on resolving the issue. The situation is currently stable and we did not see any jobs failing since the last update. Our main goal is currently to mitigate the issue before the midnight job surge. Next update in 18:30 UTC or sooner if there are any new information or situation changes.
Update 18:15 UTC: We are still working on mitigating the slowdown. We've seen only 3 related job failures since last update, so we still consider the situation stable. We think that the issue will be resolved in the following hour.
Update 19:30 UTC: We're monitoring the situation and the performance is improved and close to previous values. We should have fresh aggregated monitoring data an aprroximately 15 minutes and we expect them to show complete recovery to the standard performance.
Update 20:01 UTC: The issue has been resolved.
On Dec 1st, 2019, Snowflake will turn on strict string-to-date conversion for all Keboola Connection projects.
By introducing this change, Snowflake wants to address issues such as invalid date strings rolling over. For example, 07-32 (July 32nd) has been interpreted as August 1st. Rollover will no longer be supported for dates, times, or timestamps.
A full description of affected cases and tips on what to do are provided in the Snowflake 3.45 Release Notes.
We have a list of affected queries in the Keboola Connection projects ready and will be contacting all users that may be affected by this change. You'll have plenty of time to check everything and fix any potential issue.
Thanks a lot for your cooperation.
Since October 11, 2:00 PM UTC we are experiencing failures on the Transformation Overview page and on the Storage Table Graph page in all regions with the following error message:
Cannot load GoodData Writer: Client error response [status code] 410 [reason phrase] Gone
We're investigating the issue and will update this post with our findings.
We apologize for the inconvenience.
UPDATE October 15, 5:15 PM UTC: a new version was released and the bug was fixed.
3.6.1.Since October 10, 3:00 am UTC we are experiencing higher rate of "Obsolete output mapping" user errors in transformation jobs. This issue is caused by a recent update of the transformation component. We'll be rolling back the release as soon as possible and keep you updated on further events.
UPDATE 8:45 am UTC: We have successfully deployed a previous version and all affected configurations should be working now.
UPDATE October 13, 6:11 PM UTC: A new version with a fix has been released and deployed.
We apologize for the inconvenience.
We're investigating issues with Power BI not showing tables in your Snowflake workspace.
We're sorry for this inconvenience, we'll be updating this post with our findings.
UPDATE 17:26 CET
Snowflake confirmed they can reproduce the issue and are investigating further.
The issue affects transient tables (CREATE TABLE statement), but does not affect views. As a temporary workaround we suggest to log in directly into the Snowflake workspace and create views from all tables in the schema. These views should be immediately visible in Power BI.
UPDATE 11:20 CET
Snowflake are preparing a new version of the ODBC Driver which allows Power BI to see the tables. Use the support button in your project to contact us to obtain this driver.
We're working hard every day to make Keboola Connection faster and minimize the time you spend waiting for your results. This effort includes a wide variety of components of the platform architecture. Some of the changes are straightforward and transparent to the end users, but others are unexpectedly complicated.
This is the case of transformations. We rolled out an update earlier this year and were forced to immediately rollback to the previous version as it broke the data flow in a few projects. This time, we're more prepared. We have identified the source of the incompatibility, and we'll be rolling out the update silently, only for those projects that will not be adversely affected. The projects that would be affected will not be updated. Instead, they will be notified to take steps to fix the incompatibilities. Then they'll become eligible for the update as well.
In the original system, when all transformations in a phase are executed, the output processing starts. It takes all the transformations sequentially and processes the outputs one by one. The order of the executions is not defined, but it is predictable and, most importantly, it doesn't change between runs. Some projects rely on a specific order of output processing to achieve certain goals.
To speed up the output, we have decided to queue all outputs at once and let Storage handle all jobs as fast as possible. But, as you already may have noticed, this mean the transformations can run in any order or even in parallel. This may affect the result of the output if you relied on a particular order.
The project overview and all affected transformations will notify you about multiple outputs being written to the same table in Storage. You will be easily navigated to the places that need to be fixed. Please contact our support if you need any help doing that.
Once you have fixed all instances where multiple outputs are written to the same table in your project, you can immediately contact us using the support button. We will turn on the update in your project. Or, wait until we update your project automatically (we're watching all projects regularly).
Since 2018-12-19, 19:13:00 UTC we're experiencing a higher rate of application errors in all regions due to an outage in AWS ECR.
We're investigating the issue and update this post once there.
We apologize for the inconvenience.
Update 19:30 UTC
This outage also affects the Developer Portal.
Update 20:05 UTC
As of 19:51 UTC the issue is resolved by Amazon