During the maintenance, developers won't be able to access Developer Portal frontend. Connection and components execution won't be affected.
Update 7:45 UTC: The maintenance finished successfully and Developer Portal is up and fully operational.
Since last night (2021-11-01) we have been experiencing failing transformations with the error "Table already exists in workspace". We continue investigating and will keep you posted.
UPDATE 09:30 UTC - we have identified possible root cause and escalated the problem to Snowflake that seem to cause the errors. The problem is related to clone table input mapping, so workaround might to switch to standard copy table input mapping for the time being.
UPDATE 10:30 UTC - issues still persists, we are waiting for the findings from Snowflake.
UPDATE 11:15 UTC - We're closely communicating with Snowflake support and we've identified potential cause of the issue that's now shared with the engineering team. Next update in 30 minutes.
UPDATE 11:45 UTC - We are waiting for the Snowflake engineering team for the update. The issue seems to only be affecting clone mapping queries. We suspect that the issue might be related to a particular unhealthy server somewhere in the Snowflake infrastructure that the query is assigned to. So while we wait for more information from Snowflake's side, our suggestion for a workaround is to try running the affected job/query again if possible. Next update in 30 minutes.
UPDATE 12:15 UTC - We are still waiting for the Snowflake engineering team for the update. Next update in 30 minutes.
UPDATE 12:45 UTC - Snowflake informed us that they are still investigating the issue with their engineering team and keep us informed. Next update in 30 minutes.
UPDATE 13:15 UTC - Snowflake informed us they turned off the unhealthy server and asked us to test it. Our testing transformations work so the issue should be resolved now. We wait for Snowflake to confirm it is resolved. However we encourage to run your failed transformations.
UPDATE 14:10 UTC - Snowflake confirmed the issue should be resolved, at the same time continue investigating for the root cause. After more extensive testing we consider this to be resolved, that means all transformations in EU should succeed without any slowdown or temporal fail.
UPDATE 2021-01-19 11:18 UTC - We published post-mortem for this incident.
Dec 26, 2020 19:32 - We are investigating higher error rate of orchestrations and transformations in AWS EU region. Next update in 30 minutes.
Dec 26, 2020 19:40 - We've identified the issue on one of the worker nodes and we triggered the replacement. Everything should be back to normal. We'll keep monitoring our platform closely.
Dec 26, 2020 20:08 - Incident is now resolved. We apologize for the inconvenience caused by this incident.
ReportService.ReportQuery to download data.
Since 2020-11-25 13:45 UTC we have noticed an unfunctional synchronization of components between Keboola Connection and the Developer portal. The problem concerns AWS in the US region. Working in Keboola Connection is not affected by this error. It has an impact on the release and synchronization of components. You also cannot log in to the Developer Portal.
We are going to monitor the situation and keep you posted.
Update Nov 25, 2020 16:11 UTC - We have identified a problem. The outage is on the AWS side.
Update Nov 25, 2020 16:19 UTC - We found that the problem affects both AWS and Azure in all regions. Next update in 30 minutes.
Update Nov 25, 2020 16:59 UTC - Outage on the AWS side persists, see the status page. We continue to monitor the situation. Next update in 1 hour.
Update Nov 25, 2020 18:10 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.
Update Nov 25, 2020 20:15 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.
Update Nov 25, 2020 22:17 UTC - Issues on the AWS side are reducing. Login to the Developer Portal is already working and we are monitoring successful component synchronizations across all regions. Next update on Nov 26, 2020 06:30 UTC
Update Nov 26, 2020 07:05 UTC - Work on the AWS side to troubleshoot still continues, but the error rate is reduced. We are no longer experiencing any issues with component synchronizations or other Keboola Connection services. We continue to monitor the situation. Next update in 4 hours.
Update Nov 26, 2020 10:23 UTC - Issues on the AWS services are solved. The incident is now resolved.
We're investigating an issue with Sandbox API in our Azure regions. Next update in 5 minutes.
Update Nov 19, 2020 09:03 UTC - Configuration issue caused the release of Sandbox API today at 8:23 UTC that did not work properly. The problem was noticed and rollback began. At 8:58 UTC the release was rolled back. We're verifying that the issue is resolved.
Update Nov 19, 2020 09:10 UTC - We verified that the rollback restored the API to a fully working condition. The incident is now resolved.
We're investigating problems with configuration editor.
Update Nov 19, 2020 07:00 UTC - Configuration editor is broken across all regions. We're working on a fix that should be ready shortly. Sorry for inconvenience. Next update in 30 minutes.
Update Nov 19, 2020 07:19 UTC - Change causing the issue has been rolled back and we're in a process of verification that the issue is resolved. Next update in 30 minutes.
Update Nov 19, 2020 07:28 UTC - We verified that the change causing the issue is rolled back on all regions and the configuration editor is working normally. If you're still experiencing issues with the editor, please contact support individually using support button in your project.
A maintenance of Azure EU Keboola Connection (https://connection.north-europe.azure.keboola.com/) will take place on Sunday, Nov 15th, 2020 from 08:00 UTC and should take less than two hours.
During the maintenance, you can't access your projects. Orchestrations and running component jobs should be generally delayed, but not interrupted. However, feel free to re-schedule your sunday's orchestrations to avoid this maintenance window.
Update Nov 15, 2020 08:00 UTC - The maintenance started.
Update Nov 15, 2020 08:36 UTC - The maintenance is over. Azure EU Keboola Connection is up and fully operational.
We are encountering increased error rates from Snowflake in the US Region and are currently investigating the cause.
We will update here again as soon as we have relevant information or if 60 minutes have passed.
Update Oct 27, 2020 17:17 UTC: The error rate has returned to normal since 16:55 UTC. We are continuing to investigate the root cause.
Update Oct 27, 2020 18:57 UTC: Snowflake has released a status report. The errors continue to occur, but at a reduced rate.
Update Oct 27, 2020 19:29 UTC: Snowflake has reported the incident as resolved.