Database storing locks was restarted at 03:49 UTC which caused the job failures. Also some of the jobs were queued after this failure.
The backlog of all jobs was cleared at 06:15 UTC. The system is fully operational now. We're working on infrastructure changes which should prevent similar issues.
Storage jobs processing was delayed between 04:30 - 06:04 UTC. Worker nodes didn't started as expected to process the workload.
We are investigating the issue.
We are experiencing minor job queue issues in US region which may result in delaying orchestrations. We are investigating this issue at the moment. We are sorry for inconvenience.
UPDATE, Sep 28, 06:30 UTC - We have confirmed from Snowflake support that they are tracking the issue in the increase in the compilation time of the queries and their engineering is actively investigating the issue.
UPDATE, Sep 28, 10:45 UTC - Snowflake support reported that they are experiencing a degradation of service with elevated query failures in our US region. https://support.snowflake.net/s/feed/0D50Z00008QI7AqSAL
UPDATE, Sep 28, 15:00 UTC - Snowflake support reported that they have resolved the issue and the US west region is again operating at 100%
We have launched Keboola Connection in AUS region. You can now create your projects in the AUS region. Please contact info@keboola.com if you are interested.
According to our open-source philosophy, we always want to hear new ideas from you. To make communication easier, we have added a Feature Wishlist link to Keboola Connection sidebar which leads to productboard site where you can request new features.
Some users are not able authenticate to KBC using Google Sign In. We are investigating the issue.
In the meantime, please sign in using the standard email and password method.
UPDATE, Sep 22, 07:47 UTC - We have identified the issue. Google users with some special characters in name were affected. We have made changes in interaction with Google. It should resolve this issue. Fix should be deployed in two hours.
UPDATE, Sep 22, 09:55 UTC - Fix was deployed. You should be able to authenticate to KBC using Google Sign In.
Scheduled maintenance will start at 06:00 am UTC.
The maintenance won't be system wide. It will be performed for each project in connection.keboola.com separately and it should take few minutes for each project depending on the number of project's files. It should not affect project's running jobs, these will be queued during the short project maintenance. Only projects created before August 16, 2018 will be affected by this migration.
We'll update this status about the progress.
Update, Sep 22, 06:00 UTC - Migrations have started.
Update, Sep 22, 10:11 UTC - All projects have been migrated.
Friday, September 26, 2018 04:30 UTC
Due to a configuration error the Oracle extractor was failing to process jobs from 22:00 UTC Thursday, September 20 to around 04:20 UTC on Friday, September 21, 2018.
The error has now been corrected and all new jobs will process correctly.
Sincere apologies for the error and inconvenience.
The Microsoft SQL Server Extractor will be updated to use a new driver on October 1, 2018.
This update may effect the format of datetime columns if you are using the advanced query option in your configuration.
Standard configurations (not advanced queries) will not have any change to their output.
The old driver exports datetime columns in the format `yyyy-mm-dd hh:mm:ss` however the new driver includes milliseconds when outputting datetime values `yyyy-mm-dd hh:mm:ss.ms`.
There are 2 options to mitigate this change:
1) Recommended: Switch the configuration to select the tables and columns that you need and not use an advanced query.
2) Convert the datetime output column in the SQL query, ex: CONVERT(DATETIME2(0), my_column)
If you have any concerns or questions regarding this update please contact us using the support button in your Keboola Connection project.
Thank you
We're experiencing random connection errors to Snowflake started on Sep 12, 22:08 UTC.
These errors caused a few transformations failures. We're investigating the issue and we'll update this post with our findings.
Update, Sep 14, 07:15 UTC
These rare errors are still occuring.
Update, Sep 14, 13:07 UTC
Snowflake confirms the issue is on their side and are investigating it.
A maintenance will start at 6:00 am UTC and should take less than two hours. We'll be migrating metadata of Jobs during the maintenance. All jobs will be stopped during the maintenance and should be restarted after the maintenance. Orchestrations scheduled during the maintenance period won't be started.
All projects in connection.keboola.com will be inaccessible during the maintenance.
A maintenance will start at 6:00 am UTC and should take less than two hours. We'll be migrating metadata of Jobs during the maintenance. All jobs will be stopped during the maintenance and should be restarted after the maintenance. Orchestrations scheduled during the maintenance period won't be started.
All projects in connection.eu-central-1.keboola.com will be inaccessible during the maintenance.