Keboola Status

We're investigating problems with configuration editor. 

Update Nov 19, 2020 07:00 UTC - Configuration editor is broken across all regions. We're working on a fix that should be ready shortly. Sorry for inconvenience. Next update in 30 minutes. 

Update Nov 19, 2020 07:19 UTC - Change causing the issue has been rolled back and we're in a process of verification that the issue is resolved. Next update in 30 minutes. 

Update Nov 19, 2020 07:28 UTC - We verified that the change causing the issue is rolled back on all regions and the configuration editor is working normally. If you're still experiencing issues with the editor, please contact support individually using support button in your project. 

A maintenance of Azure EU Keboola Connection (https://connection.north-europe.azure.keboola.com/) will take place on Sunday, Nov 15th, 2020 from 08:00 UTC  and should take less than two hours.

During the maintenance, you can't access your projects. Orchestrations and running component jobs should be generally delayed, but not interrupted.  However, feel free to re-schedule your sunday's orchestrations to avoid this maintenance window.

Update Nov 15, 2020 08:00 UTC  - The maintenance started.

Update Nov 15, 2020 08:36 UTC  - The maintenance is over. Azure EU Keboola Connection is up and fully operational.

We are encountering increased error rates from Snowflake in the US Region and are currently investigating the cause.
We will update here again as soon as we have relevant information or if 60 minutes have passed.

Update Oct 27, 2020 17:17 UTC:  The error rate has returned to normal since 16:55 UTC.  We are continuing to investigate the root cause.

Update Oct 27, 2020 18:57 UTC: Snowflake has released a status report.  The errors continue to occur, but at a reduced rate.

Update Oct 27, 2020 19:29 UTC: Snowflake has reported the incident as resolved.

We are experiencing a minor slow down of performance on Snowflake in the EU Region and are currently investigating.
There are no job failures or increased queue backlog but everything seems to run slightly slower.
The next update will be in 120 minutes or as new information becomes available.

Update October 26 2020, 12:50 UTC: Performance has for the most part recovered, the rate of long execution of some queries has dropped significantly, but hasn't quite completely disappeared.  We will update again if the issue re-surfaces.

New Components

• POP3 Attachment extractor – allows you to download email attachments from a POP3 mailbox. There is no restriction on attachment type, so you can also download XLS files or images.

Updated Components

• MongoDB extractor – added an option to “Include parent in Primary Key” in Mapping mode. This will allow better PK generation and the behavior will be much more consistent with Generic Extractor.

• Looker writer – added support for Google BigQuery as the underlying database.

UI Improvements

• If there's a name collision, a numeric suffix is added (for example, if there is an account table from another bucket and you add a new bucket which has a table with the same name, it will be named account_2).
• If there are already tables from the bucket, only the remaining tables will be added.

Force Unlink

There’s a new option to “force unlink” a bucket from projects that have it linked. When you disable sharing, it will “force unlink” the bucket automatically.

Edit Bucket Sharing

From now on, if there are projects that have the bucket linked, you can only add more projects/users. Edit sharing is available from “shared from this projects” list and from “bucket detail in data catalog” pages (if you have sufficient privileges). Until the bucket is linked, sharing can be changed without limitations.

Since 2020-10-15 10:02 UTC Start investigating overload in EU. Next update in 1 hour or as new information

UPDATE 2020-10-15 11:14 UTC: All operations are back to normal and everything is fully working. No jobs should be affected.

We experienced overload in EU

Since 2020-10-12 11:40 UTC Start investigating overload in EU. Next update in 1 hour or as new information becomes available.

UPDATE 2020-10-12 12:56 UTC:  Everything should be back to normal. No jobs should be affected. We'll keep monitoring our platform closely.

UPDATE 2020-10-12 13:57 UTC: All operations are back to normal and everything is fully working.

Dear customers,

We are in the final phase of shifting and finalizing the calculation of time credits. We thank you for your patience and partnership in working with us and transitioning your contracts over to the new credit system.

As a part of finalizing, we are implementing some changes that weren’t applied previously and will begin to calculate starting now.

• Transformations with multiple transformation backends - If you have a transformation that runs an SQL transformation with a R/Python transformation following it, in the same bucket, the SQL transformation is being calculated until the R/Python transformations are finished. As a result, these SQL jobs will have its child jobs run time deducted from their own, resulting in lower time credits consumption of affected clients. The change will cover jobs since the beginning of 2020.
• SQL sandboxes time credits usage - Most of our clients don’t have SQL sandboxes consumption added to their overall consumption. In our analysis, this is a fraction of total consumption, so the impact is not very material for the majority of clients. For those clients that have a material impact, your CSM will be reaching out to you to let you know the impact, otherwise you can assume your usage is not going to impact you.
• COVID-19 Error jobs - We haven’t been calculating credits for error jobs since April 2020 due to Covid and a higher than normal rate of Snowflake related issues. We will begin adding the consumption from job errors starting October.

Telemetry Component

We’re planning to release a new Keboola Connection component, which will allow the users to get telemetry data about their project or organization. That will eventually replace our temporary solutions like GoodData telemetry dashboard or direct writing of telemetry data to some projects. In the end, all of the Keboola Connection users will have easy access to documented telemetry data on demand.

We experienced Connection API higher error rate and issue with starting new Python sandboxes between 17:45 - 17:55 UTC. The issue is now resolved, jobs processing wasn't affected. 

We're investigating an issue with Snowflake in US region which causes some Storage operations with tables to be stuck in processing state. This can cause jobs to be executing longer than expected or seemingly "forever". Terminating and restarting the job does not help in such a case. Only certain projects are affected.

Next update in 1 hour or as new information becomes available.

Update 10:50 UTC: The stuck jobs are unblocked now and should be finishing, we're monitoring the situation if the issue reappears. A post mortem will be published once we get an RCA from Snowflake.