Keboola Status

Introducing Keboola Community!

2021-01-21T16:12:37Z

We are proud to announce that we have just launched Keboola Community!

Keboola Community is the place where you can find all feature announcements, share advice, submit feedback, and contribute to discussions about Keboola. So from now on, you can find all news about Keboola on the community.keboola.com website.

This page (status.keboola.com) will still be providing updates about the performance status and ongoing incidents of the Keboola Connection platform.

We are looking forward to seeing you on both websites!

Investigating higher latency (North Europe Azure)

2021-01-21T14:28:04Z

As of January 21 12:10 UTC we are investigating higher latency on some requests in North Europe Azure stack.

It might leads to errors in the UI.

Job processing is not affected.

UPDATE 2020-01-21 18:00 UTC - All operations are back to normal and everything is fully working.

Post-mortem: Failing transformation in EU region

2021-01-19T11:14:41Z

This is a post-mortem of the Failing transformation in EU region incident.

We received a root-cause analysis from Snowflake and learned that the problem was caused by a DNS resolution issue. The nodes could not reach the NTP service, causing some clock skew and failed jobs for several Keboola customers. Snowflake replaced the nodes that were affected and resolved the issue.

We learned that even the jobs that did not fail might have produced incorrect results as a result of a few seconds of time skew. Server-time-related functions (current_timestamp(), current_time() and current_date()) returned values that were skewed by a few seconds from the actual time. The data would only be affected if the query modifying it used the above mentioned functions.

We're working on a report of affected queries internally. You can contact us via standard support channels if you think your queries are using server time in a way in which a few seconds of skew would affect the results. We'll work with you on checking the queries as soon as we have the report ready.

Investigating decreased Snowflake performance

2021-01-19T10:47:03Z

As of January 15th we are experiencing decreased Snowflake performance in peak times resulting in slower query executions or even query timeouts.

We are actively working on this case with Snowflake support and we shall have a further update at 14:00 UTC.

We are sorry for the inconvenience.

UPDATE 19.1. 14:00 UTC - Snowflake support is continuing with a deep investigation of the problem. Next update should be 20.1. around 8:30 UTC.

UPDATE 20.1. 8:30 UTC - We have boosted the Snowflake Warehouse size to compensate for longer running queries. And we are still waiting for the update from Snowflake support.

UPDATE 20.1. 14:00 UTC - Snowflake is raising the priority of the issue internally, they've acknowledged there's something wrong with the duration of some queries. Next update will be tomorrow (21.1.) morning.

UPDATE 21.1. 13:00 UTC - Snowflake engineering has made some changes (around 4AM UTC) in our account, which they believe should help get performance back on track. We will continue monitoring the case for 24 hours, to see if the changes really helped.

UPDATE 22.1. 9.00 UTC - Both Snowflake engineering and we at Keboola can confirm that queries average times have decreased dramatically over last 24 hours. It seems like the problem has been resolved by Snowflake. We have brought our Snowflake warehouse back to its original size with boost window between 00:00 and 06:30. We will continue to monitor the case for another 24 hours

Developer Portal Maintenance Announcement

2021-01-13T10:23:05Z

Maintenance of Keboola Developer Portal (https://components.keboola.com) will take place on Saturday, Jan 16, 2021, at 07:00 UTC and should take less than an hour.

During the maintenance, developers won't be able to access Developer Portal frontend. Connection and components execution won't be affected.

Update 7:45 UTC: The maintenance finished successfully and Developer Portal is up and fully operational.

[Resolved] Failing transformation in EU region

2021-01-11T08:13:42Z

Since last night (2021-11-01) we have been experiencing failing transformations with the error "Table already exists in workspace". We continue investigating and will keep you posted.

UPDATE 09:30 UTC - we have identified possible root cause and escalated the problem to Snowflake that seem to cause the errors. The problem is related to clone table input mapping, so workaround might to switch to standard copy table input mapping for the time being.

UPDATE 10:30 UTC - issues still persists, we are waiting for the findings from Snowflake.

UPDATE 11:15 UTC - We're closely communicating with Snowflake support and we've identified potential cause of the issue that's now shared with the engineering team. Next update in 30 minutes.

UPDATE 11:45 UTC - We are waiting for the Snowflake engineering team for the update. The issue seems to only be affecting clone mapping queries. We suspect that the issue might be related to a particular unhealthy server somewhere in the Snowflake infrastructure that the query is assigned to. So while we wait for more information from Snowflake's side, our suggestion for a workaround is to try running the affected job/query again if possible. Next update in 30 minutes.

UPDATE 12:15 UTC - We are still waiting for the Snowflake engineering team for the update. Next update in 30 minutes.

UPDATE 12:45 UTC - Snowflake informed us that they are still investigating the issue with their engineering team and keep us informed. Next update in 30 minutes.

UPDATE 13:15 UTC - Snowflake informed us they turned off the unhealthy server and asked us to test it. Our testing transformations work so the issue should be resolved now. We wait for Snowflake to confirm it is resolved. However we encourage to run your failed transformations.

UPDATE 14:10 UTC - Snowflake confirmed the issue should be resolved, at the same time continue investigating for the root cause. After more extensive testing we consider this to be resolved, that means all transformations in EU should succeed without any slowdown or temporal fail.

UPDATE 2021-01-19 11:18 UTC - We published post-mortem for this incident.

Orchestration and transformation errors in EU AWS region

2020-12-26T19:33:54Z

Dec 26, 2020 19:32 - We are investigating higher error rate of orchestrations and transformations in AWS EU region. Next update in 30 minutes.

Dec 26, 2020 19:40 - We've identified the issue on one of the worker nodes and we triggered the replacement. Everything should be back to normal. We'll keep monitoring our platform closely.

Dec 26, 2020 20:08 - Incident is now resolved. We apologize for the inconvenience caused by this incident.

Week in Review - December 11th, 2020

2020-12-11T08:49:47Z

New Components

CosmosDB SQL API Extractor – exports data from the NoSQL Azure Cosmos DB using the SQL API.
Azure Storage Table Extractor – exports data from the Azure Table storage or Azure Cosmos DB Table API.
Azure Cost Management Extractor – exports cost and usage data of an Azure subscription.
Azure Event Hub Writer – publishes events to the Azure Event Hubs.

Updated Components

Generic extractor – supports client certificates. This is needed if the server requires two-way SSL authentication, so in addition to the verification of the server, the server also verifies the client.

Week in Review - December 7th, 2020

2020-12-07T08:07:39Z

General updates

New wishlist – the old wishlist was replaced with a new, more transparent one. You can find it at https://ideas.keboola.com or in the "Settings Menu" in your projects.
New Keboola Connection instance running on Azure is available in Europe – https://connection.north-europe.azure.keboola.com
SQLdep removed – we were forced to remove SQLdep from Keboola Connection as the service is shutting down.
New on-boarding experience - starting from November 16, 2020, new users will no longer get Guide Mode projects. They were replaced by a new interactive wizard inside each project.
Migration of projects to Azure stacks – it's now possible to migrate projects in AWS stacks to new Azure stacks [see the documentation].
Shared workspaces – this feature is available only to customers with new transformations.

If you share a workspace, all project users will be able to use/restore it. Only workspace creators/owners can delete workspaces.

In the list of workspaces, there’s a new column indicating if a workspace is yours and if it is shared or not.

New Components

Google Ad Manager Report Service exctractor – downloads reports from the Google Ad Manager API. It uses ReportService.ReportQuery to download data.
Generic writer – sends data to REST-like APIs either as JSON or as a BINARY payload.
B!Q GoodData Telemetry extractor – uses the GoodData API to get info about users, projects, dashboards, reports, and uploaded datasets.
Klaviyo extractor – fetches customer profiles, lists, campaigns, and metrics from the Klaviyo email marketing platform.
Klaviyo writer – creates/updates customer profiles and manages a list of memberships and subscriptions on the Klaviyo email marketing platform.
Azure Blob Storage V2 extractor – extracts a single or multiple CSV files from a single or nested folders from an Azure Blob Container and stores them into Keboola Connection.
Looker Schedule Runner – forces a new cache on your dashboards by triggering a dashboard export, which will execute a query and refresh the cache of your dashboard.

Updated Components

FTP extractor – has a new option “Ignore passive address”. It solves the problem when a server offers a random port for communication but the port is blocked by firewall.

UI Improvements

Input Mapping – data types are available for the Snowflake transformation. This feature is available in new PAYG projects and for customers with new transformations only [see the documentation].
Option validate and SQLdep removed

Transformations have been split into Transformations and Workspaces (applies to new transformations only).

Component synchronization down in all regions

2020-11-25T15:55:48Z

Since 2020-11-25 13:45 UTC we have noticed an unfunctional synchronization of components between Keboola Connection and the Developer portal. The problem concerns AWS in the US region. Working in Keboola Connection is not affected by this error. It has an impact on the release and synchronization of components. You also cannot log in to the Developer Portal.

We are going to monitor the situation and keep you posted.

Update Nov 25, 2020 16:11 UTC - We have identified a problem. The outage is on the AWS side.

Update Nov 25, 2020 16:19 UTC - We found that the problem affects both AWS and Azure in all regions. Next update in 30 minutes.

Update Nov 25, 2020 16:59 UTC - Outage on the AWS side persists, see the status page. We continue to monitor the situation. Next update in 1 hour.

Update Nov 25, 2020 18:10 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.

Update Nov 25, 2020 20:15 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.

Update Nov 25, 2020 22:17 UTC - Issues on the AWS side are reducing. Login to the Developer Portal is already working and we are monitoring successful component synchronizations across all regions. Next update on Nov 26, 2020 06:30 UTC

Update Nov 26, 2020 07:05 UTC - Work on the AWS side to troubleshoot still continues, but the error rate is reduced. We are no longer experiencing any issues with component synchronizations or other Keboola Connection services. We continue to monitor the situation. Next update in 4 hours.

Update Nov 26, 2020 10:23 UTC - Issues on the AWS services are solved. The incident is now resolved.

Investigating Sandbox API issues in Azure regions

2020-11-19T09:02:53Z

We're investigating an issue with Sandbox API in our Azure regions. Next update in 5 minutes.

Update Nov 19, 2020 09:03 UTC - Configuration issue caused the release of Sandbox API today at 8:23 UTC that did not work properly. The problem was noticed and rollback began. At 8:58 UTC the release was rolled back. We're verifying that the issue is resolved.

Update Nov 19, 2020 09:10 UTC - We verified that the rollback restored the API to a fully working condition. The incident is now resolved.

Configuration editor broken across all regions

2020-11-19T07:04:14Z

We're investigating problems with configuration editor.

Update Nov 19, 2020 07:00 UTC - Configuration editor is broken across all regions. We're working on a fix that should be ready shortly. Sorry for inconvenience. Next update in 30 minutes.

Update Nov 19, 2020 07:19 UTC - Change causing the issue has been rolled back and we're in a process of verification that the issue is resolved. Next update in 30 minutes.

Update Nov 19, 2020 07:28 UTC - We verified that the change causing the issue is rolled back on all regions and the configuration editor is working normally. If you're still experiencing issues with the editor, please contact support individually using support button in your project.

Azure EU Maintenance Announcement

2020-11-11T07:35:58Z

A maintenance of Azure EU Keboola Connection (https://connection.north-europe.azure.keboola.com/) will take place on Sunday, Nov 15th, 2020 from 08:00 UTC and should take less than two hours.

During the maintenance, you can't access your projects. Orchestrations and running component jobs should be generally delayed, but not interrupted. However, feel free to re-schedule your sunday's orchestrations to avoid this maintenance window.

Update Nov 15, 2020 08:00 UTC - The maintenance started.

Update Nov 15, 2020 08:36 UTC - The maintenance is over. Azure EU Keboola Connection is up and fully operational.

Investigating Increased Snowflake Errors in the US Region

2020-10-27T16:55:28Z

We are encountering increased error rates from Snowflake in the US Region and are currently investigating the cause.
We will update here again as soon as we have relevant information or if 60 minutes have passed.

Update Oct 27, 2020 17:17 UTC: The error rate has returned to normal since 16:55 UTC. We are continuing to investigate the root cause.

Update Oct 27, 2020 18:57 UTC: Snowflake has released a status report. The errors continue to occur, but at a reduced rate.

Update Oct 27, 2020 19:29 UTC: Snowflake has reported the incident as resolved.

Investigating Snowflake Performance in the EU Region

2020-10-26T10:43:36Z

We are experiencing a minor slow down of performance on Snowflake in the EU Region and are currently investigating.
There are no job failures or increased queue backlog but everything seems to run slightly slower.
The next update will be in 120 minutes or as new information becomes available.

Update October 26 2020, 12:50 UTC: Performance has for the most part recovered, the rate of long execution of some queries has dropped significantly, but hasn't quite completely disappeared. We will update again if the issue re-surfaces.

Week in Review - October 20th, 2020

2020-10-20T14:27:39Z

New Components

POP3 Attachment extractor – allows you to download email attachments from a POP3 mailbox. There is no restriction on attachment type, so you can also download XLS files or images.

Updated Components

MongoDB extractor – added an option to “Include parent in Primary Key” in Mapping mode. This will allow better PK generation and the behavior will be much more consistent with Generic Extractor.

Looker writer – added support for Google BigQuery as the underlying database.

UI Improvements

Selecting a whole bucket in Input Mapping is now possible even if there are collisions between table names, or if there’s already a table from the specific bucket:

If there's a name collision, a numeric suffix is added (for example, if there is an account table from another bucket and you add a new bucket which has a table with the same name, it will be named account_2).
If there are already tables from the bucket, only the remaining tables will be added.

Force Unlink

There’s a new option to “force unlink” a bucket from projects that have it linked. When you disable sharing, it will “force unlink” the bucket automatically.

Edit Bucket Sharing

From now on, if there are projects that have the bucket linked, you can only add more projects/users. Edit sharing is available from “shared from this projects” list and from “bucket detail in data catalog” pages (if you have sufficient privileges). Until the bucket is linked, sharing can be changed without limitations.

Investigating EU overload

2020-10-15T10:10:47Z

We experienced overload in EU

Since 2020-10-15 10:02 UTC Start investigating overload in EU. Next update in 1 hour or as new information

UPDATE 2020-10-15 11:14 UTC: All operations are back to normal and everything is fully working. No jobs should be affected.

Investigating EU overload

2020-10-12T12:12:39Z

We experienced overload in EU

Since 2020-10-12 11:40 UTC Start investigating overload in EU. Next update in 1 hour or as new information becomes available.

UPDATE 2020-10-12 12:56 UTC: Everything should be back to normal. No jobs should be affected. We'll keep monitoring our platform closely.

UPDATE 2020-10-12 13:57 UTC: All operations are back to normal and everything is fully working.

Telemetry Data Fixes and Upcoming Telemetry Component

2020-10-12T11:09:45Z

Dear customers,

We are in the final phase of shifting and finalizing the calculation of time credits. We thank you for your patience and partnership in working with us and transitioning your contracts over to the new credit system.

As a part of finalizing, we are implementing some changes that weren’t applied previously and will begin to calculate starting now.

Transformations with multiple transformation backends - If you have a transformation that runs an SQL transformation with a R/Python transformation following it, in the same bucket, the SQL transformation is being calculated until the R/Python transformations are finished. As a result, these SQL jobs will have its child jobs run time deducted from their own, resulting in lower time credits consumption of affected clients. The change will cover jobs since the beginning of 2020.
SQL sandboxes time credits usage - Most of our clients don’t have SQL sandboxes consumption added to their overall consumption. In our analysis, this is a fraction of total consumption, so the impact is not very material for the majority of clients. For those clients that have a material impact, your CSM will be reaching out to you to let you know the impact, otherwise you can assume your usage is not going to impact you.
COVID-19 Error jobs - We haven’t been calculating credits for error jobs since April 2020 due to Covid and a higher than normal rate of Snowflake related issues. We will begin adding the consumption from job errors starting October.

Telemetry Component

We’re planning to release a new Keboola Connection component, which will allow the users to get telemetry data about their project or organization. That will eventually replace our temporary solutions like GoodData telemetry dashboard or direct writing of telemetry data to some projects. In the end, all of the Keboola Connection users will have easy access to documented telemetry data on demand.

Higher error rate in US region

2020-10-05T18:15:32Z

We experienced Connection API higher error rate and issue with starting new Python sandboxes between 17:45 - 17:55 UTC. The issue is now resolved, jobs processing wasn't affected.

Snowflake issues in US region

2020-09-29T08:49:11Z

We're investigating an issue with Snowflake in US region which causes some Storage operations with tables to be stuck in processing state. This can cause jobs to be executing longer than expected or seemingly "forever". Terminating and restarting the job does not help in such a case. Only certain projects are affected.

Next update in 1 hour or as new information becomes available.

Update 10:50 UTC: The stuck jobs are unblocked now and should be finishing, we're monitoring the situation if the issue reappears. A post mortem will be published once we get an RCA from Snowflake.

Keboola-provided credentials for Snowflake and Redshift database writers

2020-09-15T14:21:00Z

When configuring a Snowflake or Redshift database writer, you can use a Keboola-provided database.

In the past, when you selected this option, the credentials were stored in a configuration in a plain-text format. Storing the credentials this way allowed you to copy the password and use it in your favorite database client (or another system) even if you didn't copy it right after the creation.

To improve the overall security, we decided to show you your password only once and store it encrypted. From now on, when you create a new Keboola-provided database (Snowflake or Redshift), you will see the password only once, right after its creation.

Backward compatibility

The existing credentials will remain untouched. But if you delete them, there's no option to create them the old way.

Week in Review - September 14th, 2020

2020-09-14T06:57:47Z

New Components

LiveRamp Identity Resolution application - solving some of the main challenges with customer and prospect data by returning people-based identifiers and metadata for your consumer records
KBC Project Metadata extractor - Keboola metadata extractor downloads metadata about all objects in your Keboola project.
Avro2CSV processor - Avro is a row-oriented remote procedure call and data serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.

Updated Components

MySQL extractor - the "Transaction Isolation Level" setting is now configurable (documentation)

Oracle extractor / Oracle writer - the "Tnsnames" can be now used to provide login credentials to Oracle database (documentation)

Generic extractor - added option "caCertificate" which allows you configure custom certificate authority bundle in crt/pem format. (documentation)

Minor Improvements

Google BigQuery - updated google-cloud-bigquery package
Python updated to 3.8.5
Julia updated to 1.5.0

Snowflake US - Performance degradation

2020-09-02T12:47:37Z

We are investigating slight performance degradation of Snowflake in US region, there are no job failures or increased queue backlog but everything seems to run slightly slower. Degradation started around 00:00 AM UTC. Next update in 120 minutes or as new information becomes available.

UPDATE 2020-09-02 14:59 UTC: We still see slight performance degradation of some queries. We are in touch with Snowflake support. Next update tomorrow or as new information becomes available.

UPDATE 2020-09-03 06:31 UTC: The issue is now resolved, performance went back to normal around 2020-09-03 00:00 UTC. We are waiting for more details about the issue from Snowflake.

Snowflake Incident in US region

2020-09-02T06:12:10Z

On September 1st between 8:25 PM UTC and 9:48 PM UTC there was an incident on Snowflake service which led to Storage job failures.

The issue is now resolved and all systems are operational. We apologize for the inconvenience caused by this incident.

Snowflake Incident in US region

2020-08-25T08:57:54Z

Since 2020-08-25 8:35 UTC we are experiencing Storage errors in the US region due to a reported incident in Snowflake. We are going to monitor the situation and keep you posted within 90 minutes.

UPDATE 2020-08-25 9:20 UTC: Snowflake incident update:

We have identified the problem with the Snowflake Service that is interrupting the following services:
1. Access to Snowflake UI
2. Cannot execute queries

Incident Start Time: 01:20 PT Aug 25, 2020
We will provide an update within 30 minutes or as soon as we have more details on the status of the issue.

UPDATE 2020-08-25 9:50 UTC: The problem seems to disappear, we don't get any more errors since 9:30 UTC. But Snowflake hasn't updated the incident yet so we are still monitoring the situation.

UPDATE 2020-08-25 10:15 UTC: A fix to resolve the issue is applied and the situation is still being monitored.

Week in Review - August 14th, 2020

2020-08-14T11:10:36Z

Updated Components

AWS S3 Extractor supports Authentication with an AWS role (documentation)

Twitter Extractor supports Direct Messages

To extract direct messages, you must reauthorize the account since an additional permission (DMs) is needed.

MongoDB Extractor supports custom URI connection

MSSQL Extractor supports encrypted (SSL) connection

UI Improvements

Storage job detail now has a permalink

To get a permalink for the job detail, click on the job ID. A popup with the job detail will appear, and the URL in your browser will change.

Minor Improvements

Storage API List files endpoint is now returning only valid (not expired) files. Expired files can be retrieved with showExpired option, more in our documentation https://keboola.docs.apiary.io/#reference/files/list-files/list-files

GoodData Writer Failures

2020-08-13T09:28:36Z

There is some problem on GoodData API since about 01:00 CEST (23:00 UTC) which causes job timeouts. We are investigating the problem.

Next update in 60 minutes or as new information becomes available.

13 Aug 2020 10:30 UTC We are investigating the problem with GoodData support. So far it looks like problem is in WebDav integration on GoodData side. Next update in 60 minutes or as new information becomes available.

13 Aug 2020 11:30 UTC GoodData support is still investigating issue. Next update in 60 minutes or as new information becomes available.

13 Aug 2020 12:05 UTC According to our investigation it seems that all GoodData jobs are working now normally. We will continue to monitor situation. If you have GoodData job which is running unusually long, try to restart it, if it doesn't help please contact out support, we will hand over your pid to GoodData for further investigation. We are still waiting for GoodData support for detailed information's about incident. Next update in 2 hours or as new information becomes available.

13 Aug 2020 14:07 UTC We are still waiting for resolution from GoodData. Next update will be when new information becomes available. If you want detailed information for you project contact GoodData support with you pid and import time of last csv before failure.

13 Aug 2020 16:40 UTC Job failures were caused by slower GoodData DHWs, GoodData take action and fix problem, but several jobs timeout during this period. All services are stable now.

Snowflake Slowdown in the EU and US Region

2020-08-04T04:29:12Z

04 Aug 2020 04:28 UTC We're seeing a higher load and longer execution time in EU and US Snowflake warehouse queries. We are investigating the causes. Next update in 60 minutes or as new information becomes available.

04 Aug 2020 04:53 UTC We have added some processing power to the Snowflake warehouse in both regions but the backlog is still present. Snowflake identified the problem with the interruption which may cause the processing slowdown. Next update in 60 minutes or as new information becomes available.

04 Aug 2020 05:20 UTC Backlogs are cleared in both regions and the situation seems normal, but we're monitoring it closely for next couple hours. Next update in two hours or as new information becomes available.

04 Aug 2020 06:25 UTC Snowflake marked the incident as resolved, everything should be back to normal. We'll keep monitoring our platform closely.

04 Aug 2020 07:27 UTC Query times are back to normal and all the backlogs are cleared for some time already. The incident is completely resolved.

Week in Review - July 30th, 2020

2020-07-30T08:33:54Z

Updated Components

Elasticsearch Writer supports collumns mapping
PostgreSQL Extractor supports Encrypted SSL Connection
Zendesk Extractor no longer has custom tickets fields values set as a primary key in the mapping.