Keboola Status

Sep 20 15:15 UTC: We are experiencing degraded performance on our GCP EU stack (https://connection.europe-west3.gcp.keboola.com/). We are deeply sorry for the inconvenience this may cause and appreciate your patience as we work through it. We will provide further updates as soon as we have more information.

Sep 22 18:53 UTC update:We have gained additional understanding of the performance degradation. The root cause seems to be in intermittent slowdown of query execution on Snowflake. While the execution of single query is not delayed significantly, the delays do accumulate to noticeable slowdowns of minutes on transformations consisting of multiple queries and even more for entire flows. We're in touch with Snowflake support in the process of uncovering all the technical details.

The symptoms of the performance degradation include longer running times of jobs - especially on Snowflake transformations, but Data Source jobs and Data Destination jobs are also affected, because they load/unload data from a Snowflake database. The performance degradation occurrence is random and somewhat time dependent so not all flows are affected in the same manner. The performance degradation does not cause any errors.

Sep 24 07:10 UTC update: We have now confirmed the root cause to be in slower execution of Snowflake queries at certain times. We have implemented a temporary resource increase to improve the situation. This means that you should see improved job run times. We're still working with Snowflake support on the solution.

Oct 1 7:54 UTC update: We are still working on the resolution together with Snowflake support. The temporary resource increase is still in place. This means that that the situation is contained and overall stack performance should be acceptable, but not perfect. 

At this moment we don't have a solution ready, nor an ETA for one. Thank you for your understanding and patience. We will provide further updates as soon as we have more information.

There was an increased error rate for components using keboola.processor-create-manifest in configurations. This error affected only component configurations which were using empty enclosure in CSV Settings.

Affected component may include data sources like AWS S3, FTP, etc.

We are sorry for any inconvenience. Please feel free to restart your jobs manually.

2022-06-20 16:44 UTC - We are investigating increased error rate of some component using keboola.processor-create-manifest.

2022-06-21 07:22 UTC - We identified a root cause and continue working on a fix.

2022-06-21 12:30 UTC - Incident is resolved, last occurrence of the error was at 12:13 UTC.

2022-02-01 14:36 UTC We are experiencing small number of jobs in projects with Queue v2 in waiting state. We are investigating the issue.

2022-02-01 15:06 UTC Only 3 projects are affected by this issue.

2022-02-01 15:34 UTC Root cause has been identified and we are working on a fix.

2022-02-01 17:12 UTC The issue has been fixed.

2022-02-01 09:10 UTC We are experiencing number of jobs in waiting state more than usual. We are investigating the issue.

2022-02-01 10:02 UTC There was increased traffic on the EU Snowflake warehouse so that we upgraded it to larger instance and the queued jobs were immediately processed. The delay is fixed, no jobs are waiting for processing right now.

12.11.2021 10:43 CET

We are experiencing increased error rate for components communication with Google APIs

Google reports there are several services disruptions:

• https://status.cloud.google.com/
• https://www.google.com/appsstatus/dashboard/
  

We continue to monitor the situation.

12.11.2021 14:11 CET

We no longer see increased component failures, everything is working as expected.

Since 02:23 CET 2021-06-25 we're seeing a delay in processing of job events in Azure North Europe stack.

We are monitoring a situation, next update in one hour.

UPDATE 09:15 CET 2021-06-25 We increased the capacity of infrastructure and issue has been resolved. All job events have been processed. We're sorry for this inconvenience.

When configuring a Snowflake or Redshift database writer, you can use a Keboola-provided database.

In the past, when you selected this option, the credentials were stored in a configuration in a plain-text format. Storing the credentials this way allowed you to copy the password and use it in your favorite database client (or another system) even if you didn't copy it right after the creation.

To improve the overall security, we decided to show you your password only once and store it encrypted. From now on, when you create a new Keboola-provided database (Snowflake or Redshift), you will see the password only once, right after its creation.

Backward compatibility

The existing credentials will remain untouched. But if you delete them, there's no option to create them the old way.

New Features and Updates

Project Description

Looker Writer Connection Name

Deprecation of Storage API .NET Client

Renaming Storage Buckets and Tables

Selecting Bucket in Input Mapping

Bug Fixes

• Generic Extractor no longer stops after the 2nd page when downloading data in child jobs (only configurations with Limit Stop setting were affected)
• CSV import component supports full load again (due to a bug, all imports were performed incrementally).
• MySQL writer no longer writes an "empty string" instead of a null for columns with DATE and DATETIME data types.

New Components

• CSOB CEB extractor for downloading bank statements from the CSOB CEB Business Connector service
• Azure Blob Storage writer for exporting any input CSV files into designated Blob containers
  
• Sisense writer for sending tables from Keboola Connection to a Sisense database platform
• Zendesk writer for creating and updating Zendesk Support properties with the flexibility of defining their own parameters

An option to rename buckets and tables was one of the most requested features on our wishlist. It is very useful when you want to name your bucket by its contents (e.g., "email-orders") rather than "in.c-keboola-ex-gmail-587163382".

From now on, you'll be able to change the names of buckets and tables.

Rename Bucket

To rename a bucket, navigate to the bucket detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (there are some limitations though).

Rename Table

To rename a table, navigate to the table detail page, and click the pen icon next to the name parameter.

Then choose the name of your preference (the same limitations apply).

Consequent Changes

Despite the fact that adding the option to rename a bucket or a table does not look like a very big deal, we had to make some substantial changes under the hood. Some of the consequences are worth mentioning here:

Hidden "c-" prefix

We no longer show the "c-" prefix in the names of buckets and tables. It is still a part of the bucket and table ID, but the ID is no longer displayed in most cases. If you need to access the ID for some reason, it is still available on the detail page of each bucket and table.

This is an example of how buckets and tables are displayed without the "c-" prefix:

Stage Selector

When searching for a specific bucket or table, just select a stage and the buckets will be filtered by the selected stage.

We're happy to announce the arrival of Configuration Rows, our new powerful configuration format, to database writers.

From now on, you'll see a migration button in the configuration detail of each database writer (Snowflake, MySQL, SQL Server, Oracle, PostgreSQL, Impala, Hive, and Redshift).

Just click Migrate Configuration and the configuration will be migrated to the new format.

After the migration, you'll see more information about each table. All tables can be easily reordered, so you can move more important tables to the top and they will be uploaded first.

Also, you will be able to see information about each table on a new table detail page, with Last runs and Versions in a sidebar.

Underlying Important Changes

While there were certain limitations in the old configuration format, this is no longer true in the new "rows format".

The following features are worth mentioning:

• Disabled tables will no longer be exported from Storage (previously, they were exported with limit=1 and not used in the database writer).
• Each table has its own state with information about the date/time of the last import (previously, an upload of a single table cleared the state for other tables).