Keboola Status

2022-06-06 21:00 UTC We are investigating a problem with not processing jobs in AWS US stack https://connection.keboola.com/. Some jobs will probably end up with an internal error.

2022-06-06 21:25 UTC Correction - jobs are only delayed and all jobs ends successfully.

2022-06-06 21:40 UTC Unfortunately the problem still persists, we are trying to find out the cause.

2022-06-06 22:30 UTC We will update this post when we have more information.

2022-06-06 23:25 UTC We have managed to improve the situation. We will continue monitoring the situation and tomorrow we will provide more information about the problem.

2022-06-07 12:40 UTC The situation is now under control and our services are running normally.

We are sorry for the inconvenience.

Since 2022-06-03 23:30 UTC we're seeing a higher error rate in Pay As You Go projects in Azure Stack (https://connection.north-europe.azure.keboola.com/). This is caused by degraded performance of one of the Azure services. You may encounter higher loading times of the main and billing page and occasional errors when running jobs. Non Pay As You Go projects are not affected.

Update 2022-06-04 7:20 UTC: The Azure service performance was restored back to normal at approximately 2022-06-04 2:40. Our services closely followed. The issue is now resolved.

Next update in 8 hours.

UPDATE 10:00 UTC - Previously announced maintenance of connection.keboola.com will start in one hour at 11:00 UTC. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

UPDATE 11:00 UTC - AWS US stack (connection.keboola.com) maintenance started.

UPDATE 12:12 UTC - AWS US stack (connection.keboola.com) is back up and running. We continue to monitor the situation.

UPDATE 06:00 UTC - Previously announced maintenance of connection.eu-central-1.keboola.com will start in one hour at 07:00 UTC. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

UPDATE 07:00 UTC - EU stack (connection.eu-central-1.keboola.com) maintenance started.

UPDATE 09:08 UTC - EU stack (connection.eu-central-1.keboola.com) is back up and running. We continue to monitor the situation.

Previously announced maintenance of Keboola Connection AWS stacks will take place on Saturday, May 28th, 2022 and should take less than three hours.

• connection.eu-central-1.keboola.com - 7:00 AM UTC to 10:00 AM UTC
• connection.keboola.com - 11:00 AM UTC to 2:00 PM UTC

Read more in maintenance announcement.

2022-05-25 08:40 UTC We are investigating a problem where newly created workspaces (python, R, Snowflake ) are not displayed. 

We have currently solved the problem, but workspaces created from 07:40 UTC to 08:40 UTC will not be displayed. So please create worspaces again.

We are sorry for the inconvenience.

Dear Keboola Connection users,

A new version of our telemetry has been released, and you can get the new data via the Telemetry Data extractor (a data source component).

All changes mentioned in the announcement post have been applied, but there is one additional change: a new table, kbc_branch, has been added. It contains info about the main and development branches in projects and can be joined to kbc_job so you can see the branch in which the job runs. Please note that jobs on the legacy queue are always marked as being run on the main branch. Only new-queue jobs are assigned to a particular development branch (if it has been used).

We remind you that all of the tables in the new telemetry contain changed primary keys. Hence, you need to process data in full in case there is some incremental processing taking place in your project. The Telemetry Data extractor will be forcing full load until the end of May.

2022-05-18 10:00 UTC We have started investigation of Telemetry Data extractor failures

Resolved - The problem is that the table kbc_component_configuration_version is now extracted with a new column kbc_branch_id that is also a part of the primary key but is missing in the destination/existing table kbc_component_configuration_version. To fix this, delete kbc_component_configuration_version table and run the Telemetry Data extractor again - that will correctly extracts the table without an error.

2022-05-09 18:10 UTC - We are investigating delayed data updates for ECB Currency Rates Data Source. Since 2022-05-07 15:00 UTC there were no updates in currency rate tables. Next update when new information is available. 

2022-05-09 19:41 UTC - We have identified the root cause and we are working on the fix. Next update when new information is available.

2022-05-10 10:45 UTC - The issue is fixed, missing data were backfilled.

Dear Keboola Connection users,

On Monday May 16, 2022, we are releasing a new version of our telemetry.

There will be a couple of changes to the data structure:

• All platform (kbc_*) tables - Primary Keys will be slightly changed to add stack identifiers, which will help us programmatically manage the script and make implementation of the new stack easier. Example: “kbc_project_id” 9241_us-east-1_aws will be changed to 9241_kbc-us-east-1. This is important only in case you are parsing values from Primary Keys.

• Usage Metrics Values - the usage_breakdown “Snowflake Writer” will be renamed “DWH Direct Query” so that it is less confusing for the user, who is often anticipating queue jobs of the component called Snowflake Writer (Data Destination).

• KBC Project - the column “kbc_project_stack” will be added, which uniquely identifies the stack where the project exists.

The major changes are happening on the backend of the Telemetry data processing:

• All our internal Keboola Connection projects related to telemetry are managed by Keboola CLI, which allows us to easily manage updates, use GitHub reviews of the changes or initiate rollbacks in case of an issue.

• All our internal Keboola Connection projects related to telemetry are using Queue V2, so they are able to take advantage of all of the new and future Keboola Connection features.

• Using a new strictly managed multi-project architecture enables better readiness of the entire flow.

• We are addressing minor issues related to product updates that were not covered in the legacy telemetry.

How the release affects our users:

• Snowflake usage of Workspaces and Direct Queries will be adjusted, as the legacy telemetry did not correctly identify all of the Snowflake queries related to these usage breakdowns. The vast majority of users will see little to no impact from these changes and all those who will be most impacted have already been provided with further details by our Customer Success Team.

• We will force FULL load in the Telemetry Data extractor, as changes to Primary Keys would cause duplicates if incremental processing were to be used in your configuration. It also leaves one task for our users - they will need to process the data in full in their subsequent processes after the new telemetry is released in order to align their telemetry with the update.

• Telemetry data won’t be updated for up to two days so that we can ensure that users are not receiving incorrect data during the new telemetry product implementation.

In case you encounter any issues after the new telemetry is released, please contact support through the button in your project.