Increased API Error rate in AWS us-east-1 connection.keboola.com stack

2022-06-10 03:40 UTC Today since 2:50 UTC we are experiencing increased Connection api internal error rates as well as increased latency due to an incident in AWS us-east-1 region. The job processing shouldn't be affected, however there might be slow down in the UI responsiveness.

2022-06-10 04:20 UTC - AWS still haven't resolved the incident. Furthermore we are noticing some jobs processing might be affected ending in internal error due to failed connections retries to AWS API. 

2022-06-10 08:15 UTC - the issue has been resolved and since 04:30 UTC we don't see an increased api error rate.

Stuck jobs in AWS us-east-1 connection.keboola.com stack

2022-06-06 21:00 UTC We are investigating a problem with not processing jobs in AWS US stack https://connection.keboola.com/. Some jobs will probably end up with an internal error.

2022-06-06 21:25 UTC Correction - jobs are only delayed and all jobs ends successfully.

2022-06-06 21:40 UTC Unfortunately the problem still persists, we are trying to find out the cause.

2022-06-06 22:30 UTC We will update this post when we have more information.

2022-06-06 23:25 UTC We have managed to improve the situation. We will continue monitoring the situation and tomorrow we will provide more information about the problem.

2022-06-07 12:40 UTC The situation is now under control and our services are running normally.

We are sorry for the inconvenience.

Job Processing Slowing down

Since approximately 13:30 we're seeing some jobs processing longer then usual in the the us-east and eu-central stack (connection.keboola.com and connection.eu-central-1.keboola.com). We're investigating the issue. Next update in 30 minutes.

Update 14:30 UTC: We have identified the root cause of the issue. We're actively working on fixing the issue. Next update in 30 minutes.

Update 15:20 UTC: We are still working on the fix. Next update in 30 minutes.

Update 15:55 UTC: We have deployed the fix to both affected stacks and the issues should be resolved now. We will continue monitoring the situation.

Increased error rate in Pay-as-you-go projects

Since 2022-06-03 23:30 UTC we're seeing a higher error rate in Pay As You Go projects in Azure Stack (https://connection.north-europe.azure.keboola.com/). This is caused by degraded performance of one of the Azure services. You may encounter higher loading times of the main and billing page and occasional errors when running jobs. Non Pay As You Go projects are not affected.

Update 2022-06-04 7:20 UTC: The Azure service performance was restored back to normal at approximately 2022-06-04 2:40. Our services closely followed. The issue is now resolved.

Next update in 8 hours.

connection.keboola.com maintenance

UPDATE 10:00 UTC - Previously announced maintenance of connection.keboola.com will start in one hour at 11:00 UTC. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

UPDATE 11:00 UTC - AWS US stack (connection.keboola.com) maintenance started.

UPDATE 12:12 UTC - AWS US stack (connection.keboola.com) is back up and running. We continue to monitor the situation.

connection.eu-central-1.keboola.com maintenance

UPDATE 06:00 UTC - Previously announced maintenance of connection.eu-central-1.keboola.com will start in one hour at 07:00 UTC. During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

UPDATE 07:00 UTC - EU stack (connection.eu-central-1.keboola.com) maintenance started.

UPDATE 09:08 UTC - EU stack (connection.eu-central-1.keboola.com) is back up and running. We continue to monitor the situation.


New workspaces are not displayed

2022-05-25 08:40 UTC We are investigating a problem where newly created workspaces (python, R, Snowflake ) are not displayed. 

We have currently solved the problem, but workspaces created from 07:40 UTC to 08:40 UTC will not be displayed. So please create worspaces again.

We are sorry for the inconvenience.

New Telemetry Released

Dear Keboola Connection users,

A new version of our telemetry has been released, and you can get the new data via the Telemetry Data extractor (a data source component).

All changes mentioned in the announcement post have been applied, but there is one additional change: a new table, kbc_branch, has been added. It contains info about the main and development branches in projects and can be joined to kbc_job so you can see the branch in which the job runs. Please note that jobs on the legacy queue are always marked as being run on the main branch. Only new-queue jobs are assigned to a particular development branch (if it has been used).

We remind you that all of the tables in the new telemetry contain changed primary keys. Hence, you need to process data in full in case there is some incremental processing taking place in your project. The Telemetry Data extractor will be forcing full load until the end of May.

Telemetry data extractor failure

2022-05-18 10:00 UTC We have started investigation of Telemetry Data extractor failures

Resolved - The problem is that the table kbc_component_configuration_version is now extracted with a new column kbc_branch_id that is also a part of the primary key but is missing in the destination/existing table kbc_component_configuration_version. To fix this, delete kbc_component_configuration_version table and run the Telemetry Data extractor again - that will correctly extracts the table without an error.