tag:status.keboola.com,2013:/posts Keboola Status 2022-05-26T16:34:43Z Keboola Connection - Platform Status tag:status.keboola.com,2013:Post/1834009 2022-05-26T13:39:08Z 2022-05-26T13:39:08Z AWS Stacks Maintenance Announcement 2022-05-28 Reminder

Previously announced maintenance of Keboola Connection AWS stacks will take place on Saturday, May 28th, 2022 and should take less than three hours.

Read more in maintenance announcement.

tag:status.keboola.com,2013:Post/1833483 2022-05-25T08:57:11Z 2022-05-25T08:57:12Z New workspaces are not displayed

2022-05-25 08:40 UTC We are investigating a problem where newly created workspaces (python, R, Snowflake ) are not displayed. 

We have currently solved the problem, but workspaces created from 07:40 UTC to 08:40 UTC will not be displayed. So please create worspaces again.

We are sorry for the inconvenience.

tag:status.keboola.com,2013:Post/1830868 2022-05-18T20:18:43Z 2022-05-18T20:18:43Z New Telemetry Released

Dear Keboola Connection users,

A new version of our telemetry has been released, and you can get the new data via the Telemetry Data extractor (a data source component).

All changes mentioned in the announcement post have been applied, but there is one additional change: a new table, kbc_branch, has been added. It contains info about the main and development branches in projects and can be joined to kbc_job so you can see the branch in which the job runs. Please note that jobs on the legacy queue are always marked as being run on the main branch. Only new-queue jobs are assigned to a particular development branch (if it has been used).

We remind you that all of the tables in the new telemetry contain changed primary keys. Hence, you need to process data in full in case there is some incremental processing taking place in your project. The Telemetry Data extractor will be forcing full load until the end of May.

tag:status.keboola.com,2013:Post/1830715 2022-05-18T10:18:05Z 2022-05-18T10:27:59Z Telemetry data extractor failure

2022-05-18 10:00 UTC We have started investigation of Telemetry Data extractor failures

Resolved - The problem is that the table kbc_component_configuration_version is now extracted with a new column kbc_branch_id that is also a part of the primary key but is missing in the destination/existing table kbc_component_configuration_version. To fix this, delete kbc_component_configuration_version table and run the Telemetry Data extractor again - that will correctly extracts the table without an error.

tag:status.keboola.com,2013:Post/1830527 2022-05-17T08:16:23Z 2022-05-17T08:18:09Z Low Queue API errors rate in AWS us-east-1

2022-05-17 09:54 CET Due to the RDS upgrade, there was a small outage of the API queue and you may have observed a small number of errors in the connection.keboola.com UI.

The situation is now under control and our services are running normally.

Václav Eder
tag:status.keboola.com,2013:Post/1828967 2022-05-12T14:03:40Z 2022-05-12T14:29:37Z Delayed processing of jobs in AWS US stack

May 12, 14:00 UTC We are investigating a problem with waiting jobs in AWS US stack https://connection.keboola.com/

We will update this post when we have more information.

May 12, 14:30 UTC All operations are back in normal. The problems were detected only with part of jobs that ran on new queue.

Erik Žigo
tag:status.keboola.com,2013:Post/1827995 2022-05-09T18:25:09Z 2022-05-10T10:46:09Z ECB Currency Rates Data Source delayed data updates

2022-05-09 18:10 UTC - We are investigating delayed data updates for ECB Currency Rates Data Source. Since 2022-05-07 15:00 UTC there were no updates in currency rate tables. Next update when new information is available. 

2022-05-09 19:41 UTC - We have identified the root cause and we are working on the fix. Next update when new information is available.

2022-05-10 10:45 UTC - The issue is fixed, missing data were backfilled.

tag:status.keboola.com,2013:Post/1827763 2022-05-09T08:59:31Z 2022-05-10T19:49:48Z New Telemetry Release

Dear Keboola Connection users,

On Monday May 16, 2022, we are releasing a new version of our telemetry.

There will be a couple of changes to the data structure:

  • All platform (kbc_*) tables - Primary Keys will be slightly changed to add stack identifiers, which will help us programmatically manage the script and make implementation of the new stack easier. Example: “kbc_project_id” 9241_us-east-1_aws will be changed to 9241_kbc-us-east-1. This is important only in case you are parsing values from Primary Keys.

  • Usage Metrics Values - the usage_breakdown “Snowflake Writer” will be renamed “DWH Direct Query” so that it is less confusing for the user, who is often anticipating queue jobs of the component called Snowflake Writer (Data Destination).

  • KBC Project - the column “kbc_project_stack” will be added, which uniquely identifies the stack where the project exists.

The major changes are happening on the backend of the Telemetry data processing:

  • All our internal Keboola Connection projects related to telemetry are managed by Keboola CLI, which allows us to easily manage updates, use GitHub reviews of the changes or initiate rollbacks in case of an issue.

  • All our internal Keboola Connection projects related to telemetry are using Queue V2, so they are able to take advantage of all of the new and future Keboola Connection features.

  • Using a new strictly managed multi-project architecture enables better readiness of the entire flow.

  • We are addressing minor issues related to product updates that were not covered in the legacy telemetry.

How the release affects our users:

  • Snowflake usage of Workspaces and Direct Queries will be adjusted, as the legacy telemetry did not correctly identify all of the Snowflake queries related to these usage breakdowns. The vast majority of users will see little to no impact from these changes and all those who will be most impacted have already been provided with further details by our Customer Success Team.

  • We will force FULL load in the Telemetry Data extractor, as changes to Primary Keys would cause duplicates if incremental processing were to be used in your configuration. It also leaves one task for our users - they will need to process the data in full in their subsequent processes after the new telemetry is released in order to align their telemetry with the update.

  • Telemetry data won’t be updated for up to two days so that we can ensure that users are not receiving incorrect data during the new telemetry product implementation.

In case you encounter any issues after the new telemetry is released, please contact support through the button in your project.

tag:status.keboola.com,2013:Post/1826426 2022-05-05T21:18:47Z 2022-05-05T21:30:09Z Orchestration jobs end after a second but child job still running

2022-05-05 13:10 CET - Starting with 9:10 CET it could happen that a parent job (orchestration or orchestration phase) ends with success even though the child jobs are still running. The child jobs will finish their work normally, but things may appear out of sync in case of nested or chained orchestrations.

The issue is fixed for Azure stacks since 12:50 CET, the fix for AWS stacks is under way.

UPDATE 2022-05-05 14:33 CET - We discovered one of the manifestations may be that all the phases started at once.

UPDATE 2022-05-05 23:30 CET - The issue is fixed since 12:50 CET.

tag:status.keboola.com,2013:Post/1822008 2022-04-28T05:29:41Z 2022-05-26T16:34:43Z AWS Stacks Maintenance Announcement 2022-05-28

Maintenance of Keboola Connection AWS stacks will take place on Saturday, May 28th, 2022 and should take less than three hours.The following stacks will be affected:

During the maintenance, you won’t be able to access your data and projects. All network connections will be terminated by the "HTTP 503 - down for maintenance" status message.

We will monitor all running tasks and restart any affected by interruption. Orchestrations (flows) and running transformations will be generally delayed, but not interrupted. However, feel free to re-schedule your Saturday orchestrations to avoid this maintenance window.

tag:status.keboola.com,2013:Post/1819009 2022-04-15T08:31:59Z 2022-04-15T08:51:32Z Stuck jobs in AWS us-east-1 [resolved]

2022-04-15 10:30 UTC - We are investigation stuck jobs on a new queue. Some jobs will probably end up with an internal error. Next update in 15 min.

2022-04-15 10:50 UTC - The situation was resolved, however 6 jobs were terminated by an internal error with no automatic restart. We apologize for the inconvenience and please feel free to restart your jobs manually.

Affected jobs:

  • job-839828541
  • job-839828546
  • job-839830974
  • job-839831721
  • job-839831773
  • job-839831782

Václav Eder
tag:status.keboola.com,2013:Post/1817534 2022-04-11T15:11:56Z 2022-04-14T12:58:50Z Longer jobs runtime on new queue in AWS eu-central-1

2022-04-11 15:04 UTC - We are investigating transient delays in jobs processing. It manifests as a two hours gap without any activity in job events. It is happening randomly across projects and configurations, most of the occurrences are around 04:00 UTC. Only jobs running on new queue are affected. We are investigating the issue, next update in three hours or when new information will be available.

2022-04-11 16:54 UTC - We have increased minimum number of nodes which might help to avoid the issue happening again. Meanwhile we are investigating the root causes of timeouts. We are also working on decreasing timeouts from two hours to much lower value to prevent unnecessary job runtime increase in case of networking issues. Next update when new information will be available.

2022-04-14 12:54 UTC - We have reduced the timeouts from two hours to two minutes. This will prevent a job to get stuck for such a long time when a connection issue occurs. We are still investigating the root networking problem. Next update when new information is available.

tag:status.keboola.com,2013:Post/1815046 2022-04-05T05:29:09Z 2022-04-11T15:12:10Z Stuck Orchestrations [resolved]

2022-04-05 5:28 UTC - We are investigation stuck orchestration jobs on a new queue. The next update in 15 minutes.

2022-04-05 5:50 UTC - We can see the problem occurs in AWS regions, however we haven't found the root and continue investigation. Next update in 15 minutes.

2022-04-05 6:10 UTC - We rolled back previously deployed version of queue internal component and it seem to unblocked the stuck orchestrations jobs. We don't see any stuck orchestrations for now. We continue monitoring the situation and investigate for the root cause.

2022-04-05 9:10 UTC - We identified a root cause and now preparing a fix. However as of previous quick fix we are not noticing the stuck orchestrations anymore.

2022-04-05 11:40 UTC - We deployed fix and everything is running operational now. The root cause was a misconfigured network access for an internal Queue component.

tag:status.keboola.com,2013:Post/1813582 2022-04-01T07:38:07Z 2022-04-01T08:37:32Z Delayed processing of jobs in AWS EU stack

2022-04-01 07:36 UTC We are experiencing higher number of jobs in waiting state more than usual. We are investigating the issue. Next update in one hour or when new information will be available.

2022-04-01 08:36 UTC Backlog is cleared, delays were caused by increased traffic.

tag:status.keboola.com,2013:Post/1810091 2022-03-25T07:33:58Z 2022-03-31T16:59:17Z Scheduled Maintenance (Azure North Europe Region)

Scheduled maintenance of Azure North Keboola Connection (https://connection.north-europe.azure.keboola.com) will take place on Thursday, Mar 31th 2022 at 05:00 pm UTC and should take less than one hour. 

It should not affect project's running jobs, these will be queued during the short project maintenance. Orchestrations and running transformations will be generally delayed, but not interrupted. 

During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

We'll update this status about the progress.

UpdateMar 31th, 16:57 UTC - All changes done, in the end not maintenance was necessary.

tag:status.keboola.com,2013:Post/1807772 2022-03-18T10:38:28Z 2022-03-18T10:38:29Z Recursion limit for queries in snowflake about to be lifted

The 6.7 release of Snowflake platform will remove recursion limit for queries. Snowflake plans to release the change in March (no exact date). 

For more information follow this post on Keboola Community or check out the official Snowflake pending behavior change log.

tag:status.keboola.com,2013:Post/1807768 2022-03-16T12:58:04Z 2022-03-18T15:04:49Z Some Python Transformations Failing on Date Parsing Error

2022-03-16 12:45 UTC
We have discovered some failing python transformations that throw an exception with the message "bad escape \d at position".  This error was caused by a breaking change to the underlying regular expression library.

If your transformation is failing in this way you can fix it by specifying "regex==2022.03.02" in the packages input.

2022-03-18 15:00 UTC regex package is not part of our base images, so there will be no further actions. Users have set working version of package in their dependencies

tag:status.keboola.com,2013:Post/1807314 2022-03-15T08:17:51Z 2022-03-15T09:32:41Z High error rate from Sklik API

We are seeing high error rate response from Sklik API

Since 2022-03-15 00:00 UTC we are experiencing jobs failures due to a shortage in Sklik API. We are going to monitor the situation and keep you posted.

Since 2022-03-15 09:30 UTC last error from Sklik API was 8:34 UTC, for now it looks like Sklik API operates normally.

tag:status.keboola.com,2013:Post/1804953 2022-03-09T20:51:33Z 2022-03-09T21:23:22Z Errors When loading Tables in AWS US region

We're seeing failed table loads, it seems that the cause is a problem with Amazon AWS services.

Update 21:22 UTC (Resolved): AWS confirmed recovery of the services experiencing errors hence we consider the issue to be resolved as well.

Update 21:10 UTC: AWS informed about issues in us-east-1 region and started investigation however we are not seeing anymore issues on our side and US multitenant stack seem to be fully operational now. We continue monitoring the situation.

Update 20:56 UTC: The issues started occurring at approximately at 20:44 UTC. The issue is isolated to the US multitenant stack https://connection.keboola.com/. 

tag:status.keboola.com,2013:Post/1804407 2022-03-08T14:21:30Z 2022-03-09T08:20:05Z Delayed telemetry for new queue in AWS eu-central-1

2022-03-08 14:20 UTC - We are investigating delayed telemetry data for projects migrated to new queue in https://connection.eu-central-1.keboola.com/ stack. There is significant drop of consumed credits in telemetry since March 4. Next update in 24 hours.

2022-03-09 08:19 UTC - Telemetry data were backfilled and telemetry reports should be updated. Please contact support if you still see incorrect data.

tag:status.keboola.com,2013:Post/1801820 2022-03-02T19:04:36Z 2022-03-02T19:56:10Z Joining project does not work

2022-03-02 19:03 UTC We're investigating reports that project join screen does not work. Next update in 30 minutes. 

2022-03-02 19:33 UTC We've found the problematic change and initiated a rollback. Next update in 30 minutes.

2022-03-02 19:55 UTC The problem has been resolved on all stacks. Sorry for inconvenience. 

tag:status.keboola.com,2013:Post/1801296 2022-03-01T14:25:49Z 2022-03-01T14:25:49Z Migration to new Job Queue in North Europe

In the following weeks, starting tomorrow 2.3.2022, we will be continuously migrating projects in our North Europe stack to the new version of the Job Queue.

At first glance, these are mostly cosmetic changes, but under the hood we have totally reworked how jobs are processed in order to make them more reliable, faster, and makes it easier for us to bring more features to you:

  • Dynamic backends
  • Parallel execution of configuration rows
  • Long running jobs
  • Versioning and trash for Orchestrator configurations
  • More concise API of the individual services

Please read more about new Job Queue and the migration here.

We will start the migration with our internal projects and in the following weeks we will migrate projects which don't have any incompatible components. (The list of incompatible components can also be found here). 
If you are using one of the components from the list in you project and want your project to be migrated to the new Job Queue, please contact our support.

The migration of all projects should be completed within a month or two.

Please contact us on support@keboola.com if you have any questions about the upcoming migration or the new Job Queue in general.

We will keep you posted about the progress of the migration.

Miroslav Čillík
tag:status.keboola.com,2013:Post/1799220 2022-02-25T09:41:54Z 2022-02-25T09:41:55Z Multi-factor authentication devices registered via Safari do not work

There is a problem with MFA registration in Safari for both, hardware keys like Yubikey and TouchID. The registration succeeds but it's not possible to verify the MFA upon login.

If you registered a hardware key in Safari, use other browser to log in (it will work). Then remove the registered device and re-register it using the other browser.

If you registered TouchID in Safari, please contact support to have your MFA disabled to allow you to login. After that please reregister your TouchID the using different browser.

tag:status.keboola.com,2013:Post/1799296 2022-02-24T15:49:41Z 2022-02-24T17:23:56Z Failed jobs in AWS eu-central-1 stack

2022-02-24 15:47 UTC - We are investigating failures of some jobs on new queue in https://connection.eu-central-1.keboola.com/ stack. Affected jobs are throwing error: Job "XYZ" terminated unexpectedly. We're currently working on mitigating the issue and will update when it gets resolved or in 60 minutes.

2022-02-24 16:05 UTC - The errors were occurring between 15:00 - 15:30. The issue is now resolved, we are monitoring the situation and investigating root cause of error.

2022-02-24 16:10 UTC - We have seen few reoccurrences of the issue. We're currently working on mitigating the issue and will update when it gets resolved or in 60 minutes.

2022-02-24 17:19 UTC - Issue is resolved, platform is stable. Last error at 16:30. Issue was caused by overload of one of the nodes, we've upgraded the nodes and we are working on better resources management to prevent similar issues. We apologize for the inconvenience. 

tag:status.keboola.com,2013:Post/1797865 2022-02-21T13:35:18Z 2022-02-21T15:51:07Z Cannot create projects on Azure North Europe

2022-02-21 13:30 UTC We discovered an issue that may cause it not possible to create new projects on Azure North Europe. So it is not possible to create new Pay as you go projects. 

We're currently working on mitigating the issue and will update when it gets resolved or in 60 minutes. 

Update 14:30 UTC The issue is now fixed.

tag:status.keboola.com,2013:Post/1796787 2022-02-18T11:19:39Z 2022-02-18T11:29:20Z Failed jobs in eu-central and us-east stack

Due to a misconfiguration, there was a brief error present in job processing in https://connection.keboola.com/ https://connection.eu-central-1.keboola.com/ stacks which caused some jobs to fail (14 jobs was affected). The error was present from 12:10:01 to 12:14:07 CET. We apologize for the inconvenience.

tag:status.keboola.com,2013:Post/1796444 2022-02-17T16:12:14Z 2022-02-17T16:20:08Z Some orchestration phases doing no work

We see cases of orchestrations skipping their tasks in phases. The symptom is that orchestration finishes with success but in very short time with no tasks executed. We do not know yet what are the exact properties of an orchestration to trigger the issue.

We are rolling back todays changes in orchestration processing which should fix this issue. 

Update 16:15 UTC The issue is now fixed.

tag:status.keboola.com,2013:Post/1793541 2022-02-09T20:51:21Z 2022-02-09T20:51:21Z Microsoft Azure services planned maintenance

This applies only to Azure North Europe stack (https://connection.north-europe.azure.keboola.com).

Microsoft Azure will be performing maintenance on database services between 16:00 UTC on Feb 10 2022 and 08:00 UTC on 11 Feb 2022. This maintenance will last 60 to 120 seconds and may happen multiple times in this window. This may cause

  • brief platform unavailability, 
  • job retries, 
  • job errors and
  • very rarely also duplicate job execution

As we do not know when this will happen exactly we're unable to plan a maintenance window. If you encounter any issues due to this maintenance please contact our support. 

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1793325 2022-02-09T11:20:29Z 2022-02-09T15:24:46Z Azure North Europe Orchestrations with More than 10 Phases May Fail

2022-02-09 11:00 UTC - We discovered an issue that may cause orchestrations that have more than 10 phases to fail with the error "there is already X waiting orchestrations". 

The first occurrence of this error was on 22-02-08 at 6:40 AM UTC. 

We're currently working on mitigating the issue and will update when it gets resolved or in 60 minutes.

2022-02-09 12:00 UTC We've prepared a fix. We will let you know as soon as the fix is deployed.

2022-02-09 15:24 UTC The issue has been fixed. We are sorry for any inconvenience.

tag:status.keboola.com,2013:Post/1792473 2022-02-07T09:20:13Z 2022-02-07T09:20:13Z New Telemetry coming on Thursday, February 10th

On Thursday, February 10th, we will be updating our Telemetry to improve the data available from the Telemetry Data extractor. Please follow the post on our Community page for more details: https://community.keboola.com/post/new-telemetry-coming-on-thu-february-10th-6200cc1ed1d0cf25b64e9c68