Delayed processing of job in Azure North Europe stack

2021-12-16 17:40 UTC We are experiencing number of jobs in waiting state more than usual. We continue investigating the issue.

2021-12-16 18:45 UTC The issue has been resolved, everything is working as expected. 

2021-12-16 19:10 UTC Further investigation revealed the parallel config rows execution might have been affected leaving some jobs stuck. Please review your jobs run as a configuration in parallel, terminate such jobs if they seem to be stuck and run them again.

Log4j zero-day vulnerability update

Regarding the security issue (CVE-2021-44228) with the Log4j zero-day vulnerability, we have completed all necessary steps to investigate if our system had been compromised.

After a deep investigation, we can say that there were no security issues or breaches on our systems. We don't utilize Log4j for our main services.

We also checked all 3rd party services we are using, but thanks to our very strict security standards those services are not publicly accessible, they run in a separate environment (disconnected from customer's data), and cannot be used as an attack vector. We also haven't received any security issues from our SaaS partners.

We take the security of your data very seriously, so we applied additional threat detection regarding the Log4j security issue.

Please reach out if you have any questions.

Column, Table, and Bucket metadata overwritten – repair

We found a way to repair the overwritten column, table, and bucket user metadata, caused by the incident reported here: Column, table or bucket metadata possibly overwritten. 

The incident affected column, table, and bucket metadata that had two (or more) metadata with the same key but a different provider. If metadata had been updated for one provider, values were changed for all of them. This could have led to a rewrite of user-defined metadata for column type, length, or any other metadata. These metadata are used for input mapping. Existing mappings were not affected. But you may be facing a problem when you create a new input mapping and use any table with affected metadata that works in existing mappings. This may cause a problem with the newly created input mapping. As a temporary solution, you can reset this user-defined metadata for a data type manually to the correct value.

We will find all affected metadata and obtain the correct values by “replaying” update metadata storage events. For all user metadata we fix, we also update the time stamp. While repairing the metadata, we will disable a project for a short time (we expect seconds or a few minutes at most), during which you will be unable to use the project. We apologize for any inconvenience. In the following days, we will add a message (shown on the project dashboard) to the affected projects with the expected date when the process to repair corrupted metadata will start. 

Any changes to the metadata after the issue has been fixed (December 3, 9:03 UTC) will be also taken into account and will not be lost.

Corrupted telemetry data

Dec 9 2021, 11:38 UTC We are currently investigating an issue regarding corrupted data obtained via our Telemetry Data component (keboola.ex-telemetry-data).

Next update in 60 minutes.

Dec 9 2021, 13:07 UTC We have identified the issue in our telemetry data and fixed it. The issue might cause job with no existing configuration to not be assigned to its actual project telemetry data.

We have modified the component so that it now loads data using full loads only. To ensure that you have the correct telemetry data, all you need to do is run the extractor (or wait for your pipeline to run it). We will re-implement the incremental fetching in the following months.

We are very sorry for any inconvenience caused. 

High error rate in Developer Portal

Service disruption in AWS US are also causing problems in Devloper Porta (apps-api.keboola.com & components.keboola.com). You may see intermittent errors 5XX, refreshing the page can help. 

AWS acknowledge the service disruption and are active working towards recover. See https://status.aws.amazon.com/ for more details. Once the AWS service disruption is over our services should start running smoothly again. Next update in 60 minutes or when new information is available.

UPDATE 19:10 UTC Service disruption on the AWS US region persists. We continue to monitor the situation. Next update in 2 hours.

UPDATE 21:40 UTC Service disruption on the AWS US region are reducing. Our affected services are showing significant improvement. Next update in 12 hours or as new information is available.

UPDATE Dec 8th, 07:12 UTC Most services in the affected AWS region have already been repaired. Our services are operating normally. Next update in 4 hours or as new information is available.

UPDATE Dec 8th, 15:130 UTC We're sorry for the late update. AWS services have already recovered. Everything should be running without any issues now.

High error rate in AWS US stack

We're investigating high error rate in AWS US (https://connection.keboola.com) stack. Next update in 60 minutes or when new information is available. 

We're sorry for this inconvenience.

UPDATE 15:55 UTC It seems that the root cause is a service disruption in AWS. We're waiting for official confirmation of this issue. You may see intermittent errors (404 or 500), refreshing the page can help. Next update in 60 minutes or when new information is available.

UPDATE 16:25 UTC Service disruption in AWS US region may also cause issues in other Keboola Connection stacks, e. g. when running components jobs. Next update in 60 minutes or when new information is available.

UPDATE 16:55 UTC AWS acknowledge the service disruption and are active working towards recover. See https://status.aws.amazon.com/ for more details. Once the AWS service disruption is over our services should start running smoothly again. Next update in 60 minutes or when new information is available.

UPDATE 18:27 UTC Service disruption on the AWS US region persists. The availability of our services has improved slightly but we are still experiencing errors regarding Workspaces. We continue to monitor the situation. Next update in 2 hours.

UPDATE 21:40 UTC Service disruption on the AWS US region are reducing. Our affected services are showing significant improvement. Next update in 12 hours or as new information is available.

UPDATE Dec 8th, 07:12 UTC Most services in the affected AWS region have already been repaired. Our services are operating normally. Next update in 4 hours or as new information is available.


UPDATE Dec 8th, 15:130 UTC We're sorry for the late update. AWS services have already recovered. Everything should be running without any issues now.


Slow event processing in AWS US

We're investigating possible intermittent slower event processing in AWS US stack (https://connection.keboola.com/). The API response can be delayed up to 2 seconds. This may cause 

  • poor UI response and
  • slowing down of jobs that write metadata (e. g. column datatypes).

This issue does not cause any job failures.

We're sorry for this inconvenience. Next update in 24 hours or as new information is available.

UPDATE Dec 4th, 15:00 UTC We have identified a few possible root causes of this issue and minimized the impact. The situation is now stable, but we're monitoring it closely. 

This is the last update and we'll reopen the communication here only if the situation escalates. 

Column, table or bucket metadata possibly overwritten

We’re investigating possible issue with column, table and bucket metadata in all stacks. We’re seeing suspicious behaviour when running output mapping from a workspace (e. g. Snowflake transformation or SQL Workspace). Under yet unknown conditions column, table or bucket metadata may have been overwritten. This should not affect any existing configurations or jobs. 

Next update in 24 hours or when new information is available. 


UPDATE Dec 2 16:25 UTC: We can confirm issue with metadata, the issue occurs when column, table or bucket metadata has two (or more) metadata with the same key but different provider. Then if metadata is updated for one provider, values will change for all of them. We still investigating scope of issue, next update in 24 hours or when new information is available.

UPDATE Dec 3 9:03 UTC: We have fixed the issue that caused the overwriting of metadata. We found out the problem influenced only buckets, tables or columns in their own scope (no data was mixed between projects, buckets, tables and columns). Now we are investigating scope of affected projects. We're also examining the option of backfilling the overwritten data from backups.Next update in a week or sooner if there are new information available.

We are sorry for the inconvenience.

FTP Extractor new files only redownloaded

Recent release 1.7.0 of FTP extractor released on 30 Nov 2021 09:31 UTC caused misbehaviour of Only New Files flag which led to redownload of all matching files. If you encountered any problems of FTP extractor configuration after this release please contact Keboola support from your project for assistance.

We are sorry for the inconvenience.

Failing Facebook Ads and Instagram extractors

Today, 26th November 2021, between 10:00 and 11:00 UTC we experienced Facebook ads and Instagram extractors failing on an internal error. We fixed the problem and the extractors should be working as expected. If you run jobs in between the mentioned timeframe, please restart your jobs. We are sorry for the inconvenience.