High API errors rate in AWS service

The AWS service in the US-EAST-1 region where we operate the AWS US stack is disrupted by networking connectivity issues of some instances in one availability zone. So far our service does not seem to be directly affected but it may reach some of its parts eventually. We are monitoring the situation and let you know in an hour about its progress.

UPDATE: The Keboola Academy site is down too most probably due to this outage.

UPDATE 14:30 CET: The problem was identified by the AWS team (a power outage in one data center in USE1-AZ4 availability zone) and they are already restoring the power and recovering from the problem. It seems that some other services like Slack and SolarWinds Papertrail were affected too but the Connection seems to be unaffected except for some short job processing delays. We are still monitoring the situation and let you know about the situation in an hour.

UPDATE 15:30 CET: Power to all affected instances and network devices was restored and recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone is being seen. The impact on the Connection stack should be almost zero by now.

Delayed processing of job in Azure North Europe stack

2021-12-16 17:40 UTC We are experiencing number of jobs in waiting state more than usual. We continue investigating the issue.

2021-12-16 18:45 UTC The issue has been resolved, everything is working as expected. 

2021-12-16 19:10 UTC Further investigation revealed the parallel config rows execution might have been affected leaving some jobs stuck. Please review your jobs run as a configuration in parallel, terminate such jobs if they seem to be stuck and run them again.

Log4j zero-day vulnerability update

Regarding the security issue (CVE-2021-44228) with the Log4j zero-day vulnerability, we have completed all necessary steps to investigate if our system had been compromised.

After a deep investigation, we can say that there were no security issues or breaches on our systems. We don't utilize Log4j for our main services.

We also checked all 3rd party services we are using, but thanks to our very strict security standards those services are not publicly accessible, they run in a separate environment (disconnected from customer's data), and cannot be used as an attack vector. We also haven't received any security issues from our SaaS partners.

We take the security of your data very seriously, so we applied additional threat detection regarding the Log4j security issue.

Please reach out if you have any questions.

Column, Table, and Bucket metadata overwritten – repair

We found a way to repair the overwritten column, table, and bucket user metadata, caused by the incident reported here: Column, table or bucket metadata possibly overwritten. 

The incident affected column, table, and bucket metadata that had two (or more) metadata with the same key but a different provider. If metadata had been updated for one provider, values were changed for all of them. This could have led to a rewrite of user-defined metadata for column type, length, or any other metadata. These metadata are used for input mapping. Existing mappings were not affected. But you may be facing a problem when you create a new input mapping and use any table with affected metadata that works in existing mappings. This may cause a problem with the newly created input mapping. As a temporary solution, you can reset this user-defined metadata for a data type manually to the correct value.

We will find all affected metadata and obtain the correct values by “replaying” update metadata storage events. For all user metadata we fix, we also update the time stamp. While repairing the metadata, we will disable a project for a short time (we expect seconds or a few minutes at most), during which you will be unable to use the project. We apologize for any inconvenience. In the following days, we will add a message (shown on the project dashboard) to the affected projects with the expected date when the process to repair corrupted metadata will start. 

Any changes to the metadata after the issue has been fixed (December 3, 9:03 UTC) will be also taken into account and will not be lost.

FTP Extractor new files only redownloaded

Recent release 1.7.0 of FTP extractor released on 30 Nov 2021 09:31 UTC caused misbehaviour of Only New Files flag which led to redownload of all matching files. If you encountered any problems of FTP extractor configuration after this release please contact Keboola support from your project for assistance.

We are sorry for the inconvenience.

Failing Facebook Ads and Instagram extractors

Today, 26th November 2021, between 10:00 and 11:00 UTC we experienced Facebook ads and Instagram extractors failing on an internal error. We fixed the problem and the extractors should be working as expected. If you run jobs in between the mentioned timeframe, please restart your jobs. We are sorry for the inconvenience.

Support for legacy state update in configuration update API will be removed

Updating state using configuration update API call has been deprecated for some time and will soon be completely removed. Please make sure your integrations are not using it anymore.

The legacy state update API has been part of the configuration update API. The behavior was inconsistent, because updating state didn't create a new version of a configuration. The state update has therefore been moved to a dedicated API call.

Delayed processing of jobs in AWS eu-central-1 stack

2021-10-19 23:07 UTC - We are investigating  job processing delays in connection.eu-central-1.keboola.com.  Next update when new information will be available or in hour.

Update 2021-10-20 00:01 UTC We have identified the root cause and working on a fix. Next update when new information will be available or in hour.

Update 2021-10-20 00:45 UTC Everything should be running without any issues now. We're sorry for this inconvenience. 

Google AdWords extractor jobs consume all credits in PAYG

Some jobs of keboola.ex-google-adwords-reports-v201809 in the Azure North Europe stack fail immediately with an error. The job details is missing the job start date and for PAYG customers the billing consumes all available credits.

We're investigating this issue and will update this status in 60 minutes or when an update is available. 

UPDATE 18:43 UTC: We have found the root cause and we're working on a fix. Next update in 60 minutes or when an update is available. 

UPDATE 19:45 UTC: We have fixed billing stats in all affected projects. The root cause has not been fixed yet that means a new job of the keboola.ex-google-adwords-reports-v201809 component will again consume invalid number of credits. We'll update the billing stats later tonight and tomorrow early morning to keep the projects running smoothly. Next update as soon as we have any news or at 07:00 UTC. 

UPDATE Oct 11 06:55 UTC: Unfortunately we're still seeing failing jobs of the affected component after releasing the fix. We're further investigating the issue and preparing a new fix. Next update in 6 hours (13:00 UTC) or when new information is available. 

UPDATE Oct 11 11:40 UTC: We have deployed and verified the fix. We'll continue monitoring jobs closely to see double check for any re-occurrences. 

Azure EU Maintenance

Oct 9, 2021, 10:00 CET - Azure EU is down for scheduled maintenance: https://status.keboola.com/azure-eu-maintenance-announcement-1

Oct 9, 2021, 10:38 CET - Azure EU is back and fully operational. Thanks for your patience.