Delayed processing of jobs in AWS eu-central-1 stack

2021-10-19 23:07 UTC - We are investigating  job processing delays in connection.eu-central-1.keboola.com.  Next update when new information will be available or in hour.

Update 2021-10-20 00:01 UTC We have identified the root cause and working on a fix. Next update when new information will be available or in hour.

Update 2021-10-20 00:45 UTC Everything should be running without any issues now. We're sorry for this inconvenience. 

Google AdWords extractor jobs consume all credits in PAYG

Some jobs of keboola.ex-google-adwords-reports-v201809 in the Azure North Europe stack fail immediately with an error. The job details is missing the job start date and for PAYG customers the billing consumes all available credits.

We're investigating this issue and will update this status in 60 minutes or when an update is available. 

UPDATE 18:43 UTC: We have found the root cause and we're working on a fix. Next update in 60 minutes or when an update is available. 

UPDATE 19:45 UTC: We have fixed billing stats in all affected projects. The root cause has not been fixed yet that means a new job of the keboola.ex-google-adwords-reports-v201809 component will again consume invalid number of credits. We'll update the billing stats later tonight and tomorrow early morning to keep the projects running smoothly. Next update as soon as we have any news or at 07:00 UTC. 

UPDATE Oct 11 06:55 UTC: Unfortunately we're still seeing failing jobs of the affected component after releasing the fix. We're further investigating the issue and preparing a new fix. Next update in 6 hours (13:00 UTC) or when new information is available. 

UPDATE Oct 11 11:40 UTC: We have deployed and verified the fix. We'll continue monitoring jobs closely to see double check for any re-occurrences. 

Azure EU Maintenance

Oct 9, 2021, 10:00 CET - Azure EU is down for scheduled maintenance: https://status.keboola.com/azure-eu-maintenance-announcement-1

Oct 9, 2021, 10:38 CET - Azure EU is back and fully operational. Thanks for your patience.

Additional outbound IPs for connection.north-europe.azure.keboola.com

We have added new outbound IPs for connection.north-europe.azure.keboola.com stack.

Please make sure you have whitelisted all these IP addresses in your firewalls to allow Keboola Connection to successfully connect to your system:

  • 40.127.144.42
  • 20.82.252.94 (new)
  • 20.82.252.129 (new)
  • 20.82.252.124 (new)

For your convenience, you can process programmatically fetch and process the list of existing IP addresses in JSON format.

Read more about outbound IP addresses in documentation.

Azure EU higher error rate

2021-09-29 08:11 UTC - We are investigating higher error rate and increased latencies in https://connection.north-europe.azure.keboola.com stack. Next update when new information will be available or in hour.

2021-09-29 09:15 UTC - The error rate has dropped, but we are still investigating higher latencies in internal requests. Next update when new information will be available or in hour.

2021-09-29 10:12 UTC - We are still investigating higher latencies in internal requests. Next update when new information will be available or in hour.

2021-09-29 12:00 UTC - We have identified one of 3rd party services to be the possible root cause of this issue and we are working with support team of this service to fix it.

2021-09-29 13:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or in hour.

2021-09-29 14:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or in hour.

2021-09-29 16:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or  tomorrow morning.

2021-09-29 19:05 UTC - Still working with 3rd party service support on a solution. Job execution times also might be longer due to this incident. Next update when new information will be available or tomorrow morning.

2021-09-29 22:01 UTC - We've applied a fix which reduced the latencies, job processing times should be back to normal. We are  investigating occasional error responses and we are still working with 3rd party service support on a solution. Next update tomorrow morning.

2021-09-30 09:20 UTC - Platform is stabilized after yesterday's fix, however we still see occasional errors. We are still working with 3rd party service support on a solution. Next update when new information will be available or in four hours.

2021-09-30 17:57 UTC - We have implemented another fix which decreased the error rate. We continue to monitor the situation but unless the situation escalates we consider this incident resolved.

Azure EU Maintenance Announcement

A maintenance of Azure EU Keboola Connection (connection.north-europe.azure.keboola.com) will take place on Saturday, Oct 9th, 2021 from 10:00 CET and should take less than two hours.

During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

Orchestrations and running component jobs should be generally delayed, but not interrupted.  However, feel free to re-schedule your saturday orchestrations to avoid this maintenance window.

Sending emails with empty body from Connection

2021-09-21 08:27 UTC -  We have noticed that emails from our platforms are being sent with empty body (email..title, email..body etc.).  Next update in one hour.

2021-09-21 11:14 UTC  - we successfully rolled back the previous working version. 

We're sorry for this inconvenience.

Failures Saving Queries and Scripts

2021-09-13 08:15:10 UTC:  there was a release of the UI which broke the parsing of transformations into separate queries.  Queries that were not edited this morning were not affected.  

2021-09-13 11:48:29 UTC: We identified broken version and applied a fix

We advise all users to reload their browsers and resave any affected transformations