Today, 26th November 2021, between 10:00 and 11:00 UTC we experienced Facebook ads and Instagram extractors failing on an internal error. We fixed the problem and the extractors should be working as expected. If you run jobs in between the mentioned timeframe, please restart your jobs. We are sorry for the inconvenience.
12.11.2021 10:43 CET
We are experiencing increased error rate for components communication with Google APIs
Google reports there are several services disruptions:
We continue to monitor the situation.
12.11.2021 14:11 CET
We no longer see increased component failures, everything is working as expected.
Updating state using configuration update API call has been deprecated for some time and will soon be completely removed. Please make sure your integrations are not using it anymore.
The legacy state update API has been part of the configuration update API. The behavior was inconsistent, because updating state didn't create a new version of a configuration. The state update has therefore been moved to a dedicated API call.
2021-10-19 23:07 UTC - We are investigating job processing delays in connection.eu-central-1.keboola.com. Next update when new information will be available or in hour.
Update 2021-10-20 00:01 UTC We have identified the root cause and working on a fix. Next update when new information will be available or in hour.
Update 2021-10-20 00:45 UTC Everything should be running without any issues now. We're sorry for this inconvenience.
Some jobs of keboola.ex-google-adwords-reports-v201809 in the Azure North Europe stack fail immediately with an error. The job details is missing the job start date and for PAYG customers the billing consumes all available credits.
We're investigating this issue and will update this status in 60 minutes or when an update is available.
UPDATE 18:43 UTC: We have found the root cause and we're working on a fix. Next update in 60 minutes or when an update is available.
UPDATE 19:45 UTC: We have fixed billing stats in all affected projects. The root cause has not been fixed yet that means a new job of the keboola.ex-google-adwords-reports-v201809 component will again consume invalid number of credits. We'll update the billing stats later tonight and tomorrow early morning to keep the projects running smoothly. Next update as soon as we have any news or at 07:00 UTC.
UPDATE Oct 11 06:55 UTC: Unfortunately we're still seeing failing jobs of the affected component after releasing the fix. We're further investigating the issue and preparing a new fix. Next update in 6 hours (13:00 UTC) or when new information is available.
UPDATE Oct 11 11:40 UTC: We have deployed and verified the fix. We'll continue monitoring jobs closely to see double check for any re-occurrences.
Oct 9, 2021, 10:00 CET - Azure EU is down for scheduled maintenance: https://status.keboola.com/azure-eu-maintenance-announcement-1
Oct 9, 2021, 10:38 CET - Azure EU is back and fully operational. Thanks for your patience.
We have added new outbound IPs for connection.north-europe.azure.keboola.com stack.
Please make sure you have whitelisted all these IP addresses in your firewalls to allow Keboola Connection to successfully connect to your system:
- 40.127.144.42
- 20.82.252.94 (new)
- 20.82.252.129 (new)
- 20.82.252.124 (new)
For your convenience, you can process programmatically fetch and process the list of existing IP addresses in JSON format.
Read more about outbound IP addresses in documentation.
On 07 Oct 2021 we experienced failing Google Drive extractor jobs run between 12:22 UTC and 12:49 UTC in all regions. The issue is now resolved.
04 Oct 2021 07:27 UTC - We are investigating increased backlog of jobs in connection.eu-central-1.keboola.com stack. Next update in 60 minutes or as new information becomes available.
04 Oct 2021 07:45 UTC - We have temporarily boosted the Snowflake capacity and backlog is cleared. All operations are back to normal and everything is fully working.
2021-09-29 08:11 UTC - We are investigating higher error rate and increased latencies in https://connection.north-europe.azure.keboola.com stack. Next update when new information will be available or in hour.
2021-09-29 09:15 UTC - The error rate has dropped, but we are still investigating higher latencies in internal requests. Next update when new information will be available or in hour.
2021-09-29 10:12 UTC - We are still investigating higher latencies in internal requests. Next update when new information will be available or in hour.
2021-09-29 12:00 UTC - We have identified one of 3rd party services to be the possible root cause of this issue and we are working with support team of this service to fix it.
2021-09-29 13:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or in hour.
2021-09-29 14:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or in hour.
2021-09-29 16:15 UTC - Still working with 3rd party service support on a solution. Next update when new information will be available or tomorrow morning.
2021-09-29 19:05 UTC - Still working with 3rd party service support on a solution. Job execution times also might be longer due to this incident. Next update when new information will be available or tomorrow morning.
2021-09-29 22:01 UTC - We've applied a fix which reduced the latencies, job processing times should be back to normal. We are investigating occasional error responses and we are still working with 3rd party service support on a solution. Next update tomorrow morning.
2021-09-30 09:20 UTC - Platform is stabilized after yesterday's fix, however we still see occasional errors. We are still working with 3rd party service support on a solution. Next update when new information will be available or in four hours.
2021-09-30 17:57 UTC - We have implemented another fix which decreased the error rate. We continue to monitor the situation but unless the situation escalates we consider this incident resolved.