The latest version (8.2.0) of Microsoft SQL Server Extractor terminates with an internal error. This version was deployed yesterday, and we are currently performing a rollback Next update will be available in 15 minutes.
[Resolved] UTC 07:56: We have rollback to version 8.1.1, and the extractions are now functioning without any issues. We apologize for any inconvenience caused.
We have discovered that some writer jobs in the projects that were migrated to the new job queue (Queue V2) after the beginning of May are missing information about the data transferred. That information is used to calculate the number of credits consumed by those jobs.
We will deploy a fix tomorrow (10th Aug), which will add missing credits to the jobs affected. For affected projects regularly using writers, the result may be that they have a higher recorded consumption of credits.
The issue is related solely to the telemetry and does not affect Keboola Connection in any way. Moreover, it affects the telemetry only for projects that were recently migrated to Queue V2.
UPDATE 2023-08-10 11:04 UTC: The fix was deployed and the affected writer jobs show consumed credits again.
Detailed description of the issue
When a project is migrated to Queue V2, any jobs created in the past several months are also migrated, so that the user can keep track of what is going on in their Keboola project UI. Jobs in both Queue V1 (the old queue) and Queue V2 contain information about the data transferred by these jobs as different metrics. However, this information is not passed from an original job to the corresponding migrated one during the migration process.
Generally, Queue V1 jobs take precedence over Queue V2 jobs. To prevent any issues, they are used in the telemetry calculations, rather than the migrated jobs, as they have the original data.
In May, to speed up the telemetry calculations, the input mapping of Queue V1 jobs in a transformation was switched so that only data updated in the last 30 days was incrementally loaded for further processing.
As noted above, when a project was migrated to Queue V2, migrated jobs were also created. So, when processing jobs, loads of migrated jobs from the past several months were processed but only recently updated Queue V1 jobs (from the last 30 days) were processed alongside them. Thus, the older Queue V1 jobs could not take precedence over the newer migrated Queue V2 jobs, so the latter were incorrectly used for the telemetry output. Hence, information is missing about transferred data, resulting in no credits.
For the bug fix, a transformation will now always load the entire history of Queue V1 jobs to prevent migrated jobs from incorrectly being used in telemetry calculations.
We are observing an increased number of faulty storage jobs, resulting in the error message "Cannot import data from Storage API“ in connection.eu-central-1.keboola.com. The main cause has been identified and resolved, and now all systems should be running smoothly. We will continue to monitor the situation, and the next update will be provided in 30 minutes.
We apologize for any inconvenience caused.
UPDATE 7:20 UTC [resolved] All systems are functioning normally, and the incident has been resolved and closed.
Since 09:39 UTC we're seeing job starting with delays on https://connection.north-europe.azure.keboola.com/ We're investigating the situation. Next update in 30 minutes.
UPDATE 10:30 UTC we managed to find the root cause, new worker nodes have a problem authorization accessing the container registry, we are working on a fix. Next update in 30 minutes.
UPDATE 10:57UTC The problem with authorization to container registry is now solved. All systems are now operating normally.
We apologize for any inconvenience caused.
We are adding new outbound IP addresses for the connection.keboola.com and connection.eu-central-1.keboola.com stacks for Queue V2. These new addresses are available now, but are not yet being used automatically.
This update is important for Keboola Connection customers. It may affect their ability to connect to their resources, particularly if they are behind a firewall.
What are outbound IP addresses?
Outbound IP addresses are unique addresses assigned to a device for the purpose of identifying it and sending information over the Internet. When Keboola Connection customers connect to their resources (typically databases), those resources are usually behind a firewall. In order for Keboola Connection to connect to those resources, customers whitelist our outbound IP addresses.
What must I do?
If your resources are behind a firewall, ensure that all the new IP addresses are added to the whitelist, so as to enable connection to your system through Keboola Connection.
Use the “Test with new outbound IPs” feature to check the connectivity for any or all configurations in the credentials section. This will verify that your resources are accessible from the new addresses.
- If the connection works well, switch the project to the new IP addresses. In the event of problems, you can temporarily revert to your original IP address and contact our support team for assistance.
If you have multiple projects in your organization and have already tested the connection from the new IP addresses, you can ask our support team for help. They can switch all your projects at once, so you don’t have to do it individually for each one.
If you are not yet making use of Queue V2 for your projects, don’t hesitate to whitelist, as this will speed up your migration to the new queue in the future.
By when must I do it?
To ensure uninterrupted connectivity, the new IP addresses must be whitelisted by June 30, 2023. Otherwise, you run the risk of your connection not working. If the above update is not done manually by this date, Keboola Connection will perform the switch globally. To make sure of a smooth change, please add the new IP addresses to your whitelist and switch your projects as soon as you can.
Current list of outbound IP addresses
connection.keboola.com
-
Queue V2
52.7.83.13652.20.72.254-
3.222.3.15(new) -
34.206.78.206(new) -
3.213.250.110(new) -
107.22.113.103(new) -
54.144.9.113(new) -
54.204.61.145(new) -
34.239.7.70(new) -
3.217.232.144(new)
- Email delivery
149.72.196.5
- Queue V1 - legacy syrup services
34.224.0.18834.200.169.17752.206.109.12634.203.87.137
connection.eu-central-1.keboola.com
- Queue V2
3.66.248.1803.64.150.30-
35.157.62.225(new) -
3.71.156.204(new) -
3.74.28.187(new) -
18.158.155.128(new) -
35.157.208.189(new) -
3.72.243.47(new) -
18.193.225.37(new) -
3.127.158.56(new)
- Email delivery
149.72.196.5
- Queue V1 - legacy syrup services
35.157.170.22935.157.93.175
For your convenience, you can programmatically fetch and process the list of existing IP addresses in JSON format. Read more about outbound IP addresses in documentation.
We appreciate your cooperation in making this transition as smooth as possible.
2023-04-21 14:38 UTC - We experienced another incident on connection.eu-central-1.keboola.com. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.
2023-04-03 10:15 UTC - We are investigating delayed telemetry data. More information within the hour.
2023-04-03 11:30 UTC - Delayed telemetry data on all Keboola Connection stacks have been recorded since approximately 20:00 UTC on March 29. We were able to determine the root cause and perform a backfill. Now all telemetry data tables are up-to-date.
We are very sorry for the inconvenience. If you encounter any discrepancies, please contact us immediately.
We are aware of a bug in the user interface that makes it impossible to undo the action after clicking collapse tables in IM.
We are sorry for complications. A bug fix will be deployed within minutes.
Update 14:05 UTC The fix has been released to all stacks.
2022-08-04 12:05 UTC We are investigating internal job errors. Next update in 30 minutes.
2022-08-04 12:36 UTC We experienced two waves of internal errors, between 11:50-11:57 and 12:00-12:07 UTC. The root cause has been removed and I don't see any other errors in the reporting. We continue to monitoring the situation. Next update in 30 minutes.
2022-08-04 13:06 UTC All Keboola Connection services are running normally. The incident is resolved.
We apologize for any inconvenience.
2022-05-17 09:54 CET Due to the RDS upgrade, there was a small outage of the API queue and you may have observed a small number of errors in the connection.keboola.com UI.
The situation is now under control and our services are running normally.