Telemetry: Missing credits for writer jobs in projects recently migrated to the new queue [resolved]

We have discovered that some writer jobs in the projects that were migrated to the new job queue (Queue V2) after the beginning of May are missing information about the data transferred. That information is used to calculate the number of credits consumed by those jobs.

We will deploy a fix tomorrow (10th Aug), which will add missing credits to the jobs affected. For affected projects regularly using writers, the result may be that they have a higher recorded consumption of credits.

The issue is related solely to the telemetry and does not affect Keboola Connection in any way. Moreover, it affects the telemetry only for projects that were recently migrated to Queue V2.

UPDATE 2023-08-10 11:04 UTC: The fix was deployed and the affected writer jobs show consumed credits again.

Detailed description of the issue

When a project is migrated to Queue V2, any jobs created in the past several months are also migrated, so that the user can keep track of what is going on in their Keboola project UI. Jobs in both Queue V1 (the old queue) and Queue V2 contain information about the data transferred by these jobs as different metrics. However, this information is not passed from an original job to the corresponding migrated one during the migration process.

Generally, Queue V1 jobs take precedence over Queue V2 jobs. To prevent any issues, they are used in the telemetry calculations, rather than the migrated jobs, as they have the original data.

In May, to speed up the telemetry calculations, the input mapping of Queue V1 jobs in a transformation was switched so that only data updated in the last 30 days was incrementally loaded for further processing.

As noted above, when a project was migrated to Queue V2, migrated jobs were also created. So, when processing jobs, loads of migrated jobs from the past several months were processed but only recently updated Queue V1 jobs (from the last 30 days) were processed alongside them. Thus, the older Queue V1 jobs could not take precedence over the newer migrated Queue V2 jobs, so the latter were incorrectly used for the telemetry output. Hence, information is missing about transferred data, resulting in no credits.

For the bug fix, a transformation will now always load the entire history of Queue V1 jobs to prevent migrated jobs from incorrectly being used in telemetry calculations.

Storage job failures in the AWS EU stack

We are observing an increased number of faulty storage jobs, resulting in the error message "Cannot import data from Storage API“ in connection.eu-central-1.keboola.com. The main cause has been identified and resolved, and now all systems should be running smoothly. We will continue to monitor the situation, and the next update will be provided in 30 minutes.

We apologize for any inconvenience caused.

UPDATE 7:20 UTC [resolved] All systems are functioning normally, and the incident has been resolved and closed.

Slowdown of processing of jobs on Azure North Europe stack [resolved]

Since 09:39 UTC we're seeing job starting with delays on https://connection.north-europe.azure.keboola.com/ We're investigating the situation. Next update in 30 minutes.

UPDATE 10:30 UTC we managed to find the root cause, new worker nodes have a problem authorization accessing the container registry, we are working on a fix. Next update in 30 minutes.

UPDATE 10:57UTC The problem with authorization to container registry is now solved. All systems are now operating normally.

We apologize for any inconvenience caused.

New Outbound IP Addresses for Keboola Connection: Action Required

We are adding new outbound IP addresses for the connection.keboola.com and connection.eu-central-1.keboola.com stacks for Queue V2. These new addresses are available now, but are not yet being used automatically.

This update is important for Keboola Connection customers. It may affect their ability to connect to their resources, particularly if they are behind a firewall.

What are outbound IP addresses?

Outbound IP addresses are unique addresses assigned to a device for the purpose of identifying it and sending information over the Internet. When Keboola Connection customers connect to their resources (typically databases), those resources are usually behind a firewall. In order for Keboola Connection to connect to those resources, customers whitelist our outbound IP addresses.

What must I do?

  1. If your resources are behind a firewall, ensure that all the new IP addresses are added to the whitelist, so as to enable connection to your system through Keboola Connection. 

  2. Use the “Test with new outbound IPs” feature to check the connectivity for any or all configurations in the credentials section. This will verify that your resources are accessible from the new addresses.

  3. If the connection works well, switch the project to the new IP addresses. In the event of problems, you can temporarily revert to your original IP address and contact our support team for assistance.

If you have multiple projects in your organization and have already tested the connection from the new IP addresses, you can ask our support team for help. They can switch all your projects at once, so you don’t have to do it individually for each one.

If you are not yet making use of Queue V2 for your projects, don’t hesitate to whitelist, as this will speed up your migration to the new queue in the future.

By when must I do it?

To ensure uninterrupted connectivity, the new IP addresses must be whitelisted by June 30, 2023. Otherwise, you run the risk of your connection not working. If the above update is not done manually by this date, Keboola Connection will perform the switch globally. To make sure of a smooth change, please add the new IP addresses to your whitelist and switch your projects as soon as you can.

Current list of outbound IP addresses

connection.keboola.com

  • Queue V2
    • 52.7.83.136
    • 52.20.72.254
    • 3.222.3.15 (new)
    • 34.206.78.206 (new)
    • 3.213.250.110 (new)
    • 107.22.113.103 (new)
    • 54.144.9.113 (new)
    • 54.204.61.145 (new)
    • 34.239.7.70 (new)
    • 3.217.232.144 (new)
  • Email delivery
    • 149.72.196.5
  • Queue V1 - legacy syrup services
    • 34.224.0.188
    • 34.200.169.177
    • 52.206.109.126
    • 34.203.87.137

connection.eu-central-1.keboola.com

  • Queue V2
    • 3.66.248.180
    • 3.64.150.30
    • 35.157.62.225 (new)
    • 3.71.156.204 (new)
    • 3.74.28.187 (new)
    • 18.158.155.128 (new)
    • 35.157.208.189 (new)
    • 3.72.243.47 (new)
    • 18.193.225.37 (new)
    • 3.127.158.56 (new)
  • Email delivery
    • 149.72.196.5
  • Queue V1 - legacy syrup services
    • 35.157.170.229
    • 35.157.93.175

For your convenience, you can programmatically fetch and process the list of existing IP addresses in JSON format. Read more about outbound IP addresses in documentation.

We appreciate your cooperation in making this transition as smooth as possible.

Delayed telemetry data [resolved]

2023-04-03 10:15 UTC - We are investigating delayed telemetry data. More information within the hour.

2023-04-03 11:30 UTC - Delayed telemetry data on all Keboola Connection stacks have been recorded since approximately 20:00 UTC on March 29. We were able to determine the root cause and perform a backfill. Now all telemetry data tables are up-to-date.

We are very sorry for the inconvenience. If you encounter any discrepancies, please contact us immediately.

AWS us-east-1 internal job errors [resolved]

2022-08-04 12:05 UTC We are investigating internal job errors. Next update in 30 minutes.

2022-08-04 12:36 UTC We experienced two waves of internal errors, between 11:50-11:57 and 12:00-12:07 UTC. The root cause has been removed and I don't see any other errors in the reporting. We continue to monitoring the situation. Next update in 30 minutes.

2022-08-04 13:06 UTC All Keboola Connection services are running normally. The incident is resolved. 

We apologize for any inconvenience.

Stuck jobs in AWS us-east-1 [resolved]

2022-04-15 10:30 UTC - We are investigation stuck jobs on a new queue. Some jobs will probably end up with an internal error. Next update in 15 min.

2022-04-15 10:50 UTC - The situation was resolved, however 6 jobs were terminated by an internal error with no automatic restart. We apologize for the inconvenience and please feel free to restart your jobs manually.

Affected jobs:

  • job-839828541
  • job-839828546
  • job-839830974
  • job-839831721
  • job-839831773
  • job-839831782