Since 09:39 UTC we're seeing job starting with delays on https://connection.north-europe.azure.keboola.com/ We're investigating the situation. Next update in 30 minutes.

UPDATE 10:30 UTC we managed to find the root cause, new worker nodes have a problem authorization accessing the container registry, we are working on a fix. Next update in 30 minutes.

UPDATE 10:57UTC The problem with authorization to container registry is now solved. All systems are now operating normally.

We are adding new outbound IP addresses for the connection.keboola.com and connection.eu-central-1.keboola.com stacks for Queue V2. These new addresses are available now, but are not yet being used automatically.

This update is important for Keboola Connection customers. It may affect their ability to connect to their resources, particularly if they are behind a firewall.

What are outbound IP addresses?

Outbound IP addresses are unique addresses assigned to a device for the purpose of identifying it and sending information over the Internet. When Keboola Connection customers connect to their resources (typically databases), those resources are usually behind a firewall. In order for Keboola Connection to connect to those resources, customers whitelist our outbound IP addresses.

What must I do?

1. If your resources are behind a firewall, ensure that all the new IP addresses are added to the whitelist, so as to enable connection to your system through Keboola Connection. 

2. Use the “Test with new outbound IPs” feature to check the connectivity for any or all configurations in the credentials section. This will verify that your resources are accessible from the new addresses.
   

   

3. If the connection works well, switch the project to the new IP addresses. In the event of problems, you can temporarily revert to your original IP address and contact our support team for assistance.
   
   

   

If you have multiple projects in your organization and have already tested the connection from the new IP addresses, you can ask our support team for help. They can switch all your projects at once, so you don’t have to do it individually for each one.

If you are not yet making use of Queue V2 for your projects, don’t hesitate to whitelist, as this will speed up your migration to the new queue in the future.

By when must I do it?

To ensure uninterrupted connectivity, the new IP addresses must be whitelisted by June 30, 2023. Otherwise, you run the risk of your connection not working. If the above update is not done manually by this date, Keboola Connection will perform the switch globally. To make sure of a smooth change, please add the new IP addresses to your whitelist and switch your projects as soon as you can.

Current list of outbound IP addresses

connection.keboola.com

• Queue V2
  • 52.7.83.136
  • 52.20.72.254
  • 3.222.3.15 (new)
  • 34.206.78.206 (new)
  • 3.213.250.110 (new)
  • 107.22.113.103 (new)
  • 54.144.9.113 (new)
  • 54.204.61.145 (new)
  • 34.239.7.70 (new)
  • 3.217.232.144 (new)
• Email delivery
  • 149.72.196.5
• Queue V1 - legacy syrup services
  • 34.224.0.188
  • 34.200.169.177
  • 52.206.109.126
  • 34.203.87.137

connection.eu-central-1.keboola.com

• Queue V2
  • 3.66.248.180
  • 3.64.150.30
  • 35.157.62.225 (new)
  • 3.71.156.204 (new)
  • 3.74.28.187 (new)
  • 18.158.155.128 (new)
  • 35.157.208.189 (new)
  • 3.72.243.47 (new)
  • 18.193.225.37 (new)
  • 3.127.158.56 (new)
• Email delivery
  • 149.72.196.5
• Queue V1 - legacy syrup services
  • 35.157.170.229
  • 35.157.93.175

For your convenience, you can programmatically fetch and process the list of existing IP addresses in JSON format. Read more about outbound IP addresses in documentation.

We appreciate your cooperation in making this transition as smooth as possible.

2023-04-21 14:38 UTC - We experienced another incident on connection.eu-central-1.keboola.com. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

2023-04-03 10:15 UTC - We are investigating delayed telemetry data. More information within the hour.

2023-04-03 11:30 UTC - Delayed telemetry data on all Keboola Connection stacks have been recorded since approximately 20:00 UTC on March 29. We were able to determine the root cause and perform a backfill. Now all telemetry data tables are up-to-date.

We are very sorry for the inconvenience. If you encounter any discrepancies, please contact us immediately.

We are aware of a bug in the user interface that makes it impossible to undo the action after clicking collapse tables in IM.

We are sorry for complications. A bug fix will be deployed within minutes.

Update 14:05 UTC The fix has been released to all stacks.

2022-08-04 12:05 UTC We are investigating internal job errors. Next update in 30 minutes.

2022-08-04 12:36 UTC We experienced two waves of internal errors, between 11:50-11:57 and 12:00-12:07 UTC. The root cause has been removed and I don't see any other errors in the reporting. We continue to monitoring the situation. Next update in 30 minutes.

2022-08-04 13:06 UTC All Keboola Connection services are running normally. The incident is resolved. 

We apologize for any inconvenience.

2022-05-17 09:54 CET Due to the RDS upgrade, there was a small outage of the API queue and you may have observed a small number of errors in the connection.keboola.com UI.

The situation is now under control and our services are running normally.

2022-04-15 10:30 UTC - We are investigation stuck jobs on a new queue. Some jobs will probably end up with an internal error. Next update in 15 min.

2022-04-15 10:50 UTC - The situation was resolved, however 6 jobs were terminated by an internal error with no automatic restart. We apologize for the inconvenience and please feel free to restart your jobs manually.

Affected jobs:

• job-839828541
  
• job-839828546
  
• job-839830974
  
• job-839831721
  
• job-839831773
  
• job-839831782
  

A maintenance of AWS stacks Keboola Connection will take place on Saturday, Jan 8th, 2022 and should take less than three hours.

• connection.keboola.com - 09:00 CET

• connection.eu-central-1.keboola.com - 12:00 CET

During the maintenance, you can't access your data and projects. All network connections will be terminated by "HTTP 503 - down for maintenance" status message.

All running tasks will be monitored by us and restarted in case of any interruption. Orchestrations and running transformations will be generally delayed, but not interrupted. However, feel free to re-schedule your saturday's orchestrations to avoid this maintenance window.

Service disruption in AWS US are also causing problems in Devloper Porta (apps-api.keboola.com & components.keboola.com). You may see intermittent errors 5XX, refreshing the page can help. 

AWS acknowledge the service disruption and are active working towards recover. See https://status.aws.amazon.com/ for more details. Once the AWS service disruption is over our services should start running smoothly again. Next update in 60 minutes or when new information is available.

UPDATE 19:10 UTC Service disruption on the AWS US region persists. We continue to monitor the situation. Next update in 2 hours.

UPDATE 21:40 UTC Service disruption on the AWS US region are reducing. Our affected services are showing significant improvement. Next update in 12 hours or as new information is available.

UPDATE Dec 8th, 07:12 UTC Most services in the affected AWS region have already been repaired. Our services are operating normally. Next update in 4 hours or as new information is available.

UPDATE Dec 8th, 15:130 UTC We're sorry for the late update. AWS services have already recovered. Everything should be running without any issues now.