Since 09:39 UTC we're seeing job starting with delays on https://connection.north-europe.azure.keboola.com/ We're investigating the situation. Next update in 30 minutes.

UPDATE 10:30 UTC we managed to find the root cause, new worker nodes have a problem authorization accessing the container registry, we are working on a fix. Next update in 30 minutes.

UPDATE 10:57UTC The problem with authorization to container registry is now solved. All systems are now operating normally.

Since 13:40 UTC we're seeing job starting with delays on https://connection.north-europe.azure.keboola.com/ We're investigating the situation. Next update in 30 minutes.

14:14 UTC - All systems are now operating normally.

If your project run out of credits and you have enabled automatic top-up, this would have failed between approximately 13:40 to 14:10. Restarting the job will trigger automatic top-up correctly now.

We apologize for any inconvenience caused.

2023-04-26 11:00 UTC - We have discovered a problem with orchestrations not starting on the legacy queue. We are currently investigating possible causes.

2023-04-26 11:30 UTC - The problem was caused by a release earlier today, and as a result, no orchestrations on the legacy queue were run since 08:10 UTC. We have done rollback of the release and orchestrations should be functioning properly again as of 11:30 UTC. We apologize for any inconvenience caused.

We are adding new outbound IP addresses for the connection.keboola.com and connection.eu-central-1.keboola.com stacks for Queue V2. These new addresses are available now, but are not yet being used automatically.

This update is important for Keboola Connection customers. It may affect their ability to connect to their resources, particularly if they are behind a firewall.

What are outbound IP addresses?

Outbound IP addresses are unique addresses assigned to a device for the purpose of identifying it and sending information over the Internet. When Keboola Connection customers connect to their resources (typically databases), those resources are usually behind a firewall. In order for Keboola Connection to connect to those resources, customers whitelist our outbound IP addresses.

What must I do?

1. If your resources are behind a firewall, ensure that all the new IP addresses are added to the whitelist, so as to enable connection to your system through Keboola Connection. 

2. Use the “Test with new outbound IPs” feature to check the connectivity for any or all configurations in the credentials section. This will verify that your resources are accessible from the new addresses.
   

   

3. If the connection works well, switch the project to the new IP addresses. In the event of problems, you can temporarily revert to your original IP address and contact our support team for assistance.
   
   

   

If you have multiple projects in your organization and have already tested the connection from the new IP addresses, you can ask our support team for help. They can switch all your projects at once, so you don’t have to do it individually for each one.

If you are not yet making use of Queue V2 for your projects, don’t hesitate to whitelist, as this will speed up your migration to the new queue in the future.

By when must I do it?

To ensure uninterrupted connectivity, the new IP addresses must be whitelisted by June 30, 2023. Otherwise, you run the risk of your connection not working. If the above update is not done manually by this date, Keboola Connection will perform the switch globally. To make sure of a smooth change, please add the new IP addresses to your whitelist and switch your projects as soon as you can.

Current list of outbound IP addresses

connection.keboola.com

• Queue V2
  • 52.7.83.136
  • 52.20.72.254
  • 3.222.3.15 (new)
  • 34.206.78.206 (new)
  • 3.213.250.110 (new)
  • 107.22.113.103 (new)
  • 54.144.9.113 (new)
  • 54.204.61.145 (new)
  • 34.239.7.70 (new)
  • 3.217.232.144 (new)
• Email delivery
  • 149.72.196.5
• Queue V1 - legacy syrup services
  • 34.224.0.188
  • 34.200.169.177
  • 52.206.109.126
  • 34.203.87.137

connection.eu-central-1.keboola.com

• Queue V2
  • 3.66.248.180
  • 3.64.150.30
  • 35.157.62.225 (new)
  • 3.71.156.204 (new)
  • 3.74.28.187 (new)
  • 18.158.155.128 (new)
  • 35.157.208.189 (new)
  • 3.72.243.47 (new)
  • 18.193.225.37 (new)
  • 3.127.158.56 (new)
• Email delivery
  • 149.72.196.5
• Queue V1 - legacy syrup services
  • 35.157.170.229
  • 35.157.93.175

For your convenience, you can programmatically fetch and process the list of existing IP addresses in JSON format. Read more about outbound IP addresses in documentation.

We appreciate your cooperation in making this transition as smooth as possible.

2023-04-21 14:38 UTC - We experienced another incident on connection.eu-central-1.keboola.com. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

A limited service disruption on AWS US stack will start at 09:00 a.m. UTC today, as announced earlier. Storage jobs processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 08:50 a.m. UTC: The limited service disruption has begun.

Update 09:20 a.m. UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

A limited service disruption on AWS EU stack will start at 07:00 a.m. UTC today, as announced earlier. Storage jobs processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 6:55 a.m. UTC: The limited service disruption has begun.

Update 07:45 a.m. UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

2023-04-14 02:11 UTC Due to internal incident few jobs ended with incorrect user exception `Component terminated. Possibly due to out of memory error`. 

2023-04-13 07:40 UTC We are investigating failing workspace creation on EU stack (connection.eu-central-1.keboola.com). The issue is manifesting as Loading data to workspace failed: Client error:... when you try to create a workspace from python transformation. More information within the hour.

2023-04-13 08:40 UTC We are still investigating root cause. This issue happens only when creating new workspace from python transformation with empty input mapping on connection.eu-central-1.keboola.com. More information within the hour.

2023-04-13 09:15 UTC The service disruption has been resolved and the stack is now fully operational.

We are very sorry for the inconvenience.

2023-04-12 09:50 UTC Table dependency graph shows error message on projects with queue v1.  More information within the hour.

Update 2023-04-12 10:43 UTC We have deployed the latest functional version, the problem should be solved by now. 

We apologize for the inconvenience.