We are investigating a brief outage of our Billing API on the Azure North Europe stack (https://connection.north-europe.azure.keboola.com/) between 23:03 and 23:19 UTC. Non–Pay-As-You-Go projects were not affected.
Update 23:35 UTC: The service disruption also caused job delays across all projects in this stack during the affected period. All operations have since returned to normal. We apologize for the inconvenience.
On March 12, 2025, our support button was unavailable from 07:15 UTC to 07:50 UTC.
We apologize for any inconvenience this may have caused.
The button is now working again properly.
The announced partial maintenance of connection.keboola.com and connection.eu-central-1.keboola.com will start in one hour at 08:00 UTC. We will update this post with the progress of the partial maintenance for each stack.
Update 07:00 UTC: The scheduled partial maintenance for connection.keboola.com has begun.
Update 07:21 UTC: The maintenance of connection.keboola.com has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience. Maintenance of connection.eu-central-1.keboola.com will start at 08:00 UTC.
Update 08:00 UTC: The scheduled partial maintenance for connection.eu-central-1.keboola.com has begun.
Update 08:25 UTC: The maintenance of connection.eu-central-1.keboola.com has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.
2025-03-13 23:46 CET – We are investigating a service degradation on the https://connection.europe-west3.gcp.keboola.com/ stack.
2025-03-14 00:01 CET – Services on the https://connection.europe-west3.gcp.keboola.com/ stack have resumed normal operations. Component events (level INFO) display only 50% of the logs.
2025-03-14 22:09 CET – To avoid database bottleneck during peak hours this night we have implemented events sampling. Component events (level info) will display only 50% of the logs. All storage events and component events with higher level will not be affected by this change.
2025-03-12 06:00 UTC - we are investigation possible service degradation on https://connection.europe-west3.gcp.keboola.com/ stack
2025-03-12 07:15 UTC - we are still working on the issue, stack is operational, but jobs are being delayed
2025-03-12 08:15 UTC - All services are fully operational now however you may experience job delays due to limited worker capacity. We have identified a database bottleneck and are working towards getting to full capacity.
2025-03-12 23:40 UTC - To avoid database bottleneck during peak hours this night we have implemented events sampling. Component events (level info) will display only 50% of the logs. All storage events and component events with higher level will not be affected by this change.
2025-03-13 09:15 UTC - Sampling of the component events has been removed and all events created from now on are displaying.
2025-03-11 23:15 UTC - we are investigation possible service degradation on https://connection.europe-west3.gcp.keboola.com/ stack
2025-03-12 00:15 UTC - stack is now fully operational, ale jobs are being processed.
2025-03-11 10:50 UTC - we are investigating failed jobs for component keboola.ex-telemetry-data
2025-03-11 11:30 UTC - We found invalid record in telemetry data which caused telemetry extractor to fail. Record was fixed and all telemetry extractors are working as expected.
We are sorry for caused inconvenience.
2025-03-11 10:00 UTC - we are investigation possible service degradation on https://connection.eu-central-1.keboola.com stack
2025-03-11 11:10 UTC - all services are operating normally now. Only marginal number of component job was affected with delayed processing.
2025-02-08 23:55 UTC We are experiencing issues with the Storage API on our stack. This may cause slower Queue jobs and occasional downtime for the Storage API and Connection administration. The next update will be in 30 minutes.
2025-02-09 0:35 UTC We identified an issue where some parts of our system were overloaded due to a high number of jobs running during the nightly peak after 11 PM. This caused slowdowns and occasional errors in our UI and Storage API and some Queue jobs may have been delayed or failed.
We have increased system capacity to prevent this issue in the future.The incident is now resolved.
We apologize for any inconvenience caused.
2025-03-05 16:10 UTC We are currently experiencing degraded performance with workspace creation, which may impact all jobs that rely on workspace creation as part of their lifecycle.
Our team is actively investigating the issue, and we will provide updates as we have more information.
We appreciate your patience and apologize for any inconvenience this may cause.
2025-03-05 17:15 UTC We've identified the core issue and deployed a fix. All Storage jobs associated with WorkspaceCreate should now be functioning properly.