We are investigating job start delays on the https://connection.us-east4.gcp.keboola.com/ stack. We will provide updates as new information becomes available.
Update 06:45 UTC: The platform has stabilized, and the job backlog has been cleared. Thank you for your patience, and we apologize for any inconvenience caused.
We are experiencing problems on our GCP EU stack (https://connection.europe-west3.gcp.keboola.com/). We are deeply sorry for the inconvenience this may cause. In the user interface, you can issue an error alert or task slowdown processing jobs. Next update in 30 minutes.
Oct 28th 21:11 UTC: We're still investigating scheduling issues within our underlying infrastructure. Next update in 30 minutes.
Oct 28th 21:18 UTC: The issue has been resolved, and job listing should now work as expected. Thank you for your patience, and sorry for any inconvenience.
The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 30 minutes, around 12:00 UTC. We will update this post with the progress of the partial maintenance.
Update 12:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.
Update 12:22 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.
2024-10-19 10:26 UTC We have noticed a slowdown in the processing of jobs on the https://connection.eu-central-1.keboola.com stack, but the jobs shouldn't end with an error.
Update 2024-10-19 10:57 UTC The problem has been identified and solved, the platform should be stable again.
We apologize for the inconvenience.
2024-10-07 18:10 UTC - We are investigating issues on north-europe.azure.keboola.com Stack
2024-10-07 18:30 UTC - Issues are caused by Azure networking problems, we are monitoring situation.
2024-09-23 13:48 UTC - We are currently investigating potential modifications to the primary key for the FTP and S3 extractors that occurred around September 13th, 2024. The issue has already been reverted, and we are conducting an analysis. We will provide more information as soon as we have further details.
UPDATE 2024-09-23 16:36 UTC - Our analysis confirms that no projects on single-tenant stacks were affected by the issue. We are continuing with the analysis of multi-tenant stack projects and will provide more information as soon as we have further details.
UPDATE 2024-09-25 9:30 UTC - Our analysis has been completed, and we now have a list of affected configurations. The issue with potential primary key modifications may have impacted not only FTP and S3 extractors but also other components using the Processor Create Manifest. We would like to highlight that not all configurations with this processor were affected.
In cases where a configuration experienced a primary key modification, the key was automatically restored after the next run. However, a small number of configurations did not revert to their original primary key due to duplicate records in the table.
These cases are limited, and the clients affected by this issue will be contacted individually by our support team today with further steps and recommendations.
If you have any questions or concerns, please reach out to our support team.
2024-09-05 20:30 UTC - We are investigating issues on connection.north-europe.azure.keboola.com
2024-09-05 20:40 UTC - Issue is now resolved, we are monitoring situation.
2024-08-27 06:25 - We are investigating an issue with restoration of Python workspaces
2024-08-27 07:50 - Issue is resolved, python workspaces are now working normally.
We are apologize for the inconvenience.
A major difference is that the new endpoint returns results from a secondary database, which is synchronized with a slight delay.
This means a new job may not be shown right away after it's created and job detail may contain slightly outdated data if it was updated recently. The delay should be couple seconds at max during normal operation.
Other difference is the new endpoint will return maximum 500 items per page.
See the Job Queue API documentation for more details.
2024-08-19 08:05 - We are investigating an issue with synchronous action on connection.north-europe.azure.keboola.com stack
2024-08-19 08:30 - We are still investigating the issue, problem is on all Azure instances.
2024-08-19 09:00 - Issue is resolved, synchronous actions are operational on all azure stacks.