We're investigating an issue with jobs not starting on Azure North Europe stack (https://connection.north-europe.azure.keboola.com/).

We are currently investigating a problems with creation of new project for Pay as you go subscription on https://connection.us-east4.gcp.keboola.com/

Update 07:30 UTC - Issue was resolved, pay as you go wizard is now operational.

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 15 minutes, around 06:00 UTC. We will update this post with the progress of the partial maintenance.

Update 06:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 06:10 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

We are currently experiencing degraded performance on our GCP stacks due to ongoing global GCP infrastructure issues. 

The following stacks are affected:

• https://connection.us-east4.gcp.keboola.com/
• https://connection.europe-west3.gcp.keboola.com/

As a result, you may observe random job failures or delays. Jobs may fail with intermittent errors unrelated to the actual job configuration or data.

Our team is actively monitoring the situation and working to restore full functionality. We appreciate your patience and will provide updates as more information becomes available.

[Update – 2025-06-12 19:37 UTC] GCP is in the process of restoring its services. We are observing that our GCP stacks are beginning to operate as expected. API latency has returned to normal levels, and the job queue is starting to clear.

We will continue to monitor the situation closely.

[Update – 2025-06-12 20:25 UTC] The incident has been resolved. All GCP stacks are stable and the platform is fully operational. Thank you for your patience.

We are currently investigating a delay in the job listing updates on https://connection.keboola.com/. New jobs are not appearing in the job overview, and the statuses of existing jobs are not being refreshed.

This issue does not affect job processing or scheduling. Jobs are still running as expected.

We will provide the next update as soon as new information becomes available.

Update 13:15 UTC: We have identified the root cause and resolved the issue. All jobs are now listed correctly, and their statuses are updating immediately.

We are currently experiencing an outage of the Python Workspace on the stack https://connection.eu-central-1.keboola.com. Our team is actively working on resolving the issue. Please monitor this status page for further updates.

Update 2025-06-10 14:45 UTC: Python Workspaces are fully operational after a brief overload-related outage. Apologies for the inconvenience.

Update 2025-06-10 14:55 UTC: Due to the earlier Python Workspace outage, job processing was delayed by approximately 15 minutes. The issue has been stabilized.

Update 2025-06-10 14:59 UTC: The root cause of the incident was a failover on the MySQL RDS database. We apologize for the disruption and appreciate your patience.

We are currently experiencing an issue across all stacks where login to Keboola Connection is not possible. Attempting to access the application results in a 404 Not Found error.

Workaround: After login, if redirected to a 404 URL ending with `/admin`, manually remove the `/admin` part and reload the page.

Our engineering team is actively investigating the root cause. We will share an update as soon as we know more or within the next 30 minutes.

Update 2025-06-10 07:17 UTC: The issue affects only login via Google. Other login methods remain functional. We are continuing to investigate the root cause. Next update will follow within 30 minutes.

Update 2025-06-10 07:42 UTC: The issue has been resolved. Login via Google is now working as expected.

Thank you for your patience.

We would like to inform you about the planned maintenance of all Keboola stacks hosted on Azure.

During the database upgrades there will be a short service outage on all Azure stacks, including all single-tenant stacks and Azure North Europe multi-tenant stack (connection.north-europe.azure.keboola.com). This will take place on Saturday, June 21, 2025 between 05:30 and 06:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 06:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

• 05:30–06:00 UTC: processing of new jobs stops.
• 06:00–06:15 UTC: service enhancement period.
• 06:15 UTC: processing of jobs resumes.

We are investigating issue that flows generate new IDs which are not possible to set trigger with. It's currently affecting all stacks.

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 15 minutes, around 06:00 UTC. We will update this post with the progress of the partial maintenance.

Update 06:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 06:11 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

14:00 UTC: All Azure stacks are currently experiencing issues starting Python Workspaces, Data Apps and Jobs. The root cause appears to be a failure to pull the required Docker images.

On startup you may see a long start which ends with an internal error. We apologize for the situation.

UPDATE 14:52 UTC: We’ve applied a fix and all systems are now operating normally. We apologize for the disruption.

Since May 13, 20:00 UTC, we have been seeing intermittent delays when performing service management operations on Azure Storage accounts hosted in the West Europe region. Storage availability and data-processing workflows remain fully operational; however, you may notice jobs delays.

Azure’s latest update (not publicly available):

Current Status: Our monitoring shows that our mitigation strategy has worked and less than 5 % of the traffic is impacted at this stage; most customer impact should be mitigated at this stage. We continue to monitor our infrastructure and expect to see the delays decrease over the next few hours.

We will update this post as new information becomes available. If you have any questions or concerns, please reach out to our support team.

Update May 16, 08:00 UTC: Azure's latest update (not publicly available):

Service restored, and customer impact mitigated.

We can confirm the issue is resolved with our findings.

We are sorry for this inconvenience.

We are currently experiencing job failures related to the keboola.legacy-transformation component.

Our team is actively working on reverting to the previous stable release to resolve the issue.

We sincerely apologize for any inconvenience this may have caused and appreciate your patience.

Update 10:56 UTC: The issue has been fixed by reverting to previous release.

We’ve identified an issue in the Keboola UI where the list of tables in the transformation may render incorrectly after a table has been deleted from the list. This can result in visually corrupted table rows (e.g., mixed or misaligned entries).

This is a UI-only issue — data processing is not affected. Reloading the page restores the table view correctly.

We’ve successfully reproduced the problem and are working on a fix.

Update 13:50 UTC: the issue has been fixed. 

We would like to inform you about the planned maintenance of all Keboola stacks hosted on GCP.

During the database upgrades there will be a short service outage on all GCP stacks, including all single-tenant stacks and GCP US and EU multi-tenant stacks (connection.us-east4.gcp.keboola.com, connection.europe-west3.gcp.keboola.com). This will take place on Saturday, May 24, 2025 between 05:30 and 06:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 06:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

• 05:30–06:00 UTC: processing of new jobs stops.
• 06:00–06:15 UTC: service enhancement period.
• 06:15 UTC: processing of jobs resumes.

We are currently investigating an issue involving missing telemetry data that appears to affect all Azure stacks. The issue began on April 25, 2025, at approximately 08:30 UTC.

We will continue to update this article with additional information as our investigation progresses.

UPDATE 13:00 UTC

Additionally, since April 23, 2025, at approximately 08:00 UTC, we have identified missing billing data for Workspaces, DWH Direct Query, Data Streams, and Data Apps across all stacks (not limited to Azure).

All missing telemetry and billing data will be backfilled within the next few hours.

UPDATE [April 29, 2025, 05:38 UTC]

We apologize for the inconvenience during the incident.

We have identified an issue where configuring schedules via the UI orchestrator/flow leads to an off-by-one-day error. For example setting an execution for Sunday would incorrectly represent it as Saturday. See included media how to check for the inconsistency of a date.

Incident Timeline:

• Start: April 25, 2025 11:42 AM UTC

• End: April 26, 2025 10:10 AM UTC

Impact:
Schedules created through the UI during this timeframe have been translated into wrong crontab expressions, causing misaligned execution days.

Action Required:
If you configured any schedules via the UI, please review and correct them manually to ensure they are aligned with the intended execution days.

We apologize for the inconvenience

We are currently experiencing very rare encryption errors across all Azure stacks, resulting in immediate job failures. These errors occur very infrequently, approximately at a rate of 1 in 50,000 jobs.

Affected jobs have either a very short runtime (approximately 1 second) or no recorded runtime. These jobs typically lack events other than the error message itself. The specific error messages observed include:

• Internal error
• Decryption failed: Deciphering failed.
• Value "***" is not an encrypted value. 

These errors manifest in two distinct ways: transient and permanent.

Transient errors occur during the scheduling or initial starting phase of a job. Restarting the affected job resolves these transient issues.

Permanent errors occur during the saving of configurations or state, notably involving OAuth configurations when storing new refresh tokens after a job successfully completes. Subsequent job runs retrieve this improperly encrypted value, causing the job to fail with an application error. Unlike transient errors, restarting does not resolve permanent errors. To correct permanent errors, the configuration itself must be updated to remove or correct the incorrectly encrypted value.

We are proactively monitoring all occurrences of these errors. In the case of permanent failures, we are directly contacting affected customers to resolve the issue promptly.

We apologize for any inconvenience this may cause. Please do not hesitate to reach out to our support team if you have further questions or require assistance.

UPDATE 2025-04-23 

We have identified the root cause of these issues and will be carefully deploying a fix in the coming days. We continue to proactively monitor all occurrences of these errors. In the case of permanent failures, we are directly contacting affected customers to resolve the issue promptly.

Due to a bug introduced in the latest deployment, all Job Queue jobs started between 2025-04-15 10:30 UTC and 12:00 UTC are missing approximately 1/3 of their log events. The issue has been identified and resolved.

Only AWS and Azure stacks were affected — GCP was not impacted.

Audit logs were not affected and remain fully intact.

We sincerely apologize for the disruption this has caused.

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 30 minutes, around 12:00 UTC. We will update this post with the progress of the partial maintenance.

Update 12:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 12:10 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

2025-04-06 07:21 UTC We are facing a problem with the authorization of the `kds-team.ex-onedrive` component. 

The problem manifests itself in not being able to authorize a new component or reauthorize an old one. The action ends with an authorization error message. The problem appears on our AWS and Azure stacks. GCP stacks seem to be without problem.

Update 2025-04-06 09:00 UTC We have determined the cause of the problem and are working on a fix. 

Update 2025-04-06 09:30 UTC - We have solved the problem and the component authorization should be functional again on all stacks. 

We apologize for the inconvenience.

We would like to inform you about the planned maintenance of all Keboola stacks hosted on Azure.

During the database upgrades there will be a short service outage on all Azure stacks, including all single-tenant stacks and Azure North Europe multi-tenant stack (connection.north-europe.azure.keboola.com). This will take place on Saturday, April 12, 2025 between 11:30 and 12:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 12:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

• 11:30–12:00 UTC: processing of new jobs stops.
• 12:00–12:15 UTC: service enhancement period.
• 12:15 UTC: processing of jobs resumes.

We are investigating a brief outage of our Billing API on the Azure North Europe stack (https://connection.north-europe.azure.keboola.com/) between 23:03 and 23:19 UTC. Non–Pay-As-You-Go projects were not affected.

Update 23:35 UTC: The service disruption also caused job delays across all projects in this stack during the affected period. All operations have since returned to normal. We apologize for the inconvenience.

On March 12, 2025, our support button was unavailable from 07:15 UTC to 07:50 UTC.
We apologize for any inconvenience this may have caused. 
The button is now working again properly.

The announced partial maintenance of connection.keboola.com and connection.eu-central-1.keboola.com will start in one hour at 08:00 UTC. We will update this post with the progress of the partial maintenance for each stack.

Update 07:00 UTC: The scheduled partial maintenance for connection.keboola.com  has begun.

Update 07:21 UTC: The maintenance of connection.keboola.com  has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience. Maintenance of connection.eu-central-1.keboola.com will start at 08:00 UTC.

Update 08:00 UTC: The scheduled partial maintenance for connection.eu-central-1.keboola.com  has begun.

Update 08:25 UTC: The maintenance of  connection.eu-central-1.keboola.com has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience. 

2025-03-13 23:46 CET – We are investigating a service degradation on the https://connection.europe-west3.gcp.keboola.com/ stack.

2025-03-14 00:01 CET – Services on the https://connection.europe-west3.gcp.keboola.com/ stack have resumed normal operations. Component events (level INFO) display only 50% of the logs.

2025-03-14 22:09 CET – To avoid database bottleneck during peak hours this night we have implemented events sampling. Component events (level info) will display only 50% of the logs. All storage events and component events with higher level will not be affected by this change. 

2025-03-12 06:00 UTC - we are investigation possible service degradation on https://connection.europe-west3.gcp.keboola.com/ stack

2025-03-12 07:15 UTC - we are still working on the issue, stack is operational, but jobs are being delayed

2025-03-12 08:15 UTC - All services are fully operational now however you may experience job delays due to limited worker capacity. We have identified a database bottleneck and are working towards getting to full capacity.

2025-03-12 23:40 UTC - To avoid database bottleneck during peak hours this night we have implemented events sampling. Component events (level info) will display only 50% of the logs. All storage events and component events with higher level will not be affected by this change. 

2025-03-13 09:15 UTC - Sampling of the component events has been removed and all events created from now on are displaying.

2025-03-11 23:15 UTC - we are investigation possible service degradation on https://connection.europe-west3.gcp.keboola.com/ stack

2025-03-12 00:15 UTC - stack is now fully operational, ale jobs are being processed.

2025-03-11 10:50 UTC - we are investigating failed jobs for component keboola.ex-telemetry-data

2025-03-11 11:30 UTC - We found invalid record in telemetry data which caused telemetry extractor to fail. Record was fixed and all telemetry extractors are working as expected. 

We are sorry for caused inconvenience.

2025-03-11 10:00 UTC - we are investigation possible service degradation on https://connection.eu-central-1.keboola.com stack

2025-03-11 11:10 UTC - all services are operating normally now. Only marginal number of component job was affected with delayed processing.