tag:status.keboola.com,2013:/posts Keboola Status 2025-06-21T06:10:08Z Keboola Connection - Platform Status tag:status.keboola.com,2013:Post/2204272 2025-06-21T05:51:29Z 2025-06-21T06:10:08Z Planned partial maintenance on Saturday, June 21, 2025 for all Azure stacks

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 15 minutes, around 06:00 UTC. We will update this post with the progress of the partial maintenance.

Update 06:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 06:10 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2203299 2025-06-12T18:41:18Z 2025-06-12T20:56:54Z Degraded Performance on GCP Stacks

We are currently experiencing degraded performance on our GCP stacks due to ongoing global GCP infrastructure issues

The following stacks are affected:

As a result, you may observe random job failures or delays. Jobs may fail with intermittent errors unrelated to the actual job configuration or data.

Our team is actively monitoring the situation and working to restore full functionality. We appreciate your patience and will provide updates as more information becomes available.

[Update – 2025-06-12 19:37 UTC] GCP is in the process of restoring its services. We are observing that our GCP stacks are beginning to operate as expected. API latency has returned to normal levels, and the job queue is starting to clear.

We will continue to monitor the situation closely.

[Update – 2025-06-12 20:25 UTC] The incident has been resolved. All GCP stacks are stable and the platform is fully operational. Thank you for your patience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2203246 2025-06-12T12:03:04Z 2025-06-12T13:18:51Z Job Listing Not Updating on connection.keboola.com

We are currently investigating a delay in the job listing updates on https://connection.keboola.com/. New jobs are not appearing in the job overview, and the statuses of existing jobs are not being refreshed.

This issue does not affect job processing or scheduling. Jobs are still running as expected.

We will provide the next update as soon as new information becomes available.

Update 13:15 UTC:
We have identified the root cause and resolved the issue. All jobs are now listed correctly, and their statuses are updating immediately.


]]>
tag:status.keboola.com,2013:Post/2202807 2025-06-10T14:42:10Z 2025-06-10T15:01:12Z Python Workspace Outage on AWS EU

We are currently experiencing an outage of the Python Workspace on the stack https://connection.eu-central-1.keboola.com. Our team is actively working on resolving the issue. Please monitor this status page for further updates.

Update 2025-06-10 14:45 UTC: Python Workspaces are fully operational after a brief overload-related outage. Apologies for the inconvenience.

Update 2025-06-10 14:55 UTC: Due to the earlier Python Workspace outage, job processing was delayed by approximately 15 minutes. The issue has been stabilized.

Update 2025-06-10 14:59 UTC: The root cause of the incident was a failover on the MySQL RDS database. We apologize for the disruption and appreciate your patience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2202759 2025-06-10T07:11:51Z 2025-06-10T07:43:33Z Google Login to Keboola Not Working

We are currently experiencing an issue across all stacks where login to Keboola Connection is not possible. Attempting to access the application results in a 404 Not Found error.

Workaround: After login, if redirected to a 404 URL ending with `/admin`, manually remove the `/admin` part and reload the page.

Our engineering team is actively investigating the root cause. We will share an update as soon as we know more or within the next 30 minutes.

Update 2025-06-10 07:17 UTC: The issue affects only login via Google. Other login methods remain functional. We are continuing to investigate the root cause. Next update will follow within 30 minutes.

Update 2025-06-10 07:42 UTC: The issue has been resolved. Login via Google is now working as expected.

Thank you for your patience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2202145 2025-06-06T12:39:59Z 2025-06-06T12:39:59Z Scheduled Partial Maintenance of all Azure stacks – June 21, 2025

We would like to inform you about the planned maintenance of all Keboola stacks hosted on Azure.

During the database upgrades there will be a short service outage on all Azure stacks, including all single-tenant stacks and Azure North Europe multi-tenant stack (connection.north-europe.azure.keboola.com). This will take place on Saturday, June 21, 2025 between 05:30 and 06:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 06:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

  • 05:30–06:00 UTC: processing of new jobs stops.
  • 06:00–06:15 UTC: service enhancement period.
  • 06:15 UTC: processing of jobs resumes.


]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2201998 2025-06-05T16:58:16Z 2025-06-05T20:06:30Z Flows triggers are not possible to set trigger

We are investigating issue that flows generate new IDs which are not possible to set trigger with. It's currently affecting all stacks.

We will update this post as new information becomes available. If you have any questions or concerns, please reach out to our support team.

Update June 05 17:50 UTC We released fix which should solved the issue visually with 201 status response but still the schedule is not available in UI. We will investigate further where is the problem.

Update June 05 18:15 UTC We see things operating as usually. All flows configurations between 17:50 UTC and 18:15 UTC won't be properlly working with triggers. 

Also all flows created from ~June 03 15:00 UTC till June 05 17:50 UTC which were created with ULID identifiers remain unusable using table triggers. We will make sure that these will be able to be scheduled with tiggers subsequently. Current approach is to delete your flows with ULID identifiers and create new ones with integer identifiers.

We are sorry for this inconvenience.

]]>
tag:status.keboola.com,2013:Post/2199777 2025-05-24T05:45:37Z 2025-05-24T06:11:54Z Planned partial maintenance on Saturday, May 24, 2025 for all GCP stacks

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 15 minutes, around 06:00 UTC. We will update this post with the progress of the partial maintenance.

Update 06:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 06:11 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

]]>
tag:status.keboola.com,2013:Post/2199608 2025-05-23T14:08:43Z 2025-05-23T15:10:04Z Azure Stacks: Workspace, Data Apps and Jobs Startup Failures

14:00 UTC: All Azure stacks are currently experiencing issues starting Python Workspaces, Data Apps and Jobs. The root cause appears to be a failure to pull the required Docker images.

On startup you may see a long start which ends with an internal error. We apologize for the situation.

UPDATE 14:52 UTCWe’ve applied a fix and all systems are now operating normally. We apologize for the disruption.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2197572 2025-05-15T09:01:36Z 2025-05-16T08:13:05Z Degraded performance of Azure Storage accounts in all Azure West Europe stacks

Since May 13, 20:00 UTC, we have been seeing intermittent delays when performing service management operations on Azure Storage accounts hosted in the West Europe region. Storage availability and data-processing workflows remain fully operational; however, you may notice jobs delays.

Azure’s latest update (not publicly available):

Current Status: Our monitoring shows that our mitigation strategy has worked and less than 5 % of the traffic is impacted at this stage; most customer impact should be mitigated at this stage. We continue to monitor our infrastructure and expect to see the delays decrease over the next few hours.

We will update this post as new information becomes available. If you have any questions or concerns, please reach out to our support team.

Update May 16, 08:00 UTC: Azure's latest update (not publicly available):

Service restored, and customer impact mitigated.

We can confirm the issue is resolved with our findings.

We are sorry for this inconvenience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2195158 2025-05-05T10:36:29Z 2025-05-05T10:57:12Z Failing jobs of Legacy transformations

We are currently experiencing job failures related to the keboola.legacy-transformation component.

Our team is actively working on reverting to the previous stable release to resolve the issue.

We sincerely apologize for any inconvenience this may have caused and appreciate your patience.

Update 10:56 UTC: The issue has been fixed by reverting to previous release.


]]>
tag:status.keboola.com,2013:Post/2194056 2025-04-30T12:44:36Z 2025-04-30T13:51:15Z UI Rendering Issue in Transformations (fixed)

We’ve identified an issue in the Keboola UI where the list of tables in the transformation may render incorrectly after a table has been deleted from the list. This can result in visually corrupted table rows (e.g., mixed or misaligned entries).

This is a UI-only issue — data processing is not affected. Reloading the page restores the table view correctly.

We’ve successfully reproduced the problem and are working on a fix.

Update 13:50 UTC: the issue has been fixed. 



]]>
tag:status.keboola.com,2013:Post/2193791 2025-04-29T12:08:30Z 2025-04-29T12:08:30Z Scheduled Partial Maintenance of all GCP stacks – May 24, 2025

We would like to inform you about the planned maintenance of all Keboola stacks hosted on GCP.

During the database upgrades there will be a short service outage on all GCP stacks, including all single-tenant stacks and GCP US and EU multi-tenant stacks (connection.us-east4.gcp.keboola.com, connection.europe-west3.gcp.keboola.com). This will take place on Saturday, May 24, 2025 between 05:30 and 06:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 06:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

  • 05:30–06:00 UTC: processing of new jobs stops.
  • 06:00–06:15 UTC: service enhancement period.
  • 06:15 UTC: processing of jobs resumes.
]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2193506 2025-04-28T10:51:15Z 2025-04-29T05:41:34Z Missing telemetry data and billing data across all stacks [resolved]

We are currently investigating an issue involving missing telemetry data that appears to affect all Azure stacks. The issue began on April 25, 2025, at approximately 08:30 UTC.

We will continue to update this article with additional information as our investigation progresses.

UPDATE 13:00 UTC

Additionally, since April 23, 2025, at approximately 08:00 UTC, we have identified missing billing data for Workspaces, DWH Direct Query, Data Streams, and Data Apps across all stacks (not limited to Azure).

All missing telemetry and billing data will be backfilled within the next few hours.

UPDATE [April 29, 2025, 05:38 UTC]

The incident has been resolved. All missing telemetry and billing data across all stacks have now been successfully backfilled. No further impact is expected, and all systems are operating normally.  

We apologize for the inconvenience during the incident.


]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2193122 2025-04-26T10:30:22Z 2025-04-26T10:30:22Z Schedule set using UI orchestrator/flow configuration results into wrong crontab expression

We have identified an issue where configuring schedules via the UI orchestrator/flow leads to an off-by-one-day error. For example setting an execution for Sunday would incorrectly represent it as Saturday. See included media how to check for the inconsistency of a date.

Incident Timeline:

  • Start: April 25, 2025 11:42 AM UTC

  • End: April 26, 2025 10:10 AM UTC

Impact:
Schedules created through the UI during this timeframe have been translated into wrong crontab expressions, causing misaligned execution days.

Action Required:
If you configured any schedules via the UI, please review and correct them manually to ensure they are aligned with the intended execution days.

We apologize for the inconvenience

]]>
tag:status.keboola.com,2013:Post/2190653 2025-04-15T18:19:19Z 2025-04-23T08:25:33Z Rare encryption errors affecting job execution across all Azure Stacks

We are currently experiencing very rare encryption errors across all Azure stacks, resulting in immediate job failures. These errors occur very infrequently, approximately at a rate of 1 in 50,000 jobs.

Affected jobs have either a very short runtime (approximately 1 second) or no recorded runtime. These jobs typically lack events other than the error message itself. The specific error messages observed include:

  • Internal error
  • Decryption failed: Deciphering failed.
  • Value "***" is not an encrypted value. 

These errors manifest in two distinct ways: transient and permanent.

Transient errors occur during the scheduling or initial starting phase of a job. Restarting the affected job resolves these transient issues.

Permanent errors occur during the saving of configurations or state, notably involving OAuth configurations when storing new refresh tokens after a job successfully completes. Subsequent job runs retrieve this improperly encrypted value, causing the job to fail with an application error. Unlike transient errors, restarting does not resolve permanent errors. To correct permanent errors, the configuration itself must be updated to remove or correct the incorrectly encrypted value.

We are proactively monitoring all occurrences of these errors. In the case of permanent failures, we are directly contacting affected customers to resolve the issue promptly.

We apologize for any inconvenience this may cause. Please do not hesitate to reach out to our support team if you have further questions or require assistance.

UPDATE 2025-04-23 

We have identified the root cause of these issues and will be carefully deploying a fix in the coming days. We continue to proactively monitor all occurrences of these errors. In the case of permanent failures, we are directly contacting affected customers to resolve the issue promptly.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2190636 2025-04-15T11:53:45Z 2025-04-15T12:08:02Z Missing Log Events

Due to a bug introduced in the latest deployment, all Job Queue jobs started between 2025-04-15 10:30 UTC and 12:00 UTC are missing approximately 1/3 of their log events. The issue has been identified and resolved.

Only AWS and Azure stacks were affected — GCP was not impacted.

Audit logs were not affected and remain fully intact.

We sincerely apologize for the disruption this has caused.

]]>
tag:status.keboola.com,2013:Post/2189234 2025-04-12T11:32:09Z 2025-04-12T12:11:14Z Planned partial maintenance on Saturday, April 12, 2025 for all Azure stacks

The announced partial maintenance has just started. The platform has been scaled down and is no longer accepting new jobs. We expect a brief downtime in 30 minutes, around 12:00 UTC. We will update this post with the progress of the partial maintenance.

Update 12:00 UTC: The announced partial maintenance is ongoing, and we are expecting downtime to begin any minute now. We will continue to update this post as the maintenance progresses.

Update 12:10 UTC: The maintenance has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2188447 2025-04-06T08:33:29Z 2025-04-06T09:34:02Z Authorization issue with kds-team.ex-onedrive component on AWS and Azure Stacks

2025-04-06 07:21 UTC We are facing a problem with the authorization of the `kds-team.ex-onedrive` component. 

The problem manifests itself in not being able to authorize a new component or reauthorize an old one. The action ends with an authorization error message. The problem appears on our AWS and Azure stacks. GCP stacks seem to be without problem.

Update 2025-04-06 09:00 UTC We have determined the cause of the problem and are working on a fix. 

Update 2025-04-06 09:30 UTC - We have solved the problem and the component authorization should be functional again on all stacks. 

We apologize for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/2187114 2025-04-01T08:15:35Z 2025-04-01T08:15:35Z Scheduled Partial Maintenance of all Azure stacks – April 12, 2025

We would like to inform you about the planned maintenance of all Keboola stacks hosted on Azure.

During the database upgrades there will be a short service outage on all Azure stacks, including all single-tenant stacks and Azure North Europe multi-tenant stack (connection.north-europe.azure.keboola.com). This will take place on Saturday, April 12, 2025 between 11:30 and 12:30 UTC.

Effects of the Maintenance

During the above period, services will be scaled down and the processing of jobs may be delayed. For a very brief period (at around 12:00 UTC) the service will be unavailable for up to 10 minutes and APIs may respond with a 500 error code. After that, all services will scale up and start processing all jobs. No running jobs, data apps, or workspaces will be affected. Delayed scheduled flows and queued jobs will resume after the maintenance is completed.

Detailed Schedule

  • 11:30–12:00 UTC: processing of new jobs stops.
  • 12:00–12:15 UTC: service enhancement period.
  • 12:15 UTC: processing of jobs resumes.
]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2186260 2025-03-27T23:29:55Z 2025-03-27T23:43:22Z Azure North Europe stack - Billing API problems and job delays

We are investigating a brief outage of our Billing API on the Azure North Europe stack (https://connection.north-europe.azure.keboola.com/) between 23:03 and 23:19 UTC. Non–Pay-As-You-Go projects were not affected.

Update 23:35 UTC: The service disruption also caused job delays across all projects in this stack during the affected period. All operations have since returned to normal. We apologize for the inconvenience.


]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2183282 2025-03-17T08:00:57Z 2025-03-17T08:00:57Z Brief outage of support button

On March 12, 2025, our support button was unavailable from 07:15 UTC to 07:50 UTC.
We apologize for any inconvenience this may have caused. 
The button is now working again properly.

]]>
tag:status.keboola.com,2013:Post/2182748 2025-03-15T05:52:11Z 2025-03-15T08:28:17Z Planned partial maintenance on Saturday, March 15, 2025 for AWS multi-tenant stacks

The announced partial maintenance of connection.keboola.com and connection.eu-central-1.keboola.com will start in one hour at 08:00 UTC. We will update this post with the progress of the partial maintenance for each stack.

Update 07:00 UTC: The scheduled partial maintenance for connection.keboola.com  has begun.

Update 07:21 UTC: The maintenance of connection.keboola.com  has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience. Maintenance of connection.eu-central-1.keboola.com will start at 08:00 UTC.

Update 08:00 UTC: The scheduled partial maintenance for connection.eu-central-1.keboola.com  has begun.

Update 08:25 UTC: The maintenance of  connection.eu-central-1.keboola.com has been completed, and all services have been scaled back up. The platform is fully operational, and jobs are now being processed as usual. All delayed jobs will be processed shortly. Thank you for your patience. 

]]>
tag:status.keboola.com,2013:Post/2182455 2025-03-13T22:57:29Z 2025-03-14T22:09:20Z Degraded services GCP EU west3 stack

2025-03-13 23:46 CET – We are investigating a service degradation on the https://connection.europe-west3.gcp.keboola.com/ stack.

2025-03-14 00:01 CET – Services on the https://connection.europe-west3.gcp.keboola.com/ stack have resumed normal operations. Component events (level INFO) display only 50% of the logs.

2025-03-14 22:09 CET – To avoid database bottleneck during peak hours this night we have implemented events sampling. Component events (level info) will display only 50% of the logs. All storage events and component events with higher level will not be affected by this change. 

]]>
tag:status.keboola.com,2013:Post/2182059 2025-03-12T06:15:47Z 2025-03-13T23:26:23Z Degraded services GCP EU west3 stack

2025-03-12 06:00 UTC - we are investigation possible service degradation on https://connection.europe-west3.gcp.keboola.com/ stack

2025-03-12 07:15 UTC - we are still working on the issue, stack is operational, but jobs are being delayed

2025-03-12 08:15 UTC - All services are fully operational now however you may experience job delays due to limited worker capacity. We have identified a database bottleneck and are working towards getting to full capacity.

2025-03-12 23:40 UTC - To avoid database bottleneck during peak hours this night we have implemented events sampling. Component events (level info) will display only 50% of the logs. All storage events and component events with higher level will not be affected by this change. 

2025-03-13 09:15 UTC - Sampling of the component events has been removed and all events created from now on are displaying.

]]>
tag:status.keboola.com,2013:Post/2182005 2025-03-11T23:32:12Z 2025-03-12T00:14:54Z Degradated services GCP EU west3 stack

2025-03-11 23:15 UTC - we are investigation possible service degradation on https://connection.europe-west3.gcp.keboola.com/ stack

2025-03-12 00:15 UTC - stack is now fully operational, ale jobs are being processed.

]]>
tag:status.keboola.com,2013:Post/2181827 2025-03-11T10:50:02Z 2025-03-11T11:32:46Z Component keboola.ex-telemetry-data failures

2025-03-11 10:50 UTC - we are investigating failed jobs for component keboola.ex-telemetry-data


2025-03-11 11:30 UTC - We found invalid record in telemetry data which caused telemetry extractor to fail. Record was fixed and all telemetry extractors are working as expected. 

We are sorry for caused inconvenience.

]]>
tag:status.keboola.com,2013:Post/2181825 2025-03-11T10:19:44Z 2025-03-11T11:30:41Z Possible degradation of services on AWS EU Central 1 stack

2025-03-11 10:00 UTC - we are investigation possible service degradation on https://connection.eu-central-1.keboola.com stack

2025-03-11 11:10 UTC - all services are operating normally now. Only marginal number of component job was affected with delayed processing.

]]>
tag:status.keboola.com,2013:Post/2181217 2025-03-08T23:55:15Z 2025-03-09T00:34:07Z Storage API outage at GCP EU west3 stack

2025-02-08 23:55 UTC We are experiencing issues with the Storage API on our stack. This may cause slower Queue jobs and occasional downtime for the Storage API and Connection administration. The next update will be in 30 minutes.

2025-02-09 0:35 UTC 
We identified an issue where some parts of our system were overloaded due to a high number of jobs running during the nightly peak after 11 PM. This caused slowdowns and occasional errors in our UI and Storage API and some Queue jobs may have been delayed or failed.

We have increased system capacity to prevent this issue in the future.The incident is now resolved.

We apologize for any inconvenience caused.

]]>
Erik Žigo
tag:status.keboola.com,2013:Post/2180416 2025-03-05T16:11:34Z 2025-03-05T17:16:24Z Degraded Workspace Creation Performance

2025-03-05 16:10 UTC We are currently experiencing degraded performance with workspace creation, which may impact all jobs that rely on workspace creation as part of their lifecycle.

Our team is actively investigating the issue, and we will provide updates as we have more information.

We appreciate your patience and apologize for any inconvenience this may cause.


2025-03-05 17:15 UTC We've identified the core issue and deployed a fix. All Storage jobs associated with WorkspaceCreate should now be functioning properly.

]]>