Issues across all Azure stacks

We're investigating issues across all Azure stacks. Next update in 15 minutes or when new information is available.

UPDATE 14:35 CEST: The issue seems to be resolved, we're still evaluating the impact. Next update in 30 minutes or when new information is available.

UPDATE 14:55 CEST: The outage was caused by routine maintenace of databases in Auzure. All running jobs affected during this outage should restart automatically. Some of the affected Storage jobs may be executed twice and in very rare cases when using incremental load without a primary key this could lead to data duplication.

We're sorry for this inconvenience, we will be taking measures to decrease impact of future maintenance events. 

For any additional questions, please contact support.

Limited service disruption for AWS US and EU stacks on Friday, December 29

Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on December 29:

  • 13:00–14:00 UTC (14:00–15:00 CET) for the AWS EU stack
  • 15:00–16:00 UTC (07:00–08:00 PDT) for the AWS US stack

We anticipate the duration of the disruption will be approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.

During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be paused, but will resume after the upgrade. The platform will be unavailable for approximately 5 minutes. The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs or Queue v1 jobs.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.

We apologize for any inconvenience caused and thank you for your understanding.

Brief UI inaccessibility across all stacks

We experienced a brief outage in the application where the platform was inaccessible due to a UI bug. We promptly addressed the issue by rolling back the latest change, and this resolution has been deployed across all stacks.

It's important to note that no flows, running jobs, or workspaces were affected during this incident.

Planned service maintenance on April 15th in AWS US and AWS EU stacks

Regrettably, we were unable to upgrade all necessary databases during the previous planned service disruption. Due to a strict deadline imposed by our service provider (AWS), we must carry out another service disruption for maintenance purposes.

This maintenance will impact both the AWS US and AWS EU stacks.

It is scheduled for Saturday, April 15, 2023,

  • between 07:00 and 08:00 UTC (09:00 and 10:00 CEST) for the AWS EU stack, and
  • between 09:00 and 10:00 UTC (02:00 and 03:00 PDT) for the AWS US stack.

During this time, Storage jobs will be paused or delayed, and the platform will be unavailable for a brief period (approximately 5 minutes). The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs.

We sincerely apologize for any inconvenience caused and appreciate your understanding.

Limited service disruption for AWS US

A limited service disruption on AWS US stack will start at 10:00 a.m. UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 10:00 a.m. UTC: The limited service disruption has begun.

Update 10:10 a.m. UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.


Brief metadata database outage in AWS US

We have encountered a brief metadata DB outage in AWS US at 15:07 UTC. Affected services are

  • legacy Transformations
  • legacy Orchestrations
  • projects and jobs running on old Queue V1

This outage may cause some jobs being executed during the outage fail or run twice in parallel.

We're sorry for this inconvenience. 

UPDATE 15:39 UTC: All affected jobs were restarted and any duplicate executions were terminated.

Limited service disruption for AWS EU

A limited service disruption on AWS EU stack will start at 12:00 p.m. UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 12:00 p.m. UTC: The limited service disruption has begun.

Update 12:33 p.m. UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

Limited service disruption for AWS US and EU stacks on March 21st and 22nd

Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on March 21st and 22nd.

  • On Tuesday March 21st at 12:00 pm UTC, the disruption will begin for AWS EU, and
  • on Wednesday March 22nd at 10:00 am UTC, it will begin for AWS US.

We anticipate that the limited service disruption will take approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.

During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.

We apologize for any inconvenience caused and thank you for your understanding.