tag:status.keboola.com,2013:/posts Keboola Status 2024-04-13T13:30:13Z Keboola Connection - Platform Status tag:status.keboola.com,2013:Post/2103328 2024-04-13T10:44:24Z 2024-04-13T13:30:13Z AWS eu-central stack maintenance

AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance will start today, April 13, 2024, at 12:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

Maintenance should take less than three hours.

We apologize for any inconvenience caused and thank you for your understanding.

Update 12:00 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance started.

Update 13:30 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance is finished and the stack is now fully operational.

Thank you for your patience.]]>
tag:status.keboola.com,2013:Post/2101626 2024-04-06T06:00:59Z 2024-04-06T08:02:15Z AWS us-east-1 stack maintenance

AWS us-east-1 stack (connection.keboola.com) maintenance will start today, April 06, 202407:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

We apologize for any inconvenience caused and thank you for your understanding.

Update 07:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance started.

Update 08:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance is finished and the stack is now fully operational.

Thank you for your patience.

]]>
tag:status.keboola.com,2013:Post/2101225 2024-04-04T07:33:05Z 2024-04-04T07:39:50Z Error on installing python packages in transformations

7:20 UTC: We are investigating an error when installing a python package in the transformations.
You may see errors such as: Job "XXXXXXX" ended with a user error "Failed to install package: io".
Next update in 15 min.

UPDATE 7:38 UTC: We have rolled back a previous version and all operations are back normal. We're sorry for this inconvenience. 



]]>
Václav Eder
tag:status.keboola.com,2013:Post/2098416 2024-03-23T10:00:03Z 2024-03-23T11:52:55Z Limited service disruption for Azure EU
A limited service disruption on Azure EU stack will start at 11:00 a.m. UTC today, as announced earlier. Storage jobs processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

We apologize for any inconvenience caused and thank you for your understanding.

Update 11:00 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) maintenance started.

Update 11:52 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) service disruption has been resolved and the stack is now fully operational. Thank you for your patience.


]]>
tag:status.keboola.com,2013:Post/2097231 2024-03-18T10:12:14Z 2024-03-18T10:12:14Z Reminder: Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to remind you of the upcoming scheduled maintenance for Keboola stacks hosted on AWS and Azure.

The announced maintenance of the Keboola platform for both Azure and AWS stacks is scheduled for March and April 2024.

Read more in the original maintenance announcement.

]]>
tag:status.keboola.com,2013:Post/2096831 2024-03-16T09:22:39Z 2024-03-17T12:06:20Z 2024-03-16 Jobs not starting in AWS eu-central-1

9:21 UTC: We are investigating jobs not starting on AWS EU this morning. Next update in 30 minutes.

9:57 UTC: We are still investigating the issue. Next update in 30 minutes.

10:24 UTC: We have identified stuck job that was causing other jobs not starting and proceed to unblock it. The platform is operational now and all stuck jobs have started running. We continue investigating the root cause. 

]]>
tag:status.keboola.com,2013:Post/2096819 2024-03-16T07:31:35Z 2024-03-17T12:06:24Z Limited service disruption of Queue V1 for AWS EU A limited service disruption of Queue V1 on the AWS EU stack (connection.eu-central-1.keboola.com) will begin today, Saturday, March 16, at 10:00 CET, as previously announced. This will impact only a small number of projects that continue to use Queue V1. Projects on Queue V2 will not be affected.  

We apologize for any inconvenience this may cause and appreciate your understanding.

Update 2024-03-16 10:10 CET: The limited service disruption has begun.

Update 2024-03-16 10:25 CET: The limited service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

]]>
tag:status.keboola.com,2013:Post/2095155 2024-03-08T10:49:39Z 2024-03-17T12:06:26Z Limited Service Disruption: AWS EU's Queue V1 Unavailable on Saturday, March 16

Please be advised that there will be a one-hour service disruption next week. Check carefully to see if it will affect your projects.

Affected service: Queue Version 1 on connection.eu-central-1.keboola.com

Here are the details. Due to a necessary database upgrade to our AWS EU stack (connection.eu-central-1.keboola.com), there will be a limited service disruption of Queue Version 1 (V1) on Saturday, March 16, from 10:00 to 11:00 CET.

This will affect only a minority of projects that still use Queue V1. Projects on Queue V2 will not be affected.

Does your project use Queue Version 1? To find out, go to your Project Settings page and look for the queuev2 feature. If you don't see it, that means that your project does indeed use the old Queue V1, and that this service disruption will affect you.

Effects of the disruption

  1. Queue V1 and Orchestrations (Queue V1) processing will be temporarily halted.
  2. New Queue V1 jobs will be delayed until the upgrade is completed.
  3. All running Queue V1 jobs will be paused, and then restarted after the upgrade is completed.

We apologize for this inconvenience. We will appreciate your understanding as we strive to maintain and improve our service quality. If you have any questions or need further assistance, please contact our support team.


]]>
tag:status.keboola.com,2013:Post/2095145 2024-03-08T08:25:03Z 2024-03-17T12:06:30Z Job processing delays in AWS eu-central-1

2024-03-08 08:20 UTC: We are currently investigating an increase in latency within our job listings and delays in job execution on connection.eu-central-1.keboola.com. We will provide an update as soon as more information becomes available.

2024-03-08 09:05 UTC: We have identified the root cause and are actively working on a resolution. Job processing has not been affected since 08:20. We are continuing to investigate increased latency in Jobs and Flows page listings.

2024-03-08 09:15 UTC: The incident has been resolved. Latencies have returned to normal, and the platform is fully operational. We will continue to monitor the situation.

]]>
tag:status.keboola.com,2013:Post/2093693 2024-03-01T10:19:20Z 2024-03-17T12:06:28Z Failing queue jobs (There is no active transaction)

Following the release deployed at 8:30 UTC March 1st 2024, we encountered a bug that led to some queue jobs failing with the error "There is no active transaction." We addressed this issue by reverting the release at 9:50 UTC, which resolved the problem.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding and patience.

]]>
tag:status.keboola.com,2013:Post/2089600 2024-02-16T12:35:28Z 2024-03-17T12:06:34Z Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS and Azure.

We will perform the maintenance of each stack separately during March and April 2024. This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:

Impact of the Maintenance and Recommended Mitigation:

  • Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
  • Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
  • Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for March 23, April 06, and April 13 to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.

Thank you for your patience and cooperation.

]]>
tag:status.keboola.com,2013:Post/2085675 2024-02-06T15:54:11Z 2024-03-17T12:06:37Z List of tables in buckets may not work correctly

Today at 15:30 UTC we noticed a problem with the listing tables in Storage. Tables fail to display only to new users in the project. There can be related issues such as not being able to load data to workspace. All of which apply only to new users. We are seeing this problem across all stacks and regions. 

We are working on a fix, next update in 30 min.

UPDATE 16:10 We have identified the root cause and are working on a fix. Next update in 2 hours.

UPDATE 19:00 The incident is now resolved, and tables are displayed correctly in storage for all users.

We apologize for the inconvenience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2076969 2024-01-18T09:23:45Z 2024-03-17T12:06:40Z Support Service Unavailability

Due to the inoperability of the Atlassian Service Desk In the morning of January 18, 2024 our support services are temporarily unavailable.

We will provide an update as soon as services have returned to normal.

Many apologies for this inconvenience.

<January 18, 2024 12:30 CET>. Support systems have resumed normal operations

]]>
tag:status.keboola.com,2013:Post/2075541 2024-01-15T07:25:26Z 2024-03-17T12:06:41Z Stuck jobs and failures in AWS EU stack

2024-01-15 07:20 UTC - We experienced another incident on connection.eu-central-1.keboola.com between Jan 14 23:30 UTC and Jan 15 07:54 UTC. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2073494 2024-01-11T13:12:48Z 2024-03-17T12:06:43Z Email Delivery Issues

2024-01-11 13:04 - We are currently investigating a problem related to email delivery from the Keboola platform. We will provide an update as soon as we have more information.

UPDATE 13:19 - We have fixed the underlying issue (invalid SPF record). If you still have issues receiving messages from any Keboola platform please let us know. 

]]>
tag:status.keboola.com,2013:Post/2070542 2024-01-04T06:58:39Z 2024-03-17T12:06:48Z Stuck jobs and failures in AWS US stack

2024-04-01 06:35 UTC - We experienced incident on https://connection.keboola.com. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

Update 2024-04-01 06:54 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/2068075 2023-12-29T14:03:45Z 2024-03-17T12:06:50Z Limited service disruption for AWS US

A limited service disruption on AWS EU stack will start at 15:00 UTC today, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends.

We apologize for any inconvenience caused and thank you for your understanding.

Update 15:00 UTC: The limited service disruption has begun.

Update 15:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2068023 2023-12-29T11:58:02Z 2024-03-17T12:06:53Z Limited service disruption for AWS EU

A limited service disruption on AWS EU stack will start at 13:00 UTC today Friday, December 29, as announced earlier. Storage jobs, Queue v1, and Orchestration (in projects with Queue v1) processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays. We will provide an update when the service disruption starts and ends. 

We apologize for any inconvenience caused and thank you for your understanding.

Update 13:00 UTC: The limited service disruption has begun.

Update 13:35 UTC: The service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

]]>
tag:status.keboola.com,2013:Post/2064763 2023-12-20T10:46:40Z 2024-03-17T12:06:56Z Limited service disruption for AWS US and EU stacks on Friday, December 29

Due to necessary database upgrades to our AWS US and EU stacks, a limited service disruption will take place on December 29:

  • 13:00–14:00 UTC (14:00–15:00 CET) for the AWS EU stack
  • 15:00–16:00 UTC (07:00–08:00 PDT) for the AWS US stack

We anticipate the duration of the disruption will be approximately 15 minutes, but it should not exceed 60 minutes. Hopefully, this will be resolved before you return from your lunch or coffee break.

During this period, Storage jobs, Queue v1 and Orchestration (in projects with Queue v1) processing will stop, and new jobs will be delayed until the upgrade is completed. All running jobs will be paused, but will resume after the upgrade. The platform will be unavailable for approximately 5 minutes. The platform will then generate a 500 HTTP response for the majority of API requests. Throughout the remainder of the maintenance window, the platform will be fully accessible but will not process any new or existing Storage jobs or Queue v1 jobs.

All APIs and other unaffected services, such as Workspaces and Queue v2 jobs, will remain operational, though their operations may be delayed due to the Storage job delays.

We apologize for any inconvenience caused and thank you for your understanding.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2064008 2023-12-18T11:09:08Z 2024-03-17T12:06:58Z Some flows processing longer than usual

Between 9:18 and 10:42 UTC some flows were processing longer than usual with jobs "stuck" in processing or waiting state. This was caused be a recent update which was already reverted. The issue affected all stacks but only a small portion of jobs. We sincerely apologize for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/2060008 2023-12-08T15:10:34Z 2024-03-17T12:07:01Z Job processing slowing down on Azure NE stack

Since 14:00 UTC, we have been experiencing slower job processing in the Azure North Europe stack (https://connection.north-europe.azure.keboola.com). We are currently investigating the issue.

Update 16:20 UTC we have rolled back a previous version of one of our services that was causing the delays and all operations are back normal. We're sorry for this inconvenience. 

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2057008 2023-11-30T13:04:03Z 2024-03-17T12:07:03Z Event triggers in AWS US not working

We are investigating event triggers not working and also not visible in UI in AWS US stack (https://connection.keboola.com). No flows based on event triggers are being executed at the moment. Time based schedules are working as expected as well as event triggers in other stacks.

Update 13:35 UTC: We identified the root cause and will now proceed to restore event triggers configurations.

(Resolved) Update 14:05 UTC: We restored event triggers configurations, all event triggers are now working as expected and we don't observe any failures regarding even triggered flows. However if you still experience issues please contact us on support to resolve it further.


]]>
tag:status.keboola.com,2013:Post/2056513 2023-11-29T13:08:05Z 2023-11-29T13:21:13Z Investigating higher latency through all stacks
As of 29 November 13:45 UTC, we are investigating higher latency for some requests in the through all stacks.
  • It might leads to errors in the UI
  • Job processing is not affected
We'll be doing a rollback to the previous version. Next update in 30 min.

UPDATE 2023-11-29 13:20 UTC - All operations are back to normal and everything is fully working.
]]>
Václav Eder
tag:status.keboola.com,2013:Post/2056452 2023-11-29T07:53:32Z 2023-11-29T07:53:33Z Brief UI inaccessibility across all stacks

We experienced a brief outage in the application where the platform was inaccessible due to a UI bug. We promptly addressed the issue by rolling back the latest change, and this resolution has been deployed across all stacks.

It's important to note that no flows, running jobs, or workspaces were affected during this incident.

]]>
Ondrej Hlavacek
tag:status.keboola.com,2013:Post/2055206 2023-11-26T03:20:09Z 2023-11-26T04:15:20Z Job processing slowing down on Azure NE stack

2023-11-26 3:19 UTC - The processing of jobs in Azure NE stack (https://connection.north-europe.azure.keboola.com) has been slowed down, we're investigating the issue.

2023-11-26 4:15 UTC - Slowdown of starting new jobs was caused by heavy load. The backlog of jobs has been cleared, and new jobs are processing with no delays. We apologise for the inconvenience.

]]>
Erik Žigo
tag:status.keboola.com,2013:Post/2048643 2023-11-20T10:32:32Z 2023-11-20T16:10:06Z Data Apps Security Incident: November 13, 2023
We'd like to share an update about a proactive measure we've taken to ensure system security. On Monday, November 13, our diligent testing processes helped us to identify a vulnerability in the Data Apps component of our system. As part of our continuous efforts to strengthen security, we have fortified the protection against unauthorized decryption and access to some parts of our infrastructure that could have occurred under certain conditions.
We paused the provisioning of Data Apps as a precautionary step. By the day after discovering the vulnerability, November 14, our dedicated engineering team had comprehensively examined and effectively addressed the issue, ensuring the highest standards of security and reliability for our services. Following this, we thoroughly reviewed the configurations and logs of all Data Apps and found no suspicious activity.

If you have any questions or need more information, we encourage you to contact us at our support center: https://help.keboola.com/management/support.
]]>
tag:status.keboola.com,2013:Post/2043749 2023-11-03T14:47:35Z 2023-11-03T16:06:02Z Job processing slowing down on Azure NE stack

2023-11-03 14:40 UTC We're experiencing job processing slowing down on Azure NE stack (https://connection.north-europe.azure.keboola.com) due to heavy load. We apologise for the inconvenience.

Update 2023-11-03 15:10 UTC We have managed to find the cause and are working on a fix.

Update 2023-11-03 15:40 UTC We managed to mitigate the main cause. It will take some time for the platform to return to normal. 

Update 2023-11-03 16:05 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/2042188 2023-10-30T16:54:23Z 2023-10-30T19:53:44Z Delayed job logs on AWS US

2023-10-30 15:17 UTC We're experiencing log delays on running jobs on AWS US stack (https://connection.keboola.com) due to heavy load. 

Update 2023-10-30 17:34 UTC We have managed to find the cause and are working on a fix.

Update 2023-10-30 19:50 UTC The incident is now resolved, and the platform is stable again.

We apologize for the inconvenience.

]]>
tag:status.keboola.com,2013:Post/2035172 2023-10-11T05:54:21Z 2023-10-11T06:39:03Z Some jobs are failing with OAuth Broker deprecation notice

2023-10-11 5:54 UTC - Some runs of component using OAuth Broker are failing with User errors "OAuth Broker v2 has been deprecated". We are investigating the issue.

2023-10-11 6:30 UTC - We have identified the cause of the issue. Yesterday at 13:20 UTC, a new UI version was deployed that contained a bug in configuration editor.

During the configuration editing of components that are using the OAuth Broker, the version paremeter was of the broker 3 was inadvertently deleted.


It affected only configrations that were created or modified between 2023-10-10 13:20 UTC and 2023-10-11 6:00 UTC. Jobs of these configurations are ending with User Error "OAuth Broker v2 has been deprecated on September 30, 2019."


UI changes were reverted, but your failing configurations need to be fixed manually.

Go to the all version history in your configuration and use "Restore" button to revert changes you made. It will also revert OAuth configuration back to use newer broker v3. New configuration modifications won't break the settings anymore.

Second option to fix the issue is reauthorization of OAuth in your configuration.

We apologize for any inconvenience caused.


]]>
Erik Žigo
tag:status.keboola.com,2013:Post/2029548 2023-09-26T13:41:04Z 2023-09-26T13:41:05Z Deprecated Twitter Data Source

Due to the changes at Twitter we are no longer able to provide it as a Data Source component. 
We will be introducing an X Data Source if demand calls for it.

If you have any questions, or would like to push for introducing an X Data Source then please let us know at support@keboola.com.

]]>