tag:status.keboola.com,2013:/posts Keboola Status 2024-06-12T04:43:04Z Keboola Connection - Platform Status tag:status.keboola.com,2013:Post/2115892 2024-06-12T04:43:04Z 2024-06-12T04:43:04Z Stuck jobs and failures in AWS EU stack
2024-06-12 We have experienced some jobs stuck to continue and failed between  06:20 - 06:40 UTC. The root cause was invalid job node, we had to shut it down to prevent further issues.
]]>
tag:status.keboola.com,2013:Post/2113954 2024-06-06T08:25:03Z 2024-06-06T08:25:04Z Snowflake Security Notice

We want to inform our users of a recent security concern proactively announced by Snowflake. Snowflake has released important proactive measures that our Security team is currently following. Please find instructions here - https://community.snowflake.com/s/article/Communication-ID-0108977-Additional-Information

The Keboola Security team are performing queries in both Keboola managed and unmanaged Snowflake accounts to identify activity against the published list of suspected IP addresses - following the advice provided by Snowflake. If our investigations return any activity from the identified IP addresses, we will be in touch with your Keboola Admin immediately. Furthermore we will notify users once Snowflake provide an update from their ongoing inquiry.

]]>
tag:status.keboola.com,2013:Post/2112675 2024-05-28T10:47:56Z 2024-05-31T13:30:57Z Slow jobs in AWS eu-central-1 stack

2024-05-28 10:45 UTC - We are investigation slow responses from queue API. 

2024-05-28 11:25 UTCWe have identified the root cause. All services are now operating normally, but some jobs may be delayed.

]]>
tag:status.keboola.com,2013:Post/2111684 2024-05-23T12:34:08Z 2024-05-23T13:49:47Z Slow jobs listing in AWS eu-central-1 stack

2024-05-23 12:30 UTC - We are investigation slow responses in jobs list requests that affects the UI responsiveness. 

2024-05-23 13:48 UTC -  We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.

]]>
tag:status.keboola.com,2013:Post/2110921 2024-05-20T05:53:49Z 2024-05-20T06:41:08Z Job delays in AWS us-east-1 stack

2024-05-20 05:53 UTC - We are investigating job delays in AWS us-east-1 stack (connection.keboola.com). The next update will be provided when we have more information.

2024-05-20 06:38 UTC - We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.

]]>
tag:status.keboola.com,2013:Post/2110776 2024-05-19T16:25:04Z 2024-05-19T17:06:18Z Job delays in AWS us-east-1 stack

2024-05-19 16:24 UTC - We are investigating the higher error rate and job delays in AWS us-east-1 stack (connection.keboola.com). The next update will be provided when we have more information.

2024-05-19 16:49 UTC - The platform has stabilized, and the job backlog is being cleared. Jobs were delayed but not errored as a result of this incident.

 ]]>
tag:status.keboola.com,2013:Post/2109827 2024-05-15T12:28:27Z 2024-05-15T13:04:24Z Failing Facebook ads component due to Facebook outage

2024-05-15 12:14 UTC We are experiencing Facebook ads extractor component to fail due to Facebook outage. Facebook reports the outage is already solved, however we are still sill seeing components failing on "Service temporarily unavailable", which is error message directly from Facebook api. We continue monitoring the issue and wait for Facebook to resolve the outage correctly. More info on https://metastatus.com/ or https://developers.facebook.com/community/threads/3832997983612514/

]]>
tag:status.keboola.com,2013:Post/2109090 2024-05-12T03:47:40Z 2024-05-12T03:47:40Z Stuck jobs and failures in AWS EU stack

2024-05-12 We have experienced some jobs stuck to continue and failed between  03:00 - 03:45 UTC. The root cause was invalid job node, we had to shut it down to prevent further issues. 

]]>
tag:status.keboola.com,2013:Post/2108193 2024-05-07T08:07:32Z 2024-05-07T11:11:15Z Failures of dbt Cloud Job Trigger component

Since 2024-05-06 21:30 UTC, the dbt Cloud Job Trigger component has been failing due to the error 'Missing columns: scribe_enabled.' We are currently investigating the root cause and will provide more information as soon as it becomes available.

UPDATE 2024-05-07 08:43 UTC - Since 08:18 UTC, we have not observed any further errors. The DBT Cloud has resumed returning the scribe_enabled field, which has resolved the failures. We are currently investigating the cause of this change on their side.

UPDATE 2024-05-07 11:08 UTC - We have continued to observe a few occurrences of errors. A new version of the component was deployed at 10:05 UTC, designed not to fail if the scribe_enabled field is missing in the response. In such cases, scribe_enabled will be set as empty.
We have not observed any further errors since the deployment of the new version. The incident is now resolved.



]]>
tag:status.keboola.com,2013:Post/2107954 2024-05-06T07:05:29Z 2024-05-06T07:06:08Z KBC Project Metadata component failing on token endpoint
Due to change how we allow projects manage their tokens, projects using KBC Project Metadata component (kds-team.ex-kbc-project-metadata-v2) are failing while requesting token for a given project.

This component is however experimental and not longer maintained, so we're investigating best scenarios to mitigate failing jobs in affected projects. For now, we're reverting the change. It should propagate within ~90 minutes from this post.

Thanks for understanding.
]]>
tag:status.keboola.com,2013:Post/2107447 2024-05-03T12:28:00Z 2024-05-03T12:29:22Z Wrong behavior of Output mapping for DBT and no-code transformations in Azure north-europe region

All DBT and No-code transformations have been affected by a bug and no tables were imported from these transformations to Storage (in process of output mapping). This issue has been detected, root cause identified and version with this issue was rollbacked


DBT and no-code transforamtions running in interval 9AM CEST 2024-05-02 - 2pm CEST 2024-05-03 might be affected, and re-run is recommended.

This issue affected https://connection.north-europe.azure.keboola.com/ only. 

]]>
tag:status.keboola.com,2013:Post/2105945 2024-04-25T13:03:09Z 2024-04-26T07:43:24Z Queue v1 jobs and sync actions failing, Queue V2 jobs delayed in all stacks

Today since 12:00 UTC we suffer jobs as well as sync actions failing to start due to bug introduced in the last deploy. We are now reverting to the previous working version. 

Update 13:15 UTC (resolved) : We have successfully reverted backend to the previous working version and platform is back operational now.

Update 13:30 UTC: Upon further investigation and damage analysis, we found that Queue v2 jobs were delayed in starting because of this incident. That could result in delays of up to one hour.

]]>
tag:status.keboola.com,2013:Post/2105896 2024-04-25T02:24:27Z 2024-04-25T02:24:27Z Stuck jobs and failures in AWS EU stack

2024-04-25 02:23 UTC - We experienced another incident on connection.eu-central-1.keboola.com between 01:30 UTC and 02:30 UTC. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2105776 2024-04-24T14:32:18Z 2024-04-24T14:32:19Z Slowing Down Queue Jobs on AWS EU Stack

We noticed a slowdown in our AWS EU stack’s job queue from 12:00 to 14:00 UTC, due to a temporary service performance issue. We sincerely apologize for any inconvenience this may have caused.

Our team has resolved the problem and we are taking steps to prevent future occurrences. Thank you for your understanding and patience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2105351 2024-04-22T13:12:14Z 2024-04-22T15:14:42Z Error on loading data to storage from job

13:07 UTC: We're investigating an issue where import to Storage shows "Storage API: The new primary key cannot be created; duplicate values in primary key columns exist" error. 
Next update in 15 minutes. 

13:24 UTC: We've rolled back the last release. We're investigating impact of the issue. 

13:54 UTC: We're still investigating the impact. 

14:34 UTC: We're still investigating the impact. So far it seems to have only impacted a very limited number of customer jobs. We will reach out to the affected customers separately as soon as we finish the impact investigation.  

15:09 UTC: We've finished investigating the impact and we're reaching out to the affected customers. 

]]>
tag:status.keboola.com,2013:Post/2103328 2024-04-13T10:44:24Z 2024-04-13T13:30:13Z AWS eu-central stack maintenance

AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance will start today, April 13, 2024, at 12:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

Maintenance should take less than three hours.

We apologize for any inconvenience caused and thank you for your understanding.

Update 12:00 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance started.

Update 13:30 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance is finished and the stack is now fully operational.

Thank you for your patience.]]>
tag:status.keboola.com,2013:Post/2101626 2024-04-06T06:00:59Z 2024-04-06T08:02:15Z AWS us-east-1 stack maintenance

AWS us-east-1 stack (connection.keboola.com) maintenance will start today, April 06, 202407:00 UTC, as announced earlier.

You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.

We apologize for any inconvenience caused and thank you for your understanding.

Update 07:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance started.

Update 08:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance is finished and the stack is now fully operational.

Thank you for your patience.

]]>
tag:status.keboola.com,2013:Post/2101225 2024-04-04T07:33:05Z 2024-04-04T07:39:50Z Error on installing python packages in transformations

7:20 UTC: We are investigating an error when installing a python package in the transformations.
You may see errors such as: Job "XXXXXXX" ended with a user error "Failed to install package: io".
Next update in 15 min.

UPDATE 7:38 UTC: We have rolled back a previous version and all operations are back normal. We're sorry for this inconvenience. 



]]>
Václav Eder
tag:status.keboola.com,2013:Post/2098416 2024-03-23T10:00:03Z 2024-03-23T11:52:55Z Limited service disruption for Azure EU
A limited service disruption on Azure EU stack will start at 11:00 a.m. UTC today, as announced earlier. Storage jobs processing will stop and new jobs will be delayed until the upgrade is completed. All running jobs will be cancelled, but will resume after the upgrade.

We apologize for any inconvenience caused and thank you for your understanding.

Update 11:00 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) maintenance started.

Update 11:52 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) service disruption has been resolved and the stack is now fully operational. Thank you for your patience.


]]>
tag:status.keboola.com,2013:Post/2097231 2024-03-18T10:12:14Z 2024-03-18T10:12:14Z Reminder: Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to remind you of the upcoming scheduled maintenance for Keboola stacks hosted on AWS and Azure.

The announced maintenance of the Keboola platform for both Azure and AWS stacks is scheduled for March and April 2024.

Read more in the original maintenance announcement.

]]>
tag:status.keboola.com,2013:Post/2096831 2024-03-16T09:22:39Z 2024-03-17T12:06:20Z 2024-03-16 Jobs not starting in AWS eu-central-1

9:21 UTC: We are investigating jobs not starting on AWS EU this morning. Next update in 30 minutes.

9:57 UTC: We are still investigating the issue. Next update in 30 minutes.

10:24 UTC: We have identified stuck job that was causing other jobs not starting and proceed to unblock it. The platform is operational now and all stuck jobs have started running. We continue investigating the root cause. 

]]>
tag:status.keboola.com,2013:Post/2096819 2024-03-16T07:31:35Z 2024-03-17T12:06:24Z Limited service disruption of Queue V1 for AWS EU A limited service disruption of Queue V1 on the AWS EU stack (connection.eu-central-1.keboola.com) will begin today, Saturday, March 16, at 10:00 CET, as previously announced. This will impact only a small number of projects that continue to use Queue V1. Projects on Queue V2 will not be affected.  

We apologize for any inconvenience this may cause and appreciate your understanding.

Update 2024-03-16 10:10 CET: The limited service disruption has begun.

Update 2024-03-16 10:25 CET: The limited service disruption has been resolved and the stack is now fully operational. 

Thank you for your patience.

]]>
tag:status.keboola.com,2013:Post/2095155 2024-03-08T10:49:39Z 2024-03-17T12:06:26Z Limited Service Disruption: AWS EU's Queue V1 Unavailable on Saturday, March 16

Please be advised that there will be a one-hour service disruption next week. Check carefully to see if it will affect your projects.

Affected service: Queue Version 1 on connection.eu-central-1.keboola.com

Here are the details. Due to a necessary database upgrade to our AWS EU stack (connection.eu-central-1.keboola.com), there will be a limited service disruption of Queue Version 1 (V1) on Saturday, March 16, from 10:00 to 11:00 CET.

This will affect only a minority of projects that still use Queue V1. Projects on Queue V2 will not be affected.

Does your project use Queue Version 1? To find out, go to your Project Settings page and look for the queuev2 feature. If you don't see it, that means that your project does indeed use the old Queue V1, and that this service disruption will affect you.

Effects of the disruption

  1. Queue V1 and Orchestrations (Queue V1) processing will be temporarily halted.
  2. New Queue V1 jobs will be delayed until the upgrade is completed.
  3. All running Queue V1 jobs will be paused, and then restarted after the upgrade is completed.

We apologize for this inconvenience. We will appreciate your understanding as we strive to maintain and improve our service quality. If you have any questions or need further assistance, please contact our support team.


]]>
tag:status.keboola.com,2013:Post/2095145 2024-03-08T08:25:03Z 2024-03-17T12:06:30Z Job processing delays in AWS eu-central-1

2024-03-08 08:20 UTC: We are currently investigating an increase in latency within our job listings and delays in job execution on connection.eu-central-1.keboola.com. We will provide an update as soon as more information becomes available.

2024-03-08 09:05 UTC: We have identified the root cause and are actively working on a resolution. Job processing has not been affected since 08:20. We are continuing to investigate increased latency in Jobs and Flows page listings.

2024-03-08 09:15 UTC: The incident has been resolved. Latencies have returned to normal, and the platform is fully operational. We will continue to monitor the situation.

]]>
tag:status.keboola.com,2013:Post/2093693 2024-03-01T10:19:20Z 2024-03-17T12:06:28Z Failing queue jobs (There is no active transaction)

Following the release deployed at 8:30 UTC March 1st 2024, we encountered a bug that led to some queue jobs failing with the error "There is no active transaction." We addressed this issue by reverting the release at 9:50 UTC, which resolved the problem.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding and patience.

]]>
tag:status.keboola.com,2013:Post/2089600 2024-02-16T12:35:28Z 2024-03-17T12:06:34Z Scheduled Maintenance for Keboola AWS and Azure Stacks – March and April 2024

We would like to inform you about the planned maintenance of Keboola stacks hosted on AWS and Azure.

We will perform the maintenance of each stack separately during March and April 2024. This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:

Impact of the Maintenance and Recommended Mitigation:

  • Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
  • Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
  • Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for March 23, April 06, and April 13 to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.

Thank you for your patience and cooperation.

]]>
tag:status.keboola.com,2013:Post/2085675 2024-02-06T15:54:11Z 2024-03-17T12:06:37Z List of tables in buckets may not work correctly

Today at 15:30 UTC we noticed a problem with the listing tables in Storage. Tables fail to display only to new users in the project. There can be related issues such as not being able to load data to workspace. All of which apply only to new users. We are seeing this problem across all stacks and regions. 

We are working on a fix, next update in 30 min.

UPDATE 16:10 We have identified the root cause and are working on a fix. Next update in 2 hours.

UPDATE 19:00 The incident is now resolved, and tables are displayed correctly in storage for all users.

We apologize for the inconvenience.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2076969 2024-01-18T09:23:45Z 2024-03-17T12:06:40Z Support Service Unavailability

Due to the inoperability of the Atlassian Service Desk In the morning of January 18, 2024 our support services are temporarily unavailable.

We will provide an update as soon as services have returned to normal.

Many apologies for this inconvenience.

<January 18, 2024 12:30 CET>. Support systems have resumed normal operations

]]>
tag:status.keboola.com,2013:Post/2075541 2024-01-15T07:25:26Z 2024-03-17T12:06:41Z Stuck jobs and failures in AWS EU stack

2024-01-15 07:20 UTC - We experienced another incident on connection.eu-central-1.keboola.com between Jan 14 23:30 UTC and Jan 15 07:54 UTC. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.

]]>
Václav Eder
tag:status.keboola.com,2013:Post/2073494 2024-01-11T13:12:48Z 2024-03-17T12:06:43Z Email Delivery Issues

2024-01-11 13:04 - We are currently investigating a problem related to email delivery from the Keboola platform. We will provide an update as soon as we have more information.

UPDATE 13:19 - We have fixed the underlying issue (invalid SPF record). If you still have issues receiving messages from any Keboola platform please let us know. 

]]>