Since July 24th 19:30 UTC, we are seeing increased error rate jobs of kds-team.app-custom-python component, failing with "Failed to install package" error message.
We have reverted to the previous working version and component is now working as expected.
]]>2024-07-23 13:00 UTC: We are investigating problems with access to Google APIs in our UI
2024-07-23 13:10 UTC: Error about blocked account occurs when trying to browse content using file picker for components keboola.ex-google-drive, keboola.wr-google-sheets we are working on fix now.
2024-07-23 14:00 UTC: We have identified the root cause and are now waiting for Google to take action to resolve the issue.
Next update when new information is available.
]]>
We would like to inform you about the planned maintenance of Keboola stacks hosted on GCP.
This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:
GCP europe-west3: https://connection.europe-west3.gcp.keboola.com/
GCP us-east4: https://connection.us-east4.gcp.keboola.com/
Impact of the Maintenance and Recommended Mitigation:
We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.
Thank you for your patience and cooperation.
2024-07-20 7:00 UTC: We have started the maintenance.
2024-07-20 7:58 UTC: The maintenance is complete, and both stacks are now fully operational.
2024-07-12 13:20 UTC We discovered a problem with Mysql extractor with custom queries, this issue caused PK to be deleted in tables.
The problem has probably occurred since 09:50 UTC.
Update 13:28 UTC We found out that since 09:55 UTC the problem is also related to the Snowflake extractor.
Update 14:08 UTC We found that the mentioned problem started to appear from 09:50 UTC and for the mysql extractor the problem persisted until 12:50 UTC, in the case of the snowflake extractor until 13:30 UTC.2024-07-10 12:45 UTC We are experiencing some storage jobs failing with application errors since 12:00 UTC. We are currently investigating the issue and will provide an update within one hour.
2024-07-10 13:45 UTC We are still investigating the issue that is occurring since 12:00 UTC. Our team is working to identify and resolve the problem. We will provide another update within one hour.
2024-07-10 15:00 UTC We have likely identified the root cause of the problem. Our team is preparing to revert the changes that were deployed on the platform earlier today. Once the previous version is deployed, the issue should be resolved. We will provide another update within one hour.
2024-07-10 15:30 UTC The issue has been resolved by deploying the previous version. Jobs created after 15:25 UTC should no longer fail with internal errors. If you observe any issues, please feel free to contact our support team. We apologize for the inconvenience.
2024-07-12 06:50 UTC We are investigating additional potential impacts of the earlier issue. In very rare cases, it may cause table export mismatches between jobs within the same project. This is not confirmed yet. We will contact affected customers if any are identified.
]]>2024-07-05 11:06 UTC We are investigating payments on https://connection.north-europe.azure.keboola.com/ which are not transferred into credits in platform.
2024-07-05 11:15 UTC We have identified root cause of the issue and prepared fix which will be deployed approximately ~13:00 UTC, all payments will be credited when fix is deployed.
2024-07-05 13:00 UTC Issue is now resolved, all payments are credited.
We apologize for the inconvenience.
Thank you for your understanding and patience.
2024-07-03 13:15 UTC: We have identified the root cause. The issue occurs if the columns selected in the configuration are in a different order than the columns that already exist in the output table. This can result in:
2024-06-28 18:30 UTC - Approximately from 2024-06-28 18:00 UTC we noticed a problem when displaying the job detail when the detail ended with the error "Invalid date time value "Invalid Date" for field createdTimeTo".
We have resolved the issue by deploying the previous version and the error should no longer be displayed.
We apologize for the inconvenience.
2024-06-22 7:40 UTC Due to an outage in the Snowflake in one of the regions, Telemetry Data has not been updated since 4:30 UTC and will be delayed for the duration of the outage.
Other functions of our platform should operate normally and remain unaffected by this outage.
This issue affects all of our customers, regardless of the Storage backend they are using (Snowflake, BigQuery, etc.).
2024-06-22 8:40 UTC THe Snowfalke outage has been resolved. Telemetry data has been fully updated and is now up-to-date. Thank you for patience.
We want to inform our users of a recent security concern proactively announced by Snowflake. Snowflake has released important proactive measures that our Security team is currently following. Please find instructions here - https://community.snowflake.com/s/article/Communication-ID-0108977-Additional-Information
The Keboola Security team are performing queries in both Keboola managed and unmanaged Snowflake accounts to identify activity against the published list of suspected IP addresses - following the advice provided by Snowflake. If our investigations return any activity from the identified IP addresses, we will be in touch with your Keboola Admin immediately. Furthermore we will notify users once Snowflake provide an update from their ongoing inquiry.
]]>2024-05-28 10:45 UTC - We are investigation slow responses from queue API.
2024-05-28 11:25 UTC - We have identified the root cause. All services are now operating normally, but some jobs may be delayed.
]]>2024-05-23 12:30 UTC - We are investigation slow responses in jobs list requests that affects the UI responsiveness.
2024-05-23 13:48 UTC - We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.
]]>2024-05-20 05:53 UTC - We are investigating job delays in AWS us-east-1 stack (connection.keboola.com). The next update will be provided when we have more information.
2024-05-20 06:38 UTC - We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.
2024-05-19 16:24 UTC - We are investigating the higher error rate and job delays in AWS us-east-1 stack (connection.keboola.com). The next update will be provided when we have more information.
2024-05-19 16:49 UTC - The platform has stabilized, and the job backlog is being cleared. Jobs were delayed but not errored as a result of this incident.
2024-05-15 12:14 UTC We are experiencing Facebook ads extractor component to fail due to Facebook outage. Facebook reports the outage is already solved, however we are still sill seeing components failing on "Service temporarily unavailable", which is error message directly from Facebook api. We continue monitoring the issue and wait for Facebook to resolve the outage correctly. More info on https://metastatus.com/ or https://developers.facebook.com/community/threads/3832997983612514/
]]>2024-05-12 We have experienced some jobs stuck to continue and failed between 03:00 - 03:45 UTC. The root cause was invalid job node, we had to shut it down to prevent further issues.
]]>Since 2024-05-06 21:30 UTC, the dbt Cloud Job Trigger component has been failing due to the error 'Missing columns: scribe_enabled.' We are currently investigating the root cause and will provide more information as soon as it becomes available.
UPDATE 2024-05-07 08:43 UTC - Since 08:18 UTC, we have not observed any further errors. The DBT Cloud has resumed returning the scribe_enabled field, which has resolved the failures. We are currently investigating the cause of this change on their side.
UPDATE 2024-05-07 11:08 UTC - We have continued to observe a few occurrences of errors. A new version of the component was deployed at 10:05 UTC, designed not to fail if the scribe_enabled field is missing in the response. In such cases, scribe_enabled will be set as empty.
We have not observed any further errors since the deployment of the new version. The incident is now resolved.
All DBT and No-code transformations have been affected by a bug and no tables were imported from these transformations to Storage (in process of output mapping). This issue has been detected, root cause identified and version with this issue was rollbacked.
DBT and no-code transforamtions running in interval 9AM CEST 2024-05-02 - 2pm CEST 2024-05-03 might be affected, and re-run is recommended.
This issue affected https://connection.north-europe.azure.keboola.com/ only.
]]>Today since 12:00 UTC we suffer jobs as well as sync actions failing to start due to bug introduced in the last deploy. We are now reverting to the previous working version.
Update 13:15 UTC (resolved) : We have successfully reverted backend to the previous working version and platform is back operational now.
Update 13:30 UTC: Upon further investigation and damage analysis, we found that Queue v2 jobs were delayed in starting because of this incident. That could result in delays of up to one hour.
]]>2024-04-25 02:23 UTC - We experienced another incident on connection.eu-central-1.keboola.com between 01:30 UTC and 02:30 UTC. Some jobs ended in error due to an underlying node failure. We're still investigating the root cause and taking measures to prevent future incidents.
]]>We noticed a slowdown in our AWS EU stack’s job queue from 12:00 to 14:00 UTC, due to a temporary service performance issue. We sincerely apologize for any inconvenience this may have caused.
Our team has resolved the problem and we are taking steps to prevent future occurrences. Thank you for your understanding and patience.
13:07 UTC: We're investigating an issue where import to Storage shows "Storage API: The new primary key cannot be created; duplicate values in primary key columns exist" error.
Next update in 15 minutes.
13:24 UTC: We've rolled back the last release. We're investigating impact of the issue.
13:54 UTC: We're still investigating the impact.
14:34 UTC: We're still investigating the impact. So far it seems to have only impacted a very limited number of customer jobs. We will reach out to the affected customers separately as soon as we finish the impact investigation.
15:09 UTC: We've finished investigating the impact and we're reaching out to the affected customers.
]]>AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance will start today, April 13, 2024, at 12:00 UTC, as announced earlier.
You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
Maintenance should take less than three hours.
We apologize for any inconvenience caused and thank you for your understanding.
Update 12:00 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance started.
Update 13:30 UTC: AWS eu-central-1 stack (connection.eu-central-1.keboola.com) maintenance is finished and the stack is now fully operational.
AWS us-east-1 stack (connection.keboola.com) maintenance will start today, April 06, 2024, 07:00 UTC, as announced earlier.
You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
We apologize for any inconvenience caused and thank you for your understanding.
Update 07:00 UTC: AWS us-east-1 stack (connection.keboola.com) maintenance started.
7:20 UTC: We are investigating an error when installing a python package in the transformations.
You may see errors such as: Job "XXXXXXX" ended with a user error "Failed to install package: io".
Next update in 15 min.
UPDATE 7:38 UTC: We have rolled back a previous version and all operations are back normal. We're sorry for this inconvenience.
Update 11:00 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) maintenance started.
Update 11:52 UTC: Azure North Europe stack (connection.north-europe.azure.keboola.com) service disruption has been resolved and the stack is now fully operational. Thank you for your patience.
We would like to remind you of the upcoming scheduled maintenance for Keboola stacks hosted on AWS and Azure.
The announced maintenance of the Keboola platform for both Azure and AWS stacks is scheduled for March and April 2024.
Read more in the original maintenance announcement.
9:21 UTC: We are investigating jobs not starting on AWS EU this morning. Next update in 30 minutes.
9:57 UTC: We are still investigating the issue. Next update in 30 minutes.
10:24 UTC: We have identified stuck job that was causing other jobs not starting and proceed to unblock it. The platform is operational now and all stuck jobs have started running. We continue investigating the root cause.
]]>