Storage Jobs Errors

2024-07-10 12:45 UTC We are experiencing some storage jobs failing with application errors since 12:00 UTC. We are currently investigating the issue and will provide an update within one hour.

2024-07-10 13:45 UTC We are still investigating the issue that is occurring since 12:00 UTC. Our team is working to identify and resolve the problem. We will provide another update within one hour.

2024-07-10 15:00 UTC We have likely identified the root cause of the problem. Our team is preparing to revert the changes that were deployed on the platform earlier today. Once the previous version is deployed, the issue should be resolved. We will provide another update within one hour.

2024-07-10 15:30 UTC The issue has been resolved by deploying the previous version. Jobs created after 15:25 UTC should no longer fail with internal errors. If you observe any issues, please feel free to contact our support team. We apologize for the inconvenience.

2024-07-12 06:50 UTC  We are investigating additional potential impacts of the earlier issue. In very rare cases, it may cause table export mismatches between jobs within the same project. This is not confirmed yet. We will contact affected customers if any are identified.

Pay as you go payments without credits

2024-07-05 11:06 UTC We are investigating payments on https://connection.north-europe.azure.keboola.com/ which are not transferred into credits in platform.

2024-07-05 11:15 UTC We have identified root cause of the issue and prepared fix which will be deployed approximately ~13:00 UTC, all payments will be credited when fix is deployed.

2024-07-05 13:00 UTC Issue is now resolved, all payments are credited.

We apologize for the inconvenience.

Misplaced columns in Google BigQuery Extractor component

2024-07-03 10:22 UTC: On Monday, July 1st, we released version 1.9.0 of the Google BigQuery Extractor. Unfortunately, it has come to our attention that this version does not properly respect the columns selected. Instead, it retrieves all columns, which could result in having different data in different columns than intended.

Impact:
This issue may affect the accuracy of the data extracted and could lead to discrepancies in your data analysis and reporting.

Current Status:
• We have reverted the Google BigQuery Extractor to the previous version to prevent further impact.
• Our team is actively investigating the root cause of this issue.

Next Steps:
• We are working diligently to develop a fix that will ensure the correct columns are extracted as specified.
• We are also reviewing the data impacted by this issue and will provide further guidance on any corrective actions required.

What You Should Do:
• Review your recent data extractions performed since Monday July 1st 11:54 UTC. You can do that using Workspaces or Data Preview.
• If you notice any discrepancies or have concerns about your data, please contact our support team immediately.

We understand the importance of data accuracy and apologize for any inconvenience this issue may have caused. We are committed to resolving this promptly and keeping you informed throughout the process.

Thank you for your understanding and patience.

2024-07-03 13:15 UTC: We have identified the root cause. The issue occurs if the columns selected in the configuration are in a different order than the columns that already exist in the output table. This can result in:

• Data Misplacement: Data in the resulting table being placed under the wrong column.
• Import Failures: If the table has more columns than selected in the configuration, the platform attempts to add the missing columns, which caused an import failure.

Solution:
We have a solution to fix the impacted data. If you find that your data was compromised by this issue, please contact our support team for assistance. They will help you out to revert your data to previous state.

Customer Action Required:
• If you notice any discrepancies or have concerns about your data, please contact our support team immediately.
• Our support team will assist in fixing any compromised data due to this incident.

Jobs detail error on all stacks

2024-06-28 18:30 UTC - Approximately from 2024-06-28 18:00 UTC we noticed a problem when displaying the job detail when the detail ended with the error "Invalid date time value "Invalid Date" for field createdTimeTo". 

We have resolved the issue by deploying the previous version and the error should no longer be displayed.

We apologize for the inconvenience.

Delay in Telemetry Data Updates

2024-06-22 7:40 UTC Due to an outage in the Snowflake in one of the regions, Telemetry Data has not been updated since 4:30 UTC and will be delayed for the duration of the outage.

Other functions of our platform should operate normally and remain unaffected by this outage.
This issue affects all of our customers, regardless of the Storage backend they are using (Snowflake, BigQuery, etc.).

2024-06-22 8:40 UTC THe Snowfalke outage has been resolved. Telemetry data has been fully updated and is now up-to-date. Thank you for patience.

Snowflake Security Notice

We want to inform our users of a recent security concern proactively announced by Snowflake. Snowflake has released important proactive measures that our Security team is currently following. Please find instructions here - https://community.snowflake.com/s/article/Communication-ID-0108977-Additional-Information

The Keboola Security team are performing queries in both Keboola managed and unmanaged Snowflake accounts to identify activity against the published list of suspected IP addresses - following the advice provided by Snowflake. If our investigations return any activity from the identified IP addresses, we will be in touch with your Keboola Admin immediately. Furthermore we will notify users once Snowflake provide an update from their ongoing inquiry.

Slow jobs listing in AWS eu-central-1 stack

2024-05-23 12:30 UTC - We are investigation slow responses in jobs list requests that affects the UI responsiveness. 

2024-05-23 13:48 UTC -  We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.

Job delays in AWS us-east-1 stack

2024-05-20 05:53 UTC - We are investigating job delays in AWS us-east-1 stack (connection.keboola.com). The next update will be provided when we have more information.

2024-05-20 06:38 UTC - We have identified the root cause of the overload and took action to address it. The platform is stable at the moment. We continue to monitor it and are working on a permanent solution.