Blocked Google account when browsing in Keboola UI

2024-07-23 13:00 UTC: We are investigating problems with access to Google APIs in our UI

2024-07-23 13:10 UTC: Error about blocked account occurs when trying to browse content using file picker for components keboola.ex-google-drive, keboola.wr-google-sheets we are working on fix now.

2024-07-23 14:00 UTC: We have identified the root cause and are now waiting for Google to take action to resolve the issue.


Next update when new information is available.

Planned service maintenance on July 20th in GCP US and GCP EU stacks

We would like to inform you about the planned maintenance of Keboola stacks hosted on GCP.

This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:

Impact of the Maintenance and Recommended Mitigation:

  • Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
  • Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
  • Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for July 20th to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.

We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.

Thank you for your patience and cooperation.

2024-07-20 7:00 UTC: We have started the maintenance.

2024-07-20 7:58 UTC: The maintenance is complete, and both stacks are now fully operational.

Mysql extractor and Snowflake extractor job with custom query can drop PK on table

2024-07-12 13:20 UTC We discovered a problem with Mysql extractor with custom queries, this issue caused PK to be deleted in tables.

The problem has probably occurred since 09:50 UTC.

Update 13:28 UTC We found out that since 09:55 UTC the problem is also related to the Snowflake extractor.

Update 14:08 UTC We found that the mentioned problem started to appear from 09:50 UTC and for the mysql extractor the problem persisted until 12:50 UTC, in the case of the snowflake extractor until 13:30 UTC. 

In case you do not use custom query, mysql extractor or snowflake extractor this problem does not apply to you.

If you are using custom query and the job ended with an error at the incriminated time or next job ended with an error, in order fix your data you need to switch the extractor to full load and run the job again and contact support.

We are not experiencing any further problems and apologise for the inconvenience.

Storage Jobs Errors

2024-07-10 12:45 UTC We are experiencing some storage jobs failing with application errors since 12:00 UTC. We are currently investigating the issue and will provide an update within one hour.

2024-07-10 13:45 UTC We are still investigating the issue that is occurring since 12:00 UTC. Our team is working to identify and resolve the problem. We will provide another update within one hour.

2024-07-10 15:00 UTC We have likely identified the root cause of the problem. Our team is preparing to revert the changes that were deployed on the platform earlier today. Once the previous version is deployed, the issue should be resolved. We will provide another update within one hour.

2024-07-10 15:30 UTC The issue has been resolved by deploying the previous version. Jobs created after 15:25 UTC should no longer fail with internal errors. If you observe any issues, please feel free to contact our support team. We apologize for the inconvenience.

2024-07-12 06:50 UTC  We are investigating additional potential impacts of the earlier issue. In very rare cases, it may cause table export mismatches between jobs within the same project. This is not confirmed yet. We will contact affected customers if any are identified.

Pay as you go payments without credits

2024-07-05 11:06 UTC We are investigating payments on https://connection.north-europe.azure.keboola.com/ which are not transferred into credits in platform.

2024-07-05 11:15 UTC We have identified root cause of the issue and prepared fix which will be deployed approximately ~13:00 UTC, all payments will be credited when fix is deployed.

2024-07-05 13:00 UTC Issue is now resolved, all payments are credited.

We apologize for the inconvenience.

Misplaced columns in Google BigQuery Extractor component

2024-07-03 10:22 UTC: On Monday, July 1st, we released version 1.9.0 of the Google BigQuery Extractor. Unfortunately, it has come to our attention that this version does not properly respect the columns selected. Instead, it retrieves all columns, which could result in having different data in different columns than intended.

Impact:
This issue may affect the accuracy of the data extracted and could lead to discrepancies in your data analysis and reporting.

Current Status:
• We have reverted the Google BigQuery Extractor to the previous version to prevent further impact.
• Our team is actively investigating the root cause of this issue.

Next Steps:
• We are working diligently to develop a fix that will ensure the correct columns are extracted as specified.
• We are also reviewing the data impacted by this issue and will provide further guidance on any corrective actions required.

What You Should Do:
• Review your recent data extractions performed since Monday July 1st 11:54 UTC. You can do that using Workspaces or Data Preview.
• If you notice any discrepancies or have concerns about your data, please contact our support team immediately.

We understand the importance of data accuracy and apologize for any inconvenience this issue may have caused. We are committed to resolving this promptly and keeping you informed throughout the process.

Thank you for your understanding and patience.

2024-07-03 13:15 UTC: We have identified the root cause. The issue occurs if the columns selected in the configuration are in a different order than the columns that already exist in the output table. This can result in:

• Data Misplacement: Data in the resulting table being placed under the wrong column.
• Import Failures: If the table has more columns than selected in the configuration, the platform attempts to add the missing columns, which caused an import failure.

Solution:
We have a solution to fix the impacted data. If you find that your data was compromised by this issue, please contact our support team for assistance. They will help you out to revert your data to previous state.

Customer Action Required:
• If you notice any discrepancies or have concerns about your data, please contact our support team immediately.
• Our support team will assist in fixing any compromised data due to this incident.

Jobs detail error on all stacks

2024-06-28 18:30 UTC - Approximately from 2024-06-28 18:00 UTC we noticed a problem when displaying the job detail when the detail ended with the error "Invalid date time value "Invalid Date" for field createdTimeTo". 

We have resolved the issue by deploying the previous version and the error should no longer be displayed.

We apologize for the inconvenience.

Delay in Telemetry Data Updates

2024-06-22 7:40 UTC Due to an outage in the Snowflake in one of the regions, Telemetry Data has not been updated since 4:30 UTC and will be delayed for the duration of the outage.

Other functions of our platform should operate normally and remain unaffected by this outage.
This issue affects all of our customers, regardless of the Storage backend they are using (Snowflake, BigQuery, etc.).

2024-06-22 8:40 UTC THe Snowfalke outage has been resolved. Telemetry data has been fully updated and is now up-to-date. Thank you for patience.