2024-08-06 We have experienced some jobs stuck to continue and failed between 02:22 - 04:45 UTC. The root cause was invalid job node, we had to shut it down to prevent further issues.
2024-08-02 12:04 UTC: We're investigating an issue, where columns are getting mismatch for different components.
Next update in 1 hour.
2024-08-02 12:54 UTC: We have identified the root cause of the recent service disruption in the new manifest processor version released yesterday afternoon. We have now rolled back to the previous version to mitigate the issue.
Action Required:
If you have been impacted by this disruption, please delete any newly added columns and rerun your configuration. This should resolve the issue and allow your job to succeed.
If you continue to encounter problems, please do not hesitate to reach out to our support team for assistance.
We apologize for any inconvenience this may have caused and appreciate your understanding.
At 2024-07-31 12:05 UTC we've released a new version of "keboola.processor-create-manifest" processor, which introduced a bug resulting in following error message:
Sliced file "table_name" columns specification missing.We've reverted the processor to previous version. Please re-run your configurations if you've encountered this error.
We apologize for any inconvenience.
2024-07-31 12:15 UTC We have started an investigation into the Telemetry data extractor failures, which have been occurring for the last two days.
2024-07-31 17:15 UTC The telemetry extractor has been failing in projects with the native data types feature enabled. Columns set as non-nullable have their empty string records translated as NULLs, resulting in the error NULL result in a non-nullable column. We have decided to change all non-primary key columns to be nullable to mitigate this issue. To align the table column definitions in affected projects with the data source, we need to force a full load for all telemetry tables. The forced full load will be disabled on Tuesday, August 6th.
6:10 UTC We're seeing slow listing of component jobs in AWS US stack (https://connection.keboola.com/). List of jobs can take considerable time or it can timeout completely. Job processing and other functions are not affected now. Other stacks are not affected. We're investigating the issue.
Update 6:53 UTC: Job listing is still a bit slow, but it should not generally timeout any more.
Update 08:20 UTC: Job listing performance has stabilized. We continue to monitor the situation. The next update will be provided when new information is available.
Since July 24th 19:30 UTC, we are seeing increased error rate jobs of kds-team.app-custom-python component, failing with "Failed to install package" error message.
We have reverted to the previous working version and component is now working as expected.
2024-07-23 13:00 UTC: We are investigating problems with access to Google APIs in our UI
2024-07-23 13:10 UTC: Error about blocked account occurs when trying to browse content using file picker for components keboola.ex-google-drive, keboola.wr-google-sheets we are working on fix now.
2024-07-23 14:00 UTC: We have identified the root cause and are now waiting for Google to take action to resolve the issue.
Next update when new information is available.
2024-09-02 12:22 UTC: We have updated our privacy policy in accordance with new google requirements and are waiting for their approval
2024-09-26 13:47 UTC: The Google Picker UI component has been finally verified by Google and it's now working as expected.
Thank you for your patience and we're sorry for any inconvenience.
We would like to inform you about the planned maintenance of Keboola stacks hosted on GCP.
This maintenance is necessary to keep our services running smoothly and securely. Please note the following schedule and the stacks affected:
-
GCP europe-west3: https://connection.europe-west3.gcp.keboola.com/
- Date & Time: Saturday, July 20, 2024, at 07:00 UTC
- Expected Downtime: Less than 1 hour
-
GCP us-east4: https://connection.us-east4.gcp.keboola.com/
- Date & Time: Saturday, July 20, 2024, at 07:00 UTC
- Expected Downtime: Less than 1 hour
Impact of the Maintenance and Recommended Mitigation:
- Access to Data and Projects: You will not be able to log into Keboola and access your data or projects during the maintenance period. Any ongoing network connections will be terminated, and you will receive an “HTTP 503 - Service Unavailable” status message. See below for recommended steps to take to avoid any impact to scheduled jobs in Keboola.
- Running Tasks and Orchestrations: Flows (Orchestrations) and running transformations will generally be delayed but not interrupted. Please note that any schedules set to start during the maintenance window will also be delayed.
- Steps to Take: We recommend rescheduling your Saturday orchestrations that are planned for July 20th to avoid any impact from this maintenance. Refer to the maintenance schedule above for exact timings across the stacks affected.
We apologize for any inconvenience this may cause and appreciate your understanding as we work to maintain and improve our service. If you have any questions or need further assistance, please do not hesitate to contact our support team.
Thank you for your patience and cooperation.
2024-07-20 7:00 UTC: We have started the maintenance.
2024-07-20 7:58 UTC: The maintenance is complete, and both stacks are now fully operational.
2024-07-12 13:20 UTC We discovered a problem with Mysql extractor with custom queries, this issue caused PK to be deleted in tables.
The problem has probably occurred since 09:50 UTC.
Update 13:28 UTC We found out that since 09:55 UTC the problem is also related to the Snowflake extractor.
Update 14:08 UTC We found that the mentioned problem started to appear from 09:50 UTC and for the mysql extractor the problem persisted until 12:50 UTC, in the case of the snowflake extractor until 13:30 UTC.In case you do not use custom query, mysql extractor or snowflake extractor this problem does not apply to you.
If you are using custom query and the job ended with an error at the incriminated time or next job ended with an error, in order fix your data you need to switch the extractor to full load and run the job again and contact support.
We are not experiencing any further problems and apologise for the inconvenience.
2024-07-10 12:45 UTC We are experiencing some storage jobs failing with application errors since 12:00 UTC. We are currently investigating the issue and will provide an update within one hour.
2024-07-10 13:45 UTC We are still investigating the issue that is occurring since 12:00 UTC. Our team is working to identify and resolve the problem. We will provide another update within one hour.
2024-07-10 15:00 UTC We have likely identified the root cause of the problem. Our team is preparing to revert the changes that were deployed on the platform earlier today. Once the previous version is deployed, the issue should be resolved. We will provide another update within one hour.
2024-07-10 15:30 UTC The issue has been resolved by deploying the previous version. Jobs created after 15:25 UTC should no longer fail with internal errors. If you observe any issues, please feel free to contact our support team. We apologize for the inconvenience.
2024-07-12 06:50 UTC We are investigating additional potential impacts of the earlier issue. In very rare cases, it may cause table export mismatches between jobs within the same project. This is not confirmed yet. We will contact affected customers if any are identified.