Keboola Status

The current environment for transformations is fairly outdated (due to compatibility concerns) and needs an update. New transformation environments are now available with an update plan. You may choose to leave your environment in auto update mode or lock your environment to either the current or the future version.

Action needed:
If you want to prevent the new transformations from being updated automatically later this month, you must select the version you wish to use. This update is planned for June 22, 2021.

The following options are available:

• Leave your setting as is: “1.4.1 - Python 3.8.5 + Pandas 0.25.3 (updates automatically)”. The transformations will update automatically to Python 3.9.5 + Pandas 1.2.4 on June 22, 2021. We recommend choosing this option if you are not relying on Pandas, as it is very likely that your transformations will work without any changes.
• Lock your transformation environment to the current version by selecting “0.3.3 - Python 3.8.5 + Pandas 0.25.3 (locked)”. The update on June 22 will not affect the transformation.
• Lock your transformation to the latest future version: “Python 3.9.5 + Pandas 1.2.4 (locked)”. The update on June 22 will not affect the transformation.
• If you do not want to jump versions, you can choose to lock your transformation to an intermediate version: “Python 3.8.10 + Pandas 1.1.5 (locked)”. The update on June 22 will not affect the transformation.

The Sandbox and Workspace environments will keep running the current version until the switch date on June 22, 2021, at which point they will switch to the latest version (we're jumping the intermediate Python 3.8.10 + Pandas 1.1.5). If you want to continue to use the current (outdated) version of Pandas in the Sandbox/Workspace after this date, you'll have to install it manually.

In the future, we will also be offering at least two versions of the Python transformations: one that updates automatically (the default and current behavior) and one that does not auto-update. That way you can always choose whether your transformation environment is auto-updating or not.

We're seeing increased user error rate in Python transformation jobs since May 25 16:08 UTC. Most common error message is "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed." but also other errors may appear.

We're sorry for this inconvenience, we're actively investigating the issue. Next update in 30 minutes

Update 07:28 UTC: We have rolled back Python transformations from version 1.4.1 to version 1.4.0 and everything seems to be working again. If you encounter further issues please get in touch with us. We'll be monitoring this issue and post an update in one hour. 

Update 08:36 UTC: The situation is stable, we're not seeing any further errors since the rollback. Please accept our apology for the inconvenience. 

From 2021-05-18 21:13 UTC to 2021-05-19 05:06 UTC we experienced increased jobs failures in the Azure North Europe region.

These errors were caused by a faulty deploy on a single node. The deploy has been fixed and the situation is back to normal and you can restart the failed jobs.

We're sorry for this inconvenience. 

2021-05-10 13:25 CET

We have noticed a slight increase in job failures for some components today since this morning's release of the job runner. 
We are investigating the root cause of the issue and will update when more information becomes available.

2021-05-10 14:35 CET

All systems are back to fully operational status.  We are continuing to monitor for further instances of this error, and we are working on a preventative measures plan to reduce the impact of this type of incident in the future.  

Since 2021-04-28 01:00 UTC we are experiencing Facebook ads extractor failures on error "Please reduce the amount of data you're asking for, then retry your request"

It is caused by a bug in Facebook API that have been reported and currently being investigate by a Facebook backend team.

Link to the Facebook bug ticket: https://developers.facebook.com/support/bugs/503443564145524/

We continue to watch the Facebook bug ticket and will update here once we know more.

What can be done now

If you have access to the Facebook bug report you can raise importance/severity of it by leaving a comment there.

One possible workaround that might work is to retrieve data with smallest window possible that is adding .date_preset(yesterday) parameter to the query, e.g:

insights.action_attribution_windows(28d_click).action_breakdowns(action_type).level(adset).date_preset(yesterday).time_increment(1)

We found a root-cause, PHP sorting function is not guaranteed to be stable. It is fixed in PHP 8.0 (https://wiki.php.net/rfc/stable_sorting), but we used 7.4 in the extractor (which is also still supported).

Since 2021-04-21 10:00 UTC we're seeing increased error rate in Snowflake connections. Users may experience failed Snowflake transformation jobs. We're investigating the root cause. 

UPDATE 18:00 UTC: The last error occurred on 2021-04-22 11:24 UTC and we haven't seen any further failures since then. All operations back to normal. We're in touch with Snowflake support to find the root cause and prevent this from happening in the future. 

We're sorry for this inconvenience and thanks for your understanding.

Since 2021-04-20 18:30 UTC we are experiencing increased error rate on all APIs in Azure North Europe. Our engineering team is working to identify the root cause. Next update in 1 hour.

UPDATE 08:45 UTC: We have restarted a faulty container and the situation seems to be stabilised. Next update in 1 hour. 

UPDATE 09:00 UTC: The increased error rate might have caused delays in job processing. 

UPDATE 09:50 UTC: The container does not show any further symptoms of the failure, all operations are back to normal. 

We have discovered that that Snowflake Extractor can be suffering from a similar problem as the MSSQL extractor reported earlier.

The issue affects only configurations using custom query. In case the extracted table contains more then 16 columns, the data may have been swapped within the table columns. The issue was live between 19.4. 14:51 -  20.4. 11:17 UTC. 

The extraction itself does not manifest any error. It can only be discovered later in the pipeline. We're working on finding all affected tables and we'll be contacting you through our support system with possible repair options in case your project is affected.

We sincerely apologize for this problem. A postmortem will be published with the root cause once we resolve the data issues.

April 27, 2021, 06:50 UTC We found out that no job met the conditions for the error to take effect (even though the error was in the code and we have already fixed it).

April 20, 2021, 07:32 UTC We received reports about errors when loading data to workspaces. We're investigating. Next update in 15 minutes or sooner. 

April 20, 2021, 07:48 UTC Only AWS multitenant stacks are affected with the loading errors. We're still investigating. Next update in 15 minutes or sooner.

April 20, 2021, 08:03 UTC Only manual data loads from UI are affected, orchestrations and jobs work normally. Next update in 15 minutes or sooner.

April 20, 2021, 08:30 UTC We are working on a fix. Next update in 1 hour or sooner. 

April 20, 2021, 09:11 UTC The issue is resolved. You need to reload your browser window for the fix to load.