tag:status.keboola.com,2013:/posts Keboola Status 2021-04-29T09:11:26Z Keboola Connection "Data Framework" tag:status.keboola.com,2013:Post/1684872 2021-04-29T09:09:50Z 2021-04-29T09:11:26Z Failing Facebook Ads extractor

Since 2021-04-28 01:00 UTC we are experiencing Facebook ads extractor failures on error "Please reduce the amount of data you're asking for, then retry your request"

It is caused by a bug in Facebook API that have been reported and currently being investigate by a Facebook backend team.

Link to the Facebook bug ticket: https://developers.facebook.com/support/bugs/503443564145524/

We continue to watch the Facebook bug ticket and will update here once we know more.

What can be done now

If you have access to the Facebook bug report you can raise importance/severity of it by leaving a comment there.

One possible workaround that might work is to retrieve data with smallest window possible that is adding .date_preset(yesterday) parameter to the query, e.g:


tag:status.keboola.com,2013:Post/1683926 2021-04-27T07:01:46Z 2021-04-27T07:02:00Z Post-mortem: MSSQL extractor errors This is a post-mortem of the MSSQL extractor errors incident.

We found a root-cause, PHP sorting function is not guaranteed to be stable. It is fixed in PHP 8.0 (https://wiki.php.net/rfc/stable_sorting), but we used 7.4 in the extractor (which is also still supported).

We have learned that in older versions of PHP, a sort function can randomly swap elements with the same value if there are more than 16 values. As the error did not take effect with the lower number of items, our tests did not find it. 

We've fixed the bug and added a tests for sorting more than 16 items.
tag:status.keboola.com,2013:Post/1681901 2021-04-22T15:35:14Z 2021-04-28T07:37:58Z Snowflake transformation errors in Azure North Europe

Since 2021-04-21 10:00 UTC we're seeing increased error rate in Snowflake connections. Users may experience failed Snowflake transformation jobs. We're investigating the root cause. 

UPDATE 18:00 UTC: The last error occurred on 2021-04-22 11:24 UTC and we haven't seen any further failures since then. All operations back to normal. We're in touch with Snowflake support to find the root cause and prevent this from happening in the future. 

We're sorry for this inconvenience and thanks for your understanding.

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1681311 2021-04-21T07:44:29Z 2021-04-21T09:56:40Z Increased API error rate in Azure North Europe

Since 2021-04-20 18:30 UTC we are experiencing increased error rate on all APIs in Azure North Europe. Our engineering team is working to identify the root cause. Next update in 1 hour.

UPDATE 08:45 UTC: We have restarted a faulty container and the situation seems to be stabilised. Next update in 1 hour. 

UPDATE 09:00 UTC: The increased error rate might have caused delays in job processing. 

UPDATE 09:50 UTC: The container does not show any further symptoms of the failure, all operations are back to normal. 

tag:status.keboola.com,2013:Post/1680979 2021-04-20T15:38:39Z 2021-04-27T06:51:04Z Snowflake Extractor Errors

We have discovered that that Snowflake Extractor can be suffering from a similar problem as the MSSQL extractor reported earlier.

The issue affects only configurations using custom query. In case the extracted table contains more then 16 columns, the data may have been swapped within the table columns. The issue was live between 19.4. 14:51 -  20.4. 11:17 UTC. 

The extraction itself does not manifest any error. It can only be discovered later in the pipeline. We're working on finding all affected tables and we'll be contacting you through our support system with possible repair options in case your project is affected.

We sincerely apologize for this problem. A postmortem will be published with the root cause once we resolve the data issues.

April 27, 2021, 06:50 UTC We found out that no job met the conditions for the error to take effect (even though the error was in the code and we have already fixed it).

tag:status.keboola.com,2013:Post/1680898 2021-04-20T07:34:18Z 2021-04-20T11:39:12Z Data loading errors when loading data to workspaces

April 20, 2021, 07:32 UTC We received reports about errors when loading data to workspaces. We're investigating. Next update in 15 minutes or sooner. 

April 20, 2021, 07:48 UTC Only AWS multitenant stacks are affected with the loading errors. We're still investigating. Next update in 15 minutes or sooner.

April 20, 2021, 08:03 UTC Only manual data loads from UI are affected, orchestrations and jobs work normally. Next update in 15 minutes or sooner.

April 20, 2021, 08:30 UTC We are working on a fix. Next update in 1 hour or sooner. 

April 20, 2021, 09:11 UTC The issue is resolved. You need to reload your browser window for the fix to load. 

tag:status.keboola.com,2013:Post/1680882 2021-04-20T05:54:01Z 2021-04-20T10:48:52Z MSSQL extractor errors

Yesterday we have released a new version of MSSQL extractor in which a bug was present with caused failed jobs - data doesn`t end up lining up with the headers.

UPDATE 04:12 UTC: We have rollbacked the previous version. All affected configurations should be working.

We sincerely apologize for the errors. A postmortem reports will follow with further details.

UPDATE 10:47 UTC: We'll be contacting all customers possibly affected by this error through support.

tag:status.keboola.com,2013:Post/1678752 2021-04-14T18:34:31Z 2021-04-14T21:30:28Z Job delays in AWS EU region

Apr 14 21:30 UTC: Backlog is cleared and all operations are back to normal. The incident is now resolved.

Apr 14 20:15 UTC: We mitigated the issue, backlog should be cleared in 20 minutes. We continue to monitor the situation. Next update in one hour or when new information will be available.

Apr 14 19:06 UTC: We've identified the problem and added more capacity for faster backlog processing. Next update in one hour or when new information will be available.

Apr 14 18:30 UTC: We are investigating job delays in AWS EU region. We are working on resolving the situation and keep you posted.

tag:status.keboola.com,2013:Post/1678739 2021-04-14T18:17:17Z 2021-04-14T18:55:17Z Job errors in Azure North Europe region

Apr 14 18:55 UTC We don't see any connectivity failure now, hence we expect this to be resolved. To sum it up, the failing connections happened between 18:00 and 18:15 UTC and caused higher job error rates and job delay in Azure north region.

Apr 14 18:25 UTC: The system seem to back to normal, however we continue to investigate the root cause. The preliminary investigations show there was a temporary network connection failure to one of our metadata databases that caused the increased jobs error rate.

Apr 14 18:15 UTC: We are investigating higher jobs error rate and job delay in Azure north region. We are working on resolving the situation and keep you posted.

tag:status.keboola.com,2013:Post/1676094 2021-04-08T07:39:43Z 2021-04-09T06:35:32Z Recurring jobs processing delays in EU stack in morning hours

The problem with delays and prolongation of jobs processing reported yesterday was happening this morning again. Apparently, the actions we already took were not sufficient. We are investigating the situation and let you know about the resolution.

UPDATE 14:15 CET We analyzed loads of workers processing component jobs and because of some suspicious activity, we restarted their instances to ensure they will be behaving correctly. We are going to monitor the situation the next morning to ensure that jobs performance is not affected.

UPDATE April 9 8:30 CET We were closely monitoring the situation in the morning and the platform seems to be back to normal. Please let us know if you still experienced any unexpected effects.

tag:status.keboola.com,2013:Post/1675739 2021-04-07T14:34:54Z 2021-04-08T05:15:53Z Jobs processing delays in EU stack

Yesterday morning (2021-04-06 ca between 5 and 9 CET) we noticed an overall increased number of jobs in the eu-central region that resulted in prolonged orchestrations or long waiting for jobs starts. The situation repeated today and we decided to add more workers to compensate for that. 

Also, yesterday there was unusual congestion in transformation jobs in the same region but that seems to be an anomaly that did not repeat earlier nor today. 

We are going to monitor the situation in the next days and take other measures if necessary. 

tag:status.keboola.com,2013:Post/1670441 2021-03-25T13:00:38Z 2021-03-25T13:55:31Z Python Workspaces Fail to Run in North Europe stack

Since 2021-03-25 12:30 UTC we are experiencing failures during starting of Python and Python MLflow workspaces in North Europe stack. We are working on resolving the situation and keep you posted.

UPDATE 13:30 UTC: We identified the source of the problem as the lack of hardware resources in the North Europe Azure datacenter. We are working on a switch to another type of computing instances that would not be affected by this shortage.

UPDATE 13:55 UTC: We switched the infrastructure to another type of computing instances and the workspaces are running again. We are still monitoring the situation.

tag:status.keboola.com,2013:Post/1669323 2021-03-22T17:31:18Z 2021-03-23T09:56:29Z Jobs performance degradation in EU and US stack

We are investigating slower jobs performance since Sunday night (2021-03-21), in EU AWS and US AWS stacks. We're seeing the issue only for particular projects and limited number of jobs. Most of the jobs are not affected. 

We are still investigating the causes. If you are experiencing a substantial slowdown of job processing in your project please contact us via the support in the project.

UPDATE 20:28 UTC: We have restarted all job worker nodes in US AWS and EU AWS stacks and monitoring the results. This should have no impact on currently running jobs.

UPDATE March 23, 2021, 10:00 UTC: After the restart all systems are operating normally, we don't see any unusual delays in running jobs. We continue to monitor the situation but unless the situation escalates we consider this incident resolved.

tag:status.keboola.com,2013:Post/1666252 2021-03-15T20:28:48Z 2021-03-16T11:24:01Z Authentication errors across multiple Microsoft services

March 15, 2021, 20:16 UTC We're seeing an authentication errors of multiple Microsoft or Azure related services. This affects all stacks and all regions. In AWS regions only some component jobs are affected (mainly PowerBI writer, OneDrive Excel Sheets writer, OneDrive Excel Sheets extractor). in Azure regions more services may be affected.

You may see application errors or increased job running times.

See https://status.azure.com/en-us/status for more details.

Next update in 60 minutes.

UPDATE March 15, 2021, 21:44 UTC - Engineers at Azure are currently rolling out mitigation worldwide. Full mitigation expected within 60 minutes.

UPDATE March 16, 2021, 0:16 UTC - Engineers at Azure have rolled out a fix to all affected regions for Azure AD. Internal telemetry and customer reports suggests that the error rate for dependent services is rapidly decreasing. Microsoft services are in the process of recovery. Some services may have varying times of recovery following the underlying fix. The next update will be provided at 8:00 UTC.

UPDATE March 16, 2021, 7:52 UTC - The issue related to Azure Active Directory has been resolved. Azure reports problems with authentication to the Storage Service now, but we don't see any impact on Keboola Connection. The next update will be provided at 12:00 UTC.

UPDATE March 16, 2021, 11:23 UTC - Issues on the Azure services are solved. The incident is now resolved.

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1650785 2021-02-07T12:55:14Z 2021-02-07T12:55:14Z Deprecated RTB House extractor failures

We have identified a system wide failure of the deprecated RTB House extractor by Revolt BI (revolt-bi.ex-rtbhouse). The API version used in this extractor was just deprecated on the service side.

Please upgrade your configurations to the new RTB House extractor - https://components.keboola.com/components/revolt-bi.ex-rtb-house-v5

Ondrej Hlavacek
tag:status.keboola.com,2013:Post/1646207 2021-01-28T17:03:13Z 2021-01-28T18:33:58Z Orchestration Notifications incident

2021-01-28 at 17:03 UTC We're investigating an issue with Orchestration notifications in EU and US AWS stacks. You may not be receiving notifications for failed or long running orchestrations.

2021-01-28 at 17:39 UTC Notifications are being sent again.

2021-01-28 at 18:30 UTC The issue is resolved. There were a total of 11 affected orchestrations. We sent tickets to all affected projects (unless the given orchestration was failing regularly).

tag:status.keboola.com,2013:Post/1644792 2021-01-25T10:37:02Z 2021-01-25T11:01:45Z Slowdown of Orchestration processing in Azure North Europe

Beginning with 7:35 UTC 2021-01-25 we're seeing longer queue times of orchestration in Azure North Europe stack. Orchestrations are stuck in waiting phase. We're working on a fix. Next update in 30 minutes.

Update 11:58 UTC 2021-01-25 The issue is now resolved, we apologize for the delays. 

tag:status.keboola.com,2013:Post/1643081 2021-01-21T16:12:37Z 2021-01-26T12:33:38Z Introducing Keboola Community!

We are proud to announce that we have just launched Keboola Community!

Keboola Community is the place where you can find all feature announcements, share advice, submit feedback, and contribute to discussions about Keboola. So from now on, you can find all news about Keboola on the community.keboola.com website.

This page (status.keboola.com) will still be providing updates about the performance status and ongoing incidents of the Keboola Connection platform.

We are looking forward to seeing you on both websites!

tag:status.keboola.com,2013:Post/1643063 2021-01-21T14:28:04Z 2021-01-21T18:21:12Z Investigating higher latency (North Europe Azure)

As of January 21 12:10 UTC we are investigating higher latency on some requests in North Europe Azure stack.

It might leads to errors in the UI.

Job processing is not affected.

UPDATE 2020-01-21 18:00 UTC - All operations are back to normal and everything is fully working.

tag:status.keboola.com,2013:Post/1641605 2021-01-19T11:14:41Z 2021-01-20T08:30:35Z Post-mortem: Failing transformation in EU region

This is a post-mortem of the Failing transformation in EU region incident. 

We received a root-cause analysis from Snowflake and learned that the problem was caused by a DNS resolution issue. The nodes could not reach the NTP service, causing some clock skew and failed jobs for several Keboola customers. Snowflake replaced the nodes that were affected and resolved the issue. 

We learned that even the jobs that did not fail might have produced incorrect results as a result of a few seconds of time skew. Server-time-related functions (current_timestamp(), current_time() and current_date()) returned values that were skewed by a few seconds from the actual time. The data would only be affected if the query modifying it used the above mentioned functions.

We're working on a report of affected queries internally. You can contact us via standard support channels if you think your queries are using server time in a way in which a few seconds of skew would affect the results. We'll work with you on checking the queries as soon as we have the report ready.

tag:status.keboola.com,2013:Post/1642071 2021-01-19T10:47:03Z 2021-01-22T08:54:24Z Investigating decreased Snowflake performance

As of January 15th we are experiencing decreased Snowflake performance in peak times resulting in slower query executions or even query timeouts.

We are actively working on this case with Snowflake support and we shall have a further update at 14:00 UTC.

We are sorry for the inconvenience.

UPDATE 19.1. 14:00 UTC - Snowflake support is continuing with a deep investigation of the problem. Next update should be 20.1. around 8:30 UTC.

UPDATE 20.1. 8:30 UTC - We have boosted the Snowflake Warehouse size to compensate for longer running queries. And we are still waiting for the update from Snowflake support.

UPDATE 20.1. 14:00 UTC - Snowflake is raising the priority of the issue internally, they've acknowledged there's something wrong with the duration of some queries. Next update will be tomorrow (21.1.) morning.

UPDATE 21.1. 13:00 UTC - Snowflake engineering has made some changes (around 4AM UTC) in our account, which they believe should help get performance back on track. We will continue monitoring the case for 24 hours, to see if the changes really helped.

UPDATE 22.1. 9.00 UTC - Both Snowflake engineering and we at Keboola can confirm that queries average times have decreased dramatically over last 24 hours. It seems like the problem has been resolved by Snowflake. We have brought our Snowflake warehouse back to its original size with boost window between 00:00 and 06:30. We will continue to monitor the case for another 24 hours
tag:status.keboola.com,2013:Post/1639559 2021-01-13T10:23:05Z 2021-01-16T07:46:40Z Developer Portal Maintenance Announcement

Maintenance of Keboola Developer Portal (https://components.keboola.com) will take place on Saturday, Jan 16, 2021, at 07:00 UTC and should take less than an hour.

During the maintenance, developers won't be able to access Developer Portal frontend. Connection and components execution won't be affected.

Update 7:45 UTC: The maintenance finished successfully and Developer Portal is up and fully operational.

tag:status.keboola.com,2013:Post/1638529 2021-01-11T08:13:42Z 2021-01-19T11:19:55Z [Resolved] Failing transformation in EU region

Since last night (2021-11-01) we have been experiencing failing transformations with the error "Table already exists in workspace". We continue investigating and will keep you posted.

UPDATE 09:30 UTC - we have identified possible root cause and escalated the problem to Snowflake that seem to cause the errors. The problem is related to clone table input mapping, so workaround might to switch to standard copy table input mapping for the time being.

UPDATE 10:30 UTC - issues still persists, we are waiting for the findings from Snowflake.

UPDATE 11:15 UTC -  We're closely communicating with Snowflake support and we've identified potential cause of the issue that's now shared with the engineering team. Next update in 30 minutes.

UPDATE 11:45 UTC -  We are waiting for the Snowflake engineering team for the update. The issue seems to only be affecting clone mapping queries. We suspect that the issue might be related to a particular unhealthy server somewhere in the Snowflake infrastructure that the query is assigned to. So while we wait for more information from Snowflake's side, our suggestion for a workaround is to try running the affected job/query again if possible. Next update in 30 minutes.

UPDATE 12:15 UTC -  We are still waiting for the Snowflake engineering team for the update. Next update in 30 minutes.

UPDATE 12:45 UTC -  Snowflake informed us that they are still investigating the issue with their engineering team and keep us informed. Next update in 30 minutes.

UPDATE 13:15 UTC - Snowflake informed us they turned off the unhealthy server and asked us to test it. Our testing transformations work so the issue should be resolved now. We wait for Snowflake to confirm it is resolved. However we encourage to run your failed transformations.

UPDATE 14:10 UTC - Snowflake confirmed the issue should be resolved, at the same time continue investigating for the root cause. After more extensive testing we consider this to be resolved, that means all transformations in EU should succeed without any slowdown or temporal fail.

UPDATE 2021-01-19 11:18 UTC - We published post-mortem for this incident

tag:status.keboola.com,2013:Post/1632239 2020-12-26T19:33:54Z 2020-12-26T20:10:13Z Orchestration and transformation errors in EU AWS region

Dec 26, 2020 19:32 - We are investigating higher error rate of orchestrations and transformations in AWS EU region.  Next update in 30 minutes.

Dec 26, 2020 19:40  - We've identified the issue on one of the worker nodes and we triggered the replacement. Everything should be back to normal. We'll keep monitoring our platform closely.

Dec 26, 2020 20:08  - Incident is now resolved. We apologize for the inconvenience caused by this incident.

tag:status.keboola.com,2013:Post/1626404 2020-12-11T08:49:47Z 2020-12-11T08:49:47Z Week in Review - December 11th, 2020

New Components

Updated Components

  • Generic extractor – supports client certificates. This is needed if the server requires two-way SSL authentication, so in addition to the verification of the server, the server also verifies the client.

tag:status.keboola.com,2013:Post/1623577 2020-12-07T08:07:39Z 2020-12-07T08:07:39Z Week in Review - December 7th, 2020

General updates

  • New wishlist – the old wishlist was replaced with a new, more transparent one. You can find it at https://ideas.keboola.com or in the "Settings Menu" in your projects.
  • New Keboola Connection instance running on Azure is available in Europe – https://connection.north-europe.azure.keboola.com
  • SQLdep removed – we were forced to remove SQLdep from Keboola Connection as the service is shutting down.
  • New on-boarding experience - starting from November 16, 2020, new users will no longer get Guide Mode projects. They were replaced by a new interactive wizard inside each project.
  • Migration of projects to Azure stacks – it's now possible to migrate projects in AWS stacks to new Azure stacks [see the documentation].
  • Shared workspaces – this feature is available only to customers with new transformations.
If you share a workspace, all project users will be able to use/restore it. Only workspace creators/owners can delete workspaces. 
In the list of workspaces, there’s a new column indicating if a workspace is yours and if it is shared or not.

New Components

  • Google Ad Manager Report Service exctractor – downloads reports from the Google Ad Manager API. It uses ReportService.ReportQuery to download data.
  • Generic writer – sends data to REST-like APIs either as JSON or as a BINARY payload.
  • B!Q GoodData Telemetry extractor – uses the GoodData API to get info about users, projects, dashboards, reports, and uploaded datasets.
  • Klaviyo extractor – fetches customer profiles, lists, campaigns, and metrics from the Klaviyo email marketing platform.
  • Klaviyo writer – creates/updates customer profiles and manages a list of memberships and subscriptions on the Klaviyo email marketing platform.
  • Azure Blob Storage V2 extractor – extracts a single or multiple CSV files from a single or nested folders from an Azure Blob Container and stores them into Keboola Connection.
  • Looker Schedule Runner – forces a new cache on your dashboards by triggering a dashboard export, which will execute a query and refresh the cache of your dashboard.

Updated Components

  • FTP extractor – has a new option “Ignore passive address”. It solves the problem when a server offers a random port for communication but the port is blocked by firewall.

UI Improvements

  • Input Mapping – data types are available for the Snowflake transformation. This feature is available in new PAYG projects and for customers with new transformations only [see the documentation].
  • Option validate and SQLdep removed

  • Transformations have been split into Transformations and Workspaces (applies to new transformations only).

tag:status.keboola.com,2013:Post/1621133 2020-11-25T15:55:48Z 2020-11-26T10:43:12Z Component synchronization down in all regions

Since 2020-11-25 13:45 UTC we have noticed an unfunctional synchronization of components between Keboola Connection and the Developer portal. The problem concerns AWS in the US region. Working in Keboola Connection is not affected by this error. It has an impact on the release and synchronization of components. You also cannot log in to the Developer Portal.

We are going to monitor the situation and keep you posted.

Update Nov 25, 2020 16:11 UTC - We have identified a problem. The outage is on the AWS side.

Update Nov 25, 2020 16:19 UTC - We found that the problem affects both AWS and Azure in all regions. Next update in 30 minutes.

Update Nov 25, 2020 16:59 UTC - Outage on the AWS side persists, see the status page. We continue to monitor the situation. Next update in 1 hour.

Update Nov 25, 2020 18:10 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.

Update Nov 25, 2020 20:15 UTC - Outage on the AWS side persists. We continue to monitor the situation. Next update in 2 hours.

Update Nov 25, 2020 22:17 UTC - Issues on the AWS side are reducing. Login to the Developer Portal is already working and we are monitoring successful component synchronizations across all regions. Next update on Nov 26, 2020 06:30 UTC 

Update Nov 26, 2020 07:05 UTC - Work on the AWS side to troubleshoot still continues, but the error rate is reduced. We are no longer experiencing any issues with component synchronizations or other Keboola Connection services. We continue to monitor the situation. Next update in 4 hours.

Update Nov 26, 2020 10:23 UTC - Issues on the AWS services are solved. The incident is now resolved.

Václav Eder
tag:status.keboola.com,2013:Post/1618854 2020-11-19T09:02:53Z 2020-11-19T09:11:14Z Investigating Sandbox API issues in Azure regions

We're investigating an issue with Sandbox API in our Azure regions. Next update in 5 minutes. 

Update Nov 19, 2020 09:03 UTC - Configuration issue caused the release of Sandbox API today at 8:23 UTC that did not work properly. The problem was noticed and rollback began. At 8:58 UTC the release was rolled back. We're verifying that the issue is resolved. 

Update Nov 19, 2020 09:10 UTC - We verified that the rollback restored the API to a fully working condition. The incident is now resolved. 

tag:status.keboola.com,2013:Post/1618828 2020-11-19T07:04:14Z 2020-11-19T07:31:53Z Configuration editor broken across all regions

We're investigating problems with configuration editor. 

Update Nov 19, 2020 07:00 UTC - Configuration editor is broken across all regions. We're working on a fix that should be ready shortly. Sorry for inconvenience. Next update in 30 minutes. 

Update Nov 19, 2020 07:19 UTC - Change causing the issue has been rolled back and we're in a process of verification that the issue is resolved. Next update in 30 minutes. 

Update Nov 19, 2020 07:28 UTC - We verified that the change causing the issue is rolled back on all regions and the configuration editor is working normally. If you're still experiencing issues with the editor, please contact support individually using support button in your project. 

tag:status.keboola.com,2013:Post/1614860 2020-11-11T07:35:58Z 2020-11-15T08:40:55Z Azure EU Maintenance Announcement

A maintenance of Azure EU Keboola Connection (https://connection.north-europe.azure.keboola.com/) will take place on Sunday, Nov 15th, 2020 from 08:00 UTC  and should take less than two hours.

During the maintenance, you can't access your projects. Orchestrations and running component jobs should be generally delayed, but not interrupted.  However, feel free to re-schedule your sunday's orchestrations to avoid this maintenance window.

Update Nov 15, 2020 08:00 UTC  - The maintenance started.

Update Nov 15, 2020 08:36 UTC  - The maintenance is over. Azure EU Keboola Connection is up and fully operational.