Jobs failures

Several projects may have experienced errors in extractor jobs processing. Database, Zendesk, Google drive and some other extractors were affected. Issue is resolved now and we are investigation the cause of the issue.

We have restarted failed or waiting jobs. We're sorry for any inconvenience! 

Extractors failures

Paymo, Facebook, Facebook Ads and Salesforce extractors were returning curl(60) error in orchestrations from 3 PM - 11PM PST November 6th. Error was caused by invalid SSL certificates.

To finish your tasks, just re-run your orchestrators. We're sorry for any inconvenience! 

Inaccessible Storage API files

Some files were not accessible between 7 PM - 10 PM PST November 4. It caused failures of loads to storage API tables and thus also orchestration failures.

Example of failed orchestration:

It was caused by failed Elasticsearch cluster node. We are still investigating the cause of this issue. However, our whole infrastructure works smoothly at this time. To finish your tasks, just re-run your orchestrators. We're sorry for any inconvenience!

Network errors in GoodData Writer

GoodData Writer experiences occurring network errors during download of csv files from storage and that cause failures of some load data jobs. We are working on fix. Thanks for patience. 

Update: It seems that the network errors ceased. In addition we published a fix which should properly deal with such errors in the future. We apologize for any inconvenience.

Failing GoodData Writer jobs

We are experiencing problems with one of our server causing fails of some GoodData Writer jobs. They typically end with “Job execution error” message without other explanation. We are working on fix. Thanks for your patience.

Update: The problem should be fixed by now.

TAPI-A down

Our main transformation server (TAPI-A) was down between 8:21pm and 8:27pm PST. We had to restart this server to solve connectivity issues. We tried to find all crashed orchestrations and restart them. 

EDIT 8:51pm PST: We’re still having issues with TAPI-A server. Switching all transformations to OVH server.

EDIT 9:45pm PST: The issues persisted on the OVH server. We found a bug in the transformation engine and did a rollback to a previous version. Everything should be operating normally now.

We apologize for this inconvenience, if you find any failed orchestrations, that were not restarted manually, please feel free start them again.