Speech and visual representations provide complimentary information about observable behaviour. However, it is important to evaluate the model on unseen actors to ensure generalisation. This work focuses on the fusion of speech and facial features to predict an artificially constructed affective stress risk proxy. We used 480 data-points from 8 actors, each containing 39 acoustic features extracted from speech and 1280 features extracted from CNNs for facial features. The proxy is continuous between 0 and 100, and therefore split into Low Proxy and At-Risk Proxy classes at a threshold of 30. These labels were constructed heuristically.

The main experiment was performed using an 8-fold leave-one-actor-out cross-validation. Using concatenated features logistic regression achieved 86.46\% accuracy, 90.93\% F1-score and a ROC-AUC of 0.8874. A separate experiment was performed to create a speaker invariant representation, this achieved a maximum accuracy of 74.79\% and F1-score of 85.33\% for both face-invariant and joint-invariant configurations. However, this resulted in all At-Risk Proxy samples being predicted as positive, and therefore had low specificity. When evaluated on a random sample basis, accuracy was 98.96\%. However, all 8 actors were present in the training and test set, therefore does not demonstrate generalisation.

These results demonstrate that the model generalisation depends on the type of representation and evaluation performed. The importance of looking at metrics per class and per actor has been highlighted. Leakage must also be considered, and care should be taken when interpreting the results. These results show the ability to predict an artificial affective proxy.

Multimodal Learning,Speech Acoustics,Facial Representation,Feature Fusion,Leave-one-actor-out evaluation,Affective stress-risk proxy