Computer Vision & Biosignals

🧠 ELITE - Competency Analysis and Prediction System

Comprehensive system for analysing and predicting cognitive, socioemotional, and leadership competencies through multimodal data integration combining eye-tracking, facial expression recognition, audio analysis, and neuropsychological assessment. Evidence-based platform enabling objective assessment of leadership style and ability through independent virtual environments and serious games with implicit measures.

• Grant: CPP2021-008569 funded by MICIU/AEI/10.13039/501100011033 and European Union NextGenerationEU/PRTR
• Approach: Practical, realistic, evidence-based methodology for objective competency evaluation through hidden assessment (implicit measures) in serious games within virtual environments
• Module 1 (Cognitive): Processing speed, attention (divided/selective), creative thinking, self-confidence, emotional self-regulation through eye-tracking and temporal markers
• Module 2 (Socioemotional): Verbal/non-verbal/paraverbal communication, empathy, assertiveness, active listening through facial expression analysis and audio features
• Module 3 (Neuropsychological): Impulsivity (BIS), cognitive flexibility (WISC), planning (TOWERS), attention (D2), reasoning (MAT), creativity (KEY), working memory (DIGIT)
• Module 4 (Audio/Psychometric): Assertiveness (RATHUS), transformational leadership (MLQ), empathy (TECA) through advanced speech signal processing
• Technologies: Python 3.12, PyTorch, scikit-learn, XGBoost, librosa for audio processing, real-time eye-tracking analysis, independent virtual environments per competency
• Machine Learning: Ensemble models with feature extraction pipelines, trained classification models achieving multi-class prediction (HIGH/MEDIUM/LOW competency levels), clustering algorithms for behaviour prediction
• Deployment: Standalone executable via PyInstaller for cross-platform distribution, modular architecture with independent competency evaluators
• Innovation: Multimodal fusion framework integrating visual (eye-tracking), audio (prosody, acoustic features), and behavioural data for holistic competency assessment through serious games
• Output: Automated performance reports and structured JSON classifications based on validated psychometric scales and neuropsychological tests, behaviour prediction algorithms

💶 NeuroCash - Multimodal Analysis of Banknote Design Perception

Academic research project analysing cognitive and emotional responses to banknote designs through multimodal data integration. Commissioned by Banco de España, the system combines eye-tracking, gesture recognition, facial coding, speech analysis, and psychometric questionnaires to understand user perception and interaction with currency aesthetics. Winner of UPV Excellence Award for Knowledge Transfer 2025.

• Multimodal Data Sources: Eye-tracking (heatmaps, scanpaths, attention metrics), gesture recognition (flip, tilt, rotate, rub), facial expression analysis (emotion detection: happiness, anger, sadness), speech transcription and sentiment analysis, standardised questionnaire responses
• Research Questions: Impact of demographics (age, gender, education, cash usage) on gaze patterns and gestures, correlation between design ratings and behavioural responses, emotional triggers by different banknote designs, hypothesis testing (e.g., rotation behaviour for vertical vs. non-vertical designs)
• Technologies: Python 3.x, eye-tracking hardware integration, facial coding libraries, NLP modules for sentiment extraction, statistical analysis frameworks
• Segmentation Analysis: Demographic-based segmentation (gender, age groups, education levels, cash usage frequency), rating-based grouping (high vs. low scores), gesture frequency and type analysis per banknote design
• Visualisation Pipeline: Customisable plotting functions for gesture distributions, facial expression timelines, attention heatmaps, survey result comparisons, statistical significance testing
• Data Processing: Modular scripts for gesture loading, heatmap analysis, facial coding processing, speech transcription, questionnaire data integration
• Innovation: Holistic approach combining behavioural (gestures), physiological (gaze, facial expressions), linguistic (speech sentiment), and self-report data (questionnaires) for comprehensive currency design evaluation
• Impact: Provides evidence-based insights for Banco de España on banknote design optimisation, accessibility, and user experience across diverse demographics

🎯 PICTURE - VR-based Platform for Leadership Assessment (CPP2021-008517, 2022-2025)

Public-Private Partnership: LabLENI (UPV) + HGBS CONSOURCING, S.L | Funded by: MICIU/AEI/10.13039/501100011033 + European Union NextGenerationEU/PRTR

VR-based comprehensive assessment platform utilising multimodal machine learning to evaluate horizontal competencies (decision-making, interpersonal abilities) through implicit measures in hyper-realistic virtual environments. The system captures eye-tracking, facial expressions, voice analysis, psychophysiological signals, and behavioural data to provide objective leadership and cognitive-emotional profiling.

• VR Assessment Context: Hyper-realistic virtual scenarios for evaluating horizontal competencies: critical thinking, decision-making, interpersonal communication, emotional regulation, assertiveness, empathy, strategic planning
• Implicit Measures Integration: Eye-tracking metrics (AOI visits, fixation patterns, gaze transitions), voice features (pitch, intensity, jitter, shimmer, HNR, speech rate), facial expression analysis, psychophysiological signals, behavioural interaction logs
• VR Activities: Character selection & CV/video interactions, ZOO spatial planning puzzle (cognitive performance), assertiveness negotiation tasks, presentation scenarios, leadership interview simulations with Big Five personality assessment (N/E/O/A/C)
• Competencies Evaluated: Critical thinking (F1: 0.640, 80% accuracy), strategic planning, attachment style (embeddings-based), assertiveness, emotional self-regulation, empathy (F1: 0.739, 74% accuracy via presentation ET), Big Five personality traits
• Machine Learning Pipeline: RandomForestClassifier, XGBoost, GradientBoosting, AdaBoost, SVC, LogisticRegression, custom neural networks for embedding-based classification, trait-specific model optimization
• Technologies: Unity VR platform, Python 3.12, PyTorch, scikit-learn, librosa (audio analysis), RoBERTa (NLP embeddings), Tobii eye-tracking integration, modular feature extraction pipeline
• Advanced Features: Cosine similarity loss for attachment embeddings, automatic category mapping and normalisation, multimodal fusion (gaze + voice + behaviour), fine-tuned sentence transformers for verbal interaction analysis
• Data Architecture: Standardised user data structure (Activities/, EyeTrackingData/), automated transcript processing, feature consolidation across modalities, model versioning with .pkl serialisation
• Performance Highlights: Critical thinking (80% accuracy), empathy via presentation eye-tracking (74% accuracy), attachment style via NLP embeddings (62% accuracy), Big Five personality (38-55% accuracy, 3-class problem)
• Innovation: First integrated VR platform combining implicit measures (physiological, gaze, voice) with behavioural and linguistic features for objective leadership competency assessment, eliminating self-report bias through hyper-realistic immersive context

📄 Affective Robot Story-telling Human-Robot Interaction

Exploratory real-time emotion estimation analysis using facial expressions and physiological signals (EEG, BVP, GSR) in social robot interaction.

📄 Real-time Facial Expression Recognition using Smoothed Deep Neural Network Ensemble

Real-time facial expression recognition system using CNN ensemble with label smoothing. State-of-the-art on FER2013, SFEW 2.0, and ExpW (13.48 ms on GPU).

📄 Real-time Emotional Recognition for Sociable Robotics Based on Deep Neural Networks Ensemble

Emotional recognition system for social robots integrating YOLO facial detection and CNN ensemble for real-world interactions. IWINAC 2019.