Diploma in Multimodal Model Evaluation and Robustness
Online postgraduate diploma of 800 hours in multimodal model evaluation and robustness: 8 modules covering multimodal models, evaluation design, performance according to ISO/IEC TS 4213, robustness according to ISO/IEC 24029, adversarial attacks, bias and hallucinations, and lifecycle evaluation, with an assigned technical tutor.
Take the controls.
Run a real engineering trade-off from this field. Move the sliders, watch the model react in real time and beat the challenge.
What you will be able to do
Describe the architectures and tasks of multimodal models.
Design evaluation protocols with suitable metrics and benchmarks.
Measure performance with confidence intervals and by subgroup.
Assess robustness to corruptions and distribution shift.
Carry out adversarial testing and multimodal red teaming.
Analyse bias, hallucinations and interpretability.
Integrate evaluation into the regulated model lifecycle.
Who it's for
- AI evaluation engineers designing benchmarks and test protocols.
- Robustness researchers studying how models fail under perturbation.
- AI security specialists running adversarial tests and red teaming.
- Data scientists who need rigorous evidence of model performance.
- Model quality engineers responsible for release and regression testing.
- Technical AI auditors reviewing models against standards and regulation.
- Engineering or science graduates looking to move into the technology sector.
What you'll work with
- ISO/IEC TS 4213:2022 (assessment of machine learning classification performance)
- ISO/IEC 24029-2:2023 (assessment of the robustness of neural networks using formal methods)
- ISO/IEC TR 24029-1:2021 (overview of neural network robustness assessment)
- ISO/IEC TR 24027:2021 (bias in AI systems and AI-aided decision making)
- ISO/IEC 25059:2023 (quality model for AI systems)
- Regulation (EU) 2024/1689 on Artificial Intelligence (EU AI Act)
- NIST AI 100-2 E2023 (taxonomy and terminology of adversarial machine learning)
- NIST AI RMF 1.0 (AI Risk Management Framework)
- lm-evaluation-harness
- Hugging Face Evaluate
- Adversarial Robustness Toolbox
- Captum
- Giskard
8 modules
1.1Multimodal fusion architectures1.2Vision-language models CLIP and BLIP1.3Large multimodal models1.4Audio and text models1.5Video models1.6Classification, retrieval and generation tasks1.7Pre-training and fine-tuning1.8Zero-shot and few-shot1.9Industrial uses of multimodal models1.10Workshop: catalogue of multimodal tasks for a real case
2.1Evaluation objectives and scope2.2Benchmarks MMMU, VQAv2 and COCO Captions2.3Benchmark contamination2.4Custom evaluation sets2.5Accuracy and recall metrics2.6Generation metrics BLEU, CIDEr and CLIPScore2.7Human evaluation and protocols2.8Models as evaluators2.9Statistical significance of differences2.10Practical: evaluation plan for a visual assistant
3.1Confusion matrix and derived metrics3.2Calibration and reliability of probabilities3.3Performance by subgroup3.4Bootstrap confidence intervals3.5Model comparison3.6Decision thresholds3.7Evaluation with class imbalance3.8Performance report3.9Reproducibility of evaluations3.10Lab: evaluation of a vision-language classifier
4.1Robustness to natural corruptions4.2ImageNet-C and similar benchmarks4.3Noise in audio and text4.4Missing or degraded modality4.5Misalignment between modalities4.6Distribution shift4.7Assessment according to ISO/IEC 24029-24.8Formal methods for neural networks4.9Multimodal metamorphic testing4.10Lab: corruption tests with a missing modality
5.1Adversarial examples in images5.2Cross-modal attacks5.3Prompt injection in images5.4Jailbreaks in multimodal models5.5Data poisoning5.6Model extraction and inversion5.7Defences and adversarial training5.8Structured red teaming5.9NIST taxonomy of attacks5.10Practical: red teaming of a vision-language model
6.1Social biases in multimodal models6.2Measurement with ISO/IEC TR 240276.3Visual hallucinations and how to measure them6.4Grounding in visual evidence6.5Attention maps and Grad-CAM6.6Cross-modal attribution6.7Explanations for users6.8Limitations of interpretability6.9Bias mitigation6.10Case study: hallucinations in the captioning of technical images
7.1The ISO/IEC 25059 quality model7.2Requirements of the AI Act7.3Pre-deployment evaluation7.4Monitoring in production7.5Continuous evaluation and regressions7.6Automated evaluation harnesses7.7Documentation with model cards7.8Risk management of the evaluation7.9Communicating results7.10Case study: evaluation report for a risk committee
8.1Selection of the model and use case8.2Protocol design8.3Custom evaluation set8.4Performance and calibration metrics8.5Robustness campaign8.6Red teaming and security8.7Bias and hallucination analysis8.8Evaluation report8.9Report and presentation8.10Final project: final review with the tutor
How you'll learn
100% online training on the SEIUM campus, with downloadable material, recorded classes and an assigned technical tutor. There are no fixed timetables: you progress at your own pace within the intake, so you can combine it with your job.
Each module combines theory, guided reading of the original standard and applied exercises. The working software is lm-evaluation-harness, Hugging Face Evaluate, Adversarial Robustness Toolbox, Captum and Giskard, in free, academic or evaluation versions. Practical work is based on real procedures for evaluating vision-language models with custom datasets, robustness testing against corruptions and missing modalities, and multimodal red-teaming exercises. Assessment is continuous by module, plus the final project.
Standards covered: ISO/IEC TS 4213:2022, ISO/IEC 24029-2:2023, ISO/IEC TR 24029-1:2021, ISO/IEC TR 24027:2021, ISO/IEC 25059:2023, Regulation (EU) 2024/1689 on Artificial Intelligence, NIST AI 100-2 E2023, NIST AI RMF 1.0.
Build it end to end
Your final project follows the real engineering workflow, step by step, reviewed by your tutor.
Where this takes you
Requirements
- Prior technical training or professional experience in the sector. No university degree is required.
- An internet connection and a computer capable of running the programme software.
- English sufficient to read technical standards.
Ready to start?
Send your application today. Our admissions team replies within 72 hours with your study plan and funding options.
Frequently asked questions
The programme totals 800 hours over 6–8 months. You set your own pace within each intake, and your tutor helps you plan the workload around your job.
You work with lm-evaluation-harness, Hugging Face Evaluate, Adversarial Robustness Toolbox, Captum and Giskard, using free, academic or evaluation versions. Your tutor indicates the free alternative wherever one exists.
A SEIUM Institute diploma, awarded on passing the continuous assessment and the final project. It is SEIUM's own qualification, not an official state degree; its recognition depends on the programme completed and the regulations of each country.
Yes. Modules passed are recognised in a higher programme in the same area (postgraduate diploma or master's). Ask about the recognition policy when you enrol.
Prior technical training or professional experience in the sector (no university degree required), a computer able to run the programme software and English sufficient to read technical standards.
Start dates are set by SEIUM's faculty for each intake. Our admissions team will confirm the next available date when you apply.
Related programmes
Diploma in Spatiotemporal Telemetry and Risk Maps
Online postgraduate diploma of 800 hours in spatio-temporal telemetry and risk maps. 8 modules covering spatio-temporal data, telemetry and ingestion, trajectory analysis, spatial statistics, risk modelling,…
Diploma in Roadmap, Data SLAs and Service Contracts
Online postgraduate diploma of 800 hours in data roadmaps, data SLAs and service contracts, with 8 modules covering data product management, data roadmaps, data service levels,…
Diploma in Production Deployment of Solvers and Optimization APIs
Online postgraduate diploma of 800 hours in putting solvers and optimisation APIs into production. 8 modules covering optimisation-as-a-service architecture, model engineering, asynchronous API design, containerised deployment…
More in Data, AI, Simulation & Analytics
All Data, AI, Simulation & Analytics programmes → · How to apply · Fees & financing · Scholarships