Diploma in Stateful and Exactly-Once Pipeline Design
Online postgraduate diploma of 800 hours in stateful and exactly-once pipeline design. 8 modules covering processing semantics, state management, checkpoints and recovery, end-to-end transactions, event time and production operation, with an assigned technical tutor.
Take the controls.
Run a real engineering trade-off from this field. Move the sliders, watch the model react in real time and beat the challenge.
What you will be able to do
Understand delivery and processing semantics in distributed systems.
Manage keyed state efficiently and in an evolvable way.
Configure checkpoints and savepoints for reliable recovery.
Implement end-to-end exactly-once guarantees.
Handle out-of-order and late data using event time.
Design pipelines that are robust to failures and reprocessing.
Operate stateful applications in production.
Who it's for
- Streaming data engineers responsible for pipelines that must never lose or duplicate events.
- Distributed systems engineers dealing with failures, consistency and recovery.
- Flink developers who want to master state, checkpoints and transactional sinks.
- Real-time data architects designing end-to-end guarantees.
- Site reliability engineers (SRE) operating stateful streaming jobs.
- Backend data engineers integrating Kafka with databases and services.
- Engineering or science graduates looking to move into manufacturing.
What you'll work with
- Apache Kafka KIP-98 (Exactly Once Delivery and Transactional Messaging)
- Apache Kafka KIP-447 (Producer scalability for exactly once)
- Apache Flink FLIP-143 (Unified Sink API)
- CloudEvents 1.0 (CNCF event specification)
- ISO/IEC 25010:2023 (software product quality model)
- IEC 62443 (security of industrial automation and control systems)
- Apache Flink
- Apache Kafka
- RocksDB
- Flink Kubernetes Operator
- Kafka Streams
8 modules
1.1Guarantees: at most once, at least once and exactly once1.2End-to-end exactly once1.3Idempotence and deduplication1.4Processing determinism1.5Partial failures in distributed systems1.6CAP theorem and consistency1.7Clocks, ordering and causality1.8Google's Dataflow model1.9Costs of strong guarantees1.10Workshop: failure analysis of a simple pipeline
2.1Keyed state and operator state2.2State types in Flink2.3State backends: memory and RocksDB2.4Timers and state TTL2.5State size and its growth2.6Serialisation and state schema evolution2.7Broadcast state2.8Querying state2.9State anti-patterns2.10Lab: stateful function with timers in Flink
3.1Chandy-Lamport algorithm3.2Aligned and unaligned checkpoint barriers3.3Incremental checkpoints with RocksDB3.4Savepoints and application upgrades3.5Restart strategies3.6Task-local recovery3.7Recovery time and its measurement3.8Checkpoint storage3.9Tuning the checkpoint interval3.10Practical exercise: recovery after failure with savepoints
4.1Kafka idempotent producer4.2Kafka transactions4.3Reading with read_committed isolation4.4Two-phase commit protocol in Flink4.5Transactional sinks in Flink4.6Idempotent writes to databases4.7Transactional outbox pattern4.8Kafka Streams with exactly_once_v24.9Limitations and edge cases4.10Lab: exactly-once Kafka-Flink-Kafka pipeline
5.1Watermarks and their generation5.2Partition idleness5.3Late data and side outputs5.4Stateful windows and triggers5.5Recalculation and retractions5.6Stateful temporal joins5.7Deduplication with windows5.8Testing with out-of-order data5.9Clocks of industrial devices5.10Case study: deduplication of sensor events
6.1Partitioning and keys for state6.2Rebalancing hot keys6.3Scaling with state rescaling6.4Backpressure and its diagnosis6.5Error handling and dead-letter queues6.6Reprocessing and historical backfill6.7Versioning of business logic6.8Unit testing with the Flink test harness6.9Chaos testing6.10Practical exercise: historical reprocessing without duplicates
7.1Deployment with Flink Kubernetes Operator7.2Checkpoint and state metrics7.3Lag and backpressure alerts7.4Upgrades with savepoints7.5Resource and memory management7.6Cost of state and storage7.7Security of data held in state7.8Incident runbooks7.9Post-mortems of streaming incidents7.10Workshop: job recovery runbook
8.1Use case and required guarantees8.2Flow and state design8.3Implementation of the processing8.4Transactional integration of the pipeline8.5Failure and recovery testing8.6Pipeline performance testing8.7Pipeline operation plan8.8Evaluation of pipeline results8.9Report and presentation8.10Final project: final review with the tutor
How you'll learn
Training is 100% online on the SEIUM campus, with downloadable material, recorded classes and an assigned technical tutor. There are no fixed timetables: you progress at your own pace within the intake, so the programme fits around your job.
Each module combines theory, guided reading of the original specification and applied exercises. The working software is Apache Flink, Apache Kafka, RocksDB, Flink Kubernetes Operator and Kafka Streams, in free, academic or evaluation versions. Exercises are based on real procedures: fault injection and verification that no duplicates occur, recovery of applications from savepoints, and testing with out-of-order and late events. Assessment is continuous by module, plus the final capstone project.
Standards covered: Apache Kafka KIP-98, Apache Kafka KIP-447, Apache Flink FLIP-143, CloudEvents 1.0, ISO/IEC 25010:2023, IEC 62443.
Build it end to end
Your final project follows the real engineering workflow, step by step, reviewed by your tutor.
Where this takes you
Requirements
- Prior technical training or professional experience in the sector. No university degree is required.
- An internet connection and a computer capable of running the programme software.
- English sufficient to read technical standards.
Ready to start?
Send your application today. Our admissions team replies within 72 hours with your study plan and funding options.
Frequently asked questions
The programme totals 800 hours over 6–8 months. You set your own pace within each intake, and your tutor helps you plan the workload around your job.
You work with Apache Flink, Apache Kafka, RocksDB, Flink Kubernetes Operator and Kafka Streams, using free, academic or evaluation versions. Your tutor indicates the free alternative wherever one exists.
A SEIUM Institute diploma, awarded on passing the continuous assessment and the final project. It is SEIUM's own qualification, not an official state degree; its recognition depends on the programme completed and the regulations of each country.
Yes. Modules passed are recognised in a higher programme in the same area (postgraduate diploma or master's). Ask about the recognition policy when you enrol.
Prior technical training or professional experience in the sector; no university degree is required.
Start dates are set by SEIUM's faculty for each intake. Our admissions team will confirm the next available date when you apply.
Related programmes
Diploma in Root Cause Analytics and Data-Based Diagnostics
Online postgraduate diploma of 800 hours in root cause analytics and data-driven diagnosis. 8 modules covering classical root cause analysis, diagnostic data preparation, multivariate statistics, explainable…
Diploma in Partial Twins and Degradation Assessment
Online postgraduate diploma of 800 hours in partial twins and degradation assessment. 8 modules covering physical, reduced-order and hybrid degradation models, Bayesian data assimilation, remaining useful…
Diploma in Electrical Asset Twins and DERMs
Online postgraduate diploma of 800 hours in electrical asset digital twins and DERMS: 8 modules covering grid assets, asset modelling with CIM (IEC 61970/61968), asset digital…
More in Data, AI, Simulation & Analytics
All Data, AI, Simulation & Analytics programmes → · How to apply · Fees & financing · Scholarships