Real-time driver attention monitoring with closed-loop vehicle actuation, using only a webcam.
A hierarchical vision pipeline (MediaPipe Face Mesh → explainable heuristics → TCN-Attention network) decides whether the driver is attentive, drowsy or distracted, and a simulated car responds: it limits speed, slows down for microsleeps, and applies autonomous emergency braking.
Python · OpenCV · MediaPipe · PyTorch → NumPy · FastAPI · WebSockets · Canvas · ReportLab
| 👁 Multi-cue biometrics | Eye Aspect Ratio, iris gaze vector, head yaw/pitch/roll (solvePnP), mouth aspect ratio (yawns), face-cover (mask) detection, optical-flow motion energy, PERCLOS and blink rate |
| 🧠 Hierarchical decisions | Phase 1: a deterministic, explainable state machine that reacts instantly. Phase 2: a dilated causal TCN with self-attention over 30-frame windows that catches subtler drift |
| 🚗 Closed-loop actuation | V_t = min(V_max − CL·k, V_current + A_t·Δt): cognitive load lowers the speed ceiling smoothly, microsleeps slow the car, and critical lapses latch emergency braking until the driver looks back |
| 🧭 Context aware | Looking left with the left indicator on counts as a mirror check, not a distraction. Alert escalation depends on speed |
| 🔍 Transparent | Every alert says why (head turned / gaze off road / eyes closed / face lost) and for how long. The TCN shows its class probabilities and a live heat-strip of where its attention focused |
| 📄 Post-drive report | One-click PDF with KPIs, attention timeline, time-in-state, TCN probabilities, recommendations and a full event log |
| 🎛 Data → model loop | Record labelled windows from the dashboard, retrain, and the stats view switches to real-data metrics |
| 🔌 Hardware ready | Optional Arduino link over serial (LEDs, buzzer, throttle servo) and optional on-device text-to-speech |
git clone https://github.com/QuantumCoderrr/AttenDrive.git && cd AttenDrive
python3 -m venv .venv && source .venv/bin/activate # Python 3.10–3.12
pip install -r requirements.txt
python scripts/fetch_models.py # MediaPipe face landmarker (3.7 MB)
python run.py # opens http://127.0.0.1:8000A trained TCN model (models/tcn_attention.npz) ships with the repo, so no training is needed to run the app.
macOS: the first run asks for camera permission for your terminal app. Enable it in System Settings → Privacy & Security → Camera, then restart the command.
python run.py --source 1 # a different camera
python run.py --source drive.mp4 # replay a recorded video (loops)
python run.py --driver "Sandip Ghosh" # name on the PDF report
python run.py --server-tts # speak alerts on this machine (pyttsx3)
python run.py --serial /dev/tty.usbmodem101 # drive an Arduino actuator| Key | Action |
|---|---|
↑ / ↓ |
Throttle / brake |
← / → |
Steer between lanes |
Q / E |
Left / right indicator |
C |
Cruise control at the current speed |
M |
Mute alerts |
Try it: set cruise around 60 km/h, then look over your shoulder for 3 seconds, close your eyes for 2 seconds, put on a mask, or signal left and glance left.
flowchart LR
CAM[Webcam] --> FM[MediaPipe Face Mesh<br/>478 landmarks]
CAM --> OF[Farnebäck optical flow<br/>motion trigger]
FM --> FX[Feature extraction<br/>EAR · gaze · solvePnP pose · MAR · mask]
FX --> BL[Dynamic baseline<br/>first 60 frames + rolling EAR]
BL --> P1[Phase 1 · heuristic state machine<br/>timers · context · escalation]
BL --> W[30-frame window] --> P2[Phase 2 · TCN-Attention<br/>Safe / Drowsy / Distracted]
OF -. event trigger .-> P2
P1 --> FU[Fusion → alert level 0-3, focus score]
P2 --> FU
FU --> VEH[Vehicle model<br/>cognitive load · speed cap · AEB]
FU --> UI[Dashboard · voice · PDF]
VEH --> UI
VEH -. optional .-> ARD[Arduino]
| Signal | Rule | Response |
|---|---|---|
| Eyes closed | EAR below 70 % of personal baseline | blink if < 0.5 s |
| Microsleep | eyes closed > 1.5 s | level 2: drowsy, active slowdown |
| Critical closure | eyes closed > 3 s | level 3: emergency braking |
| Head turned / gaze away / looking down | |yaw| > 20°, pitch < −15°, |gaze| > 0.45 | glance if < 2 s, caution > 2 s, distracted > 4 s |
| Distracted at speed | away > 3 s while > 40 km/h | level 3: emergency braking |
| Face not visible | no face > 2 s | level 2, escalating at speed |
| Mirror check | deviation in the same direction as the active indicator | suppressed |
| Fatigue trend | PERCLOS > 25 % or ≥ 3 yawns in 5 min | level 1 |
Pose and gaze are smoothed (EMA), and a 0.4 s grace window keeps a single noisy frame from resetting a distraction timer. Face-cover detection compares forehead and lower-face HSV histograms (Bhattacharyya distance) on frontal views only, and disables mouth cues while a mask is worn.
F_t = [EAR_norm, GazeX, GazeY, Yaw_norm, Pitch_norm] window T = 30 frames (~1 s)
x ─► TemporalBlock(d=1) ─► d=2 ─► d=4 ─► d=8 ─► H causal, dilated, residual (receptive field 61)
α = softmax(wᵀ tanh(V·H)) where in the window attention broke
z = [Σ αₜ hₜ ‖ mean(H)] ─► Dense ─► ReLU ─► Dense ─► softmax {Safe, Drowsy, Distracted}
The network is trained in PyTorch and exported to a plain .npz. The app runs a pure-NumPy forward pass (≈ 0.4 ms on a laptop CPU), so no deep-learning framework is needed at runtime. tests/test_tcn.py checks that it matches PyTorch to 1e-5. Inference runs every 3 frames, immediately on an optical-flow spike, or when the heuristic level changes (event-triggered). If the heuristics say "fine" but the TCN is confident about a non-safe class for over 1 s, the system raises a cognitive-drift caution.
The bundled model is bootstrapped on a synthetic behaviour simulator (training/synth.py) that models blinks, saccades, mirror glances, slow eyelid closures, head nods, phone-down posture and more. Its metrics in the Stats view are therefore synthetic, and the app says so.
To get real-world numbers:
- In the dashboard click Record data → Safe and drive normally for a minute, then click Stop.
- Repeat for Drowsy (slow blinks, heavy lids, nodding) and Distracted (phone in lap, looking at a passenger).
- Retrain with your recordings mixed in (they are held out for evaluation too):
pip install -r requirements-train.txt
python -m training.train --real data/recordedRestart the app and the Stats view now reports accuracy, precision, recall, F1 and the confusion matrix on your recorded test windows.
attendrive/
vision/ landmarks.py (Face Mesh) · features.py (EAR, gaze, solvePnP, MAR, mask) · flow.py (optical flow)
cognition/ baseline.py (dynamic normalisation) · heuristics.py (Phase 1) · tcn.py (Phase 2, NumPy runtime)
vehicle/ dynamics.py (cognitive load, speed cap, AEB, lanes, indicators)
session/ recorder.py (events, timeline) · report.py (ReportLab PDF)
integrations/ tts.py (pyttsx3) · serial_link.py (Arduino)
server/ app.py (FastAPI + WebSocket streaming)
engine.py the real-time loop that ties it all together
config.py every threshold in one place
training/ synth.py · model.py (PyTorch) · train.py (train, evaluate, export)
web/ dashboard (vanilla JS + Canvas, no build step)
hardware/ attendrive_actuator.ino
tests/ pose conventions, state machine, vehicle model, NumPy-vs-PyTorch parity, PDF
pip install -r requirements-train.txt && pytest -q- Head pitch read about −178° for a frontal face, because the Euler angles from
solvePnPwere never unwrapped. The 3-D model now lives in camera coordinates, angles are folded into ±90°, and the sign conventions are unit-tested. - The TCN label was hard-wired to "SAFE". It is now a trained network with real probabilities, attention weights and event triggers.
- "Face cover: detected" with no mask on. Detection now compares skin histograms against the forehead, is gated to frontal views, and uses hysteresis.
- Gaze reported "UP" at rest. All signals are normalised against a per-driver baseline.
- Tkinter UI replaced with a responsive, GPU-accelerated web dashboard; added the PDF report, data recording, cruise control, lane departure, PERCLOS/blink/yawn metrics and tests.
AttenDrive is a research prototype with a simulated vehicle. It is not a certified safety system. Accuracy figures depend entirely on the data the TCN is evaluated on; the shipped numbers come from synthetic data. Head pose uses a generic face model, so absolute angles carry a per-person offset, which the calibration baseline removes.
Based on "AttenDrive: A Hierarchical Vision-Based TCN-Attention Framework for Real-Time Driver Cognitive Monitoring and Vehicle Actuation" by Sandip Ghosh, Abirbhab Adhikari, Soumya Bakshi and Sampurna Sarkar, Department of Artificial Intelligence & Machine Learning, St. Thomas' College of Engineering and Technology, Kolkata, under the guidance of Jaydeb Mondal.
Key references: Soukupová & Čech (EAR, 2016) · Bai, Kolter & Koltun (TCN, 2018) · Vaswani et al. (attention, 2017) · Farnebäck (optical flow, 2003).
MIT. See LICENSE.