ICLR 2025 Procedure planning from visual endpoints
MTID learns latent visual mid-state guidance between a start and goal observation, then conditions a diffusion planner to predict the intervening discrete action sequence.
Model output: dataset-defined action labels, not generated video frames. No annotated intermediate visual states are required.
Open full resolution
Problem & insight
Procedure planning must infer the discrete actions connecting a visual start state Vs to a goal state Vg, even though every state in between is unobserved.
01 · Prior supervision
Earlier approaches often construct intermediate guidance from task labels, events, graphs, or language. These signals help identify what should happen, but describe less of the visual evolution between the endpoints.
02 · MTID
MTID generates learnable latent features between the encoded endpoints, refines their temporal dependencies, and injects them directly into the action planner through cross-attention.
Intermediate features are learned from the two observed endpoints without synthesizing pixels.
The predicted task restricts initial action noise to the associated action vocabulary.
A weighted masked loss emphasizes actions near the observed states and discourages irrelevant predictions.
Method
The task classifier is trained first. The latent interpolation module and diffusion planner are then optimized together, so the generated guidance adapts to action reasoning.
Task prediction
A Transformer classifier predicts the task from the start and goal observations. The class conditions planning and defines the task-specific action mask.
Latent temporal interpolation
An observation encoder maps the endpoints to Ls and Lg. Learnable interpolation weights generate intermediate features, which Transformer blocks refine.
Ij = (1 − φj)Ls + φjLg
Masked diffusion planning
The task mask is applied to the initial action noise—not at every denoising iteration. DDIM then denoises all T action steps jointly in a temporal U-Net.
Open full resolution
Open full resolution
Selected results
Values are percentage points. CrossTask uses HowTo100M features under the PDPP setting for the dagger-marked rows in the paper’s main table.
CrossTask · T=3 · SR
40.45 +1.52 over SCHEMA in Table 1CrossTask · T=3 · mAcc
67.19 +2.45 over KEPP in Table 1CrossTask · main table
3 best SR results across T=3–5CrossTask
| Method | T=3 | T=4 | T=5 | T=6 |
|---|---|---|---|---|
| PDPP | 37.20 | 21.48 | 13.45 | 8.41 |
| KEPP | 38.12 | 24.15 | 14.20 | 9.27 |
| SCHEMA | 38.93 | 24.50 | 14.75 | 10.53 |
| MTID | 40.45 | 24.76 | 15.26 | 10.30 |
MTID leads at T=3–5. At T=6, SCHEMA reports 10.53 versus MTID’s 10.30.
COIN & NIV · KEPP setting
22.74 SR
49.90 mAcc
24.89 SR
44.54 mAcc
mIoU is reported in the paper for completeness, but MTID is not the leading method on mIoU in these tables; this page therefore avoids an across-the-board state-of-the-art claim.
Same-setting gain over PDPP · CrossTask T=3
+3.25 SRSame-setting gain over PDPP · CrossTask T=3
+2.52 mAccSame-setting gain over PDPP · CrossTask T=3
+2.60 mIoUAblations
The experiments isolate what to interpolate, when to apply the task mask, and how to weight the objective. They are not presented as one cumulative ladder.
Interpolation design · T=3
Mask placement · T=3
Training objective · T=5
Benchmarks & scope
Evaluation protocol
Videos are randomly split into 70% training and 30% testing within each task. Results measure in-distribution procedure planning rather than unseen-task generalization.
Known dependency
Planning depends on the predicted task and dataset-specific action vocabulary. A task-class mismatch can therefore affect both conditioning and the action mask.
Output boundary
MTID predicts discrete benchmark actions from endpoint features. It is not a video-generation model, deployed robot policy, or real-time control system.
Resources
Everything needed to read the work, inspect the implementation, and prepare the benchmark features.
Citation
If this work supports your research, please cite the ICLR 2025 paper.
@inproceedings{zhou2025masked,
title = {Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos},
author = {Yufan Zhou and Zhaobo Qi and Lingshuai Lin and Junqi Jing and Tingting Chai and Beichen Zhang and Shuhui Wang and Weigang Zhang},
booktitle = {The Thirteenth International Conference on Learning Representations},
year = {2025},
url = {https://openreview.net/forum?id=HnpDHiItd2}
}