Predicting the next frames of a surgical clip is the question this project sits on. The model is a VAE–Transformer. In my experiments it improved PSNR by 2.36 dB over the selected baselines and ran at 22 FPS with FP16 mixed precision. The code is on GitHub.
Fig. 1The FET-VAE architecture used for autoregressive surgical-video prediction. Architecture schematic. The t=20 results are the repository’s reported JIGSAWS test-set figures; the shape of the quality-decay curve is illustrative.
