Structure-from-motion rebuilds a scene in 3D from ordinary photographs. It works by finding the same points in pairs of images and solving for where the cameras must have been. On easy pairs, a modern matcher does this almost perfectly. On hard pairs it falls apart: two photos with little overlap, a big change in viewpoint, repeated textures such as windows on a façade, or two different buildings that look alike.
\nThe usual response is a better matcher. This project tries something else. It trains a small multimodal model to decide how to handle each pair: which tool to call, on which crop, and when to give up on a match that looks wrong. A frozen matcher applies one strategy to every pair; an agent can change strategy per pair.
\nThe idea is Gabriele Berton's. He posted it publicly on LinkedIn on 3 July 2026, as his research idea number 11, and he has not published a paper on it. This repository is my implementation, and it is still in progress.
\nThe agent talks to a small tool server over HTTP. It can:
\nA rollout is capped at 10 tool calls and 15 turns, so the agent cannot solve a pair by brute force.
\nThe reward follows what structure-from-motion actually cares about, which is getting the camera pose right. The main term checks whether the recovered relative pose is within 5°, 10° and 20° of the truth, averaged as a pass or fail at each threshold. That is a coarse stand-in for the standard pose AUC. Smaller terms reward the inlier count and correctly formatted tool calls, and a penalty discourages invalid ones.
\nThe first design also charged a small cost for every tool call. The current configuration switches that off in favour of an accumulative tool coefficient, taken from PyVision-RL, which shapes tool use over the whole rollout rather than taxing each call.
\nThe policy is Qwen/Qwen3.5-2B with a LoRA adapter (rank 32, α 64). It is trained with GRPO in groups of eight, with the clip-higher and dynamic-sampling changes from recent work. The current configuration uses S-GRPO. The repository carries both a custom GRPO loop and a VeRL configuration.
It runs on three NVIDIA L40S GPUs on the AIRE cluster:
\nThe training set is 1,535 hard pairs from MegaDepth, drawn from 102 scenes. The curriculum starts with medium pairs for 10 epochs, then medium and hard pairs for 20, then everything for 20, at a learning rate of 1 × 10⁻⁵.
\nBefore any training, I checked whether a capable model could simply be prompted to use the tools well. On a small, synthetic set of hard pairs it could not. The zero-shot agent averaged 8.9 inliers, against 12.4 for calling a matcher directly.
\nA second check on 12 real MegaDepth pairs showed why. The untrained model asked for full-frame crops every time, so its matches were identical to direct matching. It had tools and did not use them. That result is the reason for the reinforcement learning, rather than prompting.
\nThe Phase 1 run is under way. Over the first 30 logged steps, the mean reward rose from 0.374 to 0.496, with an epoch-one mean of 0.388. The run was resumed from its latest checkpoint after I fixed a logging bug.
\nThese are the best matches the agent produced at steps 10, 20 and 30. They scored 432 inliers (85%), 540 inliers (93%) and 295 inliers (67%), with rewards of 0.58, 0.58 and 0.57.
\n","A rising training reward is not an evaluation. I have not yet compared the trained policy against direct matching on held-out pairs, and until I have, there is no result to claim beyond "the policy is learning something the reward measures".
\nPhase 2 moves from pairs to whole scenes: the agent builds a reconstruction from a set of images instead of a single pair. That pipeline is implemented and passes its 180 tests, and it is being validated now. Phase 3, a proper benchmark against standard SfM on hard scenes, is planned.
\nThe adapters from each stage are on Hugging Face: pair-level fine-tuning, the Phase 1 GRPO checkpoint, and scene-level fine-tuning.
\n"],"figures":[1],"words":802,"minutes":3}