Solving inverse problems for the analysis of fast moving objects
The aim of the project was to significantly improve image and video quality despite the limited technical capabilities of recording devices. The method, developed in collaboration with the CTU - Centre for Machine Perception, leverages the fact that more information can be extracted from a blurred video than from a single sharp image — such as the 3D trajectory and angular velocity of an object. This enables the reconstruction of the shape, appearance, and motion of fast-moving objects. The results of the project have applications in scientific experiments, defense, healthcare, and other fields.
Even the best video cameras can not capture fast-moving objects—like a football or passing cars—without motion blur. The developed method can significantly improve image quality by estimating the shape, appearance, and 2D trajectory of a fast-moving object from blurred video. This can produce video of higher quality than what even the best current cameras can capture. When additional a priori information about the object is included, it is even possible to recover complete 3D trajectories and angular velocities.
The method is based on solving a complex inverse problem, which we refer to as deblatting. This process simultaneously solves the inverse problems of deblurring (removing blur) and matting (foreground-background separation). The method assumes a forward image-formation model:
where the individual variables have the following meanings:
The method has already attracted media attention by successfully identifying birds of prey in footage originally believed to show a UFO. Read more in the media.
Selected Results on Real Data:
Reconstruction of the appearance and shape of fast-moving cars.
Reconstruction of the 2D trajectory, appearance, and shape of free-flying objects.
Temporal video super-resolution (artificial frame rate increase).
The research behind this work was supported by a GACR project and received the GACR President's Award 2022.
Related publications:
- Rozumnyi D., Kotera Jan, Šroubek Filip, Matas J. : Tracking by Deblatting, International Journal of Computer Vision vol.129, 9 (2021), p. 2583-2604.
- Kotera Jan, Matas J., Šroubek Filip : Restoration of Fast Moving Objects, IEEE Transactions on Image Processing vol.29, 1 (2020), p. 8577-8589.
- Rozumnyi D., Kotera Jan, Šroubek Filip, Matas J. : Sub-Frame Appearance and 6D Pose Estimation of Fast Moving Objects, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6777-6785, 2020.
- Kotera Jan, Rozumnyi D., Šroubek Filip, Matas J. : Intra-Frame Object Tracking by Deblatting, Proceedings of the IEEE International Conference on Computer Vision 2019 (ICCV), p. 2300-2309, 2019.
- Rozumnyi D., Kotera Jan, Šroubek Filip, Matas J. : Non-causal Tracking by Deblatting, Pattern Recognition : 41st DAGM German Conference, DAGM GCPR 2019, p. 122-135.
- Rozumnyi D., Kotera Jan, Šroubek Filip, Novotný L., Matas J. : The World of Fast Moving Objects, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 5203-5211.