We have presented FlexISP, a framework and a system that replaces the traditional image processing pipeline for reconstructing photographs from raw sensor data by a single, integrated, and flexible system that is based on global optimization.
The image formation model and any image priors and regularization terms are expressed as a single objective function, which is solved using a proximal operator framework.
Proximal operators decouple the individual terms in the objective function in a principled way, making it possible to separately implement the operators for the data term and each regularizer.
This approach enables mixing and matching different, highly optimized implementations of data terms and regularizers.
Our framework therefore achieves both the improved image quality of a fully integrated optimization and the separation of concerns that gave rise to the traditional pipeline approach.
We detail and analyze our design choices, and conclude that a single specific set of priors can be used for a variety of applications, and still outperforms other state-of-the-art methods. While we demonstrate significant improvements in quality and simplicity for traditional camera designs, we believe that our approach will achieve its full potential with future computational cameras that have significantly more complex image formation models.