|
By co-designing optics and algorithms, computational cameras can do more than regular cameras - they can see in the extreme dark, measure 3D, be extremely compact, record different wavelengths of light, or capture the phase of light. These computational imagers are powered by algorithms which uncover the signal from encoded or noisy measurements. Over the years the classic methods to recover information from computational cameras have been based on minimizing an optimization problem consisting of a data fidelity and hand-picked prior term. More recently, deep learning has been applied to these problems, but often has no way to incorporate known optical characteristics, requires large training datasets, and results in black-box models that cannot easily be interpreted. In this talk, I will introduce physics-informed machine learning for computational imaging, which is a middle ground approach that combines elements of classic methods with deep learning. I will demonstrate this approach through two examples on real computational cameras: a tiny, cheap lensless camera and a high-end low-light camera for nighttime videography. In each case incorporating knowledge of imaging system physics into neural networks can improve image quality and performance beyond what is feasible with either classic or deep methods. For lensless imaging, physics-informed machine learning can speed up the reconstruction time by an order of magnitude and improve the perceptual image quality. For nighttime videography, we can learn a physics-informed noise generator that can realistically synthesize noise at extremely high-gain, low-light settings. Using this learned noise model, we can take videos of moving objects on a clear, moonless night with no external illumination (submillilux) for the first time, pushing the limit of what cameras can see in the extreme dark by an order of magnitude.
|
