As most cameras are currently built to be used alongside machine learning algorithms, image quality requirements still emanate from human perception. To redefine key performance indicators (KPI) for machine vision, optical designs are tested and optimized before their conception using differentiable simulation methods and gradient backpropagation to jointly train an optical design and a neural network. Although this helps to design optical systems for improved machine learning performance, it remains unstable and computationally expensive to model complex compound optics such as wide-angle cameras. We focus on optimizing the distortion profile of ultra wide-angle designs as it constitutes the main KPI during the optical design. Along the way, we highlight the benefits of controlling the distortion profile of such systems, as well as the challenges related to using learning-based methods for optical design.