Differentiable Optics for Automated Optical Design and End-to-end Computational Imaging

Overview

Computational imaging relies on tight integration between optical capture and computational reconstruction, yet the two parts have traditionally been designed separately. Differentiable optics, which builds optical simulation with automatic differentiation frameworks, opens a new paradigm. First, end-to-end co-design jointly optimizes optics and reconstruction algorithms for optimal system-level performance. Second, by combining accurate and efficient gradient backpropagation with modern optimization algorithms, differentiable optics enables automated optical design of complex systems from scratch without human intervention. However, existing optical simulations are either too idealized for accuracy or too computationally expensive for differentiable simulation, and optimization methods remain limited in robustness and scalability, hindering automated and scalable design for complex systems.

This dissertation develops accurate, efficient, and fully differentiable optical simulation models, aiming to automate the design of complex optical systems and bridge the gap between simulation and reality for computational imaging. Specifically, we extend differentiable ray tracing to incorporate diffractive and polarization optics, unify ray and wave optics, and develop non-sequential models, supporting various optical systems including hybrid refractive-diffractive lenses and geometric waveguide displays. To improve simulation efficiency, we explore hierarchical strategies spanning pipeline optimizations and neural surrogate models. We further study novel optimization methods to address non-convexity and large-scale challenges, enabling stable optical design from scratch.

Leveraging this, we demonstrate three directions: (1)automated design from scratch for complex optical systems including multi-element lenses, hybrid refractive-diffractive systems, and geometric waveguides, minimizing expert input; (2)accurate optical simulation for computational imaging including depth estimation from focal stacks and defocus deblurring; and (3)end-to-end co-design for computational cameras, enhancing extended depth-of-field and task-specific vision. Across these applications, realistic optical simulation reduces the gap between idealized training assumptions and physical imaging behavior, leading to improved robustness and system-level performance.

In summary, differentiable optics provides a foundation for next-generation optical design and computational imaging, bridging physics, simulation, and learning to optimize systems directly for target tasks beyond classical metrics. By developing differentiable optical simulation and optimization methods, this dissertation advances automated design of complex optical systems and enables more accurate and robust computational imaging, paving the way for future innovations in optics and vision.