Differentiable display photometric stereo is a computational imaging framework that jointly optimizes surface reflectance, lighting, and display parameters by differentiating through a display rendering model. By treating screen pixels as controlled light sources, this approach enables precise shape reconstruction and material estimation under controllable illumination.
This technique bridges inverse optics, neural rendering, and display hardware, allowing images to be synthesized under new lighting conditions while remaining physically consistent with measurable surface properties.
| Core Objective | Key Inputs | Typical Outputs | Display Integration Benefit |
|---|---|---|---|
| Estimate surface shape and reflectance | Multi-view images, known lighting directions | Normal map, albedo, roughness | Use display as controllable lighting proxy |
| Optimize display radiance patterns | Target image, inverse render model | Optimized pixel intensity patterns | Close the loop between capture and rendering |
| Enable shape-aware relighting | Reconstructed geometry, measured BRDF | New synthetic renders under arbitrary lights | Generate photoreal results with fewer real lights |
| Support hardware-in-the-loop calibration | Display response measurements, sensor data | Calibrated light transport parameters | Reduce cross-device variability |
Photometric Stereo Under Differentiable Rendering
Classic photometric stereo estimates surface normals from multiple images lit by known light sources. Differentiable display photometric stereo reformulates this by encoding light transport in a differentiable rendering pipeline, where display pixels act as structured light emitters and gradients guide optimization.
This formulation allows end-to-end learning or calibration of lighting, camera, and display interactions, supporting both inverse reconstruction and forward synthesis under novel lighting conditions.
Display Modeling for Inverse Rendering
From Measured Intensity to Controlled Radiance
Instead of relying solely on physical light sources, this method parameterizes the display as a set of basis radiance patterns and optimizes their weights through gradient descent. Camera images and display outputs are jointly explained by a shading model that accounts for display non-linearities and spatial interactions.
Accurate camera response calibration, screen emissivity characterization, and per-pixel radiance estimation are essential to ensure that gradients backpropagated from images correspond to physically meaningful display and surface changes.
Shape and Material Estimation with Closed-Loop Illumination
Hybrid Inverse-Forward Optimization
Optimization alternates between inferring surface geometry and material properties from observed images and synthesizing target patterns on the display that probe specific surface directions. Closed-loop feedback from photometric error enables the system to refine both the display radiance and the estimated BRDF over multiple iterations.
By leveraging displays as structured probes, the approach can capture subtle interreflections, anisotropic scattering, and view-dependent effects that are difficult to measure with conventional light fixtures alone.
Integration with Display Hardware and Calibration
Hardware-Aware Regularization and Noise Robustness
Display non-linearities, bit-depth limitations, and temporal artifacts are incorporated into the loss using hardware-aware regularizers. Multi-camera rigs, synchronized acquisition, and display gamma correction are modeled explicitly to prevent ill-conditioned optimization and ghosting artifacts.
Calibration routines estimate per-screen radiance transfer functions, camera projective geometry, and extrinsic alignment to lighting-controlled surfaces, enabling reproducible results across different devices and environments.
Key Takeaways for Practitioners and Researchers
- Treat the display as a differentiable light source to close the loop between capture and rendering.
- Jointly estimate surface shape, material properties, and display radiance patterns via gradient-based optimization.
- Invest in radiometric calibration, multi-camera geometry, and display characterization to stabilize results.
- Combine classic photometric stereo priors with modern rendering to exploit both physical and learned components.
- Plan for controlled environments to mitigate interreflections, intervisibility issues, and display-induced artifacts.
FAQ
Reader questions
How does differentiable display photometric stereo differ from classic photometric stereo?
Classic photometric stereo relies on a small set of physical lights with known directions, while differentiable display photometric stereo treats the display as a programmable light source, jointly optimizing display patterns and surface properties through gradients rendered from a shading model.
What types of materials and scenes can this method capture reliably?
It performs well for matte and mildly glossy surfaces where linear or single-scattering BRDF approximations hold, and where interreflections and complex view-dependent effects are secondary; transparent or highly glossy samples may require extended models.
Can this approach be used for real-time relighting of captured objects?
Yes, after geometry and reflectance are estimated, novel lighting synthesis can be computed efficiently; however, display-driven optimization is typically used offline to capture data, with real-time rendering delegated to shaded meshes or neural proxies. Key constraints include precise camera and display synchronization, accurate radiometric calibration, controlled ambient lighting, and sufficient multi-view image coverage; motion blur from display refresh and camera shutter mismatch can further limit resolution and acquisition speed.