Generative models have recently demonstrated impressive capabilities in producing high-quality 3D shapes from a variety of user inputs (e.g., text or images). However, generated objects often lack physical integrity. We introduce PhysiOpt, a differentiable physics optimizer designed to improve the physical behavior of 3D generative outputs, enabling them to transition from virtual designs to physically plausible, real-world objects. While most generative models represent geometry as continuous implicit fields, physics-based approaches often rely on the finite element method (FEM), requiring ad hoc mesh extraction to perform shape optimization. In addition, these methods are typically slow, limiting their integration in fast, iterative generative design workflows. Instead, we bridge the representation gap and propose a fast and effective differentiable simulation pipeline that optimizes shapes directly in the latent space of generative models using an intuitive and easy-to-implement differentiable mapping. This approach enables fast optimization while preserving semantic structure, unlike traditional methods relying on local mesh-based adjustments. We demonstrate the versatility of our optimizer across a range of shape priors, from global and part-based latent models to a state-of-the-art large-scale 3D generator, and compare it to a traditional mesh-based shape optimizer. Our method preserves the native representation and capabilities of the underlying generative model while supporting user-specified materials, loads, and boundary conditions. The resulting designs exhibit improved physical behavior, remain faithful to the learned priors, and are suitable for fabrication. We demonstrate the effectiveness of our approach on both virtual and fabricated objects.
Our approach supports a variety of input modalities (Inputs, left), including libraries of shapes and parts, images, and text prompts, which are converted into the latent parameters of a given generative model. Users can also specify materials, loads, and boundary conditions. Given a latent parameter $\boldsymbol{\pi}$, PhysiOpt decodes it into an implicit field $\phi(\cdot, \boldsymbol{\pi})$, which is voxelized to produce a sparse set of density-weighted finite elements. Under the user-defined conditions, we solve for displacements using linear static analysis and compute a physics-based loss $J(\boldsymbol{\pi})$. The entire process is fully differentiable, allowing iterative updates of $\boldsymbol{\pi}$ towards more physically compliant designs (Differentiable Optimization, right).
PhysiOpt is compatible with various 3D generative models including global-shape latent models (e.g., DeepSDF), part-based latent models and state-of-the-art large-scale 3D generators (e.g., TRELLIS).
Our method fully leverages the flexibility of the finite element method (FEM) by allowing users to define custom boundary conditions (black) and apply loads to specific regions (red with arrows). This enables adaptation to a wide range of real-world scenarios.
By operating directly in the latent space of a generative model, PhysiOpt retains its full capabilities and enables multi-step design phases without cumbersome and “lossy” conversions between representations. For example, with TRELLIS, users can start by generating a shape from an input image, optimize it under specific loads (e.g., an additional load on the front porch), modify the geometry via inpainting (e.g., by adding a side platform with a flowerpot), and then re-optimize the updated shape to account for the new geometry.
Constrary to traditional mesh-based shape optimization methods (e.g., DiffIPC), PhysiOpt enables semantically consistent changes to geometry rather than vertex perturbations. Operating in the latent space of generative models also leads to faster convergence.
We demonstrate the effectiveness of our optimizer by fabricating various shapes before and after optimization. In this example, the unoptimized octopus chair (left) deforms and its legs touch the ground, while the unoptimized flamingo glass (right) tips over with minimal weight.
@inproceedings{physiopt,
title = {PhysiOpt: Physics-Driven Shape Optimization for 3D Generative Models},
author = {Zhan, Xiao and Jambon, Cl{\'e}ment and Thompson, Evan and Ng, Kenney and Konakovi{\'c} Lukovi{\'c}, Mina},
booktitle = {ACM SIGGRAPH Asia 2025 Conference Proceedings},
year = {2025},
note = {To appear in ACM SIGGRAPH Asia 2025 Conference Proceedings.}
}
