Domain Expansion of Image Generators

Abstract

Can one inject new concepts into an already trained generative model, while respecting its existing structure and knowledge? We propose a new task - domain expansion - to address this. Given a pretrained generator and novel (but related) domains, we expand the generator to jointly model all domains, old and new, harmoniously. First, we note the generator contains a meaningful, pretrained latent space. Is it possible to minimally perturb this hard-earned representation, while maximally representing the new domains? Interestingly, we find that the latent space offers unused, dormant directions, which do not affect the output. This provides an opportunity: By repurposing these directions, we can represent new domains without perturbing the original representation. In fact, we find that pretrained generators have the capacity to add several - even hundreds - of new domains! Using our expansion method, one expanded model can supersede numerous domain-specific models, without expanding the model size. Additionally, a single expanded generator natively supports smooth transitions between domains, as well as composition of domains.

How does it work?

We start by performing an orthogonal decomposition of the model's latent space and identify hundreds of low-magnitude directions which we qualify as dormant. Dormant directions have no effect while other directions are sufficient to represent the original domain. We now pick one dormant direction for every domain we wish to add. We call the space spanned by other directions the "base subspace". For each new domain, we transport the base space along its dormant direction, defining another subspace we call the "repurposed space". To capture the new domain, we apply an off-the-shelf domain adaptation method, modified to operate only on latent codes from the repurposed space. A regularization loss is applied on the base subspace to ensure that the original domain is preserved. Thanks to the subspaces being parallel and the latent space being disentangled, the original factors of variation from the original domain are introduced to the new domains.

The repurposed latent directions now control gradual transitions between the original and new domains. The new domain is now represented like other factors of variation that existed in the original generator. Expanded directions are meaningful not only between the two spaces! Extrapolations represent the opposite relationship between domains or exaggerations of it.

Images generated from the repurposed space itself are almost identical to those generated by a generator that was completely adapted to the new domain. Top two blocks display images generated by two distinct generators, adapted with StyleGAN-NADA and MyStyle. Both bottom blocks are generated from a single generator expanded with 105 domains!

Since the domains are represented as disentangled linear directions, we can trivially compose them with other domains, creating hybrids, or with other factors of variation that exist in the source domain. This works regardless of the domain adaptation method used to learn the new domain! Starting from a latent code that generates the top-left image, we traverse along two latent directions. For each direction, we note the associated domain, its ordinal number in the latent space's basis, and the training method used to learn the domain.

Citation