AI Style Transfer Explained: From Neural Style to Diffusion Style References, Uses, Copyright and Artist Protections

How AI style transfer evolved from the 2015 neural style algorithm to diffusion models with ControlNet and image style references, how creators use it in design, animation and photography, where the law stands on artistic style, and tools such as Glaze that artists use to resist mimicry.

PUBLISHED // UPDATED
DURATION // 06 MIN READ
AI Style Transfer Explained: From Neural Style to Diffusion Style References, Uses, Copyright and Artist Protections
FIG.01 // VISUAL DOCUMENTATION AI Style Transfer Explained: From Neural Style to Diffusion Style References, Uses, Copyright and Artist Protections

In 2015, a paper titled “A Neural Algorithm of Artistic Style” showed that a neural network could separate the content of a photo from the style of a painting and recombine them, turning a snapshot into something resembling a Van Gogh. It was slow and the results were rough, but it launched a field. A decade later, style transfer is built into phone filters, design tools and image generators, and it sits at the centre of debates about imitation, authorship and artists’ livelihoods. This guide explains how the technology works, how it is used and what the legal and ethical issues are.

How style transfer works

The original approach. Gatys and colleagues used the internal features of an image-recognition network: deeper layers capture an image’s content, while correlations between features capture textures, brushstrokes and colours, which the authors called style. Their method optimised a new image step by step to match the content of one picture and the style of another. Each image took many iterations of computation.

Fast feed-forward networks. In 2016, researchers trained networks that apply a given style in a single pass, fast enough for video and phone apps. A year later, Adaptive Instance Normalization (AdaIN) allowed one model to apply arbitrary styles rather than one style per model. These methods powered the wave of artistic filter apps.

Diffusion models. Modern image generators approach style differently. Rather than transferring texture statistics from one image to another, they generate a new image guided by a text prompt, a reference image or both. Image-to-image modes start from an existing photo and repaint it; tools such as ControlNet preserve the structure, such as edges, depth or pose, while the style changes; and image-prompt adapters let a model take visual cues from reference pictures. Commercial tools offer style-reference features built on similar ideas: Midjourney, for example, lets users supply reference images to steer the look of new generations.

GenerationMethodStrengthsLimits
2015Optimisation-based neural styleStriking texture transferSlow; distorts content
2016–2017Feed-forward networks, AdaINReal time; arbitrary stylesMostly surface texture and colour
2022 onwardsDiffusion with image prompts, ControlNet and adaptersChanges style while preserving structure; text controlCan drift from the source; raises imitation concerns

How creators use it

Design exploration. Designers use style references and image-to-image modes to try many visual treatments quickly: a product concept in watercolour, flat vector or photorealistic styles, or a fashion sketch in different fabrics and patterns. The aim is usually to explore options before committing to a direction, not to produce final artwork.

Consistent visual identity. Studios and brands use their own artwork as style references, or fine-tune small style adapters on it, so that many generated images share a look. Using your own material avoids most of the ethical and legal questions that arise from imitating others.

Animation and film. Style transfer can give live action or 3D renders a painted or hand-drawn look, though keeping the style stable from frame to frame remains a challenge. Netflix Japan’s 2023 short “The Dog and the Boy” drew criticism for using AI-generated background art, showing how sensitive audiences and artists are to such uses in animation.

Photography and social media. Artistic filters in phone and editing apps are the most familiar form, turning photos into sketches, paintings or comics.

Copyright generally protects specific works, not an artistic style. Anyone may paint in an impressionist manner, and in principle an AI tool may produce images in a recognisable style without copying a particular work. The legal questions that matter more are how models were trained and whether outputs reproduce protected works or characters.

Those questions are being litigated. A group of artists sued Stability AI, Midjourney and other companies in 2023, alleging their works were used for training without permission. In August 2024, as The Verge reported, Judge William Orrick allowed the case to proceed with some claims dismissed, adding a claim of induced copyright infringement against Stability. In the US, the Copyright Office’s position is that purely AI-generated material cannot be registered, though human contributions can be (U.S. Copyright Office).

When OpenAI’s GPT-4o image generation launched in March 2025, a viral wave of “Studio Ghibli-style” images reopened the debate about imitating a studio’s look without its involvement, even where no specific frame was copied.

Protections for artists

Researchers at the University of Chicago developed Glaze, which adds small perturbations to artwork that are hard for people to see but disrupt a model’s ability to learn and mimic the artist’s style (Glaze paper, arXiv:2302.04222). Such tools raise the cost of mimicry but are not a guarantee, because new models and techniques can reduce their effect.

Other options include opting out of training where platforms and datasets allow it, licensing work only to providers with clear terms, and adding Content Credentials to published work to record authorship.

Using style transfer responsibly

SituationApproach
Your own artwork or brand assets as referencesGenerally fine; you control the source
Public-domain art (for example, old masters)Generally fine
Licensed style packs or commissioned referencesFine within the licence terms
A living artist’s name or work to produce saleable images in their styleAvoid: damages trust, may breach platform rules and can create legal risk
Recognisable characters or specific artworksAvoid without permission: these are protected

A short technical history

  • Gatys, Ecker and Bethge (2015): showed that convolutional neural network features can separate an image’s “content” from its “style”, which made neural style transfer possible (arXiv:1508.06576).
  • Johnson, Alahi and Fei-Fei (2016): trained feed-forward networks with perceptual losses, making style transfer fast enough for real time (arXiv:1603.08155).
  • Huang and Belongie (2017): Adaptive Instance Normalization (AdaIN) allowed arbitrary styles in a single model (arXiv:1703.06868).
  • Zhang, Rao and Agrawala (2023), ControlNet: added structural conditioning such as edges, depth and pose to diffusion models, so style can change while composition stays fixed (arXiv:2302.05543).
  • Ye et al. (2023), IP-Adapter: let diffusion models take a reference image as a prompt alongside text, a common basis for style-reference features (arXiv:2308.06721).

Our analysis

Style transfer has evolved from a striking research demo into an everyday creative tool, and diffusion models with image references have made it far more controllable. Its most defensible uses rely on your own references, public-domain art or licensed material. Imitating living artists is where the technology meets real harm and unresolved law, and artists’ protections such as Glaze are helpful but imperfect. For creators, the practical rule is to treat style as something to develop, not borrow.