AI Style Transfer Explained: From Neural Style to Diffusion Style References, Uses, Copyright and Artist Protections
How AI style transfer evolved from the 2015 neural style algorithm to diffusion models with ControlNet and image style references, how creators use it in design, animation and photography, where the law stands on artistic style, and tools such as Glaze that artists use to resist mimicry.
In 2015, a paper titled “A Neural Algorithm of Artistic Style” showed that a neural network could separate the content of a photo from the style of a painting and recombine them, turning a snapshot into something resembling a Van Gogh. It was slow and the results were rough, but it launched a field. A decade later, style transfer is built into phone filters, design tools and image generators, and it sits at the centre of debates about imitation, authorship and artists’ livelihoods. This guide explains how the technology works, how it is used and what the legal and ethical issues are.
How style transfer works
The original approach. Gatys and colleagues used the internal features of an image-recognition network: deeper layers capture an image’s content, while correlations between features capture textures, brushstrokes and colours, which the authors called style. Their method optimised a new image step by step to match the content of one picture and the style of another. Each image took many iterations of computation.
Fast feed-forward networks. In 2016, researchers trained networks that apply a given style in a single pass, fast enough for video and phone apps. A year later, Adaptive Instance Normalization (AdaIN) allowed one model to apply arbitrary styles rather than one style per model. These methods powered the wave of artistic filter apps.
Diffusion models. Modern image generators approach style differently. Rather than transferring texture statistics from one image to another, they generate a new image guided by a text prompt, a reference image or both. Image-to-image modes start from an existing photo and repaint it; tools such as ControlNet preserve the structure, such as edges, depth or pose, while the style changes; and image-prompt adapters let a model take visual cues from reference pictures. Commercial tools offer style-reference features built on similar ideas: Midjourney, for example, lets users supply reference images to steer the look of new generations.
| Generation | Method | Strengths | Limits |
|---|---|---|---|
| 2015 | Optimisation-based neural style | Striking texture transfer | Slow; distorts content |
| 2016–2017 | Feed-forward networks, AdaIN | Real time; arbitrary styles | Mostly surface texture and colour |
| 2022 onwards | Diffusion with image prompts, ControlNet and adapters | Changes style while preserving structure; text control | Can drift from the source; raises imitation concerns |
How creators use it
Design exploration. Designers use style references and image-to-image modes to try many visual treatments quickly: a product concept in watercolour, flat vector or photorealistic styles, or a fashion sketch in different fabrics and patterns. The aim is usually to explore options before committing to a direction, not to produce final artwork.
Consistent visual identity. Studios and brands use their own artwork as style references, or fine-tune small style adapters on it, so that many generated images share a look. Using your own material avoids most of the ethical and legal questions that arise from imitating others.
Animation and film. Style transfer can give live action or 3D renders a painted or hand-drawn look, though keeping the style stable from frame to frame remains a challenge. Netflix Japan’s 2023 short “The Dog and the Boy” drew criticism for using AI-generated background art, showing how sensitive audiences and artists are to such uses in animation.
Photography and social media. Artistic filters in phone and editing apps are the most familiar form, turning photos into sketches, paintings or comics.
Style, copyright and the courts
Copyright generally protects specific works, not an artistic style. Anyone may paint in an impressionist manner, and in principle an AI tool may produce images in a recognisable style without copying a particular work. The legal questions that matter more are how models were trained and whether outputs reproduce protected works or characters.
Those questions are being litigated. A group of artists sued Stability AI, Midjourney and other companies in 2023, alleging their works were used for training without permission. In August 2024, as The Verge reported, Judge William Orrick allowed the case to proceed with some claims dismissed, adding a claim of induced copyright infringement against Stability. In the US, the Copyright Office’s position is that purely AI-generated material cannot be registered, though human contributions can be (U.S. Copyright Office).
When OpenAI’s GPT-4o image generation launched in March 2025, a viral wave of “Studio Ghibli-style” images reopened the debate about imitating a studio’s look without its involvement, even where no specific frame was copied.
Protections for artists
Researchers at the University of Chicago developed Glaze, which adds small perturbations to artwork that are hard for people to see but disrupt a model’s ability to learn and mimic the artist’s style (Glaze paper, arXiv:2302.04222). Such tools raise the cost of mimicry but are not a guarantee, because new models and techniques can reduce their effect.
Other options include opting out of training where platforms and datasets allow it, licensing work only to providers with clear terms, and adding Content Credentials to published work to record authorship.
Using style transfer responsibly
| Situation | Approach |
|---|---|
| Your own artwork or brand assets as references | Generally fine; you control the source |
| Public-domain art (for example, old masters) | Generally fine |
| Licensed style packs or commissioned references | Fine within the licence terms |
| A living artist’s name or work to produce saleable images in their style | Avoid: damages trust, may breach platform rules and can create legal risk |
| Recognisable characters or specific artworks | Avoid without permission: these are protected |
A short technical history
- Gatys, Ecker and Bethge (2015): showed that convolutional neural network features can separate an image’s “content” from its “style”, which made neural style transfer possible (arXiv:1508.06576).
- Johnson, Alahi and Fei-Fei (2016): trained feed-forward networks with perceptual losses, making style transfer fast enough for real time (arXiv:1603.08155).
- Huang and Belongie (2017): Adaptive Instance Normalization (AdaIN) allowed arbitrary styles in a single model (arXiv:1703.06868).
- Zhang, Rao and Agrawala (2023), ControlNet: added structural conditioning such as edges, depth and pose to diffusion models, so style can change while composition stays fixed (arXiv:2302.05543).
- Ye et al. (2023), IP-Adapter: let diffusion models take a reference image as a prompt alongside text, a common basis for style-reference features (arXiv:2308.06721).
Our analysis
Style transfer has evolved from a striking research demo into an everyday creative tool, and diffusion models with image references have made it far more controllable. Its most defensible uses rely on your own references, public-domain art or licensed material. Imitating living artists is where the technology meets real harm and unresolved law, and artists’ protections such as Glaze are helpful but imperfect. For creators, the practical rule is to treat style as something to develop, not borrow.