Abstract
Multi-teacher distillation has emerged as a way to combine complementary teacher models into a single student model that exhibits the strengths of all its teachers. The student is trained to mimic the output of the teachers on a set of images, typically the union of the individual teacher's training sets, assuming this data is available. In this paper, we question that assumption and explore alternative options. We first study how far one can go when distilling from teachers fed with different types of noise. Then, we show that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation: we propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation. Experiments show that our most effective samples, IDeaL, lead to strong students that successfully capture complementary information from the teachers, yielding surprisingly competitive results that substantially narrow the gap with students distilled from real images. Moreover, given a limited budget of 1K images for distillation, students distilled using our IDeaL samples match or surpass the performance of those distilled using a 1K-image subset of ImageNet.
From Dead Leaves to IDeaL: optimizing pixels for
multi-teacher distillation
We ask how far we can get with no real data at all when distilling from multiple teachers. Starting from synthetic noise, we let the teachers optimize its pixels into samples that are maximally informative to all of them at once. The result, IDeaL, closes most of the gap to real images and even outperforms on small budgets.
More qualitative examplesNot all noise is created equal
We first replace real images with Gaussian Noise and procedural synthetic noise (Dead Leaves) during the distillation process and observed that Dead Leaves perform best among alternatives. Thanks to our pixel-optimization method, students distilled with improved dead leaves, (IDeaL), achieves much better performances.
Gaussian Noise
Every patch looks the same. Self-attention has nothing to attend to, and the student learns almost nothing.
Dead Leaves
Overlapping shapes randomly drawn by a formula.
IDeaL (ours)
Dead Leaves samples, pixel-optimized through the frozen teachers to be maximally informative for all of them at once.
Method
How to optimize the pixels? Two losses, one idea: decorrelate
Real images are visually rich, which leads to diverse patch representations in ViT self-attention layers. In contrast, noise yields nearly homogeneous patch representations, limiting the expressiveness of self-attention layers. So we treat pixels as parameters and push representations apart: both for each image, and across the batch. No privileged information from the training datasets are used, no class labels, and no dataset-specific statistics.
Patch decorrelation lossLPD
We extract the attention output at each layer and average across heads to obtain per-patch representations. We compute the pairwise cosine similarity matrix, where entry (j,k) is the cosine similarity between the representations of the j-th and k-th patches. The loss encourages the off-diagonal similarities to be 0.
patches
sim. matrix → identity
Image decorrelation lossLID
For each image in the batch, we extract the final-layer CLS embedding and compute the pairwise cosine similarity matrix on image level. Next, we minimize the global representations cosine similarity, so that each generated image has a distinct global representation as perceived by each teacher.
feature space
sim. matrix → identity
Visual examples
These samples never try to look real; they try to be informative for the teachers that they have been optimized.
Flip the losses on and off
Each loss optimizes the images differently and it affects the student performances distilled from them. All results are evaluated using 10k samples, with the Dead Leaves as our baseline.
ImageNet
66.6
Transfer
65.1
Seg.
29.1
Depth
0.655
Results
How close can IDeaL get?
We adopt the UNIC multi-teacher distillation framework and evaluation protocol to compare the performance of students distilled with Gaussian Noise, Dead Leaves, ImageNet and IDeaL, on four downstream tasks: ImageNet classification, transfer learning, semantic segmentation and depth estimation.
BibTeX
@inproceedings{yavuz2026ideal,
title = {{IDeaL}: Data-Free Multi-Teacher Distillation via Improved Dead Leaves},
author = {Yavuz, Feyza and Sar{\i}y{\i}ld{\i}z, Mert B{\"u}lent and Larlus, Diane},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2026}
}