Diffusion beats GPT at speed, but can it match its IQ?
bytedance's 8b illada diffusion model matches qwen2.5 base benchmarks, proving non-autoregressive architectures can compete with gpt-style transformers.
researchers from bytedance and renmin university introduced illada, an 8b parameter diffusion language model trained from scratch on 12 trillion tokens. on standard benchmarks, the base model achieved an average score of 63.9, slightly edging out the autoregressive qwen2.5 7b at 63.3. however, after instruction tuning, illada scored 67.1, significantly trailing qwen2.5 instruct's 77.1 due to gaps in math and code. the model relies on parallel bidirectional processing similar to image diffusion models, offering an alternative to standard token-by-token generation.