DiScoFormer unifies density and score estimation in a single transformer
huggingface released discoformer, a transformer that estimates both density and score across distributions without retraining. this generalizes classical methods to high dimensions and enables on-the-fly adaptation for generative modeling and scientific simulation.
huggingface published research on discoformer, a transformer designed to estimate the density and score of data distributions simultaneously without retraining. the model uses cross-attention to act as a learnable generalization of kernel density estimation, maintaining accuracy in high dimensions where classical methods fail. it features a shared backbone with two heads, utilizing a label-free consistency loss to adapt to out-of-distribution data instantly without ground truth. this approach bridges the gap between flexible neural networks and interpretable classical methods, with applications in diffusion-based generation and scientific simulations like plasma modeling.