Text degeneration creates inference bottlenecks
huggingface research identifies text degeneration as a structural failure in autoregressive models, causing small numbers of requests to consume nearly half of total inference time.
researchers found that requests hitting maximum-token limits and exhibiting n-gram repetition patterns drastically increased inference latency. standard fixes like repetition penalties or temperature adjustments are insufficient because they fail to address the root cause: the model's training objective. maximum-likelihood estimation optimizes for immediate next-token probability without considering the holistic sequence quality, leaving models prone to infinite loops that never emit end-of-sequence tokens.