ByteDance study finds QA training beats OCR for long-context multimodal models
a bytedance study shows that training multimodal models on question-answer pairs improves long-context performance, while text transcription training actively hurts it.
bytedance and hkust released a study demonstrating that standard character recognition tasks disrupt multimodal model training for long contexts. instead, using question-answer pairs forces the model to learn efficient retrieval and information filtering. the resulting model, mmprolong, maintains stability up to 512,000 tokens and outperforms massive open-source rivals on benchmarks. additionally, the researchers found that a mix of document lengths is superior to exclusive long-context training, and these gains transfer to video understanding without specific training.