Introduction
In the rapidly evolving field of multimodal machine learning, H Company has introduced NeoMME, a novel family of single-tower, bidirectional multimodal encoders that challenge conventional architectures by eliminating both the pretrained vision tower and the causal decoder. This approach represents a significant departure from the dominant paradigm of using separate components for vision and language processing, as seen in models like ColPali. The NeoMME family includes two variants: one with 260 million parameters and another with 800 million, each designed for efficient and effective retrieval tasks across multilingual text and low-resolution images.
What is NeoMME?
NeoMME is a multimodal encoder—a type of neural network architecture designed to process and encode information from multiple modalities (e.g., text and images) into a shared embedding space. Unlike traditional architectures that rely on a vision tower (a pretrained component specialized for image processing) and a causal decoder (used for generating sequential outputs, like in language models), NeoMME processes both text and image inputs within a single Transformer architecture. This means all components are trained jointly, without the need for separate pretraining stages or specialized modules for vision.
How Does NeoMME Work?
The core innovation in NeoMME lies in its masked discrete-diffusion pretraining objective. This method trains the model by reconstructing input data (both text and image patches) from corrupted versions. Specifically, it employs a discrete diffusion process—a technique that gradually corrupts data through a series of noise steps and then trains the model to reverse this process. For image inputs, the model receives 32×32 patches, which are processed in a unified manner alongside text tokens.
NeoMME also features a dual retrieval head structure: a dense retrieval head for fast similarity matching and a late-interaction retrieval head for more nuanced, context-aware matching. This dual setup allows the model to be both efficient and accurate in retrieval tasks.
Additionally, the model supports index compression, specifically a 255× compression ratio, enabling efficient storage and retrieval. With a throughput of 51.3 pages per second on a single NVIDIA L40S GPU, NeoMME is optimized for real-time applications while maintaining high performance.
Why Does It Matter?
NeoMME addresses key limitations in current multimodal architectures. By removing the vision tower and causal decoder, it reduces computational overhead and simplifies training pipelines. This is particularly valuable in scenarios where efficiency and scalability are critical, such as in large-scale search systems or real-time applications. Furthermore, the model’s ability to process multilingual text and low-resolution images within a single encoder demonstrates the potential for more unified and adaptable multimodal systems.
The ViDoRe v3 benchmark results are also significant: the 260M variant achieves a nDCG@10 score of 0.523, indicating strong retrieval performance. This validates the effectiveness of the proposed architecture and training method in handling complex multimodal retrieval tasks.
Key Takeaways
- NeoMME is a single-tower multimodal encoder that avoids the need for a pretrained vision tower or causal decoder.
- It employs a masked discrete-diffusion pretraining objective for joint text and image processing.
- The model supports dual retrieval heads for efficiency and accuracy in retrieval tasks.
- It achieves high indexing throughput (51.3 pages/sec) and significant compression (255×), making it suitable for scalable applications.
- Performance on ViDoRe v3 demonstrates competitive retrieval capabilities, with a 0.523 nDCG@10 for the 260M variant.
