The Transformer architecture changed sequence modeling by introducing attention, but its computational cost rises sharply as sequence length grows. That limits its use for very long contexts, including full-document understanding and genomic analysis.
Researchers have explored alternatives such as efficient linear recurrent neural networks (RNNs) and state space models (SSMs) like Mamba-2. These approaches scale faster by compressing context into a fixed-size state, but that can miss important information in very long sequences.
In two new papers, Titans and MIRAS, the authors introduce an architecture and theoretical blueprint that combine the speed of RNNs with the accuracy of transformers. Titans is the specific architecture, while MIRAS is the theoretical framework for generalizing these approaches.
Together, they advance test-time memorization, which allows an AI model to maintain long-term memory by using more powerful “surprise” metrics while the model is running and without dedicated offline retraining.
The MIRAS framework, as demonstrated by Titans, moves toward real-time adaptation. Instead of compressing information into a static state, the architecture actively learns and updates its own parameters as data streams in. That could help a model incorporate new, specific details into its core knowledge instantly.
Source: research.google.
Companies can share verified announcements through Newz9’s international press release submission page.
