What If LLMs Remembered in Vectors Instead of Words?
What happens when you give a language model a 60,000 token document and ask it a question about paragraph 47? It forgets. Or worse, it makes something up. This is the long context problem and it is one of the most important open challenges in language modeling today. Context windows keep growing (Gemini has 1M tokens, Claude has 200K) but models still struggle with information buried deep in the middle of long inputs....