Zyphra - 1B context window for any LLM

Google currently has a competitive advantage with excellent recall on very large context windows on Gemini.

Most other LLMs, even with nominal context size in the 100k+ tokens category, show significantly degraded performance above 32k tokens.

If confirmed, this research will enable the competition to close the gap, potentially eliminating some of Google’s lead.