Teaching LLMs to Remember: Internalizing Documents Without Fine-Tuning

By Salvatore Raieli · Published March 4, 2026 · 1 min read · Source: Level Up Coding

Member-only story

|LLM|ADAPTERS|CONTEXT INTERNALIZATION|

Teaching LLMs to Remember: Internalizing Documents Without Fine-Tuning

Never forget a document: Moving from fine-tuning to context internalization

Salvatore Raieli10 min read·Just now

Long contexts are crucial for LLM reasoning but make inference slow and memory-intensive due to quadratic attention. While context distillation embeds prompt knowledge into model weights, it is too costly per document. Doc-to-LoRA (D2L) addresses this by meta-learning a hypernetwork that generates LoRA adapters in a single forward pass, internalizing context without retraining. It reduces latency and memory, outperforms standard CD under constraints, and enables fast, scalable model adaptation. — image created by the author using AI

There are lots of people who mistake their imagination for their memory. — Josh Billings

Large language models (LLMs) are increasingly used to analyze long documents. In the standard setup, the document must be inserted into the prompt each time the model is used. This approach is transient: the model does not retain the document after the interaction. It is also computationally expensive, as inference cost grows rapidly with context length, increasing latency and usage costs.

Supervised fine-tuning offers an alternative by embedding the document’s knowledge directly into the model’s parameters. However, fine-tuning is itself resource-intensive and impractical for frequent or dynamic updates.

Is there a way to internalize the knowledge of a document efficiently? One that is reusable, fast, and does not require repeatedly writing the full text as part of the prompt?

This article was originally published on Level Up Coding and is republished here under RSS syndication for informational purposes. All rights and intellectual property remain with the original author. If you are the author and wish to have this article removed, please contact us at [email protected].