Friday, November 20, 2026


TECH


To adapt an AI model to a small device, keep its training notes concise

MIT CSAIL and Harvard researchers have introduced a memory-saving training method for large language models that could make on-device AI more practical. The work, published on the arXiv preprint server, challenges the common assumption that parameter-efficient fine-tuning, or PEFT, automatically lowers memory use because it updates only a fraction of a model’s parameters. According to the team, memory demand still spikes during training because models store extensive intermediate calculations, known as intermediate activation storage, rather than summarizing them.

To address that bottleneck, MIT EECS Ph.D. student and CSAIL researcher Irene Tenison and colleagues developed Low-memory Activation Rank Subspace, or LARS. The method pools those intermediate details into a more concise representation instead of retaining every calculation. The researchers report that LARS reduces LLM memory footprint by more than 23 percent on GPUs and about 47 percent on CPUs compared with leading PEFT techniques. On a Raspberry Pi 5 edge device with an 8 GB limit, LARS saved 9 percent more memory than the top PEFT approach, LoRA, at a context length of 128,000, with savings increasing as context length grew.

The team also found a trade-off: LARS saved significantly more memory than all baselines but was slightly less accurate than LoRA when models answered detailed questions on chips with more memory. Still, it saved the most memory and performed competitively on long-context prompts requiring detailed summarization. Across all questions, LARS saved about 10 GB more than the leading baseline on high-performance NVIDIA GPUs and twice as much memory on less powerful AMD CPUs. The efficiency gain also affected energy use: on a CPU, LARS used roughly 22 percent less power and energy than comparable baselines across short, intermediate and long prompts.

The researchers say the approach could support on-device LLMs that analyze long contexts, such as a fitness watch tracking heart rate over several minutes, real-time factory unit tracking, or robots processing surroundings locally without cloud dependence. LARS is also compatible with complementary techniques such as attention optimization, layer freezing and quantization, suggesting further memory and energy savings when used in tandem.

Imagine trying to run AI models directly on a device with very limited computing power, like a smartphone or health wearable. It needs to respond in real time, so instead of training it in the cloud, you have to do that on-device, where it eats up a lot of gigabytes (GBs). The likely result is an out-of-memory error.

To make training more efficient, engineers have often used a technique called “Parameter-Efficient Fine-Tuning” (PEFT). It tweaks only a fraction of a model’s parameters — the values that capture patterns learned when taking in data — under the assumption that fewer updates mean less memory usage. But that’s not exactly the case, according to new research from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Harvard University.

The issue, they argue, is that the models write out all their mental notes during training. As an LLM is learning from different data, it uses something called “intermediate activation storage” to essentially describe its extensive calculations in a hidden place. During these moments, memory usage peaks sharply, so CSAIL engineers developed the “Low-memory Activation Rank Subspace” (LARS) method, which pools those details into something more concise, instead of stashing every calculation. The approach reduces LLMs’ memory footprint by over 23% on GPUs and about 47% on CPUs compared to top PEFT techniques.

“Imagine you’re attending a lecture,” says Irene Tenison, MIT electrical engineering and computer science (EECS) PhD student and CSAIL researcher. “And instead of summarizing what you’ve learned by transcribing it all, you recap it in a couple of sentences. That’s basically what LARS is doing to help us run AI on tiny devices. After all, models are fine-tuning their parameters more efficiently, but the hardware they rely on has memory limits, which we don’t upgrade as often.”

Tenison adds that LARS could be helpful when you want an on-device LLM to analyze long contexts. For example, if you’re wearing a fitness watch, you may want it to analyze your heart rate over the span of a few minutes. On a tiny device, this is challenging because of memory limits, but CSAIL’s approach could make that kind of intelligence possible directly on your hardware.

LARS might also be useful in tracking different units in a factory in real time, one day. That way, AI models can oversee workflows without relying on the cloud to process all their data. Likewise, this more efficient training approach could potentially help a robot process its surroundings locally.

Memory (and energy) saved...To better understand how much memory LARS saves, Tenison and her colleagues tried their method on Raspberry Pi 5, a small edge computing device with an 8 GB limit. They used LARS and top PEFT approach, “LoRA,” to adjust an LLM’s parameters (also called fine-tuning), where their approach saved 9% more memory at a 128,000 context length. Notably, savings increased with context length.

When the team asked LLMs to answer more detailed questions on chips with more memory available, the researchers noticed a subtle tradeoff. LARS saved significantly more GBs than all baselines, but was slightly less accurate than LoRA. Fittingly, MIT CSAIL’s approach saved the most GBs and performed the most competitively on long-context prompts, where it had to summarize detailed information. Across all questions, LARS saved about 10 GB more than its leading baseline on high-performance NVIDIA GPUs, and on less powerful AMD CPUs, it saved twice as much memory.

But LARS’ memory efficiency also has a downstream effect on energy consumption, as Tenison and her coauthors later found out. They tasked an LLM with short, intermediate, and long prompts on a CPU and observed that their method used roughly 22% less power and energy than comparable baselines.“The data we’ve collected so far shows LARS may be a more sustainable LLM fine-tuning approach,” says Tenison.  

Shrinking the memory footprint even further...To maximize memory savings, MIT CSAIL’s method can be combined with approaches that address other parts of the training process, or the ways it completes prompts.

For instance, tools that optimize attention mechanisms (basically which parts of a prompt a model focuses on or how the prompts are processed within the kernels), or freeze certain layers processing the data, or partly or fully quantize a model, are compatible with LARS. This is because other tools don’t address models’ intermediate, hidden calculations.

“These approaches are actually complementary,” says Tenison. “We think that if you use them in tandem, you’ll save even more energy.”

The project was introduced in a paper that was presented at the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL) in November. Tenison authored it with several MIT CSAIL colleagues: Stella Ahn ‘25, SM ‘26; visiting researcher and Harvard student Miriam Kim; Broad Institute of MIT and Harvard researcher Ebtisam Alshehri; and senior author Lalana Kagal, who is a Principal Research Scientist at the lab.

Researchers from MIT CSAIL and Harvard University introduced LARS (Low-memory Activation Rank Subspace), a fine-tuning method that compresses intermediate calculation "notes" to cut memory use on small devices. Instead of saving every intermediate calculation during training, LARS pools those details into a more concise form.

Key memory and energy gains:

• GPU & CPU memory cuts: Reduces LLM memory footprint by over 23% on GPUs and about 47% on CPUs compared to top Parameter-Efficient Fine-Tuning (PEFT) baselines.

• Long-context savings: Saves 9% more memory than LoRA at a 128,000-token context length, with savings growing as context length increases.

• Hardware impact: Saves roughly 10 GB more on high-performance NVIDIA GPUs and twice as much memory on AMD CPUs across tested questions, while cutting CPU power and energy use by about 22%.

How it works & trade-offs:

• Compressing "mental notes": Models normally store extensive intermediate activations during training, causing sharp memory spikes on memory-constrained hardware. LARS condenses these intermediate calculations rather than recording everything.

• Accuracy vs. efficiency: LARS can be slightly less accurate than LoRA when answering detailed questions on higher-memory chips, though it remains competitive on long-context summarization.

• Compatibility: Works alongside complementary optimizations—such as attention optimization, layer freezing, and quantization—because it targets intermediate hidden calculations rather than those stages.

Potential small-device uses:

• Wearables: Analyzing health data like heart rate over several minutes directly on a fitness watch.

• Edge & robotics: Helping robots process surroundings locally or tracking factory units in real time without relying on the cloud.

Researchers from MIT CSAIL and Harvard have developed the LARS (Low-memory Activation Rank Subspace) method to reduce memory usage during the fine-tuning of language models on small devices.

How it works:

• Fewer intermediate "notes": Instead of storing extensive intermediate calculations during training, LARS condenses this information into a more compact format.

• Focus on long contexts: Memory savings increase as the size of the context processed on the device grows.

• Compatibility with other techniques: It can be used alongside attention optimization, layer freezing, and quantization.

Results and limitations:

• Reduced memory usage: It lowers memory consumption by more than 23% on GPUs and approximately 47% on CPUs compared to advanced Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA.

• Gains in long-context scenarios and energy efficiency: It saved 9% more memory than LoRA with a 128,000-token context, spared about 10 GB of memory on NVIDIA GPUs, and reduced energy consumption by approximately 22% on CPUs.

• Trade-off: It shows slightly lower accuracy than LoRA when handling detailed queries on chips with larger memory capacities, yet it maintains competitive performance for long-context summarization tasks.

mundophone

No comments:

Post a Comment

  TECH Taming blue light: Researchers bring precision lasers onto a chip A newly developed chip-scale photonic system has united frequency s...