Researchers have introduced LiME (Lightweight Mixture of Experts), a parameter-efficient method for adapting large AI models to dozens of tasks at once — achieving competitive accuracy with up to 4x fewer trainable parameters and 29% faster training than current approaches, according to the authors.

Multi-task learning — training a single model to handle many different tasks simultaneously — has become a central challenge as AI systems are deployed across diverse applications. Current approaches combine Mixture of Experts (MoE) with parameter-efficient fine-tuning (PEFT), a technique that adapts large pre-trained models by adding small, trainable modules rather than retraining every parameter. The problem: existing MoE-PEFT methods require a separate adapter module for each expert, so the number of trainable parameters grows linearly as you add more experts, quickly making these systems expensive to train.

How LiME Replaces Adapter Replication

LiME's core insight is that you don't need a separate adapter per expert to achieve specialisation. Instead, the method uses a single shared PEFT module and applies lightweight "expert vectors" to modulate its output — think of it as adjusting the knobs on one shared piece of equipment rather than building a separate machine for each task. This design keeps the parameter count low while still allowing each expert to behave differently.

Because the approach doesn't depend on any specific adapter architecture, it generalises to virtually any PEFT method, including LoRA and others, rather than being locked to one design.

The method achieves expert specialisation through lightweight modulation rather than adapter replication, reducing expert parameters while generalising to any PEFT method.

The authors also introduce what they call zero-parameter routing — a mechanism for deciding which expert handles which input without adding any learned router parameters per layer. Standard MoE systems require these routers, which add parameters and complexity. LiME instead leverages representations already present in the frozen and adapted parts of the model to make routing decisions, contributing to its overall efficiency.

New Routing Mechanisms: N-gram Windows and Auto Top-K

Beyond the core architecture, LiME adds two routing innovations. The first is n-gram windowed routing, which considers a short window of context when making routing decisions rather than looking at tokens in isolation — relevant for tasks involving sequential data like text or video. The second is Auto Top-K, an adaptive expert selection mechanism that adjusts how many experts are activated based on the model's own confidence in its routing decisions, rather than using a fixed number.

The authors also provide theoretical backing for their design. They prove two properties: first, that using more experts preserves more task-relevant information; second, that LiME's modulation approach can approximate the performance of full expert-specific PEFT modules with a bounded error — meaning the gap in capability has a mathematical ceiling, not an open-ended unknown.

Tested Across 47 Tasks Spanning Text, Image, and Video

The benchmark used for evaluation is MMT-47, a multimodal multi-task benchmark covering 47 tasks across text, image, and video modalities. Evaluating across this range is important because real-world deployments increasingly require models to handle different types of data, not just one.

Across these tasks, LiME matched or outperformed existing MoE-PEFT baselines. These benchmarks are self-reported by the research team and have not yet undergone independent peer review, as the paper was posted directly to ArXiv. The efficiency gains are concrete: up to 4x reduction in trainable parameters and up to 29% faster training compared to the baselines the authors selected for comparison.

The reduction in parameters matters for practical deployment. Fine-tuning large models is computationally expensive, and methods that reduce this cost without sacrificing performance open up multi-task adaptation to organisations and researchers with more limited infrastructure. Faster training also shortens iteration cycles, making it easier to experiment and improve models.

Broader Implications for Efficient Multi-Task AI

LiME sits within a broader research trend of making large model adaptation more tractable. PEFT methods like LoRA have already dramatically reduced the cost of fine-tuning; combining them with MoE to handle multiple tasks has been an active area, but the linear parameter scaling of existing approaches has been a recognised bottleneck. LiME's approach — shared modules plus lightweight modulation — offers a practical path around that bottleneck without sacrificing architectural flexibility.

The zero-parameter routing mechanism is particularly notable. Router parameters are a persistent overhead in MoE systems, and eliminating them per layer without a drop in performance, if the results hold up under independent scrutiny, would represent a meaningful simplification of the architecture.

Next steps for this line of research will involve independent replication on different model families and benchmarks, as well as testing whether LiME's performance advantages hold as model scale increases.

What This Means

For teams looking to deploy large models across multiple task types without prohibitive compute costs, LiME offers a credible, architecturally flexible alternative to current MoE fine-tuning methods — though independent validation beyond the authors' own benchmarks will test its claims.