PorTAL: Portable Task Adaptation for LoRA

Parameter-efficient fine-tuning (e.g. LoRA) adapts a frozen LLM to a task, but the resulting adapter is welded to one base model: when a new model is released, the adaptation must be re-learned from scratch. We study portable task adaptation — learning a task adaptation once, in a base-agnostic form, and transferring it to new frozen models by refitting only a small per-model component.

Concretely, we learn a base-agnostic task latent $z_t$ together with a per-base hypernetwork decoder $D_b$ that generates per-layer LoRA for a frozen base; to port to a new base we freeze $z_t$ and the shared decoder core, and refit only a thin per-base alignment on a small amount of data.

This works both within a model family and, more strikingly, across model families, recovering most of the accuracy gain that a from-scratch per-task LoRA provides over the unadapted base. Freezing a task latent and shared decoder core learned on Qwen3-1.7B and 4B, then refitting only a thin per-base alignment, recovers ~98% of LoRA's lift on an unseen Qwen3-8B, and ~94% on Gemma-3, a different model family. Additionally the refit is data-efficient: PorTAL reaches the from-scratch-LoRA accuracy plateau with roughly half the calibration data, and at equal accuracy is consistently better-calibrated (lower held-out log-loss) than a from-scratch LoRA at every data size. It far outperforms transferring an existing adapter: a faithful Cross-LoRA baseline that aligns a strong source LoRA into the new model's activation space recovers only ~14% of the gain on the unseen 8B, versus our ~98% — learning a shared latent beats translating an existing adapter.

#Introduction and motivation

New language models arrive at an accelerating pace: the number of notable foundation models released per year rose from 2 in 2020 to 9 (2021), 32 (2022), and 149 in 2023Stanford HAI — AI Index Report 2024¹, and by 2024–2025 the frontier had become so crowded that the Elo gap between the top and 10th-ranked model collapsed from 11.9% to 5.4% in a single yearStanford HAI — AI Index Report 2025². In practice, a stronger base model — often several — ships every few months.

Adapting a model to a task, however, is a per-model cost that does not amortize across these releases. A fine-tune — full or LoRA — is welded to one base model's weight space; when the next model ships, the adaptation must be redone on the new base. Parameter-efficient methods lowered the unit cost (a LoRA on a 7B model runs ~$1–3k vs ~$12k for full fine-tuningAlloc Labs — The Hidden Cost of LLM Fine-Tuning³) but not its structure: you still pay data curation + a training run + evaluation once per (task, model), and full fine-tuning cost still scales with model size, which keeps growing.

The result is an economic treadmill: maintaining a portfolio of fine-tuned capabilities on the current frontier model costs roughly (cost-per-adaptation) × (#tasks) × (#model releases tracked). Because per-adaptation cost is roughly fixed while release cadence rises, this total grows over time — re-fine-tuning per model becomes the dominant, ever-growing cost of keeping a system specialized while also gaining the raw intelligence of each newer, smarter base.

#Related work

Our contribution is a combination and framing rather than a wholly new primitive, so we position it against three lines of work and credit each.

Hypernetworks that generate LoRA for a single base. Text-to-LoRA (Charakorn et al., ICML 2025)⁵, SHINE (2026)⁶, and Profile-to-PEFT (2025)⁷ amortize per-task or per-user adaptation into a single forward pass, but target a fixed base and generalize across tasks or users, not across models (Text-to-LoRA explicitly leaves cross-model transfer open).

Generating LoRA across architectures. LoRAGen (2025)⁸ uses a structural embedding (latent + module/layer embeddings) to emit LoRA for different bases, but is trained by reconstructing existing LoRAs; we share its decoder shape yet train end-to-end on task loss, and crucially, freeze a shared task latent and a shared core, refitting only a thin per-base alignment to reach an unseen base.

Transferring an existing LoRA across models. Cross-LoRA (Xia et al., 2025)⁹, LoRA-X (2025)¹⁰, and CAST (2025)¹¹ target the same goal we do, but by translating one already-trained adapter via subspace or activation-manifold alignment; we instead learn a base-agnostic latent and regenerate the adapter per base. Head-to-head this matters: a faithful Cross-LoRA (strong source + activation alignment, subsuming CAST) recovers ~14% of LoRA's lift on the unseen 8B versus our ~98% (§6.2).

In short, the pieces — hypernetwork→LoRA, cross-architecture generation, and cross-model transfer — all have prior art. Our contribution is combining them into a freeze the latent and a shared core, refit only a thin per-base alignment porting recipe, framed as a maintenance-cost answer to an accelerating model-release cadence and shown to dominate the adapter-transfer line empirically.

#Background: LoRA and LoRA hypernetworks

LoRA.LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)¹² For a frozen weight $W \in \mathbb{R}^{d_{out}\times d_{in}}$, LoRA learns a low-rank update $\Delta W = \tfrac{\alpha}{r} B A$ with $A \in \mathbb{R}^{r\times d_{in}}$, $B \in \mathbb{R}^{d_{out}\times r}$, $r \ll d$, applied as $y = Wx + \tfrac{\alpha}{r} B(Ax)$. Only $A,B$ train.

LoRA hypernetworks. Rather than training $(A,B)$ directly, a hypernetwork generates them from a conditioning input. Text-to-LoRA trains a hypernetwork to emit a full LoRA for a single base model from a task-description embedding, end-to-end through the frozen base. This amortizes per-task optimization into one forward pass — but is single-base and generalizes across tasks, not across models. Separately, Cross-LoRA, LoRA-X, and CAST transfer a single existing LoRA across bases via SVD-subspace or activation-manifold alignment (per-task, training-free, no shared latent). Our design borrows the hypernetwork-generates-LoRA idea but targets the orthogonal axis — cross-base transfer of a shared, learned task representation.

#Method

Let a frozen base $b$ have transformer layers $\ell = 1,\dots,L_b$ and adapted projections $W_{\ell,m} \in \mathbb{R}^{d^{out}_{\ell,m}\times d^{in}_{\ell}}$ for modules $m \in \{q\_proj, v\_proj\}$ (we extend $m$ to all attention and MLP projections in the full-module variant.) Let $\theta_b$ denote the frozen base parameters.

Task latent. Each task $t$ is represented by a base-agnostic latent $z_t \in \mathbb{R}^{d_z}$ ($d_z = 256$).

Per-base decoder (hypernetwork). A small network $D_b$ maps the latent + a layer embedding $e_\ell$ to the LoRA factors of each (layer, module):

Internally $D_b$ computes a per-layer trunk with FiLM conditioning on $z_t$,

then per-module heads emit low-rank factors that a per-base linear alignment maps to $(A_{\ell,m}, B_{\ell,m})$ (the Canonical decoder below), with the $B$-path zero-initialized so $\Delta W_{\ell,m}=0$ at start (training begins from the clean base). The generated adapter is injected as a standard LoRA delta:

Canonical decoder — a portable factorization. To make porting cheap, $D_b$ factors into a large base-agnostic core, shared across all bases, plus a thin per-base alignment. The core maps the latent and layer embedding to low-rank factors in a fixed canonical width $d_c$ that is independent of any base's dimensions; the per-base alignment is a learned per-layer embedding plus a linear map ($A = \hat A\,P^{in}$, $B = P^{out}\hat B$) projecting those canonical factors into the target base's actual per-module dimensions. The shared core is trained once on the seen bases; porting to a new base reuses the core verbatim and fits only the thin linear alignment — a small, near-linear cross-model map.

Objective (source training). Only $\{z_t\}$ and $D_b$ are trainable; $\theta_b$ is frozen. We minimize the gold-continuation NLL (loss only on answer tokens):

Multi-task training uses balanced per-task steps with EMA loss-normalization (without it, hard tasks collapse to chance).

Shared latent across bases. For a set of seen bases $\mathcal{B}$, we share both $\{z_t\}$ and the base-agnostic core across all bases, keeping only a per-base alignment, and optimize $\sum_{b\in\mathcal{B}}\sum_t \mathcal{L}_{b,t}$. Naively, a small base monopolizes the shared latent's gradient; we apply gradient-norm balancing on $z_t$, rescaling each base's accumulated gradient on $z_t$ to equal norm before the optimizer step.

Portability (the transfer step). Given an unseen base $b'$, we freeze $\{z_t\}$ and the shared core, and fit only the thin per-base alignment of a fresh decoder $D_{b'}$ on a small calibration set:

#Experimental setup

Tasks (14, standard multiple-choice). TruthfulQA, RTE, CB, COPA, WiC, WSC (SuperGLUE + TruthfulQA; higher-headroom), and BoolQ, ARC-Easy, ARC-Challenge, HellaSwag, OpenBookQA, WinoGrande, CommonsenseQA, SciQ (broader/bigger-eval).

Metric. Length-normalized log-likelihood over choices (acc_norm); we also report held-out log-loss (token-mean NLL of the gold continuation). §6.1–6.3 use best-epoch held-out selection (per-epoch eval) while §6.4 uses final-epoch eval. All are 3-seed means ± std.

Data. Up to 2,000 examples/task — a hard cap applied to both source training and the per-base alignment refit (some smaller tasks have fewer). §6.1–6.3 fit on the full 2,000/task; the data-efficiency study (§6.4) shows far fewer suffices. Evaluation sets range from 56 (CB) to 1,000 (BoolQ/WinoGrande/CSQA/SciQ); ~7,200 eval examples total on the 14-task suite.

Models. Seen bases: Qwen3-1.7B, Qwen3-4B. Unseen bases: Qwen3-8B (same family), Gemma-3-4B (different family). Per-task LoRA baselines: rank 16 on q/k/v/o + MLP. LoRA Hypernet/PorTAL (§6.1–6.3): rank 8 on q/v, $d_z=256$.

Experiments reported. (i) LoRA Hypernet vs per-task LoRA; (ii) portability to unseen bases within and across families; (iii) data-efficiency of the per-base alignment refit.

#Results

#Source base: LoRA Hypernet matches per-task LoRA

We first confirm that a hypernetwork — a learned task latent $z$ and the decoder, trained jointly on the source base — can match per-task LoRAs trained independently on the same base. It can: the generated LoRA Hypernet recovers ~94% of per-task LoRA's lift on average.

#Within-family portability: unseen Qwen3-8B

We then test the central portability claim: a latent and core learned on the seen bases should port to an unseen base by refitting only the thin alignment. It does — on an unseen Qwen3-8B, refitting only the per-base alignment recovers ~98% of the from-scratch LoRA's lift, reaching 0.792 against the 0.795 LoRA ceiling and far above the best cross-model transfer baseline. Freezing the latent is free, not a tax: training it natively on 8B from scratch reaches 0.785 ± 0.002 (~92%), statistically on par with the ported latent, so reusing a latent learned on other models costs essentially no quality while sparing the cost of re-deriving it per base.

#Cross-family portability

Finally, the cross-family test: does the same Qwen-trained frozen latent port to a different model family, not just a larger Qwen? It does — on Gemma-3, refitting only the lightweight per-base alignment recovers ~94% of its from-scratch LoRA's lift, so cross-family transfer is nearly lossless.

#Data efficiency: PorTAL vs from-scratch LoRA

We hypothesize that PorTAL amortizes task adaptation: a latent and core learned once on the seen bases should make every subsequent model cheap to adapt, so porting to a new base needs far less data than training a LoRA from scratch there. We test this on the unseen Qwen3-8B, sweeping the refit-set size for three methods: PorTAL q/v r8, PorTAL full r8, and a separate per-task r16-full LoRA. All use the same per-size schedule and eval over 3 seeds, reporting final-epoch acc and log-loss. Raw 14-task averages use a base-8B floor acc of 0.667 and log-loss of 3.819.

We find PorTAL is substantially more data-efficient. It reaches the strongest from-scratch LoRA's ~0.77 accuracy max with roughly 2× less calibration data: PorTAL q/v r8 hits that max at ~500 examples/task and full r8 at ~750, while from-scratch r16-full LoRA only settles there around ~1,000. PorTAL even ends slightly above it at the high end, 0.769 vs 0.762 at 2,000. Because the frozen base dominates per-step cost, reaching the target with half the data is also roughly half the adaptation FLOPs — the amortization we set out to test, since the latent and core are paid for only once.

PorTAL is also better-calibrated, with lower held-out log-loss than from-scratch LoRA at every data size. We read this as a calibration effect rather than a separate efficiency gain: held-out NLL penalizes overconfidence and a smaller adapter makes gentler logit edits, so the gap tracks adapter capacity.

#Future work

Capacity competition on a cluster of hard tasks. With best-epoch selection most tasks recover (10/14 reach ~80–110% of LoRA's lift), but a small cluster of harder commonsense/knowledge tasks underfits — OpenBookQA (~42% of lift), WinoGrande (~57%), HellaSwag (~61%) — the most-distinct tasks losing the shared rank-8 decoder's gradient competition. This is an optimization limit, not a capacity one: a rank-16 generated adapter did not help (it regressed and merely shifted which task starved). In the future we hope to pursue better multi-task optimization (per-task capacity, curriculum), or a small per-task residual on top of the shared decoder.

Amortized text-description variant. A natural extension replaces the free per-task latent with an encoder over a task description ($z_t = E(\mathrm{emb}(\text{desc}_t))$), so a brand-new task could be adapted zero-shot from its description alone (à la Text-to-LoRA), with no per-task training. We leave a full study to future work.

Other directions. Larger and instruction/generation tasks beyond multiple-choice; and theory on when a frozen latent suffices vs. when base-specific adaptation is required.

#Appendix A — Architecture and training details

#Architecture

Concrete configuration and dimensions for the §4 decoder:

• Task latent $z_t$: 256-d (a free per-task vector; or the encoder output in the amortized variant).
• Layer embedding $e_\ell$: learned nn.Embedding(L, 32) — breaks symmetry so one decoder yields a different adapter per layer.
• Trunk: Linear(256+32 → 512), GELU; FiLM modulate ($(1+\gamma)\odot h + \beta$, with $\gamma,\beta$ linear in $z_t$); Linear(512 → 512), GELU.
• Heads (one A- and one B-head per module $m\in\{q,v\}$): A-head Linear(512 → $r\cdot d^{in}$) reshaped to $[r,d^{in}]$; B-head Linear(512 → $d^{out}\cdot r$) reshaped to $[d^{out},r]$.
• Adapter: rank $r=8$ on q_proj/v_proj at every layer; scaling $\alpha/r=2$ ($\alpha=16$); injected by wrapping the module's real forward and adding $\text{scaling}\cdot(xA^\top)B^\top$.
• Init: B-heads and FiLM $\gamma,\beta$ zero-initialized, so the generated adapter is the identity ($\Delta W=0$) at start.
• Per-base alignment: a learned per-layer embedding plus per-module linear maps ($P^{in}$ projecting the canonical $d_c$ factors to the base's input width, $P^{out}$ to its output width).
• Parameters: the core holds the bulk; the latent table is 256 × #tasks.

#Training and hyperparameters

#Appendix B — Metrics: recovered lift vs retention

We report recovered lift; prior cross-model-transfer papers (Cross-LoRA, CAST) instead report retention (a ratio of accuracies). For a method $m$, unadapted base $b$, and from-scratch per-task LoRA $L$:

Retention is near 100% whenever there is little headroom ($\mathrm{acc}_b \approx \mathrm{acc}_L$) — the regime those papers operate in (their trained LoRA adds only ~1% over base), so it is not discriminative. We evaluate in a higher-headroom setting and therefore use recovered lift. For comparability, in retention terms our faithful Cross-LoRA reimplementation scores ~86% (within CAST's reported 85–95% band) while recovering only ~14% of the lift, whereas our porting scores ~99% retention / ~98% recovered lift — i.e. we win on both metrics, and the gap is only visible under the stricter one.

Research author — Ben Geist @b_geist

Want to keep up with our next AI experiments? Subscribe here and follow us on @RampLabs. We're also hiring across roles at Ramp.

#Citation

1.Stanford HAI — AI Index Report 2024

2.Stanford HAI — AI Index Report 2025

3.Alloc Labs — The Hidden Cost of LLM Fine-Tuning

4.Platonic Representation Hypothesis (Huh et al., 2024)

5.Text-to-LoRA (Charakorn et al., ICML 2025)

6.SHINE (2026)

7.Profile-to-PEFT (2025)

8.LoRAGen (2025)

9.Cross-LoRA (Xia et al., 2025)

10.LoRA-X (2025)

11.CAST (2025)

12.LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

Ben Geist

·July 1, 2026·Research