Soft Prompt

A soft prompt is a set of learnable vectors — often called virtual tokens — that are concatenated with a model's input embeddings and trained by gradient descent to steer a frozen language model toward a task. It is the continuous-valued counterpart to a hard prompt, the ordinary handwritten instruction you type into a model. Where a hard prompt searches for the right words, a soft prompt skips words entirely and searches the model's embedding space directly for whatever vectors best trigger the desired behavior.

Hard versus soft

Hard prompts are discrete: sequences of real tokens from the vocabulary, human-readable and human-written. Their weakness is brittleness — accuracy can swing wildly on a rephrasing, and finding a good one is manual trial-and-error over a space that gradient descent cannot search, because discrete tokens are not differentiable. Soft prompts dissolve that constraint. The virtual tokens live in continuous embedding space, unbound by the vocabulary: they need not correspond to any real word, and in practice they usually do not — inspecting a trained soft prompt's nearest vocabulary neighbors yields little that reads as an instruction. What is lost in interpretability is gained in optimization: the prompt is trained end-to-end against task data, and a tuned soft prompt reliably outperforms handcrafted text on the task it was trained for. Crucially, only the soft-prompt tensor updates during training; the model's own weights stay frozen. One base model can therefore serve many tasks, each specialization living in a prompt artifact of a few kilobytes to a few megabytes — swap the vectors, swap the skill.

The soft-prompt family

Several parameter-efficient methods build on this one idea and differ mainly in where the learnable vectors go and how they are produced. Prompt tuning is the minimal form: directly optimized vectors prepended at the input embedding layer only — famously approaching full fine-tuning quality as model scale grows. Prefix tuning injects learned key-value prefixes into every transformer layer via a reparameterization network, buying more steering capacity for harder generation tasks. P-tuning generates its virtual tokens through a small prompt encoder (typically an LSTM) and may interleave them anywhere in the sequence rather than only at the front. All of them sit inside the broader PEFT landscape alongside weight-adapting methods like LoRA — the practical dividing line being that soft prompts modify what the model reads, while adapters modify what the model is.

Practical notes for sovereign builders

For anyone specializing open-weight models on their own hardware, soft prompts are among the cheapest levers available: training touches only thousands to a few million parameters, memory overhead is minimal, and the artifact is small enough to version, share, and hot-swap trivially — one frozen base model plus a drawer of task prompts is a genuinely practical serving architecture for a home lab. Three honest caveats. Capacity: for hard generative tasks, weight-adapting methods such as LoRA usually win, so match the tool to the job. Context cost: virtual tokens occupy sequence positions, spending a slice of your context window on every request. Portability: a soft prompt is meaningless outside the exact base model it was trained on — its vectors are coordinates in that specific model's embedding space, a useful reminder that in the soft-prompt world, as in the rest of the sovereign stack, the value lives in what you can run and reproduce yourself, not in what you must rent. Compare full fine-tuning for the maximal-capacity, maximal-cost end of the same spectrum.

A closing perspective: soft prompts quietly reframe what "a model" is. Once the base weights are frozen and shared, the valuable, differentiated artifact becomes the small trained steering vector — cheap to make, trivial to distribute, useless without the commons it plugs into. That is a healthy shape for an ecosystem: heavyweight foundations maintained openly, lightweight specializations owned by whoever trains them. It mirrors the arrangement we favor everywhere else in the sovereign stack — shared open protocols, personal keys — and it is a large part of why local fine-tuning is economically viable for individuals at all.