Definition
Masked language modeling (MLM) is a self-supervised pretraining task, popularized by BERT, in which some tokens of an input sentence are hidden and the model must predict the originals, essentially a "fill in the blanks" exercise. Because the model can use the words on both sides of a blank, it learns deeply bidirectional representations of language.
How BERT Does It
BERT randomly selects 15% of the tokens in each sequence. Of those, 80% are replaced with a special [MASK] token, 10% are swapped for a random word, and 10% are left unchanged. The model is then trained with a cross-entropy objective to recover the original token at each masked position, conditioning on the full surrounding context to the left and the right. That random mix keeps the model from over-relying on the literal [MASK] symbol, which never appears at inference time.
Why It Matters
MLM produces general-purpose language representations from nothing but raw text, no manual labels required. Those pretrained weights can then be fine-tuned for search, classification, or extraction on your own corpus, on your own hardware. For a sovereign builder, that means you can adapt a capable language model to private documents without ever uploading them to an external service.
MLM is a canonical example of self-supervised learning, and the representations it produces are sized by the model's embedding dimension.
In Simple Terms
Masked language modeling (MLM) is a self-supervised pretraining task, popularized by BERT, in which some tokens of an input sentence are hidden and the model…