Blog

Introducing NandiBERT

Open encoders that are state of the art in Indic and among the best in English, with Bahasa Indonesia, Malay and Arabic. Two sizes: 150M with a 16K window and 450M with a 32K window.

NandiBERT 150M and 450M: one token attending to every other token in both directions, with up to 32K tokens of context
Every token in a NandiBERT encoder attends to every other token, in both directions, across up to 32,768 tokens.

Encoders do the quiet work behind search, classification, tagging and retrieval-augmented generation. Until now, a team building for English and Indic languages had to choose: an English encoder that is strong in English, or an Indic encoder that is strong in Indic. NandiBERT is built so that choice goes away.

We are releasing two sizes with open weights under Apache 2.0. NandiBERT-150M reads 16,384 tokens and NandiBERT-450M reads 32,768. Both share one 100K-token vocabulary across English, 11 Indic languages, Bahasa Indonesia, Malay and Arabic.

Until now, IndicBERT v2 was the state of the art for Indic encoders. NandiBERT takes that place: NandiBERT-450M beats IndicBERT v2 on all seven Indic tasks we report, and NandiBERT-150M beats it on average with 45% fewer parameters. In English, both models are among the best, ranking second on average against English-only encoders of their size, ahead of ModernBERT and Ettin. This post covers the design and the numbers.

Overview

One Encoder, Many Languages

NandiBERT is a family of bidirectional Transformer encoders trained with masked language modeling. Both sizes share the same design and the same tokenizer, so what you learn on one carries to the other.

2sizes

149M and 449M parameters, open weights under Apache 2.0

32Kcontext

16,384 tokens for 150M, 32,768 for 450M

100Kvocabulary

100,352 tokens, shared across the family

1.1Ttokens

in the pre-training corpus

SpecNandiBERT-150MNandiBERT-450M
ArchitectureBidirectional Transformer encoderBidirectional Transformer encoder
Parameters149M449M
Layers2828
Hidden size7681,024
AttentionGrouped-query: 6 query heads, 2 key-value heads, head dimension 128, QK-normGrouped-query: 8 query heads, 2 key-value heads, head dimension 128, QK-norm
Feed-forwardSwiGLU, intermediate size 1,280SwiGLU, intermediate size 3,712
EmbeddingsFactorized, rank 192Factorized, rank 512
Vocabulary100,352 tokens100,352 tokens
Context16,384 tokens32,768 tokens
PositionsRoPE, base 1,000,000RoPE, base 1,000,000
ObjectiveMasked language modeling, whole-word maskingMasked language modeling, whole-word masking
LicenseApache 2.0Apache 2.0

Source: the NandiBERT-150M and NandiBERT-450M model cards and configs.

Architecture

Deep, Narrow and Built for Long Inputs

Both models are 28 layers deep. The 150M model is 768 wide and the 450M model is 1,024 wide, so most of the capacity goes into depth, where an encoder refines what each token means in context.

Attention

Grouped-Query, QK-Norm

Query heads share two key-value heads, which keeps attention memory low on long inputs. Queries and keys are normalized before attention.

Feed-forward

SwiGLU

A gated feed-forward layer: 1,280 wide in the 150M model and 3,712 in the 450M model.

Positions

RoPE, Base One Million

Rotary positions with a base of 1,000,000, so positions stay distinct across 32,768 tokens.

A Big Vocabulary Without the Weight

A 100,352-token vocabulary is what lets one model cover many scripts without breaking them into fragments. Stored as a plain table, it would cost 77M parameters in the 150M model, half its budget. So the table is factored: tokens are looked up in a narrow table and a small projection lifts them to the model's width.

Figure 1 · Embedding Parameters, Full Against Factorized

NandiBERT-150MFull table · 100,352 × 76877.1MFactorized · rank 19219.4MNandiBERT-450MFull table · 100,352 × 1,024102.8MFactorized · rank 51251.9M
Parameters in the token embedding, in millions. Factorized figures include the projection to the model width.

That returns about 58M parameters to the encoder layers in NandiBERT-150M and about 51M in NandiBERT-450M.

Indic

State of the Art in Indic

Until now, IndicBERT v2 was the state of the art among Indic encoders. We compared both NandiBERT models with it on seven tasks: four for classification (IndicSentiment, IndicXNLI, IndicCOPA and IndicXParaphrase), two for structure prediction (Naamapadam NER and MASSIVE slot-filling) and one for question answering (IndicQA).

Figure 2 · NandiBERT-450M Against IndicBERT v2

IndicBERT v2 · 272MNandiBERT-450M · 449MIndicSentiment88.392.9IndicXNLI74.381.3IndicCOPA63.076.1IndicXParaphrase57.077.8Naamapadam NER72.472.5MASSIVE Slot-fill57.361.0IndicQA49.251.0Average65.973.2
Scores on seven Indic tasks, plus their average. All panels share one 0–100 scale. Higher score in each panel in bold.

NandiBERT-450M is ahead on all seven tasks and averages 73.2 against 65.9. The widest margins are on IndicXParaphrase (77.8 against 57.0) and IndicCOPA (76.1 against 63.0). That makes NandiBERT-450M the new state of the art for Indic encoders.

NandiBERT-450M against IndicBERT v2
ModelParamsIndicSentimentIndicXNLIIndicCOPAIndicXParaphraseNaamapadam NERMASSIVE Slot-fillIndicQAAvg.
IndicBERT v2272M88.374.3635772.457.349.265.93
NandiBERT-450M449M92.981.376.177.872.5615173.23

Figure 3 · NandiBERT-150M Against IndicBERT v2

IndicBERT v2 · 272MNandiBERT-150M · 149MIndicSentiment91.291.3IndicXNLI73.775.9IndicCOPA64.666.4IndicXParaphrase56.960.6Naamapadam NER73.277.4MASSIVE Slot-fill58.154.0IndicQA48.346.0Average66.667.4
Scores on seven Indic tasks, plus their average. All panels share one 0–100 scale. Higher score in each panel in bold.

NandiBERT-150M is ahead on five of seven tasks and on average (67.4 against 66.6), with 149M parameters to IndicBERT v2's 272M. IndicBERT v2 keeps the lead on MASSIVE slot-filling and IndicQA.

NandiBERT-150M against IndicBERT v2
ModelParamsIndicSentimentIndicXNLIIndicCOPAIndicXParaphraseNaamapadam NERMASSIVE Slot-fillIndicQAAvg.
IndicBERT v2272M91.273.764.656.973.258.148.366.57
NandiBERT-150M149M91.375.966.460.677.4544667.37

IndicBERT v2 scores as reported on each NandiBERT model card. Higher score in each column in bold.

English

Among the Best in English

We compared each NandiBERT model with English encoders of similar size on ten benchmarks: five for classification (GLUE, SQuAD 2.0, MultiCoNER v2, Few-NERD and RewardBench) and five for retrieval (BEIR, RAR-b Code, RAR-b Math, MLDR and LongEmbed). Every other model in these comparisons is English-only, so NandiBERT competes with specialists on their own ground.

Figure 4 · English Average, 150M Class

UltraBERT-150M152M63.2NandiBERT-150M149M61.0Ettin-Encoder-150M149M58.8ModernBERT-base149M58.6GTE-en-MLM-base137M54.1NomicBERT-2048137M51.9MosaicBERT-base137M50.6RoBERTa-base125M50.4BERT-base110M47.0
Mean of the ten English benchmarks. Higher is better.

NandiBERT-150M averages 61.0, second to UltraBERT-150M and ahead of Ettin-Encoder-150M and ModernBERT-base. It ranks first on Few-NERD, and second on SQuAD 2.0, RewardBench and all five retrieval tasks.

English benchmarks, 150M class
ModelParamsGLUESQuAD 2.0MultiCoNER v2Few-NERDRewardBenchBEIRRAR-b CodeRAR-b MathMLDRLongEmbedAvg.
BERT-base110M84.776.353.866.659.436.76.627.52929.346.99
RoBERTa-base125M86.483.756.567.163.239.611.733.428.733.850.41
MosaicBERT-base137M85.479.457.865.762.541.412.730.929.640.550.59
NomicBERT-2048137M8482.760.166.964.14213.830.229.94551.87
GTE-en-MLM-base137M85.682.759.266.463.442.41233.637.158.554.09
ModernBERT-base149M88.486.757.466.76842.629.646.137.763.258.64
Ettin-Encoder-150M149M88.987.457.466.570.342.431.847.135.960.158.78
UltraBERT-150M152M89.589.257.967.47345.239.155.443.57263.22
NandiBERT-150M149M87.187.656.968.171.244.532.849.441.271.160.99

Figure 5 · English Average, 400M Class

UltraBERT-380M380M66.8NandiBERT-450M449M63.3Ettin-Encoder-400M395M63.2LFM2.5-Encoder-350M355M62.8ModernBERT-large395M61.0GTE-en-MLM-large434M55.0RoBERTa-large355M54.0NeoBERT222M50.5BERT-large334M49.9
Mean of the ten English benchmarks. Higher is better.

NandiBERT-450M averages 63.3, second to UltraBERT-380M and ahead of Ettin-Encoder-400M, LFM2.5-Encoder-350M and ModernBERT-large. It ranks first on Few-NERD. At 449M parameters it is also the largest model in this table.

English benchmarks, 400M class
ModelParamsGLUESQuAD 2.0MultiCoNER v2Few-NERDRewardBenchBEIRRAR-b CodeRAR-b MathMLDRLongEmbedAvg.
BERT-large334M85.281.857.767.862.638.612.23329.83049.87
RoBERTa-large355M88.989.462.368.168.542.820.337.829.532.253.98
GTE-en-MLM-large434M87.687.26367.162.239.413.53240.85754.98
NeoBERT222M8986.657.466.457.642.917.333.421.732.450.47
LFM2.5-Encoder-350M355M89.287.262.767.968.547.338.650.244.571.762.78
Ettin-Encoder-400M395M90.890.462.367.876.745.938.55440.964.363.16
ModernBERT-large395M90.489.860.967.672.845.23446.139.963.561.02
UltraBERT-380M380M9190.963.568.378.847.74761.744.174.966.79
NandiBERT-450M449M89.989.761.468.574.746.137.452.841.4671.463.34

UltraBERT leads the English average in both size classes, and it is English-only. NandiBERT carries Indic, Bahasa Indonesia, Malay and Arabic in the same weights, and still places second.

Long Context

Whole Documents in One Pass

A contract, a court judgment or a research paper rarely fits in 512 tokens. NandiBERT-150M reads 16,384 tokens and NandiBERT-450M reads 32,768, so a whole document can go through the encoder at once instead of being cut into chunks that lose their context.

LongEmbed

71.1 and 71.4

Long-document retrieval for NandiBERT-150M and NandiBERT-450M. The 150M model is second of nine in its class.

MLDR

41.2 and 41.5

Long-document retrieval for the two sizes. The 150M model is second of nine in its class.

Family

The NandiBERT Family

Both encoders are on Hugging Face with open weights.

Encoder

NandiBERT-150M

149M parameters and a 16K window, for fast classification, tagging and reranking.

Encoder

NandiBERT-450M

449M parameters and a 32K window, for long documents and the strongest Indic results.

Embeddings

EmbeddingNandi: Coming Soon

An encoder becomes a search engine once it learns to turn a whole passage into one vector. EmbeddingNandi is that next step: sentence-embedding models fine-tuned from NandiBERT-150M and NandiBERT-450M, for semantic search, retrieval-augmented generation, clustering and deduplication, across the same languages.

Run

Run It Yourself

The architecture is custom, so loading it needs trust_remote_code=True. Everything else is standard Hugging Face.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "FrontiersMind/NandiBERT-150M"  # or FrontiersMind/NandiBERT-450M
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(
    model_id, dtype=torch.bfloat16, trust_remote_code=True, device_map="cuda"
)

texts = [
    f"The capital of France is {tokenizer.mask_token}.",
    f"भारत की राजधानी {tokenizer.mask_token} है।",
]

for text in texts:
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**inputs).logits
    mask_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero()[0, 1]
    print(tokenizer.decode(logits[0, mask_index].argmax()))

For classification, tagging or retrieval, fine-tune the encoder on your task as you would any BERT-style model.

Fit

Where NandiBERT Fits

Reach for NandiBERT-150M when latency and memory matter: high-volume classification, tagging and reranking, on hardware you own. Reach for NandiBERT-450M when inputs are long or Indic accuracy matters most. Both are a strong place to start a fine-tune in any of the languages they cover, because one model serves all of them.

Know their edges. These are base encoders, so they need fine-tuning for most tasks, and they do not generate text. They are not sentence-embedding models yet; that is what EmbeddingNandi is for. And in English alone, an English-only specialist still leads the average in each size class.

The Takeaway

NandiBERT is one encoder for English, Indic, Bahasa Indonesia, Malay and Arabic. It is the state of the art in Indic, among the best in English against English-only specialists, it reads whole documents, and its weights are open for you to run on your own infrastructure. That is where Sovereign Superintelligence starts.