the Silent Killer: How mDeBERTa's Float16 Weights Broke Every Training Run

Anmol
March 18, 2026

The Silent Killer: How mDeBERTa's Float16 Weights Broke Every Training Run

March 2026 · Nirdesh Guragain · Debugging NLP Training

We spent an entire day chasing NaN losses across six different training configurations — and the root cause turned out to be a single default parameter in HuggingFace's AutoModel.from_pretrained().

If you're fine-tuning microsoft/mdeberta-v3-base and your training dies at ~500 steps with loss=0.0000 and grad_norm=nan, this post is for you.

The Setup

We were building a toxicity classifier for the EEUCA 2026 Shared Task on the GameTox dataset — 53K World of Tanks chat messages, 6 classes, extreme imbalance. Our architecture: mDeBERTa-v3-base with 22 auxiliary POS/lexicon features concatenated to the [CLS] embedding before a classification head.

Simple enough. Except nothing worked.

The Symptom

Every run followed the exact same pattern. Training would start fine — loss around 0.5-0.7, gradients healthy. Then somewhere between step 400-550:

Step 450 | loss=0.5955 | grad_norm=1.587 | lr=5.40e-06
Step 500 | loss=0.5471 | grad_norm=nan   | lr=6.00e-06
Step 550 | loss=0.0000 | grad_norm=nan   | lr=6.60e-06
Step 600 | loss=0.0000 | grad_norm=nan   | lr=7.20e-06
... (dead forever)

Loss collapses to exactly zero. Gradients become NaN. The model is brain-dead but doesn't crash — it just produces garbage for the rest of training.

What We Tried (None of This Worked)

Over the course of ~8 hours, we systematically eliminated every suspect:

1. Focal Loss → Regular Cross-Entropy. Maybe focal loss was crushing confident predictions to zero? Nope. Same crash at step ~500 with plain nn.functional.cross_entropy.

2. Removed class weights entirely. Maybe extreme inverse-frequency weights (50x for rare classes) were destabilizing? Set alpha=None. Same crash.

3. fp16=False, bf16=False. RTX 4090 should handle bf16 natively. Tried both, tried neither. Same crash. (bf16 actually triggered a CUDA driver failure that required a full machine reboot.)

4. Ditched HuggingFace Trainer for manual training loop. We noticed our working scripts all used manual loss.backward() → clip_grad_norm_ → optimizer.step(). Rewrote everything. Same crash.

5. Lowered learning rate from 2e-5 to 5e-6. Increased weight decay to 0.06. Added label smoothing. Lowered grad clip to 0.5. Each individually. All combinations. Same crash.

6. Removed auxiliary features. Maybe the 22 POS/lexicon features were introducing NaN through the concatenation? Ran with just [CLS] → classifier. Same crash.

7. Initialized classifier head with std=0.02 (matching HuggingFace's internal init). Didn't help.

The maddening part: our older scripts using the exact same FocalLoss, same data, same hyperparameters worked perfectly — but they used DebertaV2PreTrainedModel.from_pretrained() instead of AutoModel.from_pretrained().

Finding the Root Cause

The breakthrough came from instrumenting the model's forward pass with debug prints on every tensor:

Batch 1: loss=0.6824 | encoder hidden: finite ✓ | cls: finite ✓
Batch 2: loss=NaN    | encoder hidden: NaN ✗    | cls: NaN ✗

The encoder itself was producing NaN on batch 2. Not the classifier, not the features, not the loss function — the backbone transformer weights were corrupted after the very first optimizer update.

We inspected the model's embedding layer:

>>> model.encoder.embeddings.word_embeddings.weight.dtype
torch.float16

There it was.

The Bug

AutoModel.from_pretrained("microsoft/mdeberta-v3-base") loads the checkpoint weights in whatever dtype they were saved in. For mDeBERTa-v3-base, that's float16.

When AdamW performs a weight update on float16 parameters:

weight = weight - lr * (grad + weight_decay * weight)

Float16 has ~3.3 decimal digits of precision. The learning rate (2e-5) multiplied by the gradient produces updates smaller than what float16 can represent. The result isn't zero — it's undefined behavior that silently produces NaN or Inf, which then cascades through the entire network.

This didn't crash immediately because:

  • Batch 1: The random initialization of the classifier head dominates the loss. Gradients flowing back to the encoder are small. The float16 update barely corrupts anything.
  • Batch 2-500: Corruption accumulates with each update. Some embedding rows get NaN. When a batch happens to contain tokens that hit those corrupted rows → instant death.

The timing (~step 500) was consistent because that's roughly when enough embedding rows had been updated enough times to hit the precision limit.

The Fix

One argument:

# Before (broken)
self.encoder = AutoModel.from_pretrained("microsoft/mdeberta-v3-base")

# After (works)
self.encoder = AutoModel.from_pretrained("microsoft/mdeberta-v3-base", torch_dtype=torch.float32)

That's it. Forces all weights to float32 regardless of checkpoint dtype.

Why Our Old Scripts Worked

Our working JointBERT and DeBERTa scripts used:

model = DebertaV2PreTrainedModel.from_pretrained("microsoft/deberta-v3-base", ...)

HuggingFace's PreTrainedModel.from_pretrained() with a specific model class handles dtype casting internally during weight loading. AutoModel.from_pretrained() does not — it trusts whatever the checkpoint contains.

This is particularly dangerous because:

  • Setting fp16=False in TrainingArguments only controls training precision (mixed precision), not model weight dtype
  • The model reports device=cuda and appears healthy — no error, no warning
  • nvidia-smi shows normal memory usage
  • The first few hundred steps look completely fine

How to Check If You're Affected

from transformers import AutoModel

model = AutoModel.from_pretrained("microsoft/mdeberta-v3-base")
print(model.embeddings.word_embeddings.weight.dtype)
# If this prints torch.float16 → you have the bug
# If this prints torch.float32 → you're safe

Affected Models

This affects any HuggingFace checkpoint that stores weights in float16, loaded via AutoModel into a standard (non-mixed-precision) training loop. mDeBERTa-v3-base is the one we hit, but the pattern applies to any model where the checkpoint dtype doesn't match your training dtype expectations.

Models loaded via their specific class (BertModel, DebertaV2Model, etc.) through .from_pretrained() are generally safe because the class-specific loading handles dtype conversion.

TL;DR

If you're using AutoModel.from_pretrained() and getting NaN losses after a few hundred steps:

  1. Check model.embeddings.word_embeddings.weight.dtype
  2. If it's float16, add torch_dtype=torch.float32 to from_pretrained()
  3. That's the fix. Don't waste 8 hours like we did.

This bug was discovered while building a toxicity classifier for the EEUCA 2026 Shared Task on the GameTox dataset. The full system — including POS/lexicon feature engineering and keyword override post-processing — is described in our upcoming workshop paper.