Researchers at NYU and the University of Calgary propose deletion tuning, which retains pretrained weights while selectively setting fewer than 0.05% to zero. Llama 8B’s GSM8K accuracy rose from 60.27% to 83.55%, comparable to 83.24% with full fine-tuning. Compressed task-specific information required 8.4MB versus 30.6GB for the full tuned result, but the base model is still required separately. Tests reached 30 billion parameters, and some smaller models underperformed full fine-tuning. Storage savings do not imply proportional training-cost reductions or guaranteed gains on every model.
Back to Tech
Deletion tuning improves post-training performance by disabling selected weights
Task-specific storage shrinks, but a base model and training remain necessary; effects on frontier commercial models are untested.