Model Adaptation · Software component
Full-Parameter Fine-Tuner
Software componentModel AdaptationModelsarc:FullParameterFineTuner
A fine-tuning pipeline that updates all model weights, storing gradients and optimizer states for every parameter.
Responsibility. Fine-tunes every parameter of a model.
Also known as: Full fine-tuning
Variant of Fine-Tuning Pipeline abstract
When to choose. Choose when substantial multi-GPU infrastructure is available and all model weights are to be modified.
Relationships
deployed on structural
alternative to variability
- LoRA Fine-Tuner abstract Ch3.5
- QLoRA Fine-Tuner Ch3.5
Design guidance
- SHOULD start with data parallelism and add tensor, then pipeline, then context parallelism only when the model or sequence does not fit (Ref7.06).
- SHOULD confine tensor parallelism to a single NVLink domain (intra-node, TP <= 8) (Ref7.06, Ref7.05).
Quantitative guidance
As stated by the sources; verify before use.
- A 70B-parameter model requires hundreds of GB (~400GB) of GPU memory for full fine-tuning (Ch3.5).
- 70B FP16 (~140 GB) with TP=8 needs ~17.5 GB per GPU (Ref7.06).
- Targets: >80% GPU utilisation and >90% scaling efficiency (Ref7.06).
Classification
- Patterns
- Data parallelismTensor parallelismPipeline / virtual pipeline parallelismContext (sequence) parallelismFully sharded data parallelismDistributed optimizerCommunication/computation overlap
- Technologies
- NVIDIA NeMo FrameworkMegatron FSDPPyTorch FSDPNCCL
- Quality attributes
- Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Sources
- Ch3.5: T. Nguyen, "Prompt Optimization, Few-Shot Learning, Fine-Tuning," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.5. ISBN: 9798244538229.
- Ref7.06: NVIDIA, "Performance Tuning Guide," Megatron Bridge Documentation. Accessed: Sep. 27, 2026. [Online]. Available: https://docs.nvidia.com/nemo/megatron-bridge/latest/performance-guide.html