Skip to content

Commit 21cb353

Browse files
casincastevhliu
andauthored
[docs] Typos - Single GPU efficient training features (huggingface#38964)
* Typos - corrected bf16 training argument - corrected header for SDPA * improved readability for SDPA suggested by @stevhliu Co-authored-by: Steven Liu <[email protected]> --------- Co-authored-by: Steven Liu <[email protected]>
1 parent f9be71b commit 21cb353

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

docs/source/en/perf_train_gpu_one.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Refer to the table below to quickly help you identify the features relevant to y
3131
| data preloading | yes | no |
3232
| torch_empty_cache_steps | no | yes |
3333
| torch.compile | yes | no |
34-
| PEFT | no | yes |
34+
| scaled dot production attention (SDPA) | yes | yes |
3535

3636
## Trainer
3737

@@ -128,7 +128,7 @@ fp16 isn't memory-optimized because the gradients that are computed in fp16 are
128128

129129
[bf16](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus) trades off some precision for a much larger dynamic range, which is helpful for avoiding overflow and underflow errors. You can use bf16 without adding any loss scaling methods like you would with fp16. bf16 is supported by NVIDIAs Ampere architecture or newer.
130130

131-
Configure [`~TrainingArguments.fp16`] in [`TrainingArguments`] to enable mixed precision training with the bf16 data type.
131+
Configure [`~TrainingArguments.bf16`] in [`TrainingArguments`] to enable mixed precision training with the bf16 data type.
132132

133133
```py
134134
from transformers import TrainingArguments

0 commit comments

Comments
 (0)