Repository navigation
[codex] Add Nemotron-CLIMB data-curation recipe - #2138
Conversation
Signed-off-by: Lawrence Lane <llane@nvidia.com>
| --output-path /data/climb/mixtures_3 \ | ||
| --metric valid_avg \ | ||
| --num-mixtures 16 | ||
| ``` |
There was a problem hiding this comment.
Incorrect embedding dimension for stella_en_400M_v5
The page documents --embedding-dim 3072 as "the conservative estimate for Stella embeddings" and tells readers to "Adjust it for the model's embedding width." However, NovaSearch/stella_en_400M_v5 produces 1024-dimensional embeddings by default. A reader following the advice to set this flag to the model's embedding width would arrive at 1024, not 3072.
If 3072 is deliberately set higher than the actual dimension to create more, smaller Parquet groups for cuDF memory safety, that intent should be stated explicitly. As written, the passage implies 3072 is the actual embedding width, which is misleading.
|
|
||
| **Key configuration** | ||
|
|
||
| - A cluster survives only when its average is at least every corresponding `--pruning-threshold`. |
There was a problem hiding this comment.
Grammatically ambiguous — "is at least every threshold" reads as a comparison against an implicit zero rather than against each threshold value. The suggested phrasing makes the per-threshold comparison explicit.
| - A cluster survives only when its average is at least every corresponding `--pruning-threshold`. | |
| - A cluster survives only when its average score meets or exceeds every corresponding `--pruning-threshold`. |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| <Card title="Tutorials" href="/curate-text/tutorials"> | ||
| Run end-to-end text curation recipes, including Nemotron-CLIMB data-mixture optimization | ||
| recipes | ||
| hands-on | ||
| pretraining | ||
| </Card> |
There was a problem hiding this comment.
The card description ends without a period, unlike every other card description in this file. Inconsistent punctuation is a minor style issue, but fixing it keeps the page uniform.
| <Card title="Tutorials" href="/curate-text/tutorials"> | |
| Run end-to-end text curation recipes, including Nemotron-CLIMB data-mixture optimization | |
| recipes | |
| hands-on | |
| pretraining | |
| </Card> | |
| <Card title="Tutorials" href="/curate-text/tutorials"> | |
| Run end-to-end text curation recipes, including Nemotron-CLIMB data-mixture optimization. | |
| recipes | |
| hands-on | |
| pretraining | |
| </Card> |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-codex-docs-nemotron-climb-recipe.docs.buildwithfern.com/nemo/curator Here are the markdown pages you've updated: |
sarahyurick
left a comment
There was a problem hiding this comment.
Thanks! Added a bunch of comments about some things I thought about while reading. Will leave it up to your decision which to add versus which do not need adding and/or maybe were already explained elsewhere.
| - Stage 4 tokenizes on Ray workers. Stages 5 and 8 are CPU-only; stage 8 requires enough evaluated mixtures to form nonempty training and validation splits. | ||
| - `6_train.sh`, `7_evaluate.sh`, and `e2e.sh` use GNU/Linux shell features such as `mapfile`, `readarray`, and GNU `find` options. | ||
|
|
||
| The supplied proxy configuration is single-node. Adapt the launch command and Megatron parallelism settings before using multiple nodes. The default proxy run exits after approximately 110 minutes, saves every 1,000 iterations, and targets 10,000 iterations. |
There was a problem hiding this comment.
As a general comment I would expect the run to reach the 110 minute timeout with well under 10,000 iterations completed (how many iterations will depend on how many/what type of GPUs are available, model size, i.e., width and depth, etc.). So like 10,000 is just an upper ceiling I chose.
| | `computed_embeddings/` | Text, generated unique IDs, and embedding vectors | Yes, through clustering | | ||
| | `clusters/` and `centroids/` | Documents partitioned by centroid and fitted K-means centroids | Yes, through pruning | | ||
| | `pruned_clusters/` | Quality-scored JSONL grouped into retained or merged domains | Yes, through tokenization | | ||
| | `domains/` | One merged `.bin`/`.idx` pair per domain | Yes, for every search round and final training | |
There was a problem hiding this comment.
Once the domains are generated then the embeddings, clusters, and pruned clusters are not needed for steps 6-8.
There was a problem hiding this comment.
(This is assuming the user is just running each step themselves, as opposed to the e2e script which does depend if the output directory produced by each script already exists or not.)
| | `mixtures_*/` | Shell-readable mixture files containing weight and extensionless data-prefix pairs | Yes, paired with evaluation results | | ||
| | `megatron_exp_*/` | Proxy checkpoints, data cache, and TensorBoard logs | Yes, until evaluation succeeds | | ||
| | `lm_eval_results_*/` | LM Evaluation Harness JSON results | Yes, for predictor fitting | |
There was a problem hiding this comment.
The proxy checkpoints are not needed after the evals are done on them (mentioning this because the proxy checkpoints can take up a huge chunk of disk space very quickly). Of course if there are other types of evaluations, etc. that the user wants to do with the proxy models then they should be kept.
|
|
||
| ```text | ||
| /data/climb/ | ||
| ├── source/ |
There was a problem hiding this comment.
| ├── source/ | |
| ├── source/ | |
| ├── source_split/ # if split_large_files was needed |
| ├── mixtures_1/ | ||
| ├── lm_eval_results_1/ |
There was a problem hiding this comment.
Not sure if you want to include it but the final subdirectories will look something like:
| ├── mixtures_1/ | |
| ├── lm_eval_results_1/ | |
| ├── mixtures_1/ | |
| ├── lm_eval_results_1/ | |
| ├── mixtures_2/ | |
| ├── lm_eval_results_2/ | |
| ├── mixtures_3/ | |
| ├── lm_eval_results_3/ |
| **Inputs** | ||
|
|
||
| - Stage 3 `centroid=<id>/` directories containing JSONL by default. | ||
| - A Hugging Face tokenizer identifier or local path. The default Llama 2 model is gated. |
There was a problem hiding this comment.
The tokenizer used here is important since the same tokenizer must be used for steps 6 and 7. If using the proxy base models that we published on Hugging Face (62m or 350m) then it must be the Llama 2 tokenizer, to match those base models. Else it is up to whatever the user wants to use.
|
|
||
| **Inputs** | ||
|
|
||
| Stage 4 domain pairs. At least one `.bin` must exist and the combined `.bin` size must be nonzero. |
There was a problem hiding this comment.
Really at least 2 bin files must exist. Since if we just had 1 bin/idx file pair then there would be no data weighting needed.
| /data/climb/mixtures_1/n1.sh \ | ||
| /checkpoints/climb/megatron_exp_1/n1 \ | ||
| /models/Llama-2-7b/tokenizer.model \ | ||
| /models/nemotron_climb_proxy_model_350m |
There was a problem hiding this comment.
Note if the user uses the 350m or 62m then they should update the 6_train.sh script to use the appropriate Megatron-LM parameters such as model width, depth, etc. The proxy model Hugging Face page should have more information about what values are appropriate.
|
|
||
| ## 7. Evaluate Proxy Models | ||
|
|
||
| `7_evaluate.sh` discovers all `n<integer>` work directories under one training-round directory and evaluates their checkpoints on ARC-Easy, HellaSwag, and PIQA. |
There was a problem hiding this comment.
The user can override and do something other than the ARC-Easy + HellaSwag + PIQA combination that we have hardcoded. But if they do that then they will need to update 8_predict.py accordingly (since it hardcodes the benchmark names).
| /models/Llama-2-7b/tokenizer.model | ||
| ``` | ||
|
|
||
| To fine-tune a compatible proxy checkpoint, add the optional fifth positional argument: |
There was a problem hiding this comment.
Add link to https://huggingface.co/nvidia/nemotron-climb-proxy-models if we don't have it somewhere already?
|
@meatybobby friendly ping on this docs PR for the #2118 release-docs workstream — when you have a moment, could you take a look at the Nemotron-CLIMB data-curation recipe? Docs preview: https://nvidia-preview-codex-docs-nemotron-climb-recipe.docs.buildwithfern.com/nemo/curator Thanks! |
|
@sarahyurick Addressed your review comments in
|
096fd0d to
e1b531a
Compare
Clarify proxy timeout/iteration behavior, disk retention after domains and evaluation, manual vs e2e restart semantics, directory layout, resumability via checkpoint_path, tokenizer/proxy-model requirements, mixture minimums, benchmark overrides, and fit-data-fraction usage. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
e1b531a to
c6ed962
Compare
|
/ok to test c6ed962 |
|
/ok to test 7958eac |
Summary
Why
The complete workflow currently lives only in the tutorial README and scripts, which makes it difficult to discover from the documentation site and leaves important production concerns—particularly expensive stages, intermediate artifacts, and partial-run recovery—implicit.
Validation
cd fern && npm run check— 0 errors; 103 existing removed-reference redirect warningsbash -nfor6_train.sh,7_evaluate.sh, ande2e.shast.parseargparsedefinitions (including shared Ray flags)--helpfor4_tokenize.pyand5_mixture.py; the remaining entry points require optional CUDA, FastText, or LightGBM packages not present in the local validation environmentCloses #2119