Skip to content

[codex] Add Nemotron-CLIMB data-curation recipe - #2138

Merged
lbliii merged 8 commits into
NVIDIA-NeMo:mainfrom
lbliii:codex/docs-nemotron-climb-recipe
Jul 8, 2026
Merged

lbliii merged 8 commits into
NVIDIA-NeMo:mainfrom
lbliii:codex/docs-nemotron-climb-recipe

Conversation

@lbliii

@lbliii lbliii commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an operator-focused Nemotron-CLIMB recipe covering prerequisites, compute and storage planning, external model assets, quick validation, the full 64 → 32 → 16 → 1 search, and safe restart behavior
  • add a detailed stage reference for scripts 1–8 with exact commands, inputs, outputs, key controls, artifact contracts, completion checks, and Megatron tokenizer-output merging
  • add the recipe to Fern navigation and link it from the text-curation and text-tutorial landing pages

Why

The complete workflow currently lives only in the tutorial README and scripts, which makes it difficult to discover from the documentation site and leaves important production concerns—particularly expensive stages, intermediate artifacts, and partial-run recovery—implicit.

Validation

  • cd fern && npm run check — 0 errors; 103 existing removed-reference redirect warnings
  • bash -n for 6_train.sh, 7_evaluate.sh, and e2e.sh
  • parsed all seven tutorial Python files with ast.parse
  • validated every documented Python command flag against the current argparse definitions (including shared Ray flags)
  • directly exercised --help for 4_tokenize.py and 5_mixture.py; the remaining entry points require optional CUDA, FastText, or LightGBM packages not present in the local validation environment

Closes #2119

Signed-off-by: Lawrence Lane <llane@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@lbliii lbliii self-assigned this Jun 30, 2026
@lbliii
lbliii marked this pull request as ready for review July 2, 2026 14:53
@lbliii
lbliii requested a review from a team as a code owner July 2, 2026 14:53
@lbliii
lbliii requested review from meatybobby and removed request for a team July 2, 2026 14:53
@greptile-apps

greptile-apps Bot commented Jul 2, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds documentation for the Nemotron-CLIMB data curation recipe — a compute-intensive, multi-stage workflow that discovers optimal pretraining data mixtures via embedding, clustering, proxy model training, and LightGBM-based predictor fitting.

  • fern/versions/main.yml: Converts the flat "Tutorials" page entry into a nested section containing the new Nemotron-CLIMB recipe and stage reference pages.
  • index.mdx (curate-text and tutorials): Adds a "Tutorials" card to the text curation landing page and a "Nemotron-CLIMB Data Curation" card to the tutorials index.
  • nemotron-climb/index.mdx (new): Full recipe overview covering prerequisites, storage planning, software/model assets, run profiles (quick validation vs. full-scale search), and safe restart behavior.
  • nemotron-climb/stages.mdx (new): Detailed stage reference for scripts 1–8 with exact commands, input/output contracts, key configuration flags, and completion checks for each stage.

Confidence Score: 5/5

Pure documentation addition with no code changes; navigation slugs and cross-links are internally consistent and validated by the author's npm check run.

All five changed files are documentation or navigation config. The two new MDX pages contain factually accurate stage descriptions (embedding-dim rationale, cumulative proxy model count, restart behavior) that are internally consistent with each other. Previously flagged issues have been addressed in this revision. No executable code is introduced or modified.

No files require special attention; all changes are documentation-only.

Important Files Changed

Filename Overview
fern/versions/main.yml Converts Tutorials from a flat page to a section with correct slug hierarchy; URLs and slugs for Recipe and Stage Reference are consistent with cross-links in the new MDX pages.
fern/versions/main/pages/curate-text/index.mdx Adds a Tutorials Card with a description ending in a period, consistent with other cards. Tags and href are correct.
fern/versions/main/pages/curate-text/tutorials/index.mdx Adds the Nemotron-CLIMB card to the tutorials landing page; href and description are accurate and consistent with the new recipe page.
fern/versions/main/pages/curate-text/tutorials/nemotron-climb/index.mdx Comprehensive recipe overview page; the full-scale search table is arithmetically correct and restart/resume guidance is internally consistent with stage behavior.
fern/versions/main/pages/curate-text/tutorials/nemotron-climb/stages.mdx Stage-by-stage reference with commands, artifact contracts, and restart checks; the deliberate 3x embedding-dim overestimate is now explicitly explained in Stage 2, and all cross-links resolve to the correct slugs.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant User
    participant S1 as 1_embed.py
    participant S2 as 2_cluster.py
    participant S3 as 3_prune.py
    participant S4 as 4_tokenize.py
    participant S5 as 5_mixture.py
    participant S6 as 6_train.sh
    participant S7 as 7_evaluate.sh
    participant S8 as 8_predict.py
    participant FS as Full-scale Training

    User->>S1: JSONL / Parquet source
    S1-->>User: computed_embeddings/ (ID + text + vector)

    User->>S2: computed_embeddings/
    S2-->>User: clusters/ + centroids/

    User->>S3: clusters/ + centroids/ + FastText models
    S3-->>User: pruned_clusters/

    User->>S4: pruned_clusters/
    S4-->>User: domains/ (.bin + .idx per domain)

    User->>S5: domains/ (--num-mixtures 64)
    S5-->>User: mixtures_1/ (n1.sh...n64.sh)

    loop "Iterative search (rounds 1->3)"
        User->>S6: mixture nX.sh x N models
        S6-->>User: megatron_exp_R/ checkpoints

        User->>S7: megatron_exp_R/
        S7-->>User: lm_eval_results_R/ JSON

        User->>S8: lm_eval_results_1..R + mixtures_1..R
        S8-->>User: "mixtures_(R+1)/ (32 -> 16 -> 1 candidates)"
    end

    User->>FS: "optimal_mixture/n1.sh -> --train-data-path"
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant User
    participant S1 as 1_embed.py
    participant S2 as 2_cluster.py
    participant S3 as 3_prune.py
    participant S4 as 4_tokenize.py
    participant S5 as 5_mixture.py
    participant S6 as 6_train.sh
    participant S7 as 7_evaluate.sh
    participant S8 as 8_predict.py
    participant FS as Full-scale Training

    User->>S1: JSONL / Parquet source
    S1-->>User: computed_embeddings/ (ID + text + vector)

    User->>S2: computed_embeddings/
    S2-->>User: clusters/ + centroids/

    User->>S3: clusters/ + centroids/ + FastText models
    S3-->>User: pruned_clusters/

    User->>S4: pruned_clusters/
    S4-->>User: domains/ (.bin + .idx per domain)

    User->>S5: domains/ (--num-mixtures 64)
    S5-->>User: mixtures_1/ (n1.sh...n64.sh)

    loop "Iterative search (rounds 1->3)"
        User->>S6: mixture nX.sh x N models
        S6-->>User: megatron_exp_R/ checkpoints

        User->>S7: megatron_exp_R/
        S7-->>User: lm_eval_results_R/ JSON

        User->>S8: lm_eval_results_1..R + mixtures_1..R
        S8-->>User: "mixtures_(R+1)/ (32 -> 16 -> 1 candidates)"
    end

    User->>FS: "optimal_mixture/n1.sh -> --train-data-path"
Loading

Reviews (9): Last reviewed commit: "Merge branch 'main' into codex/docs-slur..." | Re-trigger Greptile

--output-path /data/climb/mixtures_3 \
--metric valid_avg \
--num-mixtures 16
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Incorrect embedding dimension for stella_en_400M_v5

The page documents --embedding-dim 3072 as "the conservative estimate for Stella embeddings" and tells readers to "Adjust it for the model's embedding width." However, NovaSearch/stella_en_400M_v5 produces 1024-dimensional embeddings by default. A reader following the advice to set this flag to the model's embedding width would arrive at 1024, not 3072.

If 3072 is deliberately set higher than the actual dimension to create more, smaller Parquet groups for cuDF memory safety, that intent should be stated explicitly. As written, the passage implies 3072 is the actual embedding width, which is misleading.


**Key configuration**

- A cluster survives only when its average is at least every corresponding `--pruning-threshold`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Grammatically ambiguous — "is at least every threshold" reads as a comparison against an implicit zero rather than against each threshold value. The suggested phrasing makes the per-threshold comparison explicit.

Suggested change
- A cluster survives only when its average is at least every corresponding `--pruning-threshold`.
- A cluster survives only when its average score meets or exceeds every corresponding `--pruning-threshold`.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +70 to +75
<Card title="Tutorials" href="/curate-text/tutorials">
Run end-to-end text curation recipes, including Nemotron-CLIMB data-mixture optimization
recipes
hands-on
pretraining
</Card>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The card description ends without a period, unlike every other card description in this file. Inconsistent punctuation is a minor style issue, but fixing it keeps the page uniform.

Suggested change
<Card title="Tutorials" href="/curate-text/tutorials">
Run end-to-end text curation recipes, including Nemotron-CLIMB data-mixture optimization
recipes
hands-on
pretraining
</Card>
<Card title="Tutorials" href="/curate-text/tutorials">
Run end-to-end text curation recipes, including Nemotron-CLIMB data-mixture optimization.
recipes
hands-on
pretraining
</Card>

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

@sarahyurick sarahyurick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Added a bunch of comments about some things I thought about while reading. Will leave it up to your decision which to add versus which do not need adding and/or maybe were already explained elsewhere.

- Stage 4 tokenizes on Ray workers. Stages 5 and 8 are CPU-only; stage 8 requires enough evaluated mixtures to form nonempty training and validation splits.
- `6_train.sh`, `7_evaluate.sh`, and `e2e.sh` use GNU/Linux shell features such as `mapfile`, `readarray`, and GNU `find` options.

The supplied proxy configuration is single-node. Adapt the launch command and Megatron parallelism settings before using multiple nodes. The default proxy run exits after approximately 110 minutes, saves every 1,000 iterations, and targets 10,000 iterations.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general comment I would expect the run to reach the 110 minute timeout with well under 10,000 iterations completed (how many iterations will depend on how many/what type of GPUs are available, model size, i.e., width and depth, etc.). So like 10,000 is just an upper ceiling I chose.

Comment on lines +57 to +60
| `computed_embeddings/` | Text, generated unique IDs, and embedding vectors | Yes, through clustering |
| `clusters/` and `centroids/` | Documents partitioned by centroid and fitted K-means centroids | Yes, through pruning |
| `pruned_clusters/` | Quality-scored JSONL grouped into retained or merged domains | Yes, through tokenization |
| `domains/` | One merged `.bin`/`.idx` pair per domain | Yes, for every search round and final training |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the domains are generated then the embeddings, clusters, and pruned clusters are not needed for steps 6-8.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This is assuming the user is just running each step themselves, as opposed to the e2e script which does depend if the output directory produced by each script already exists or not.)

Comment on lines +61 to +63
| `mixtures_*/` | Shell-readable mixture files containing weight and extensionless data-prefix pairs | Yes, paired with evaluation results |
| `megatron_exp_*/` | Proxy checkpoints, data cache, and TensorBoard logs | Yes, until evaluation succeeds |
| `lm_eval_results_*/` | LM Evaluation Harness JSON results | Yes, for predictor fitting |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proxy checkpoints are not needed after the evals are done on them (mentioning this because the proxy checkpoints can take up a huge chunk of disk space very quickly). Of course if there are other types of evaluations, etc. that the user wants to do with the proxy models then they should be kept.


```text
/data/climb/
├── source/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
├── source/
├── source/
├── source_split/ # if split_large_files was needed

Comment on lines +25 to +26
├── mixtures_1/
├── lm_eval_results_1/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if you want to include it but the final subdirectories will look something like:

Suggested change
├── mixtures_1/
├── lm_eval_results_1/
├── mixtures_1/
├── lm_eval_results_1/
├── mixtures_2/
├── lm_eval_results_2/
├── mixtures_3/
├── lm_eval_results_3/

**Inputs**

- Stage 3 `centroid=<id>/` directories containing JSONL by default.
- A Hugging Face tokenizer identifier or local path. The default Llama 2 model is gated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tokenizer used here is important since the same tokenizer must be used for steps 6 and 7. If using the proxy base models that we published on Hugging Face (62m or 350m) then it must be the Llama 2 tokenizer, to match those base models. Else it is up to whatever the user wants to use.


**Inputs**

Stage 4 domain pairs. At least one `.bin` must exist and the combined `.bin` size must be nonzero.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really at least 2 bin files must exist. Since if we just had 1 bin/idx file pair then there would be no data weighting needed.

/data/climb/mixtures_1/n1.sh \
/checkpoints/climb/megatron_exp_1/n1 \
/models/Llama-2-7b/tokenizer.model \
/models/nemotron_climb_proxy_model_350m

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note if the user uses the 350m or 62m then they should update the 6_train.sh script to use the appropriate Megatron-LM parameters such as model width, depth, etc. The proxy model Hugging Face page should have more information about what values are appropriate.


## 7. Evaluate Proxy Models

`7_evaluate.sh` discovers all `n<integer>` work directories under one training-round directory and evaluates their checkpoints on ARC-Easy, HellaSwag, and PIQA.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The user can override and do something other than the ARC-Easy + HellaSwag + PIQA combination that we have hardcoded. But if they do that then they will need to update 8_predict.py accordingly (since it hardcodes the benchmark names).

/models/Llama-2-7b/tokenizer.model
```

To fine-tune a compatible proxy checkpoint, add the optional fifth positional argument:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add link to https://huggingface.co/nvidia/nemotron-climb-proxy-models if we don't have it somewhere already?

@lbliii

lbliii commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

@meatybobby friendly ping on this docs PR for the #2118 release-docs workstream — when you have a moment, could you take a look at the Nemotron-CLIMB data-curation recipe?

Docs preview: https://nvidia-preview-codex-docs-nemotron-climb-recipe.docs.buildwithfern.com/nemo/curator

Thanks!

@lbliii

lbliii commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

@sarahyurick Addressed your review comments in 096fd0dba:

  • Proxy timeout/iterations: clarified that the 110-minute exit usually happens well before 10,000 iterations.
  • Disk retention: noted that embeddings/clusters/pruned clusters can be dropped after domains/ exists, and proxy checkpoints can be dropped after stage 7 unless needed for other work.
  • Manual vs e2e.sh restart: clarified that directory-existence skipping applies to the driver script, not necessarily manual stage reruns.
  • Directory layout: added source_split/ and round-specific mixtures_* / lm_eval_results_* examples.
  • Resumability: documented optional pipeline.run(checkpoint_path=...) for stage 1 and linked to the resumable-processing guide.
  • --fit-data-fraction: noted it is for cases where embedding data cannot fit in GPU memory.
  • Tokenizer/proxy models: documented Llama 2 requirement for published 62M/350M checkpoints and linked the proxy-model card.
  • Mixture minimum: clarified that at least two domain .bin files are required.
  • Benchmark overrides: noted that changing stage 7 benchmarks requires matching updates in 8_predict.py.

fern check passes with 0 errors.

@lbliii
lbliii force-pushed the codex/docs-nemotron-climb-recipe branch from 096fd0d to e1b531a Compare July 7, 2026 18:26
Clarify proxy timeout/iteration behavior, disk retention after domains
and evaluation, manual vs e2e restart semantics, directory layout,
resumability via checkpoint_path, tokenizer/proxy-model requirements,
mixture minimums, benchmark overrides, and fit-data-fraction usage.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
@lbliii
lbliii force-pushed the codex/docs-nemotron-climb-recipe branch from e1b531a to c6ed962 Compare July 7, 2026 18:26
@lbliii

lbliii commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test c6ed962

@lbliii

lbliii commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7958eac

This branch was previously deployed

3 inactive deployments
nemo-ci — 7958eac7 Deployed Jul 8, 2026 by copy-pr-bot[bot] via L0_Unit_Test_GPU-serve #4898
public — 7958eac7 Deployed Jul 8, 2026 by copy-pr-bot[bot] via release / finalize / notify #495
test — 7958eac7 Deployed Jul 8, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #4898
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Docs] Add a Nemotron-CLIMB data-curation recipe

2 participants