Repository navigation
Adding Streaming Sortformer Stage - #1573
Conversation
6566d15 to
00e745f
Compare
Greptile SummaryThis PR adds
Confidence Score: 4/5Safe to merge for tutorial/evaluation use, but several pre-existing P1s should be resolved before production use. Multiple P1-level issues from prior rounds remain open (map_location hardcoded to cuda, IndexError on malformed string segments, DER double-counting), plus two new P2 findings (incomplete inputs() declaration and a test that silently validates the wrong path). The new findings are P2, but the carry-over P1s keep the score at 4. nemo_curator/stages/audio/inference/sortformer.py (map_location, IndexError on malformed segments, inputs() validation); tests/stages/audio/inference/test_sortformer.py (setup_on_node success-path coverage); tutorials/audio/callhome_diar/run.py (DER double-counting) Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant InferenceSortformerStage
participant NeMo as SortformerEncLabelModel
participant HFHub as HuggingFace Hub
participant RTTM as RTTM File
User->>InferenceSortformerStage: setup_on_node()
InferenceSortformerStage->>HFHub: snapshot_download(model_name)
HFHub-->>InferenceSortformerStage: repo_dir (sets model_path)
User->>InferenceSortformerStage: setup()
alt diar_model injected
InferenceSortformerStage->>NeMo: eval()
else model_path set
InferenceSortformerStage->>NeMo: restore_from(model_path, map_location="cuda")
InferenceSortformerStage->>NeMo: eval()
end
InferenceSortformerStage->>NeMo: configure_streaming(chunk_len, fifo_len, ...)
User->>InferenceSortformerStage: process(AudioTask)
InferenceSortformerStage->>InferenceSortformerStage: validate_input(task)
InferenceSortformerStage->>NeMo: diarize([audio_filepath])
NeMo-->>InferenceSortformerStage: raw_segments
InferenceSortformerStage->>InferenceSortformerStage: _parse_sortformer_segments(raw_segments)
opt rttm_out_dir set
InferenceSortformerStage->>RTTM: _write_rttm(segments, sess_name)
end
InferenceSortformerStage-->>User: AudioTask(diar_segments=[...])
Reviews (13): Last reviewed commit: "Merge branch 'main' into sortformer_stag..." | Re-trigger Greptile |
| def setup_on_node(self, _node_info: NodeInfo | None = None, _worker_metadata: WorkerMetadata = None) -> None: | ||
| """Pre-download model weights on the node so actors load from cache.""" | ||
| if self.model_path is not None: | ||
| return | ||
| try: | ||
| snapshot_download(repo_id=self.model_name) | ||
| except Exception: # noqa: BLE001 | ||
| logger.info(f"Could not pre-cache {self.model_name}; actors will download on first use") | ||
|
|
||
| def setup(self, _worker_metadata: WorkerMetadata = None) -> None: |
There was a problem hiding this comment.
Incorrect type annotation — WorkerMetadata is not nullable
Both setup_on_node and setup accept None as their _worker_metadata default, but the type annotation says WorkerMetadata (non-optional). The base class ProcessingStage declares these methods with WorkerMetadata | None = None, so the override is inconsistent with the parent signature and will fail static analysis tools.
| def setup_on_node(self, _node_info: NodeInfo | None = None, _worker_metadata: WorkerMetadata = None) -> None: | |
| """Pre-download model weights on the node so actors load from cache.""" | |
| if self.model_path is not None: | |
| return | |
| try: | |
| snapshot_download(repo_id=self.model_name) | |
| except Exception: # noqa: BLE001 | |
| logger.info(f"Could not pre-cache {self.model_name}; actors will download on first use") | |
| def setup(self, _worker_metadata: WorkerMetadata = None) -> None: | |
| def setup_on_node(self, _node_info: NodeInfo | None = None, _worker_metadata: WorkerMetadata | None = None) -> None: |
And similarly for setup:
| def setup_on_node(self, _node_info: NodeInfo | None = None, _worker_metadata: WorkerMetadata = None) -> None: | |
| """Pre-download model weights on the node so actors load from cache.""" | |
| if self.model_path is not None: | |
| return | |
| try: | |
| snapshot_download(repo_id=self.model_name) | |
| except Exception: # noqa: BLE001 | |
| logger.info(f"Could not pre-cache {self.model_name}; actors will download on first use") | |
| def setup(self, _worker_metadata: WorkerMetadata = None) -> None: | |
| def setup(self, _worker_metadata: WorkerMetadata | None = None) -> None: |
| def test_unrecognised_format_warns(self) -> None: | ||
| out = _parse_sortformer_segments([42]) | ||
| assert out == [] |
There was a problem hiding this comment.
Test name claims a warning is asserted but warning is never checked
test_unrecognised_format_warns only verifies the empty-list return value. The actual logger.warning(...) call in _parse_sortformer_segments is never asserted. Readers of the test suite will expect the warning to be verified given the method name.
Consider capturing the loguru output with caplog or loguru's capfd sink and asserting the warning text:
def test_unrecognised_format_warns(self, caplog) -> None:
with caplog.at_level("WARNING"):
out = _parse_sortformer_segments([42])
assert out == []
assert "Unrecognised segment format" in caplog.text(Adjust the capture approach for loguru if the project already has a standard fixture for it.)
| "speaker": parts[2] if len(parts) > 2 else "unknown", # noqa: PLR2004 | ||
| } | ||
| ) | ||
| elif hasattr(seg, "start") and hasattr(seg, "end"): | ||
| segments.append( | ||
| { | ||
| "start": float(seg.start), | ||
| "end": float(seg.end), |
There was a problem hiding this comment.
IndexError on malformed string segments
If the NeMo model returns a string segment with fewer than 2 space-separated tokens (e.g., an empty string "" or a single-token string like "0.00"), parts = seg.strip().split() will produce a list with 0 or 1 elements. Accessing parts[0] or parts[1] on it will immediately raise an IndexError, crashing the entire batch.
The existing guard len(parts) > 2 on line 52 only protects the third element (speaker), not the first two, so malformed segments will propagate as unhandled exceptions.
| "speaker": parts[2] if len(parts) > 2 else "unknown", # noqa: PLR2004 | |
| } | |
| ) | |
| elif hasattr(seg, "start") and hasattr(seg, "end"): | |
| segments.append( | |
| { | |
| "start": float(seg.start), | |
| "end": float(seg.end), | |
| if isinstance(seg, str): | |
| parts = seg.strip().split() | |
| if len(parts) < 2: # noqa: PLR2004 | |
| logger.warning(f"Unrecognised segment format (too few fields): {seg!r}") | |
| continue | |
| segments.append( | |
| { | |
| "start": float(parts[0]), | |
| "end": float(parts[1]), | |
| "speaker": parts[2] if len(parts) > 2 else "unknown", # noqa: PLR2004 | |
| } | |
| ) |
| from nemo.collections.asr.models import SortformerEncLabelModel | ||
| except ImportError as e: | ||
| msg = ( | ||
| "NeMo ASR is required for InferenceSortformerStage. " | ||
| "Install via: pip install 'git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]'" |
There was a problem hiding this comment.
Pre-loaded model not put into eval mode
When a diar_model is injected at construction time, setup() calls _configure_streaming() and returns early — skipping the self.diar_model.eval() call that is present in the other branch (line 158). If the caller passes a model that is still in training mode (e.g., a freshly instantiated or fine-tuned model), dropout and batch-norm layers will behave randomly during inference, silently producing non-deterministic and degraded diarization results.
| from nemo.collections.asr.models import SortformerEncLabelModel | |
| except ImportError as e: | |
| msg = ( | |
| "NeMo ASR is required for InferenceSortformerStage. " | |
| "Install via: pip install 'git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]'" | |
| def setup(self, _worker_metadata: WorkerMetadata = None) -> None: | |
| """Load Sortformer model from Hugging Face or a local .nemo file.""" | |
| if self.diar_model is not None: | |
| self.diar_model.eval() | |
| self._configure_streaming() | |
| return |
| else: | ||
| self.diar_model = SortformerEncLabelModel.from_pretrained(self.model_name) | ||
|
|
||
| self.diar_model.eval() | ||
| self._configure_streaming() | ||
|
|
There was a problem hiding this comment.
map_location hardcoded to "cuda" for local checkpoint loading
When model_path is provided, restore_from() is called with map_location="cuda". If CUDA is not available (e.g., a CI runner, a CPU-only node, or a node with CUDA temporarily unavailable), this will fail with a device error. The from_pretrained path on line 156 lets NeMo decide the device, making the two code paths inconsistent.
Consider deriving the map location dynamically so the stage degrades gracefully:
import torch
...
map_location = "cuda" if torch.cuda.is_available() else "cpu"
self.diar_model = SortformerEncLabelModel.restore_from(
restore_path=self.model_path,
map_location=map_location,
strict=False,
)7e24cac to
090bfab
Compare
| todo_hashes: list[str] = [] | ||
|
|
||
| for task in current_tasks: | ||
| h = task._metadata.get(CKPT_HASH_KEY) or _task_hash(task) | ||
| if h in existing_hashes: | ||
| cached_tasks.append(_load_task(sdir / f"{h}.json")) | ||
| else: | ||
| task._metadata[CKPT_HASH_KEY] = h | ||
| todo_tasks.append(task) | ||
| todo_hashes.append(h) |
There was a problem hiding this comment.
todo_hashes is populated but never read
todo_hashes is built in every iteration of the stage loop but is never referenced after the loop body. The same pattern appears in single_speaker_filter/run.py (lines 222–232). This is dead code that wastes allocations and misleads readers into thinking the list is used for deduplication or lookup.
| todo_hashes: list[str] = [] | |
| for task in current_tasks: | |
| h = task._metadata.get(CKPT_HASH_KEY) or _task_hash(task) | |
| if h in existing_hashes: | |
| cached_tasks.append(_load_task(sdir / f"{h}.json")) | |
| else: | |
| task._metadata[CKPT_HASH_KEY] = h | |
| todo_tasks.append(task) | |
| todo_hashes.append(h) | |
| cached_tasks: list[AudioBatch] = [] | |
| todo_tasks: list[AudioBatch] = [] | |
| for task in current_tasks: | |
| h = task._metadata.get(CKPT_HASH_KEY) or _task_hash(task) | |
| if h in existing_hashes: | |
| cached_tasks.append(_load_task(sdir / f"{h}.json")) | |
| else: | |
| task._metadata[CKPT_HASH_KEY] = h | |
| todo_tasks.append(task) |
| cached_tasks: list[AudioBatch] = [] | ||
| todo_tasks: list[AudioBatch] = [] | ||
| todo_hashes: list[str] = [] | ||
|
|
||
| for task in current_tasks: | ||
| h = task._metadata.get(CKPT_HASH_KEY) or _task_hash(task) | ||
| if (sdir / f"{h}.json").exists(): | ||
| cached_tasks.append(_load_task(sdir / f"{h}.json")) | ||
| else: | ||
| task._metadata[CKPT_HASH_KEY] = h | ||
| todo_tasks.append(task) | ||
| todo_hashes.append(h) |
There was a problem hiding this comment.
todo_hashes populated but never used
todo_hashes is declared and appended to but is never read after the loop. This is the same dead-code pattern as in callhome_diar/run.py. Remove the list to keep the intent clear.
| cached_tasks: list[AudioBatch] = [] | |
| todo_tasks: list[AudioBatch] = [] | |
| todo_hashes: list[str] = [] | |
| for task in current_tasks: | |
| h = task._metadata.get(CKPT_HASH_KEY) or _task_hash(task) | |
| if (sdir / f"{h}.json").exists(): | |
| cached_tasks.append(_load_task(sdir / f"{h}.json")) | |
| else: | |
| task._metadata[CKPT_HASH_KEY] = h | |
| todo_tasks.append(task) | |
| todo_hashes.append(h) | |
| cached_tasks: list[AudioBatch] = [] | |
| todo_tasks: list[AudioBatch] = [] | |
| for task in current_tasks: | |
| h = task._metadata.get(CKPT_HASH_KEY) or _task_hash(task) | |
| if (sdir / f"{h}.json").exists(): | |
| cached_tasks.append(_load_task(sdir / f"{h}.json")) | |
| else: | |
| task._metadata[CKPT_HASH_KEY] = h | |
| todo_tasks.append(task) |
| ``` | ||
| COMPLETED: 139 files evaluated (collar=0.25s) | ||
|
|
||
| --- Macro-Average (equal weight per file) --- | ||
| DER: 6.2% | ||
| Miss: 1.5% | ||
| FA: 3.4% | ||
| Confuse: 1.3% | ||
| Correct: 97.2% | ||
|
|
||
| --- Speaker Count --- | ||
| Exact match: 109/139 (78%) | ||
| ``` |
There was a problem hiding this comment.
Example output in README does not match what _print_summary actually produces
The README shows section headers like --- Macro-Average (equal weight per file) --- and --- Speaker Count ---, with field labels like DER:, Miss:, Confuse:, and Correct:. The actual _print_summary function in run.py produces a completely different layout:
============================================================
COMPLETED: 139 files evaluated (collar=0.25s)
============================================================
Macro-avg DER=6.2% Miss=1.5% FA=3.4% Conf=1.3%
Weighted DER=6.0% ...
Speaker count match: 109/139 (78%)
Best 5: ...
Worst 5: ...
Key differences:
- The
=-border header ("COMPLETED: ...") replaces the---headers. - The function also prints a
Weightedrow (absent from the README example). - Field names are
Macro-avg/Weightedinstead of--- Macro-Average ---, and key names areConfnotConfuse, noCorrectrow. - The
Best 5/Worst 5lines are missing entirely from the README example.
Please update the example output block to reflect the actual printed output so users know what to expect.
| def _write_rttm(segments: list[dict[str, Any]], sess_name: str, rttm_out_dir: str) -> None: | ||
| """Write diarization segments to an RTTM file.""" | ||
| os.makedirs(rttm_out_dir, exist_ok=True) | ||
| rttm_path = os.path.join(rttm_out_dir, f"{sess_name}.rttm") | ||
| with open(rttm_path, "w") as f: | ||
| for seg in segments: | ||
| duration = seg["end"] - seg["start"] | ||
| f.write(f"SPEAKER {sess_name} 1 {seg['start']:.3f} {duration:.3f} <NA> <NA> {seg['speaker']} <NA> <NA>\n") |
There was a problem hiding this comment.
_write_rttm does not validate that segment duration is positive
duration = seg["end"] - seg["start"] is written directly to the RTTM file without any guard. A zero-duration or negative-duration segment (e.g. if the model returns a degenerate segment where end ≤ start) will produce an invalid RTTM entry. Standard RTTM parsers (e.g. dscore, pyannote) reject non-positive durations and will fail on the output file.
Consider adding an assertion or a warning + skip:
duration = seg["end"] - seg["start"]
if duration <= 0:
logger.warning(f"Skipping degenerate segment with non-positive duration: {seg!r}")
continue224efda to
1bc1b80
Compare
| def process(self, task: AudioBatch) -> AudioBatch: | ||
| """Run speaker diarization on each audio file in the task.""" | ||
| if not self.validate_input(task): | ||
| msg = f"Task {task!s} failed validation for stage {self}" | ||
| raise ValueError(msg) | ||
|
|
||
| files = [item[self.filepath_key] for item in task.data] | ||
| session_names = [item.get("session_name") if isinstance(item, dict) else None for item in task.data] | ||
|
|
||
| all_segments = self.diarize(files) |
There was a problem hiding this comment.
Cryptic AttributeError when setup() is never called
If neither diar_model is injected at construction time nor setup() has been called, self.diar_model stays None. The validate_input call on line 190 only validates task structure (required attrs from inputs()), so process() passes validation and proceeds to line 197 where self.diarize(files) calls self.diar_model.diarize(...) on None, raising:
AttributeError: 'NoneType' object has no attribute 'diarize'
This gives no indication that setup() needs to be called first. Adding an explicit guard at the top of process() would surface the error clearly:
def process(self, task: AudioBatch) -> AudioBatch:
if self.diar_model is None:
msg = (
f"{type(self).__name__}.process() was called before setup(). "
"Either call setup() first or pass diar_model= at construction time."
)
raise RuntimeError(msg)
...| def _compute_der( # noqa: C901, PLR0912 | ||
| self, gt: list[dict], pred: list[dict], uem_start: float, uem_end: float, | ||
| ) -> dict | None: | ||
| """Frame-level DER restricted to UEM region with collar tolerance.""" | ||
| if not gt or not pred: | ||
| return None | ||
|
|
||
| pred = [ | ||
| {"speaker": s["speaker"], "start": max(s["start"], uem_start), "end": min(s["end"], uem_end)} | ||
| for s in pred if s["end"] > uem_start and s["start"] < uem_end | ||
| ] | ||
| pred = [s for s in pred if s["end"] > s["start"]] | ||
| if not pred: | ||
| return None | ||
|
|
||
| collar_zones: list[tuple[float, float]] = [] | ||
| if self.collar > 0: | ||
| for s in gt: | ||
| collar_zones.append((s["start"] - self.collar, s["start"] + self.collar)) | ||
| collar_zones.append((s["end"] - self.collar, s["end"] + self.collar)) | ||
|
|
||
| def in_collar(t: float) -> bool: | ||
| return any(lo <= t <= hi for lo, hi in collar_zones) | ||
|
|
||
| # Greedy speaker mapping by overlap | ||
| mv: Counter = Counter() | ||
| for g in gt: | ||
| for p in pred: | ||
| ov = min(g["end"], p["end"]) - max(g["start"], p["start"]) | ||
| if ov > 0: | ||
| mv[(g["speaker"], p["speaker"])] += ov | ||
| used, mapping = set(), {} | ||
| for _v, gs, ps in sorted([(v, g, p) for (g, p), v in mv.items()], reverse=True): | ||
| if gs not in mapping and ps not in used: | ||
| mapping[gs] = ps | ||
| used.add(ps) | ||
| inv = {v: k for k, v in mapping.items()} | ||
|
|
||
| # Frame-level scoring | ||
| step = 0.01 | ||
| nf = int((uem_end - uem_start) / step) + 1 | ||
| gf: dict[int, set] = {} | ||
| pf: dict[int, set] = {} | ||
| for s in gt: | ||
| for i in range(max(0, int((s["start"] - uem_start) / step)), min(nf, int((s["end"] - uem_start) / step))): | ||
| gf.setdefault(i, set()).add(s["speaker"]) | ||
| for s in pred: | ||
| mm = inv.get(s["speaker"], f"x_{s['speaker']}") | ||
| for i in range(max(0, int((s["start"] - uem_start) / step)), min(nf, int((s["end"] - uem_start) / step))): | ||
| pf.setdefault(i, set()).add(mm) | ||
|
|
||
| miss = fa = conf = correct = total = 0 | ||
| for i in range(nf): | ||
| t = uem_start + i * step | ||
| if in_collar(t): | ||
| continue | ||
| gs = gf.get(i, set()) | ||
| ps = pf.get(i, set()) | ||
| if gs: | ||
| total += len(gs) | ||
| for s in gs: | ||
| if s in ps: | ||
| correct += 1 | ||
| elif ps: | ||
| conf += 1 | ||
| else: | ||
| miss += 1 | ||
| fa += len(ps - gs if gs else ps) | ||
|
|
||
| ts = total * step | ||
| if ts == 0: | ||
| return None | ||
| return { | ||
| "der": (miss + fa + conf) * step / ts * 100, "miss": miss * step / ts * 100, | ||
| "fa": fa * step / ts * 100, "conf": conf * step / ts * 100, "correct": correct * step / ts * 100, | ||
| "gt_speech_s": ts, "pred_speech_s": sum(s["end"] - s["start"] for s in pred), | ||
| "gt_speakers": len({s["speaker"] for s in gt}), "pred_speakers": len({s["speaker"] for s in pred}), |
There was a problem hiding this comment.
Confusion frames are double-counted, inflating DER
The current frame-level scoring adds both conf and fa for the same confusion frame. Consider a frame where the reference has speaker A but the hypothesis (after global mapping) has speaker C (i.e., wrong speaker):
gs = {A}, ps = {C} (C is a valid GT speaker, just the wrong one)
→ conf += 1 (A not found in ps)
→ fa += len({C} - {A}) = 1 (C is not A, counted as a false alarm too)
Standard NIST DER counts this as one confusion error. Here it is counted as two (conf + fa), making the reported DER approximately 2× the standard value for the confusion component. Under-collar segments mask some of this, but any true speaker-confusion frames outside collar zones are penalised twice.
Consequence: the headline 6.2 % macro-DER may be significantly higher than what dscore or pyannote.metrics would report for the same RTTM files.
To align with standard NIST DER, fa should only count frames where the hypothesis speaks but the reference is completely silent, not frames where a (mapped) hypothesis speaker merely disagrees with the reference speaker:
if gs:
total += len(gs)
for s in gs:
if s in ps:
correct += 1
elif ps:
conf += 1 # counted once, not also as FA
else:
miss += 1
# Only count predicted speakers with no reference counterpart as FA
fa += len(ps - gs)
else:
# No reference speech at this frame → all hypothesis speech is FA
fa += len(ps)With this change fa only grows for the "reference has silence" case, matching the NIST definition.
sarahyurick
left a comment
There was a problem hiding this comment.
Thanks @melllinia ! I did an initial pass for now and added comments for some things that stuck out to me.
sarahyurick
left a comment
There was a problem hiding this comment.
Thanks @melllinia ! Added some more comments.
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
…and format cleanup Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
02bd3c8 to
2990a2d
Compare
|
/ok to test 981ab03 |
Signed-off-by: mmkrtchyan <mmkrtchyan@nvidia.com>
|
/ok to test bc9eb5c |
sarahyurick
left a comment
There was a problem hiding this comment.
Thanks! Also, either in this PR or as a follow-up it will need to add benchmarking: https://github.com/NVIDIA-NeMo/Curator/tree/main/benchmarking
|
|
||
|
|
||
| @dataclass | ||
| class CallHomeReaderStage(ProcessingStage[_EmptyTask, AudioTask]): |
There was a problem hiding this comment.
Can any of these stages be consolidated and/or child stages of existing audio stages?
There was a problem hiding this comment.
SingleSpeakerFilterStage could reuse PreserveByValueStage for the filtering part, but we'd first need a separate stage to count speakers from the segments. Felt simpler to keep it as one stage for the tutorial.
I think the other stages (reader, DER) are dataset-specific with no existing equivalents to reuse.
| logger.info(f"Running stage {idx}/{len(stages) - 1}: {stage._name} ({len(todo_tasks)} tasks)") | ||
| stage_t0 = time.time() | ||
| pipeline = Pipeline(name=f"stage_{idx}_{stage._name}", stages=[stage]) | ||
| output = pipeline.run(executor=executor, initial_tasks=todo_tasks) |
There was a problem hiding this comment.
We typically try to avoid running pipelines with non-empty initial_tasks.
There was a problem hiding this comment.
We use initial_tasks here for checkpointing here, so on resume only the unfinished tasks get re-run. This avoids reprocessing work that already completed.
|
/ok to test 13d6a2c |
Summary
InferenceSortformerStagefor speaker diarization using Streaming Sortformer (NeMo)Test plan
pytest tests/stages/audio/inference/test_sortformer.py— 19/19 passedruff checkandruff formatclean