Some Matroska/webm videos yield zero extracted frames (empty output / 500)

[keighrim]

Running app-swt-detection (v8.6, classifier path) over some Matroska/webm videos extracts zero frames: the classifier loop never runs and the request ends in a 500 with TypeError: 'NoneType' object is not iterable at the output stage. Root cause not yet confirmed. I observed the frameCount value is correct immediately after vdh.capture() (cv2-based video reader code), but the value used a few lines later to compute the total timeline has become 0, so the image extraction start/end window is 0..0 and nothing is sampled. With zero frames classified, the result accumulator stays None and the output loop (which has no empty-input guard) raises the TypeError.

Suspected area

After the PyAV migration (#379) , SWT's classifier path still uses the older cv2-backed helpers (vdh.capture, vdh.extract_frames_as_images) alongside newer code (vdh.get_framerate, tuh.convert). The frameCount appears correct when cv2 sets it but 0 by the time it is consumed, so a mismatch between the cv2 and PyAV code paths (or between how a document property is written vs. read) is the leading suspect. Not confirmed.

Open questions

• Why does frameCount change from a correct value to 0 between being set and being read, with no explicit reset in between?
• Is the collapse specific to the cv2 path, the PyAV path, or their interaction?
• Independent of the above: SWT should handle "zero frames extracted" gracefully rather than 500-ing with an opaque TypeError.

[keighrim]

The issue has three layers of problems, each caused by the one below it.

The direct cause of the empty output is that the document's frameCount prop (written at first file read time by SDK) becomes 0 when being read (for image extraction), so the timeline computes as 0, the sampling window collapses to [0..0], no frames are extracted from zero-length sequence, and the empty result trips the TypeError. capture() (cv2) writes a correct frameCount, but the very next get_framerate() call can't see it ans returns 0.

That was caused by get_framerate() needlessly re-opening the container. It decides whether fps is already known with if key in video_document; the test fails to match the value capture() just set, so it falls through to open_container(), whose PyAV path derives frameCount/duration from per-stream fields that some Matroska/webm files leave unset (stream.frames == 0, stream.duration is None) and writes 0, ignoring the container-level duration — overwriting the good value.

And that membership test fails because of a bug in __contains__ membership check in the serialization layer. Document.add_property() cannot write into the document's own JSON (a Document is treated as read-only — issue #226), so app-added properties land in _props_pending. Document.get/get_property was overridden to read _props_pending, but __getitem__/__contains__ were not — they inherit Annotation.__getitem__, which consults only _props_original and _props_ephemeral. So a freshly added property is readable through get_property yet invisible to the in operator until the property is serialized and deserlized again. It becomes visible to in only after a serialize -> deserialize round trip, when it migrates _props_pending -> capital Annotation -> _props_ephemeral; the bug is therefore strictly intra-process.

This answers the three questions in the issue: the reset comes from get_framerate() -> open_container(); it originates in the PyAV path overwriting cv2's value while the membership test silently fails; and the app should also guard the zero-frame case.

Proposed fix

The failure spans mmif.serialize and mmif.utils.video_document_helper. The serialize defect is the root cause and must be fixed regardless of anything else; the larger video-loader rework is orthogonal cleanup, not an alternative to it.

Required: fix the getter/membership asymmetry (serialization layer)

Override Document.__getitem__ (and therefore __contains__) to also consult _props_pending, so the getter and the in operator agree. This makes get_framerate()'s existing membership test correct without touching video_document_helper, and closes the whole class of "property added via add_property is invisible to in until serialized" bugs for every caller, not just SWT.

Required: container-level duration fallback in open_container()

Current implementation (mmif/utils/video_document_helper.py, in open_container()) — both frame_count and duration_ms fall through to 0 when the per-stream fields are unset, which is exactly what webm/mkv hit:

mmif-python/mmif/utils/video_document_helper.py

Lines 104 to 116 in 21ca1e0

	if stream.frames > 0:
	frame_count = stream.frames
	elif stream.duration is not None and stream.average_rate is not None:
	frame_count = int(round(float(stream.duration) * time_base
	* float(stream.average_rate)))
	else:
	frame_count = 0
	if stream.duration is not None:
	duration_ms = int(round(float(stream.duration) * time_base * 1000))
	elif frame_count > 0 and fps > 0:
	duration_ms = int(round(frame_count / fps * 1000))
	else:
	duration_ms = 0

(problematic matroska files land L110 and L116)

We can add a new conditional branch to fall back to the container-level duration (container.duration / av.time_base, microseconds -> seconds) when the per-stream fields are absent, and derive frameCount from the resolved duration, so webm/mkv get non-zero duration/frameCount. MP4 behavior is unchanged (should still takes the stream.duration branch). See webm-duration-bug.md. This is needed independent of the getter fix: once get_framerate() stops needlessly re-opening the container, open_container() is still the metadata reader and must produce correct values on these files. Removing cv2 (below) makes it the only metadata reader, so its correctness matters more, not less.