Reject vertex/triangle counts whose allocation size would wrap 32 bits

[Fedr]

Summary

Follow-up to the _ctmStreamReadSTRING heap-overflow fix (#2), addressing the secondary item flagged in the same coordinated disclosure by Kamal Sentassi of S9S Security Research: the file-declared vertex and triangle counts drive malloc directly.

ctmLoadCustom reads mVertexCount and mTriangleCount as 32-bit values straight from the file, then allocates count * stride for the mesh arrays and for every temporary buffer in the RAW/MG1/MG2 decoders. On a platform where size_t is 32-bit — the wasm32 target MeshLib builds OpenCTM for under Emscripten — that multiplication wraps, producing an undersized buffer that the decoder then overruns. This is the same integer-overflow-to-heap-write class as #2, reachable from the public ctmLoad on a single crafted file.

Independently of size_t width, some count * stride products are computed in 32-bit CTMuint and wrap on any platform: _ctmAllocateFloatMaps truncated aChannels * sizeof(CTMfloat) * mVertexCount into a CTMuint size, and the packed int/float temp buffers in stream.c allocate aCount * aSize * 4 with 32-bit operands. On a 64-bit size_t these sit behind the multi-GB mesh allocations, but under Linux memory overcommit those can succeed, so the truncated float-map write is reachable there too — not merely a large-allocation DoS.

Fix (lib/openctm.c)

Immediately after the counts are read, reject any mVertexCount or mTriangleCount above 0xFFFFFFFF / 16, 16 bytes (4 floats per vertex) being the largest real stride. The file is rejected with CTM_BAD_FORMAT. A single up-front check covers every count-driven allocation in the loader; legitimately large-but-non-wrapping counts still fall through to the existing CTM_OUT_OF_MEMORY path.

The bound is UINT_MAX / 16, not SIZE_MAX / 16: the counts are 32-bit, and count * 3, count * 4 and the stream.c temp buffer sizes are still computed in 32-bit CTMuint on every platform, so count * stride has to stay below 2^32 regardless of size_t width. On 32-bit size_t the two bounds coincide anyway. On 64-bit this essentially matches the pre-existing effective ceiling: the 4-channel attribute-map allocation and packed buffers already wrapped their 32-bit products at 2^28 elements; only meshes without attribute maps between 2^28 and ~3.6e8 elements loaded before and are rejected now.

The float-map allocation size is also computed in size_t so it is correct by construction, independent of the guard.

Also (lib/compressMG1.c)

_ctmUncompressMesh_MG1 leaked its temporary indices buffer when the packed-int read of the INDX chunk failed. Freed on that path, matching the sibling error paths.

Verification

Built the library for i686 (32-bit size_t, matching wasm32) and for x86-64, driven through the public ctmLoad.

• Before, 32-bit: a RAW file declaring 0x15555556 vertices (whose count * 4 * 3 wraps to 8 bytes) segfaults as the decoder writes past the 8-byte buffer.
• After, 32-bit: the same file is rejected with CTM_BAD_FORMAT and no crash.
• After, 64-bit: an oversized count is rejected by the same guard (previously it fell through); a merely large-but-non-wrapping count still degrades to a clean error with no ASan report.
• Regression: a valid RAW mesh still loads on both builds.

Notes

• The _ctmClearMesh call added on the mVertices failure path is a consistency/future-proofing measure — mIndices/mNormals are still NULL at that point, so it frees nothing new; it is not itself a leak fix. It matches the sibling error paths and mFileComment remains owned by the context until ctmFreeContext.
• A very large mUVMapCount / mAttribMapCount drives a long allocation loop rather than a single wrapping multiply, so it self-limits via CTM_OUT_OF_MEMORY; left as-is. The other error-path leaks flagged in the original report were addressed in Fix heap-buffer-overflow in _ctmStreamReadSTRING and stop parsing on string read errors #2.