gh-111178: fix UBSan failures in Modules/overlapped.c

[picnixz]

This PR fixes the UBSan failures and addresses some minor cosmetic changes. PEP-7 changes were not applied since they could scramble the diff but other semantic changes affecting the signature of touched functions may have been done.

• Issue: UBSan: Calling a function through pointer to incorrect function type is undefined behavior #111178

[picnixz]

There seems to be a Windows warning that I didn't see as it was in a non-compiled section for me.

[chris-eibl]

Yeah, the whole file overlapped.c is only compiled on Windows platforms.

To get rid of the warnings, I think, all those Overlapped_clear(self) can replaced with (void)Overlapped_clear(op);, which you already did in one spot.

I assume, there you prefixed with (void) to suppress warnings like "unused return value"?

Maybe instead change the return type of Overlapped_clear to void, since it is static and its return type is never used?

[picnixz]

No, the problem here is that self is not a PyObject * so I just need to change it in the functions that call Overlapped_Clear(). The unused return is not the problem (it's just that I fixed the file without any IDE help...)

[chris-eibl]

Sure! Having another thought, then maybe not change Overlapped_Clear() to take a PyObject *, i.e. keep it unchanged and all of its occurrences wouldn't have to be touched and the PR could be kept smaller?

Is this possible?

[picnixz]

Sure! Having another thought, then maybe not change Overlapped_Clear() to take a PyObject *, i.e. keep it unchanged and all of its occurrences wouldn't have to be touched and the PR could be kept smaller?

No that's the problem of the PR. The signature of the clear function must be int (*)(PyObject *) and not int (*)(OverlappedObject *). What I'm trying to fix are undefined behaviours reported by clang (currently, those are suppressed because we are adding an explicit cast, but we don't want that to be kept)

[chris-eibl]

Yeah, I know. But AFAICT Overlapped_clear() is not used in OverlappedType.tp_clear, so there shouldn't be a problem?
It is used here only as an internal static function and can always get the OverlappedObject *self = OverlappedObject_CAST(op); parameter?

In contrast to e.g. Overlapped_dealloc, which of course needs this fix ...

[picnixz]

OH! There wasn't an issue with the clear! I... missed the comment... oook. I'll improve the comment as well and will shorten the PR. My bad! Thanks for noticing this!

[chris-eibl]

Oh that looks so much nicer now :)

Sorry for expressing myself clumsy in the first place. I am not a native English speaker.

Your UBSan fixes are a great work towards a better code base and a lot of effort!

Thank you very much for all of them!