gh-118814: Fix crash in _PyArg_UnpackKeywordsWithVararg

[picnixz]

The cause of the crash, if I'm not mistaken, is that we currently have:

if (nkwargs) {
current_arg= ...
}
else {
current_arg=NULL;
}
if (nargs<vararg&&i!=vararg) { buf[i] =current_arg;
}
else {
buf[i+1] =current_arg;
}
if (current_arg) {
Py_DECREF(current_arg);
--nkwargs;
}

If we have the following unrolling:

• first iteration: current_arg != NULL and use of buf[i + 1] = current_arg
• second iteration: current_arg == NULL -> buf[i] = NULL so we smash what we just stored, hence the subsequent segfaults. The patch consists of

if (nargs<vararg&&i<vararg) {
if (current_arg!=NULL) {
buf[i] =current_arg;
}
}

so that we are sure that we don't overwrite something. I don't think it's also correct to overwrite buf[i] if the latter is not NULL but I may be wrong. If I'm wrong, the patch should simply check that we did not set buf[i + 1] to a non-NULL current_arg in the previous iteration.

• Issue: TypeVar with bad arguments segfault/misbehavior #118814

[picnixz]

Ok, it's not fixed. I found another bug and the fix doesn't work. I need to think a bit more.

[picnixz]

Hopefully it's now fixed (the issue, if I'm correct, was about the check i != vararg instead of i < vararg (if i comes after vararg, then it should also be offsetted).

[erlend-aasland]

This looks correct; I'll be able to review it in a day or two.

[erlend-aasland]

This looks good to me, but I'd like @serhiy-storchaka's eyes on this, as he's more knowledgable about the arg parsing corners of the code base.

[serhiy-storchaka]

The design of this function is questionable:

• It seems that vararg is always equal maxpos. So this parameter is redundant.
• Placing the tuple of var-positional arguments between keyword-or-positional and keyword-only arguments complicates the code.
• This is the only buf item that contains non-borrowed reference. This is error prone.
• Not always we need a tuple. Creating a tuple can add an overhead. This also makes impossible a fast path.

I will address this in other issue.

[serhiy-storchaka]

Check the result for NULL.

[serhiy-storchaka]

There may be less than vararg items in args.

[picnixz]

Right and this is something I didn't check. I need to add tests for that (I would like to suggest a way to generate more clinic tests because there is a lack of coverage).

[serhiy-storchaka]

This leaves buf[i] not initialized, isn't?

[picnixz]

Urgh... yes. I should probably change the way it's being checked... like a boolean checking whether I replaced something before or not. But depending on whether you want to change the logic of this function, I may delay the commit.

[picnixz]

Urgh, actually there's another issue:

vararg_with_many_argsa: objectb: objectc: objectd: object*args: objecte: boolf: bool=Falseg: bool=False

• Calling ac_tester.vararg_with_many_args(1, 2, 3, 4, e=5) -> says that 'f' is required
• Calling ac_tester.vararg_with_many_args(1, 2, 3, 4, e=5, f=6) -> segfault

I wanted to avoid rewriting the entire function but I think it'll just be easier to just rewrite entirely... I just need to understand the internals a bit more (the variable naming doesn't help IMO...)

[picnixz]

Placing the tuple of var-positional arguments between keyword-or-positional and keyword-only arguments complicates the code.

And yes... I'm pretty sure that my issue is about indices... (I like coding and so, but the topic that I always disliked whether it's in maths or in dev is about indices!!! (the irony being I'm doing stuff that requires indices a lot...)))

[picnixz]

@serhiy-storchaka I think it's better to re-address the design of the function itself. Do you want to create a separate issue and work on that? (I'm only available today and will be leaving on holidays for roughly 10 days)

I should also confess that I don't have the courage to think of it today...

[serhiy-storchaka]

The solution for this issue can be simpler: #122664.

There are unresolved issues

[picnixz]

Closing in favor of #122664.

If more test coverage is needed, we'll just extract my tests from this one in another PR.