DosForceDup (INT 21h/AH=46h) does not validate NewHandle: out-of-bounds JFT access + dup2(h,h) closes the handle

[geraldo-netto]

DosForceDup (INT 21h/AH=46h) does not validate NewHandle: out-of-bounds JFT access + dup2(h,h) closes the handle

Summary

DosForceDup() in kernel/dosfns.c validates OldHandle but never validates
NewHandle against ps_maxfiles, and it has no OldHandle == NewHandle
short-circuit. Because NewHandle comes straight from the caller's CX in
INT 21h/AH=46h (the DOS "force duplicate handle" / dup2 primitive), any
unprivileged program can:

1. force an out-of-bounds read, and conditionally an out-of-bounds write,
   into memory adjacent to the Job File Table (JFT); and
2. make dup2(h, h) close an open handle instead of being a no-op, which is
   a fully deterministic, always-reproducible failure.

Both stem from the same missing validation and are fixed by the same small patch.

Affected code

kernel/dosfns.c (line numbers from current master):

COUNT DosForceDup(unsigned OldHandle, unsigned NewHandle)
{
 psp FAR *p = MK_FP(cu_psp, 0);
 sft FAR *Sftp;

 /* Get the SFT block that contains the SFT */
 if ((Sftp = get_sft(OldHandle)) == (sft FAR *) - 1) /* OldHandle validated */
 return DE_INVLDHNDL;

 /* now close the new handle if it's open */
 if ((UBYTE) p->ps_filetab[NewHandle] != 0xff) /* <-- OOB read: NewHandle unchecked */
 {
 COUNT ret;

 if ((ret = DosClose(NewHandle)) != SUCCESS)
 return ret;
 }

 /* If everything looks ok, bump it up. */
 p->ps_filetab[NewHandle] = p->ps_filetab[OldHandle]; /* <-- OOB write: NewHandle unchecked */
 /* possible hazard: integer overflow ska*/
 Sftp->sft_count += 1;
 return SUCCESS;
}

Reached from the INT 21h dispatcher in kernel/inthndlr.c:

 /* Force Duplicate File Handle */
 case 0x46:
 rc = DosForceDup(lr.BX, lr.CX); /* lr.CX = NewHandle, fully caller-controlled */
 goto short_check;

Contrast with the correct guard used everywhere else in the kernel,
get_sft_idx() in kernel/dosfns.c:

int get_sft_idx(unsigned hndl)
{
 psp FAR *p = MK_FP(cu_psp, 0);
 int idx;

 if (hndl >= p->ps_maxfiles) /* the check DosForceDup is missing */
 return DE_INVLDHNDL;
 ...
}

Why it is wrong

The JFT is ps_maxfiles bytes long, pointed to by ps_filetab (see
hdr/process.h: default ps_files[20] at PSP offset 0x18, ps_maxfiles at
0x32, ps_filetab at 0x34). Valid handle indices are 0 .. ps_maxfiles-1.

DosForceDup indexes ps_filetab[NewHandle] for read (the "is it open?"
test) and, when the byte read is 0xFF, for write, with NewHandle taken
verbatim from CX (0x0000 .. 0xFFFF).

• The read at ps_filetab[NewHandle] is unconditional and out of bounds for
  any NewHandle >= ps_maxfiles. ps_filetab is a FAR pointer, so the effective
  address is FP_SEG(ps_filetab):(FP_OFF(ps_filetab) + NewHandle) with the
  offset wrapping modulo 64 KiB inside the segment. With the default in-PSP JFT
  at offset 0x18, a large CX reads elsewhere in the PSP segment (environment
  pointer, PSP header, etc.).
• The write ps_filetab[NewHandle] = ps_filetab[OldHandle] corrupts that
  same out-of-bounds byte whenever the pre-existing byte reads as 0xFF
  ("unused slot"), which is common in adjacent DOS memory. The written value is
  OldHandle's SFT index — attacker-influenced.
• MS-DOS returns error 06h (invalid handle) when CX is out of range for
  AH=46h. FreeDOS instead performs the access and can return SUCCESS.

Separately, when OldHandle == NewHandle and the handle is open, the
function first calls DosClose(NewHandle) — which sets ps_filetab[NewHandle] = 0xFF and decrements the SFT count — and then executes
ps_filetab[NewHandle] = ps_filetab[OldHandle], i.e. ps_filetab[h] = ps_filetab[h], copying back the now-closed 0xFF. The handle is left closed
even though the call reports SUCCESS. MS-DOS defines dup2(h, h) as a
successful no-op that leaves the handle open.

Reproduction

Case A — deterministic: dup2(h, h) closes an open handle

100% reproducible on real hardware or any emulator (DOSBox, 86Box, QEMU), no
memory-layout assumptions.

Assemble as a .COM (NASM: nasm -f bin dup2same.asm -o dup2same.com):

; dup2same.asm - demonstrates that DosForceDup mishandles OldHandle == NewHandle
 org 100h

 ; open this program's own file read-only to obtain a real handle
 mov ax, 3D00h ; INT 21h/AH=3Dh open, AL=0 read-only
 mov dx, fname
 int 21h
 jc fail_open
 mov [handle], ax ; save handle h (e.g. 5)

 ; dup2(h, h): AH=46h, BX = OldHandle = h, CX = NewHandle = h
 mov ah, 46h
 mov bx, [handle]
 mov cx, [handle]
 int 21h ; MS-DOS: success, handle still open
 ; FreeDOS: returns success but CLOSES the handle

 ; probe the handle: seek to current position (AH=42h, AL=1, offset 0)
 mov ax, 4201h
 mov bx, [handle]
 xor cx, cx
 xor dx, dx
 int 21h
 jc handle_closed ; FreeDOS: CF set, AX=6 (invalid handle) -> BUG
 ; MS-DOS: CF clear -> handle survived, correct

 mov dx, msg_ok
 jmp print_exit
handle_closed:
 mov dx, msg_bug
 jmp print_exit
fail_open:
 mov dx, msg_openfail
print_exit:
 mov ah, 09h
 int 21h
 mov ax, 4C00h
 int 21h

handle dw 0
fname db "DUP2SAME.COM",0
msg_ok db "OK: handle survived dup2(h,h)",13,10,'$'
msg_bug db "BUG: dup2(h,h) closed the handle (AX=6)",13,10,'$'
msg_openfail db "could not open own file",13,10,'$'

Expected on MS-DOS / PC-DOS / DR-DOS: OK: handle survived dup2(h,h).
Observed on FreeDOS: BUG: dup2(h,h) closed the handle (AX=6).

Equivalent under DEBUG.COM (open handle 5 first, then):

-A
xxxx:0100 MOV AH,46
xxxx:0102 MOV BX,0005
xxxx:0105 MOV CX,0005
xxxx:0108 INT 21
xxxx:010A INT 20
xxxx:010C
-G =100 10A

After this, JFT entry 5 reads FF (closed) on FreeDOS; on MS-DOS it is unchanged.

Case B — out-of-bounds JFT access (security impact)

The missing bounds check lets CX address up to 64 KiB from the JFT. Minimal
trigger (any .COM):

 mov ah, 46h
 mov bx, 0001h ; OldHandle = STDOUT (open, valid)
 mov cx, 0FFFFh ; NewHandle = 65535, far out of range
 int 21h
 ; A conforming DOS returns CF=1, AX=6.
 ; FreeDOS performs ps_filetab[0xFFFF] read, and if that byte == 0xFF,
 ; writes ps_filetab[OldHandle] there -> out-of-bounds write, CF=0.

Whether a given CX yields a crash, silent corruption, or an accidental error
depends on the byte currently at the target address, which is exactly why an
unchecked, caller-controlled index here is dangerous. Sweeping CX from
ps_maxfiles upward and watching for a CF=0 (success) return reliably
exhibits the out-of-bounds write in an emulator with a debugger attached.

Suggested fix

Two guards are added, and their order matters:

1. Bounds-check NewHandle first, before it is ever used as a JFT index.
   This closes both the out-of-bounds read and the out-of-bounds write for any
   caller-supplied CX.
2. Short-circuit OldHandle == NewHandle to SUCCESS, but only after
   get_sft(OldHandle) has confirmed the source handle is actually open. Placing
   the no-op check after the get_sft validation is deliberate: dup2(h, h)
   where h is in range but not open must still fail with DE_INVLDHNDL
   (matching MS-DOS), so the short-circuit must not precede that check.

Explicit diff against current master:

--- a/kernel/dosfns.c
+++ b/kernel/dosfns.c
@@ -683,24 +683,34 @@ COUNT DosForceDup(unsigned OldHandle, unsigned NewHandle)
 COUNT DosForceDup(unsigned OldHandle, unsigned NewHandle)
 {
 psp FAR *p = MK_FP(cu_psp, 0);
 sft FAR *Sftp;
 
+ /* Validate NewHandle before it is ever used as a JFT index. NewHandle
+ is the caller's CX from INT 21h/AH=46h; without this check an
+ out-of-range value indexes ps_filetab[] outside the ps_maxfiles-entry
+ Job File Table, causing an out-of-bounds read and (when the byte read
+ is 0xFF) an out-of-bounds write. A conforming DOS returns error 06h. */
+ if (NewHandle >= p->ps_maxfiles)
+ return DE_INVLDHNDL;
+
 /* Get the SFT block that contains the SFT */
 if ((Sftp = get_sft(OldHandle)) == (sft FAR *) - 1)
 return DE_INVLDHNDL;
 
+ /* dup2(h, h) is a successful no-op in MS-DOS. This must come AFTER the
+ get_sft(OldHandle) check above, so that duplicating an unopened handle
+ onto itself still fails. Without it, the code below would DosClose()
+ the handle and then copy the just-cleared 0xFF back over it, leaving a
+ handle that the caller believes is open but is actually closed. */
+ if (OldHandle == NewHandle)
+ return SUCCESS;
+
 /* now close the new handle if it's open */
 if ((UBYTE) p->ps_filetab[NewHandle] != 0xff)
 {
 COUNT ret;
 
 if ((ret = DosClose(NewHandle)) != SUCCESS)
 return ret;
 }
 
 /* If everything looks ok, bump it up. */
 p->ps_filetab[NewHandle] = p->ps_filetab[OldHandle];
 /* possible hazard: integer overflow ska*/
 Sftp->sft_count += 1;
 return SUCCESS;
 }

Post-patch behavior:

• Case A prints OK: handle survived dup2(h,h) — the handle is left open.
• Case B returns CF=1, AX=06h (invalid handle) with no memory access beyond
  the JFT.
• dup2(closed_handle, closed_handle) still returns CF=1, AX=06h, because the
  no-op short-circuit sits behind the get_sft(OldHandle) validation.

Notes

• Still present on origin/UNSTABLE (the NewHandle index there, lines 704/716,
  remains unchecked; only an SFT-status guard was added around the write).
• This defect was found using adversarial AI agents performing a static
  review of the 16-bit resident kernel — multiple independent agents auditing the
  same code and cross-checking each other's findings against the source. The
  result was then verified by hand against the kernel sources. It has not yet
  been build-tested; the reproducers above are the recommended validation.