feat: Add region-based in-place splicing and force overwrite option for `extract` #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

adrianbrowning wants to merge 6 commits into main from 21-extract-destructive-full-file-overwrite-instead-of-in-place-region-splice-aliased-file-paths-clobber-each-other

.bumpy/extract-region-splice-force.md

-Original file line number
+Diff line change
@@ -0,0 +1,21 @@
+    ---
+    "mdcode-ts": minor
+    ---
+    **Behaviour change:** `extract` no longer overwrites pre-existing files that a block describes in
+    full. Such targets are skipped with a warning; pass `--force` for the old behaviour. When anything is
+    skipped the count is reported even under `--quiet` and the CLI exits with status 2, so a pipeline
+    cannot mistake "wrote nothing" for success.
+    `extract` now splices `region=` blocks in place instead of rewriting the whole file, so surrounding
+    code and untouched regions survive, and aliased `file=` paths pointing at one file resolve to a
+    single write. Region markers are written and matched in the block language's own comment syntax, so
+    shell, SQL, CSS and block-comment markers splice instead of being duplicated.
+    A target is left byte-identical rather than spliced when it is a symlink, is not valid UTF-8, has a
+    region it never closes or names inconsistently, declares a region twice, or when the blocks for one
+    file mix `region=` with whole-file blocks. Writes go through a temp file and `rename`, preserving the
+    target's permission bits, so an interrupted write cannot truncate a source file.
+    Install and import docs now name the published package `mdcode-ts`; they previously pointed at a
+    name that resolved to the upstream fork.

CLAUDE.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -6,24 +6,23 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
  
    TypeScript port of [szkiba/mdcode](https://github.com/szkiba/mdcode) - a Markdown code block authoring tool for extracting, updating, and managing code blocks within markdown documents.

    This is a **pnpm workspace monorepo** with three packages:

    - `packages/mdcode` - Main library and CLI

    - `packages/usage` - E2E integration tests

    - `packages/example` - Example usage (no tests)

    This is a **pnpm workspace monorepo** with two packages:

    - `packages/mdcode` - Main library and CLI, published as `mdcode-ts`

    - `packages/usage` - Integration tests that consume the built package

    ## Essential Commands

    ### Testing (ALWAYS RUN BOTH)

    ```bash

    # Run ALL tests (mdcode-ts + usage packages = 138 total tests)

    # Run ALL tests (mdcode-ts + usage packages )

    pnpm test

    # Watch mode during development

    pnpm --filter mdcode-ts test:watch

    # Individual packages

    pnpm --filter mdcode-ts test    # 51 unit tests

    pnpm --filter usage test        # 87 E2E tests

    pnpm --filter mdcode-ts test    # unit tests

    pnpm --filter usage test        # E2E tests

    ```

    **NOTE**: `pnpm test` is `pnpm -r test` — it already covers both packages. `pnpm test:all` also exists but just re-runs `usage` a second time.

    @@ -91,7 +90,7 @@ Type signature: `(options: {tag, meta, code}) => string | Promise<string>`
  
    Use `defineTransform()` helper for type safety:

    ```typescript

    import { defineTransform } from 'mdcode';

    import { defineTransform } from 'mdcode-ts';

    export default defineTransform(({tag, code}) => {

      if (tag === 'sql') return code.toUpperCase();

    @@ -123,10 +122,11 @@ Original design used `unified` + `remark-parse`, but switched to custom state ma
  
    - No AST overhead for simple code block extraction

    ### Test Organization

    - **Root tests/** - Parser and transformer unit tests (not used currently)

    - **packages/mdcode/src/*.test.ts** - Co-located unit tests (16 tests)

    - **packages/usage/tests/** - E2E workflow tests (5 tests)

    - Import path from root: `../packages/mdcode/src/...`

    - **packages/mdcode/src/\*\*/\*.test.ts** - Co-located unit tests (parser, region, commands/extract, commands/update)

    - **packages/mdcode/tests/examples/** - Fixture-driven tests over the worked examples

    - **packages/usage/tests/** - Integration tests; `cli-integration.test.ts` spawns the **built** `dist/main.js`, so run `pnpm build` first

    See `TESTING.md` for the full layout.

    ### File Imports Must Use .ts Extension

    TypeScript config uses `allowImportingTsExtensions: true`:

    @@ -139,6 +139,6 @@ import { parse } from './parser';
  
    ```

    ## Before Committing

    1. `pnpm test` - Ensure ALL 138 tests pass

    1. `pnpm test` - Ensure ALL tests pass

    2. `pnpm build` - Ensure build succeeds

    3. `pnpm -r lint:ts` - Type check all packages

TESTING.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -2,52 +2,64 @@
  
    ## Running All Tests

    ### Full Test Suite (ALWAYS RUN THIS)

    ```bash

    pnpm -w run test:all

    pnpm test

    ```

    This runs tests for all packages in the workspace:

    - Main mdcode package (16 unit tests)

    - Usage package (5 E2E tests)

    Total: **21 tests**

    **Note**: `pnpm test` only runs the mdcode unit tests. Use `pnpm -w run test:all` to run everything.

    `pnpm test` is `pnpm -r test`: it already runs **both** packages — `mdcode-ts` unit tests and the

    `usage` integration tests. (`pnpm test:all` also exists, but it just runs the `usage` package a

    second time.)

    Test counts are deliberately not recorded here; they rot. Run the suite to see them.

    ### Individual Package Tests

    ```bash

    # Main mdcode package (unit tests)

    pnpm --filter @gcm/mdcode test

    # mdcode-ts unit tests

    pnpm --filter mdcode-ts test

    # Usage package (E2E tests)

    pnpm --filter @gcm/mdcode-usage test

    cd packages/usage && pnpm test  # Alternative

    # Watch mode

    pnpm --filter mdcode-ts test:watch

    # usage integration tests

    pnpm --filter usage test

    ```

    ## Test Structure

    Two packages, `packages/mdcode` (published as `mdcode-ts`) and `packages/usage`.

    ### Test Locations

    - **Root tests**: `/tests/` - Core functionality tests

      - `parser.test.ts` - Markdown parsing and walking (8 tests)

      - `transform.test.ts` - Transformer functionality (8 tests)

    - **E2E tests**: `packages/usage/tests/e2e.test.ts` - End-to-end integration (5 tests)

    - **Example package**: `packages/example/` - No tests, just runnable examples

    - **Co-located unit tests** — `packages/mdcode/src/**/*.test.ts`

      - `region.test.ts` — marker matching, region splicing, and its refusal cases

      - `parser.test.ts` — info-string and fenced-block parsing

      - `commands/extract.test.ts` — in-place splicing, `--force`, and every refusal path

      - `commands/update.test.ts` — filling blocks from source regions

    - **Fixture-driven tests** — `packages/mdcode/tests/examples/integration.test.ts`, against the

      worked examples under `packages/mdcode/tests/examples/`

    - **Integration tests** — `packages/usage/tests/`

      - `cli-integration.test.ts` spawns the **built** CLI at `packages/mdcode/dist/main.js`

      - the rest exercise the public library API as an external consumer would

    ### Important Notes

    - Import paths from root tests must use `../packages/mdcode/src/...`

    - **ALWAYS test all packages** - run `pnpm test` from workspace root

    - E2E tests validate extract/update workflows with real files

    - `packages/usage/tests/cli-integration.test.ts` runs `dist/`, not `src/`. **Run `pnpm build` before

      it** or you will be testing the previous build.

    - Region fixtures live in `packages/mdcode/tests/testdata/region/` and are compared byte-for-byte, so

      trailing newlines matter.

    - Tests that assert on warnings use `mock.method(console, "error", …)` with `mock.restoreAll()` in a

      `finally`, so a failing assertion cannot leak the stub into sibling tests.

    ## Before Committing

    1. **`pnpm -w run test:all`** - Ensure ALL 21 tests pass (mdcode + usage packages)

    2. **`pnpm build`** - Ensure build succeeds

    3. **Test examples** - Manually run at least one example:

       ```bash

       pnpm --filter @gcm/mdcode-example example:list

       pnpm --filter @gcm/mdcode-example example:extract

       ```

    1. **`pnpm test`** — all packages pass

    2. **`pnpm build`** — build succeeds, and refreshes `dist/` for the CLI tests

    3. **`pnpm -r lint`** — type check and ESLint are clean

    ## Adding New Tests

    - **Parser/core**: Add to `/tests/parser.test.ts`

    - **Transform**: Add to `/tests/transform.test.ts`

    - **E2E workflows**: Add to `packages/usage/tests/e2e.test.ts`

    - **New commands**: Add unit tests to root `/tests/` or create new test file

    - **Region/marker behaviour** → `packages/mdcode/src/region.test.ts`

    - **A command's behaviour** → `packages/mdcode/src/commands/<command>.test.ts`

    - **CLI flags, exit codes, stderr** → `packages/usage/tests/cli-integration.test.ts`

    - **Public API as a consumer sees it** → `packages/usage/tests/library-usage.test.ts`

examples/CLI_EXAMPLES.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -22,7 +22,7 @@ Comprehensive guide to using mdcode from the command line.
  
    ### Global Install (npm)

    ```bash

    npm install -g mdcode

    npm install -g mdcode-ts

    ```

    After installation, you can run `mdcode` from anywhere:

    @@ -36,7 +36,7 @@ mdcode list README.md
  
    ### Global Install (pnpm)

    ```bash

    pnpm install -g mdcode

    pnpm install -g mdcode-ts

    ```

    Usage is identical to npm installation:

    @@ -51,16 +51,16 @@ mdcode --help
  
    No installation required - run directly:

    ```bash

    pnpm dlx mdcode list README.md

    pnpm dlx mdcode extract --lang js docs/*.md

    pnpm dlx mdcode update --transform ./my-transformer.js README.md

    pnpm dlx mdcode-ts list README.md

    pnpm dlx mdcode-ts extract --lang js docs/*.md

    pnpm dlx mdcode-ts update --transform ./my-transformer.js README.md

    ```

    ### Run Without Installing (npx)

    ```bash

    npx mdcode list README.md

    npx mdcode --help

    npx mdcode-ts list README.md

    npx mdcode-ts --help

    ```

    ---

    @@ -212,6 +212,27 @@ mdcode list --json -l python -m type=example docs/
  
    Extract code blocks to files based on their `file` metadata.

    ### Existing Files Are Not Overwritten

    `extract` never destroys work you did not ask it to touch:

    - Blocks with `region=` are **spliced in place** — surrounding code and untouched regions survive.

    - Blocks without `region=` describe a whole file. If that file already exists it is **skipped with a

      warning**; pass `--force` to overwrite it.

    - A target is also skipped, and left byte-identical, when it is a symlink, is not valid UTF-8, has a

      region the file never closes, or when the blocks for one file mix `region=` with whole-file blocks.

    When anything is skipped, `extract` reports the count even under `--quiet` and exits with a non-zero

    status, so a pipeline cannot mistake "wrote nothing" for success.

    ```bash

    # Skipped with a warning if the target already exists

    mdcode extract README.md

    # Overwrite whole-file targets

    mdcode extract --force README.md

    ```

    ### Basic Usage

    ```bash

    @@ -291,6 +312,22 @@ curl https://example.com/docs.md | mdcode extract -q
  
    Update markdown code blocks from source files or transform them with custom functions.

    ### `file=` Is Trusted, By Design

    `update` reads whatever path a block's `file=` names, including paths that leave the markdown's own

    directory — `file=../src/app.js` from a `docs/` folder is normal and supported. The path is an

    explicit instruction from whoever wrote the markdown, so it is honoured as written and is **not**

    confined to `--base-path`.

    The consequence is that `update` will inline the contents of any file the process can read, and those

    contents land in the markdown. Treat markdown from an untrusted source the way you would treat a

    script from an untrusted source: review it before running `update` over it, and do not run `update`

    on contributor-supplied markdown in an environment holding secrets.

    (`extract`, which *writes*, is confined to `--dir` and refuses paths that escape it. The asymmetry is

    deliberate: reading a path you named is what you asked for, whereas writing outside the output

    directory never is.)

    ### Update from Source Files (Default Mode)

    Updates code blocks by reading from files specified in the `file` metadata attribute:

    @@ -699,7 +736,7 @@ mdcode update -l sql -f queries.sql --stdout README.md
  
    ## Comparison with Original mdcode

    This TypeScript implementation is a **drop-in replacement** for the original Go-based [szkiba/mdcode](https://github.com/szkiba/mdcode). It maintains 100% CLI compatibility.

    This TypeScript implementation is a near **drop-in replacement** for the original Go-based [szkiba/mdcode](https://github.com/szkiba/mdcode), with one deliberate difference: `extract` refuses to overwrite pre-existing whole-file targets unless `--force` is given, and exits non-zero when it skips anything. See [Existing Files Are Not Overwritten](#existing-files-are-not-overwritten).

    ### Feature Parity

    @@ -750,7 +787,7 @@ These features are **not** in the original but are available in this implementat
  
    2. **Library API** - Use mdcode programmatically in Node.js/TypeScript projects

       ```javascript

       import mdcode from 'mdcode';

       import mdcode from 'mdcode-ts';

       const result = await mdcode('README.md', transformer);

       ```

    @@ -779,9 +816,9 @@ sudo apt remove mdcode  # Linux
  
    ```bash

    # Global install

    npm install -g mdcode

    npm install -g mdcode-ts

    # or

    pnpm install -g mdcode

    pnpm install -g mdcode-ts

    ```

    ### Step 3: Verify Installation

    @@ -812,7 +849,8 @@ mdcode list -l js README.md
  
    - ✅ Metadata parsing is the same

    - ✅ Region extraction works the same

    - ✅ Stdin/stdout behavior is identical

    - ✅ Exit codes match original behavior

    - ⚠️ `extract` skips existing whole-file targets instead of overwriting them — add `--force` to keep

      the original behaviour, and expect exit code 2 when files are skipped

    ### Scripts and Automation

    @@ -841,7 +879,7 @@ Once migrated, you can optionally explore the bonus features:
  
    mdcode update --transform ./my-transformer.js README.md

    # Use as a library in your Node.js projects

    npm install mdcode

    npm install mdcode-ts

    ```

    ### Getting Help

    @@ -851,7 +889,7 @@ If you encounter any issues:
  
    1. Check the help output: `mdcode --help`

    2. Run with verbose errors (stderr will show details)

    3. Compare output with original using `--json` flag

    4. Open an issue at: https://github.com/adrianbrowning/mdcode/issues

    4. Open an issue at: https://github.com/adrianbrowning/mdcode-ts/issues

    ---

    @@ -860,8 +898,10 @@ If you encounter any issues:
  
    ### Workflow: Extract, Modify, Update

    ```bash

    # 1. Extract code blocks to files

    mdcode extract -d ./src README.md

    # 1. Extract code blocks to files.

    #    --force is needed on re-runs: whole-file targets that already exist are

    #    skipped by default. Region blocks splice in place and never need it.

    mdcode extract --force -d ./src README.md

    # 2. Edit the extracted files

    vim ./src/app.js

    @@ -918,7 +958,9 @@ mdcode extract -m type=example -d ./docs/examples README.md
  
    set -e

    echo "Extracting code blocks..."

    mdcode extract -q -d ./temp README.md

    # --force so a re-run overwrites the previous run's scratch files rather than

    # skipping them and exiting non-zero.

    mdcode extract -q --force -d ./temp README.md

    echo "Running linter..."

    mdcode run -l js "eslint {file}" README.md

    @@ -948,8 +990,8 @@ mdcode run -k -l js "node {file}" README.md
  
    ### 2. Combining with Other Tools

    ```bash

    # Format code blocks with prettier

    mdcode extract -l js -d temp README.md && \

    # Format code blocks with prettier (--force so re-runs refresh temp/)

    mdcode extract -l js --force -d temp README.md && \

      prettier --write temp/**/*.js && \

      mdcode update README.md

    @@ -1012,14 +1054,14 @@ mdcode provides a powerful CLI for working with code blocks in Markdown files:
  
    Install globally and start using it today:

    ```bash

    npm install -g mdcode

    npm install -g mdcode-ts

    mdcode list README.md

    ```

    Or try it without installing:

    ```bash

    pnpm dlx mdcode list README.md

    pnpm dlx mdcode-ts list README.md

    ```

    For more information, visit: https://github.com/adrianbrowning/mdcode

    For more information, visit: https://github.com/adrianbrowning/mdcode-ts

Navigation Menu

feat: Add region-based in-place splicing and force overwrite option for `extract` #24

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

feat: Add region-based in-place splicing and force overwrite option for extract #24

Are you sure you want to change the base?

Uh oh!

feat: Add region-based in-place splicing and force overwrite option for extract #24

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

feat: Add region-based in-place splicing and force overwrite option for `extract` #24

feat: Add region-based in-place splicing and force overwrite option for `extract` #24