Epic: Move away from `pages` folder

[ovflowd]

Back in 2023/2024 when we were releasing the redesigned Node.js Website, we had to create a custom dynamic router around our infrastructure (Next.js) because pages folder structure is what @nodejs/collaborators were used to work with. This was intentionally done to not make the move to Next.js feel as big as an atomic move as it could be. This was also done to keep DX for existing collaborators simple.

Fast forwarding to now, release blog post generation is getting automated and releasers and other collaborators that have worked with us are used and more familiar with our Next.js environment.

The time has probably come. It's time to move away from our custom next.dynamic.mjs router (including all the bells-and-whistles) and adopt Next.js's built-in MDX, which uses mdx-rs, a Rust-based MDX parser/compiler (https://github.com/web-infra-dev/mdx-rs) and that at the moment also supports custom JS-based Rehype and Remark Plugins.

This work effectively means that:

• Our custom Frontmatter Engine for Layouts will be gone. Frontmatter would still exist, but Layouts are decided by the App router.
  • i.e. /app/blog/layout.tsx, etc.
  • each section (learn, download, etc) use their own layouts.
• Each page (.mdx) file under /pages would be moved under the respective folder under /app/ router
• The use of catch-all segments would effectively be diminished (except for some rare situations such as blog category pages and pagination and the index page (that is also a blog category page)
• Adoption of Next.js's built-in MDX parser/compiler that is much faster and also enables support of some real cool stuff
• Our custom glob/mdx parsing and caching engine will be gone

Some of the benefits:

• Much faster compilation and dev runtime
• 1:1 (or almost 1:1, because there are still some magic routes such as blog categories) file-system to route
• More out of the box Next.js experience and easier to maintain and understand codebase
• Fast-refresh / HMR (Hot-Module-Reload) support for MDX
• Simplification of our OpenNext.js infrastructure. No more need of the hacky node:fs polyfills and whatnot

The cons?

• Poor @avivkeller is going to work overtime on this LOL (/s)

cc @nodejs/web-infra @nodejs/nodejs-website

[AugustinMauroy]

feel -0.5 to put MDX on app dir because I don't like to mix routing and content.

feel +1 to have a content dir that store md/mdx and these are processed by the Next.js app router. And for layout instead of injecting them by frontMatter we can just define it on the app router.

[ovflowd]

feel -0.5 to put MDX on app dir because I don't like to mix routing and content.

feel +1 to have a content dir that store md/mdx and these are processed by the Next.js app router. And for layout instead of injecting them by frontMatter we can just define it on the app router.

I get the feeling, but right now we have some pretty custom infrastructure that deviates from standard Next.js and App router was made for content too. This differentiation has been the root cause for many problems with our Next.js upgrades + the main piece of effort and issues with our Cloudflare and OpenNext integration. So yeah, I definitely would like to move away from there :)

[avivkeller]

My main concern here is that every blog post would need to be its own folder, correct? At least, that's my understanding of https://nextjs.org/docs/app/guides/mdx

[avivkeller]

As I mentioned above, and as @ovflowd agreed on Slack, the "one page per folder" structure is a major blocker for this change. It's infeasible for us to make a new folder for every new blog post and learn article we have.

[bmuenzenmeyer]

is the reasoning beyond busy work? we can script/automate that away

[avivkeller]

is the reasoning beyond busy work? we can script/automate that away

The reasoning is DX. The pages folder, IMO, is very neat. Making a new folder for every page is a lot less... 'neat'.

[avivkeller]

Regarding a script, however, I wrote:

import fs from 'node:fs/promises';
import path from 'node:path';

const base = new URL(import.meta.resolve('./apps/site/'));
const pagesBase = new URL('./pages/', base);

const pages = await Array.fromAsync(
 fs.glob('**/*.{md,mdx}', { cwd: pagesBase })
);

for (const page of pages) {
 const src = new URL(page, pagesBase);

 const ext = path.extname(page);
 const parsed = path.parse(page);

 // If the file is named "index", don't create a subdirectory
 const destPath =
 parsed.name === 'index'
 ? path.join('app', parsed.dir, `page${ext}`)
 : path.join('app', parsed.dir, parsed.name, `page${ext}`);

 await fs.cp(src, new URL(destPath, base));
}