Which Generator builds Markdown the fastest?

Blogging. The quintessential starter project for most—if not all—site generators. Here’s a few samples:

• Next.js guides new developers to Learn Next.js “by creating a very simple blog app.”
• Gatsby’s tutorial guides folks to create their “first Gatsby site: a blog site…”
• Remix’s home page has a prominent Get Started call to action button that links to their Blog Tutorial.

Given that Markdown is a very popular document format for blogging, this gives us the opportunity to compare the performance of different site generators for this pervasive use case (highlighted on each generator’s documentation, as noted above).

Let’s see how each generator stacks up when consuming markdown files from the local project’s file system and building into a production-ready project.

(Order is alphabetical. Disclosure: I am the maintainer of Eleventy)

Benchmark Results #

Times shown are in seconds. Lower is better.

Each run was repeated 3 times and the lowest/fastest time was selected. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5, 16 GB memory.

All of the code for this benchmark is fully open source and welcomes review.

Test Notes #

For each generator sample I attempted to create a reduced project with the sole use case of processing markdown files. I opt-ed out of TypeScript when options were presented in various cli tools to do so. Output folders and framework specific cache folders were deleted before each run.

• For Astro I used the Blog example which uses a pre-release of Astro 1.0.
• For Eleventy I used eleventy-base-blog, even though it has a few extra plugins and templates in play on top of the barebones core experience.
• For Gatsby, I used npm init gatsby with Markdown support (not MDX).
• For Hugo, I went through the Quickstart (skipping the theme)
• For Next.js I deleted a bunch of things out of the blog-starter example linked from the docs on Static Generation.
• For Remix I went through the Blog tutorial but did not use a database. (Read the Remix addendum below).

I put out a Twitter poll to gauge how folks felt about project sizes. 1000 files was considered a Large project by 58.8% of voters, Medium for 36.8% of voters. Markdown samples borrowed from Sean C Davis’s SSG Build Performance Comparison repository. I have not yet added tests for Jekyll or Nuxt but I’m open to it!

Summary #

Updated August 3, 2022 with notes about file-based routing of markdown in Remix and Next.js, as well as Astro MDX.

1. Hugo remains the undisputed speed champ—no question about that.
2. Eleventy was the fastest JavaScript-based generator.
3. Generators that create per-page JavaScript bundles (for single page apps, primarily) are usually slower to build, unsurprisingly. Heavier pages aren’t exclusively slower for end users—they’re slower for developers too.
4. File-based routing of markdown files in both Next.js and Remix is a bit slow! Take care to add the additional boilerplate routing code needed to load markdown files more efficiently.
   • Read more on (nextjs.org) Advanced Features: Using MDX with Next.js and (remix.run) MDX and on the Remix addendum below.
5. Astro was on-par with Next.js at mid-range (1k) and on-par with Gatsby at the upper-range (4k) of this benchmark.
   • Jon Neal submitted a PR to do Astro markdown processing via the MDX plugin. I re-ran the tests but the data for Astro didn’t change in a statistically significant way (yet?). Likely more to come there, this feature is brand new!
6. I would welcome a code review on the Remix site—it scaled so poorly that I suspect that I may have misconfigured something? I would be happy to update with corrections.
   • As noted in point 4 above, it is recommended for Remix projects by the Remix team to not use file-system based routing for .mdx or .md files. (Read the Remix addendum below).

Bonus: npm install Benchmarks #

Times shown are in seconds. Lower is better.

Each run was repeated 5 times and the lowest/fastest time was selected. npm cache clean --force was run before each to ensure a cold install. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5, 16 GB memory.

Remix Addendum #

Updated August 3, 2022 (this entire section is new)

I understand why Remix folks think this benchmark is unfair to Remix. I’ve heard lots of feedback and some of it has been decidedly unfriendly—but I do get it. I don’t really have any desire to wade into some of the larger debates about Remix or Jamstack on this post.

From my perspective, the benchmark encountered a build performance-related bug in how Remix—a request-time architected tool—performs a build-time precompilation of Markdown routes. It doesn’t have to be any more than a build-performance related bug. Once the bug is fixed (or if the approach is deprecated), I’ll make updates to this post.

I maintain that markdown file-based routing is a useful feature and it seems likely that the Remix folks will be able to (at some point in the future) allow a peaceful coexistence of markdown routes without a heavy build-time compilation step. The demand clearly exists as many other frameworks do support it!

But until then the Remix team recommends to opt out of file-based routing for Markdown to bypass the costly build-time compilation step. You can use a scripted route file to load Markdown files from the file system in a different source directory (not app/routes/). You can see an implementation example from @ebey_jacob in the bench-framework-markdown repository.

One last note about Build Performance #

“While you were waiting for your static site to build, distributed web infra­structure got really good. Break through the static.”—https://remix.run/

The only other thing I’d like to say here is that whether a tool spits out bundled JavaScript for runtime server deployment on the Edge or HTML in a static build for the CDN, whether it’s categorized as a Static Site Generator or a Site Generator or none of the above: all of the tools tested in this benchmark are performing a build step to generate code for deployment and can encounter build-related performance issues.

All that to say: I think we can agree that build performance is not a concern exclusive to static site generators! There will always be a healthy tension between request-time and build-time and the tools that make use of both to complement each other will likely end up being the most useful to developers.

Keep measuring and keep building—appreciate y’all!