middleware + pages routes + getInitialProps setting 404 = results in hard route

[adanperez]

Link to the code that reproduces this issue

https://github.com/adanperez/nextjs-issue-pages-middlware-404

To Reproduce

Setup

This linked repo demonstrates an issue we are seeing with the following setup:

• Next.js Middleware.
• Pages Router
• Dynamic page using getInitialProps
• getInitialProps

Issue

• Client navigation from / -> /soft will do a soft route to the new page as expected.
• Client navigation from / -> /dynamic/test will do a hard route to the new page which is NOT expected.

Current vs. Expected behavior

Current behavior

The following URL of /_next/data/development/dynamic/test.json?slug=test is routed to the page src/pages/dynamic/[...slug].tsx because of Next.js middleware.
This unexpected URL does not match the expected path structure we look for in the getInitialProps function of our src/pages/dynamic/[...slug].tsx page.

We do not find the correct ID of test because of the extra path structure /_next/data/development which gives us the result of data instead.

We return a 404 here since the result of our pretend DB lookup is not found when our ID is not test.

The returning 404 from the /_next/data/development/dynamic/test.json?slug=test request causes middleware/Next.js to hard route to this page instead of soft routing.

Expected behavior

We expect a soft route instead of a hard route when returning a 404.

Thing to note
This 404 / hard route behavior does not happen with Next.js version 14.1.4

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.5.0: Wed May  1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000
  Available memory (MB): 32768
  Available CPU cores: 10
Binaries:
  Node: 20.10.0
  npm: 10.2.3
  Yarn: N/A
  pnpm: 9.4.0
Relevant Packages:
  next: 14.2.5 // Latest available version is detected (14.2.5).
  eslint-config-next: 14.2.5
  react: 18.3.1
  react-dom: 18.3.1
  typescript: 5.5.4
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Middleware, Pages Router

Which stage(s) are affected? (Select all that apply)

next dev (local), Other (Deployed)

Additional context

This 404 / hard route behavior does not happen with Next.js version 14.1.4.

[rameardo]

You are experiencing hard routing issues when navigating from / to /dynamic/test with getInitialProps. This issue is caused by the unexpected path structure of the prefetch request (/_next/data/development/dynamic/test.json?slug=test). The problem does not occur in Next.js version 14.1.4.

Solution Breakdown

1. Update Path Parsing in getInitialProps: Ensure the path parsing logic handles the _next/data structure correctly.
2. Adjust Middleware Handling: Modify the middleware to avoid interfering with the routing logic.
3. Use getServerSideProps: Consider using getServerSideProps for more predictable server-side rendering behavior.

Update Path Parsing in getInitialProps

Modify getInitialProps to correctly parse paths regardless of the request source:

Dynamic.getInitialProps = async (ctx: NextPageContext) => {
  let idToLookUp = '';

  // Check if the request is coming from _next/data
  if (ctx.asPath.startsWith('/_next/data')) {
    const parts = ctx.query.slug;
    idToLookUp = Array.isArray(parts) ? parts[0] : '';
  } else {
    idToLookUp = ctx.asPath.split('/')[2];
  }

  console.log('▼▼▼▼▼ Dynamic.getInitialProps ▼▼▼▼▼');
  console.table({ path: ctx.asPath, idToLookUp });

  if (idToLookUp !== 'test') {
    if (ctx.res) {
      ctx.res.statusCode = 404;
    }
  }

  console.log('▲▲▲▲▲ Dynamic.getInitialProps ▲▲▲▲▲');
  return { test: true };
};

Adjust Middleware Handling

Ensure middleware does not affect routing unnecessarily:

import { NextResponse } from 'next/server';

export const middleware = (req) => {
  const url = req.nextUrl.clone();

  // Avoid middleware modification for _next/data paths
  if (url.pathname.startsWith('/_next/data')) {
    return NextResponse.next();
  }

  const requestHeaders = new Headers();
  requestHeaders.set('x-hello-from-middleware1', 'hello');
  return NextResponse.next({
    request: {
      headers: requestHeaders,
    },
  });
};

Using getServerSideProps

If the above changes do not resolve the issue, consider using getServerSideProps:

export async function getServerSideProps(context) {
  const { slug } = context.params;
  const idToLookUp = slug[0];

  console.log('▼▼▼▼▼ Dynamic getServerSideProps ▼▼▼▼▼');
  console.table({ slug, idToLookUp });

  if (idToLookUp !== 'test') {
    return {
      notFound: true,
    };
  }

  console.log('▲▲▲▲▲ Dynamic getServerSideProps ▲▲▲▲▲');
  return {
    props: { test: true },
  };
}

export default function Dynamic({ test }) {
  const router = useRouter();
  return (
    <div>
      <h1>Dynamic / {router.query.slug}</h1>
      <li>
        <Link href="/">Home</Link>
      </li>
    </div>
  );
}

Ensuring Soft Routing on 404

If you still face issues with hard routing on 404, ensure your custom 404 page does not force a hard navigation. Next.js should handle soft routing by default, but custom behaviors in _app.tsx or _document.tsx could affect this.

Here's a basic custom 404 page for reference src/pages/404.tsx:

import Link from 'next/link';

export default function Custom404() {
  return (
    <div>
      <h1>404 - Page Not Found</h1>
      <Link href="/">Go back home</Link>
    </div>
  );
}

Summary

By addressing the path parsing logic, ensuring middleware does not interfere with _next/data paths, and considering getServerSideProps, you can mitigate the hard route issues you're experiencing. Ensure all custom routing logic respects Next.js's soft routing mechanisms, especially around error handling and 404 pages.