How can I use raw loaders with Turbopack for binary files?

[luwes]

Summary

Hi, I'm one of the authors of next-video.

With webpack there is the option to receive the binary contents in the loader via
https://webpack.js.org/api/loaders#raw-loader

export const raw = true;

When I tried to create a custom loader for video files for example Turbopack seemed to try and convert the imported file contents to a string and errored and failed to get the returned ESM code from the loader.

Is there a raw option for Turbopack?

Additional information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6031
  Available memory (MB): 36864
  Available CPU cores: 14
Binaries:
  Node: 20.12.2
  npm: 10.8.1
  Yarn: 1.22.19
  pnpm: 9.12.2
Relevant Packages:
  next: 15.0.1 // There is a newer version (15.0.2) available, upgrade recommended!
  eslint-config-next: 15.0.1
  react: 19.0.0-rc-69d4b800-20241021
  react-dom: 19.0.0-rc-69d4b800-20241021
  typescript: 5.6.3
Next.js Config:
  output: N/A
 ⚠ There is a newer version (15.0.2) available, upgrade recommended!
   Please try the latest canary version (`npm install next@canary`) to confirm the issue still exists before creating a new issue.
   Read more - https://nextjs.org/docs/messages/opening-an-issue

Example

No response

[gadcam]

Hello @luwes,
raw-loader is already supported cf
https://nextjs.org/docs/app/api-reference/config/next-config-js/turbo#supported-loaders
It seems to be the case since March 2024 at least.

The way to specify a loader is explained here
https://nextjs.org/docs/app/api-reference/config/next-config-js/turbo#configuring-webpack-loaders

Did you already try this option ?

[luwes]

yes, that's a confusing name of that loader. raw-loader imports files as string, not as raw binary which I'm after. thanks

[gadcam]

Ok I get it!
Sorry about the confusion!

I also though we could try to use file-loader but it seems that we cannot.

If anyone not familiar with this topic wants to dig in this question here is a LLM generated comparison of raw option vs file-loader I made to understand better (extracted from https://chatgpt.com/share/67816c92-62a8-800f-b491-5996a8d9ab58)

---

The choice between using file-loader and the "raw" loader option in Webpack depends on your specific use case and what you intend to do with the resource files.

1. Use file-loader When:

• Goal: You need to include the file as a separate asset in the output directory, and you want Webpack to handle resolving its URL.
• Usage: file-loader is ideal for static assets like images, fonts, and media files where you need a direct reference to the file (e.g., a URL or path).
• Output: The file is copied to the output directory, and a URL is returned to use in your code.
• Example:

module: {
  rules: [
    {
      test: /\.(png|jpg|gif|svg)$/,
      use: [
        {
          loader: 'file-loader',
          options: {
            name: '[name].[hash].[ext]',
            outputPath: 'images/',
          },
        },
      ],
    },
  ],
}

2. Use the "Raw" Loader Option When:

• Goal: You need to work directly with the file's contents (as a string or buffer) in your application or another loader.
• Usage: The "raw" loader option is suitable when you want to process the file content in your loader or JavaScript code, such as embedding it in your bundle.
• Output: The file's raw content is passed as a string (or buffer if raw is set to true) to your code.
• Example:

module: {
  rules: [
    {
      test: /\.txt$/,
      use: [
        {
          loader: 'raw-loader',
        },
      ],
    },
  ],
}

With the raw-loader, the content of a file (e.g., a .txt file) is imported as a string:

import fileContent from './example.txt';
console.log(fileContent); // Logs the content of example.txt

Key Differences:

Feature	file-loader	"Raw" Loader Option
Purpose	Copies file to the output directory and resolves as a URL.	Processes and includes the file content directly in the bundle.
Output	A file in the output directory with a URL returned.	File content (as string or buffer) included in the bundle.
Use Case	Static assets like images, fonts, and media files.	Text or binary files that need to be processed or embedded.
Example File	image.png	example.txt, data.json

When to Use Each:

• Use file-loader if you need to reference a file by URL in your application (e.g., <img src="image.png">).
• Use the "raw" loader option if you need to read or manipulate the file contents in your application or pass them to another loader.

If you're writing your custom loader like raw-loader.js, you might set module.exports.raw = true if the content should be passed as a Buffer for binary processing. This enables more low-level control over file data.

[gadcam]

Hello @anthonyshew,
Do you know if it is possible to reproduce the behavior of export const raw = true; of webpack with the current setup ?
If not, do you have an idea of what could be implemented to get the same behavior ?
If it is a small/medium effort maybe I could implement it but I would need the first directions.
Thank you a lot for your help!

[mischnic]

This "RPC" call here

next.js/turbopack/crates/turbopack-node/src/transforms/webpack.rs

Lines 242 to 259 in 306de72

	let config_value = evaluate_webpack_loader(WebpackLoaderContext {
	module_asset: webpack_loaders_executor,
	cwd: project_path,
	env,
	context_ident_for_issue: this.source.ident().to_resolved().await?,
	asset_context: evaluate_context,
	chunking_context,
	resolve_options_context: Some(transform.resolve_options_context),
	args: vec![
	ResolvedVc::cell(content.into()),
	// We need to pass the query string to the loader
	ResolvedVc::cell(resource_path.to_string().into()),
	ResolvedVc::cell(this.source.ident().query().await?.to_string().into()),
	ResolvedVc::cell(json!(*loaders)),
	ResolvedVc::cell(transform.source_maps.into()),
	],
	additional_invalidation: Completion::immutable().to_resolved().await?,
	})

ends up calling this JS function

next.js/turbopack/crates/turbopack-node/js/src/transforms/webpack-loaders.ts

Line 182 in 306de72

const transform = (

The unfortunate bit is that we don't really know beforehand (on the Rust side) if it's a raw transformer or not.
So I guess the only solution would be: try to convert to string (as we currently do), otherwise send it over in buffer, just like we already do for the loader result (see BytesBase64 and where it's used). That is probably fine for now. And then the JS side would have to ensure the correct behavior (and potentially do some conversions)