No clear documentation on how to use registerTransformStream

[mcshaman]

I have been searching the internet for hours now trying to find documentation on how to use the registerTransformStream() method to modify files.

The documentation is very light on and says that you can pretty much use any gulp plugins. This is great however there are no example how to pipe them in a generator.

It is recommended to use gulp-if or gulp-filter to subset any transforms... However I cannot work out how you would effectively pipe to other transformers. Do I register all the transformers in the order that I want them piped?

[SBoudrias]

@mcshaman to answer the question, yes you just call registerTransformStream() for each transformer in the order you want to pipe.

Let's leave this ticket open though, the documentation could be greatly improved around that point. (especially as gulp isn't used too much anymore)

[simbo1905]

I was looking for a none trivial example and agreed with this ticket.

Eventually, I worked out the following to be able to use gulp-filter to only rewrite .json files and ignore other types. Hopefully, this will be of use to others looking for a realistic example:

const jsonTransform = require('gulp-json-transform');
const filter = require('gulp-filter');
const beautify = require('gulp-beautify');

const jsonFilter = filter(['**/*.json'], { restore: true });

module.exports = class extends Generator {

  _private_scmsecret() {
    return this.options.scmsecret || false
  }

  // The name `constructor` is important here
  constructor(args, opts) {
    super(args, opts);

    // "--scmsecret" command line flag toggles write feature of json template
    this.option('scmsecret');

    const self = this;

    // this pipe will pass through (restore) anything that doesn't match jsonFilter
    this.registerTransformStream(
      [
        jsonFilter,
        jsonTransform(function(data, file) {
          // based on the flags passed to the generator we may or may not edit the json structure
          if( !self._private_scmsecret() ){
           // edit the json removing the scmsecret object attributes
            ...
            return data
          }
          else return data
        }),
        beautify({indent_size: 2 }),
        jsonFilter.restore
      ]
    );

[robblovell]

I have also been trying to use this feature to do a string replace using gulp-replace without success. It appears that when the files are being written out using this.fs.copyTpl, the writer becomes disconnected from the "this" pointer and the result of the "registerTransformStream" doesn't stick when it gets to "_writeFiles(done)" code. (The debugger shows an array with a transformer in it at the end of "registerTransformStream's" execution , but on a breakpoint in "_writeFiles(done)" it shows an empty array).

Example code:

const Generator = require('yeoman-generator');
const replace = require('gulp-string-replace');

module.exports = class extends Generator {

  constructor(args, opts) {
    super(args, opts);
    this.answers = {};
  }

  prompting() {
    return this.prompt(
      [
        {
          type: 'input',
          name: 'appname',
          message: 'What is the application name?',
          store: true,
          default: ''
        },
      ]
    ).then((answers) => {
      this.answers = {...this.answers, ...answers};
    });
  }

  writing() {
    this.destinationRoot("./");

    this.registerTransformStream([
      replace("thing", "yo-test")
    ]);

    this.fs.copyTpl(
      this.templatePath('*'),
      this.destinationPath(),
      this.answers
    );
  }
};

templates/ directory contains a file thing.txt with the contents:

thing
<%= appname %>

That file gets written out with "thing" unchanged but "appname" appropriately replaced. The expected behavior is that the file will contain:

yo-test
yo-test

Instead it writes:

thing
yo-test

[robblovell]

There appears to be some weirdness with the "this" pointer here. Splicing in a hack into "_writeFiles" shows that if the transformStreams stay connected to this, the code works as expected and "thing" is replaced by "yo-test":

    // hack code:
    const replace = require('gulp-string-replace');

    this.registerTransformStream([
      replace("thing", "yo-test",{logs: {enabled: true}})
    ]);
    // end of hack test.
    const transformStreams = this._transformStreams.concat([conflictChecker]);
    this.fs.commit(transformStreams, () => {
      done();
    });

Somewhere in "run" things are getting messed up. The run code looks pretty complex so it will take a while to learn and debug. This looks like where things might possibly go pear shaped is in the "run(cb)" code where this segment is calling _writeFiles:

      const writeFiles = () => {
        this.env.runLoop.add('conflicts', this._writeFiles.bind(this), {
          once: 'write memory fs to disk'
        });
      };

      this.env.sharedFs.on('change', writeFiles);
      writeFiles();

[robblovell]

It appears that if you use "composeWith" that you loose some connections downstream. In the project where there is a problem, the example code that isn't working is chained like so:

const Generator = require('yeoman-generator');

module.exports = class extends Generator {

  constructor(args, opts) {
    // Calling the super constructor is important so our generator is correctly set up
    super(args, opts);
    this.argument('name', {type: String, required: false});
  }

  initializing() {
    if (!this.options.test) {
      this.composeWith(require.resolve('../entry-point'));
    }
  }
};

If the example code (in entry-point/index.js) is not linked via "composeWith", then it works as expected.

[SBoudrias]

@robblovell yeah, composed generators registered transformed aren't going to run. That was reported somewhere on https://github.com/yeoman/generator/issues (sorry I don't have time to dig it up right now). There was some context around this issue and we might have come to an agreement about what we should do about this...

[SBoudrias]

@robblovell yeah, composed generators registered transformed aren't going to run. That was reported here yeoman/generator#819. There was some context around this issue and we might have come to an agreement about what we should do about this...

[robblovell]

Cool. I did back off from using compseWith in favor of a branching questioning strategy with my own branching organization. I was hoping to use the compse with to manage different templates as the question tree branches. I am off and running with this and I am moving forward. I guess this issue is being moved forward on that other tree.

[exiadbq]

@robblovell I am looking for something similar with yeoman, so I setup a "demo" project(easy to run and for team contribution), then string replace all vars..Is there any update on that or it's your final solution?
Basiclly like https://github.com/dotnet/dotnet-template-samples/blob/4af2d4d475bcd9657c0c0cee1bc352bee8b7a595/02-add-parameters/MyProject.Con/.template.config/template.json#L19

[e-dong]

FYI Just opened a new issue #838, this function appears to be renamed to queueTransformStreams

[UlisesGascon]

This is now fixed as #820 was merged.

Note: the change will be in live docs once #834 is solved

[mischah]

The changes are live on https://yeoman.io/authoring/file-system.