Allow for multiple preprocessors to be chained

[dignifiedquire]

As discussed before we need the ability to chain preprocessors. I suggest the following api.

preprocessors = {
  '**/coverage/*.coffee': ['coffee', 'coverage'],
  '**/other/*.coffee': 'coffee'
};

This would allow you to specify an array of preprocessors in exactly the order you want them to be executed or if you just define one, only this one will be executed.

Gonna implement this later this day if I find the time.

[vojtajina]

Reopening the issue, I had to revert this commit as it breaks the preprocessing functionality.

Otherwise I totally agree, we want to allow multiple preprocessors, I will get back to this.

Preprocessors can change the file name as well (eg. coffee, html2js do that). It might be desired to apply a preprocessor based on this renaming as well. Need to think about it though...

[petebacondarwin]

With preprocessors are you not starting to step into the arena of build tools?

Would it not be better to somehow create a better integration with one or more build tools so that one didn't need to set up multiple tools to do build and testing?
Perhaps there is not an appropriate build tool out there and there is a project in itself to create one??
Pete

[dignifiedquire]

Great idea. What about the possibillty to inject your gruntfile and have it executed before files are served.

[vojtajina]

Interesting idea, +1 for researching integration with grunt.

I used to use grunt watch for compiling coffee script, before adding preprocessors into Testacular. I had two issues:

• need to start another thing (testacular + grunt watch)
• had to hack grunt to actually compile only the files that changed (I believe this got fixed already).

[dignifiedquire]

• We could just spawn a grunt process with the specified gruntfile/options or setup something more sophisticated. I will have a closer look into this.
• Not sure about that, I think it's possible with grunt 0.4 but the tasks have to get updated to take advantage of that.

[dignifiedquire]

I've started a new discussion about only compiling changed files in gruntjs/grunt-contrib-watch#14. It seems that it hasn't been solved yet.

[dignifiedquire]

I had another idea, if we rewrite the webserver using connect, we could add the ability to include arbitrary connect middleware to handle files before they get served. That might be much more efficient as these middlewares are already built for this purpose.
@vojtajina What do you think?

[vojtajina]

Rewriting the web server with connect is for sure.

Using a middleware for this is an interesting idea, but I'm afraid it still needs to be coupled to the watcher/fileList, because it needs to be preprocessed when the file changes. Web server does not have this information.

[dignifiedquire]

Actually it does. I've looked at some of connects asset compile middleware
and they automatically reconpile if the source file is newer than the
conpiled version. Performace wise this should be great as this is only done
if a request comes in.
I'll test it out as soon as I find some time.

On Monday, December 10, 2012, Vojta Jina wrote:

Rewriting the web server with connect is for sure.

Using a middleware for this is an interesting idea, but I'm afraid it
still needs to be coupled to the watcher/fileList, because it needs to be
preprocessed when the file changes. Web server does not have this
information.

—
Reply to this email directly or view it on GitHubhttps://github.com/vojtajina/testacular/issues/161#issuecomment-11185385.

[vojtajina]

Just forwarding @dignifiedquire 's comment from #334

---

I've been thinking about how to handle multiple preprocessors.
The best idea I came up with so far is to make them all work on streams, so we can just pipe the stream we get from reading the file directly into each preprocessor after another without even creating tmp files. (They can still have side effects like creating a new file if needed)

Short example

var fs = require('fs');
var es = require('event-stream');

// this is an array of all preprocessors that we want to execute
// this need to be "pipeable"
var preprocessors = createPreprocessors(filename, config.preprocessors);

var file = fs.createReadStream(filename, {flags: 'r'});

// pipe all the things
var result = es.pipeline.apply(this, [fileStream].concat(preprocessors));

Update Use event-stream.

[vojtajina]

I propose a change to the config file - remove the preprocessors and allow them in files:

files = [
  'lib/jquery.js',
  {pattern: 'src/*.coffee', preprocess: ['coffee', ...]},
  'test/*.js'
];

No reason to specify the matching patterns multiple times. Explicit order of preprocessors.

[dignifiedquire]

I'm afraid I don't think this will improve anything. For example if you have written everything in CoffeeScript, right now you can specify to preprocess all .coffee files in one line. With this new syntax you would need to add that to every single import.

// Old
files = [
  'src/*.coffee',
  'test/unit/*.coffee'
];
preprocessors = {
  '**/*.coffee': 'coffee'
}
// New
files = [
  {pattern: 'src/*.coffee', preprocess: ['coffee']},
  {pattern: 'test/unit/*.coffee', preproces: ['coffee']}
];

If you have a large list of files this can get quickly out of hand in my opinion.

[vojtajina]

That's a good point.

[vojtajina]

Mutiple preprocessors is already done, so I'm closing this issue.

There were some other interesting points in this discussion that I will keep in mind (esp. changing preprocessors to work with streams is neat and will happen; hopefully ;-))

[ghost]

How are multiple preprocessors implemented now?
I couldn't find anything in the docs (http://karma-runner.github.io/0.8/config/files.html).

[vojtajina]

You can chain multiple preprocessors like this:

preprocessors = {
  '**/*.coffee': ['one', 'two', ...]
};

Please feel free to send a PR with imrpoving the docs...

[ghost]

When I chan multiple preprocessors like so:

preprocessors =
  'src/**/*.coffee': ['coffee', 'coverage']

I get:

Fatal error: Object coffee,coverage has no method 'charAt'

[vojtajina]

@bekite sorry, I forgot this is only in 0.9 (which is not stable release yet).