Suggested Refinements on the Purpose and Scope of Webpacker

[justin808]

Current Description:

Use Webpack to manage app-like JavaScript modules in Rails

I'd suggest the following description:

Provide Rails support for the webpack module bundler for client-side assets.

I had a nice discussion with @gauravtiwari. Based on that discussion, and his example of the asset_pack_tag, I'd like to suggest the following guidelines for this gem:

1. Provide no interference with the legacy Asset Pipeline. /app/assets contains files used by the Asset Pipeline.
2. Provide no requirement that sprockets-rails be included in the Gemfile.
3. Require that the webpack configs always use the ManifestPlugin per the documentation regarding webpack caching and filename hashes. This is the magic glue that will allow the helpers to know the file paths for the view and controller helpers.
4. Provide controller helpers for getting the path to the created webpack JavaScript or CSS bundles.
5. Provide view helpers for exposing webpack created JavaScript and stylesheet tag helpers that are the webpack equivalents of the javascript_include_tag and the stylesheet_link_tag. These view helpers are smart enough to understand the 3 modes of running webpack to create assets:
6. Static files created for development.
7. Running the webpack-dev-server for providing hot-reloadable files for development.
8. Static, hashed files created for production deployment, using the chunkhash technique for asset caching.
9. Automatically insert a precompile task to build the production assets for webpack (unless overridden in the config/webpack.rb file.
10. Via a new file, /config/webpack.rb, allow configuration of the controller and view helpers in a convenient manner that does not mandate any particular location of the webpack configuration files nor the subdirectory under the /public directory for deployment.

A gem like the above can be usable for any Rails 4.x and greater project. So the current Rails codebase would love this.

Optional for Rails 5.1

• Provide a generator to create the simplest possible example of using Webpack for assets with clear documentation to consult the current Webpack v2 docs for more information. Any parts of this work will depend on Rails 5.1 should be put into a separate gem so the core gem can be usable by Rails 4 and above.

Out of Scope

• Integrations with other JavaScript frameworks, such as React, Angular, etc. These can be done in other gems.
• Anything beyond a basic webpack config, as the maintenance of this config would be far greater than the basics needed to support the scope of the gem.

Documentation

• The Rails Guide for the Asset Pipeline should be updated to reflect usage of either Sprockets or Webpack. For backward compatibility reasons, it might make sense to clearly leave /app/assets/XXXX as Sprockets supported, and /app/[good-name-for-webpack-stuff] as for the webpack parts. Some parts of this Rails Guide apply to topics outside of Sprockets, namely Section 4: In Production.

Conclusion

A minimal, yet complete, standard mechanism to integrate the webpack JS and CSS output into the Rails views will foster the development of Rails gems to provide integrations of JS frameworks with Rails. An example of such a webpack based framework is React on Rails.

[rupurt]

@justin808 this is a great summary and I'd love to help out. The killer part as you mentioned, is that it can be used for legacy Rails projects!

I've just got a few questions:

2. The manifest strategy & standard Rails helpers can be used by any of the build systems (rollup, browserify, gulp etc...). How would you feel about splitting this into 2 projects? e.g.

• frontend_on_rails (or some cooler name...): This manages the location of the manifest file, calling the precompile task
• webpacker: This manages webpack config, dev server, manifest plugin setup

3 & 4) By using the manifest files you shouldn't need to create new helpers. You should be able to use all of the existing ones.

[dleavitt]

Good stuff! Some feedback:

4. Provide view helpers for exposing webpack created JavaScript and stylesheet tag helpers that are the webpack equivalents of

We shouldn't need new helpers once we're using a manifest everywhere. Webpacker can patch the built-in Rails helpers to be webpacker-aware.

This is what compute_asset_path is for. Webpacker should patch this and then fallback to sprockets if the asset isn't found (or vice-versa.) This is exactly how sprockets-rails does it right now - looks in the sprockets manifest and if not found falls back to the the default Rails behavior (look in the public folder.)

9. Via a new file, /config/webpack.rb

This shouldn't be needed. Sensible defaults should be set, and they can be overridden in application.rb or one of the environment config files.

[justin808]

@dleavitt could a /config/webpack.rb by the webpack equivalent of assets.rb? I'd like to see the default configs listed and commented out, so it's clear what can be overridden.

Regarding the built-in rails helpers...what if somebody wants to turn off sprockets? Maybe if put the helpers in a separate very lean, webpack 2.2+ focused gem, then Sprockets can refer to that gem to patch the existing helpers? The unusual thing about Webpack integration is that we want to enable a hot reloading server for the assets sometimes, but not always (per ENV value), for development mode.

Maybe we can patch the javascript_include_tag and stylesheet_link_tag so that they would first pick a webpack source? Another issue is whether the options that make sense for these helper methods would also apply to webpack files.

If the options for webpack bundle files should be different, then, IMHO, it would be more confusing to overload the old helper names, and new helper names should be used.

I created a label for open issues on React on Rails pertaining to Webpacker and Sprockets. @rupurt has done some work here in issue #696. I recently merged PR 671: Ability to work without Sprockets, which brings up another issue in that there is no assets:precompile if Sprockets is not used. I'm going to update the top description so that it reflects the ability to work without sprockets as a requirement.

[rupurt]

@justin808 I'm interested to hear more about the use case of completely removing sprockets. We still use sprockets to inline styles for our mailers and then we also use it for the manifest file mapping when we use the helpers.

At this point I don't think we should try to remove sprockets but rather leverage what it already gives us. I'd love to know more about how an app works with it removed though i.e. how do you reference assets?

Have you had a chance to look at configuring the asset host in development for the dev server? You can guard it with an environment conditional and you don't need to change anything to do with the helpers. https://github.com/rupurt/webpack-sprockets-rails-manifest-plugin#hot-module-replacement

[dleavitt]

I don't think folks are talking about removing sprockets, just making sure that webpacker doesn't depend on its presence.

[dleavitt]

word, initializer makes more sense for webpack.rb. +1 I think all the functionality you're describing can be added to `compute_asset_path` in a way that's agnostic to the presence of sprockets. Basically if the webpack asset is missing it falls back to whatever the next strategy is (either sprockets, or straight to generating a public path.) This seems like a good idea because you can keep using the same helpers and they mostly just work if you switch from sprockets to webpack. No duplicate code, no duplicate documentation, etc. I'd be curious to get weigh in from someone on the rails core team though.

…

On Feb 28, 2017, at 11:54 AM, Justin Gordon ***@***.***> wrote: @dleavitt <https://github.com/dleavitt> could a /config/webpack.rb by the webpack equivalent of assets.rb? I'd like to see the default configs listed and commented out, so it's clear what can be overridden. Regarding the built-in rails helpers...what if somebody wants to turn off sprockets? Maybe if put the helpers in a separate very lean, webpack 2.2+ focused gem, then Sprockets can refer to that gem to patch the existing helpers? The unusual thing about Webpack integration is that we want to enable a hot reloading server for the assets sometimes, but not always (per ENV value), for development mode. Maybe we can patch the javascript_include_tag <https://github.com/rails/rails/blob/be5ddc250874e6c7f4d22e34791f5187448ebb0e/actionview/lib/action_view/helpers/asset_tag_helper.rb#L75> and stylesheet_link_tag <https://github.com/rails/rails/blob/be5ddc250874e6c7f4d22e34791f5187448ebb0e/actionview/lib/action_view/helpers/asset_tag_helper.rb#L111> so that they would first pick a webpack source? Another issue is whether the options that make sense for these helper methods would also apply to webpack files. If the options for webpack bundle files should be different, then, IMHO, it would be more confusing to overload the old helper names, and new helper names should be used. I created a label for open issues on React on Rails pertaining to Webpacker and Sprockets <https://github.com/shakacode/react_on_rails/issues?q=is%3Aopen+is%3Aissue+label%3Awebpacker>. @rupurt <https://github.com/rupurt> has done some work here in issue #696 <shakacode/react_on_rails#696>. I recently merged PR 671: Ability to work without Sprockets <shakacode/react_on_rails#671>, which brings up another issue in that there is no assets:precompile if Sprockets is not used. I'm going to update the top description so that it reflects the ability to work without sprockets as a requirement. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#139 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEYBvrtPKgDEokcqrLFFXKJyzvRsl8Aks5rhHtkgaJpZM4MOGqZ>.

[justin808]

@rupurt

Have you had a chance to look at configuring the asset host in development for the dev server? You can guard it with an environment conditional and you don't need to change anything to do with the helpers. https://github.com/rupurt/webpack-sprockets-rails-manifest-plugin#hot-module-replacement

Very cool! It just won't work if you need to be selective about which assets in case you have some old sprockets and some new webpack ones.

[rupurt]

@justin808 I deal with that by using a suffix for webpack generated files i.e. my-js-bundle. I then add a regex check for -bundle in the asset host configuration.

# application.rb
# ...

if ENV["ASSET_HOST"].present?
   config.action_controller.asset_host = proc { |source, _request|
      if Rails.env.development?
         if source =~ /-bundle/
            ENV["ASSET_HOST"]
         # else
            # return nil and use the server host
         end
      else
         ENV["ASSET_HOST"]
      end
   end
end

# ...

[dhh]

Just to set realistic expectations here: Rails 5.1 is essentially a wrap. We're not going to add any additional provisions for non-JS webpack helpers in that release. The next stop will be either Rails 5.2 or Rails 6.0. So we'll have a good amount of time (6-9 months, at least) to sort out how it should all feel.

I'm not big on overwriting the existing helpers in any case, though. We should provide a smooth transition path from asset pipeline to webpack and allow people to combine them both in a single app. That's the magic behind Webpacker and the javascript pack tag. They can just coexist. So let's continue down along that path.

We've discussed the HELLOs in other tickets, btw. For now they stay to provide the bare minimum bootstrapping. Then other gems and efforts can go above and beyond that.

[rupurt]

@dhh good to know.

I wrapped up my ideas on the integration into a separate gem webpack_on_rails. It contains an install generator which creates the Webpack config files in a similar way to webpacker as well as configuring the location of the manifest file and disabling asset debug in development mode. This allows the standard view helpers javascript_include_tag, stylesheet_link_tag, etc... to be used instead of the new javascript_pack_tag.

e.g.

<% # app/views/layouts/application.html.erb %>
<!DOCTYPE html>
<html lang="en">
  <head>
    <%= stylesheet_link_tag "webpack-application", media: :all %>
  </head>
  <body>
    <%= yield %>

    <%= javascript_include_tag "webpack-application" %>
  </body>
</html>

As I mentioned in the previous comment, the core of this idea can be used to integrate any build system (grunt, rollup, browserify...). The manifest location & disabling asset debug in development are the common pieces, and it would be up to the build system to write a compatible manifest file.

If we improved sprockets slightly, and reloaded the manifest file when it changes in development, we could also enable asset fingerprints for the development workflow.

[dhh]

@rupurt I think that's totally fine to keep as a separate project. I don't think Rails out-of-the-box needs to support anything beyond the asset pipeline and webpack, but it's good for us to make it easy to for others to provide integration with other build systems 👍

[justin808]

Just to set realistic expectations here: Rails 5.1 is essentially a wrap. We're not going to add any additional provisions for non-JS webpack helpers in that release. The next stop will be either Rails 5.2 or Rails 6.0. So we'll have a good amount of time (6-9 months, at least) to sort out how it should all feel.

In the early days of React on Rails, we had our generators try to set up CSS modules, linting, hot reloading, etc. This added a TON of extra complexity. There's no reason that the current asset pipeline can't continue to handle non-JS assets as it has always done. That being said, if you turn hot reloading on for JS, the CSS won't hot reload. It might be worth considering if hot reloading (vs. hitting CMD-R) is really worth the maintenance.

I'm not big on overwriting the existing helpers in any case, though. We should provide a smooth transition path from asset pipeline to webpack and allow people to combine them both in a single app. That's the magic behind Webpacker and the javascript pack tag. They can just coexist. So let's continue down along that path.

💯 agree on keeping the helpers separate!

[gauravtiwari]

@justin808 Should we close this one now? #153 is merged