Create pages from _content.gotmpl

[bep]

Note

There will be support for other data sources, e.g. _content.json, but that's outside the scope of this proposal.

I have experimented and thought long and hard about where to start re. generating Pages from data sources. The only scripting language we have built into Hugo is currently Go Templates. It works great as a templating language for HTML markup, but it's a little clunky for more advanced logic. But it's what we have, it's very familiar to many, and we have already a large API set up ready to use.

So, I suggest we start there by introducing a new reserved name identifier in the content file system _content.* where the first extension will be gotmpl, i.e. _content.gotmpl:

content
├── _index.md
└── posts
    ├── _content.gotmpl
    └── _index.md

A short example of how it may look:

{{ $logo := resources.Get "logo.jpg" }}
{{ $mydataFile := resources.Get "mydata.json" }}
{{ $eTag := $mydataFile | md5 }}

{{/* Allow Hugo to use a cached version if possible. */}}
{{ if not ($.UseCached $eTag) }}
   {{ $m :=  $mydataFile | transform.Unmarshal }}
   {{ range $k, $v := $m }}
       {{ $ppath := $k }}
       {{ $content := dict "type" "text" "value" "**Hello World**"  "markup" "markdown" }}
       {{ $.AddPage  (dict "kind" "page" "path" $ppath "title" $k "content" $content) }}
       {{ $.AddResource  (dict "path" (printf "%s/%s" $ppath $logo.Name) "resource" $logo "name" $logo.Name) }}
   {{ end }}
{{ end }

For bundled Resources I think we can say that you would either:

• Reference an existing Resource. It would be accessible in a page's .Resources, but the .Permalink would point to the original's.
• Or pass it in as content.

As this is early in the build process, the context passed to the template will be limited:

• You can access .Site.Title and similar and i18n would work as expected.
• But .Site.RegularPages will return empty.
• All paths will be relative to the _content.gotmpl file in the content tree.

/cc @jmooring

[jmooring]

Thoughts and questions...

Concept

Historically we instruct users that the content directory is for content, and the layouts directory is for templates. This introduces a blended structure, clearly identified by the _content keyword, where some of the content is generated from a template. I don't see any problem with this, but I suspect this will touch the docs in a few places. It's a new way of thinking, and that's fine.

Naming

Should these be called "content templates" or something else? We currently define an archetype as a "template for new content" but this can easily be changed. And we also have content view templates. Possibilities:

• content template
• content from data template (yuck)
• content creation template
• content generation template

Extension

The example above uses the .gotmpl file extension, presumably to clearly identify these "content templates" as templates rather than content or resources with an .html extension. There is also this proposal to allow the .gohtml extension for templates. Can or should these be implemented concurrently? Conceptually I like the idea of using something other than .html for all templates.

Page kinds

Can I use .AddPage to create section, taxonomy, term, and home pages?

Front matter limitations

Would the .AddPage method support all of the existing front matter fields? For example, headless, draft, aliases, type, layout, build, cascade, etc?

Path

It looks like the .AddPage method takes a path as its first argument, but path is also a front matter field with v0.123.0 and latter. Would it be better to do:

{{ .AddPage (dict "path" "foo" "lang" "de" "kind" "page" "title" "My Page Title") }}

[bep]

@jmooring thanks for the feedback, much appreciated. I'll put my comments below the same headings as you. A general comment is that part of this needs to be ironed out when testing this. I have the building blocks to relatively quickly get a version ready, and I think it would be good to release a version saying that "the API for this new feature may change slightly in the upcoming versions depending on feedback from users".

Concept

content directory is for for content.

I have thought a lot about creating something "new" on the side of everything, but

1. These new _content files will only produce content which will logically flow down in the file tree from where it is defined.
2. Having all content defined (and merged) in the content union file system is, in my head, both simple to understand and very powerful. And with the mounts setup you can easily avoid mixing Markdown and Gotmpl files.
3. It maps pretty nicely into how we do partial updates on file changes today.

Naming

Well, I don't know. But I can list the different types of _content files we'll end up with:

• _content.gotmpl
• _content.{json,yaml,toml}, one of
  1. List of Pages, Resources, no scripting
  2. A plugin definition. Would use ExecRPC under the hood, with Go server plugins defined as Hugo Modules (both easy to use and, in my simple head, as secure as we can get it).

Extension

I wanted something that didn't have HTML in it, and gotmpl is a term used in the Go project (at least in the source). I agree that we should do something about the other templates re. this, but I don't wan't to expand the scope too much here.

Page kinds

My example wasn't great. You would set the kind attribute and create whatever. In its first iteration (at least), the path given needs to be relative, so it would only be possible to create the home page from the root.

Front matter limitations

No limitations that I can think of. From your list, cascade would still be relevant and important.

Path

Yea, that was a mistake. I have adjusted my examples. I thought I would make it explicit that this was a set operation (that you would replace the old if you added the same path more than once), but we have the concept of languages (or in the future possible other dimensions) that needs to be handled.

[jmooring]

Naming:

• content connector? "A content connector may be a template, a data source, or a plugin."
• content interface? "A content interface may be a template, a data source, or a plugin."
• content adapter? "A content adapter may be a template, a data source, or a plugin."

[bep]

content adapter? "A content adapter may be a template, a data source, or a plugin."

That I like.

[bep]

Here's how I imagine the context/API we would pass to the _content.gotmpl execution:

type PagesFromDataTemplateContext interface {
	// UseCached returns whether Hugo can use a cached version
	// matching the given ETag.
	UseCached(eTag any) bool

	// AddPage adds a new page to the site.
	// The first return value will always be an empty string.
	AddPage(any) (string, error)

	// AddResource adds a new resource to the site.
	// The first return value will always be an empty string.
	AddResource(any) (string, error)

	// The site to which the pages will be added.
	Site() page.Site

	// The same template may be executed multiple times for multiple languages.
	// The Store can be used to store state between these invocations.
	Store() *maps.Scratch

	// By default, the template will be executed for the language
	// defined by the _content.gotmpl file (e.g. its mount definition).
	// This method can be used to activate the template for all languages.
	// The return value will always be an empty string.
	SetForAllLanguages() string
}

[jmooring]

// The site to which the pages will be added.
Site() page.Site

I assume this means that AddPage will ignore the lang front matter value (which may be different than the current site).

// The same template may be executed multiple times for multiple languages.
// The Scratch can be used to store state between these invocations.
Scratch() *maps.Scratch

I haven't spent any time looking into it, but is there ever a reason to use .Scratch instead of .Store? That may not be applicable here, but has been a lingering question in my mind.

[bep]

I assume this means that AddPage will ignore the lang front matter value (which may be different than the current site).

That or throw an error. My initial idea was to allow setting lang in .AddPage and allow mixing and matching languages. But then the .Site context doesn't make much sense ... also the i18n would not work as expected ... and people would want those ... We could possibly work around that, but I suspect that simpler is better. But I could be wrong ...

I haven't spent any time looking into it, but is there ever a reason to use .Scratch instead of .Store? That may not be applicable here, but has been a lingering question in my mind.

I rename it to Store, thanks. Yes, it doesn't really matter, but it ... sounds better.

[jmooring]

simpler is better

I agree. I'd say throw an error if lang is specified, or if specified and not equal to the current language/site.

[bep]

@jmooring can I borrow your eyes for a minute.

For attaching content to pages and resources, I have introduced a new Source type. It has a type, but in all my examples the type can be inferred from the Go type, so I think I'll drop that for now.

With some examples, I think it would look like this:

{{/* Add pages  */}}
{{ $.AddPage (dict "path" "p1" "content" (dict "value" "some *markdown*" "mediaType" "text/markdown") }}
{{ $.AddPage (dict "path" "p2" "content" (dict "value" "some <b>HTML</b>" "mediaType" "text/html") }}

{{/* Add resources  */}}
{{ $resource := resources.Get "mydata.yaml" }}
{{/* The resource can be accessed via p1's .Resources but it will link to $resource.RelPermalink  */}}
{{/* Only path and content needs to be provided.  */}}
{{ $.AddResource (dict "path" "p1/data1.yaml" "content" (dict "value" $resource ) "name" "mydata.yaml" "title" "My Data" "params" dict }}
{{ $.AddResource (dict "path" "p1/mytext.txt" "content" (dict "value" "some text" "mediaType" "text/plain") }}

Comments:

• I understand that we have a front matter field called markup, but that feels a bit foreign to use -- and it does not fit in both scenarios above. Also, if we introduce a type attribute, we could support content" (dict type "url" "value" "https://example.com/mydata" "mediaType" "application/json") or something.
• For pages, content.value is always a string in v1 of this, for resources we support both string and Resource.

Thoughts?

[jmooring]

{{/* Add pages  */}}
{{ $.AddResource (dict "path" "p1" "content" (dict "value" "some *markdown*" "mediaType" "text/markdown") }}
{{ $.AddResource (dict "path" "p2" "content" (dict "value" "some <b>HTML</b>" "mediaType" "text/html") }}

Is this a typo? Should the last two lines above begin with $.AddPage instead of $.AddResource?

[bep]

Is this a typo?

Yes, copy and paste ... Fixed.

[jmooring]

1) I am in favor of not using a "markup" field in the content map. In addition to not supporting your AddResource examples above, the markup identifier:extension relationship is imperfect at the moment.

2) While documenting/testing the current implementation, the type field was a little bit confusing (i.e., always set to "text") when adding pages. The specificity of media type is quite clear.

3) "For pages, content.value is always a string." I think that's fine. You can always do $resource.Content to populate content.value.

As always, I'm sure I'll have more comments and questions after taking this for a spin, but your proposed implementation above looks good to me... it's quite clear.

[jmooring]

One more comment...

Initially I suspect that > 90% of usage will be AddPagewith a remote data source. If the remote data source has an "image" field/array, it's trivial to use resources.GetRemote when rendering the page (i.e., in single.html).

[bep]

Initially I suspect that > 90% of usage will be AddPagewith a remote data source. If the remote data source has an "image" field/array, it's trivial to use resources.GetRemote when rendering the page (i.e., in single.html).

Yea; but I suspect AddResource will be useful to be able to reuse all hooks/templates that looks for featured images inside .Resources etc.

Note that mediaType currently has no effect (will get to it soonish), other than that it should work reasonably OK.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.