LearnOSM need to be tidy

[pyrog]

Hi,

One of the big problem is that this site is a… mess 😉

It use essentially _posts but this is not a blog, it's a book. We don't need to put chapters in _posts.
So the actually structure of learnOSM is too complicated, difficult to understand, translate and maintain.

We should tidy it and separate better data and code:

• data : chapters in .md format without liquid markup
• code: use "well" layouts and/or gems

Examples:

• All chapters code could be only in the layout
• same thing for Moving Forward
• Front matter are too complicated, we don't need:
  • permalink: if we will not use _posts
  • lang:, layout could determine it easily from the page.path 😉
  • category: same as permalink.
• Images:
  • their names are too complicated
  • some of them should translated, internationalized
    (so we should have the source image and separately the anotations written on them)
• layouts are too complicated:
  • a layout could call another one. Hi levels layout represent structures (menu, header, footer, traffic analysis Web Traffic Analytics? #294, …), low levels contains HTML
  • a layout could also include other files

[pyrog]

You could have a look to the result : http://pyrog.github.io/learnosm-sandbox/en/
The Jekyll source code is here : https://github.com/pyrog/learnosm-sandbox

Note: the german page about is missing to check what happened when a page is untranslated (see #252)

[wonderchook]

Hi @pyrog if I understand you are saying "tidy" to mean two different things?

1. File structure behind the site
2. Design of the site

I don't see there being a lot of different simple moving the files out of the "_posts" they are already in there in individual language folders. There could be better organization within there.

As for the resulting site you created I'm a big confused, it looks like a default Jekyll site. The LearnOSM site as it was done now was created with a design process and I think is a modern site and a good look.

[pyrog]

• Mostly tidy file structure behind the site 😄
• For the "design" I mean "coding design", not "graphical design".
  For example, if we want to add a feature like "Next chapter" (see link or button to the next page or next section #270), the changes should be done only once by devs, to avoid editing all page.md (and probably forget some).

As for the resulting site you created […] it looks like a default Jekyll site.

Of course, I use the default template 😉
I would prove that _posts are not mandatory, because the official Jekyll documentation is not very clear:

page.url
The URL of the Post without the domain, but with a leading slash, e.g.  /2008/12/14/my-post.html

[althio]

Hi @wonderchook
@pyrog wants to help only with the structure, not the graphical design.
The staging site he presented is only a proof-of-concept to show that jekyll can handle something else than blog structure. Little step I know, but step by step... :)

It is also an available sandbox where we can test anything around structure.
But -- you are right -- it is not intended as a proposal for graphical design and look.

I don't see there being a lot of different simple moving the files out of the "_posts" they are already in there in individual language folders. There could be better organization within there.

Generally (in issues+sandbox+discussion+lists...) we would like to assess the pros and cons of blog structure and regular structure.
The regular structure might be more adapted to handle organised and hierarchical content like ours instead of chronological content like a blog.

As it stands with the _posts folder, category organisation, dated and ordered filenames... It looks slightly odd and we are trying to understand the current implementation as blog. My feeling (for the time being!) is that it is not completely suited and requires a bit of shoehorn. I would like to know more about the history and previous choices. I would love to get critical feedback on any structure.

[pyrog]

The regular structure might be more adapted to handle organised and hierarchical content like ours

With the suggested design, it will simpler and obvious 😄 :

• source file structure = static file structure
• source filenames = static file names
• no need to add/edit permalink: in each files front matter

So currently we have for the file /fr/beginner/moving-forward.html

His jekyll source filename:

/_posts/fr/beginner/0200-12-21-moving-forward.md

His frontmatter:

---
layout: doc
title: Pour aller plus loin
permalink: /fr/beginner/moving-forward/
lang: fr
category: beginner
otherguides: "Les autres niveaux"
---

After the tidying of Mr Clean 😉:

/fr/beginner/moving-forward.md

Is front matter will be:

---
layout: doc
title: Pour aller plus loin
category: beginner
otherguides: "Les autres niveaux"
---

[pyrog]

Tidying images:

In the images folder, many images are redundant:

• a flat file structure LC_GUIDE_CHAPTER_imageXX.png
  ie. en_beg_ch8_image00.png
• another one CHAPTER_imageXX_LC.png
  i.e.. start_josm_image01_en.png
• a file structure LC/GUIDE/XX_CHAPTER/LC_GUIDE_XX_CHAPTER_imageXX_TITLE.png
  ie. en/beginner/01_introduction/en_beg_01_introduction_image00_village-in-indonesia.png
• a lot (all??) the images are not localized (fr, de, es… images are the same than en images)
• inside .md files, two markdown syntax are used to display an image:
• inside .md files, two markdown syntax are used to display an image:

![paper scanned][]
…
[paper scanned]: /images/en/beginner/06_field-papers/en_beg_06_field-papers_image20_watchsnapshot.png

![image](/images/fr/0300-12-23-private-data-store/image19.png)

With the regular expression below, I found only 1 796 images used by .md files in _posts/

!\[\w*\]

The regexp found images like:

![image20][]
![]({{site.baseurl}}/images/en/intermediate/en_int_ch6_image14.png)

If I search only in the _posts/en folder, only 236 are really used !!

Finally, there is 3 036 images in the folder images !!

[Nick-Tallguy]

@pyrog thanks for your work on this. I'm no expert, but reading through what you are proposing, it looks as if we would have a far easier to understand 'filing system' for the files needed to display the site if we adopted it.
I know that I found it a bit of a headache working out what to use as the header on .md files - I can't say as I understand it now, I just know how to make it work, but the logic of it seems to defeat me - comments about a blog make more sense when I look at the system we currently have.
I've looked through, but can't see where & how images would be filed? (If they end up in the same folder as the .md file, this could simplify pdf production for some of the software I've looked at.)

I'd be interested to see what the proposal would be for a transition - How easy would it be to change from one filing system to another, and would the site be available throughout? The site is constantly being updated, so a transfer would get a little complicated. I'm guessing we could branch, change the structure in the branch, alter the file which shows the web address, and if there is a problem quickly change back until it is sorted - the proposed site can be viewed on jekyll when being 'built' - need the advice of someone with more knowledge though.

My own level of knowledge is such that I would not attempt to start something like this proposal unless there were assurances from users such as yourself who understand how this should work, that they would stay with the project & help with ironing out any glitches.

Whatever system we end up with needs better documentation of 'howto' for those who are willing to write articles about mapping, HOT, etc., than we currently have. For the future, and those who follow on from us, it would be good to have a more logical system so mappers who wish to help others can easily write articles & submit them for inclusion.

Could the change over be managed incrementally?

I am concerned that the amount of effort involved could outweigh the benefits - I need someone with more knowledge on the subject to offer advice so we could then discuss & decide what to do.

[pyrog]

I've looked through, but can't see where & how images would be filed?

Where you want inside the project tree 😉

(If they end up in the same folder as the .md file, this could simplify pdf production for some of the software I've looked at.)

I just made a test and it works 😄

In /en/beginner/introduction.md

![A village in Indonesia](images/village-in-indonesia.png)

But if you want to use the same image from the french translation, it could be tricky:

• ![Un village en Indonésie](images/village-in-indonesia.png) don't work 😞
• ![Un village en Indonésie](/en/beginner/images/village-in-indonesia.png) work 😄

The simplest solution for editors is to choose the first one, but dev need to:

1. add a symlink from /fr/beginner/images → /en/beginner/images, or
2. write a plugin that automatically fix the image url if it is missing in the "local" folder
3. use a redirection mechanism in 404 layout ? (no tested)

Example for case 2.

In /fr/beginner/introduction.md

![A village in Indonesia](images/village-in-indonesia.png)

1. search for /fr/beginner/images/village-in-indonesia.png
2. found nothing
3. replace fr by en /en/beginner/images/village-in-indonesia.png

Currently, I use the KIS method : /images/beginner/village-in-indonesia.png

If an image should be localized, I suggest to add the language code in the prefix:

• /images/beginner/save-button.png (for the english page)
  
• /images/beginner/save-button.fr.png (for the french page)
  

[pyrog]

I'm guessing we could branch, change the structure in the branch, alter the file which shows the web address, and if there is a problem quickly change back until it is sorted

Yes, it's easy with git and github

[pyrog]

Tidiying images see discussion in #94

[pyrog]

I updated http://pyrog.github.io/learnosm-sandbox/
The title search engine is functionnal 😄

PS: type "iD" : it find 2 pages with the same title. They are in 2 different guides 😉

[lxbarth]

@pyrog - the proposed simplifications sound great at first read. Would be awesome to see a pull request against this repo.

[althio]

@lxbarth
Thanks for the initial feedback on @pyrog's proposals.
I am still unsure for how long we should work on the sandbox and when we should try an actual pull request against this (live) repo. Your thoughts?

If we agree to make this switch, what about these steps:

• 1. building the general structure, a few sections and a few plugins in the sandbox
• 2. demonstrating and testing sandbox
• 3. decision: approval to move forward
• 4. upgrading sandbox to full content
• 5. freeze live site/repo, fork old as backup
• 6. decision: approval to move forward
• 7. move content from sandbox to live repo

[lxbarth]

Work in a fork right away, pull request as soon as you want a specific
discussion aboit changes. Aka now?

On Tuesday, February 10, 2015, althio notifications@github.com wrote:

@lxbarth https://github.com/lxbarth
Thanks for the initial feedback on @pyrog https://github.com/pyrog's
proposals.
I am still unsure for how long we should work on the sandbox
http://pyrog.github.io/learnosm-sandbox/ and when we should try an
actual pull request against this (live) repo. Your thoughts?

If we agree to make this switch, what about these steps:

• 1. building the general structure, a few sections and a few plugins
     in the sandbox
• 2. demonstrating and testing sandbox
• 3. decision: approval to move forward
• 4. upgrading sandbox to full content
• 5. freeze live site/repo, fork old as backup
• 6. decision: approval to move forward
• 7. move content from sandbox to live repo

—
Reply to this email directly or view it on GitHub
#306 (comment).

[althio]

@lxbarth
This is a complete change of structure (no more _posts folder) and some configuration tweaks after that.
Even if the literal content is the same and the resulting website very similar, it is significantly different in the repo.

That is why @pyrog worked in a brand new repo, not a regular fork.
https://github.com/pyrog/learnosm-sandbox/commits/master?page=2
I don't think it is a good idea to create a pull request for a specific change.
It is a total internal makeover.

Based on that, could you have a look at, comment and review:
Jekyll source code: https://github.com/pyrog/learnosm-sandbox
Resulting website: http://pyrog.github.io/learnosm-sandbox/en/

[jmarlena]

Hey @pyrog! I am excited about the prospects of changing the site structure to better fit the learning materials. I am interested in better understanding how removing the _posts folder affects the site.

I tried to test the existing sandbox site with the "next & previous" buttons from the code I used in issue #270 but it doesn't seem to be working at all and I am not sure why. Any ideas?

Also, above you mention taking out the permalinks and leaving categories but that's not reflected in the sandboxrepo, has this just not been updated yet or was there another reason for this?

Here's the code I used for the next/previous buttons (same as in #270):

[Nick-Tallguy]

Copy & paste of a summary comment from @pyrog concerning a filing system for images on potential new site discussed within comments on pull request #327

---

In "my" sandbox, I use only one image folder but there will be to many images.
I thought we could use a folder by guide, but I like 4/.

But I prefer to avoid language subfolder and use a language extension (See Content negotiation concept).

/images/remote/iD_01.png for the english version
/images/remote/iD_01.fr.png

If an image is "neutral language" (like a picture of a mapper using a gps receiver) all "translations" use the "english" image (/images/gps/mapper-holding-a-gps-receiver.png)

But if the image contain important text or have a cultural meaning, for example french translation of a chapter use /images/field-papers/website.fr.png if french translators provide it, otherwise they use the "english" version.

See Internationalization and localization of images.

[pyrog]

@jmarlena
Your code work only for posts, not for pages. See jekyll/lib/jekyll/page.rb jekyll/lib/jekyll/post.rb

{% if page.previous.url %} 
    <a rel="prev" href="{{page.previous.url}}">&larr; {{page.previous.title}}</a> 
  {% endif %} 
  {% if page.next.url %} 
    <a rel="next" href="{{page.next.url}}">{{page.next.title}} &rarr;</a> 
  {% endif %} 

It's possible to use a plugin like Jekyll-pagination but I worked on a TOC plugin. See pyrog/learnosm-mm

But the current issue is how to use Jekyll plugins (or other static web pages generator) with github-pages. Do you have some tricks from Travis.CI team ?

[jmarlena]

@pyrog
The documentation and several blogs I've read on customizing your Jekyll site all say that custom plugins don't work with github pages but I found a blog post describing a "work around" that requires maintaining a master & gh-pages branch. Below is a link of how they tried to streamline the process of maintaining both branches.
http://blog.nitrous.io/2013/08/30/using-jekyll-plugins-on-github-pages.html

I will ask the Travis.CI team if they have any different ideas/input on how to use custom Jekyll plugins with github-pages. I will also try experimenting with this. I am not sure maintaining multiple branches for this site is feasible with making learnosm.org more user friendly. The prospect of switching over to another static site generator seems more and more feasible to me.

Also, I presume that TOC = Table of Contents plugin?

[Nick-Tallguy]

@jmarlena & @pyrog for your info - we have a new intern starting with us shortly - not sure if they already have a username on github. @althio & I have put together https://hackpad.com/Outreachy-LearnOSM-content-ecmiGI21ZlM in an attempt to get Anwar up to speed as quickly as possible.

I've started to update the files & images in line with the agreements so far, but there is a lot to do - any help appreciated. We may / will need to start allocating which files will be updated by which person to avoid us wasting time by working on the same file at the same time.

As we have someone translating the site from English to Swahili, I've concentrated on making sure the English section, starting with Beginner, is up to date before it is translated, so that the new files in the Swahili section, and the images associated with it, are already in the correct format.

Any help / suggestions appreciated.

[jmarlena]

@Nick-Tallguy @althio @pyrog

I added a comment on the hackpad under the comment section on UX resources. My hope is that the UX terms and resources I shared will help the new intern find relevant help and a framework for understanding the site's problems and possible solutions. For example, here's a learosm_site_audit google docs spreadsheet I started from a Content Audit template, which I originally found from this helpful UX Recipe Checklist. I think using google docs is a good way to manage content inventory/audit.

I also added some comments on the hackpad summarizing the technical challenges I encountered with the site and some possible solutions.

[althio]

@jmarlena
Thank you for the detailed input and the ideas on 'content audit' in the internship hackpad.
Much appreciated!

[Nick-Tallguy]

@jmarlena @althio @lxbarth @pyrog

I've added a second sheet to the spreadsheet at learosm_site_audit which shows the progress of the 'tidying', as well as some anomalies on files present in different languages.

As I've been working through I've often renamed all the files in a folder so that it is easier to make additions later (only while we are still working with the '_posts' blog system) Basically I've spread the 'dates' so we can more easily see what is needed, and what we have.

I haven't updated the wiki, as althio & I have set that as a target for our intern. I'm also hoping for help with the tidying.

Although the 'tidying' is time consuming, I've made considerable inroads on the number of deleted redundant images (I'm checking that none are 'localized' before doing so), and making other slight improvements as I go - correcting broken links, adding a comment on the top of 'old' files that a newer version exists in another language, etc..

My intention at present is:

• Try to keep ahead of anyone doing translating so they translate on a 'clean & tidy' section
• If another section is needed urgently, try to have that folder updated in 'en' first so we have a system in place that may help others to help us - they can see where we are going
• Complete all of the 'en' folders - most translations seem to come from there (I'm willing to be corrected on that)
• Then work my way down the spreadsheet with the other languages - hopefully with help!
• Once all the duplicate images are sorted, it should be a relatively simple task to rename the files, taking out the 'date' element at the start of them - this needs to take place at the same time as enhancement to the .yml files, and possibly enhancements using Jekyll plugins and other features for the creation of the static site. (At or just before this point need to establish how I/we can view our proposed changes before publication - at present I use Jekyll & rdiscount, but it doesn't render everything exactly the same as the current site)

[Nick-Tallguy]

@pyrog I'm closing this now but I agree and it's a long term aim. Using our current system we' re gradually improving the situation. I'll keep working on this even though the issue is closed - trying to tidy the issues as well!