Breaking Change: Rethink Lesson Folder Organization

[zkamvar]

At the moment, the folders in the lesson are all organized underneath the episodes folder. This is.... not optimal because it breaks the relationship between the folder structure you see in the repository with the folder structure that you see on the website. Ideally, you would want to have the folder structure reflect the structure of the website menu.

All of this can change, but it would be a good idea to make sure that we can switch between folder structures (heavily related to #19).

At the moment, it would be good to have a similar structure to what we have for the current template with the flexibility and notion that it could change.

I think the idea behind this is that the folders could look like this:

./
|-- CONDUCT.md
|-- SETUP.md
|-- episodes/
|-- extras/
|-- something/ 
|   `-- hidden/
`- LICENSE.md

The issue with this is how you tell {sandpaper} that this is the configuration you want without getting too in the weeds with yaml files. Why, you ask? Because I want to avoid behemoth config files that look like this: https://github.com/grunwaldlab/Population_Genetics_in_R/blob/master/_site.yml

It might be possible to do something like this, but it might be too complex 😬

- episodes
  path: episodes/
  order:
  - introduction.md
  - getting-to-know-vectors.Rmd
- extras
  path: extras/
  order:
  - factors.Rmd
  - classes.Rmd
- hidden
  path: something/hidden/
  order:
  - hidden-extra.md

[fmichonneau]

If we could avoid having to specify a yaml file for content organization that would be best.

Would it help if we used a content folder with everything site-related in it?

./
|-- CONDUCT.md
|-- SETUP.md
|-- content/
|   `-- episodes/
|    |-- extras/
|    |-- something/ 
|        `-- hidden/
`- LICENSE.md

[zkamvar]

I need to keep remembering to add discussions related to issues in the issue itself. François described the situation perfectly:

I only see 2 options: either you rely on the file names or you provide the order in another file. Neither option really sparks joy.

[emphasis: Zhian]

The problem with specifying file order via file names (which is our current system) is the fact that if you want to create a new episode in the middle of the lesson, then you must rename all of the episodes after it or name it something really awkward. Moreover, when you rename an episode, all the internal links will need to change.

The problem with specifying file order via yaml file is that a) the authors have to abide by yaml syntax and b) the authors must remember to update the file if the file names changes (either via name change or deletion) or a new file is added.

Again, neither of these solutions spark joy, but I think the yaml solution at least provides an easier way to fix mistakes and modify lessons. Moreover, with the yaml specification, it's possible to have files in development that don't need to go into production. You cannot do that if you have a file-based ordering system.

[zkamvar]

As per our discussion today, there are a couple of issues here:

1. The current navigation bar that contains a folder of "extra" items is a trash can of materials and definitely needs a rethink
2. The maintainers need to have a way of specifying the order of items within dropdown menus, but not necessarily the order of the navigation bar.
3. While we may want for this package to be of use outside of The Carpentries, we have to make sure that we decrease as much complexity for our maintainers as possible.
4. There is the expert awareness gap of writing YAML that must be addressed.

Items 1 and 3 are tough to tackle and I think are best left until we have a clear path forward about how we want to template order of files in episodes. I am still an advocate for using a separate YAML file to specify episode order because of a couple of things:

1. Of all the templating syntaxes, YAML is the simplest.
2. The depth of YAML required for ordering episodes is a single level, so any mistakes should be fixable by adding or removing a character.
3. We can include a YAML linter in the check for an episode.
4. Current episodes do this for objectives, questions, and keypoints at the same level we are proposing.
5. Any non-trivial RMarkdown document will use nested levels of YAML elements.

[fmichonneau]

Thanks for this summary. It captures the various issues nicely.

I agree that YAML is the least worse option. Some of the drawbacks can be addressed by providing helpers to write and manipulate the YAML files (hand typing possible for people comfortable doing so, but providing helpers to make it easier for people who are not as familiar with YAML). These helpers can be developed later and we can move forward with using YAML to specify the order of the episodes.

Regarding the extra folder, after thinking a little more about it, a possible solution would be to split it into "Resources for learners" (setup, reference, ...) and "Resources for Instructors" (instructor guide, discussion, ...). This is not perfect but I think taking this role-based approach could help organize the mess that is the extra folder and the dropdown. For other things, the distinction might not be as clear: for instance, the recently proposed carpentries/styles#517 could fit in either. However, @tobyhodges and I had spoken in the past about including more information about lesson design decisions that could fit well under the "Resources for Instructors".

For item 2 in your first list, I also think we could also use YAML for the order of the navigation bar, and enforce the order as part of the checks that would be Carpentries-specific. This approach should make extensibility outside of the Carpentries more straightforward.

[tobyhodges]

However, @tobyhodges and I had spoken in the past about including more information about lesson design decisions that could fit well under the "Resources for Instructors".

Yes I would really love a way to do this, so lesson developers could keep a record of design choices: links to concept maps, high-level discussion and decisions about the dataset users/narrative of the lesson, learner profiles, etc I'm currently designing a study group-style series of sessions to guide people through the lesson design/development process, and it would be great for them to have a place to keep materials like this that they develop during that process in the lesson pages/repository itself.

To address this discussion more generally: I really like the idea of separate dropdowns for Extras intended for learners and instructors. Regarding reliance on more YAML to specify the order of such things in the lesson navigation: right now, lesson developers are required to be able to work with some basic YAML in order to write the page front matter correctly, which IMO makes the YAML array used to specify episode_order and extras_order not much of an additional hurdle. Are you planning to introduce some other way of specifying learning objectives, etc in each episode file, which wouldn't involve large blocks of YAML?

[zkamvar]

Regarding the extra folder, after thinking a little more about it, a possible solution would be to split it into "Resources for learners" (setup, reference, ...) and "Resources for Instructors" (instructor guide, discussion, ...).

I like this approach. It definitely helps differentiate resources for learners and instructors. In terms of https://github.com/caprentries/styles/issues/517, this could be a whole different dropdown menu and folder/markdown file.

One of the things I'm torn between is where to store these pages on the website itself. At the moment, AFAICT, all of the pages are stored at the top of the site, so if you go to site.carpentries/intro-to-r or site.carpentries/some-resource, you will get to that page, no matter what menu it lives under. Do we still want to keep this paradigm?

Regarding reliance on more YAML to specify the order of such things in the lesson navigation: right now, lesson developers are required to be able to work with some basic YAML in order to write the page front matter correctly, which IMO makes the YAML array used to specify episode_order and extras_order not much of an additional hurdle.

I agree with this. At the moment, people have do write each Objective as a yaml list like:

objectives:
- "stay the course"
- "defeat my enemies"
- "seek the golden fleece"

This is exactly the syntax I expect for listing the episodes, but instead of each entry being string of words, it will be a file name. I do not expect it to be any more complicated than that.

Are you planning to introduce some other way of specifying learning objectives, etc in each episode file, which wouldn't involve large blocks of YAML?

YES. @fmichonneau and I want to keep the yaml structure for the lessons as simple as possible. The specifications I have in mind for yaml are here: https://github.com/zkamvar/stunning-barnacle/blob/main/Landscape.md#yaml-metadata. The lesson, with the special sections, would look like this.

[zkamvar]

That being said.... at the moment, I'm going to keep the navigation bar fixed, because my current strategy for parsing the file doesn't allow for that.... but in the future it should be possible.

[tobyhodges]

This sounds really good to me @zkamvar. Especially when I factor in the discussion below the example you linked, about removing the need to follow the section separators (::::::) with a heading.

[tobyhodges]

One comment on that, though: I think it would be good to allow for exercise blocks to have custom headings, rather than all of them being called "Exercise." Having individual titles for exercises helps us support learners with questions, by making each exercise more quickly identifiable. That also helps people give feedback e.g. in GitHub issues by being able to refer to a particular exercise by name rather than e.g. "the second exercise in the Watercolors episode."

Is it possible to have a default "Exercise" heading added automatically, unless that default is overridden by the author specifying a heading for that section? (Such an override would probably only be necessary for exercises, and shouldn't be included for Objectives, Questions, Key Points.)

[fmichonneau]

Is it possible to have a default "Exercise" heading added automatically, unless that default is overridden by the author specifying a heading for that section? (Such an override would probably only be necessary for exercises, and shouldn't be included for Objectives, Questions, Key Points.)

That's what I'd like to see as well. I don't think we could do this with pure CSS but should be quite doable as part of the processing.

[zkamvar]

Is it possible to have a default "Exercise" heading added automatically, unless that default is overridden by the author specifying a heading for that section? (Such an override would probably only be necessary for exercises, and shouldn't be included for Objectives, Questions, Key Points.)

Yes. This has been my plan all along. As @fmichonneau mentioned, it will be doable in the processing from static markdown to HTML via pandoc lua filters.