Proposal for choosing RSS vs. episode JSON

[theDanielJLewis]

Building on my soapbox thread about how we name things, I want to propose something that could help us keep the RSS feed optimized while also guiding us for how to implement future features.

First, let's ditch the "chapters file" name. It's a file that can hold far more data about an episode than merely chapters. I think we should call it an "episode data file," or something similar.

So then to my point of this proposal: what should go in the episode data file, and what should go in the RSS feed?

I think we can make this easy: anything that's not required until opening/playing an episode should be in the episode data file.

For example, I don't think audiences need to see this information until they open or play an episode, and thus would be best in the episode data file:

• Episode notes (Proposal - <podcast:notes> #399)
• Transcripts (although we have separate files for these)
• Chapters
• Image galleries and chapter text blocks (Proposal: Extend chapter capabilities #400)
• Soundbites

But the following information should be visible before someone opens or plays an episode, and thus would be best in the RSS feed:

• Episode title
• Episode description
• Season numbers and titles
• Episode numbers
• Episode types (Proposal: <podcast:episodeType> #398)
• Episode length
• Episode image

And here's information that could fit well in either RSS or episode JSON:

• Person tags—Some podcast apps might want to show the people in the episode list, others might not show the people until the episode is opened/played, and others might use people for search.
• Episode-level value tags—I actually don't see a need for this to be in the RSS feed except that it can be supported without having to upload a separate episode data file.

I think this starts to reveal a difference between the features, and thus guides where they go. Heavier, more involved features that might be added or edited later (notes, transcripts, chapters, and image galleries or chapter text blocks) are good for the episode data file, where updates can be released without triggering any RSS refreshes, and those updates will be loaded when the episode is played.

On the flip side, the shorter, simpler fields most likely exposed in podcast apps (through browsing, searching, and previewing) are good for the RSS feed and will most likely not be updated after publishing the episode for which these fields are used.

So if the data is needed before opening or playing an episode, it goes in the RSS feed. If it's not needed until opening or playing an episode, it goes in the episode data file (JSON).

[theDanielJLewis]

This would give the added benefit of refreshing the RSS feed only when a new item (live or episode) or top-level information is updated.

[theDanielJLewis]

Soundbites would be something else I think best suited for the episode data file.

[theDanielJLewis]

I couldn't find where I've mentioned this previously, but another way to look at this is that the RSS feed should at least expose the existence of features a podcast app wants (thus keeping the <podcast:chapters> tag in the feed), but these deeper pieces of data can be deferred to the episode data file (like how <podcast:chapters> merely points to the file for chapters).

So stuff like <podcast:notes> would also contain a link to the episode data file that has both the chapters and the notes, and other deeper-experience features could also be exposed in the feed but given details from the episode data file.

[ryan-lp]

First, let's ditch the "chapters file" name. It's a file that can hold far more data about an episode than merely chapters. I think we should call it an "episode data file," or something similar.

I agree that a good principle to follow is that data can go in the same file if it is always downloaded at the same time. But, another good principle to follow is that data can go in the same file if it is updated at the same time or by the same author.

When we consider the authoring tools, it is reasonable that we have different authoring tools for the podcast feed itself, for the transcripts, and for the chapters. The podcast author will likely author the episode title, description and also the show notes, and may leave chapters and transcripts to others, either to 3rd party service or to people in the listener community.

If you put unrelated things into the same file only because they are downloaded at the same time, it may make that data difficult to author and update.

[theDanielJLewis]

I suspect that case of letting the audience create chapters is going to be extremely rare—maybe only NA-inspired podcasts.

Yes, there are team responsibilities, but I think anyone who creates chapters would also probably be involved in managing the other episode data.

Do we really want separate files for chapters, transcripts, notes, people, soundbites, etc.? Or can we simplify the management of these—both for developers and podcasters—by consolidating as much as possible?

[ryan-lp]

Do we really want separate files for chapters, transcripts, notes, people, soundbites, etc.? Or can we simplify the management of these—both for developers and podcasters—by consolidating as much as possible?

You might be underestimating how much development work goes into just one of these tools that you think combining them into one would make things easier for developers. I have spent the better part of the last year developing a transcription tool, and that is surely taxing enough. There are many intricacies to transcription to make it work well, from the natural language processing, to the editing, and to the UI/UX design. I have no interest to spread myself thin by adding sound bites or podcast metadata or chapters to this. Transcripts are what I specialise in, and I don't have the resources to tackle anything else. Nor would all users necessarily want to be locked into one vendor that does everything.

What we need is the flexibility to allow different vendors to focus on one thing, and do one thing well, and to also develop better integrations between these different vendors so that different parts of the ecosystem can work together and provide consumer choice.

Separate files doesn't preclude someone from developing a monolithic authoring tool, it just makes it possible to choose either a monolithic or a specialised tool. It doesn't matter whether their are multiple files, authoring tools can manage multiple files just fine. I wouldn't put every single little thing in its own file, that's too extreme, but when you look at the authoring process, I think you can see that chapters has its own timeline of a certain type of media, while transcripts has its own timeline of a different kind of media. The authoring tools will be different, and the artifacts produced by these tools should more conveniently be in separate files for better flexibility.

[theDanielJLewis]

@ryan-lp, for my context, can you please remind me which thing it is that you work on? I apologize that I can't remember.

There are many intricacies to transcription to make it work well, from the natural language processing, to the editing, and to the UI/UX design.

That's a good point, and I think transcripts still have a logical reason to be a separate file.

What we need is the flexibility to allow different vendors to focus on one thing, and do one thing well, and to also develop better integrations between these different vendors so that different parts of the ecosystem can work together and provide consumer choice.

If we consider things like chapters, notes, soundbites, and galleries for a single episode metadata file, I can foresee allowing multiple apps/services to add to that file just like multiple apps/services can already add to MP3s: audio processing, ID3 text and images, dynamic content-insertion, and such.

[jamescridland]

I can foresee allowing multiple apps/services to add to that file just like multiple apps/services can already add to MP3s: audio processing, ID3 text and images, dynamic content-insertion, and such.

This is an imperfect comparison. MP3s are published in one place, and these services are normally chained together upon publish. It's not possible for multiple services to independently hook into MP3s and reformat them.

Because a transcript is a different file, hosted in a different place, that means that I could have in my own RSS feed a link to a third-party provider of transcripts, and those could be directly provided by that one third party. That isn't possible - or, at least, much harder - with one file containing lots of bits of content.

[theDanielJLewis]

So we keep transcripts separate. I think that's fine and makes sense because there are already standard file formats for transcripts. So publishing those same files makes the transcripts fully compliant with countless existing systems.

[keunes]

From the initial post:

So if the data is needed before opening or playing an episode, it goes in the RSS feed. If it's not needed until opening or playing an episode, it goes in the episode data file (JSON).

Just to note: what one would expect not to be needed before, some of our (AntennaPod) users disagree: AntennaPod/AntennaPod#6258

[theDanielJLewis]

This is still opening the episode. A chapters indicator can be triggered by the chapters tag in the item, and even if a UI existed to open the chapters separate from the episode, then the app could simply download the episode metadata file to load the chapters from there on tap.

[keunes]

Sure. But that'd mean we'd have to download an extra file each time you view/open an episode. Not really a problem for chapters I'd say, but it would be for description (which I understand might get replaced by the notes tag). In AntennaPod you can swipe through/between episodes – imagine the lag we'd be introducing if we'd have to download the description each time swiping to (= opening) the next episode.

[theDanielJLewis]

The description would remain in the feed.

To clarify, I think there are 4 kinds of text that can represent podcast content:

1. Description/summary: this is essentially a paragraph that condenses the whole episode.
2. Article/post: this is the easily readable full representation of the content, as if it was written to be read.
3. Transcript: the verbatim (or close with some editing for clarity) record of everything spoken in the podcast.
4. Notes: the outline and actionable resources, similar to what a handout might be for a conference presentation or church sermon.

Of these, #1 is best in the feed, #2 is best on the site and for RSS readers (but not podcast apps), #3 is best in the <podcast:transcript> tag, and #4 is what I think should be in podcast apps and tied with chapters since the notes sections would nearly always align with chapters.

[keunes]

Fair enough. From some earlier comments somewhere it seemed that all was put on a big pile (specifically: that notes would replace/is the same as description).

[keunes]

I have a question: what might be the impact of merging these elements on the file size?

AntennaPod already has support for chapters. Let's say that we don't want to implement notes. Then we'll still have to pull in the notes because they're in the same file as the chapters. What would be the impact for end-users with slow, expensive and/or flaky connection?

[theDanielJLewis]

My proposal for #400 is to put notes in the chapters.

[keunes]

Ok, wrong introduction to the question from my side. Key question was/is:

What would be the impact [of putting everything in 1 file] for end-users with slow, expensive and/or flaky connection?

[theDanielJLewis]

I think it would be minimal because I don't think the metadata would get much larger than 8 KB uncompressed.

[theDanielJLewis]

I want to resurrect this because I've been thinking about the Value Time Split proposal (which Alecks won't let me comment on).

We're stuffing more and more in the RSS feed, like Value Time Splits, and it's causing me to wonder if chapters should really be an external file anymore.

I still think we should either have everything (excluding large content like images, multimedia, and transcripts) in the feed, or in external files (or a single external file).

Why not make value time splits a JSON format so it could be in the same episode metadata file with chapters?

Or why not bring chapters back into the feed so it can pair easily with Value Time Split?