add short description of what each "app" does

[rahulbot]

It is a little hard to figure out exactly what each "app" container is for, even thought the names are descriptive. I think it'd be helpful to have a high-level, one-sentence description of each all together in one file. Perhaps make a mediacloud/apps/README.md that looks like this:

• rescrape-media - When a new media source is added, scrape it to find RSS/Atom feeds to add.
• crawler-provider - Pull stories from RSS feeds by periodically adding jobs to a "fetch-RSS" queue
• crawler-fetcher - Grab fetch-RSS jobs off a queue and grab any new stories
• etc.

Just one sentence for each app explaining what its primary function is. I think that'd help quite a bit in terms of spinning up new people (and communicating amongst ourselves). I suggest putting it in with the code because this is about "what" the code in that folder is, while most of the stuff in docs seems to be about "how" and "why".

[pypt]

Super low effort attempts of mine to do that were made in both docker-compose.dist.yml (e.g. https://github.com/berkmancenter/mediacloud/blob/master/apps/docker-compose.dist.yml#L1189-L1191) plus at the top of each Dockerfile. Sure, I could have been less terse with those.

[rahulbot]

Oh, those are great! Didn't know they were there. Many are good, but I agree some are too short - for instance "Crawler fetcher" requires a fair amount of understanding about our system to get. I think ones like "Fetch finished podcast transcripts and store them locally" are quite good. Maybe just doing one more iteration on the very short ones in there would address the difficulty I was having?

[esirK]

This was really helpful on this issue #734 I can add the content to mediacloud/apps/README.md and create a PR 😁

[pypt]

@esirK if you're up for it, I think it would be better to extend annotations that we currently have in docker-compose.dist.yml, e.g.:

https://github.com/mediacloud/backend/blob/master/apps/docker-compose.dist.yml#L1164-L1166

This is the file that we constantly edit and update, and those Markdown files tend to get outdated pretty quickly :)