Collections of Projects

[shubham76]

Collections of Projects

This project involves building a user interface for groups of projects in Read the Docs (Collections).
Users would be allowed to create, publish, and search a Collection of projects that they care about.
We would also allow for automatic creation of Collections based on a project's setup.py or requirements.txt.

Once a user has a Collection,
we would allow them to do a few sets of actions on them:

• Search across all the projects in the Collection with one search dialog
• Download all the project's documentation (PDF, HTMLZip, Epub) for offline viewing
• Build a landing page for the collection that lists out all the projects, and could even have a user-editable description, similar to our project listing page.

There is likely other ideas that could be done with Collections over time.

[RichardLitt]

I don't know the answer, but I do know that if you're kind enough to open an issue, than it is likely that other people didn't. We should edit the docs to include this information once we figure it out.

[shubham76]

Oh! Great! Thank you. :)
@ericholscher & @humitos will be able to give more insight, may be?
Thank you for your time and sorry for the trouble :)

[agjohnson]

I'm re-purposing this issue to address implementing this feature.

[dojutsu-user]

@ericholscher

We would also allow for automatic creation of Collections based on a project's setup.py or requirements.txt.

What is the meaning of this line. I believe that 'Collections` is specific to a user and not specific to a project. Like there might be a user with zero projects built in RTD but may have one or more collections.

I have some loose ideas about features of collections:

1. Multiple Collection - A user can have multiple collections and not just one.
2. Name and Description - Each collection can be given a specific name so to make it shareable.
3. (From the issue description) Search across all the projects in the Collection with one search dialog.
4. (From the issue description) Download all the project's documentation (PDF, HTMLZip, Epub) for offline viewing.
5. (From the issue description) Landing page for each collection listing out all the projects.
6. Social sharing links for each version from the landing page.

Other features we might want:

1. Forking of a collection. It is just like github repo forking. Like I can fork a collection and add more projects to it.
2. Collections of collections - Like... These are my favourite collections.
3. Projects in a collection can be public and private. Like if a project is private in a collection, it is not visible to other users other than the admin on the collections page.

I would like to discuss this Feature more in detail.

cc: @agjohnson @humitos @stsewd @safwanrahman

[agjohnson]

In thinking about how to implement this, I'm completely overwhelmed by the number of features we're tacking on here and I'm starting to have reservations about collections as a feature. I think this would benefit from a deeper initial planning phase and would definitely benefit from discussions with users.

We need to start by at least defining who wants this and how they would use it, as we've not discussed the feature at this level yet. Preferably we have actual input from users here, because I feel like we're making assumptions about how this feature would be used. So before discussing what additional features we can tack on here, I'd rather step way back and look at this from the user perspective.

We haven't even defined who this feature is for, to start. Is is a maintainer feature for organizing or showing off a group of projects they maintain? Is it a non-RTD user feature for maintaining a private list of projects they commonly search? Is it both? Who wants this right now? If this is for possibly non-RTD users, how do we even get them using RTD for this feature?

After that, we still have a lot of UX and product questions we should be asking ourselves. @dojutsu-user raised some good points, but now I'm just swimming in questions, which means we haven't planned this out enough to start or guide someone on the project.

So, before we over-engineer something, let's plan this at a much deeper level and perhaps discuss the feature with some users. Do we know of any that have asked for a feature like this and would be willing to talk with us?

[dojutsu-user]

@agjohnson
I can think of some use cases for this -

1. A user can create a collection of dependencies for a larger project.
   Example - A user have a large project that uses certain versions of some projects. So instead of going to one of the dependency project and manually select its version, he/she can create a collection of particular versions of projects to ease the task.
   Like I am using certain version of requests in my project and I don't want to open its documentation and select that specific version every time I want to read its documentation. So I add it to my "dependency-collection"

2. I can make a collections of some documentations that are a good sources of knowledge (let's say... beginner-friendly python sources)... and I can share it.

3. A collection of my talks/presentation. (I went to PyCon India 2018, and in one of the workshop I attended, teacher was using Read the Docs for his talk.)

There can be many more use cases like this.
I would like to request the team to give this discussion a priority because it is one of the GSoC ideas.

[humitos]

We haven't even defined who this feature is for, to start. Is is a maintainer feature for organizing or showing off a group of projects they maintain? Is it a non-RTD user feature for maintaining a private list of projects they commonly search? Is it both? Who wants this right now? If this is for possibly non-RTD users, how do we even get them using RTD for this feature?

@agjohnson when I think about Collections it comes to me organizations/groups like: encode, pallets and jazzband. So, maybe they are good people to ask about how they could be benefits from this.

On the other hand, what would be the differences between a Collections of projects and the concept of Organization that we have in the corporate site. Maybe it worth to make more effort in merging our code base instead of "copying" a feature/concept from the corporate into the community site.

To me, it seems that a combination of Organizations + Projects + Subprojects together with a better UX is what it's more related to what I think we want.

[ericholscher]

My initial thoughts around this were mostly for search. Specifically:

• It would be cool to be able to create custom search indexes of arbitrary sets of projects
  • Those could then be exposed to other users, or just used for yourself

A lot of the ideas that @dojutsu-user mentioned in this comment capture my thoughts.

Thinking more about this, I agree there's a few similar but overlapping ideas:

• My original idea, which is a user feature
• A similar idea to my original, but adding the ability to specify that Collection as the search for a project, which would be a maintainer feature. This would replace the "project + subprojects" search index that we automatically use, and make it more configurable for users, so that we could more explicitly define a custom search index for their project. This would be pretty lightweight.
• An autogenerated landing page that shows the collection of projects in a nice UI. This is a maintainer feature. It could be attached to a specific "project & subprojects" instead of a Collection, but I think Collections make it much more flexible (eg. you can add projects you don't own into your Collection). This is a bit complicated because we might want to support custom Domain's and other traditionally Project focused things.
• A replacement or enhancement of Organizations. This would be a much larger project, but could bring Organizations (similar to GitHub) to the .org, allowing the sharing of permissions around a set of projects. This feels different though, and I think we'd want both Organizations and Collections.

I think the main value we get from Collections is having an object we can attach Project's to that isn't itself a Project. I think that Organizations could have some value here for attaching permission around Projects, but I think it would be useful to have another grouping for other functionality; eg. search, specific versions, downloads, and other specific metadata. I guess one way of thinking about this is taking the "Project & Subprojects" concept and building it into a more fully formed idea, with a proper abstraction and UX that is more powerful.

I don't have a fully formed idea of the UI/UX here. I mostly was thinking about it from the search side of things, and I think there's value in that implementation without adding all the rest of this complexity. I think we would find the abstraction useful over time, and consider adding more functionality onto Collections as we got more comfortable with the ideas.

[dojutsu-user]

@ericholscher
For the UI/UX, we can use the existing styles for maintaining the consistency.

It would be cool to be able to create custom search indexes of arbitrary sets of projects

Wouldn't it create too many redundant indexes, like many users can have collections of different names but have same project in their collection?

Also thanks for the clarification. I was having the doubt about whether this will this remain as an idea or not.

[agjohnson]

For the UI/UX, we can use the existing styles

@dojutsu-user When we say we're not sure of the UX, we're more concerned about how users interact with and use the feature at this stage, not necessarily the styles/css/or technical implementation.

@ericholscher agreed that we are discussing many different core feature sets here.

Collections as a reader user feature has me thinking instead about a third party project that interacts with our API, instead of an RTD feature. I think we've described a lot of features here which would be great for reader personalized docsets. Exploring this as a separate product entirely would keep the delineation between reader and maintainer users more clear on RTD, and would make the UX around sharing collections/etc much more clear. I do worry this might be overlapping with Dash/Zeal, or other API/doc browsers, so maybe that is a different way we can take this feature -- by supporting Dash/Zeal, or others, directly.

So I think this is the definition of "collections" that we're excited by. The other features are separately useful, but maybe we need to break them away from the "collections" nomenclature.

We might have already talked about an artbitrary search index configuration, which is a feature we need on the .com and more generally. Also, subproject nesting for search is not great UX to begin with. This could potentially be controlled by a config file option. Configuration would need both a list of search index ids to index to, and a configuration option for the search index to query on in-doc search. This would be a great addition!

A feature that organizes project nesting/hierarchy could be helpful, though would be a complicated change given how deeply subprojects are implemented already. As much as I'd like to see better options for nesting here, it's maybe a low priority? The lack of multiple nesting levels, hardcoded /projects/<slug> URLs, and using subprojects as our only search index compounding mechanism are all subpar UX however.

Organization landing pages is hard UX to tackle, given how domains are tied to projects and how common it is to customize HTML output for these pages. So my vote would be to make a project type that facilitates landing pages more easily (probably with a good template project), and rely on the existing mechanisms for domain handing, project nesting, and HTML output. We can maybe focus on making UX here a lot better instead of reimplementing these features for organization landing pages? It's also going to be a feature that isn't used heavily ultimately, so we probably don't need to make a huge investment here, I see more value in having good resources and examples for organizations to use. Also, a strong APIv3 will make this all much more natural and easier to create an automated landing page project.

[dojutsu-user]

@agjohnson
Thanks for the explanation.

I still have some doubts though.

• Collections as a reader user feature has me thinking instead about a third party project that interacts with our API, instead of an RTD feature.

  Is this going to be a completely different project interacting with the RTD API ? If yes, this can be implemented with ReactJs.

• This could potentially be controlled by a config file option. Configuration would need both a list of search index ids to index to, and a configuration option for the search index to query on in-doc search.

  Sorry for so much questions, but I am not able to understand this. Can you please explain it a bit more.

• Also, I am unable to visualize the problem with landing pages. For the collections landing page, which will contains the list of projects in it, we can just list out the name and maintainers and link them to readthedocs.org/projects/<project_slug>.

Sorry for so much questions and not able to understand things in one go. But I am really interested in this project and want to understand it better.

[stsewd]

Just an update here, related to search we now can search for several projects and different versions at the same time

readthedocs.org/readthedocs/search/faceted_search.py

Lines 54 to 55 in 8c2f334

	:param projects: A dictionary of project slugs mapped to a `VersionData` object.
	Results are filter with these values.