Add WebAPI for fetching torrent metadata #21015

Piccirello · 2024-07-01T06:11:00Z

This PR implements two new APIs for fetching a torrent's metadata. The APIs accept a magnet URI, torrent hash, .torrent URL, or uploaded .torrent file, and return the torrent's associated metadata. This PR also modifies the /torrents/add API to support downloading a torrent whose metadata has been previously fetched. The ultimate goal is for the WebUI to provide an Add Torrent experience equivalent to that of the GUI, where content can be reprioritized/unchecked before the torrent is added.

`/fetchMetadata` API

HTTP request

To request metadata for a torrent, specify the torrent in the source query parameter of a GET request to /api/v2/torrents/fetchMetadata. Torrents are supported in the following formats:

magnet URI (e.g. magnet:?xt=urn:btih:a8eeefc8a0dc402b24686ddfd775a409fe4b00e0&dn=example)
hash (e.g. a8eeefc8a0dc402b24686ddfd775a409fe4b00e0)
.torrent URL (e.g. https://example.com/example.torrent)

HTTP response

Given the asynchronous nature of retrieving metadata, there are two successful HTTP status codes used.

When metadata is requested for a torrent that requires asynchronous background work (i.e. connecting to DHT/peers), the client will receive a 202. A 202 indicates that the request was successful, but additional background work must be completed before a meaningful response can be provided. The response will contain the info hashes, if available, or an empty object.

GET /api/v2/torrents/fetchMetadata?source=abc
HTTP/1.1 202 OK
{
  "hash": string,
  "infohash_v1": string,
  "infohash_v2": string
} | {}

When metadata is available for the torrent, either because the torrent exists in the transfer list or because the metadata has been retrieved from a prior request, the client will receive a 200.

GET /api/v2/torrents/fetchMetadata?source=abc
HTTP/1.1 200 OK
{
  "comment": string,
  "created_by": string,
  "creation_date": int,
  "files": [],
  "hash": string,
  "infohash_v1": string,
  "infohash_v2": string,
  "name": string,
  "piece_size": int,
  "pieces_num": int,
  "private": bool,
  "total_size": int,
  "trackers": [],
  "webseeds": []
}

Retrieved metadata will be cached in the current web session. Subsequent requests performed within the same web session will return the metadata immediately, while other web sessions will be required to reretrieve the torrent's metadata from peers. Once a torrent is added using the cached metadata, the metadata is removed from the cache.

`/parseMetadata` API

HTTP request

To request metadata for one to several .torrent file(s), you may upload the files to the /parseMetadata API. To do so, submit the file(s) as multipart MIME data. You may use any key for the uploaded value.

To test file upload using curl, specify the -F flag (e.g. curl https://127.0.0.1:8080/api/v2/torrents/parseMetadata --get -F file=@"/root/example.torrent").

HTTP response

This API always responds to successful requests (i.e. valid torrent file(s)) with a 200. The response will contain the full metadata for the uploaded torrent(s). The response object is keyed off of the uploaded file's name.

GET /api/v2/torrents/parseMetadata
HTTP/1.1 200 OK
{
  "example.torrent": {
    "comment": string,
    "created_by": string,
    "creation_date": int,
    "files": [],
    "hash": string,
    "infohash_v1": string,
    "infohash_v2": string,
    "name": string,
    "piece_size": int,
    "pieces_num": int,
    "private": bool,
    "total_size": int,
    "trackers": [],
    "webseeds": []
  }
}

`/add` API

The existing /add API now supports using the metadata cache that's populated by the new metadata APIs. When specifying a url and/or torrent file to download, the metadata cache is first checked for the torrent. If found, the metadata is used directly from the cache, rather than needing to re-retrieve it. Note that when specifying the name of a .torrent file uploaded via the parseMetadata API, you must prepend file: to the file name. For example, if you uploaded example.torrent to the parseMetadata API, you can add this torrent via the /add API by specifying a url of file:example.torrent.

When metadata is retrieved directly from the cache, you may also specify a new filePriorities parameter. This parameter allows for specifying the file priority of each file in the torrent. This parameter may only be specified when adding a single torrent.

Alternatives:

I explored having the metadata API leave the request open until the metadata was available. Once the metadata was fetched, it would be returned directly in the response of the original request. One downside of this approach is that metadata retrieval can take an arbitrary long amount of time. This could result in torrents whose metadata could never be retrieved via this API (e.g. due to the retrieval taking longer than the client's/reverse proxy's request timeout). This approach would also require some further modification of qBittorrent's web application layer to suppress the default behavior of returning a blank response.

Future work:

Modify the WebUI to make use of the new /fetchMetadata API. This will likely mean splitting the current Add/Upload Torrent dialog into two dialogs. The first dialog will support specifying the URL(s)/.torrent file(s) to submit, while the second dialog will display the torrent's metadata and allow for modification of file priorities.
Support downloading the retrieved metadata as a .torrent file (as supported in the GUI)

Closes #20966.

src/base/bittorrent/torrentinfo.h

src/webui/api/serialize/serialize_torrent.h

src/webui/api/torrentscontroller.cpp

src/webui/api/torrentscontroller.h

glassez

Only preliminary comments after brief review.

src/webui/api/apicontroller.h

src/webui/api/serialize/serialize_torrent.h

src/webui/api/torrentscontroller.cpp

src/webui/api/torrentscontroller.h

glassez · 2024-07-01T12:48:39Z

@Piccirello
I wonder if it's difficult to implement client side bencode parser in JS? It is very possible that one (or more) already exists.
I believe that if we had used one, we would have got a more universal solution. Since users often add local (for the WebUI) .torrent files, we would not have to send it to the server first for parsing, and then again to add the torrent. As for magnet links, it would also look easier if you sent raw metadata to the client and forget about it. Otherwise, how do you propose to add such torrents (after WebUI user selects file priorities etc.)? Of course, you can store all this metadata in a session... But this seems to be a more cumbersome implementation.

In any case, this part of the API should be thought out, implying the subsequent addition of torrents. Otherwise, we may end up with something little useful in practice.

src/webui/api/torrentscontroller.cpp

Piccirello · 2024-07-01T18:18:18Z

Since users often add local (for the WebUI) .torrent files, we would not have to send it to the server first for parsing, and then again to add the torrent.

I'm not convinced that's the approach we'll eventually take. I can imagine sending the .torrent file once, returning the metadata to the client, and then allowing the torrent to be added without needing to re-send the file (likely by transmitting the info hash).

As for magnet links, it would also look easier if you sent raw metadata to the client and forget about it. Otherwise, how do you propose to add such torrents (after WebUI user selects file priorities etc.)? Of course, you can store all this metadata in a session... But this seems to be a more cumbersome implementation.

To me the session seems like an appropriate place to store this. I don't think the client should be responsible for parsing this data. It would also mean each client (official and unofficial) would need to implement it.

In any case, this part of the API should be thought out, implying the subsequent addition of torrents. Otherwise, we may end up with something little useful in practice.

I agree with this. I'll try to sketch out what the next steps would look like and how this API would be used.

NikcN22 · 2024-07-01T18:57:20Z

@Piccirello I wonder if it's difficult to implement client side bencode parser in JS? It is very possible that one (or more) already exists. I believe that if we had used one, we would have got a more universal solution. Since users often add local (for the WebUI) .torrent files, we would not have to send it to the server first for parsing, and then again to add the torrent. As for magnet links, it would also look easier if you sent raw metadata to the client and forget about it. Otherwise, how do you propose to add such torrents (after WebUI user selects file priorities etc.)? Of course, you can store all this metadata in a session... But this seems to be a more cumbersome implementation.

In any case, this part of the API should be thought out, implying the subsequent addition of torrents. Otherwise, we may end up with something little useful in practice.

I think there is no need to embed the Bencode decoder directly into the client API. This is quite easy to do directly in the “graphical” part. More interesting is the need to provide the ability to assign priority to files in the add method.

Chocobo1 · 2024-07-02T05:33:26Z

I wonder if it's difficult to implement client side bencode parser in JS? It is very possible that one (or more) already exists.

FYI, there certainly exists bencode encoder/decoder library in JS however I'm not aware that they are compatible with bittorrent v2. In the past, I had to mod one to suit my need.

glassez · 2024-07-03T05:45:51Z

FYI, there certainly exists bencode encoder/decoder library in JS however I'm not aware that they are compatible with bittorrent v2.

"bencode" format is independent from BitTorrent so generic "bencode" decoder should not depend on it too. Or do you refer to torrent file specific parsers?

Chocobo1 · 2024-07-03T07:10:17Z

"bencode" format is independent from BitTorrent so generic "bencode" decoder should not depend on it too.

They had deficiencies in their implementations. Not fully conform with the spec.

glassez · 2024-07-03T07:19:38Z

"bencode" format is independent from BitTorrent so generic "bencode" decoder should not depend on it too.

They had deficiencies in their implementations. Not fully conform with the spec.

It seems to be the same problem as with bencode editors. I couldn't find BitTorrent independent editor for Linux.

Piccirello · 2024-07-09T04:16:52Z

I ended up exploring how retrieved metadata would tie into the /add API, resulting in some changes to the /metadata API. Namely, the /metadata API now supports processing multiple sources at once. I've also made the necessary changes to the /add API to support downloading a torrent whose metadata has been previously retrieved via /metadata. This allows for adding the torrent with custom file priorities, which will enable a future PR to modify the WebUI's Add Torrent experience to mimic that of the GUI. PR description has been modified with the full changes.

src/webui/api/torrentscontroller.cpp

glassez · 2024-07-12T14:12:46Z

sources can be delineated by a (non-url encoded) comma

I suppose you're talking about percent-encoding, right?
I don't believe it is valid requirement. IIRC, in URL some of characters MUST be percent-encoded and other CAN be percent-encoded.

glassez · 2024-07-12T14:16:33Z

When metadata is requested for multiple torrents, a 202 will be returned if any of the torrents requires asycnhronous background work. This means that a 202 may include data for some of the request torrents.

It would look much simpler and more convenient if "metadata" endpoint accepted only single source.

Piccirello · 2024-07-12T17:04:32Z

sources can be delineated by a (non-url encoded) comma

I suppose you're talking about percent-encoding, right? I don't believe it is valid requirement. IIRC, in URL some of characters MUST be percent-encoded and other CAN be percent-encoded.

, is a reserved character and so it must be percent encoded.

It would look much simpler and more convenient if "metadata" endpoint accepted only single source.

This was my initial approach but I changed it because of how it will integrate with the Add Torrent dialogs in the WebUI. With proper documentation and status codes, I think this API will be easy for consumers/clients to understand.

glassez · 2024-07-12T18:21:31Z

, is a reserved character and so it must be percent encoded.

Then why do you suggest using it non-encoded?

I changed it because of how it will integrate with the Add Torrent dialogs in the WebUI.

Do "Add Torrent dialogs in the WebUI" really require API endpoint to allow accepting multiple sources at once?

With proper documentation and status codes, I think this API will be easy for consumers/clients to understand.

If "metadata" accepts only single source then 202 status would definitely mean that the response does not contain metadata, and 200 - on the contrary, that the metadata is available in the response.
If "metadata" accepts multiple sources then 202 status is ambiguous and you will still have to check the response body to understand which metadata is immediately available and which is not. In addition, it is possible that one of the sources is invalid. What should you do in this case? In single source approach you could just return error status...

src/webui/api/torrentscontroller.cpp

Piccirello · 2024-10-04T14:51:01Z

@qbittorrent/bug-handlers this PR has been in a ready state for several weeks. Can I get a thorough review and/or an approval?

glassez · 2024-10-04T15:52:54Z

@qbittorrent/bug-handlers this PR has been in a ready state for several weeks. Can I get a thorough review and/or an approval?

It's still on my to-do list. Unfortunately it's hard to review such changes using the mobile web version of GitHub, which I usually use to review the PRs. And there is still not enough time at the computer for it, since there are many other tasks.

Piccirello · 2024-10-04T21:00:34Z

@qbittorrent/bug-handlers this PR has been in a ready state for several weeks. Can I get a thorough review and/or an approval?

It's still on my to-do list. Unfortunately it's hard to review such changes using the mobile web version of GitHub, which I usually use to review the PRs. And there is still not enough time at the computer for it, since there are many other tasks.

Understood. You've already given several rounds of reviews, so I invite other folks to take a look too.

Chocobo1

I only commented on coding style.

src/webui/api/apicontroller.cpp

Chocobo1 · 2024-10-05T07:53:58Z

src/webui/api/torrentscontroller.h

+#include "base/bittorrent/torrentdescriptor.h"
+#include "base/bittorrent/torrentinfo.h"
+#include "base/net/downloadmanager.h"


Since you already forward declared them:

Suggested change

#include "base/bittorrent/torrentdescriptor.h"

#include "base/bittorrent/torrentinfo.h"

#include "base/net/downloadmanager.h"

So how about this?

src/webui/api/torrentscontroller.cpp

Chocobo1

Last two comments for coding style.

src/webui/api/apicontroller.h

src/webui/api/torrentscontroller.cpp

Piccirello · 2024-10-19T18:52:40Z

Any more feedback on this or is it ready for approvals?

glassez · 2024-10-19T20:17:55Z

I still haven't gotten to the detailed code review, but at the same time I had the opportunity to rethink its design.
IMO, this would eliminate some questionable aspects if it did not claim to have a general purpose, but would be presented in a more utilitarian sense as some kind of companion to the torrent addition function, that is, to expand the current torrents/add function and supplement it with the torrents/prepareAdd one.
What advantages could it have?

This could be the only function that accepts both links to torrents (magnet links) and existing files (as you originally wanted to do, but unlike that time, this behavior looks legitimate in this interpretation, since it does what it says).
There is no need to do anything doubtful if the requested torrent has already been added. In this case, torrents/prepareAdd naturally fails.

I will not insist (for subjective reasons). If someone explicitly approves of the current design, then so be it. Your responsibility.
Moreover, if someone (for example, @Chocobo1) finds it important to approve and merge it faster (without further code review), then you can do so. Maybe I'll come back to it later, when it's already been merged. Nothing prevents we from fixing the shortcomings of its implementation later (if any). The main thing in this matter is its basic design (and interface).

glassez · 2024-09-28T19:01:51Z

src/webui/api/apicontroller.h

@@ -67,6 +69,8 @@ class APIController : public ApplicationComponent<QObject>
    void setResult(const QJsonObject &result);
    void setResult(const QByteArray &result, const QString &mimeType = {}, const QString &filename = {});

+    void setStatus(APIStatus status);


Maybe just add another parameter to setResult?

void setResult(const QJsonObject &result, APIStatus status = APIStatus::Ok);

How would this work for cases where we don't want to send any data? For example, setting an explicit 204.

src/webui/api/torrentscontroller.h

glassez · 2024-09-28T19:47:52Z

src/webui/api/torrentscontroller.cpp

+    if (m_torrentMetadata.contains(infoHash))
+    {
+        const BitTorrent::TorrentDescriptor &torrentDescr = m_torrentMetadata[infoHash];
+        return torrentDescr.info().has_value() && torrentDescr.info().value().isValid();


I believe the following check is enough:

Suggested change

return torrentDescr.info().has_value() && torrentDescr.info().value().isValid();

return torrentDescr.info().has_value();

src/webui/api/torrentscontroller.cpp

glassez · 2024-10-05T09:20:35Z

src/webui/api/torrentscontroller.cpp

+            if (iter != m_torrentSource.end() && isMetadataDownloaded(iter.value()))
+            {
+                const BitTorrent::InfoHash &hash = iter.value();
+                const BitTorrent::TorrentDescriptor &torrentDescr = m_torrentMetadata[hash];


After using m_torrentSource.constFind, this should change to:

I don't understand what the relationship is... Why to copy it?

src/webui/api/torrentscontroller.cpp

Signed-off-by: Thomas Piccirello <thomas@piccirello.com>

Piccirello commented Jul 1, 2024

View reviewed changes

src/base/bittorrent/torrentinfo.h Outdated Show resolved Hide resolved

Piccirello commented Jul 1, 2024

View reviewed changes

src/webui/api/serialize/serialize_torrent.h Outdated Show resolved Hide resolved

Piccirello commented Jul 1, 2024

View reviewed changes

src/webui/api/torrentscontroller.cpp Outdated Show resolved Hide resolved

Piccirello commented Jul 1, 2024

View reviewed changes

src/webui/api/torrentscontroller.h Outdated Show resolved Hide resolved

glassez self-assigned this Jul 1, 2024

glassez added the WebAPI WebAPI-related issues/changes label Jul 1, 2024

glassez reviewed Jul 1, 2024

View reviewed changes

src/webui/api/torrentscontroller.cpp Outdated Show resolved Hide resolved

Piccirello force-pushed the metadata-api branch 2 times, most recently from cec83e2 to f857170 Compare July 1, 2024 18:38

Piccirello force-pushed the metadata-api branch 2 times, most recently from 1490741 to f736e62 Compare July 1, 2024 20:22

Piccirello force-pushed the metadata-api branch from f736e62 to fcbbac3 Compare July 9, 2024 04:14

github-advanced-security bot found potential problems Jul 9, 2024

View reviewed changes

src/webui/api/torrentscontroller.cpp Fixed Show resolved Hide resolved

Piccirello force-pushed the metadata-api branch from fcbbac3 to a7da584 Compare July 9, 2024 04:26

Piccirello mentioned this pull request Jul 9, 2024

Add WebAPI for managing torrent WebSeeds #21043

Merged

Piccirello marked this pull request as ready for review July 9, 2024 21:42

Piccirello requested a review from a team July 9, 2024 21:42

glassez reviewed Sep 28, 2024

View reviewed changes

src/webui/api/torrentscontroller.cpp Outdated Show resolved Hide resolved

Piccirello force-pushed the metadata-api branch 2 times, most recently from 778a776 to dee0d8c Compare September 30, 2024 12:34

Chocobo1 reviewed Oct 5, 2024

View reviewed changes

Piccirello force-pushed the metadata-api branch from dee0d8c to 6e68324 Compare October 12, 2024 00:26

Piccirello requested a review from Chocobo1 October 12, 2024 00:52

Chocobo1 reviewed Oct 12, 2024

View reviewed changes

src/webui/api/apicontroller.h Outdated Show resolved Hide resolved

src/webui/api/torrentscontroller.cpp Outdated Show resolved Hide resolved

Piccirello force-pushed the metadata-api branch from 6e68324 to 64d1af9 Compare October 12, 2024 14:08

Piccirello force-pushed the metadata-api branch from 64d1af9 to 59eb3bd Compare October 19, 2024 22:24

Piccirello mentioned this pull request Oct 19, 2024

Add new Add Torrent experience to WebUI #21645

Draft

glassez reviewed Oct 24, 2024

View reviewed changes

Piccirello force-pushed the metadata-api branch 9 times, most recently from b58e754 to 2e36fd9 Compare October 29, 2024 07:15

Piccirello added 4 commits October 31, 2024 01:06

Support sending custom status codes via WebAPI

e8520a5

Signed-off-by: Thomas Piccirello <thomas@piccirello.com>

Add WebAPI for fetching torrent metadata

729e795

Signed-off-by: Thomas Piccirello <thomas@piccirello.com>

Support downloading torrent from previously fetched metadata

60116a4

Signed-off-by: Thomas Piccirello <thomas@piccirello.com>

Add WebAPI for downloading torrent metadata

8127516

Signed-off-by: Thomas Piccirello <thomas@piccirello.com>

Piccirello force-pushed the metadata-api branch from 2e36fd9 to 8127516 Compare October 31, 2024 08:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add WebAPI for fetching torrent metadata #21015

Add WebAPI for fetching torrent metadata #21015

Piccirello commented Jul 1, 2024 •

edited

Loading

glassez left a comment

glassez commented Jul 1, 2024

Piccirello commented Jul 1, 2024

NikcN22 commented Jul 1, 2024

Chocobo1 commented Jul 2, 2024 •

edited

Loading

glassez commented Jul 3, 2024

Chocobo1 commented Jul 3, 2024 •

edited

Loading

glassez commented Jul 3, 2024

Piccirello commented Jul 9, 2024

glassez commented Jul 12, 2024

glassez commented Jul 12, 2024

Piccirello commented Jul 12, 2024

glassez commented Jul 12, 2024

Piccirello commented Oct 4, 2024

glassez commented Oct 4, 2024

Piccirello commented Oct 4, 2024

Chocobo1 left a comment

Chocobo1 Oct 5, 2024

glassez Oct 23, 2024

Chocobo1 left a comment

Piccirello commented Oct 19, 2024

glassez commented Oct 19, 2024

glassez Sep 28, 2024

Piccirello Oct 24, 2024

glassez Sep 28, 2024

glassez Oct 5, 2024

	#include "base/bittorrent/torrentdescriptor.h"
	#include "base/bittorrent/torrentinfo.h"
	#include "base/net/downloadmanager.h"

	return torrentDescr.info().has_value() && torrentDescr.info().value().isValid();
	return torrentDescr.info().has_value();

Add WebAPI for fetching torrent metadata #21015

Are you sure you want to change the base?

Add WebAPI for fetching torrent metadata #21015

Conversation

Piccirello commented Jul 1, 2024 • edited Loading

/fetchMetadata API

HTTP request

HTTP response

/parseMetadata API

HTTP request

HTTP response

/add API

Alternatives:

Future work:

glassez left a comment

Choose a reason for hiding this comment

glassez commented Jul 1, 2024

Piccirello commented Jul 1, 2024

NikcN22 commented Jul 1, 2024

Chocobo1 commented Jul 2, 2024 • edited Loading

glassez commented Jul 3, 2024

Chocobo1 commented Jul 3, 2024 • edited Loading

glassez commented Jul 3, 2024

Piccirello commented Jul 9, 2024

glassez commented Jul 12, 2024

glassez commented Jul 12, 2024

Piccirello commented Jul 12, 2024

glassez commented Jul 12, 2024

Piccirello commented Oct 4, 2024

glassez commented Oct 4, 2024

Piccirello commented Oct 4, 2024

Chocobo1 left a comment

Choose a reason for hiding this comment

Chocobo1 Oct 5, 2024

Choose a reason for hiding this comment

glassez Oct 23, 2024

Choose a reason for hiding this comment

Chocobo1 left a comment

Choose a reason for hiding this comment

Piccirello commented Oct 19, 2024

glassez commented Oct 19, 2024

glassez Sep 28, 2024

Choose a reason for hiding this comment

Piccirello Oct 24, 2024

Choose a reason for hiding this comment

glassez Sep 28, 2024

Choose a reason for hiding this comment

glassez Oct 5, 2024

Choose a reason for hiding this comment

Piccirello commented Jul 1, 2024 •

edited

Loading

`/fetchMetadata` API

`/parseMetadata` API

`/add` API

Chocobo1 commented Jul 2, 2024 •

edited

Loading

Chocobo1 commented Jul 3, 2024 •

edited

Loading