RFE: Option in pbench-move-results command to output results direct link

[pravins]

Is there an existing issue for this?

• I have searched the existing issues

Description

As a user, I want a direct link from where i can download results uploaded to the server so i can do further analysis on the uploaded results to find required information.

As of now, we do not have any such option and end user need to construct direct link from config-file+ results + user + benchmark directory

To Reproduce

Run pbench-move-results

Actual Results

It does not output direct link for results.

Expected Results

Some option to get direct link to the uploaded results.

Additional information

None

[dbutenhof]

It's worth noting here that with "ownership" and "curation" this becomes a bit more nuanced than it was for 0.69. Yes, if one knew the benchmark name, the config string, and the run timestamp, plus the hostname and port of the Pbench Server, one could generate a URL to pull the tarball or any artifact from the tarball via unauthenticated Apache file sharing.

We're moving away from all that. Unless the dataset is "public", only the authenticated user will be able to do this. And, of course, we're changing the structure of the URL as it'll go through http[s]://<hostname>:port/api/v1/inventory/<resource_id> with authorization via the Pbench Server APIs. (The resource_id is just the tarball MD5 as this time, but we're also talking about switching to SHA256 or something, and we don't want to lock ourselves in too tightly to representation.)

So given pbench-results-move --access=public --token=<token>, then, yes, we could generate a URL to access to the tarball. One possibility would be to return a Location: <URI> response header. This is commonly a REST GET URI to return the full REST resource just created. I'm not entirely sure what that would mean as we don't really visualize our datasets as a "full resource" at this time... so, possibly, we could "fudge" by returning the inventory API for the tarball. Knowing relative paths within the tarball, you could easily append that path to get individual artifacts, and that might be useful.

But, again, unless you use --access=public, the GET will require authentication as the owning user.

[pravins]

Agree.
With data protection perspective, we have to make sure, user will only able to access his own tarballs or public tarballs.

[jhutar]

Hello. I'm uploading with:

pbench-move-results --user jhutar --prefix spc

and data seems to be publicly available

[pravins]

@jhutar New server is still not deployed. With new implementation of user notion, user has to generate a token and use that token to upload the data.
Once its uploaded, only authenticated user will able to see the data. User will be able to mark their data public, so anyone can access it using direct link.
With present 0.69 server, all uploaded data is public.

[dbutenhof]

pbench-move-results --user jhutar --prefix spc

As Pravin said, the --user from 0.69 has nothing to do with dataset ownership: it's effectively metadata. (In fact, if you're accessing results on 0.69 you'll only find any trace of this if you open metadata.log; what it does is control a link under the users results tree.) When you move to Pbench Server 1.0, you'll specify an API authentication token to pbench-results-move --token which identifies the user which will own the data. (You'll also be able to specify --access public if you want your data to initially be accessible to everyone rather than private.)

Also, note that --prefix goes away completely: it's not supported by pbench-results-move.

[webbnh]

@pravins, I'm not sure that this is a direction that we want to take.

My understanding of the model is that pbench-results-move "simply" takes an existing file tree from the current system and uploads it to the Pbench Server. Thus, the user already has access to their result on the local system, and doesn't need a URL from the Server in order to find it (locally). If a user wants to download a result somewhere else, they can (or will be able to) do that via the Pbench Dashboard (which will handle all of the authentication and access issues). I don't see how it will improve things to add generation of a URL to pbench-results-move.

Now, if the issue is that a user is having trouble locating their result locally, that seems like something we might consider creating a command for. (Such a task should be simple, well-bounded, and not subject to implementation details of the Pbench Server.)

[dbutenhof]

A little more nuanced. First, the default behavior of pbench-results-move deletes the local tarball after successfully uploading it. (Although there is a --no-delete approximating the behavior of the old pbench-results-copy.)

We also don't currently have dashboard mechanism to return dataset inventory. (Either individual files, which might be displayed inline, or the full tarball, which would presumably be downloaded to the browser's local filesystem.)

There's no reason that anyone can't just use curl or equivalent to pull files from a Pbench Server. It might be perfectly reasonable for PUT, for example, to return a Location response header with the http://<host>/api/v1/inventory/<resource_id> URL; which, if used directly on a GET, would return the entire dataset tarball. And, if we did that, there's no reason that pbench-results-move couldn't display that header for whatever good it might do the user.

So ultimately, whether or not we do this, and however we prioritize it, I don't think there's anything inherently wrong with the request.

[webbnh]

Diving into the nuance, the RFE requests the ability for a user to "do further analysis on the uploaded results to find required information." There are two ways for the user to engage with that: either locally or on some other system.

If the user wants to work locally, then there is no reason to involve pbench-results-move or the Pbench Server -- the user already has the results (assuming that pbench-results-move doesn't remove them) -- and the issue devolves to whether the user can readily find them (which I think it makes sense to help with, if needed).

However, if the user wants to work on another system, then having the result's URL made available on the system where they ran pbench-results-move is somewhat inconvenient, since that's not where they want the results to be downloaded to. Also, contrary to your assertion, a user won't necessarily be able to "just use curl", because there is (or will, or should be!) an authentication and authorization process involved in accessing the result on the Pbench Server.

Thus, I think it would be better to refer the user to the Pbench Dashboard:

• it will handle the authentication and authorization details
• it can collaborate with the Pbench Server to hide any details around the URI, access, and download (particularly as these change over time)
• and, for specific analysis tasks, it will evolve to offer support which frees the users from having to do it themselves.

That is, I think that using the Pbench Dashboard (in collaboration with the Pbench Server) should (and will eventually) be better than accessing the result directly...and so I don't see any value to trying to help the user to avoid using the Dashboard.

[dbutenhof]

Yes, obviously one needs an authorization header; but once we adopt the "API token" mechanism that's not onerous. We don't want to position our dashboard as the only gateway into the server APIs, for lots of reasons, including the fact that we don't expect our dashboard to be able to do everything someone might want. Making it easy for them to get at the raw archived data provides a hook for all sorts of expansions.

This is all way beyond "exposing the URI in pbench-move-results" (and, by extension, our PUT API), but all I'm arguing is that I can see viable uses.

[webbnh]

We don't want to position our dashboard as the only gateway into the server APIs

I concur -- we want to facilitate the development of other client programs, as well. But, equally, I don't think we should expect our (human) end-users to be using the Pbench Server APIs directly (e.g., via curl), and so handing out URLs (especially if they are returned on "the wrong node") isn't likely to produce as good a UX as the Dashboard would/should (and, using the Dashboard will facilitate covering the arcana like how to provide the API token, etc. in the access request to the Server).

we don't expect our dashboard to be able to do everything someone might want. Making it easy for them to get at the raw archived data provides a hook for all sorts of expansions.

No, we don't...but wouldn't it be cool if it did?! And, if the Dashboard is the first point of contact, then we're much more likely to find out when it doesn't do something which a user needs. (If the default is that we give users a URL for their result, then, instead of getting RFE's for the Dashboard, we'll be getting requests for help making curl work. 😛)

I would really like to see "access to the result" be a Pbench Server API request, e.g., for timeseries data, possibly facilitated by the Dashboard, and not a process where the user pulls the tarball, unpacks it, and then does something like manually ingest the guts into Elasticsearch.... 😢

[jhutar]

Hello @webbnh . I need that URL because my use-case is "run the test, upload results and cleanup" and I need to add that URL to the metadata I store.