DM-45211: Update shared dataset policies #692
Conversation
usdf/datasets.rst
Outdated
|
|
||
| All newly-added datasets, including external datasets, must follow the guidelines for supplying a :ref:`README <readme_guidelines_usdf>` file. Updates to the readme should be reviewed on subsequent Jira tickets. | ||
|
|
||
| Requests for new shared directories for should be sent to Kian-Tat Lim. Users are often given ownership of the shared directory and are responsible for setting appropriate permissions. |
There was a problem hiding this comment.
| Requests for new shared directories for should be sent to Kian-Tat Lim. Users are often given ownership of the shared directory and are responsible for setting appropriate permissions. | |
| Requests for new shared directories for should be sent to ``#ops-usdf-data-curation`` in the Rubin Observatory Slack. Members of the ``rubinmgr`` group will handle these, including having quotas applied. Requesting users are often given initial ownership of the shared directory and are responsible for setting appropriate permissions. If the shared dataset needs central curation, ownership may be set to ``rubinmgr`` after it is initially populated. |
There was a problem hiding this comment.
Are you going to create that channel? I don't see it listed, unless it's private.
usdf/datasets.rst
Outdated
|
|
||
| Policy | ||
| ====== | ||
| New shared datasets should be added to ``/shared``, following policies to be detailed in `DM-31704 <https://rubinobs.atlassian.net/browse/DM-31704>`_. Additions or modifications to any dataset that is to be used in regular (re-)processing must have a corresponding RFC. The RFC should contain information about: |
There was a problem hiding this comment.
It's not exactly clear from this whether new shared datasets need an RFC. We should say what the current status is, even if the policy is up in the air. If we do require an RFC, then we should make clear that the request for space allocation below is after the RFC has been adopted.
There was a problem hiding this comment.
See the new line. I figure anything getting ingested must have an RFC, and anything being used for regular processing but not in any butler (I'm not sure what that would be, but maybe some camera/summit stuff?) ought to be too. Anything else I suppose doesn't need an RFC, although I'm not sure what the leaves. I guess common git repos like testdata_*?
| @@ -77,67 +93,19 @@ Here is a template for what each refcat's readme should contain: | |||
| The configuration that was used to ingest the data is included in this | |||
| directory as `IngestIndexedReferenceTask.py`. | |||
There was a problem hiding this comment.
We can actually do something like the immutability/sharing below, as long as the modification access is coming from the interactive or batch nodes (not Kubernetes/notebooks). But we can put that back in when it's better-defined.
There was a problem hiding this comment.
Isn't that basically what's described with setting permissions above?
There was a problem hiding this comment.
Changing ownership is kind of a brute-force means of doing this; we can do something a bit more sophisticated for "temporary unlock to allow modification" or even "permanently allow curation by this group of people other than rubinmgr".
taranu
force-pushed
the
tickets/DM-45211
branch
from
d63599c
to
76eaf93
Compare
usdf/datasets.rst
Outdated
|
|
||
| All newly-added datasets, including external datasets, must follow the guidelines for supplying a :ref:`README <readme_guidelines_usdf>` file. Updates to the readme should be reviewed on subsequent Jira tickets. | ||
|
|
||
| Requests for new shared directories for should be sent to ``#ops-usdf-data-curation`` in the Rubin Observatory Slack (lsstc). |
There was a problem hiding this comment.
Let's just make this usdf-help@slac.stanford.edu for now.
taranu
force-pushed
the
tickets/DM-45211
branch
from
76eaf93
to
1f98c5c
Compare
usdf/datasets.rst
Outdated
|
|
||
| All newly-added datasets, including external datasets, must follow the guidelines for supplying a :ref:`README <readme_guidelines_usdf>` file. Updates to the readme should be reviewed on subsequent Jira tickets. | ||
|
|
||
| Requests for new shared directories for should be emailed to ``usdf-help@slac.stanford.edu``. |
There was a problem hiding this comment.
| Requests for new shared directories for should be emailed to ``usdf-help@slac.stanford.edu``. | |
| Requests for new shared directories should be emailed to ``usdf-help@slac.stanford.edu``. |
taranu
force-pushed
the
tickets/DM-45211
branch
2 times, most recently
from
4e58a83
to
3a80a2a
Compare
taranu
force-pushed
the
tickets/DM-45211
branch
from
3a80a2a
to
07ca0d4
Compare
No description provided.