-
Notifications
You must be signed in to change notification settings - Fork 4
feat: new job wrapper DIRAC Integration #86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
| if os.getenv("DIRAC_PROTO_LOCAL") == "1": | ||
| self._sandbox_store_client = MockSandboxStoreClient() | ||
| else: | ||
| DIRAC.initialize() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's okay for now but just for your information:
- in the current DIRAC code, the initialization is performed in the
JobWrapperTemplate: https://github.com/DIRACGrid/DIRAC/blob/11bc43dda994b41832aa29237400f9c2b16c0f2f/src/DIRAC/WorkloadManagementSystem/JobWrapper/JobWrapperTemplate.py#L24 - at some point we want to use the diracx endpoints (Use diracx-api jobs.py module to interact with the sandbox store endpoint (instead of the one from DIRAC) #73). In that case you would need to replace this
initializeby providing theDIRACX_URLenvironment variable in thejob_wrapper_templateI think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems Script.parseCommandLine() needs to be run before importing any DIRAC component. So the job wrapper template would look like this. If it's ok for you, then it can replace the DIRAC.initialize() without any problem.
from cwl_utils.parser import load_document_by_uri
from cwl_utils.parser.cwl_v1_2_utils import load_inputfile
from DIRAC.Core.Base.Script import Script # type: ignore[import-untyped]
from ruamel.yaml import YAML
if os.getenv("DIRAC_PROTO_LOCAL") != "1":
Script.parseCommandLine()
from dirac_cwl_proto.job.job_wrapper import JobWrapper
from dirac_cwl_proto.submission_models import JobModel
def main():
"""Execute the job wrapper for a given job model."""
if len(sys.argv) != 2:
...It also may change the default logging configuration a bit, but it shouldn't be too much of an issue.
For the DIRACX_URL env var I'm not yet sure how to get the url but I guess we will see when it's needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes that solution sound good indeed!
For the DIRACX_URL, indeed we can see that later when we will use the diracx API instead of the DIRAC one.
|
|
Fixes problems related to the execution of the job wrapper on pilots (i.e. cwl ids link to local path).
Also initialize DIRAC in the job wrapper as it's needed to access Dirac Resources (i.e. /Resources/FileCatalogs).
See #44