AWS Sagemaker (SNMG & MNMG)

[mmccarty]

[skirui-source]

Experiencing the following blockers:

• [ RESOLVED ] The notebook examples in the provided repo is 2+ years old; the cuml and cudf API have drastically changed ever since. For instance, some of the hyperparameteres for cuml RandomForestClassifer have been deprecated -- had to exclude them

• [ RESOLVED ] sagemaker-containers (package that allows rapids container to be compatible with SageMaker) has also been deprecated , and is replaced with sagemaker-training and sagemaker-inference toolkits

• Lots of dependency conflicts btw the latest rapids release v22.10 and sagemaker-training v3.4 (conflicting version requirements for some packages: cuda-python, cupy-cuda, protobuf, botocore)

• re AWS configuration, I don't have access to a permanent secret access key/ID --- when testing locally, i have to keep generating temporary access tokens that expire in an hour-- keeps timing out while debugging is in-progress

• [ RESOLVED ] Configuring SageMaker's CreateTrainingJob API to use images from private docker registry for training (ie. rapids image from DockerHub in this case)
  ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: TrainingImageConfig with TrainingRepositoryAccessMode set to VPC must be provided when using a training image from a private Docker registry. Please provideTrainingImageConfig and TrainingRepositoryAccessMode set to VPC when using a training image from a private Docker registry.

• [ RESOLVED ] Having trouble with creating ECR repo due to the following error:
  (AccessDeniedException) when calling the CreateRepository operation: User: arn:aws:sts::561241433344:assumed-role/AmazonSageMakerServiceCatalogProductsUseRole/SageMaker is not authorized to perform: ecr:CreateRepository on resource: arn:aws:ecr:us-west-1:561241433344:repository/test-sagemaker-rapids-skirui because no identity-based policy allows the ecr:CreateRepository action

[skirui-source]

Seeing this error with rapids_estimator.fit({'dataset': s3_data_dir}), blocking a completed training job:

I 2022-11-16 03:05:01.454 ServerApp] jupyterlab | extension was successfully loaded.
[W 2022-11-16 03:05:01.454 ServerApp] jupyterlab_nvdashboard | extension failed loading with message: 'NoneType' object is not callable
[E 2022-11-16 03:05:01.454 ServerApp] jupyterlab_nvdashboard | stack trace
    Traceback (most recent call last):
      File "/opt/conda/envs/rapids/lib/python3.9/site-packages/jupyter_server/extension/manager.py", line 355, in load_extension
        extension.load_all_points(self.serverapp)
      File "/opt/conda/envs/rapids/lib/python3.9/site-packages/jupyter_server/extension/manager.py", line 229, in load_all_points
        return [self.load_point(point_name, serverapp) for point_name in self.extension_points]
      File "/opt/conda/envs/rapids/lib/python3.9/site-packages/jupyter_server/extension/manager.py", line 229, in <listcomp>
        return [self.load_point(point_name, serverapp) for point_name in self.extension_points]
      File "/opt/conda/envs/rapids/lib/python3.9/site-packages/jupyter_server/extension/manager.py" , line 222, in load_point
        return point.load(serverapp)
      File "/opt/conda/envs/rapids/lib/python3.9/site-packages/jupyter_server/extension/manager.py", line 148, in load
        return loader(serverapp)
    TypeError: 'NoneType' object is not callable

[jacobtomlinson]

Hrm that error is related to nvdashboard which I don't think is relevant to the .fit call. Is there any other output or logs?

[skirui-source]

The training job fails with this output:

2022-11-22 06:37:45 Starting - Starting the training job...
2022-11-22 06:38:08 Starting - Preparing the instances for trainingProfilerReport-1669099065: InProgress
...............
2022-11-22 06:40:41 Downloading - Downloading input data....................Running SageMaker HPO entrypoint.
@ entrypoint -> launching training script 
2022-11-22 06:43:58,978     INFO hpo_log 
parsing configuration from environment settings...
2022-11-22 06:43:58,978     INFO hpo_log   Dataset: Airline
  Compute: single-GPU
  Algorithm: XGBoost
  CV_folds: 3
2022-11-22 06:43:58,978     INFO hpo_log parsing model hyperparameters from command line arguments...log
2022-11-22 06:43:58,979     INFO hpo_log {    'gamma': 0.0,
     'lambda': 1,
     'learning_rate': 0.3,
     'max_depth': 5,
     'num_boost_round': 10,
     'objective': 'binary:logistic',
     'random_state': 0,
     'seed': 0,
     'tree_method': 'gpu_hist',
     'verbosity': 0}
2022-11-22 06:43:58,980     INFO hpo_log CSV input files detected
['/opt/ml/input/data/training/HIGGS.csv']
2022-11-22 06:43:58,980     INFO hpo_log detected 1 files as input
Traceback (most recent call last):
  File "cuda/_cuda/ccuda.pyx", line 3671, in cuda._cuda.ccuda._cuInit
  File "cuda/_cuda/ccuda.pyx", line 435, in cuda._cuda.ccuda.cuPythonInit
RuntimeError: Failed to dlopen libcuda.so
Exception ignored in: 'cuda._lib.ccudart.utils.cudaPythonGlobal.lazyInitGlobal'
Traceback (most recent call last):
  File "cuda/_cuda/ccuda.pyx", line 3671, in cuda._cuda.ccuda._cuInit
  File "cuda/_cuda/ccuda.pyx", line 435, in cuda._cuda.ccuda.cuPythonInit
RuntimeError: Failed to dlopen libcuda.so
/opt/rapids_cloudml/entrypoint.sh: line 45:    15 Segmentation fault      (core dumped) python train.py

2022-11-22 06:44:14 Training - Training image download completed. Training in progress.
2022-11-22 06:44:14 Uploading - Uploading generated training model
2022-11-22 06:44:14 Failed - Training job failed

[skirui-source]

^^ Here is the complete Traceback message from the output above:

---------------------------------------------------------------------------
UnexpectedStatusException                 Traceback (most recent call last)
<timed eval> in <module>

~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/workflow/pipeline_context.py in wrapper(*args, **kwargs)
    270             return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, **kwargs)
    271 
--> 272         return run_func(*args, **kwargs)
    273 
    274     return wrapper

~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
   1126         self.jobs.append(self.latest_training_job)
   1127         if wait:
-> 1128             self.latest_training_job.wait(logs=logs)
   1129 
   1130     def _compilation_job_name(self):

~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/estimator.py in wait(self, logs)
   2240         # If logs are requested, call logs_for_jobs.
   2241         if logs != "None":
-> 2242             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
   2243         else:
   2244             self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
   3891 
   3892         if wait:
-> 3893             self._check_job_status(job_name, description, "TrainingJobStatus")
   3894             if dot:
   3895                 print()

~/anaconda3/envs/python3/lib/python3.8/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
   3429                     actual_status=status,
   3430                 )
-> 3431             raise exceptions.UnexpectedStatusException(
   3432                 message=message,
   3433                 allowed_statuses=["Completed", "Stopped"],

UnexpectedStatusException: Error for Training job sagemaker-rapids-2022-11-22-06-37-45-441: Failed. Reason: AlgorithmError: , exit code: 139

[jacobtomlinson]

The RuntimeError: Failed to dlopen libcuda.so error suggests that the VMs the estimator spins up have some driver or container runtime issue. You might want to check in with the cuml folks in case they have some thoughts.

[skirui-source]

Seems like the segfault error is due to outdated version of cuda-python dependency, following this PR, ---- waiting on RAPIDS v22.12 to be pushed to Docker Hub for testing

[jacobtomlinson]

Are you able to test with the current nightly?