Sampler failing when setting skip_transpilation to True

[luciacuervovalor]

Describe the bug

When setting skip_transpilation to True in transpilation_settings in the run method of the Sampler, jobs fail randomly. For example, evaluating a 6 qubit Bell state circuit with no parameters failed twice in 1000 trials (job ids of the jobs which failed were cdd3msvhc7fs715q3t10 and cdd3s52auedvlnhhqc0g). When a job fails, an AlgorithmError is raised which prevents the program from continuing.

Steps to reproduce

from qiskit import IBMQ
IBMQ.load_account()
from qiskit_ibm_runtime import Session, Sampler
from qiskit.circuit import QuantumCircuit
            
qc = QuantumCircuit(6)
for i in range(6):
    qc.h(i)
for i in range(0,6,2):
    qc.cx(i,i+1)
qc.measure_all()

with Session(backend='ibmq_qasm_simulator') as session:
    sampler = Sampler(session=session)
    for iteration in range(1000):
        if iteration%25==0:
            print(f"Iteration {iteration} completed")
        job = sampler.run(
                            [qc], [], 
                            run_options={"shots": 1024}, 
                            transpilation_settings={"skip_transpilation": True}
                          )
        result = job.result()

Expected behavior

Evaluates all 1000 circuits without failure

Suggested solutions

To prevent the program from breaking, I added a patch which resends a job when it randomly fails, and also prints the job id, in case I want to view the error log. This doesn't solve the core problem but at least allows the program to continue. The patch was:

class RetrySampler(Sampler):
    
    def __init__(self, *args, max_retries: int = 5, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.max_retries = max_retries
    
    def run(self, circuits, parameter_values, **kwargs):
        result = None
        for i in range(self.max_retries):
            try:
                job = super().run(circuits, parameter_values, **kwargs)
                result = job.result()
                if result is not None:
                    return job
            except:
                if i != self.max_retries-1:
                    print(f"Sampler job with ID {job.job_id} failed... Starting trial number {i+2} for this iteration")
                pass
        if result is None:
            raise RuntimeError(f"Runtime job failed! Maximum number of retries {self.max_retries} exceeded")

where then we replace Sampler with RetrySampler in the above code.

Additional Information

• qiskit-ibm-runtime version: 0.7.0rc3
• Python version: 3.10.4
• Operating system: macOS Monterey 12.6 M1 2020

[luciacuervovalor]

Update

This issue can also occur when skip_transpilation is set to False. Further, other errors may randomly arise. Below is the updated version of the patch which catches these other errors.

import signal, time

from qiskit_ibm_runtime import Sampler, Session
from qiskit.providers import JobStatus

def timeout_handler(signum, frame):
    raise Exception('Iteration timed out')

class RuntimeRetrySampler(Sampler):
    """RuntimeRetrySampler class.
    
    This class inherits from Qiskit IBM Runtime's Sampler and overwrites its run method such that it retries calling it
    a maximum of 'max_retries' consecutive times, if it encounters one of the following randomly occuring errors:
    
    * A Sampler error (in this case "Job.ERROR" is printed, and the job is cancelled automatically)
    * A timeout error where the job either remains running or completes but does not return anything, for a time larger 
      than 'timeout' (in this case the job is cancelled by the patch and "Job.CANCELLED" is printed)
    * A creation error, where the job fails to be created because connection is lost between the runtime server and the
      quantum computer (in this case "Failed to create job." is printed). If this error occurs, the patch connects the user
      to a new Session (to be handled with care! also, this will unfortunately put the next job in the queue). 
    """
    
    def __init__(self, *args, backend: str, max_retries: int = 5, timeout: int = 3600, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.max_retries = max_retries
        self.timeout = timeout
        self.backend = backend
        signal.signal(signal.SIGALRM, timeout_handler)
    
    def run(self, circuits, parameter_values, **kwargs):
        result = None
        for i in range(self.max_retries):
            try:
                job = super().run(circuits, parameter_values, **kwargs)
                while job.status() in [JobStatus.INITIALIZING, JobStatus.QUEUED, JobStatus.VALIDATING]:
                    time.sleep(5) # Check every 5 seconds whether job status has changed
                signal.alarm(self.timeout) # Once job starts running, set timeout to 1 hour by default
                result = job.result()
                if result is not None:
                    signal.alarm(0) # reset timer
                    return job
            except Exception as e:
                if 'job' in locals(): # Sometimes job fails to create
                    if job.status() not in [JobStatus.DONE, JobStatus.ERROR, JobStatus.CANCELLED]:
                        job.cancel()
                    if i != self.max_retries-1:
                        print("\nSomething went wrong...")
                        print(f"Job ID: {job.job_id}. Job status: {job.status()}.")
                        print(f"\n\nERROR MESSAGE:\n{e}\n\n")
                        print(f"Starting trial number {i+2}...\n")   
                else:
                    if i != self.max_retries-1:
                        print("\nSomething went wrong...")
                        print(f"Failed to create job. ")
                        print(f"\n\nERROR MESSAGE:\n{e}\n\n")
                        print(f"Creating new session...\n") 
                        self._session = Session(backend=self.backend)
                signal.alarm(0) # reset timer
        if result is None:
            raise RuntimeError(f"Program failed! Maximum number of retries ({self.max_retries}) exceeded")

[luciacuervovalor]

Update

Here is the patch updated such that it works for the Estimator too (i.e. the retry logic here is common to both the Sampler and the Estimator)

import signal, time

from qiskit_ibm_runtime import Sampler, Estimator, Session
from qiskit.providers import JobStatus

def timeout_handler(signum, frame):
    raise Exception('Iteration timed out')

class RetryPrimitiveMixin:
    """RetryPrimitive class.
    
    This class inherits from Qiskit IBM Runtime's Primitives and overwrites its run method such that it retries calling it
    a maximum of 'max_retries' consecutive times, if it encounters one of the following randomly occuring errors:
    
    * A Primitive error (in this case "Job.ERROR" is printed, and the job is cancelled automatically)
    * A timeout error where the job either remains running or completes but does not return anything, for a time larger 
      than 'timeout' (in this case the job is cancelled by the patch and "Job.CANCELLED" is printed)
    * A creation error, where the job fails to be created because connection is lost between the runtime server and the
      quantum computer (in this case "Failed to create job." is printed). If this error occurs, the patch connects the user
      to a new Session (to be handled with care! also, this will unfortunately put the next job in the queue). 
    """
    
    def __init__(self, *args, max_retries: int = 5, timeout: int = 3600, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.max_retries = max_retries
        self.timeout = timeout
        self.backend = super().session._backend
        signal.signal(signal.SIGALRM, timeout_handler)
        
    def run(self, *args, **kwargs):
        result = None
        for i in range(self.max_retries):
            try:
                job = super().run(*args, **kwargs)
                while job.status() in [JobStatus.INITIALIZING, JobStatus.QUEUED, JobStatus.VALIDATING]:
                    time.sleep(5) # Check every 5 seconds whether job status has changed
                signal.alarm(self.timeout) # Once job starts running, set timeout to 1 hour by default
                result = job.result()
                if result is not None:
                    signal.alarm(0) # Reset timer
                    return job
            except Exception as e:
                signal.alarm(0) # Reset timer
                print("\nSomething went wrong...")
                print(f"\n\nERROR MESSAGE:\n{e}\n\n")
                if 'job' in locals(): # Sometimes job fails to create
                    print(f"Job ID: {job.job_id}. Job status: {job.status()}.")
                    if job.status() not in [JobStatus.DONE, JobStatus.ERROR, JobStatus.CANCELLED]:
                        job.cancel()
                else:
                    print("Failed to create job.")
                    try:
                        super().session.close()
                        print("Current session was closed.")
                    except:
                        print("Current session could not be closed. Will leave it to close automatically.")
                    print(f"Creating new session...\n")
                    self._session = Session(backend=self.backend)
                print(f"Starting trial number {i+2}...\n")
                signal.alarm(0) # Reset timer
        if result is None:
            raise RuntimeError(f"Program failed! Maximum number of retries ({self.max_retries}) exceeded")
            
class RetrySampler(RetryPrimitiveMixin, Sampler):
    pass

class RetryEstimator(RetryPrimitiveMixin, Estimator):
    pass

[jyu00]

I'm going to close this issue since it doesn't seem to be related to skip_transpilation. The stability of Qiskit Runtime service has improved greatly since 1H of 2023.

@luciacuervovalor if you encounter other failures, please create new issues or contact support.