Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARM] Fix: az bicep install: Use CA bundle environment variables if set #26013

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

wwmoraes
Copy link
Contributor

@wwmoraes wwmoraes commented Mar 31, 2023

Related command
az bicep install

Description
Added a fallback order equivalent to the one found in the requests package. Also removed the unneeded setdefaults. See #21807 for history, and the recent comment done by @jiasli

closes #26007

Testing Guide
A clean environment should use the certifi CA bundle path:

env -i az bicep install

If either REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE is set, then it should be picked up:

env -i REQUESTS_CA_BUNDLE=/path/to/cacert1.pem az bicep install
env -i CURL_CA_BUNDLE=/path/to/cacert1.pem az bicep install

If both REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE are set, then it should use REQUESTS_CA_BUNDLE (this is by definition of the requests package).

History Notes

[ARM] az bicep install: Use CA bundle environment variables if set


This checklist is used to make sure that common guidelines for a pull request are followed.

@azure-client-tools-bot-prd
Copy link

azure-client-tools-bot-prd bot commented Mar 31, 2023

❌AzureCLI-FullTest
️✔️acr
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️acs
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️advisor
️✔️latest
️✔️3.11
️✔️3.9
️✔️ams
️✔️latest
️✔️3.11
️✔️3.9
️✔️apim
️✔️latest
️✔️3.11
️✔️3.9
️✔️appconfig
️✔️latest
️✔️3.11
️✔️3.9
️✔️appservice
️✔️latest
️✔️3.11
️✔️3.9
️✔️aro
️✔️latest
️✔️3.11
️✔️3.9
️✔️backup
️✔️latest
️✔️3.11
️✔️3.9
️✔️batch
️✔️latest
️✔️3.11
️✔️3.9
️✔️batchai
️✔️latest
️✔️3.11
️✔️3.9
️✔️billing
️✔️latest
️✔️3.11
️✔️3.9
️✔️botservice
️✔️latest
️✔️3.11
️✔️3.9
️✔️cdn
️✔️latest
️✔️3.11
️✔️3.9
️✔️cloud
️✔️latest
️✔️3.11
️✔️3.9
️✔️cognitiveservices
️✔️latest
️✔️3.11
️✔️3.9
️✔️config
️✔️latest
️✔️3.11
️✔️3.9
️✔️configure
️✔️latest
️✔️3.11
️✔️3.9
️✔️consumption
️✔️latest
️✔️3.11
️✔️3.9
️✔️container
️✔️latest
️✔️3.11
️✔️3.9
️✔️containerapp
️✔️latest
️✔️3.11
️✔️3.9
️✔️core
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️cosmosdb
️✔️latest
️✔️3.11
️✔️3.9
️✔️databoxedge
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️dla
️✔️latest
️✔️3.11
️✔️3.9
️✔️dls
️✔️latest
️✔️3.11
️✔️3.9
️✔️dms
️✔️latest
️✔️3.11
️✔️3.9
️✔️eventgrid
️✔️latest
️✔️3.11
️✔️3.9
️✔️eventhubs
️✔️latest
️✔️3.11
️✔️3.9
️✔️feedback
️✔️latest
️✔️3.11
️✔️3.9
️✔️find
️✔️latest
️✔️3.11
️✔️3.9
️✔️hdinsight
️✔️latest
️✔️3.11
️✔️3.9
️✔️identity
️✔️latest
️✔️3.11
️✔️3.9
️✔️iot
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️keyvault
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️kusto
️✔️latest
️✔️3.11
️✔️3.9
️✔️lab
️✔️latest
️✔️3.11
️✔️3.9
️✔️managedservices
️✔️latest
️✔️3.11
️✔️3.9
️✔️maps
️✔️latest
️✔️3.11
️✔️3.9
️✔️marketplaceordering
️✔️latest
️✔️3.11
️✔️3.9
️✔️monitor
️✔️latest
️✔️3.11
️✔️3.9
️✔️mysql
️✔️latest
️✔️3.11
️✔️3.9
️✔️netappfiles
️✔️latest
️✔️3.11
️✔️3.9
️✔️network
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️policyinsights
️✔️latest
️✔️3.11
️✔️3.9
️✔️privatedns
️✔️latest
️✔️3.11
️✔️3.9
️✔️profile
️✔️latest
️✔️3.11
️✔️3.9
️✔️rdbms
️✔️latest
️✔️3.11
️✔️3.9
️✔️redis
️✔️latest
️✔️3.11
️✔️3.9
️✔️relay
️✔️latest
️✔️3.11
️✔️3.9
❌resource
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
❌latest
❌3.11
Type Test Case Error Message Line
Failed test_use_bicep_cli_from_path_false_after_install self = <azure.cli.command_modules.resource.tests.latest.test_resource_bicep.TestBicep testMethod=test_use_bicep_cli_from_path_false_after_install>
isfile_stub = <function request at 0x7fe2d8781f80>
dirname_stub = <MagicMock name='isfile' id='140612264686800'>
exists_stub = <MagicMock name='dirname' id='140612271070736'>
mock_requests_get = <MagicMock name='exists' id='140612271070800'>
open_stub = <MagicMock name='open' id='140612270917264'>
buffered_writer_stub = <MagicMock name='BufferedWriter' id='140612266147344'>
stat_stub = <MagicMock name='stat' id='140612264662672'>
chmod_stub = <MagicMock name='chmod' id='140612264404944'>

    @mock.patch("os.chmod")
    @mock.patch("os.stat")
    @mock.patch("io.BufferedWriter")
    @mock.patch("azure.cli.command_modules.resource.bicep.open")
    @mock.patch("os.path.exists")
    @mock.patch("os.path.dirname")
    @mock.patch("os.path.isfile")
    @mock.patch('requests.request', autospec=True)
    def test_use_bicep_cli_from_path_false_after_install(self, isfile_stub, dirname_stub, exists_stub, mock_requests_get, open_stub, buffered_writer_stub, stat_stub, chmod_stub):
        isfile_stub.return_value = False
        dirname_stub.return_value = "tmp"
        exists_stub.return_value = True
        buffered_writer_stub.write.return_value = None
    
        stat_result = mock.Mock()
        stat_result.st_mode = 33206
        stat_stub.return_value = stat_result
    
        chmod_stub.return_value = None
    
        response = mock.MagicMock()
        response.headers = {}
        response.status_code = 200
        response.content = b"test"
        mock_requests_get.return_value = response
    
>       ensure_bicep_installation(self.cli_ctx, release_tag="v0.14.85", stdout=False)

src/azure-cli/azure/cli/command_modules/resource/tests/latest/test_resource_bicep.py:65: 
                                        
src/azure-cli/azure/cli/command_modules/resource/bicep.py:116: in ensure_bicep_installation
    installed_version = get_bicep_installed_version(installation_path)
src/azure-cli/azure/cli/command_modules/resource/bicep.py:264: in get_bicep_installed_version
    installed_version_output = run_command(bicep_executable_path, ["--version"])
src/azure-cli/azure/cli/command_modules/resource/bicep.py:302: in run_command
    process = subprocess.run(
/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/subprocess.py:548: in run
    with Popen(*popenargs, **kwargs) as process:
/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/subprocess.py:1026: in init
    self.execute_child(args, executable, preexec_fn, close_fds,
 
 
 
 
 
 
 
 
                               _ 

self = Popen: returncode: 255 args: ['/home/cloudtest/.azure/bin/bicep', '--version']
args = ['/home/cloudtest/.azure/bin/bicep', '--version']
executable = b'/home/cloudtest/.azure/bin/bicep', preexec_fn = None
close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None
creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = 16
c2pwrite = 17, errread = 18, errwrite = 19, restore_signals = True, gid = None
gids = None, uid = None, umask = -1, start_new_session = False
process_group = -1

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session, process_group):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and process_group == -1
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        process_group, gid, gids, uid, umask,
                        preexec_fn, _USE_VFORK)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                if err_msg == "noexec:chdir":
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                elif err_msg == "noexec":
                    err_msg = ""
                    err_filename = None
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
                if err_filename is not None:
>                   raise child_exception_type(errno_num, err_msg, err_filename)
E                   FileNotFoundError: [Errno 2] No such file or directory: '/home/cloudtest/.azure/bin/bicep'

/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/subprocess.py:1953: FileNotFoundError
azure/cli/command_modules/resource/tests/latest/test_resource_bicep.py:38
❌3.9
Type Test Case Error Message Line
Failed test_use_bicep_cli_from_path_false_after_install self = <azure.cli.command_modules.resource.tests.latest.test_resource_bicep.TestBicep testMethod=test_use_bicep_cli_from_path_false_after_install>
isfile_stub = <function request at 0x7f8ff61909d0>
dirname_stub = <MagicMock name='isfile' id='140256285882592'>
exists_stub = <MagicMock name='dirname' id='140256287485472'>
mock_requests_get = <MagicMock name='exists' id='140256287559152'>
open_stub = <MagicMock name='open' id='140256287452224'>
buffered_writer_stub = <MagicMock name='BufferedWriter' id='140256287488896'>
stat_stub = <MagicMock name='stat' id='140256287548560'>
chmod_stub = <MagicMock name='chmod' id='140256287511456'>

    @mock.patch("os.chmod")
    @mock.patch("os.stat")
    @mock.patch("io.BufferedWriter")
    @mock.patch("azure.cli.command_modules.resource.bicep.open")
    @mock.patch("os.path.exists")
    @mock.patch("os.path.dirname")
    @mock.patch("os.path.isfile")
    @mock.patch('requests.request', autospec=True)
    def test_use_bicep_cli_from_path_false_after_install(self, isfile_stub, dirname_stub, exists_stub, mock_requests_get, open_stub, buffered_writer_stub, stat_stub, chmod_stub):
        isfile_stub.return_value = False
        dirname_stub.return_value = "tmp"
        exists_stub.return_value = True
        buffered_writer_stub.write.return_value = None
    
        stat_result = mock.Mock()
        stat_result.st_mode = 33206
        stat_stub.return_value = stat_result
    
        chmod_stub.return_value = None
    
        response = mock.MagicMock()
        response.headers = {}
        response.status_code = 200
        response.content = b"test"
        mock_requests_get.return_value = response
    
>       ensure_bicep_installation(self.cli_ctx, release_tag="v0.14.85", stdout=False)

src/azure-cli/azure/cli/command_modules/resource/tests/latest/test_resource_bicep.py:65: 
                                        
src/azure-cli/azure/cli/command_modules/resource/bicep.py:116: in ensure_bicep_installation
    installed_version = get_bicep_installed_version(installation_path)
src/azure-cli/azure/cli/command_modules/resource/bicep.py:264: in get_bicep_installed_version
    installed_version_output = run_command(bicep_executable_path, ["--version"])
src/azure-cli/azure/cli/command_modules/resource/bicep.py:302: in run_command
    process = subprocess.run(
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/subprocess.py:505: in run
    with Popen(*popenargs, **kwargs) as process:
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/subprocess.py:951: in init
    self.execute_child(args, executable, preexec_fn, close_fds,
 
 
 
 
 
 
 
 
                               _ 

self = Popen: returncode: 255 args: ['/home/cloudtest/.azure/bin/bicep', '--version']
args = ['/home/cloudtest/.azure/bin/bicep', '--version']
executable = b'/home/cloudtest/.azure/bin/bicep', preexec_fn = None
close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None
creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = 16
c2pwrite = 17, errread = 18, errwrite = 19, restore_signals = True, gid = None
gids = None, uid = None, umask = -1, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: '/home/cloudtest/.azure/bin/bicep'

/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/subprocess.py:1837: FileNotFoundError
azure/cli/command_modules/resource/tests/latest/test_resource_bicep.py:38
️✔️role
️✔️latest
️✔️3.11
️✔️3.9
️✔️search
️✔️latest
️✔️3.11
️✔️3.9
️✔️security
️✔️latest
️✔️3.11
️✔️3.9
️✔️servicebus
️✔️latest
️✔️3.11
️✔️3.9
️✔️serviceconnector
️✔️latest
️✔️3.11
️✔️3.9
️✔️servicefabric
️✔️latest
️✔️3.11
️✔️3.9
️✔️signalr
️✔️latest
️✔️3.11
️✔️3.9
️✔️sql
️✔️latest
️✔️3.11
️✔️3.9
️✔️sqlvm
️✔️latest
️✔️3.11
️✔️3.9
️✔️storage
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️synapse
️✔️latest
️✔️3.11
️✔️3.9
️✔️telemetry
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9
️✔️util
️✔️latest
️✔️3.11
️✔️3.9
️✔️vm
️✔️2018-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.11
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.11
️✔️3.9
️✔️latest
️✔️3.11
️✔️3.9

@ghost ghost added the customer-reported Issues that are reported by GitHub users external to the Azure organization. label Mar 31, 2023
@ghost
Copy link

ghost commented Mar 31, 2023

Thank you for your contribution wwmoraes! We will review the pull request and get back to you soon.

@yonzhan
Copy link
Collaborator

yonzhan commented Mar 31, 2023

ARM

@ghost ghost added the Auto-Assign Auto assign by bot label Mar 31, 2023
@ghost ghost requested a review from yonzhan March 31, 2023 13:09
@ghost ghost assigned zhoxing-ms Mar 31, 2023
@ghost ghost added the ARM az resource/group/lock/tag/deployment/policy/managementapp/account management-group label Mar 31, 2023
@ghost ghost requested a review from wangzelin007 March 31, 2023 13:09
@ghost ghost assigned jiasli Mar 31, 2023
@ghost ghost added the Installation label Mar 31, 2023
@ghost ghost requested a review from jiasli March 31, 2023 13:09
@anthony-c-martin
Copy link
Member

@wwmoraes, thanks for picking this up! Does this address #26007?

@wwmoraes
Copy link
Contributor Author

wwmoraes commented Apr 4, 2023

@wwmoraes, thanks for picking this up! Does this address #26007?

Yes it does. I've referenced it for closure, thanks!

shenglol
shenglol previously approved these changes Apr 11, 2023
@shenglol
Copy link
Contributor

@zhoxing-ms Could you take a look at the PR? Thanks.

@wwmoraes It seems there are code style errors to fix. The CI logs expired, but you can see the errors locally by running azdev style resource and azdev linter resource (suppose you have azdev configured according to the contribution guide).

@zhoxing-ms
Copy link
Contributor

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

Comment on lines 135 to 140
context = ssl.create_default_context(cafile=(
os.environ.get("REQUESTS_CA_BUNDLE")
or os.environ.get("CURL_CA_BUNDLE")
or certifi.where()
))
request = urlopen(_get_bicep_download_url(system, release_tag, target_platform=target_platform), context=context)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we consider using requests.get instead of urlopen, which may make the code more concise and consistent with the approach of get_bicep_available_release_tags() and get_bicep_latest_release_tag()

@zhoxing-ms
Copy link
Contributor

Running flake8 on modules...
ERROR: /mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/resource/_bicep.py:136:11: E121 continuation line under-indented for hanging indent
/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/resource/_bicep.py:137:11: W503 line break before binary operator
/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/resource/_bicep.py:138:11: W503 line break before binary operator
/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/resource/_bicep.py:136:11: E121 continuation line under-indented for hanging indent
/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/resource/_bicep.py:137:11: W503 line break before binary operator
/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/resource/_bicep.py:138:11: W503 line break before binary operator
2 E121 continuation line under-indented for hanging indent
4 W503 line break before binary operator

Could you please resolve these flake8 related issues?

Copy link
Member

@jiasli jiasli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directly using urllib.request.urlopen should be forbidden. Please see #26008

@zhoxing-ms
Copy link
Contributor

@wwmoraes Any update?

added a fallback order equivalent to the one found in the requests
package. Also removed the unneeded setdefaults.
@azure-client-tools-bot-prd
Copy link

azure-client-tools-bot-prd bot commented Jun 6, 2023

️✔️AzureCLI-BreakingChangeTest
️✔️Non Breaking Changes

@wwmoraes
Copy link
Contributor Author

wwmoraes commented Jun 6, 2023

The individual test case reported as an error works fine when ran directly with azdev test test_use_bicep_cli_from_path_false_after_install and with a pre-existing binary in the path. It fails when ran without an installed bicep version or if the entire suite is ran as azdev test TestBicep, with or without a binary on the path. Running the tests with --series does not solve it either. This relates to how the test functions do not run within an isolated temporary directory. This means all tests share the environment, which leads to unpredictable behaviour like the one seen here. It doubles down as the test depends on a pre-existing environment state, which makes the test non-atomic and non-reproducible.

It is beyond the objective of this PR to refactor this entire suite of unit tests. @zhoxing-ms would you be so kind as to review it once again?

@zhoxing-ms
Copy link
Contributor

zhoxing-ms commented Jun 13, 2023

Type Test Case Error Message Line
Failed test_use_bicep_cli_from_path_false_after_install E FileNotFoundError: [Errno 2] No such file or directory: '/home/cloudtest/.azure/bin/bicep' azure/cli/command_modules/resource/tests/latest/test_resource_bicep.py:36

I guess we can solve this CI problem through mock _get_bicep_installation_path and _get_bicep_installed_version. But sorry, due to recent busy work, I don't have enough time to validate this idea.

@mock.patch("azure.cli.command_modules.resource._bicep._get_bicep_installation_path")
@mock.patch("azure.cli.command_modules.resource._bicep._get_bicep_installed_version")

@shenglol Do you have any thoughts or suggestions regarding this testing issue?

@wwmoraes
Copy link
Contributor Author

Type Test Case Error Message Line
Failed test_use_bicep_cli_from_path_false_after_install E FileNotFoundError: [Errno 2] No such file or directory: '/home/cloudtest/.azure/bin/bicep' azure/cli/command_modules/resource/tests/latest/test_resource_bicep.py:36
I guess we can solve this CI problem through mock _get_bicep_installation_path and _get_bicep_installed_version

@mock.patch("azure.cli.command_modules.resource._bicep._get_bicep_installation_path")
@mock.patch("azure.cli.command_modules.resource._bicep._get_bicep_installed_version")

@shenglol Do you have any thoughts or suggestions regarding this testing issue?

It seems the os.path.isfile stub is not working as expected. It is set to return False, so it should force the first if block in ensure_bicep_installation to be skipped, making no calls to _get_bicep_installed_version. Yet we see it being called and erroring during tests. 🤷‍♂️

@shenglol
Copy link
Contributor

I think the errors are due to test cases sharing the same DummyCli instance, resulting in a race condition. I've committed a fix in #26797. @wwmoraes You may integrate the changes once that PR is merged to get unblocked.

@zhoxing-ms
Copy link
Contributor

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@zhoxing-ms
Copy link
Contributor

/azp run

Copy link

Azure Pipelines successfully started running 3 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ARM az resource/group/lock/tag/deployment/policy/managementapp/account management-group Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bicep] Accessing https://downloads.bicep.azure.com/ from Mariner 2.0 fails with ssl.SSLCertVerificationError
8 participants