{Core} Add random_config_dir param in DummyCli #25689

bebound · 2023-03-06T06:54:49Z

Description

ARM64 machines are multi-core, and some errors occur because of concurrency, as all tests use same config dir.

I add a random_config_dir param in DummyCli. If it is True, the config dir will be ~/.azure/dummy_cli_config_dir/xxxxxxx.

If random config is enabled by default, the config file is created for each test, and the performance decrease a lot.
Azure.azure-cli Full Test spends 24 min instead of 16min.

I enable random_config_dir for the tests which use az config or az configure.
Example usage:

class ConfigTest(ScenarioTest):

    def __init__(self, *arg, **kwargs):
        super().__init__(*arg, random_config_dir=True, **kwargs)

2023-03-03T11:05:05.5590254Z [gw2] [ 65%] FAILED tests/test_cloud.py::TestCloud::test_remove_known_cloud 
2023-03-03T11:05:05.5590455Z 
2023-03-03T11:05:05.5590884Z =================================== FAILURES ===================================
2023-03-03T11:05:05.5591177Z ______________________ TestCloud.test_remove_known_cloud _______________________
2023-03-03T11:05:05.5592042Z [gw2] linux -- Python 3.10.10 /opt/az/bin/python3
2023-03-03T11:05:05.5592295Z self = <core.tests.test_cloud.TestCloud testMethod=test_remove_known_cloud>
2023-03-03T11:05:05.5592422Z 
2023-03-03T11:05:05.5592577Z     def test_remove_known_cloud(self):
2023-03-03T11:05:05.5592756Z >       cli = DummyCli()
2023-03-03T11:05:05.5592824Z 
2023-03-03T11:05:05.5593120Z /opt/az/lib/python3.10/site-packages/azure/cli/core/tests/test_cloud.py:216: 
2023-03-03T11:05:05.5593362Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2023-03-03T11:05:05.5593706Z /opt/az/lib/python3.10/site-packages/azure/cli/core/mock.py:25: in __init__
2023-03-03T11:05:05.5593932Z     super(DummyCli, self).__init__(
2023-03-03T11:05:05.5594263Z /opt/az/lib/python3.10/site-packages/azure/cli/core/__init__.py:59: in __init__
2023-03-03T11:05:05.5594502Z     super(AzCli, self).__init__(**kwargs)
2023-03-03T11:05:05.5594818Z /opt/az/lib/python3.10/site-packages/knack/cli.py:82: in __init__
2023-03-03T11:05:05.5595028Z     self.config = config_cls(
2023-03-03T11:05:05.5595524Z /opt/az/lib/python3.10/site-packages/knack/config.py:75: in __init__
2023-03-03T11:05:05.5595793Z     self._config_file_chain.append(_ConfigFile(self.config_dir, self.config_path))
2023-03-03T11:05:05.5596156Z /opt/az/lib/python3.10/site-packages/knack/config.py:195: in __init__
2023-03-03T11:05:05.5596418Z     self.config_parser.read(config_path, encoding=CONFIG_FILE_ENCODING)
2023-03-03T11:05:05.5596652Z /opt/az/lib/python3.10/configparser.py:699: in read
2023-03-03T11:05:05.5596854Z     self._read(fp, filename)
2023-03-03T11:05:05.5597051Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2023-03-03T11:05:05.5597146Z 
2023-03-03T11:05:05.5597321Z self = <configparser.ConfigParser object at 0xffff9808a770>
2023-03-03T11:05:05.5597664Z fp = <_io.TextIOWrapper name='/root/.azure/config' mode='r' encoding='utf-8'>
2023-03-03T11:05:05.5597957Z fpname = '/root/.azure/config'
2023-03-03T11:05:05.5598040Z 
2023-03-03T11:05:05.5598203Z     def _read(self, fp, fpname):
2023-03-03T11:05:05.5598387Z         """Parse a sectioned configuration file.
2023-03-03T11:05:05.5598554Z     
2023-03-03T11:05:05.5598736Z         Each section in a configuration file contains a header, indicated by
2023-03-03T11:05:05.5598980Z         a name in square brackets (`[]`), plus key/value options, indicated by
2023-03-03T11:05:05.5599218Z         `name` and `value` delimited with a specific substring (`=` or `:` by
2023-03-03T11:05:05.5599400Z         default).
2023-03-03T11:05:05.5599542Z     
2023-03-03T11:05:05.5599721Z         Values can span multiple lines, as long as they are indented deeper
2023-03-03T11:05:05.5600070Z         than the first line of the value. Depending on the parser's mode, blank
2023-03-03T11:05:05.5600312Z         lines may be treated as parts of multiline values or ignored.
2023-03-03T11:05:05.5600485Z     
2023-03-03T11:05:05.5600673Z         Configuration files may include comments, prefixed by specific
2023-03-03T11:05:05.5600918Z         characters (`#` and `;` by default). Comments may appear on their own
2023-03-03T11:05:05.5601152Z         in an otherwise empty line or may be entered in lines holding values or
2023-03-03T11:05:05.5601417Z         section names. Please note that comments get stripped off when reading configuration files.
2023-03-03T11:05:05.5601624Z         """
2023-03-03T11:05:05.5601770Z         elements_added = set()
2023-03-03T11:05:05.5601956Z         cursect = None                        # None, or a dictionary
2023-03-03T11:05:05.5602124Z         sectname = None
2023-03-03T11:05:05.5602282Z         optname = None
2023-03-03T11:05:05.5602434Z         lineno = 0
2023-03-03T11:05:05.5602582Z         indent_level = 0
2023-03-03T11:05:05.5602759Z         e = None                              # None, or an exception
2023-03-03T11:05:05.5602950Z         for lineno, line in enumerate(fp, start=1):
2023-03-03T11:05:05.5603141Z             comment_start = sys.maxsize
2023-03-03T11:05:05.5603465Z             # strip inline comments
2023-03-03T11:05:05.5603763Z             inline_prefixes = {p: -1 for p in self._inline_comment_prefixes}
2023-03-03T11:05:05.5604005Z             while comment_start == sys.maxsize and inline_prefixes:
2023-03-03T11:05:05.5604200Z                 next_prefixes = {}
2023-03-03T11:05:05.5604382Z                 for prefix, index in inline_prefixes.items():
2023-03-03T11:05:05.5604589Z                     index = line.find(prefix, index+1)
2023-03-03T11:05:05.5604760Z                     if index == -1:
2023-03-03T11:05:05.5604914Z                         continue
2023-03-03T11:05:05.5605086Z                     next_prefixes[prefix] = index
2023-03-03T11:05:05.5605373Z                     if index == 0 or (index > 0 and line[index-1].isspace()):
2023-03-03T11:05:05.5605592Z                         comment_start = min(comment_start, index)
2023-03-03T11:05:05.5605797Z                 inline_prefixes = next_prefixes
2023-03-03T11:05:05.5605976Z             # strip full line comments
2023-03-03T11:05:05.5606312Z             for prefix in self._comment_prefixes:
2023-03-03T11:05:05.5606506Z                 if line.strip().startswith(prefix):
2023-03-03T11:05:05.5606686Z                     comment_start = 0
2023-03-03T11:05:05.5606840Z                     break
2023-03-03T11:05:05.5607001Z             if comment_start == sys.maxsize:
2023-03-03T11:05:05.5607180Z                 comment_start = None
2023-03-03T11:05:05.5607376Z             value = line[:comment_start].strip()
2023-03-03T11:05:05.5607552Z             if not value:
2023-03-03T11:05:05.5607730Z                 if self._empty_lines_in_values:
2023-03-03T11:05:05.5607927Z                     # add empty line to the value, but only if there was no
2023-03-03T11:05:05.5608119Z                     # comment on the line
2023-03-03T11:05:05.5608294Z                     if (comment_start is None and
2023-03-03T11:05:05.5608466Z                         cursect is not None and
2023-03-03T11:05:05.5608632Z                         optname and
2023-03-03T11:05:05.5608798Z                         cursect[optname] is not None):
2023-03-03T11:05:05.5609098Z                         cursect[optname].append('') # newlines added at join
2023-03-03T11:05:05.5609286Z                 else:
2023-03-03T11:05:05.5609444Z                     # empty line marks end of value
2023-03-03T11:05:05.5609627Z                     indent_level = sys.maxsize
2023-03-03T11:05:05.5609790Z                 continue
2023-03-03T11:05:05.5609940Z             # continuation line?
2023-03-03T11:05:05.5610134Z             first_nonspace = self.NONSPACECRE.search(line)
2023-03-03T11:05:05.5610360Z             cur_indent_level = first_nonspace.start() if first_nonspace else 0
2023-03-03T11:05:05.5610584Z             if (cursect is not None and optname and
2023-03-03T11:05:05.5610779Z                 cur_indent_level > indent_level):
2023-03-03T11:05:05.5610961Z                 cursect[optname].append(value)
2023-03-03T11:05:05.5611153Z             # a section header or option header?
2023-03-03T11:05:05.5611322Z             else:
2023-03-03T11:05:05.5611483Z                 indent_level = cur_indent_level
2023-03-03T11:05:05.5611660Z                 # is it a section header?
2023-03-03T11:05:05.5611831Z                 mo = self.SECTCRE.match(value)
2023-03-03T11:05:05.5611996Z                 if mo:
2023-03-03T11:05:05.5612241Z                     sectname = mo.group('header')
2023-03-03T11:05:05.5612426Z                     if sectname in self._sections:
2023-03-03T11:05:05.5612627Z                         if self._strict and sectname in elements_added:
2023-03-03T11:05:05.5612844Z                             raise DuplicateSectionError(sectname, fpname,
2023-03-03T11:05:05.5613028Z                                                         lineno)
2023-03-03T11:05:05.5613206Z                         cursect = self._sections[sectname]
2023-03-03T11:05:05.5613392Z                         elements_added.add(sectname)
2023-03-03T11:05:05.5613747Z                     elif sectname == self.default_section:
2023-03-03T11:05:05.5613945Z                         cursect = self._defaults
2023-03-03T11:05:05.5614099Z                     else:
2023-03-03T11:05:05.5614259Z                         cursect = self._dict()
2023-03-03T11:05:05.5614443Z                         self._sections[sectname] = cursect
2023-03-03T11:05:05.5614648Z                         self._proxies[sectname] = SectionProxy(self, sectname)
2023-03-03T11:05:05.5614855Z                         elements_added.add(sectname)
2023-03-03T11:05:05.5615136Z                     # So sections can't start with a continuation line
2023-03-03T11:05:05.5615328Z                     optname = None
2023-03-03T11:05:05.5615501Z                 # no section header in the file?
2023-03-03T11:05:05.5615669Z                 elif cursect is None:
2023-03-03T11:05:05.5615870Z                     raise MissingSectionHeaderError(fpname, lineno, line)
2023-03-03T11:05:05.5616063Z                 # an option line?
2023-03-03T11:05:05.5616211Z                 else:
2023-03-03T11:05:05.5616510Z                     mo = self._optcre.match(value)
2023-03-03T11:05:05.5616672Z                     if mo:
2023-03-03T11:05:05.5616951Z                         optname, vi, optval = mo.group('option', 'vi', 'value')
2023-03-03T11:05:05.5617150Z                         if not optname:
2023-03-03T11:05:05.5617332Z                             e = self._handle_error(e, fpname, lineno, line)
2023-03-03T11:05:05.5617548Z                         optname = self.optionxform(optname.rstrip())
2023-03-03T11:05:05.5617742Z                         if (self._strict and
2023-03-03T11:05:05.5617923Z                             (sectname, optname) in elements_added):
2023-03-03T11:05:05.5618134Z >                           raise DuplicateOptionError(sectname, optname,
2023-03-03T11:05:05.5618323Z                                                        fpname, lineno)
2023-03-03T11:05:05.5618826Z E                           configparser.DuplicateOptionError: While reading from '/root/.azure/config' [line 31]: option 'cache_ttl' in section 'core' already exists
2023-03-03T11:05:05.5619016Z 
2023-03-03T11:05:05.5619208Z /opt/az/lib/python3.10/configparser.py:1098: DuplicateOptionError

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.

azure-client-tools-bot-prd · 2023-03-06T06:54:51Z

️✔️acr

️✔️acs

️✔️advisor

️✔️ams

️✔️apim

️✔️appconfig

️✔️appservice

️✔️aro

️✔️backup

️✔️batch

️✔️batchai

️✔️billing

️✔️botservice

️✔️cdn

️✔️cloud

️✔️cognitiveservices

️✔️config

️✔️configure

️✔️consumption

️✔️container

️✔️core

️✔️cosmosdb

️✔️databoxedge

️✔️dla

️✔️dls

️✔️dms

️✔️eventgrid

️✔️eventhubs

️✔️feedback

️✔️find

️✔️hdinsight

️✔️identity

️✔️iot

️✔️keyvault

️✔️kusto

️✔️lab

️✔️managedservices

️✔️maps

️✔️marketplaceordering

️✔️monitor

️✔️netappfiles

️✔️network

️✔️policyinsights

️✔️privatedns

️✔️profile

️✔️rdbms

️✔️redis

️✔️relay

️✔️resource

️✔️role

️✔️search

️✔️security

️✔️servicebus

️✔️serviceconnector

️✔️servicefabric

️✔️signalr

️✔️sql

️✔️sqlvm

️✔️storage

️✔️synapse

️✔️telemetry

️✔️util

️✔️vm

yonzhan · 2023-03-06T07:11:03Z

CI

bebound · 2023-03-09T01:39:50Z

src/azure-cli/azure/cli/command_modules/role/tests/latest/test_role.py

@@ -568,7 +571,7 @@ def test_role_assignment_handle_conflicted_assignments(self, resource_group):

                self.assertGreaterEqual(len(local_defaults_config), 1)
                actual = set([(x['name'], x['source'], x['value']) for x in local_defaults_config if x['name'] == 'group'])
-                expected = set([('group', os.path.join(temp_dir, '.azure', 'config'), self.kwargs['rg'])])
+                expected = set([('group', os.path.join(temp_dir, os.path.basename(self.cli_ctx.config.config_dir), 'config'), self.kwargs['rg'])])


The config path is temp_dir+basename(config_dir)+'config' when --scope local

Let's provide some concrete example.

If work dir is C:\Users\hanglei\AppData\Local\Temp\tmplxf_89ez, local config file is C:\Users\hanglei\AppData\Local\Temp\tmplxf_89ez\0azXbKR9OdJuZPFS\config

0azXbKR9OdJuZPFS is the base name of config dir: ~/.azure/dummy_cli_config_dir/0azXbKR9OdJuZPFS/

The local config folder was constructed by knack.config.CLIConfig:

https://github.com/microsoft/knack/blob/e496c9590792572e680cb3ec959db175d9ba85dd/knack/config.py#L63

config_dir_name = os.path.basename(self.config_dir) current_config_dir = os.path.join(current_dir, config_dir_name)

current_config_dir is constructed by combining current_dir with the random dir name, making the path a little bit weird.

bebound · 2023-03-09T05:27:50Z

src/azure-cli-testsdk/azure/cli/testsdk/base.py

@@ -181,7 +186,8 @@ class LocalContextScenarioTest(ScenarioTest):
    def __init__(self, method_name, config_file=None, recording_name=None, recording_processors=None,
                 replay_processors=None, recording_patches=None, replay_patches=None, working_dir=None):
        super(LocalContextScenarioTest, self).__init__(method_name, config_file, recording_name, recording_processors,
-                                                       replay_processors, recording_patches, replay_patches)
+                                                       replay_processors, recording_patches, replay_patches,
+                                                       random_config_dir=True)


ConfigureGlobalDefaultsTest runs config param-persist off in tearDown. I think use random_config_dir is necessary

Local Context/Param Persist has been deprecated. Consider deleting related tests instead: #24726

jiasli · 2023-03-17T06:30:33Z

src/azure-cli-core/azure/cli/core/mock.py

+            config_dir=os.path.join(GLOBAL_CONFIG_DIR, 'dummy_cli_config_dir',
+                                    random_string()) if random_config_dir else GLOBAL_CONFIG_DIR,


As an alternative, we may also consider using tempfile module: https://docs.python.org/3/library/tempfile.html

Well, sorry I didn't make my point clear. I am simply providing an alternative solution. You solution is actually better since your location is easy to find. 😉

jiasli · 2023-03-17T06:31:51Z

src/azure-cli-testsdk/azure/cli/testsdk/base.py

+            import shutil
+            shutil.rmtree(self.cli_ctx.config.config_dir, ignore_errors=True)


Well... yes. rmtree can fail intermittently on Windows. Consider using

azure-cli/src/azure-cli-core/azure/cli/core/util.py

Line 1317 in 5757778

def rmtree_with_retry(path):

Oh, yes. I forget that Windows bug.

jiasli · 2023-03-17T06:38:01Z

src/azure-cli-core/azure/cli/core/mock.py

        from azure.cli.core._config import GLOBAL_CONFIG_DIR, ENV_VAR_PREFIX
        from azure.cli.core._help import AzCliHelp
        from azure.cli.core._output import AzOutputProducer

        from knack.completion import ARGCOMPLETE_ENV_NAME

+        random_config_dir = kwargs.get('random_config_dir', False)


Consider making it an explicit kwarg.

jiasli · 2023-08-14T12:18:50Z

src/azure-cli/azure/cli/command_modules/role/tests/latest/test_role.py

@@ -255,6 +255,9 @@ def test_role_definition_scenario(self):

 class RoleAssignmentScenarioTest(RoleScenarioTestBase):

+    def __init__(self, *arg, **kwargs):
+        super().__init__(*arg, random_config_dir=True, **kwargs)


random_config_dir=True make tests under RoleAssignmentScenarioTest fail during live run, because the login context is lost.

Yes, we need to merge #26475 first.

Use random config dir in DummyCli

43813d2

ghost added Auto-Assign Auto assign by bot CI CI labels Mar 6, 2023

ghost requested review from wangzelin007, yonzhan, jiasli, kairu-ms and jsntcy March 6, 2023 06:55

ghost assigned wangzelin007 Mar 6, 2023

ghost added this to the March 2023 (2023-04-04) milestone Mar 6, 2023

bebound added 7 commits March 6, 2023 15:15

Use subdir

109201c

Fix test

b94d580

Remove warning

1b87eea

Fix test

869aacb

Get real config dir from cli_ctx.config

99aeaf0

Fix path

c29ff94

Add random_config_dir param

6a6ca82

bebound changed the title ~~{CI} Use random config dir in DummyCli~~ {Core} Add random_config_dir param in DummyCli Mar 8, 2023

bebound commented Mar 9, 2023

View reviewed changes

bebound marked this pull request as ready for review March 9, 2023 03:59

bebound requested review from evelyn-ys, calvinhzy, zhoxing-ms, yanzhudd and necusjz as code owners March 9, 2023 03:59

Minor change

60afa32

bebound commented Mar 9, 2023

View reviewed changes

jiasli reviewed Mar 17, 2023

View reviewed changes

bebound mentioned this pull request Mar 23, 2023

Improve Unit Test Experience for Functionality using Azure CLI Configuration #25796

Closed

bebound added 4 commits March 24, 2023 12:36

Use random folder

7a8d2bf

Use rmtree

d777ac0

Minor fix

7f9e0d4

Minor fix

0ec2406

yonzhan modified the milestones: March 2023 (2023-04-04), April 2023 (2023-04-25) Mar 31, 2023

jiasli approved these changes Apr 12, 2023

View reviewed changes

zhoxing-ms approved these changes Apr 12, 2023

View reviewed changes

bebound merged commit b223dd9 into Azure:dev Apr 13, 2023

bebound deleted the config-dir branch April 13, 2023 01:47

bebound mentioned this pull request Apr 13, 2023

{Core} Move clouds.config to config dir #26138

Draft

3 tasks

zhoxing-ms mentioned this pull request May 17, 2023

[Container] az container create: Add new parameters for container security context for confidential ContainerGroupSku #26273

Merged

3 tasks

bebound mentioned this pull request May 18, 2023

{Core} Copy login credential when run test with --live and random_config_dir is enabled #26475

Merged

3 tasks

jiasli reviewed Aug 14, 2023

View reviewed changes

jiasli mentioned this pull request Aug 14, 2023

{Role} Bump role_definitions API version to 2022-05-01-preview #26577

Merged

avgale pushed a commit to avgale/azure-cli that referenced this pull request Aug 24, 2023

{Core} Add random_config_dir param in DummyCli (Azure#25689)

fb845fe

bebound mentioned this pull request Mar 5, 2024

test_batch_general_arm_cmd randomly fails #28500

Closed

This was referenced Mar 8, 2024

{ARM} Fix race condition in bicep unit tests #26797

Merged

{Batch} Fix race condition in test #28508

Merged

This was referenced Apr 1, 2024

When AZURE_CONFIG_DIR is set, login context gets cleared by setting random_config_dir=True #28668

Closed

{Core} Fix random_config_dir when AZURE_CONFIG_DIR is set #28673

Merged

jiasli mentioned this pull request Apr 29, 2024

{testsdk} Initialize random config dir in setUp #28849

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{Core} Add random_config_dir param in DummyCli #25689

{Core} Add random_config_dir param in DummyCli #25689

bebound commented Mar 6, 2023 •

edited

Loading

azure-client-tools-bot-prd bot commented Mar 6, 2023 •

edited

Loading

yonzhan commented Mar 6, 2023

bebound Mar 9, 2023 •

edited

Loading

jiasli Mar 17, 2023

bebound Mar 24, 2023 •

edited

Loading

jiasli Mar 29, 2023

bebound Mar 9, 2023

jiasli Mar 17, 2023 •

edited

Loading

jiasli Mar 17, 2023 •

edited

Loading

jiasli Mar 27, 2023

jiasli Mar 17, 2023 •

edited

Loading

bebound Mar 17, 2023

jiasli Mar 17, 2023

jiasli Aug 14, 2023

bebound Aug 14, 2023

		config_dir=os.path.join(GLOBAL_CONFIG_DIR, 'dummy_cli_config_dir',
		random_string()) if random_config_dir else GLOBAL_CONFIG_DIR,

		import shutil
		shutil.rmtree(self.cli_ctx.config.config_dir, ignore_errors=True)

{Core} Add random_config_dir param in DummyCli #25689

{Core} Add random_config_dir param in DummyCli #25689

Conversation

bebound commented Mar 6, 2023 • edited Loading

azure-client-tools-bot-prd bot commented Mar 6, 2023 • edited Loading

yonzhan commented Mar 6, 2023

bebound Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bebound Mar 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiasli Mar 17, 2023 • edited Loading

Choose a reason for hiding this comment

jiasli Mar 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiasli Mar 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bebound commented Mar 6, 2023 •

edited

Loading

azure-client-tools-bot-prd bot commented Mar 6, 2023 •

edited

Loading

bebound Mar 9, 2023 •

edited

Loading

bebound Mar 24, 2023 •

edited

Loading

jiasli Mar 17, 2023 •

edited

Loading

jiasli Mar 17, 2023 •

edited

Loading

jiasli Mar 17, 2023 •

edited

Loading