[ai-did-you-mean-this] Add custom telemetry module and improve handling of different failure scenarios. #2191
Conversation
(cherry picked from commit 512cb17)
…matching, store version in centralized location, improve debug logger handling, expand unit test coverage. use type hints in some files to improve intellisense
… handle suggestion parse failure, rework telemetry naming and descriptions, make enums easier to use, fix odd copyright header formatting,
…sponse, update suggestion encoder to use correct delimiters.
…parameter blocklist, refactor command parameter normalization unit tests, simplify some code and improve readability, rename `UnknownException` to something more pythonic.
|
ai-did-you-mean-this |
|
hi @bim-msft could you pls review this PR? thanks |
| if arguments_len < parameters_len: | ||
| missing_argument_count = parameters_len - arguments_len | ||
| for _ in range(missing_argument_count): | ||
| self.arguments.append('') |
There was a problem hiding this comment.
how to align parameters with argument?
There was a problem hiding this comment.
The model used by the service should already have its parameters and arguments aligned. The checks performed here are intended to help test the model and validate results returned by the service.
Example of this behavior in action:
Scenario(
cli_command=CliCommand('account get-access-token', ['--test', '--debug'], 'a'),
expected_user_fault_type=UserFaultType.UNRECOGNIZED_ARGUMENTS
)Specifying no argument for the --debug flag results in an empty string being added to the arguments array.
The only way to have more arguments returned than parameters is some sort of model error or client-side parsing error.
|
|
||
|
|
||
| class ExtensionTelemterySession(): | ||
| def __enter__(self): |
There was a problem hiding this comment.
any exception handling on enter besides clear()?
| _set_exception_func_orig = telemetry.set_exception | ||
| _add_extension_event_func_orig = telemetry.add_extension_event | ||
|
|
||
| def _set_exception_hook(exception: Exception, fault_type: str, *args, summary: str = None, **kwargs): |
| return hash(self.value) | ||
|
|
||
|
|
||
| class TelemetryProperty(Enum): |
There was a problem hiding this comment.
curious, where to read your ai key?
There was a problem hiding this comment.
Addressed comment in offline conversation.
…nner. Better invalid suggestion handling. Move telemetry property prefix to the `_telemetry` file to make it easier to find.
This checklist is used to make sure that common guidelines for a pull request are followed.
General Guidelines
azdev style <YOUR_EXT>locally? (pip install azdevrequired)python scripts/ci/test_index.py -qlocally?