Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google Assistant SDK: support audio response playback #85989

Merged
merged 4 commits into from
Jan 24, 2023

Conversation

tronikos
Copy link
Member

@tronikos tronikos commented Jan 16, 2023

Proposed change

Support Google Assistant's audio response playback by adding an optional media_player field in google_assistant_sdk.send_text_command service.

Underlyging library changes: tronikos/gassist_text@0.0.7...0.0.8

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

@home-assistant
Copy link

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (media_player) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of media_player can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Change the title of the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign media_player Removes the current integration label and assignees on the issue, add the integration domain after the command.

@tronikos tronikos mentioned this pull request Jan 16, 2023
19 tasks
Copy link
Contributor

@allenporter allenporter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally look good to me. I have a few specific comments on how the audio responses are managed, exposed via urls, and test coverage for cleanup.

@@ -22,7 +22,10 @@
from .const import CONTENT_AUTH_EXPIRY_TIME, MediaClass, MediaType

# Paths that we don't need to sign
PATHS_WITHOUT_AUTH = ("/api/tts_proxy/",)
PATHS_WITHOUT_AUTH = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While there is not a super high security bar here, uuid1 paths seem below the bar as my impression is those are on the easier side to predict. Is there a technical motivation for opting out of signed paths?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just followed what tts_proxy does.

See comment below where this constant is used:

    elif parsed.path.startswith(PATHS_WITHOUT_AUTH):
        # We don't sign this path if it doesn't need auth. Although signing itself can't
        # hurt, some devices are unable to handle long URLs and the auth signature might
        # push it over.
        pass

I just did some tests on my setup and I don't have any such issues on any of my media players. So I removed the change in this file and also set requires_auth = True in GoogleAssistantSDKAudioView.

If anyone complains that playback doesn't work because of long URLs we can consider adding this back.


audio_view = GoogleAssistantSDKAudioView(hass)
hass.http.register_view(audio_view)
hass.data[DOMAIN][entry.entry_id][DATA_AUDIO_VIEW] = audio_view
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A more commonly seen pattern is to use something like an audio storage manager, rather than passing around references to the View directly. Other code can stick stuff in the manger and the view can interact with that.

(You could also consider having that intermediate manager thing be single object that holds the session, though not sure)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -180,6 +189,78 @@ async def test_send_text_command_expired_token_refresh_failure(
assert any(entry.async_get_active_flows(hass, {"reauth"})) == requires_reauth


async def test_send_text_command_media_player(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expiration also seems like an important thing to test. Typically this is done with async_fire_time_changed . Speculation:... though you may need to use async_call_later or one of the home assistant variants to make this happen, rather than calling call later on the loop directly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
I copied this call later on the loop directly from tts, where it's untested. I wanted to test it but I couldn't figure out how. Thanks for pointing me to the right functions!

@allenporter
Copy link
Contributor

Also, i'm ignoring the version bump part of this for now assuming that will get sorted out in the other PR before we merge this

@tronikos
Copy link
Member Author

Thanks for your review!
I'm pumping the version to 0.0.8 (instead of 0.0.10) that only contains the support for audio responses so please don't ignore it. The other PR might take a while and I'd prefer this to be merged before the next release.

@tronikos tronikos requested a review from allenporter January 17, 2023 07:38
@tronikos
Copy link
Member Author

Since I reverted the change in media_player/browse_media.py could you fix the labels and remove core from reviewers?

@tronikos
Copy link
Member Author

Actually the version bump includes a div id change (assistant-card-content instead of show_text_content). Anyway it doesn't matter much and in the other PR once we sort it out we will either remove HTML parsing or do HTML stripping instead. I prefer merging this if possible before the other PR.

@tronikos
Copy link
Member Author

Gentle ping. It would be great if this could be approved and merged before the beta release.

@allenporter allenporter merged commit 0daaa37 into home-assistant:dev Jan 24, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Jan 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants