Capture Alexa Responses

[kingbear2]

Hello, first off, thanks for a great project! I just got started and am loving what I'm seeing. I have an idea on how to capture Alexa's responses that I think would be very beneficial.

What is the need?

There are millions of things you can ask Alexa that she can answer, but you don't have a way of generating the same queries in HA. I'll give 2 examples below, and while it's likely that the first example I'm giving is in fact possible in HA, it's really the concept that matters for other things that are not possible.

Example 1: Let's say I want HA to know when the next Bulls game is. I can ask Alexa, and she will respond: "The Bulls and Nuggets are facing off on Thursday, October 12 at 7:00pm." I would love to see that info in HA in case I want to have a dashboard that shows me when the next Bulls game is. But currently I have not found (after many hours of searching) how to capture the Alexa response in text.

Example 2: I have a skill for a Sub-Zero fridge freezer. There is currently no integration for Sub-Zero in HA. There's also no way for Alexa to expose a skill entity or sensor into HA. I'd like to have a dashboard showing me the current temperature of the fridge and freezer. I can ask "Alexa, what's the freezer temperature?" and Alexa will respond "Freezer Freezer Temp is 0 Fahrenheit." Again, there's no way to capture this response and put it into HA.

Proposed Solution

I noticed you are missing a key "endpoint" for Alexa from your list of "Known Endpoints". And that is: https://www.amazon.com/alexa-privacy/apd/rvh/customer-history-records. This endpoint requires a "startTime" & "endTime" variable to be passed to it. If we use media_player.play_media to send a command to Alexa (asking the freezer temp), there's an attribute we can grab of media_player.name_of_player.last_called_timestamp. If we pass that through to the URL above adding in the last_called_timestamp as the startTime and the endTime, we can actually see what Alexa responded. For example, https://www.amazon.com/alexa-privacy/apd/rvh/customer-history-records?startTime=1691004814627&endTime=1691004814627 would retrieve the following (redacted for privacy):

{
"customerHistoryRecords":[
{
"recordKey":"REDACTED_FOR_PRIVACY",
"recordType":"VOICE_HISTORY",
"timestamp":1691004814627,
"customerId":"REDACTED_FOR_PRIVACY",
"device":{
"deviceName":"REDACTED Echo Show",
"deviceEntityId":null
},
"isBinaryFeedbackProvided":false,
"isFeedbackPositive":false,
"utteranceType":"GENERAL",
"domain":"HomeAutomation",
"intent":"GetSettingsDetailsIntent",
"skillName":"",
"voiceHistoryRecordItems":[
{
"recordItemKey":"REDACTED_FOR_PRIVACY",
"recordItemType":"CUSTOMER_TRANSCRIPT",
"utteranceId":"REDACTED_FOR_PRIVACY",
"timestamp":1691004814625,
"transcriptText":"alexa what's the freezer temperature",
"agentVisualName":"",
"personsInfo":[
]
},
{
"recordItemKey":"REDACTED_FOR_PRIVACY",
"recordItemType":"ALEXA_RESPONSE",
"utteranceId":"REDACTED_FOR_PRIVACY",
"timestamp":1691004814629,
"transcriptText":"Freezer Freezer Temp is 0 Fahrenheit",
"agentVisualName":"",
"personsInfo":[
]
}
],
"personsInfo":[
],
"cardResponse":null
}
],
"encodedRequestToken":null,
"noDataFoundWithinTimeLimit":false,
"lastRecordTimestamp":null,
"voiceProfileAssociatedWithPerson":false
}

We can then grab the "transcriptText" to get the response. Once we have the response, we can use markup to parse it as we please. But I see no reason why Alexa Media Player, with its already authenticated session with Alexa/Amazon, can't retrieve the response and add it as an attribute.

This would open up thousands of possibilities of working with skills that are only available on Alexa and don't integrate with HA directly, nor are exposed entities on Alexa that HA can pick up on. We can have automations asking Alexa the score in a game every 30 seconds and then update an HA dashboard for example. (Again, sports scores are probably a bad example because there is likely already an HA integration for this, but you get the point.)

Curious the communities and devs thoughts on adding this functionality in. It would really open up a world of possibilities. And as mentioned, I searched for hours and could not find any other way to retrieve a response from Alexa into HA (or anywhere for that matter).

(Just a note of reference, the website I inspected to find the API URL is: https://www.amazon.com/alexa-privacy/apd/rvh which shows a list of all questions and answers to Alexa).

Thank you for your time.

[wheeller123]

Hi @kingbear2, did you ever find a way to do this?

[kingbear2]

Nope, the best thing I found was to use Google Assistant SDK Custom instead, and retrieve the responses. This would obviously only work if the skill you are looking for has an associated Google Skill as well. To get specific information inside the response you would need to use a helper. Here are some screenshots of my example, which asks Google to give me the word of the day, and then extracts the type of word (noun, adjective, verb, etc).

First is the script:

Then a template helper:

With the options as follows:

I'm sure someone who is more familiar with markdown code can script it to find the actual "type" versus specifically looking for the word "verb" and then changing the variable to "verb".

But this only helps when the response is consistent. But it can be used and should work decently. I haven't deployed it in production yet, I've only tested it.