[Amazon Alexa] Nachfragen durch Alexa an den User

[MadMax-FHEM]

Hi Paulo,

sorry did not have time yet for looking/checking "if that makes any sense"

But I think there are only two ways of receiving results:

- Custom Skill (or equivalent) -> open Port...

- a server in the internet and somehow bringing local fhem and Amazon together like alexa-fhem Connector together with the fhem Vereinsserver...

But I am not sure if that with the "server in the middle" also works for that kind...

I hope you keep on!
And maybe also Andre (justme1968) finds this thread and the idea behind interesting!
Because then the chance that this will work are getting much higher...

So easter ist coming, maybe/hpefully I find some time then

Bye, Joachim

[plopes9000]

Hi Joachim,

I've got it working, sort of a hack in many ways and several details still being considered, but it is working.

Code Auswählen Erweitern


define MQTT2_zigbee_study_motion_sensor_doif DOIF ([MQTT2_zigbee_study_motion_sensor:occupancy] eq "true" and [MQTT2_zigbee_study_motion_sensor:illuminance_lux]<50 )
    (set MQTT2_zigbee_study_desk_lamp on) 
  DOELSEIF ([MQTT2_zigbee_study_desk_lamp:state] eq "ON")
    ( set ECHO_G091GG10145703NX announcement {"message" : "I've not detected motion in the Study for a while, would you like me to keep the desk lights on?", "event_id" : "study_desk_lights_off", "response_intents" : <- "AMAZON.YesIntent", "AMAZON.NoIntent" -> } )

The '<-' '->' are used here instead of the json array characters '[' ']' since the DOIF code does a search and replace on those characters.

The response appears as a reading on the ECHO device

Code Auswählen Erweitern

study_desk_lights_off AMAZON.YesIntent

Code Auswählen Erweitern

define ECHO_G091GG10145703NX_notify notify ECHO_G091GG10145703NX:study_desk_lights_off:.*(AMAZON.NoIntent|NoAnswer) { fhem 'set MQTT2_zigbee_study_desk_lamp off'; }


The custom skill uses the intents of the home assistant actionable notifications solution for now.
Would probably drop the "Selection".

Code Auswählen Erweitern

{
    "interactionModel": {
        "languageModel": {
            "invocationName": "fhem custom skill",
            "intents": [
                {
                    "name": "AMAZON.CancelIntent",
                    "samples": []
                },
                {
                    "name": "AMAZON.HelpIntent",
                    "samples": []
                },
                {
                    "name": "AMAZON.StopIntent",
                    "samples": []
                },
                {
                    "name": "AMAZON.NavigateHomeIntent",
                    "samples": []
                },
                {
                    "name": "AMAZON.YesIntent",
                    "samples": [
                        "yes"
                    ]
                },
                {
                    "name": "AMAZON.NoIntent",
                    "samples": [
                        "no"
                    ]
                },
                {
                    "name": "Select",
                    "slots": [
                        {
                            "name": "Selections",
                            "type": "Selections"
                        }
                    ],
                    "samples": [
                        "{Selections}"
                    ]
                },
                {
                    "name": "Number",
                    "slots": [
                        {
                            "name": "Numbers",
                            "type": "AMAZON.FOUR_DIGIT_NUMBER"
                        }
                    ],
                    "samples": [
                        "{Numbers}"
                    ]
                },
                {
                    "name": "Duration",
                    "slots": [
                        {
                            "name": "Durations",
                            "type": "AMAZON.DURATION"
                        }
                    ],
                    "samples": [
                        "{Durations}"
                    ]
                },
                {
                    "name": "Date",
                    "slots": [
                        {
                            "name": "Dates",
                            "type": "AMAZON.DATE"
                        },
                        {
                            "name": "Times",
                            "type": "AMAZON.TIME"
                        }
                    ],
                    "samples": [
                        "{Dates} at {Times}",
                        "at {Times}",
                        "{Dates}"
                    ]
                }
            ],
            "types": [
                {
                    "name": "Selections",
                    "values": [
                        {
                            "name": {
                                "value": "Amazon Prime",
                                "synonyms": [
                                    "Amazon",
                                    "Amazon Video"
                                ]
                            }
                        },
                        {
                            "name": {
                                "value": "Hulu"
                            }
                        },
                        {
                            "name": {
                                "value": "YouTube"
                            }
                        },
                        {
                            "name": {
                                "value": "Netflix"
                            }
                        }
                    ]
                }
            ]
        }
    }

The custom skill uses the @justme-1968 lambda code https://github.com/justme-1968/alexa-fhem/blob/master/lambda.js.

37_echodevice.pm

Code Auswählen Erweitern

        } elsif ($type eq "announcement") {
                #Allgemeine Veariablen
                $SendUrl   .= "/api/behaviors/preview";
                $SendMetode = "POST";

                $SendData=~s/<-/[/g;
                $SendData=~s/->/]/g;
                my $json = eval { JSON->new->utf8(0)->decode($SendData) };

                if (ref($json) eq "HASH") {
                  if (defined($json->{message}) && defined($json->{event_id}) && defined($json->{response_intents}) && ref($json->{response_intents}) eq "ARRAY" ) {
                        my $announcement=$json->{message};
                        my $event_id=$json->{event_id};
                        my $response_intents=JSON->new->utf8(1)->encode($json->{response_intents});
                        $response_intents=~s/"/\\"/g;
                        my $skill_id="amzn1.ask.skill.xxxxx";
                        my $skill_args='\"echo_id\":\"'.$name.'\",\"event_id\":\"'.$event_id.'\",\"message\":\"'.$announcement.'\",\"response_intents\":'.$response_intents;

                        Log3 $name, 2, "[$name] [echodevice_SendCommand] announcement: $announcement";
                        Log3 $name, 2, "[$name] [echodevice_SendCommand] event_id: $event_id";
                        Log3 $name, 2, "[$name] [echodevice_SendCommand] response_intents: $response_intents";
                        Log3 $name, 2, "[$name] [echodevice_SendCommand] skill_args: $skill_args";

                        $SendData = '{"behaviorId":"PREVIEW","sequenceJson":"{\"@type\":\"com.amazon.alexa.behaviors.model.Sequence\",\"startNode\":{\"@type\":\"com.amazon.alexa.behaviors.model.SerialNode\",\"nodesToExecute\":[{\"@type\":\"com.amazon.alexa.behaviors.model.OpaquePayloadOperationNode\",\"type\":\"Alexa.Operation.SkillConnections.Launch\",\"operationPayload\":{\"targetDevice\":{\"deviceSerialNumber\":\"'.$hash->{helper}{".SERIAL"}. '\",\"deviceType\":\"'.$hash->{helper}{DEVICETYPE}.'\"},\"locale\":\"en-UK\",\"customerId\":\"' . $hash->{IODev}->{helper}{".CUSTOMER"} .'\",\"connectionRequest\":{\"uri\":\"connection://AMAZON.Launch/'.$skill_id.'\",\"input\":{'.$skill_args.'}}}}]}}","status":"ENABLED"}';
                        $SendDataL  = $SendData;
                  }
                }
        }



server.js

Code Auswählen Erweitern


diff /usr/local/lib/node_modules/alexa-fhem-backup/lib/server.js /usr/local/lib/node_modules/alexa-fhem/lib/server.js
613a614,631
> Server.prototype.setreading = function(device, reading, value) {
>   log.info ("Reading " + reading + " set to " + value + " for device " + device);
>   if ( this.connections )
>     for( let fhem of this.connections ) {
>       if (!fhem.alexa_device) continue;
>       fhem.execute('setreading ' + device + ' ' + reading + ' ' + value);
>     }
> }
>
> Server.prototype.deletereading = function(device, reading) {
>   log.info ("Delete reading " + reading + " for device " + device);
>   if ( this.connections )
>     for( let fhem of this.connections ) {
>       if (!fhem.alexa_device) continue;
>       fhem.execute('deletereading ' + device + ' ' + reading);
>     }
> }
>
1541a1560
>     log.info('#################### sessionId: ' +  event.session.sessionId);
1545a1565
>
1581c1601
<                          text: 'Hallo.'
---
>                          text: 'Hello.'
1582a1603,1611
>                      card: {
>                        type: 'Simple',
>                          title: 'fhem',
>                          content: 'Ok'
>                      //    image: {
>                      //     smallImageUrl: 'https://url-to-small-card-image...',
>                      //     largeImageUrl: 'https://url-to-large-card-image...'
>                      //    }
>                      },
1584c1613
<                      }
---
>                      },
1589,1608d1617
<       response.response.outputSpeech.text = 'Hallo. Wie kann ich helfen?';
< if( 0 ) {
<       response.response.directives = [
<       {
<         "type": "AudioPlayer.Play",
<         "playBehavior": "REPLACE_ALL",
<         "audioItem": {
<           "stream": {
<             "token": "string",
<             "url": "string",
<             "offsetInMilliseconds": 0
<           }
<         }
<       }
<     ];
< }
<       if( fhem && fhem.alexaConfirmationLevel < 2 )
<         response.response.outputSpeech.text = 'Hallo.';
<
<       response.response.reprompt = { outputSpeech: {type: 'PlainText', text: 'Noch jemand da?' } };
1614a1624,1655
>       log.info('###### session: ' +  session);
>       if (event.request.body)
>         log.info('###### event.request.body: ' +  JSON.stringify(event.request.body) );
>       if   (event.request.body
>          && event.request.body.event_id
>        && event.request.body.message
>        && event.request.body.echo_id
>        && event.request.body.response_intents
>        && Array.isArray(event.request.body.response_intents) ) {
>         if( !sessions[session] ) {
>           sessions[session] = {};
>           sessions[session].in_session = true;
>         }
>         sessions[session].event_id = event.request.body.event_id;
>         sessions[session].echo_id = event.request.body.echo_id;
>         sessions[session].message = event.request.body.message;
>         sessions[session].response_intents = event.request.body.response_intents;
>         log.info('#### sessions[session].event_id: ' + sessions[session].event_id);
>         log.info('#### : sessions[session].echo_id ' + sessions[session].echo_id);
>         log.info('#### : sessions[session].response_intents ' + sessions[session].response_intents);
>
>         this.deletereading(sessions[session].echo_id, sessions[session].event_id );
>         response.response.outputSpeech.text = sessions[session].message;
>         if( fhem && fhem.alexaConfirmationLevel < 2 )
>           response.response.outputSpeech.text = 'Hello.';
>         response.response.reprompt = { outputSpeech: {type: 'PlainText', text: sessions[session].message } };
>       } else {
>         response.response.outputSpeech.text = 'Hello. How can I help you?';
>         if( fhem && fhem.alexaConfirmationLevel < 2 )
>           response.response.outputSpeech.text = 'Hello.';
>         response.response.reprompt = { outputSpeech: {type: 'PlainText', text: 'Someone there?' } };
>       }
1616c1657,1660
<       in_session = false;
---
>       if    (sessions[session] && sessions[session].event_id
>           && sessions[session].echo_id && sessions[session].response_intents
>           && ! sessions[session].response_intent)
>         this.setreading(sessions[session].echo_id, sessions[session].event_id ,"NoAnswer");
1622a1667,1668
>
>       log.info('#### sessionId: ' +  event.session.sessionId);
1709c1755,1783
<         }
---
>         } else if (intent_name == 'Number'
>               || intent_name == 'Duration'
>               || intent_name == 'Date'
>               || intent_name == 'Select'
>               || intent_name == 'AMAZON.YesIntent'
>               || intent_name == 'AMAZON.NoIntent') {
>         if    (sessions[session]
>             && sessions[session].event_id
>             && sessions[session].echo_id
>             && sessions[session].response_intents)
>         {
>             const r_intent = sessions[session].response_intents.find(r_intent => r_intent == intent_name);
>           if (r_intent) {
>               log.info('#### 1 sessions[session].event_id: ' + sessions[session].event_id);
>               sessions[session].response_intent = intent_name;
>               this.setreading(sessions[session].echo_id, sessions[session].event_id ,intent_name);
>             match = true;
>             in_session=false;
>               response.response.shouldEndSession = true;
>           } else {
>               log.info('#### 2 sessions[session].event_id: ' + sessions[session].event_id);
>               response.response.outputSpeech.text = 'Sorry, did not get that. ' + sessions[session].message;
>               response.response.reprompt = { outputSpeech: {type: 'PlainText', text: sessions[session].message } };
>               response.response.shouldEndSession = false;
>               callback( response );
>               return;
>           }
>         }
>       }
1879a1954,1957
>       if    (sessions[session] && sessions[session].event_id
>           && sessions[session].echo_id && sessions[session].response_intents
>           && ! sessions[session].response_intent)
>         this.setreading(sessions[session].echo_id, sessions[session].event_id,"AMAZON.StopIntent");
1881c1959
<         response.response.outputSpeech.text = 'Bis bald.';
---
>         response.response.outputSpeech.text = 'OK.';
1887a1966
>
1888a1968,1971
>       if    (sessions[session] && sessions[session].event_id
>           && sessions[session].echo_id && sessions[session].response_intents
>           && ! sessions[session].response_intent)
>         this.setreading(sessions[session].echo_id, sessions[session].event_id,"AMAZON.CancelIntent");
1896d1978
<
1899a1982,1986
>       if    (sessions[session] && sessions[session].event_id
>           && sessions[session].echo_id && sessions[session].response_intents
>           && ! sessions[session].response_intent)
>         this.setreading(sessions[session].echo_id, sessions[session].event_id,"AMAZON.HelpIntent");
>
2014c2101
<           response.response.outputSpeech.text = 'Das habe ich leider nicht verstanden.';
---
>           response.response.outputSpeech.text = 'Sorry, I did not get that.';
2255,2256c2342
<         response.response.outputSpeech.text = 'Das habe ich leider nicht verstanden';
<
---
>         response.response.outputSpeech.text = 'Sorry, I did not get that.';
2267a2354
>     response.response.card.content = response.response.outputSpeech.text;



/media/etc/fhem/alexa-fhem.cfg

Code Auswählen Erweitern

{
    "alexa": {
        "port": 3000,
        "name": "fhem custom skill",
        "keyFile": "./certs/server-key.pem",
        "certFile": "./certs/server-cert.pem",
        "applicationId": "amzn1.ask.skill.xxxx",
        "oauthClientID": "amzn1.application-oa2-client.xxxx"
    },
   "connections" : [
      {
         "server" : "127.0.0.1",
         "name" : "FHEM",
         "port" : "8083",
         "webname" : "fhem",
         "filter" : "alexaName=..*",
         "uid" : 999,
         "ssl": true,
         "auth": {"user": "xxx", "pass": "xxx"}
      }
   ],
   "sshproxy" : {
      "description" : "FHEM Connector",
      "ssh" : "/usr/bin/ssh"
   }
}


key open topics:
   - have not found a way around opening port 3000 ....
   - for some reason, I'm not getting a SessionEndedRequest when there is no response from the user.
      In the scenario of, "... no motion, shall I turn off the lights", not receiving an answer is exactly the use case where one would want to turn off the light .... hence not getting a SessionEndedRequest is a problem.
      Really would not like to hack a solution based on a timer or similar, SessionEndedRequest must come.
   - clean up
        custom skill id is hardcoded in 37_echodevice
        server.js code clean up


Any suggestions regarding the current design and implementation?

Cheers,
Paulo
       

[MadMax-FHEM]

Hey Paulo,

that sounds really GREAT!

Maybe I'll (have to   ) get some time before Easter...

But (sorry): no ideas regarding your open points...

Timer also was my first idea or (in my case) an at in fhem.
So "asking" and defmod an at for x seconds and if no response the "default action" is performed...

For the port 3000 issue I think that will remain?
If there would be a solution also for Custom Skills I think Andre would have posted one...

Keep on going!!

Bye, Joachim

[plopes9000]

Hi Joachim,

Regarding the SessionEndedRequest I just tested on an echo dot and it works fine, I do get a SessionEndedRequest. Hence the issue is restricted to echo show type devices.

I've also removed the hardcoded skill id on the 37_echodevice.pm file in favour of an attribute at either the echomaster or a specific echo device.

Code Auswählen Erweitern

echomaster attributes
userattr annoucement_skill_id
annoucement_skill_id amzn1.ask.skill.xxxx


37_echodevice.pm

Code Auswählen Erweitern


                my $skill_id = AttrVal($hash->{IODev}{NAME},"annoucement_skill_id","");
                $skill_id = AttrVal($name,"annoucement_skill_id","") if($skill_id eq "");
                if($skill_id eq "") {
                   Log3 ($name, 4, "[$name] [echodevice_SendMessage, announcement] missing annoucement_skill_id attribute");
                   return;
                }




updated key open topics:
   - have not found a way around opening port 3000 ....
   - not getting a SessionEndedRequest on echo show devices when there is no response from the user.
   - clean up
        server.js code clean up



Best Regards,
Paulo

[justme1968]

falls es noch von interesse ist:

wenn man dinge probieren möchte die mit dem smart home api nicht gehen bietet sich fast immer custom skill variante an.

es gibt eigentlich 'nur' zwei nachteile: man muss den skill namen aktiv mit ansagen und man muss sich etwas einarbeiten . dafür kann man so ziemlich alles umsetzen was man möchte.


die wiki beschreibung zur einrichtung auf amazon seite ist leider veraltet, aber das prinzip ist immer noch das gleiche. auf fhem seite ist die nötige funktionalität wie gehabt im alexa modul und alexa-fhem enthalten. eine alexa-fhem installation kann problemlos den fhem connector, einen custom skill mit port weiterleitung und belieb viele fhem instanzen gleichzeitig bedienen. das alexa modul unterstützt beim erstellen der skill konfiguration.

die eigentliche logik wird komplett auf fhem seite abgedeckt, fhem kann auf variable parameter im kommando zu greifen, fhem gibt eine gesprochen antwort über den echo und kann optional auf die fortsetzung des dialoges warten. in der fhem rückgabe können auch alle möglichkeiten von ssml (https://developer.amazon.com/de/blogs/alexa/post/5c631c3c-0d35-483f-b226-83dd98def117/new-ssml-features-give-alexa-a-wider-range-of-natural-expression) wie z.b. flüstern oder unterschiedliche stimmen genutzt werden.

fhem hat hier auch zugriff auf den raum in dem ein echo steht und auf die stimme des sprechenden falls eingerichtet.

ein beispiel das anhand eines sprach kommandos rechenoperationen durchführt findet sich z.b. hier: https://forum.fhem.de/index.php/topic,67490.msg589378.html#msg589378 .

[plopes9000]

@justme1968, that is indeed an interesting functionality of alexa-fhem that I was not ware of.

However since one has to call up the skill, it does not really fit the use case here - we would want the echo device to prompt a question on demand (eg. based on an automated trigger) rather them by user request.

I did get it to work well, except for the points I last raised:
   a) have not found a way around opening port 3000 ....
   b) not getting a SessionEndedRequest on echo show devices when there is no response from the user.

b) is a general problem with alexa skills behaviour on echo shows for which there does not appear to be a solution at the moment - since a while actually.

In the end after getting it to work, the benefit versus having to leave port 3000 open i.e. "a)", with the unsurmountable issue of "b)" - I have 10+ echo dot/flex and 4 echo shows - does not make it feel worthwhile for me.

Hence I have disabled it for now.

Thank you for responding and hinting at other options and thank you very much for all the work put into alexa-fhem, connector, etc.
Paulo

[balli1187]

@justme1968: danke für deine Rückmeldung.

Leider ist das "Aktive Ansagen des Skill-Namen" genau der Punkt, der mich am meisten vom Custom-Skill abhält. Ich persönlich empfinde das einfach nicht als "natürlich" (aber das ist natürlich mein Problem).

Vielleicht probiere ich es trotzdem mal aus, um mich mal ein wenig damit auseinander zu setzen. Bei den ganzen vorherigen Posts von Paulo, komme ich nicht so recht mit...
Vielleicht könnte man aber so ein paar Sachen des Custom Skills mit Hilfe des echodevice-Moduls triggern. Dieses bietet ja die Möglichkeit per textcommand einen (ansonsten gesprochenen) Befehl an amazon zu übergeben.
Wenn ich jetzt keinen Knoten im Hirn habe, könnte ich damit ja vielleicht die ersten Punkt umgehen?!
Bleibt nur die Frage ob sich die Abläufe dann so gestalten lassen, dass im Anschluss an ein (fiktives, ggf. kryptisches) Kommando die richtigen Abläufe (in dem Fall die Nachfragen auf einem bestimmtem Echo) abbilden lassen....

[locodriver]

Ich habe etwas gefunden, was thematisch hier rein passt:

Ab 17:21 wird eine Querlüftung beschrieben und in der Videobeschreibung auf den ursprünglichen Beitrag verwiesen.

K.A., ob das hilfreich ist, hier in dieser Richtung weiter zu kommen...

https://www.youtube.com/watch?v=KEe1FJ6szKw

https://github.com/fabiankrauss/iobroker.scripte

Vielleicht können die "Experten" damit was für fhem basteln...?!