Integrating Tmall Genie with Raspberry Pi

0x00 Introduction

This idea has been around for a long time, but the information on Baidu is mostly about binding Tmall Genie’s voice skills with software like Domoticz or Home Assistant. When I tried to use it, I encountered issues such as software not being installed or operations failing. So I thought about manually opening the GPIO pins with PHP, but the output voltage was not very stable. When reading the level with Arduino, it kept jumping between high and low (maybe the Raspberry Pi outputs digital signals?), making it impossible to determine the switch signal… Therefore, I used serial communication.

0x01 Materials

• Raspberry Pi 4B (Other models: 3A+, 3B+, 3B, 2B, 1B+, 1A+)

• Arduino development board (any model)

• 2 female-to-male Dupont wires

• 2 crocodile clip wires

• One LED, buzzer, or other components that can be used

• MicroSD card (for installing Raspbian, at least 8GB)

• Card reader (for installing Raspbian)

• One VPS (for Frp internal penetration; if you choose Ngrok or SakuraFrp, or use DDNS (or fixed IP) and open port 80 for WebHook installation, you may not need a VPS)

  

0x02 Environment

Please refer to the official tutorial for installing Raspbian: Installing Images

After installation, you need to install Nginx and PHP packages, or directly install the LNMP/LAMP stack.

sudo apt-get install nginx php7.0-fpm php7.0-cli php7.0-curl php7.0-cgi php7.0-devsudo service nginx startsudo service php7.0-fpm restart

Then link PHP to Nginx by modifying /etc/nginx/sites-available/default

Find the index in the configuration file and add index.php:

index index.php index.html index.htm index.nginx-debian.html;

Then find # pass PHP scripts to FastCGI server and modify part of the configuration below it:

location ~ \.php$ {include snippets/fastcgi-php.conf;## # With php-fpm (or other unix sockets):fastcgi_pass unix:/run/php/php7.3-fpm.sock;# # With php-cgi (or other tcp sockets):#fastcgi_pass 127.0.0.1:9000;

sudo systemctl restart nginx # Restart Nginx

Then create a phpinfo.php in /var/www/html with the content:

<?php phpinfo(); ?>

Then visit http:// Raspberry Pi ip/phpinfo.php to see if you can access it.

If it works, continue.

pip3 install pyserial

C++ serial communication is too troublesome, so here we use Python’s PySerial module to communicate with Arduino.

At this point, the runtime environment is set up. Here’s a wiring diagram:

0x03 Code

Now you need to know the path corresponding to the USB port you plugged into the Raspberry Pi.

Do not plug in the development board yet, execute once:

ls /dev/tty*

Then plug in the development board and execute the above command again to see which file has been added.

For example, the USB port I plugged into the Raspberry Pi is /dev/tty-ACM0 (some models show as /dev/ttyUSB0).

Remember it for next time when you plug in this port.

Then plug the development board into your computer and start coding (using Arduino IDE).

void setup() {Serial.begin(9600);pinMode(6,OUTPUT);}void loop() {  if(Serial.available() > 0){    int data=Serial.read();    if(data=='a'){      digitalWrite(6,HIGH);      Serial.println("ON");    }else{    if(data=='b'){      digitalWrite(6,LOW);      Serial.println("OFF");    }    }  }}

The Arduino board’s code mainly consists of two parts: setup and loop, structured by two void functions.

The setup part of the code executes only once when the development board is powered on.

void setup(){Serial.begin(9600);Serial.println("U can only c me once");}void loop(){}

However, the loop part of the code will execute repeatedly until the board loses power or receives a RST signal.

void setup(){Serial.begin(9600);Serial.println("U can only c me once");}void loop(){Serial.println("I'm everywhere");}

The second line of the code uses the Serial.begin() function to open the USB serial port of the development board with a baud rate of 9600.

The third line uses the pinMode() function to set pin 6 (if you want to change to another pin, you need to modify lines 9 and 13 as well) to output mode. At this point, pin 6’s voltage is 0V (without setting the pin mode, each pin’s voltage is about 1.2V) to avoid interference.

The sixth line uses the Serial.available() function to check if there is input on the USB serial port. If there is input, it checks whether the input data is ‘a’ or ‘b’ (customizable, if modified the Python script needs to be adjusted accordingly) and adjusts the voltage level of pin 6 accordingly (a = high level = ON, b = low level = OFF) and writes the operation result (ON/OFF, this can be unified as ok or other data, but after unifying the data, the Python script needs to define a variable to identify the operation for WebHook PHP recognition).

It is important to note that when outputting data, you must use println instead of print, otherwise the Python script will timeout when reading data.

Python script:

import sysimport serialser=serial.Serial("/dev/ttyACM0",9600,timeout=10)if sys.argv[1]=="on":    ser.write("a")    rtn=ser.readline()    if rtn=="ON\r\n":        print "ok",        ser.close()    else:        print "fail",        ser.close()if sys.argv[1]=="off":    ser.write('b')    rtn=ser.readline()    if rtn=="OFF\r\n":        print "ok",        ser.close()    else:        print "fail",        ser.close()if sys.argv[1]!="on" and sys.argv[1]!="off":    print "fail",    ser.close()

Lines 1 and 2 import the supporting libraries. The sys library mainly provides the argv array, just like in C++:

int main(int argc,char* argv[]){return 0;}

The third line uses the PySerial’s Serial function to open a serial port, with the usage as follows:

serial.Serial (string serial port path, which is the path of the development board obtained in 0x01, int baud rate, int maximum wait time, must assign a value as timeout)

Lines 4, 13, and 22 check the command line arguments passed in. Note that argv[0] is the script path (e.g., /home/pi/light.py), and argv[1] starts with the command line arguments.

Lines 5 and 6 write data to the serial port and use the readline function to read the returned data from the serial port.

The development board uses println to output serial data, and a CR LF is used to create a newline. When comparing data in Python, you need to include \r\n.

If you changed the control instructions (a and b) when burning the development board, the Python script here must also be modified accordingly.

Then output ok or fail based on the returned data (it is important not to unify the data, as you need to determine whether the operation was successful). The operations of the Python script are mainly these.

Then start setting up the PHP webhook but first do some operations.

ps -aux | grep php# This command finds all users running PHP programssudo chmod 777 /etc/sudoers

Then find root ALL=(ALL) ALL in /etc/sudoers.

Add a line below it:

www-data ALL=(ALL) NOPASSWD: ALL

The bolded www-data needs to be found by executing the ps -aux | grep php command to find the username corresponding to your php-fpm program. For example, here is:

So www-data is the user corresponding to your php-fpm program (Do not find the program with master), replace it to write into the /etc/sudoers file.

Then restore the permissions of the sudoers file, or else you won’t be able to use sudo:

pkexec chmod 0440 /etc/sudoers

Then log in to the Tmall Genie account bound to your Taobao account. After logging in, a prompt will pop up for authentication; authenticate it.

Then

After creating the skill, we need to train it a bit.

Then enter the intentions and start adding single-turn dialogue materials.

Try to input as many phrases as users might say, one at a time, and press enter after writing each one.

After writing, use the mouse to select user operations (ON, OFF), and it will let you choose an entity.

Click on the entity you just added; it will automatically match the remaining phrases.

Then use the same method to associate [ON] with the entity.

Next, configure the [Parameters] below.

If you don’t do this, the Tmall Genie will become a little mute when it cannot match parameters, neither turning on the light nor responding.

Setting required parameters and allowing the genie to ask follow-up questions can help execute user commands when NLU cannot recognize parameters.

Then we need to start setting up the WebHook.

First, here’s an example request from Tmall:

POST http://your-webhook-service.com/skill/weather
 Headers: // Defaults are filled in, specifies the data format in the request body Content-type: application/json // User-defined headers //key1: value1 //key2: value2
POST body:{ "sessionId": "b112a091-1523-4d2d-8059-e09461dafd73", "utterance": "What’s the weather today in Shanghai?", "token": "ozkYw9Y8*******lffDM", "requestData": { "userOpenId": "XXXXXXXX==", "deviceOpenId": "YYYYYYYYY==", "city": "Shanghai" }, "botId": 10, "domainId": 12345, "skillId": 23456, "skillName": "Super Weather", "intentId": 34567, "intentName": "weather", "slotEntities": [ { "intentParameterId": 45678, "intentParameterName": "city", "originalValue": "Shanghai", "standardValue": "Shanghai", "liveTime": 0, "createTimeStamp": 1564110905331, "slotName": "city:city", "slotValue": "Shanghai" }, { "intentParameterId": 56789, "intentParameterName": "time(公共实体)", "originalValue": "today", "standardValue": "today", "liveTime": 0, "createTimeStamp": 1564110905331, "slotName": "time(公共实体):sys.time", "slotValue": "today" } ], "requestId": "20190726111511958-508551760", "device": { }}

Here we only need to pay attention to one value, which is the standardValue inside slotEntities. This is the entity parameter matched after NLU understands the semantics, and we only need to extract this part of the information.

However, PHP only stores the source content when the Content-Type header is application/x-www-form-urlencoded or multipart/form-data.[Math Processing Error]

In POST, Tmall Genie’s request Content-Type is application/json, and the POST data will not be stored.POST inside

How can we get the POST data in this case?

PHP provides a file path for the script to obtain the original POST data.

$json=file_get_contents("php://input");

Using the file_get_contents function to get php://input, PHP returns the raw data of the current POST request.

With this, we can parse the JSON sent by Tmall Genie.

$json=file_get_contents("php://input");$data=json_decode($json,true);$opt=$data['slotEntities'][0]['standardValue'];/*In the example JSON, slotEntities is an array (internal data is enclosed in square brackets), when parsing in PHP, it needs to include a [0] to indicate this is the first array (PHP array index starts at 0, most programming languages do the same, except for Easy Language). When Tmall Genie matches multiple parameters, it returns multiple data in slotEntities, and we need to use the count function to get the number of array members and then read each one with a for loop. Here there are only the ON and OFF parameters, so there is only one member in the array, no need to count.*/$output=[2]; // Define an array to store the return data of the exec functionif($opt=="ON"){  exec("sudo python /home/pi/light.py on 2>&1",$output); // The significance of adding the www-data user in sudoers is that serial communication requires root permissions. Sudo requires a password, but PHP cannot do this. Setting NOPASSWD in sudoers allows PHP to use sudo without a password. verify(0,$output[1]); // Verify the output of the python script: why use the second array member is because the first member is content 2, the second member is the return content}elseif($opt=="OFF"){    exec("sudo python /home/pi/light.py off 2>&1",$output);    verify(1,$output[1]);}else{  echo("fail"); // The incoming JSON does not follow the rules, you throw an error and I throw an error}

At this point, the content processing of the WebHook and the light switch operation are completed. Next, we need to return a response JSON to the server.

First, here’s a sample response JSON:

{ "returnCode": "0", "returnErrorSolution": "", "returnMessage": "", "returnValue": { "reply": "Today’s weather in Shanghai is cloudy, with a temperature of 15 to 20 degrees, southeast wind level 3", "resultType": "RESULT", "executeCode": "SUCCESS", "msgInfo": "" }}

This format is much simpler and easy to construct.

function rtn($a,$isSuccess){// Parameter a is the operation passed from the verify function, 0 for turning on the light, 1 for turning it off. // isSuccess indicates whether the python script returns ok if($isSuccess==1){ if($a==0){ $rtntext="Okay master, the light has been turned on for you."; }elseif($a==1){ $rtntext="Okay master, the light has been turned off for you."; } }else{ $rtntext="The operation failed, master needs to modify the code."; } $rtn_array=array(); // Define an array $rtn_array['returnCode']=0; // Return code is 0, indicating success $rtn_array['returnErrorSolution']=""; // Error solution, leave blank $rtn_array['returnMessage']=""; // Return message, leave blank $rtn_array['returnValue']['reply']=$rtntext; // Reply content, already defined at the beginning $rtn_array['returnValue']['resultType']="RESULT"; // Reply type, generally RESULT, ASK_INF for insufficient parameters and need to continue asking $rtn_array['returnValue']['executeCode']="SUCCESS"; // Execution status, success SUCCESS $rtn_array['returnValue']['msgInfo']=""; // Leave blank $response_json=json_encode($rtn_array); // Convert array to JSON format echo($response_json); // Output JSON/* Note that when writing returnValue data, it cannot be written as ['returnValue'][0]['xxxx'], because you do not need to return an array to let Tmall Genie say multiple messages, just need to include several sub-items. */}

At this point, the WebHook has been completed. Combine the above code to form the entire light.php.

/* *0=POWER ON & FALSE *1=POWER OFF & TRUE */function rtn($a,$isSuccess){ if($isSuccess==1){ if($a==0){ $rtntext="Okay master, the light has been turned on for you."; }elseif($a==1){ $rtntext="Okay master, the light has been turned off for you."; } }else{ $rtntext="The operation failed, master needs to modify the code."; } $rtn_array=array(); $rtn_array['returnCode']=0; $rtn_array['returnErrorSolution']=""; $rtn_array['returnMessage']=""; $rtn_array['returnValue']['reply']=$rtntext; $rtn_array['returnValue']['resultType']="RESULT"; $rtn_array['returnValue']['executeCode']="SUCCESS"; $rtn_array['returnValue']['msgInfo']=""; $response_json=json_encode($rtn_array); echo($response_json);}function verify($b,$out){ if($out=="ok"){ rtn($b,1); }else{ rtn($b,0); }}$json=file_get_contents("php://input");$data=json_decode($json,true);$opt=$data['slotEntities'][0]['standardValue'];$output=[2];if($opt=="ON"){ exec("sudo python /home/pi/light.py on 2>&1",$output); verify(0,$output[1]);}elseif($opt=="OFF"){ exec("sudo python /home/pi/light.py off 2>&1",$output); verify(1,$output[1]);}else{ echo("fail");}

Download the authentication file and place it in the /var/www/html/aligenie directory. There is definitely no aligenie folder, create one and put the authentication file inside, then

Fill in your light.php address after internal penetration, for example, mine is http://47.***.***.***/light.php.

After filling it in, save it, and the system will automatically verify your authentication file.

If it prompts [The server is having a small issue, fixing it], do not think it is Alibaba’s problem; that is your server access timeout, check it out.

Then we can go to the online test to debug it.

After testing and confirming there are no issues, there is a real machine test.

Then you can say to your cat genie to turn the light on or off.

0x04 References

(1) Custom Skill Access – AliGenie Voice Skill Open Platform

(2) Step-by-step guide on using frp for internal penetration, for remote desktop and HTTP access – Jianshu Lastly, please everyone double-click! Double-click! Double-click!