Configuration Tutorial for Liuli on Raspberry Pi

This article introduces the configuration and deployment tutorial of Liuli on Raspberry Pi.

Liuli^[1] aims to help you build a multi-source, clean, and personalized reading environment in one place. It is still in slow development and iteration, and more friends are welcome to participate. The current applicable scenarios include:

• Building a pure RSS public account information flow based on Liuli
• Following updates & reading novels based on Liuli

It is recommended to read the article above to have a general impression. Next, I will introduce how to install Liuli on Raspberry Pi in the following ways:

• Docker deployment
• Source code deployment

PS: This project is open source, see github.com/liuli-io/liuli, welcome to give a Star🤟🏻

Install MongoDB [Important]

Since Liuli depends on MongoDB as its database, and enabling MongoDB under Docker on Raspberry Pi is a bit confusing, we will install it directly on the system. Additionally, some friends may not have high configurations and cannot run Docker, so if deploying from source later, it is necessary to install it on the system.

Install MongoDB:

# Upgrade and update
sudo apt update
sudo apt upgrade

# Add the software source in `/etc/apt/sources.list`
wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list

# Install MongoDB
sudo apt-get update
sudo apt-get install -y mongodb-org

# Start
sudo systemctl daemon-reload
sudo systemctl enable mongod
sudo systemctl start mongod
# Check status
sudo systemctl status mongod

# Enter MongoDB
mongo

Create user:

use admin
db.createUser(
    {
        "user": "liuli",
        "pwd": "liuli",
        "roles": [
            "userAdminAnyDatabase",
            "dbAdminAnyDatabase",
            "readWriteAnyDatabase"
        ]
    }
)
exit

Configure operation sudo vim /etc/mongod.conf, change as follows:

# Change bindIp to 0.0.0.0, accessible externally
net:
  port: 27017
  bindIp: 0.0.0.0
# Add login verification
security:
   authorization: enabled

Restart to take effect:

sudo systemctl restart mongod

At this time, log in:

# Note your own ip
sudo mongo --host "192.168.0.2" -u "liuli" -p "liuli"

# View databases
> db.adminCommand({listDatabases: 1})

Docker Deployment

Overall, Docker deployment is still relatively convenient, especially for friends who are not familiar with Python, so this method is placed first.

Install Docker

First, let’s install Docker:

sudo apt-get install apt-transport-https ca-certificates software-properties-common -y

curl -fsSL https://get.docker.com -o get-docker.sh
# If the network speed is slow, you can try: sudo sh get-docker.sh --mirror Aliyun
sudo sh get-docker.sh

# Add key
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/raspbian/gpg | sudo apt-key add -

# Add software source in `/etc/apt/sources.list`
sudo add-apt-repository \
    "deb [arch=armhf] https://mirrors.aliyun.com/docker-ce/linux/raspbian \
    $(lsb_release -cs) \
    stable"

sudo groupadd docker
sudo usermod -aG docker $USER

# Update
sudo apt-get update
sudo apt-get upgrade

# Refresh
newgrp docker

# Start
sudo systemctl enable docker
sudo systemctl start docker

# Test
sudo docker run hello-world

You can consider introducing portainer for management:

sudo mkdir -p /data/docker_data/portainer_data
sudo docker run -d -p 8001:8000 -p 9001:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /data/docker_data/portainer_data:/data portainer/portainer-ce

Docker Deployment

Next is the official deployment:

# Download api and schedule images
sudo docker pull liuliio/api:v0.1.3
sudo docker pull liuliio/schedule:v0.2.4

# Create configuration directory
sudo mkdir -p liuli/liuli_config
sudo touch liuli/pro.env
sudo touch liuli/liuli_config/wechat.json

First, let’s look at the pro.env configuration:

PYTHONPATH=${PYTHONPATH}:${PWD}
LL_M_USER="liuli"
LL_M_PASS="liuli"
# Your externally accessible ip
LL_M_HOST="192.168.0.2"
LL_M_PORT="27017"
LL_M_DB="admin"
LL_M_OP_DB="liuli"
LL_FLASK_DEBUG=0
LL_HOST="0.0.0.0"
LL_HTTP_PORT=8765
LL_DOMAIN=""
LL_WORKERS=1
# The above configurations do not need to be changed, the following need to be configured individually
# Whether to send notifications to DingTalk
LL_D_TOKEN=""
# Whether to send notifications to WeChat
LL_WECOM_ID=""
LL_WECOM_AGENT_ID="-1"
LL_WECOM_SECRET=""
# Create a repository liuli_backup on your github
# You can also leave it blank, which means no backup to github
LL_GITHUB_TOKEN=""
LL_GITHUB_REPO="{your_github_name}/liuli_backup"
LL_GITHUB_DOMAIN="https://{your_github_name}.github.io/liuli_backup/"

Please refer to the following documents for specific parameter configuration, it’s very simple:

• Dispatcher Configuration^[2]: DingTalk WeChat configuration tutorial
• Backup Configuration^[3]: GitHub backup configuration tutorial
• Parameter Description^[4]: Detailed introduction of each parameter

After configuring, please write the above content to sudo vim liuli/pro.env.

Next, directly configure the public account related, directly use Liuli official configuration sudo vim liuli/liuli_config/wechat.json:

{
    "name": "wechat",
    "author": "liuli_team",
    "doc_source": "liuli_wechat",
    "collector": {
        "wechat": {
            "wechat_list": [
                "Small Crowd News"
            ],
            "delta_time": 5,
            "spider_type": "sg_ruia",
            "spider_type_des": "When the image is schedule:playwright_*, spider_type can be filled in sg_playwright"
        }
    },
    "processor": {
        "before_collect": [],
        "after_collect": [{
            "func": "ad_marker",
            "cos_value": 0.6
        }, {
            "func": "to_rss",
            "doc_source_list": ["liuli_wechat"],
            "link_source": "github"

        }]
    },
    "sender": {
        "sender_list": ["wecom"],
        "query_days": 7,
        "delta_time": 3
    },
    "backup": {
        "backup_list": ["mongodb"],
        "query_days": 7,
        "delta_time": 3,
        "init_config": {},
        "after_get_content": [{
            "func": "str_replace",
            "before_str": "data-src=\"",
            "after_str": "src=\"https://images.weserv.nl/?url="
        }]
    },
    "schedule": {
        "period_list": [
            "00:10",
            "12:10",
            "21:10"
        ]
    }
}

At this time, the folder directory is as follows:

(base) ➜  cd liuli
(base) ➜  liuli tree -L 2
.
├── liuli_config
│   └── wechat.json
└── pro.env

Configuration is complete, and now you can start:

# Start API
sudo docker run -d -it --restart=always -p 8765:8765 -v $PWD/pro.env:/data/code/pro.env --name liuli_api liuliio/api:v0.1.3
# Start schedule
sudo docker run -d -it --restart=always -v $PWD/pro.env:/data/code/pro.env -v $PWD/liuli_config:/data/code/liuli_config --name liuli_schedule liuliio/schedule:v0.2.4

Successful logs are as follows:

At this time, you can verify by accessing the following interface:

• Get the public account rss address: http://0.0.0.0:8765/rss/liuli_wechat/Small Crowd News/
• View the articles backed up from the target public account: http://0.0.0.0:8765/backup/liuli_wechat/Public Account Name/Article Name

At this point, you can subscribe to and view the articles of the target public account.

Note ⚠️: Fill in the IP based on the IP of the service you deployed.

The public account rss address effect:

The target public account backup article:

If you need WeChat message prompts, please visit: Dispatcher Configuration | DingTalk WeChat Configuration Tutorial^[5]

Source Code Deployment

If you are looking at the source code deployment directly, please read the Docker Deployment section first, copy the following files:

• wechat.json
• pro.env

First, download the source code:

git clone https://github.com/liuli-io/liuli.git

# Remove the following lines in Pipfile
# black = "*"
# isort = "*"
# pylint = "*"
# pytest = "*"
# pandas = "*"
# numpy = "*"
# pypinyin = "*"
# playwright = "*"

# Ensure you have a python3.7+ environment
pip install pipenv
pipenv install --python {your_python3.7_path} --skip-lock --dev

# Configuration
# Copy the contents of pro.env above
vim pro.env
# Delete all contents of the liuli_config folder
# Copy wechat.json to liuli_config/wechat.json

# Start API
bash ./start.sh api pro
# Start schedule
bash ./start.sh schedule pro

View logs:

((liuli) ) (base) ➜  liuli git:(main) ✗ bash ./start.sh api pro
Start api(pro) serve: PIPENV_DOTENV_LOCATION=./pro.env pipenv run gunicorn -c src/config/gunicorn.py src.api.http_app:app
Loading .env environment variables...
[2022-04-28 15:00:50 +0800] [29074] [INFO] Starting gunicorn 20.1.0
[2022-04-28 15:00:50 +0800] [29074] [INFO] Listening at: http://0.0.0.0:8765 (29074)
[2022-04-28 15:00:50 +0800] [29074] [INFO] Using worker: gevent
[2022-04-28 15:00:50 +0800] [29079] [INFO] Booting worker with pid: 29079
[2022:04:28 15:00:54] INFO  Liuli API server(v0.1.3) started successfully :)
...

Instructions

The above process refers to the following materials:

• Install & Configure MongoDB on the Raspberry Pi^[6]
• How to install Conda and Docker on your Raspberry Pi^[7]
• Docker Entry to Practice – Raspberry Pi^[8]
• Docker Official Documentation Install Docker Engine^[9]

Hello to friends who see this, I tried the new feature of WeChat Quick Private Message, I don’t know what it is, everyone can help me try it out.

If you have any questions during use, feel free to join the group for discussion:

Reference Materials