Raspberry Pi 4B Guide: Setting Up Your Own Private Cloud Storage

Introduction

It has been almost a month since I unboxed the Raspberry Pi 4B. During this time, I have tried running several services on the Raspberry Pi. Overall, the 8GB version’s memory is indeed excessive; except for once when I built an aarch64 architecture Docker image, the memory usage peaked at 40%, while at other times it was around 20%. Today, I will talk about how to use the Raspberry Pi to create a private cloud storage. Uploading and downloading at 50MB/S is quite feasible, and the two open-source cloud storage solutions I will introduce have cross-platform clients and support streaming playback, essentially making it a budget NAS.

Situation Overview

My Raspberry Pi is connected directly to a gigabit router via an Ethernet cable, and the cables in my room are all category six, which barely counts as a gigabit internal network. Previously, I was using a 64GB Kingston TF card in the Raspberry Pi, but using a memory card for cloud storage has its limitations, not to mention the read/write bottleneck and capacity issues. (This issue can be alleviated by attaching an external hard drive.) Instead, I chose to boot the system directly from a 1TB mechanical drive via USB 3.0, completely discarding the TF card.

In my room, it basically looks like this, together with the router, emmmm it counts as a file server and also a small aarch64 server. In the future, it might also serve as a data processing center for home environmental sensors? (Liu is trying to force in some high-level vocabulary again orz)

The Raspberry Pi 4B supports native USB booting, but it requires a relatively new bootloader version, which might necessitate flashing the EEPROM. I won’t elaborate on this part; interested readers can check out my previous article. (Oops, I forgot I haven’t written that article yet).

I am using Ubuntu Server 20.10 for the system. The reason for choosing 20.10 instead of 20.04 is that 20.04 cannot boot directly from USB and requires additional configurations (which cause a significant drop in hard disk read/write performance; I haven’t found the reason for this, but booting the hard disk on 20.04 via TF card does not have this issue), whereas version 20.10 can boot directly. Of course, if you run Ubuntu 20.04 on a TF card, following this tutorial will work without any issues.

Why choose Ubuntu? Because the Raspberry Pi is an officially certified device for Ubuntu, ensuring strong reliability.

Ubuntu 20.04 LTS is certified for the Raspberry Pi^[1]

Raspberry Pi and Ubuntu: 2020 roundup^[2]

Regarding the choice of cloud storage, I opted for ownCloud and nextCloud, deploying them using Docker containers due to ease of deployment and good isolation.

Without further ado, let’s get started, mainly command configurations, and following the tutorial should minimize errors.

Preparation

Using Tsinghua University’s Source

Since Ubuntu’s own source is somewhat slow to access domestically, we will use the Tsinghua source. Please choose the corresponding source based on your system version.

# First, back up the source list file
sudo cp -p /etc/apt/sources.list /etc/apt/sources.list.bak
# Modify the apt source file, commenting out Ubuntu's own sources.
sudo vim /etc/apt/sources.list

Ubuntu 20.04.1 LTS

# The source mirror for the source code is commented out by default to speed up apt update, uncomment if needed
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-security main restricted universe multiverse

# Pre-release software sources are not recommended to enable
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-proposed main restricted universe multiverse

Ubuntu 20.10

# The source mirror for the source code is commented out by default to speed up apt update, uncomment if needed
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-security main restricted universe multiverse

# Pre-release software sources are not recommended to enable
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ groovy-proposed main restricted universe multiverse

After completing the above, execute the following commands to update the software list and software.

sudo apt update
sudo apt upgrade

Install Dependencies and Docker

1. Install some dependencies and docker-compose. If you are using the 20.10 system, you may need to build it yourself; just be patient during this process.

sudo apt-get update && sudo apt-get install -y vim python3-pip curl git
pip3 install --upgrade pip
pip install docker-compose

1. Install Docker. The installation of Docker may be slow, so please be patient.

# Domestic users can install Docker with one click, which is faster
sudo curl -sSL https://get.daocloud.io/docker | sh 

# Install using Docker's installation script
sudo curl -sSL get.docker.com | sh

1. Use docker -v and docker-compose -v commands to verify if the installation was successful. If similar information appears, the installation was successful.

root@Raspberry-Pi4:/# docker -v
Docker version 20.10.2, build 2291f61

root@Raspberry-Pi4:/# docker-compose -v
docker-compose version 1.28.0, build unknown

1. So far, our preliminary tools have been basically installed. Before starting the cloud storage installation, we will configure a mirror source for Docker because of network issues in China, pulling images may often fail. Execute the following commands in order.

sudo mkdir -p /etc/docker

sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["http://hub-mirror.c.163.com"]
}
EOF

sudo systemctl daemon-reload
sudo systemctl restart docker

Additionally, I will provide a few mirror addresses. You can replace the address in the brackets as needed.

Docker China official mirror
https://registry.docker-cn.com

University of Science and Technology of China
https://docker.mirrors.ustc.edu.cn

Aliyun's mirror accelerator requires applying for your own address
https://cr.console.aliyun.com/

The dedicated address you apply for looks like this: https://xxxxxx.mirror.aliyuncs.com

Install ownCloud

1. Create a folder named ownCloud under /home/ and navigate into this directory.

sudo mkdir /home/ownCloud
cd /home/ownCloud

1. Create a configuration file for the cloud storage. This configuration file will be passed as parameters during the cloud storage initialization, with the username and password set to admin by default, and it will work on port 8080 by default, which can be customized according to your needs.

cat << EOF > .env
OWNCLOUD_VERSION=10.6
OWNCLOUD_DOMAIN=localhost:8088
ADMIN_USERNAME=admin
ADMIN_PASSWORD=admin
HTTP_PORT=8088
EOF

1. Create a docker-compose.yml file. The relevant passwords and information for the database are written in this file, refer to the environment parameter list of ownCloud. It is not recommended for beginners to modify this file arbitrarily.

version: '2.1'

volumes:
  files:
    driver: local
  mysql:
    driver: local
  backup:
    driver: local
  redis:
    driver: local

services:
  owncloud:
    image: owncloud/server:${OWNCLOUD_VERSION}
    restart: always
    ports:
      - ${HTTP_PORT}:8080
    depends_on:
      - db
      - redis
    environment:
      - OWNCLOUD_DOMAIN=${OWNCLOUD_DOMAIN}
      - OWNCLOUD_DB_TYPE=mysql
      - OWNCLOUD_DB_NAME=owncloud
      - OWNCLOUD_DB_USERNAME=owncloud
      - OWNCLOUD_DB_PASSWORD=owncloud
      - OWNCLOUD_DB_HOST=db
      - OWNCLOUD_ADMIN_USERNAME=${ADMIN_USERNAME}
      - OWNCLOUD_ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - OWNCLOUD_MYSQL_UTF8MB4=true
      - OWNCLOUD_REDIS_ENABLED=true
      - OWNCLOUD_REDIS_HOST=redis
    healthcheck:
      test: ["CMD", "/usr/bin/healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - files:/mnt/data

  db:
    image: webhippie/mariadb:latest
    restart: always
    environment:
      - MARIADB_ROOT_PASSWORD=owncloud
      - MARIADB_USERNAME=owncloud
      - MARIADB_PASSWORD=owncloud
      - MARIADB_DATABASE=owncloud
      - MARIADB_MAX_ALLOWED_PACKET=128M
      - MARIADB_INNODB_LOG_FILE_SIZE=64M
    healthcheck:
      test: ["CMD", "/usr/bin/healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - mysql:/var/lib/mysql
      - backup:/var/lib/backup

  redis:
    image: webhippie/redis:latest
    restart: always
    environment:
      - REDIS_DATABASES=1
    healthcheck:
      test: ["CMD", "/usr/bin/healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - redis:/var/lib/redis

1. Use the docker-compose up -d command to start the cloud storage. The first time you start, it will need to pull the images, and this process will depend on the network environment. When the image pull is complete and you see the following output, it means the container has started.

root@Raspberry-Pi4:/home/ownCloud# docker-compose up -d
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/

Starting owncloud_redis_1 ... done
Starting owncloud_db_1    ... done
Starting owncloud_owncloud_1 ... done

At this point, you can check the status of the containers using the docker-compose ps command.

root@Raspberry-Pi4:/home/ownCloud# docker-compose ps

       Name                      Command                  State               Ports         
--------------------------------------------------------------------------------------------
owncloud_db_1         /usr/bin/entrypoint /bin/s ...   Up (healthy)   3306/tcp              
owncloud_owncloud_1   /usr/bin/entrypoint /usr/b ...   Up (healthy)   0.0.0.0:8088->8080/tcp
owncloud_redis_1      /usr/bin/entrypoint /bin/s ...   Up (healthy)   6379/tcp     

Wait a few minutes for the related initialization work to complete automatically. At this point, you can access http://your-raspberry-pi-ip:8088 to start using the cloud storage.

1. Troubleshooting: If the login page does not appear when accessing port 8088, it may be due to the following reasons.

• Port 8088 is not allowed. Execute sudo ufw allow 8088 to allow the port.
• The cloud storage initialization has not been completed. Use the docker-compose logs --follow owncloud command to check if the initialization is complete. If you see the following output, it confirms that the initialization is complete.

owncloud_1  | Writing apache config...
owncloud_1  | Enabling cron background...
owncloud_1  | Set mode for background jobs to 'cron'
owncloud_1  | Touching cron configs...
owncloud_1  | Starting cron daemon...
owncloud_1  | Starting apache daemon...

Interspersed Knowledge of Docker and docker-compose

After reading the above tutorial, you might still not fully understand some of the commands executed. Here, I will intersperse some explanations about docker and docker-compose commands.

Common Management Commands for docker-compose.

Before using the docker-compose command, we need to ensure that the command line has entered the directory where the docker-compose.yml file is located; if not, we need to specify the path to the docker-compose.yml in the command.

For example, docker-compose -f /home/ownCloud/docker-compose.yml ps.

• docker-compose up -d starts the container and runs it in the background.
• docker-compose stop stops the running container.
• docker-compose restart restarts the container.
• docker-compose ps lists the containers.
• docker-compose logs views the logs.
• docker-compose down stops and removes the containers.
• docker-compose help views all commands.

All these commands can be followed by the container name to specify which container to operate on.

For example, docker-compose stop owncloud_redis_1

Docker Commands

Here I will only talk about one command for viewing data volumes, because using the down command to remove the cloud storage container does not mean your previous data is gone. When you start the container again, this data will still exist.

Thus, we can use the docker volume ls command to view all data volumes.

root@Raspberry-Pi4:/# docker volume ls
DRIVER    VOLUME NAME
local     owncloud_backup
local     owncloud_files
local     owncloud_mysql
local     owncloud_redis

Use docker volume rm name1 name2 to remove these data volumes. If you do not remove them, the data will still exist the next time you run up -d.

Use docker volume inspect volume-name to see their specific locations.

root@Raspberry-Pi4:/# docker volume inspect owncloud_files 
[
    {
        "CreatedAt": "2021-01-26T12:46:24+08:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "owncloud",
            "com.docker.compose.version": "1.28.0",
            "com.docker.compose.volume": "files"
        },
        "Mountpoint": "/var/lib/docker/volumes/owncloud_files/_data",
        "Name": "owncloud_files",
        "Options": null,
        "Scope": "local"
    }
]

Install nextCloud

For nextCloud, I will only introduce the installation method using the PostgreSQL database.

1. Create a folder named nextCloud under /home/ and navigate into this directory.

sudo mkdir /home/nextCloud
cd /home/nextCloud

1. Create the cloud storage configuration files; these files will be used during service initialization. The two nextcloud_admin_xxx.txt files correspond to your login username and password, and postgres_db.txt corresponds to the database name, and so on.

echo admin > nextcloud_admin_user.txt
echo admin > nextcloud_admin_password.txt
echo nextcloud > postgres_db.txt
echo nextcloud > postgres_user.txt
echo nextcloud > postgres_password.txt

1. Create the docker-compose.yml file. The default startup port is 8080; if you need to specify a port, just change the ports under the app field to the port you want to set. For example, - 9000:80; note that you only need to modify the left-side port number.

version: '3.2'

services:
  db:
    image: postgres
    restart: always
    volumes:
      - db:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
    secrets:
      - postgres_db
      - postgres_password
      - postgres_user

  app:
    image: nextcloud
    restart: always
    ports:
      - 8080:80
    volumes:
      - nextcloud:/var/www/html
    environment:
      - POSTGRES_HOST=db
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
      - NEXTCLOUD_ADMIN_PASSWORD_FILE=/run/secrets/nextcloud_admin_password
      - NEXTCLOUD_ADMIN_USER_FILE=/run/secrets/nextcloud_admin_user
    depends_on:
      - db
    secrets:
      - nextcloud_admin_password
      - nextcloud_admin_user
      - postgres_db
      - postgres_password
      - postgres_user

volumes:
  db:
  nextcloud:

secrets:
  nextcloud_admin_password:
    file: ./nextcloud_admin_password.txt # put admin password to this file
  nextcloud_admin_user:
    file: ./nextcloud_admin_user.txt # put admin username to this file
  postgres_db:
    file: ./postgres_db.txt # put postgresql db name to this file
  postgres_password:
    file: ./postgres_password.txt # put postgresql password to this file
  postgres_user:
    file: ./postgres_user.txt # put postgresql username to this file

1. Use the docker-compose up -d command to start.

The first time you start, it will need to pull the images, and this process will depend on the network environment. When the image pull is complete and you see the following output, it means the container has started. Please wait a few minutes to open http://your-raspberry-pi-ip:8080 to log in and use the cloud storage.

root@Raspberry-Pi4:/home/nextCloud# docker-compose up -d
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "nextcloud_default" with the default driver
Creating nextcloud_db_1 ... done
Creating nextcloud_app_1 ... done

Note: When the container startup status is done, it does not mean that the service has been initialized. Please use docker-compose logs --follow app to check the logs to ensure that the initialization is complete.

1. Modify the configuration file. During the first login, you may encounter the following prompt.

At this point, we refer back to the command for checking the data volume location, yes, it is the docker volume command. Use docker volume inspect nextcloud_nextcloud.

root@Raspberry-Pi4:/home/nextCloud# docker volume inspect nextcloud_nextcloud 
[
    {
        "CreatedAt": "2021-01-25T20:35:29+08:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "nextcloud",
            "com.docker.compose.version": "1.28.0",
            "com.docker.compose.volume": "nextcloud"
        },
        "Mountpoint": "/var/lib/docker/volumes/nextcloud_nextcloud/_data",
        "Name": "nextcloud_nextcloud",
        "Options": null,
        "Scope": "local"
    }
]

Then we can cd to /var/lib/docker/volumes/nextcloud_nextcloud/_data directory and use nano config/config.php to modify the trusted_domains configuration.

'trusted_domains' => 
  array (
    0 => 'localhost',
    1 => 'replace with your Raspberry Pi's actual IP',
  ),

After refreshing, we can log in~~~

nextCloud Optimization

1. Optimize upload speed. Execute docker exec -u 33 -it nextcloud_app_1 bash to enter the container as user 33. Execute php occ config:app:set files max_chunk_size --value 0 to remove the block size limit.

root@Raspberry-Pi4:/home/nextCloud# docker exec -u 33 -it nextcloud_app_1 bash
www-data@e0dd14717e86:~/html$ php occ config:app:set files max_chunk_size --value 0
Config value max_chunk_size for app files set to 0

1. Use nginx as a reverse proxy, for example, listening on port 80 and forwarding to 8080; no further elaboration.

Experience the upload and download speeds~~

References