Introduction

Apologies: Sorry, I was a bit foggy yesterday and posted the wrong content. I’m resending it today.

Based on my personal work and interests, I wanted to set up a server cluster at home for testing and validation of container/K8s solutions.

I considered using second-hand servers, like the Dell R730, and even made a configuration list, as follows:

•Dell R730•3.5-inch hard drives•CPU: 2686v4*2•Memory: 16g*8•Storage: 480G Intel SSD system disk + 6T SAS Seagate*2 data disks•RAID card: H730•Power supply: single 750w•Full disk rack•Price is around 3130 yuan

Then I thought about a server cabinet…

However, considering the power and noise levels were too high, and I didn’t have a suitable space at home for such a large server, I ultimately gave up. 😂

I also considered using single-board ARM development boards, but I would need several of them, then compile the system, flash the firmware, find suitable cases, power cables, and network cables, and connect them to my home network… Setting them up one by one was too time-consuming.

Even if I managed to get everything running, I probably wouldn’t have enough network ports at home and would need to buy a switch to connect them…

In the end, perseverance paid off, and I found a combination of both forms on Xianyu, looking like a server: the ARM development board matrix – Firefly Cluster Server. It fits my needs perfectly 👍️, let’s take a look together.

Firefly Cluster Server – ARM Development Board Matrix

Overview of the Server

Let’s first take a look at its overall appearance:

It looks like a small server, but don’t be fooled by its appearance. Its internal structure is completely different from that of x86 servers.

This is based on the Firefly Cluster Server R1^[1] – core-3399-jd4 * 11 customized version, which has fewer interfaces. However, the internal structure is identical.

This is its back, with 2 small fans + power interface.

Let’s open the top cover and explore its internal structure:

Its cooling system consists of: passive cooling from heatsinks on each CPU + active cooling from the entire chassis’s fans.

Aside from the power supply, there’s a motherboard (baseboard), which is quite interesting. It consists of the following components:

•11 golden finger interfaces for inserting Firefly core-3399-jd4 core boards, one interface for each. This means there are a total of 11 core boards. Correspondingly, there are 11 ARM servers, each with configurations described later.•One is the main core, used as the management side, similar to a server’s BMC to manage the other 10 servers•The other 10 are worker nodes, used for actual load running.•Switch chip for internal Ethernet communication among these 11 core boards.•4 Ethernet interfaces, including: •1 for the main core•The other 3 connect to the switch chip on the baseboard

The specific network topology is illustrated below, which gives a clear understanding of the server’s internal structure:

Server Core – Firefly core-3399-jd4

This is the core of the server: the Firefly core-3399-jd4 in hand, which is quite small:

Each Firefly core-3399-jd4 core board is an ARM server with the following configuration:

•CPU: Rockchip RK3399 six-core ARM64 CPU (this chip is built on a 28nm process and generates a decent amount of heat)•Memory: LPDDR4, options of 2G/4G•NPU: optional, the one I purchased does not have this chip•Storage: eMMC, options of 8G – 128G, mine is 32G.

This means that this server, once booted, will have: 11 units with 6 cores, 2/4G memory, and 32G storage available.

For the configuration of 4G/32G, in total, there will be:

•66 CPU cores•44G memory•352G storage

available.

Flashing the System

This type of server is primarily intended for use as a cloud phone matrix. Therefore, it originally comes with an Android system.

Based on my needs, I flashed the entire system to Ubuntu.

The specific flashing process won’t be detailed here; if you’re interested, you can refer to this:

Sub-board Firmware Upgrade – Product Introduction — Firefly Wiki (t-firefly.com)^[2]

Maincore – BMC Management Side

The BMC management side of this server is also quite interesting, with functionalities including:

•Device list•Device monitoring dashboard•Shell terminal window•Sub-board firmware upgrades

As shown in the following images:

Does this interface look a bit familiar? 😄😄😄

Actually, this management side is based on a modified version of Grafana.

•The UI interface is: Grafana + some customized panels•Monitoring data is based on: Prometheus + node exporter•The shell terminal window is implemented based on ADB (📝 The 11 core boards of this server, aside from being interconnected through the Ethernet switch, will also connect to the main core as USB devices)•Sub-board firmware upgrades are implemented using Rockchip’s Linux flashing tools

Practical Use

The initial startup effect of this server is as follows:

After being tinkered with many times, I ultimately broke 4 devices 😂😂😂, leaving only 7 usable ones.

My heart is bleeding… 💀💀💀

Power Consumption

Its power consumption stabilizes at 30 – 40 W (7 boards, each consuming around 5 W), as shown in the following image:

The power consumption is relatively low. This is one of the advantages of ARM chips.

Conclusion

Based on my personal work/interests, I wanted:

•Multiple servers•Always on•Low noise•Low power consumption

This device, which looks like a server but is actually composed of 11 ARM development boards, meets my needs.

•After breaking 4 development boards, I still have 7 left to use•Can run continuously•Noise is still a bit loud (mainly due to the aggressive fans; I plan to find a quiet fan for installation later)•Low power consumption

It basically meets my needs. In the future, I will use it to set up a HashiCorp Nomad cluster.

Stay tuned.

📚️ Reference Documents

•CSR1-N10R3399 Data Download | Firefly | Making Technology Simpler and Life Smarter (t-firefly.com)^[3]•Welcome to Cluster-Server-R1 Manual — Firefly Wiki (t-firefly.com)^[4]•Welcome to Core-3399-JD4 Manual — Firefly Wiki (t-firefly.com)^[5]•Core-3399-JD4 Data Download | Firefly | Making Technology Simpler and Life Smarter (t-firefly.com)^[6]

References

[1] Firefly Cluster Server R1: https://wiki.t-firefly.com/zh_CN/Cluster-Server-R1/Firefly_Cluster_server_product_documentation.html[2] Sub-board Firmware Upgrade – Product Introduction — Firefly Wiki (t-firefly.com): https://wiki.t-firefly.com/zh_CN/Cluster-Server-R1/Firefly_Cluster_server_product_documentation.html#zi-ban-gu-jian-sheng-ji[3] CSR1-N10R3399 Data Download | Firefly | Making Technology Simpler and Life Smarter (t-firefly.com): https://www.t-firefly.com/doc/download/82.html[4] Welcome to Cluster-Server-R1 Manual — Firefly Wiki (t-firefly.com): https://wiki.t-firefly.com/zh_CN/Cluster-Server-R1/index.html[5] Welcome to Core-3399-JD4 Manual — Firefly Wiki (t-firefly.com): https://wiki.t-firefly.com/zh_CN/Core-3399-JD4/index.html[6] Core-3399-JD4 Data Download | Firefly | Making Technology Simpler and Life Smarter (t-firefly.com): https://www.t-firefly.com/doc/download/page/id/66.html