🚀 Essential Linux Command Collection for Big Data Development: A Must-Read Guide for Programmers!
In big data development, Linux is our “fighter jet”.Core components like Hadoop, Hive, Kafka, Flink, and Spark all run in a Linux environment.If you are not familiar with Linux commands, troubleshooting issues, optimizing performance, or even simple data file operations can be quite challenging.
Today, I have compiled a collection of essential Linux commands for big data development, covering file operations, process management, network troubleshooting, compression and decompression, permission management, and performance monitoring. Save this article, and your development and debugging efficiency will double 🚀!
📌 1. Basic Operation Commands (Files & Directories)
| Command | Function | Example |
|---|---|---|
<span>pwd</span> |
View current directory | <span>pwd</span> |
<span>ls -lh</span> |
View file list (with units) | <span>ls -lh /data</span> |
<span>cd</span> |
Change directory | <span>cd /opt/hive</span> |
<span>mkdir -p</span> |
Create directory (recursively) | <span>mkdir -p /data/hive/ods</span> |
<span>rm -rf</span> |
Delete file/directory (use with caution ⚠️) | <span>rm -rf /data/tmp</span> |
<span>cp -r</span> |
Copy file/directory | <span>cp -r conf /backup/</span> |
<span>mv</span> |
Move/rename | <span>mv test.log old_test.log</span> |
<span>cat</span> |
View file content | <span>cat /data/input.txt</span> |
<span>tail -f</span> |
Real-time view logs | <span>tail -f hive.log</span> |
<span>head -n</span> |
View the first N lines of a file | <span>head -n 20 data.csv</span> |
👉 When checking Hive/Spark logs, <span>tail -f</span> + <span>grep</span> is definitely a powerful tool for troubleshooting bugs.
📌 2. Permission and User Management
| Command | Function | Example |
|---|---|---|
<span>chmod 755 file</span> |
Change permissions | <span>chmod 644 hive-site.xml</span> |
<span>chown user:group file</span> |
Change owner | <span>chown hive:hadoop data.txt</span> |
<span>su - user</span> |
Switch user | <span>su - hadoop</span> |
<span>whoami</span> |
View current user | <span>whoami</span> |
<span>id</span> |
View UID, GID | <span>id hadoop</span> |
👉 On big data platforms, encountering <span>Permission denied</span> is common, and at this point, <span>chmod + chown</span> comes in handy.
📌 3. File Search and Text Processing
| Command | Function | Example |
|---|---|---|
<span>find / -name "*.log"</span> |
Find files | <span>find /var/log -name "hive*"</span> |
<span>grep "error" file.log</span> |
Text search | <span>grep "Exception" hive.log</span> |
<span>grep -r "hdfs" /etc/hadoop/</span> |
Recursive search | <span>grep -r "namenode" conf/</span> |
<span>awk '{print $1,$3}' file</span> |
Text splitting and processing | <span>awk -F"," '{print $1,$2}' data.csv</span> |
<span>sed -i 's/old/new/g' file</span> |
Text replacement | <span>sed -i 's/dev/prod/g' core-site.xml</span> |
<span>wc -l file</span> |
Count lines | <span>wc -l access.log</span> |
👉 After Hive SQL outputs files, <span>awk/sed</span> are commonly used for secondary cleaning.
📌 4. Process and Task Management
| Command | Function | Example |
|---|---|---|
| `ps -ef | grep hive` | View processes |
<span>top</span> |
View system resources | <span>top</span> |
<span>htop</span> |
Color version of top (requires installation) | <span>htop</span> |
<span>kill -9 pid</span> |
Force kill process | <span>kill -9 12345</span> |
<span>jps</span> |
View Java processes (essential for Hadoop) | <span>jps</span> |
<span>jobs</span> |
View background tasks | <span>jobs</span> |
<span>fg %1</span> |
Bring task to foreground | <span>fg %1</span> |
<span>nohup command &</span> |
Run task in background | <span>nohup hive -f job.sql &</span> |
👉 On big data clusters, using <span>jps</span> + <span>grep</span> is common to confirm whether NameNode, DataNode, Kafka Broker are functioning properly.
📌 5. Network Troubleshooting
| Command | Function | Example |
|---|---|---|
<span>ping host</span> |
Test network connectivity | <span>ping namenode1</span> |
<span>curl http://ip:port</span> |
Test interface | <span>curl http://localhost:8080</span> |
<span>wget url</span> |
Download file | <span>wget http://apache.org/hadoop.tar.gz</span> |
<span>netstat -tunlp</span> |
View port usage | `netstat -tunlp |
<span>telnet ip port</span> |
Test port | <span>telnet 127.0.0.1 9092</span> |
<span>ss -lntp</span> |
Efficiently view ports | `ss -lntp |
👉 When there is a port conflict with Kafka, <span>netstat</span> + <span>grep 9092</span> will quickly reveal the issue.
📌 6. Compression and Decompression
| Command | Function | Example |
|---|---|---|
<span>tar -czvf file.tar.gz dir</span> |
Package and compress | <span>tar -czvf logs.tar.gz /var/log/</span> |
<span>tar -xzvf file.tar.gz</span> |
Decompress | <span>tar -xzvf hadoop-3.3.1.tar.gz</span> |
<span>zip -r file.zip dir</span> |
Compress | <span>zip -r data.zip ods/</span> |
<span>unzip file.zip</span> |
Decompress | <span>unzip data.zip</span> |
<span>gzip file</span> |
Compress single file | <span>gzip bigdata.log</span> |
<span>gunzip file.gz</span> |
Decompress single file | <span>gunzip bigdata.log.gz</span> |
👉 When launching big data projects, it is common to use <span>tar -xzvf</span><span> to decompress Hadoop/Hive installation packages.</span>
📌 7. System Performance Monitoring
| Command | Function | Example |
|---|---|---|
<span>df -h</span> |
View disk space | <span>df -h</span> |
<span>du -sh dir</span> |
View directory size | <span>du -sh /data/hdfs/</span> |
<span>free -m</span> |
View memory usage | <span>free -m</span> |
<span>iostat -x 1</span> |
Disk IO (requires sysstat installation) | <span>iostat -x 1</span> |
<span>vmstat 1</span> |
View CPU, memory, IO | <span>vmstat 1</span> |
<span>uptime</span> |
View system load | <span>uptime</span> |
👉 When facing data skew or tasks not running, <span>vmstat</span><span> + </span><code><span>iostat</span> are commonly used to troubleshoot machine bottlenecks.
📌 8. Additional Common Commands for Hadoop & Hive
| Command | Function | Example |
|---|---|---|
<span>hadoop fs -ls /</span> |
View HDFS files | <span>hadoop fs -ls /user/hive/warehouse</span> |
<span>hadoop fs -put local remote</span> |
Upload file | <span>hadoop fs -put data.txt /ods/</span> |
<span>hadoop fs -get remote local</span> |
Download file | <span>hadoop fs -get /ods/data.txt ./</span> |
<span>hadoop fs -du -h /</span> |
View HDFS usage | <span>hadoop fs -du -h /ods/</span> |
<span>hive -e "select * from table limit 10;"</span> |
Command line query for Hive | <span>hive -e "show tables;"</span> |
<span>beeline -u jdbc:hive2://ip:10000</span> |
Connect to HiveServer2 | <span>beeline -u jdbc:hive2://localhost:10000</span> |
🎯 Conclusion
Familiarity with Linux commands is like equipping yourself with a “Swiss Army knife” that can handle 90% of issues in big data development.
👉 Recommendations:
-
Newcomers should practice 5 commands daily, learning while searching
-
Experienced users should develop the habit of using Linux tools for troubleshooting at the first sign of issues
With consistent practice, you will find that Linux is a prerequisite for big data development, and mastering these commands will truly allow you to stand firm on the big data development stage 🚀.
📌 If you find this article helpful, feel free to like 👍, bookmark ⭐, and follow me for more practical experience sharing! If you wish to discuss specific project practices, feel free to leave a comment