Original link: https://blog.opskumu.com/graceful-shutdown-docker.html
Recently, I reread the official Docker Reference documentation and found that there are still many details worth exploring. When writing a Dockerfile, most of the time you can just follow the examples without any major issues, but a deeper understanding can make it even more interesting.
When it comes to gracefully shutting down a container, we must mention the concept of signals and the ENTRYPOINT and CMD instructions in the Dockerfile. Before discussing graceful shutdown, let’s first understand the basic concept of signals in Linux.
1 Signals
A signal is a notification mechanism to a process when an event occurs, sometimes referred to as a software interrupt.
There are different types of signals, and Linux numbers standard signals from 1 to 31, which can be obtained using kill -l:
# kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT
4) SIGILL 5) SIGTRAP 6) SIGABRT
7) SIGBUS 8) SIGFPE 9) SIGKILL
10) SIGUSR1 11) SIGSEGV 12) SIGUSR2
13) SIGPIPE 14) SIGALRM 15) SIGTERM
... ...
In fact, the listed signals exceed 31; some are synonyms with different names, while others are defined but unused. Here are a few commonly used signals:
-
SIGHUP is sent to the controlling process of a terminal when the terminal disconnects (hangs up). SIGHUP can also be used by daemon processes (e.g., init). Many daemons will reinitialize and reload configuration files upon receiving the SIGHUP signal.
-
SIGINT is sent to the foreground process group when the user types an interrupt character in the terminal (usually Control-C). The default behavior of this signal is to terminate the process.
-
SIGQUIT is sent to the foreground process group when the user types an exit character on the keyboard (usually Control-“). By default, this signal terminates the process and generates a core dump file for debugging. It is suitable to use SIGQUIT when a process is stuck in an infinite loop or is not responding.
-
SIGKILL is a “sure kill” signal that the processor cannot block, ignore, or catch, hence it will always terminate the program.
-
SIGTERM is the standard signal used to terminate processes, and it is the default signal sent by the kill, killall, and pkill commands. Well-designed applications should set up a handler for the SIGTERM signal so that they can clean up temporary files and release other resources before exiting. Therefore, one should always try to terminate a process using the SIGTERM signal first, reserving SIGKILL as a last resort for those unresponsive processes.
-
SIGTSTP is the stop signal for job control, sent to the foreground process group when the user types a suspend character (usually Control-Z).
It is worth noting that Control-D does not initiate a signal; it represents EOF (End-Of-File) and closes the standard input (stdin) pipe (for example, you can exit the current shell using Control-D). If a program does not read the current input, it is unaffected by Control-D.
Programs can catch signals and execute corresponding functions:
Most of the above knowledge comes from the “Linux/UNIX System Programming Handbook”. For more information, you can refer to chapters 20, 21, and 22 of the book.
2 ENTRYPOINT and CMD
Some may ask, what do signals have to do with gracefully shutting down containers? While it may not be related to money, it is closely related to how to gracefully shut down a container.
Continuing on the ENTRYPOINT and CMD instructions in the Dockerfile, their main function is to specify the program that runs when the container starts.
CMD has three formats:
-
CMD [“executable”,”param1″,”param2″] (exec format, recommended to use this format)
-
CMD [“param1″,”param2”] (as parameters to the ENTRYPOINT instruction)
-
CMD command param1 param2 (shell format, default /bin/sh -c)
ENTRYPOINT has two formats:
-
ENTRYPOINT [“executable”, “param1”, “param2”] (exec format, recommended to use this format)
-
ENTRYPOINT command param1 param2 (shell format)
Regardless of which instruction you use in your Dockerfile, both instructions are recommended to use exec format rather than shell format. The reason is that using shell format starts the program as a subprocess of /bin/sh -c, and the shell format does not pass any signals to the program. This means that when you stop the container using docker stop, the program running in this format cannot catch the sent signals, and thus cannot achieve graceful shutdown.
➜ ~ docker stop --help
Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop one or more running containers
Options:
--help Print usage
-t, --time int Seconds to wait for stop before killing it (default 10)
When stopping a container with docker stop, it sends a SIGTERM signal by default, and if the container does not stop within 10 seconds, it will forcibly stop the container with SIGKILL. You can set the waiting time using the -t option.
➜ ~ docker kill --help
Usage: docker kill [OPTIONS] CONTAINER [CONTAINER...]
Kill one or more running containers
Options:
--help Print usage
-s, --signal string Signal to send to the container (default "KILL")
You can also specify the signal sent to the container using the -s option of docker kill.
So, after all this, if the Dockerfile executes the container startup command in exec format, will everything be fine? Of course not; it is not that simple. Next, let’s look at specific effects through examples.
3 Examples
Let’s write a simple signal handler in Go:
➜ ~ cat signals.go
package main
import (
"fmt"
"os"
"os/signal"
"syscall"
)
func main() {
sigs := make(chan os.Signal, 1)
done := make(chan bool, 1)
signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
go func() {
sig := <-sigs
fmt.Println()
fmt.Println(sig)
done <- true
}()
fmt.Println("awaiting signal")
<-done
fmt.Println("exiting")
}
3.1 Example 1
➜ ~ GOOS=linux GOARCH=amd64 go build signals.go
➜ ~ ls
Dockerfile signals signals.go
➜ ~ cat Dockerfile
FROM busybox
COPY signals /signals
CMD ["/signals"] # exec format execution
➜ ~ docker build -t signals .
Open two panels using tmux, one running the container and the other executing docker stop:
➜ ~ docker run -it --rm --name signals signals
awaiting signal
terminated
exiting
➜ ~ time docker stop signals
signals
docker stop signals 0.01s user 0.02s system 4% cpu 0.732 total
➜ ~
As we can see, before the container stops, the program receives the signal and outputs the corresponding information, and the total stopping time is 0.732 seconds, achieving a graceful effect.
Now let’s modify the CMD execution format in the Dockerfile and perform the same operation:
➜ ~ cat Dockerfile
FROM busybox
COPY signals /signals
CMD /signals # shell format execution
➜ ~ docker build -t signals .
➜ ~ docker run -it --rm --name signals signals
awaiting signal
➜ ~
➜ ~ time docker stop signals
signals
docker stop signals 0.01s user 0.01s system 0% cpu 10.719 total
After using the shell format, we can see that the program did not receive any signal before the container stopped, and the stopping time was 10.719 seconds, indicating that the container was forcibly stopped.
The conclusion is clear: to gracefully exit a container, we should use the exec format.
3.2 Example 2
In Example 1, we all learned to use exec format in the Dockerfile to execute the program. But what if the executed program is also a shell script?
➜ ~ ls
Dockerfile signals signals.go start.sh
➜ ~ cat Dockerfile
FROM busybox
COPY signals /signals
COPY start.sh /start.sh # introduce shell script to start
CMD ["/start.sh"]
➜ ~ cat start.sh
#!/bin/sh
/signals
➜ ~
Let’s test using the same method as in Example 1:
➜ ~ docker run -it --rm --name signals signals
awaiting signal
➜ ~
➜ ~ time docker stop signals
signals
docker stop signals 0.01s user 0.02s system 0% cpu 10.765 total
➜ ~
We can see that even when using exec format in the CMD instruction of the Dockerfile, the program in the container did not receive the signal and was ultimately forcibly closed. This is due to the execution of the shell script, which prevents the signal from being passed. We need to modify the shell script:
➜ ~ cat start.sh
#!/bin/sh
exec /signals # add exec execution
➜ ~ docker build -t signals .
➜ ~ docker run -it --rm --name signals signals
awaiting signal
terminated
exiting
➜ ~ time docker stop signals
signals
docker stop signals 0.02s user 0.02s system 4% cpu 0.744 total
➜ ~
As we can see, after adding the exec command, the program can receive the signal and exit normally. However, if your CMD in the Dockerfile is run in shell format, adding exec in the startup script will still be ineffective. Furthermore, if your program itself cannot handle signals, then graceful shutdown is not possible.
– END –
Recommended Reading
31 Days to Get the Most Valuable CKA+CKS Certificate in K8S!
50 Common Interview Questions for Linux Operations Engineers
How to Design a Monitoring System Based on Prometheus!
A Complete Guide to Linux Performance Analysis Tools
How to Check for Abnormal Disk Space Usage in Linux?
K8s Cluster Incidents Caused by Linux Kernel Parameters
Essential Kubectl Commands Every K8S Operator Should Know
16 Hard-Hitting Diagrams Explaining Kubernetes Networking
The Most Comprehensive Jenkins Pipeline Explanation
A Few Kubernetes Logical and Architecture Diagrams
9 Practical Shell Scripts You Should Keep!
40 Commonly Asked Nginx Interview Questions
A Comprehensive Chinese Guide to Kubernetes Network Troubleshooting
Customize Your Own Docker Image with Dockerfile - Extremely Detailed!
Mainstream Monitoring System Prometheus Learning Guide
Master Ansible for Automated Operations in One Article
100 Classic Linux Shell Script Examples (with PDF)
Build a Complete Enterprise-Level K8s Cluster (using kubeadm)
Light it up, keep the server running for three years without downtime