01
Introduction
In our daily work, we often need to execute system commands, which requires the use of the subprocess library. However, there are places where we use subprocess.run and others where we use subprocess.Popen. What are the differences between these two? What is their relationship? Today, we will analyze this issue.
subprocess is a module in Python’s standard library used to start new processes, connect to their input/output/error pipes, and obtain return codes. It has been part of Python for a long time, replacing older functions like os.system, os.spawn*, os.popen*, etc., providing more flexible and powerful process management capabilities. With the iteration of versions, its functionality has gradually been updated and improved.
Returning to the question, what exactly are the differences between subprocess.run and subprocess.Popen? Let’s analyze them one by one.
02
subprocess.run
subprocess.run is a high-level API introduced in Python 3.5, designed to simplify the execution of system commands. To understand this API, let’s first look at the Python source code.
def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs): """Run command with arguments and return a CompletedProcess instance.
The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them, or pass capture_output=True to capture both.
If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute, and output & stderr attributes if those streams were captured.
If timeout (seconds) is given and the process takes too long, a TimeoutExpired exception will be raised.
There is an optional argument "input", allowing you to pass bytes or a string to the subprocess's stdin. If you use this argument you may not also use the Popen constructor's "stdin" argument, as it will be used internally.
By default, all communication is in bytes, and therefore any "input" should be bytes, and the stdout and stderr will be bytes. If in text mode, any "input" should be a string, and stdout and stderr will be strings decoded according to locale encoding, or by "encoding" if set. Text mode is triggered by setting any of text, encoding, errors or universal_newlines.
The other arguments are the same as for the Popen constructor. """ if input is not None: if kwargs.get('stdin') is not None: raise ValueError('stdin and input arguments may not both be used.') kwargs['stdin'] = PIPE
if capture_output: if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None: raise ValueError('stdout and stderr arguments may not be used with capture_output.') kwargs['stdout'] = PIPE kwargs['stderr'] = PIPE
with Popen(*popenargs, **kwargs) as process: try: stdout, stderr = process.communicate(input, timeout=timeout) except TimeoutExpired as exc: process.kill() if _mswindows: # Windows accumulates the output in a single blocking # read() call run on child threads, with the timeout # being done in a join() on those threads. communicate() # _after_ kill() is required to collect that and add it # to the exception. exc.stdout, exc.stderr = process.communicate() else: # POSIX _communicate already populated the output so # far into the TimeoutExpired exception. process.wait() raise except: # Including KeyboardInterrupt, communicate handled that. process.kill() # We don't call process.wait() as .__exit__ does that for us. raise retcode = process.poll() if check and retcode: raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
return CompletedProcess(process.args, retcode, stdout, stderr)From the source code, we can see that subprocess.run is a high-level wrapper around subprocess.Popen. It uses a context manager with Popen and communicates through the communicate interface. While this simplifies the API logic, it also brings issues, as it blocks execution after the system command completes, either directly upon completion or upon an exception.
From here, we can see the advantages of this API:
– The API is simple and easy to call
– The capture_output parameter automatically captures standard output (stdout) and error output (stderr)
– Returns a CompletedProcess object, which contains command parameters, command exit code, standard output (stdout), error output (stderr), and other information.
Disadvantages:
– It is blocking, meaning no other operations can be performed after the command execution completes
– It interacts with the program only once, which is not flexible enough
Application scenarios:
– Suitable for scenarios where you want to immediately obtain the execution result after the command completes
Usage:
import subprocess
subprocess.run(["ls", "-l"])# CompletedProcess(args=['ls', '-l'], returncode=0)03
subprocess.Popen
subprocess.Popen is a low-level API that provides more powerful and flexible functionality. Correspondingly, the code is more complex, requiring manual management of process input, output, and state.
<pfor advantages="" api="" are:
– The API is flexible, allowing control over the command execution process
– Flexible interaction with the program, fully controlled by the caller
– It is non-blocking, allowing other operations to continue after the command execution, unless explicitly calling wait or communicate to wait for command completion
– Returns a Popen object for interacting with the process
Disadvantages:
– The API is complex, requiring manual management of process input, output, and state
Applicable scenarios:
– When asynchronous command execution or executing multiple commands is needed
– When interaction with the subprocess is required
– When fine-grained management of the subprocess is needed
Usage:
import subprocess
p = subprocess.Popen(["ls", "-l"])
p.wait()04
Conclusion
Both subprocess.run and subprocess.Popen provide the capability to execute system commands. In comparison, subprocess.run is convenient and easy to use, suitable for simple command execution scenarios; subprocess.Popen is more complex and is used to handle more complicated application scenarios. You can choose based on your actual situation; there is no right or wrong.
Scan to follow
