Article catalog

• preface
  • 1. System related
  • 2. File and directory operation
  • 3. Execute orders

preface

If we want to operate files and directories, we can enter various commands provided by the operating system under the command line. Such as dir, cd and other commands. What if you want to perform the operations of these directories and files in a python program? In fact, the commands provided by the operating system simply call the interface functions provided by the operating system. The os module built in Python can also directly call the interface functions provided by the operating system.

os module is a module in Python standard library, which is used to access the functions related to the operating system.

The main functions of os module: system related, directory and file operation, command execution and process management

Ps: the process management function is mainly Linux related and will not be discussed in this section.

Using the os module:

import os

When using the os module, if there is a problem, an OSError exception will be thrown, indicating an invalid pathname or The file name or path name (file name) cannot be accessed, or the current operating system does not support this operation.

>>> import os
>>> os.chdir("d:\11")
Traceback (most recent call last):
 File "", line 1, in <module>
 os.chdir("d:\11")
OSError: [WinError 123] Incorrect syntax for file name, directory name, or volume label.: 'd:\11'

1. System related

os module provides some operating system related variables, which can provide support in cross platform, which is convenient to write code with high portability and good usability. Therefore, when operating system related operations are involved, please try to use the methods provided in this module instead of the specific usage or format of the current platform. Otherwise, once transplanted to other platforms, it may cause difficulties that are difficult to solve.

The following is a table listing the methods and variables commonly used in the os module and their purpose explanations.

Usage example:

2. File and directory operation

os module contains a series of functions related to file operation, some of which are special methods for Linux platform. Linux is written in C. the underlying libc library and system call interfaces are C API s. The Python os module includes the Python implementation of these interfaces. Through the Python os module, you can call some underlying functions of the Linux system for system programming. For Linux related methods, you can consult the official documents according to your needs. Here we only introduce some common methods that are common to all platforms.

In Python, you must be careful when using the file path of windows. For example, you should refer to 1.0 under disk D Txt file, the path should be written as'd: \ 1 Txt 'or r'd: \ 1 txt’. The previous method is to use the double slash of windbos as the path separator. The latter uses the form of the original string. The strings starting with r are considered as the original string, indicating that all special symbols in the string are played in their natural color without escape. At this time, the path representation under ordinary windows can be used. Either of these two methods can be used, but they cannot be mixed.

The following are some examples of use. It is recommended that everyone follow them (some of them are error demonstrations to make you more aware of its usage).

>>> os.getcwd() #Get current directory
'C:\\Python36'
>>> os.chdir("d:") #Enter D: Directory
>>> os.getcwd() 
'D:\\'
>>> os.curdir #Returns the current directory
'.'
>>> os.pardir #Return to parent directory
'..'
>>> os.makedirs("1\\2") #Create directory
>>> os.removedirs("1\\2") Delete directory
>>> os.listdir()
['$360Section', '$RECYCLE.BIN', '1.txt', 'MobileFile', 'pymysql_test.py', 'System 
Volume Information', 'User directory']
>>> os.mkdir("1")
>>> os.listdir()
['$360Section', '$RECYCLE.BIN', '1', '1.txt', 'MobileFile', 'pymysql_test.py', 'System 
Volume Information', 'User directory']
>>> os.rmdir("1")
>>> os.rename('1.txt','2.txt')
>>> os.listdir()
['$360Section', '$RECYCLE.BIN', '2.txt', 'MobileFile', 'pymysql_test.py', 'System 
Volume Information', 'User directory']
>>> os.remove('1.txt')
Traceback (most recent call last):
 File "", line 1, in <module>
 os.remove('1.txt')
FileNotFoundError: [WinError 2] The system cannot find the specified file.: '1.txt'
>>> os.remove('2.txt')
>>> os.stat()
Traceback (most recent call last):
 File "", line 1, in <module>
 os.stat()
TypeError: Required argument 'path' (pos 1) not found
>>> os.stat(os.getcwd())
os.stat_result(st_mode=16895, st_ino=1407374883553285, st_dev=2431137650, 
st_nlink=1, st_uid=0, st_gid=0, st_size=32768, st_atime=1505824872, 
st_mtime=1505824872, st_ctime=1445187376)
>>> os.path.abspath(os.getcwd()) #Get absolute path
'D:\\test'
>>> os.path.split(os.getcwd()) #Split into directories and files
('D:\\', 'test')
>>> cp = os.getcwd()
>>> os.path.dirname(cp)
'D:\\'
>>> os.path.basename(cp)
'test'
>>> os.path.exists(cp)
True
>>> os.path.exists("d:\\123\123") #Determine whether the path exists
False
>>> os.path.isabs(cp)
True
>>> os.path.isabs("11\\1.py")
False
>>> os.path.isfile(cp)
False
>>> os.path.isfile("d:\\1.txt")
False
>>> os.path.isdir(cp)
True
>>> os.path.join(cp, "test.py")
'D:\\test\\test.py'
>>> os.path.getatime(cp) #Last storage time
1505825113.4970243
>>> os.path.getmtime(cp) #Last modification time
1505825113.4970243
>>> os.path.getsize(cp)
0

os.walk(top, topdown=True, οnerrοr=None, followlinks=False)

walk method is a very important and powerful method in os module. It can help us to recursively traverse the directory tree from top to bottom or from bottom to top, and return a ternary group (dirpath, dirnames, filenames) for each directory.

Of which:

dirpath - the location of the directory tree where the traversal is located. It is a string object

dirnames - a list of subdirectories in the directory tree, excluding ("." and "...")

filenames - a list of files in the directory tree

If the optional parameter topdown = True or is not specified, the top-down method is used for directory traversal, that is, step-by-step in-depth traversal from the parent directory to the child directory. If topdown = False, the bottom-up method is used for directory traversal, that is, printing the child directory first and then the parent directory.

If the optional parameter onerror is specified, onerror must be a function with a parameter of the OSError instance, so that the OS will not be interrupted even when an error occurs Or throw an exception and terminate the OS Run of walk(). Generally speaking, this parameter is defined to specify the processing method when an error occurs.

By default, OS When walk() traverses, it will not enter the symbolic link. If the optional parameter followlinks = True is set, it will enter the symbolic link. Note that this may lead to a traversal loop, because symbolic links may link themselves instead of OS Walk () doesn't have that high IQ and can't find it.

The following example prints out all the files and subdirectories in the C: \ Python 36 directory.

import os
try:
    for root, dirs, files in os.walk(r"c:\python36"):
        print("directory", root)
        for directory in dirs:
            print(" %s" % directory)
        for file in files:
            print("\t\t%s" % file)
except OSError as ex:
    print(ex)

The following example will count the size of all subdirectories under the c:/python36/Lib/email directory, except the CVS directory.

import os
from os.path import join, getsize
for root, dirs, files in os.walk('c:/python36/Lib/email'):
    print(root, "consumes", end=" ")
    print(sum(getsize(join(root, name)) for name in files), end=" ")
    print("bytes in", len(files), "non-directory files")
    if 'CVS' in dirs:
        dirs.remove('CVS') # Do not traverse CVS directory

Operation results:

C:\Python36\python.exe F:/Python/pycharm/201705/1.py -
c:/python36/Lib/email consumes 377849 bytes in 21 non-directory files
c:/python36/Lib/email\mime consumes 12205 bytes in 9 non-directory files
c:/python36/Lib/email\mime\__pycache__ consumes 30289 bytes in 27 nondirectory files
c:/python36/Lib/email\__pycache__ consumes 741924 bytes in 60 non-directory files

The following example will recursively delete all contents of the directory. It's dangerous. Don't try at will!

import os
for root, dirs, files in os.walk(top, topdown=False):
    for name in files:
        os.remove(os.path.join(root, name))
    for name in dirs:
        os.rmdir(os.path.join(root, name))

3. Execute orders

We can interact with the operating system through command line instructions on almost any operating system. So how do we complete the execution of these command-line instructions through Python? First of all, we should know that the execution of command line instructions usually has two results we pay more attention to:

1. Status code of command execution – indicates whether the command execution is successful
2. Output result of command execution – output after successful command execution

In the early Python versions, we mainly used OS system(),os.popen().read() and other functions to execute command-line instructions. In addition, there is a rarely used commands module. However, starting from Python 2.4, the official documents recommend the use of the subprocess module, so the related functions of the OS module and commands module only provide a simple use example here. We need to introduce the subprocess module.

os.system(command) Run the operating system command to display the results directly. However, if the return value is 0 or - 1, the data displayed on the screen cannot be obtained. Command is the command string to execute.

If we use IDLE to run OS under windows environment System ('ipconfig /all '), you will find that the command terminal interface flashes by, and there is no time to see anything at all. At this time, you'd better enter the cmd environment and use the python command to enter the interactive interface to see the information on the screen.

Because there are often inexplicable errors when using this function, but there is no problem executing the command directly, it is generally recommended not to use it.

os.popen(command, [mode, [bufsize]])

Start a child process, execute the command specified in the command parameter, and establish a pipe between the parent process and the child process for communication between the parent and child processes. This method returns a file object, which can be read or written, depending on the parameter mode. If the mode specifies read-only, only the file object can be read. If the mode parameter specifies write only, only the file object can be written.

In short, popen can also run operating system commands and return the command results through the read() method. Unlike system, it can only see and can't save. This can save!

>>> os.popen('ipconfig')
<os._wrap_close object at 0x0000000002BB8EF0>
>>> ret = os.popen('ipconfig')
>>> ret.read()
'\nWindows IP to configure\n\n\n ethernet adapter  Bluetooth Network connection 2:\n\n Media form
 state . . . . . . . . . . . . : Media disconnected\n Connect specific DNS suffix . . . . . . . : \n\n Wireless local area
 Network adapter wireless network connection 2:\n\n Media status . . . . . . . . . . . . : Media disconnected\n Connect special
 Fixed DNS suffix . . . . . . . : \n\n Wireless LAN adapter wireless network connection:\n\n Connect specific
DNS suffix......

subprocess module

The subprocess module is mainly used to create subprocesses, connect their input, output and error pipelines, and obtain their return status. Generally speaking, through this module, you can execute operating system level commands in Python code, such as "ipconfig" and so on.

Common functions in the subprocess module

In most cases, it is recommended to use the run() method to call sub processes and execute operating system commands.

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, shell=False, timeout=None, check=False, encoding=None, errors=None)

Function: execute the command represented by args parameter, wait for the command to end, and return an object of type CompletedProcess.

Note that the run() method does not return the execution result or related information we want, but an object of type CompletedProcess.

The above parameter table shows only some commonly used, and there are many real situations.

args: indicates the command to execute. Must be a string, string parameter list

stdin, stdout, and stderr: Standard inputs, outputs, and errors for child processes. Its value can be subprocess PIPE,subprocess.DEVNULL, an existing file descriptor, an open file object, or None. subprocess.PIPE means to create a new pipe for the child process.

subprocess.DEVNULL indicates OS devnull. None is used by default, which means nothing is done. In addition, stderr can be merged into stdout and output together.

Timeout: sets the command timeout. If the command execution time expires, the child process will be killed and a timeout expired exception will pop up.

check: if this parameter is set to True and the process exit status code is not 0, a CalledProcessError exception will pop up.

Encoding: if this parameter is specified, stdin, stdout and stderr can receive string data and encode it in this encoding method. Otherwise, only bytes of data will be received.

Shell: if this parameter is True, the specified command will be executed through the shell of the operating system.

The return value of the run() method indicates that a process has ended.

The CompletedProcess class has the following properties:

• args parameter to start the process, usually a list or string.
• returncode process end status return code. 0 indicates success status.
• Stdout gets the stdout of the child process. It is usually a byte type sequence, and None indicates that there is no captured value. If you set the parameter stderr = subprocess when calling the run() method Stdout, the error message will be output together with stdout. At this time, stderr The value of is None.
• stderr gets the error information of the child process. It is usually a byte type sequence, and None indicates that there is no captured value.
• check_returncode() is used to check the return code. If the returned status code is not zero, a CalledProcessError exception pops up.

subprocess.DEVNULL

A special value passed to the stdout, stdin, and stderr parameters. Indicates the use of OS Devnull as parameter Value.

subprocess.PIPE

Pipe, which can be passed to stdout, stdin, and stderr parameters.

subprocess.STDOUT

A special value that can be passed to the stderr parameter, indicating the combined output of stdout and stderr. args and shell parameters

The args parameter can receive a string similar to 'dir d: \' or pass a string split list similar to ['dir','d: \ ']. The shell parameter is False by default. When set to True, it means that the command is executed using the shell of the operating system. Let's look at the combination of the two.

Test in the windows system and execute the following statements independently:

ret = subprocess.run('dir d:\\')
ret = subprocess.run('dir d:\\', shell=True)
ret = subprocess.run(['dir', 'd:\\'])
ret = subprocess.run(['dir', 'd:\\'], shell=True)
ret = subprocess.run('ipconfig /all')
ret = subprocess.run('ipconfig /all', shell=True)
ret = subprocess.run(['ipconfig', '/all'])
ret = subprocess.run(['ipconfig', '/all'], shell=True)

The results show that in windows, the combination of args and shell parameters is complex, and there are different situations according to different commands. It is recommended that the shell be set to True.

Get execution results The run() method returns an object of type CompletedProcess, which cannot directly obtain the results we usually want. To obtain the result or information of the command execution, specify when calling the run() method

stdout=subprocess.PIPE. 
>>> ret = subprocess.run('dir', shell=True)
>>> ret
CompletedProcess(args='dir', returncode=0)
>>> ret = subprocess.run('dir', shell=True, stdout=subprocess.PIPE)
>>> ret
CompletedProcess(args='dir', returncode=0, stdout=b' \xc7\xfd\xb6\xaf\xc6\xf7 ......')
>>> ret.stdout
b' \xc7\xfd\xb6\xaf\xc6\xf7 C \xd6\xd0\xb5\xc4\xbe\xed\xca\xc7 ......'
>>> ret.stdout.decode('gbk')
' Driver C The volume in is a system\r\n The serial number of the volume is C038-3181\r\n\r\n C:\\Python36 Directory of
\r\n\r\n2017/08/11 10:14 ...... 15,275,020,288 Available bytes\r\n'

From the example, we can see that if you do not set stdout = subprocess Pipe, then the stdout attribute will not be included in the return value CompletedProcess(args = 'dir', returncode=0). Otherwise, the result will be saved in the ret.stdout attribute as bytes. Note: the Chinese windows system uses GBK code. You need decode('gbk ') to see familiar Chinese.

interactive input Not all operating system commands simply return execution results like 'dir' or 'ipconfig'. There are many interactive commands like 'python'. You need to enter something and then it returns the execution results. How do I input into stdin using the run() method?

Refer to the following usage in a 1 Txt file, and then:

import subprocess
fd = open("d:\\1.txt") 
ret = subprocess.run("python", stdin=fd, stdout=subprocess.PIPE,shell=True)
print(ret.stdout)
fd.close()

Although this can achieve the goal, it is inconvenient and not code driven. At this time, we can use the Popen class.

subprocess.Popen() class

The usage and parameters are basically the same as those of the run() method, but its return value is a Popen object, not a CompletedProcess object.

>>> ret = subprocess.Popen("dir", shell=True)
>>> type(ret)
<class 'subprocess.Popen'>
>>> ret
<subprocess.Popen object at 0x0000000002B17668>
Popen Object stdin,stdout and stderr Are three file handles that can be read and written like files.
>>>s = subprocess.Popen("ipconfig", stdout=subprocess.PIPE, shell=True)
>>>print(s.stdout.read().decode("GBK"))

To implement the above 'python' command function, you can follow the following example:

import subprocess
s = subprocess.Popen("python", stdout=subprocess.PIPE, stdin=subprocess.PIPE, 
shell=True)
s.stdin.write(b"import os\n")
s.stdin.write(b"print(os.environ)")
s.stdin.close()
out = s.stdout.read().decode("GBK")
s.stdout.close()
print(out)

Via s.stdin Write () can input data, while s.stdout Read() can output data.