The most common task done with Python is to read and write files. Whether it's writing simple text files, reading complex server logs, or analyzing original byte data. All of these situations require reading or writing files.
In this tutorial, you will learn:
- The composition of the file and why this is important in Python
- The basis of reading and writing files in Python
- Some scenarios of reading and writing files in Python
This tutorial is mainly for beginners to intermediate Python developers, but here are some tips for more advanced programmers to benefit from.
What is a file?
Before we begin to study how to use files in Python, it is important to understand what files are and how modern operating systems deal with some aspects of them.
In essence, documents are For storing data A contiguous set of bytes. These data are organized in a specific format, which can be any data as simple as a text file or as complex as a program executable. Finally, these byte files are translated into binary files 1, 0 for easier processing by the computer.
Files on most modern file systems consist of three main parts:
- **Header: * * metadata about file content (file name, size, type, etc.)
- **Data: * * file content written by creator or editor
- **End of file (EOF): * * indicates a special character at the end of the file
[
](https://link.juejin.cn/?target=https%3A%2F%2Fp1-jj.byteimg.com%2Ftos-cn-i-t2oaga2asx%2Fgold-user-assets%2F2019%2F2%2F24%2F1691d8e383f30621~tplv-t2oaga2asx-image.image "https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2019/2/24/1691d8e383f30621~tplv-t2oaga2asx-image.image")
The content of the data representation depends on the format specification used and is usually represented by an extension. For example, the extension is gif files are most likely to match Graphics Interchange Format standard. There are hundreds, if not thousands File extension . For this tutorial, you will only deal with txt or csv file extension.
File path
When accessing files on the operating system, a file path is required. The file path is a string representing the location of the file. It is divided into three main parts:
- **Folder path: * * folder location on the file system. Subsequent folders are separated by forward slash / (Unix) or backslash \ (Windows)
- File name: the actual name of the file
- **Extension: * * a period (.) is preset at the end of the file path, Used to represent the file type
This is a simple example. Suppose you have a file in the file structure, as follows:
/ │ ├── path/ | │ │ ├── to/ │ │ └── cats.gif │ │ │ └── dog_breeds.txt | └── animals.csv
Suppose you want to access the cats GIF file, and your current location is at the same level as the path in the folder. To access the file, you need to browse the path folder, then view the to folder, and finally reach the cats GIF file. The folder path is path/to /. The file name is cats. The file extension is gif. So the complete path is path/to / cats gif.
Now assume that your current location or current working directory (cwd) is in the to folder of our sample folder structure. You can simply refer to the file cats. Net by file name and extension Gif instead of referring to cats Gif full path path / to / cats gif .
/ │ ├── path/ | │ | ├── to/ ← Your current working directory (cwd) is here | │ └── cats.gif ← Accessing this file | │ | └── dog_breeds.txt | └── animals.csv
But for dog_breeds.txt how to access it? If you don't use the full path, how will you access it? You can use the special character double dot (..) To move a directory forward. This means that you can use it in the to directory/ dog_breeds.txt reference dog_breeds.txt file.
/ │ ├── path/ ← Referencing this parent folder | │ | ├── to/ ← Current working directory (cwd) | │ └── cats.gif | │ | └── dog_breeds.txt ← Accessing this file | └── animals.csv
Double point (..) You can connect together to traverse multiple directories before the current directory. For example, in the to folder, you want to access animals CSV, you will use.. // animals.csv.
End of line
One of the problems often encountered when processing file data is the representation of new lines or the end of lines. The end of the line originated in the Morse code era, The use of a specific symbol is used to indicate the end of a transmission or the end of a line.
Later, the international organization for Standardization (ISO) and the American Standards Association (ASA) For teletypewriter Standardization was carried out. ASA standard specifies that carriage return (sequence CR or \ r) should be used at the end of the line_ And_ Line feed (LF or \ n) character (CR+LF or \ r\n). However, the ISO standard allows CR+LF characters or only lf characters.
Windows uses CR+LF characters Represents a new line, while Unix and newer Mac versions use only LF characters. This can lead to some complications when you deal with files from different operating systems. This is a simple example. Assume that dog is created on our Windows file system_ breeds. txt:
Pug\r\n Jack Russel Terrier\r\n English Springer Spaniel\r\n German Shepherd\r\n Staffordshire Bull Terrier\r\n Cavalier King Charles Spaniel\r\n Golden Retriever\r\n West Highland White Terrier\r\n Boxer\r\n Border Terrier\r\n
The same output will be interpreted differently on Unix devices:
Pug\r \n Jack Russel Terrier\r \n English Springer Spaniel\r \n German Shepherd\r \n Staffordshire Bull Terrier\r \n Cavalier King Charles Spaniel\r \n Golden Retriever\r \n West Highland White Terrier\r \n Boxer\r \n Border Terrier\r \n
This may cause repeated problems on each line, which you may need to consider.
Character encoding
Another common problem you may face is the encoding of byte data. Encoding is the conversion from byte data to human readable characters. This is usually done by specifying the encoding format. The two most common codes are ASCII and UNICODE Format. ASCII can only store 128 characters , and Unicode can contain up to 1114112 characters.
ASCII is actually a subset of Unicode (UTF-8), which means that ASCII and Unicode share the same numeric character values. It is important to note that parsing files with incorrect character encoding may lead to character conversion failure and error. For example, if a file is created using UTF-8 encoding and you try to parse it using ASCII encoding, an error will be raised if there are characters beyond these 128 values.
Opening and closing files in Python
When you want to use a file, the first thing to do is open it. This is done by calling open() Built in functions. open() has a required parameter, which is the path to the file. open() has a return, which is the name of the file File object:
file = open('dog_breeds.txt')
After opening the file, the next thing to learn is how to close it.
**Warning: * * you should_ Always_ Make sure you close the open file correctly.
It is important to remember that it is your responsibility to close the file. In most cases, the file will eventually be closed when the application or script terminates. But there is no guarantee of what will actually happen. This can lead to unnecessary behavior, including resource leakage. This is also a best practice in Python to ensure that your code runs in a clearly defined way and reduces any unwanted behavior.
When you work with a file, there are two ways to ensure that the file is closed correctly, even if there are errors. The first way to close a file is to use a try finally block:
reader = open('dog_breeds.txt')
try:
# Further file processing goes here
finally:
reader.close()
If you are not familiar with the contents of the try finally block, please check Python Exceptions: An Introduction.
The second way to close a file is to use the following with statement:
with open('dog_breeds.txt') as reader:
# Further file processing goes here
Using the with statement, the system will automatically close the file once the with block is left or even in case of error. I strongly recommend that you use the with statement as much as possible because its code is clearer and makes it easier for you to deal with any unexpected errors.
Most likely, you also want to use the second location parameter mode. This parameter is a string containing multiple characters to indicate how you want to open the file. The default and most common is' r ', which means that the file is opened as a text file in read-only mode:
with open('dog_breeds.txt', 'r') as reader:
# Further file processing goes here
See other modes Online documents , but the most common modes are as follows:
| pattern | meaning |
|---|---|
| 'r' | Read only mode on (default) |
| 'w' | When write mode is on, the file will be overwritten |
| 'rb' or 'wb' | Open in binary mode (read / write with byte data) |
Let's go back to file objects. File objects are:
"Expose file oriented API s (using methods such as read()or write()) to objects of underlying resources." ( source)
There are three different types of file objects:
- text file
- Buffered binary
- Original binary
The of each of these file types is defined in the io module. These three types are briefly introduced here.
text file
Text files are the most common files you will encounter. Here are some examples of how to open these files:
open('abc.txt')
open('abc.txt', 'r')
open('abc.txt', 'w')
For this type of file, open() will return a TextIOWrapper file object:
>>> file = open('dog_breeds.txt')
>>> type(file)
<class '_io.TextIOWrapper'>
This is the file object returned by open() by default.
Buffer binary file type
The buffered binary file type is used to read and write binaries. Here are some examples of how to open these files:
open('abc.txt', 'rb')
open('abc.txt', 'wb')
For this type of file, open() will return a BufferedReader or BufferedWriter file object:
>>> file = open ('dog_breeds.txt' ， 'rb' )
>>> type (file )
<class'_io.BufferedReader'>
>>> file = open ('dog_breeds.txt' ， 'wb' )
> >> type (file )
<class'_io.BufferedWriter'>
Original file type
The original file type is:
"Typically used as a low-level building block for binary and text streams." ( source)
Therefore, it is not usually used.
The following is an example of how to open these files:
open('abc.txt', 'rb', buffering=0)
For this type of file, open() will return a FileIO file object:
>>> file = open('dog_breeds.txt', 'rb', buffering=0)
>>> type(file)
<class '_io.FileIO'>
Read and write open files
After opening the file, you will need to read or write the file. First, let's read a file. Several methods can be called on file objects:
| method | describe |
|---|---|
| .read(size=-1) | This will be read from the file based on the number of size bytes. If no parameter or None or - 1 is passed, the entire file is read. |
| .readline(size=-1) | This will read the maximum number of characters from this line. Until the end of the line, then to the next line. If no parameter is passed or None or - 1, the whole line (or the rest of the line) is read out. |
Use the dog used above_ breeds. Txt file, let's look at some examples of how to use these methods. Here's how to use it An example of the read() command opening and reading an entire file:
>>> with open('dog_breeds.txt', 'r') as reader:
>>> # Read & print the entire file
>>> print(reader.read())
Pug
Jack Russel Terrier
English Springer Spaniel
German Shepherd
Staffordshire Bull Terrier
Cavalier King Charles Spaniel
Golden Retriever
West Highland White Terrier
Boxer
Border Terrier
This is a how to use Example of readline() reading 5 bytes at a time in one line:
>>> with open('dog_breeds.txt', 'r') as reader:
>>> # Read & print the first 5 characters of the line 5 times
>>> print(reader.readline(5))
>>> # Notice that line is greater than the 5 chars and continues
>>> # down the line, reading 5 chars each time until the end of the
>>> # line and then "wraps" around
>>> print(reader.readline(5))
>>> print(reader.readline(5))
>>> print(reader.readline(5))
>>> print(reader.readline(5))
Pug
Jack
Russe
l Ter
rier
Translator's note: call reader for the first time ReadLine (5) actually prints out Pug\r\n, so you can see that there is a line feed output
The following are used readlines() example of reading the entire file as a list:
>>> f = open('dog_breeds.txt')
>>> f.readlines() # Returns a list object
['Pug\n', 'Jack Russel Terrier\n', 'English Springer Spaniel\n', 'German Shepherd\n', 'Staffordshire Bull Terrier\n', 'Cavalier King Charles Spaniel\n', 'Golden Retriever\n', 'West Highland White Terrier\n', 'Boxer\n', 'Border Terrier\n']
The above example can also be completed by using list() to create a list from a file object:
>>> f = open('dog_breeds.txt')
>>> list(f)
['Pug\n', 'Jack Russel Terrier\n', 'English Springer Spaniel\n', 'German Shepherd\n', 'Staffordshire Bull Terrier\n', 'Cavalier King Charles Spaniel\n', 'Golden Retriever\n', 'West Highland White Terrier\n', 'Boxer\n', 'Border Terrier\n']
Iterate over each line in the file
A common thing when reading a file is to iterate over each line. Here's how to use it Example of readline() performing this iteration:
>>> with open('dog_breeds.txt', 'r') as reader:
>>> # Read and print the entire file line by line
>>> line = reader.readline()
>>> while line != '': # The EOF char is an empty string
>>> print(line, end='')
>>> line = reader.readline()
Pug
Jack Russel Terrier
English Springer Spaniel
German Shepherd
Staffordshire Bull Terrier
Cavalier King Charles Spaniel
Golden Retriever
West Highland White Terrier
Boxer
Border Terrier
Another way to iterate over each line in a file is to use readlines() file object. Remember readlines() returns a list where each element in the list represents a line in the file:
>>> with open('dog_breeds.txt', 'r') as reader:
>>> for line in reader.readlines():
>>> print(line, end='')
Pug
Jack Russell Terrier
English Springer Spaniel
German Shepherd
Staffordshire Bull Terrier
Cavalier King Charles Spaniel
Golden Retriever
West Highland White Terrier
Boxer
Border Terrier
However, the above example can be further simplified by iterating over the file object itself:
>>> with open('dog_breeds.txt', 'r') as reader:
>>> # Read and print the entire file line by line
>>> for line in reader:
>>> print(line, end='')
Pug
Jack Russel Terrier
English Springer Spaniel
German Shepherd
Staffordshire Bull Terrier
Cavalier King Charles Spaniel
Golden Retriever
West Highland White Terrier
Boxer
Border Terrier
The final method is more python, which can be faster and more efficient. Therefore, I suggest you use it instead.
**Note: * * some of the above examples include print('some text', end =' '). This end = '' is to prevent Python from adding extra line breaks to the text being printed and printing only what is read from the file.
Now let's delve into the document. There are many ways to read and write files, just like files:
| method | describe |
|---|---|
| .write(string) | Writes a string to a file. |
| .writelines(seq) | Writes the sequence to a file. No terminator is appended to each sequence item. It's up to you to add the appropriate terminator. |
The following are used A simple example of write() and writelines():
with open('dog_breeds.txt', 'r') as reader:
# Note: readlines doesn't trim the line endings
dog_breeds = reader.readlines()
with open('dog_breeds_reversed.txt', 'w') as writer:
# Alternatively you could use
# writer.writelines(reversed(dog_breeds))
# Write the dog breeds to the file in reversed order
for breed in reversed(dog_breeds):
writer.write(breed)
Use byte
Sometimes you may need to use Byte string Process files. This can be done by adding the 'b' character to the mode parameter. All the same methods apply to file objects. However, each method expects and returns a bytes object:
>>> with open(`dog_breeds.txt`, 'rb') as reader: >>> print(reader.readline()) b'Pug\n'
Using the b flag to open text files is not so interesting. Suppose we have a lovely picture of Jack Russell Terrier (jack_russell.png):
You can open the file in Python and check the contents! because . png file format As defined, the title of the file is 8 bytes, as follows:
| value | describe |
|---|---|
| 0x89 | A "magic" number indicates that this is the beginning of a PNG |
| 0x50 0x4E 0x47 | ASCII for PNG |
| 0x0D 0x0A | End of DOS style line \ r\n |
| 0x1A | DOS style EOF characters |
| 0x0A | End of a Unix style line \ n |
When you open the file and read these bytes separately, you can see that this is indeed a problem png header file:
>>> with open('jack_russell.png', 'rb') as byte_reader:
>>> print(byte_reader.read(1))
>>> print(byte_reader.read(3))
>>> print(byte_reader.read(2))
>>> print(byte_reader.read(1))
>>> print(byte_reader.read(1))
b'\x89'
b'PNG'
b'\r\n'
b'\x1a'
b'\n'
A complete example: dos2unix.py
Let's sort out the knowledge points and see a complete example of how to read and write files. Here is a dos2unix A similar tool converts it to a file and converts the end of its line \ R \ nto \ n.
The tool is divided into three main parts. The first is str2unix(), which converts a string from the end of \ \ R \ \ nline to \ \ n. The second is that dos2unix() converts a string containing \ r\n characters to \ n. Dos2unix() calls str2unix(). Finally, yes__ main__ Block, which is called only when the file is executed as a script.
"""
A simple script and library to convert files or strings from dos like
line endings with Unix like line endings.
"""
import argparse
import os
def str2unix(input_str: str) -> str:
r"""\
Converts the string from \r\n line endings to \n
Parameters
----------
input_str
The string whose line endings will be converted
Returns
-------
The converted string
"""
r_str = input_str.replace('\r\n', '\n')
return r_str
def dos2unix(source_file: str, dest_file: str):
"""\
Coverts a file that contains Dos like line endings into Unix like
Parameters
----------
source_file
The path to the source file to be converted
dest_file
The path to the converted file for output
"""
# NOTE: Could add file existence checking and file overwriting
# protection
with open(source_file, 'r') as reader:
dos_content = reader.read()
unix_content = str2unix(dos_content)
with open(dest_file, 'w') as writer:
writer.write(unix_content)
if __name__ == "__main__":
# Create our Argument parser and set its description
parser = argparse.ArgumentParser(
description="Script that converts a DOS like file to an Unix like file",
)
# Add the arguments:
# - source_file: the source file we want to convert
# - dest_file: the destination where the output should go
# Note: the use of the argument type of argparse.FileType could
# streamline some things
parser.add_argument(
'source_file',
help='The location of the source '
)
parser.add_argument(
'--dest_file',
help='Location of dest file (default: source_file appended with `_unix`',
default=None
)
# Parse the args (argparse automatically grabs the values from
# sys.argv)
args = parser.parse_args()
s_file = args.source_file
d_file = args.dest_file
# If the destination file wasn't passed, then assume we want to
# create a new file based on the old one
if d_file is None:
file_path, file_extension = os.path.splitext(s_file)
d_file = f'{file_path}_unix{file_extension}'
dos2unix(s_file, d_file)
Tips and tricks
Now that you have mastered the basics of reading and writing files, here are some tips and techniques to help you improve your skills.
__file__
Should__ file__ Module properties are Special properties , similar to__ name__. It is:
"If it is loaded from a file, it is the pathname of the file where the module is loaded,"( source)
Note:__ file__ Return_ Phase_ For the path to call the initial Python script. If you need a complete system path, you can use OS Getcwd() gets the current working directory where the code is executed.
This is a real example. In my past job, I tested the hardware equipment many times. Each test is written in Python script, and the test script file name is used as the title. These scripts are then executed and used__ file__ Special properties print their status. This is an example folder structure:
project/ | ├── tests/ | ├── test_commanding.py | ├── test_power.py | ├── test_wireHousing.py | └── test_leds.py | └── main.py
Run main Py produces the following:
>>> python main.py tests/test_commanding.py Started: tests/test_commanding.py Passed! tests/test_power.py Started: tests/test_power.py Passed! tests/test_wireHousing.py Started: tests/test_wireHousing.py Failed! tests/test_leds.py Started: tests/test_leds.py Passed!
Add file content
Sometimes you may want to append to a file or start writing at the end of an existing file. This can be done by appending the 'a' character to the parameter mode:
with open('dog_breeds.txt', 'a') as a_writer:
a_writer.write('\nBeagle')
When it comes to dog_breeds.txt when you check again, you will see that the beginning of the file has not changed and Beagle has now been added to the end of the file:
>>> with open('dog_breeds.txt', 'r') as reader:
>>> print(reader.read())
Pug
Jack Russel Terrier
English Springer Spaniel
German Shepherd
Staffordshire Bull Terrier
Cavalier King Charles Spaniel
Golden Retriever
West Highland White Terrier
Boxer
Border Terrier
Beagle
Use two files at the same time
Sometimes you may want to read a file and write to another file at the same time. If you use the example shown when learning how to write a file, it can actually be incorporated into the following:
d_path = 'dog_breeds.txt'
d_r_path = 'dog_breeds_reversed.txt'
with open(d_path, 'r') as reader, open(d_r_path, 'w') as writer:
dog_breeds = reader.readlines()
writer.writelines(reversed(dog_breeds))
Create your own context manager
Sometimes you may need to better control file objects by putting them in custom classes. When doing this, you cannot use the with statement unless you add some magic methods: by adding__ enter__ And__ exit__， You will create what is called Context manager.
__ enter__ () called when the with statement is called__ exit__ () is called when exiting from the with statement block.
This is a template that can be used to create custom classes:
class my_file_reader():
def __init__(self, file_path):
self.__path = file_path
self.__file_object = None
def __enter__(self):
self.__file_object = open(self.__path)
return self
def __exit__(self, type, val, tb):
self.__file_object.close()
# Additional methods implemented below
Now that you have a custom class with a context manager, you can use it the same way you use the built-in open():
with my_file_reader('dog_breeds.txt') as reader:
# Perform custom class operations
pass
This is a good example. Remember when we had the cute image of Jack Russell? Maybe you want to open another one png file, but I don't want to parse the header file every time. This is an example of how to do this. This example also uses a custom iterator. If you are not familiar with them, please check them out Python iterator:
class PngReader():
# Every .png file contains this in the header. Use it to verify
# the file is indeed a .png.
_expected_magic = b'\x89PNG\r\n\x1a\n'
def __init__(self, file_path):
# Ensure the file has the right extension
if not file_path.endswith('.png'):
raise NameError("File must be a '.png' extension")
self.__path = file_path
self.__file_object = None
def __enter__(self):
self.__file_object = open(self.__path, 'rb')
magic = self.__file_object.read(8)
if magic != self._expected_magic:
raise TypeError("The File is not a properly formatted .png file!")
return self
def __exit__(self, type, val, tb):
self.__file_object.close()
def __iter__(self):
# This and __next__() are used to create a custom iterator
# See https://dbader.org/blog/python-iterators
return self
def __next__(self):
# Read the file in "Chunks"
# See https://en.wikipedia.org/wiki/Portable_Network_Graphics#%22Chunks%22_within_the_file
initial_data = self.__file_object.read(4)
# The file hasn't been opened or reached EOF. This means we
# can't go any further so stop the iteration by raising the
# StopIteration.
if self.__file_object is None or initial_data == b'':
raise StopIteration
else:
# Each chunk has a len, type, data (based on len) and crc
# Grab these values and return them as a tuple
chunk_len = int.from_bytes(initial_data, byteorder='big')
chunk_type = self.__file_object.read(4)
chunk_data = self.__file_object.read(chunk_len)
chunk_crc = self.__file_object.read(4)
return chunk_len, chunk_type, chunk_data, chunk_crc
You can open it now png files and correctly resolve them using the custom context manager:
>>> with PngReader('jack_russell.png') as reader:
>>> for l, t, d, c in reader:
>>> print(f"{l:05}, {t}, {c}")
00013, b'IHDR', b'v\x121k'
00001, b'sRGB', b'\xae\xce\x1c\xe9'
00009, b'pHYs', b'(<]\x19'
00345, b'iTXt', b"L\xc2'Y"
16384, b'IDAT', b'i\x99\x0c('
16384, b'IDAT', b'\xb3\xfa\x9a$'
16384, b'IDAT', b'\xff\xbf\xd1\n'
16384, b'IDAT', b'\xc3\x9c\xb1}'
16384, b'IDAT', b'\xe3\x02\xba\x91'
16384, b'IDAT', b'\xa0\xa99='
16384, b'IDAT', b'\xf4\x8b.\x92'
16384, b'IDAT', b'\x17i\xfc\xde'
16384, b'IDAT', b'\x8fb\x0e\xe4'
16384, b'IDAT', b')3={'
01040, b'IDAT', b'\xd6\xb8\xc1\x9f'
00000, b'IEND', b'\xaeB`\x82'
Don't build wheels again
You may encounter common situations when working with files. In most cases, other modules can be used. The two common file types you may need to use are csv and json. _ Real Python_ I've compiled some wonderful articles on how to deal with these contents:
In addition, there are built-in libraries that can be used to help you:
- wave : read and write WAV files (audio)
- aifc : read and write AIFF and AIFC files (audio)
- sunau : reading and writing Sun AU files
- tarfile : read and write tar Archive
- zipfile : use ZIP archive
- configparser : easily create and parse configuration files
- xml.etree.ElementTree : create or read XML based files
- msilib : reading and writing Microsoft Installer files
- plistlib : generate and parse Mac OS X Plist file
There's more. In addition, PyPI has more third-party tools available. Some popular are the following:
- PyPDF2 : PDF Toolkit
- xlwings : reading and writing Excel files
- Pillow : image reading and manipulation
summary
You now know how to use Python to process files, including some advanced technologies. Using files in Python is now easier than ever, and it's a good feeling when you start doing so.
In this tutorial, you have learned:
- What is a file
- How to open and close files correctly
- How to read and write files
- Some advanced techniques for working with files
- Some libraries use common file types
Finally, I'll share a full set of Python learning materials to help those who want to learn Python!
1, Python learning routes in all directions
All directions of Python is to sort out the commonly used technical points of Python and form a summary of knowledge points in various fields. Its purpose is that you can find corresponding learning resources according to the above knowledge points to ensure that you learn more comprehensively.
2, Learning software
If a worker wants to do well, he must sharpen his tools first. The commonly used development software for learning Python is here, which saves you a lot of time.
3, Getting started video
4, Actual combat cases
5, Interview materials
If necessary, you can scan the CSDN official authentication QR code below on wechat and get it for free
