Chapter 10: Files and Data Formatting
As the scale of programs increases, we no longer only deal with temporary data in memory, but need to save data for the long term or share it across programs. Files are the most basic means of data persistence and an important bridge for Python to communicate with the outside world.
In modern programming, files are not just about “reading and writing text”; they also represent forms of data organization.
Through different file formats (such as .txt, .csv, .json), we can store and process one-dimensional lists, two-dimensional tables, and even high-dimensional structured data.
Learning Objectives:
(1) Understand the concept, classification, and role of files in programs.
(2) Master the operations of opening, reading, writing, and closing files in Python.
(3) Proficiently use the with context and exception handling to achieve safe file operations.
(4) Understand the organizational dimensions of data (one-dimensional, two-dimensional, high-dimensional) and common formats (CSV, JSON).
(5) Be able to save program data as text or structured files and reload them for use.
10.1 Overview of Files
When a program ends, all variables and data will disappear from memory. To save the results of program execution, log records, or share information between different programs, we need to use files.
Files are the long-term storage form of data on external storage media (such as disks) and are the core means of achieving data persistence.
Python provides a simple yet powerful file operation interface, making it very convenient to save results and load data.
10.1.1 Files and Paths
Files consist of a filename and a path. The path describes the location of the file in computer storage.
(1) Absolute Path: The complete path starting from the root directory of the disk.
For example:
"C:\Users\mediaTEA\Documents\data.txt"(2) Relative Path: The path relative to the directory where the current program is located.
For example:
"data.txt" # File in the current directory"data\result.txt" # File in the data folder in the current directoryIn Python, the commonly used os or pathlib modules handle paths:
from pathlib import Path
cwd = Path.cwd()
print(cwd)
data_dir = cwd / "data"
print(data_dir.exists())10.1.2 File Types
Files are mainly divided into two categories based on their storage form.
(1) Text Files
• Store data in character form, such as .txt, .csv, .json.
• Can be opened with text editors.
• Usually require specifying encoding (e.g., UTF-8).
(2) Binary Files
• Store data in byte form, such as images, audio, model files (.jpg, .mp3, .pkl).
• Cannot be viewed directly in text form.
10.1.3 File Encoding
The default encoding in text mode for open() depends on the locale settings of the running environment, so it is recommended to explicitly specify encoding=”utf-8″ to avoid cross-platform encoding issues.
with open("data.txt", "r", encoding="utf-8") as f: print(f.read())Tip:
You can check the system locale using the standard library locale.getpreferredencoding().
10.1.4 Dimensions of Data Organization
Files are not just containers for saving content; they also determine how data is organized.
Python supports various dimensions of data representation, from one-dimensional sequences to two-dimensional tables and high-dimensional nested structures. Different dimensions of data correspond to different file formats and reading methods.
10.2 Using Files
The file operation process can be summarized in three steps:
Open (open) → Operate (read/write) → Close (close)
In Python, you can manually control file closure or automatically manage resources using the with context.
10.2.1 Opening and Closing Files
Use the built-in function open() to open a file.
Basic syntax structure:
f = open(file, mode, encoding=None) # Open file, return a file objectCommon mode descriptions:
'r': Read-only mode. Default mode, raises an exception if the file does not exist.'w': Write mode (overwrite). Creates the file if it does not exist, clears it completely before writing if it exists.'x': Write mode (create only). Raises an error if the file exists.'a': Append mode. Creates the file if it does not exist, appends content to the end if it exists.'b': Binary mode (e.g., 'rb', 'wb'), does not use encoding.'t': Text mode (default).'+': Read-write mode (used in combination with r/w/x/a). 'r+' reads and writes from the beginning; 'w+' truncates before reading and writing; 'a+' appends and can read.Example:
f = open("test.txt", "w", encoding="utf-8")
f.write("Hello, file world!\n")
f.close()Tip: Always call .close() after writing to close the file and release resources.
10.2.2 with Context Manager
The with statement can automatically handle file closure, making it the recommended way to operate files.
with open("demo.txt", "a", encoding="utf-8") as f: f.write("Appending a line of text.\n")Equivalent to:
f = open("demo.txt", "a", encoding="utf-8")
try: f.write("Appending a line of text.\n")
finally: f.close()Advantages: Even if an error occurs in the program, the file will be safely closed.
10.2.3 Reading and Writing Files
(1) Reading Files
Related methods:
.read(size = -1)
Reads the entire content of the file or a string or byte stream of specified size.
.readline(size = -1)
Reads a line or a string or byte stream of specified size from that line.
Note: The parameter size is the same as that of .read(), indicating the maximum number of characters (text mode) or bytes (binary mode) to return; when size is specified and less than the remaining content length, it will return an incomplete line/data fragment.
.readlines(hint = -1)
Reads all lines and returns a list of line elements. hint is a suggested byte/character size threshold (approximate), not a strict line count limit.
lines = f.readlines(1024) # Approximately read ~1024 bytes of lines, returning a list of lines.seek(offset, whence=0)
Changes the position of the current file operation pointer. offset is the offset relative to whence (in bytes or characters, depending on the file mode). whence specifies the relative position: 0 (beginning of the file), 1 (current position), 2 (end of the file).
# Move pointer to the beginning of the file
f.seek(0, 0)
# Move pointer to the end of the file
f.seek(0, 2) # or f.seek(0, os.SEEK_END)Example 1:
with open("C:\myPRJ\将进酒.txt", "r", encoding="utf-8") as f: text = f.read() # Read all content print(text)Example 2:
with open(r"C:\myPRJ\将进酒.txt", "r", encoding="utf-8") as f: text = f.readlines() # Read all lines, returning a list for line in text: print(line)Example 3:
with open("huge.log", "r", encoding="utf-8") as f: for line in f: # Iterate by line, low memory overhead print(line, end="") Note:
When the file is large, it is recommended to use the line-by-line iteration method for memory efficiency.
(2) Writing Files
Related methods:
.write(s)
Writes a string or byte stream to the file.
.writelines(lines)
Writes a list of strings as a whole to the file.
Example 1:
with open("note.txt", "w", encoding="utf-8") as f: f.write("Recording the first experiment results.\n") f.write("Loss=0.45, Accuracy=0.82\n")Example 2:
ls = ["岱宗夫如何？齐鲁青未了。\n", "造化钟神秀，阴阳割昏晓。\n", "荡胸生曾云，决眦入归鸟。\n", "会当凌绝顶，一览众山小。\n"]
with open("C:\myPRJ\望岳.txt", "w", encoding="utf-8") as f: f.write("《望岳》\n唐 · 杜甫\n") f.writelines(ls)Note:
.writelines() does not automatically add newline characters; if you want to add newlines automatically, the list elements must include \n, and you can also use print(line, file=f) or f.write(“\n”.join(lines)).
10.2.4 File Exception Handling
Common exceptions in file operations include:
• FileNotFoundError: The file does not exist.
• PermissionError: No access permission.
• IsADirectoryError: The path is a folder, not a file.
Example:
try: with open("missing.txt", "r", encoding="utf-8") as f: data = f.read()except FileNotFoundError: print("The file does not exist, please check the path.")except PermissionError: print("No permission to access this file.")except Exception as e: print("An error occurred while reading the file:", e)10.3 Organizing and Processing One-Dimensional Data
Python provides flexible tools to read, store, and process one-dimensional data, allowing us to easily complete tasks such as statistical analysis of experimental results, log file analysis, text data cleaning or filtering, and extracting model prediction results.
10.3.1 Representation and Storage of One-Dimensional Data
The most common one-dimensional data structures are lists (list) and strings (str).
Lists are suitable for storing numeric or mixed data, while strings are often used for raw text.
Example 10.3.1: Writing model loss values to a text file
loss_values = [0.92, 0.84, 0.76, 0.69]
with open("loss.txt", "w", encoding="utf-8") as f: for val in loss_values: f.write(f"{val}\n")
print("Model training loss values have been saved.")The content of the saved file is:
0.920.840.760.69Note:
This method is suitable for linear data that changes in chronological order, such as temperature records, experimental results, training logs, etc.
10.3.2 Reading and Processing One-Dimensional Data
After reading the file, further analysis can be performed through type conversion and built-in functions.
Example 10.3.2: Calculating the average loss value
with open("loss.txt", "r", encoding="utf-8") as f: values = [float(line.strip()) for line in f if line.strip()]if values: avg = sum(values) / len(values) print(f"Average loss value: {avg:.3f}")else: print("No valid data.")Output:
Average loss value: 0.802Further Example: Filtering data within a specific range
filtered = [x for x in values if x < 0.8]
print("Filtered data:", filtered)Output:
Filtered data: [0.76, 0.69]Note:
This filtering operation is very common in AI training monitoring, such as extracting only the losses below a threshold to analyze the convergence stage.
10.3.3 Applications of One-Dimensional Data
Text files can not only be used to save numeric data but are also often used to store event records or system logs.
We can use string methods to search, filter, and count key information.
Example 10.3.3: Filtering abnormal lines in AI training logs
with open("train_log.txt", "r", encoding="utf-8") as f: lines = f.readlines()
# Filter lines containing "Error" or "Fail"
errors = [line.strip() for line in lines if "ERROR" in line or "FAIL" in line]
print(f"Detected {len(errors)} error logs:")for e in errors: print(e)Example input file (train_log.txt):
[INFO] Epoch 1: Loss=0.48[INFO] Epoch 2: Loss=0.46[ERROR] Model failed to converge[INFO] Epoch 3: Loss=0.44Output:
Detected 1 error log: [ERROR] Model failed to convergeNote:
This text filtering approach is consistent with real-world log monitoring systems and is an important skill in AI engineering and system maintenance.
Example 10.3.4: Keyword Counter
Count the occurrences of specified keywords in one-dimensional text data.
from pathlib import Path
def keyword_count(filename, keyword): with open(filename, "r", encoding="utf-8") as f: text = f.read() return text.count(keyword) # .count matches segments and is case-sensitive
cwd = Path.cwd()
data_dir = cwd / "poem" / "将进酒.txt"count = keyword_count(data_dir, "酒")
print(f"Keyword '酒' occurrence count: {count}")Example output:
Keyword '酒' occurrence count: 3Note:
This method can be used for word frequency analysis in natural language processing (NLP) and can be extended for keyword extraction or sentiment statistics tasks.
📘 Summary
This lesson introduced the basic concepts of Python file operations, path handling, file types and encoding, as well as methods for opening, closing, reading, writing, and exception handling of files. Examples demonstrated the saving, reading, filtering, and analysis of one-dimensional data, as well as keyword statistics in logs and texts.
The next lesson will further explain the organization and processing of two-dimensional and high-dimensional data (such as CSV and JSON files), and will combine comprehensive cases for practical file applications, laying a solid foundation for actual data processing and AI training monitoring.
“Likes are beautiful, appreciation is encouragement”