Skip to content

Python File I/O: Reading, Writing, and pathlib

  • Opening and reading files with open() and context managers.
  • Write modes, text vs binary, and encoding.
  • Using pathlib.Path for cross-platform path operations.
  • Reading and writing CSV and JSON — the two most common file formats in practice.
  • Safe patterns for temporary files and atomic writes.

Python provides simple built-in file I/O through open() and the pathlib module. The key rule: always use a with statement when opening files. It guarantees the file handle is closed even if an exception occurs mid-read.

Save examples as .py and run with python examples.py (Python 3.9+).

# Reading a text file
with open("orders.txt", encoding="utf-8") as f:
content = f.read() # read entire file as a string
with open("orders.txt", encoding="utf-8") as f:
lines = f.readlines() # list of lines, each including \n
with open("orders.txt", encoding="utf-8") as f:
for line in f: # most memory-efficient: iterate line by line
print(line.rstrip()) # strip trailing newline
Mode Meaning
"r" Read text (default)
"w" Write text, truncate file if it exists
"a" Append text, create if missing
"x" Exclusive create — fail if file exists
"r+" Read and write, file must exist
"rb", "wb" Read/write binary (images, PDFs, etc.)
# Writing a text file — creates or overwrites
with open("report.txt", "w", encoding="utf-8") as f:
f.write("Order Report\n")
f.write("=============\n")
# Appending — does not truncate existing content
with open("report.txt", "a", encoding="utf-8") as f:
f.write("ORD-001: BensonCo $108.00\n")

Always specify encoding="utf-8" explicitly. The default encoding is platform-dependent (cp1252 on Windows, utf-8 on Linux/macOS), which causes silent data corruption when files are shared across systems.

pathlib.Path is the right way to work with file paths in Python 3. It is cross-platform, composable, and more readable than string concatenation with os.path.

from pathlib import Path
# Build paths without os.path.join
base = Path("/var/data/orders-portal")
reports_dir = base / "reports"
today_report = reports_dir / "2026-11-13.csv"
print(today_report.name) # 2026-11-13.csv
print(today_report.stem) # 2026-11-13
print(today_report.suffix) # .csv
print(today_report.parent) # /var/data/orders-portal/reports
# Check existence
print(today_report.exists()) # False (doesn't exist yet)
print(today_report.is_file()) # False
print(reports_dir.is_dir()) # depends on the filesystem
# Create directories
reports_dir.mkdir(parents=True, exist_ok=True) # like mkdir -p
# Read and write through pathlib (no open() needed for simple cases)
today_report.write_text("order_id,amount\n", encoding="utf-8")
content = today_report.read_text(encoding="utf-8")
# Glob for all CSV files
for csv_file in reports_dir.glob("*.csv"):
print(csv_file.name)
# Recursive glob
for py_file in Path(".").rglob("*.py"):
print(py_file)
# Rename and delete
backup = today_report.with_suffix(".bak")
today_report.rename(backup)
backup.unlink() # delete the file
import csv
from pathlib import Path
orders = [
{"order_id": "ORD-001", "customer": "BensonCo", "amount": "108.00"},
{"order_id": "ORD-002", "customer": "NovaCorp", "amount": "270.00"},
]
# Write CSV
out_path = Path("orders.csv")
with out_path.open("w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["order_id", "customer", "amount"])
writer.writeheader()
writer.writerows(orders)
# Read CSV
with out_path.open(encoding="utf-8") as f:
reader = csv.DictReader(f)
for row in reader:
print(row["order_id"], row["amount"])
# ORD-001 108.00
# ORD-002 270.00

Pass newline="" when opening CSV files for writing — the csv module handles line endings itself. Without it, you get extra blank lines on Windows.

import json
from pathlib import Path
config = {
"app": "OrdersPortal",
"tax_rate": 0.08,
"features": ["invoicing", "shipping", "analytics"],
}
config_path = Path("config.json")
# Write JSON
config_path.write_text(json.dumps(config, indent=2), encoding="utf-8")
# Read JSON
loaded = json.loads(config_path.read_text(encoding="utf-8"))
print(loaded["tax_rate"]) # 0.08

For large JSON files that don’t fit in memory, use the json module’s streaming alternatives from third-party libraries like ijson.

# Read a PDF or image as bytes
pdf_path = Path("invoice.pdf")
raw_bytes = pdf_path.read_bytes()
# Write binary data
output = Path("invoice_copy.pdf")
output.write_bytes(raw_bytes)

A common production pattern: write to a temporary file first, then rename. The rename is atomic on most operating systems — a reader either sees the old version or the new one, never a half-written file.

import tempfile
from pathlib import Path
def write_atomically(target: Path, content: str) -> None:
"""Write content to target path atomically via a temp file."""
target.parent.mkdir(parents=True, exist_ok=True)
# Write to a temp file in the same directory
with tempfile.NamedTemporaryFile(
mode="w",
encoding="utf-8",
dir=target.parent,
delete=False,
suffix=".tmp",
) as tmp:
tmp.write(content)
tmp_path = Path(tmp.name)
# Atomic rename
tmp_path.replace(target) # replace() is cross-platform
write_atomically(Path("data/report.txt"), "Order Report\n")
  • Missing encoding="utf-8". The default encoding is platform-dependent. Files written on Windows (cp1252) and read on Linux (utf-8) will corrupt characters above ASCII 127.
  • Not using with. If an exception occurs between open() and close(), the file handle leaks. Always use with open(...) as f:.
  • Opening a CSV with newline=None for writing. On Windows, this adds an extra \r before each \n, producing double line breaks when the file is opened in a text editor.
  • Reading a whole large file into memory with f.read(). For files that are larger than available RAM, iterate line by line with for line in f:.
  • Hardcoding path separators. "data\\reports" breaks on Linux/macOS. Use Path("data") / "reports" or Path("data/reports") — pathlib normalizes separators automatically.

1. Why should you always use a with statement when opening files? The with statement calls __enter__ on entry and guarantees __exit__ is called on exit, even if an exception is raised. For file objects, __exit__ closes the file handle. Without with, a forgotten close() call or an exception mid-read leaves the file descriptor open, which on systems with low file descriptor limits (e.g. Linux default 1024) causes OSError: [Errno 24] Too many open files under load.

2. What is the difference between Path.rename() and Path.replace()? rename() raises FileExistsError on Windows if the destination exists. replace() atomically replaces the destination on both POSIX and Windows — it is the correct method to use for atomic file replacement and is safer in production.

3. Senior scenario: an OrdersPortal batch job writes a 500 MB CSV export to disk every hour. The file is served to users via a web endpoint. How do you prevent users from downloading a partially written file? Write to a .tmp file in the same directory, then call tmp_path.replace(final_path). The rename is atomic — the web server either serves the previous complete file or the new complete file, never an incomplete write in progress. Additionally, serve the file from an object store (S3, Azure Blob) rather than the local filesystem for durability and horizontal scaling; the upload itself can be atomic using multipart upload with a final commit step.

  • Always use with open(...) as f: — it guarantees the file is closed.
  • Always specify encoding="utf-8" for text files.
  • Use pathlib.Path for all path manipulation — it is cross-platform and composable.
  • Pass newline="" when opening CSV files for writing.
  • Use temp file + atomic rename for files that are read while being written.