Skip to content

Jupyter Best Practices and Tools

Notebooks are excellent for exploring and explaining, and fragile as soon as they are shared, scheduled or committed. This page collects the habits and tools that keep them reliable: a clear structure, a pinned environment, clean diffs, parameterized headless runs, tests, and the extensions and servers that teams use around JupyterLab. It assumes the basics from Jupyter notebooks: kernels, magics and rich output.

The examples use a small regional revenue report for an invented OrdersPortal shop. Command-line samples were run with Python 3.13, papermill 2.7, nbconvert 7.17, nbval 0.11, nbstripout 0.9 and Jupytext 1.19 (November 2026); outputs are shown where they matter.

Topic What it answers
Notebook structure How to lay out a notebook that reads top to bottom
Reproducibility Environments, seeds, paths and restart-and-run-all
Version control Clean diffs with nbstripout, nbdime and Jupytext
Parameterized runs with papermill Running one notebook many times with different inputs
Converting and executing nbconvert to HTML, PDF, scripts and slides
Testing notebooks Executing notebooks in CI and testing their functions
Generating notebooks Building .ipynb files from code with nbformat
JupyterLab extensions Useful extensions and what is already built in
Sharing and serving Voilà dashboards, JupyterHub, remote servers and VS Code
Secrets and performance Keeping keys out of notebooks and keeping them fast

Write a notebook like a short report that someone can read without running it:

  1. A Markdown title cell that states the question and the data source.
  2. Imports and configuration (constants, paths, parameters) in one place.
  3. Data loading.
  4. Exploration, with a Markdown cell before each step saying what you are checking.
  5. Processing or modeling.
  6. Results and conclusions in Markdown, written after the numbers are in.

Keep one logical step per cell. A cell that loads, cleans, trains and plots cannot be re-run in part, and its failure tells you little. When a cell grows past a screen, move the logic into a function in a .py module, import it, and keep the notebook as the narrative. That also makes the logic testable.

Pitfall: a notebook as the only copy of business logic. If two notebooks paste the same cleaning code, they will drift. Put shared code in a package (ordersportal_analytics/cleaning.py) and import it, with %autoreload 2 while you develop.

A result you cannot reproduce is an anecdote. Four habits cover most problems:

  • Pin the environment. Keep a requirements.txt or a lock file (uv lock, pip-tools, conda env export) next to the notebook, and print key versions in the first cell so a saved output shows what produced it.
  • Fix random seeds for anything stochastic: random.seed(42), numpy.random.default_rng(42), torch.manual_seed(42). GPU kernels can still be non-deterministic; note it when it matters.
  • Use relative paths from the project root (Path("data/orders.csv")), never C:\Users\... or /home/....
  • Restart and run all (Kernel menu) before you commit, share or quote a number. It is the only check against hidden state.
# First cell of the notebook: record what produced these outputs
import sys
import numpy as np
import pandas as pd
print(f"Python {sys.version.split()[0]} | pandas {pd.__version__} | numpy {np.__version__}")
rng = np.random.default_rng(42)

An .ipynb file stores outputs, execution counts and metadata alongside the code, so every run changes the file and diffs are full of base64 images. Three tools fix this:

  • nbstripout removes outputs and execution counts. nbstripout --install adds a Git filter to the repository so outputs are stripped when files are staged, while your working copy keeps them. In the test run, a notebook with seven outputs had zero after nbstripout notebook.ipynb.
  • nbdime gives notebook-aware diff and merge. nbdiff a.ipynb b.ipynb shows cell-level changes, nbdiff-web shows them in the browser, and nbdime config-git --enable makes git diff use it.
  • Jupytext pairs each notebook with a plain-text version (a .py file in “percent” format or Markdown) that is easy to review. jupytext --set-formats ipynb,py:percent notebook.ipynb keeps both in sync; you can commit only the .py file and regenerate the notebook.
Terminal window
python -m pip install nbstripout nbdime jupytext
nbstripout --install # per repository: strip outputs on git add
nbdime config-git --enable # notebook-aware git diff and merge
jupytext --set-formats ipynb,py:percent regional_report.ipynb

Also add .ipynb_checkpoints/ to .gitignore.

Pitfall: stripping outputs from notebooks that are the deliverable. If a rendered report is the point (for example a notebook published as documentation), commit an executed HTML export or let CI execute it, rather than committing large outputs.

papermill executes a notebook headless with injected parameters, which turns an exploratory notebook into a repeatable report. Tag one cell parameters (in JupyterLab: the property inspector, Common Tools > Cell Tags) holding the defaults. papermill inserts a new cell tagged injected-parameters after it with your values, runs everything, and saves the executed copy.

The report notebook used here has a parameters cell with region = "EU" and min_total = 0.0, then a cell that filters the orders and prints a summary:

Terminal window
papermill regional_report.ipynb out_us.ipynb -p region US -p min_total 50

The executed copy out_us.ipynb prints:

US: 1 orders, 89.00

and its cell tags are [], ['parameters'], ['injected-parameters'], [], so the run’s inputs are visible in the output file. From Python, papermill.execute_notebook("regional_report.ipynb", "out_us.ipynb", parameters={"region": "US"}) does the same, which makes it easy to loop over regions or months from a scheduler. If any cell raises, papermill stops and still writes the partial output notebook, so the failure is easy to inspect.

nbconvert turns notebooks into other formats and can execute them first.

Terminal window
jupyter nbconvert --to html --execute regional_report.ipynb --output report_eu # run, then export HTML
jupyter nbconvert --to script regional_report.ipynb # .py file with the code cells
jupyter nbconvert --to webpdf regional_report.ipynb # PDF through a headless browser (needs nbconvert[webpdf])
jupyter nbconvert --to pdf regional_report.ipynb # PDF through LaTeX (needs a TeX installation)
jupyter nbconvert --to slides regional_report.ipynb --post serve # reveal.js slides

The HTML export in the test run wrote report_eu.html (about 280 KB, most of it the embedded styles). Add --no-input to hide code cells in an export for non-technical readers.

Two levels of testing are useful:

  • The notebook runs. The cheapest test is executing it in CI. pytest --nbval-lax notebook.ipynb runs every cell and fails if one raises an error; plain --nbval also compares outputs with the saved ones, which is only useful for deterministic notebooks. In the test run, pytest -q --nbval-lax regional_report.ipynb reported 2 passed (one test item per code cell).
  • The logic is right. Test functions in the package the notebook imports with ordinary pytest. testbook can also call functions defined inside a notebook (tb.ref("add")), but it has not had a release since 2021, so prefer moving logic into modules.

Pitfall: tests that need production data or credentials. Give the notebook a small fixture dataset through a papermill parameter (data_path) so CI can run it without secrets.

nbformat reads and writes .ipynb files, and nbclient executes them, which is how the samples on these pages were produced:

import nbformat
from nbformat.v4 import new_notebook, new_code_cell, new_markdown_cell
from nbclient import NotebookClient
params = new_code_cell('region = "EU"\nmin_total = 0.0')
params.metadata["tags"] = ["parameters"]
nb = new_notebook(cells=[
new_markdown_cell("# Regional revenue report"),
params,
new_code_cell(
'import pandas as pd\n'
'orders = pd.DataFrame({"region": ["EU", "US", "EU", "APAC"],\n'
' "total": [129.50, 89.00, 240.25, 56.75]})\n'
'subset = orders[(orders.region == region) & (orders.total >= min_total)]\n'
'print(f"{region}: {len(subset)} orders, {subset.total.sum():.2f}")'),
])
nb.metadata["kernelspec"] = {"name": "python3", "display_name": "Python 3", "language": "python"}
NotebookClient(nb, timeout=60, kernel_name="python3").execute()
print(nb.cells[2].outputs[0].text, end="")
nbformat.write(nb, "regional_report.ipynb")
EU: 2 orders, 369.75

JupyterLab 4 extensions are prebuilt and installed with pip into the environment that runs JupyterLab; restart JupyterLab afterwards and check them with jupyter labextension list. Several features that used to need extensions are now built in: the table of contents, the visual debugger (for kernels that support it, including ipykernel) and a property inspector for cell tags.

Package What it adds
jupyterlab-git A Git panel: stage, commit, diff and push from the sidebar
jupyterlab-execute-time Start time and duration under each executed cell
jupyterlab-code-formatter Format cells with Black, Ruff or isort
jupyterlab-spellchecker Spell checking in Markdown cells
jupyter-resource-usage Memory and CPU use of the kernel in the status bar
lckr-jupyterlab-variableinspector A panel listing variables, types and shapes
Terminal window
python -m pip install jupyterlab-git jupyterlab-execute-time jupyter-resource-usage
jupyter labextension list

Classic Notebook extensions (jupyter_contrib_nbextensions, last released in 2022) only work with Notebook 6 and nbclassic, not with Notebook 7 or JupyterLab.

  • Voilà serves a notebook as a web app with the code hidden and widgets live: pip install voila, then voila regional_report.ipynb. Each visitor gets their own kernel, so plan capacity accordingly.
  • JupyterHub runs a multi-user Jupyter service for a team or a class, with authentication and a separate server per user, on a single machine or on Kubernetes (Zero to JupyterHub).
  • Remote servers. Start JupyterLab on the remote machine bound to localhost (jupyter lab --no-browser --port 8888) and forward the port over SSH (ssh -N -L 8888:localhost:8888 analyst@gpu01.example.test). Do not bind to 0.0.0.0 on a public network: anyone with the token can run code as your user.
  • VS Code opens .ipynb files natively with its Jupyter extension, including a variable viewer, debugging and the same kernels.

Never type API keys into a cell: the value ends up in the saved output, the execution history and often the Git history. Read them from environment variables, a .env file excluded from Git (python-dotenv), or the platform’s secret store, such as Colab Secrets.

import os
from dotenv import load_dotenv # pip install python-dotenv
load_dotenv() # reads .env if present
api_key = os.environ["ORDERSPORTAL_API_KEY"] # fails loudly if missing

For speed, the usual Python advice applies more strongly in notebooks because cells are re-run often: use vectorized pandas or NumPy operations instead of Python loops, read large CSV files in chunks (pd.read_csv(path, chunksize=100_000)) or as Parquet, profile with %prun or %timeit before optimizing, and cache expensive intermediate results to disk so a kernel restart does not mean an hour of recomputation.

1. How do you keep notebooks reviewable in Git? Strip outputs on commit with nbstripout, use nbdime for notebook-aware diffs, and for heavily reviewed work pair the notebook with a Jupytext .py file so reviewers read plain code. Ignore .ipynb_checkpoints/.

2. What does papermill add over running jupyter nbconvert --execute? Parameters: it injects values into a tagged cell, records them in the output notebook, and is callable from Python, which makes one notebook a reusable report for many inputs. nbconvert executes as-is and focuses on format conversion.

3. What is the minimum CI check for a repository of notebooks? Execute each notebook on a small fixture dataset, for example with pytest --nbval-lax or papermill, in a pinned environment. It catches broken imports, renamed columns and hidden-state notebooks that only worked out of order.

4. When should code move out of a notebook? When it is reused across notebooks, needs unit tests, or runs on a schedule. Move it into a module or package, import it into the notebook, and keep the notebook for narrative and results.

5. Senior scenario: 40 analysts share notebooks on laptops with different package versions. What platform do you propose? A JupyterHub (often on Kubernetes) with a curated, versioned image per team, shared storage for data, and per-user servers sized to the work; Git with nbstripout and Jupytext for collaboration; papermill jobs for scheduled reports; and Voilà or exported HTML for consumers who should not run code. Upgrades roll out by publishing a new image version that users opt into, so a pandas upgrade never silently changes last quarter’s numbers.