Jupyter Best Practices and Tools
Notebooks are excellent for exploring and explaining, and fragile as soon as they are shared, scheduled or committed. This page collects the habits and tools that keep them reliable: a clear structure, a pinned environment, clean diffs, parameterized headless runs, tests, and the extensions and servers that teams use around JupyterLab. It assumes the basics from Jupyter notebooks: kernels, magics and rich output.
The examples use a small regional revenue report for an invented OrdersPortal shop. Command-line samples were run with Python 3.13, papermill 2.7, nbconvert 7.17, nbval 0.11, nbstripout 0.9 and Jupytext 1.19 (November 2026); outputs are shown where they matter.
Topics on this page
Section titled “Topics on this page”| Topic | What it answers |
|---|---|
| Notebook structure | How to lay out a notebook that reads top to bottom |
| Reproducibility | Environments, seeds, paths and restart-and-run-all |
| Version control | Clean diffs with nbstripout, nbdime and Jupytext |
| Parameterized runs with papermill | Running one notebook many times with different inputs |
| Converting and executing | nbconvert to HTML, PDF, scripts and slides |
| Testing notebooks | Executing notebooks in CI and testing their functions |
| Generating notebooks | Building .ipynb files from code with nbformat |
| JupyterLab extensions | Useful extensions and what is already built in |
| Sharing and serving | Voilà dashboards, JupyterHub, remote servers and VS Code |
| Secrets and performance | Keeping keys out of notebooks and keeping them fast |
Notebook structure
Section titled “Notebook structure”Write a notebook like a short report that someone can read without running it:
- A Markdown title cell that states the question and the data source.
- Imports and configuration (constants, paths, parameters) in one place.
- Data loading.
- Exploration, with a Markdown cell before each step saying what you are checking.
- Processing or modeling.
- Results and conclusions in Markdown, written after the numbers are in.
Keep one logical step per cell. A cell that loads, cleans, trains and plots cannot be re-run in part, and its failure tells you little. When a cell grows past a screen, move the logic into a function in a .py module, import it, and keep the notebook as the narrative. That also makes the logic testable.
Pitfall: a notebook as the only copy of business logic. If two notebooks paste the same cleaning code, they will drift. Put shared code in a package (ordersportal_analytics/cleaning.py) and import it, with %autoreload 2 while you develop.
Reproducibility
Section titled “Reproducibility”A result you cannot reproduce is an anecdote. Four habits cover most problems:
- Pin the environment. Keep a
requirements.txtor a lock file (uv lock,pip-tools,conda env export) next to the notebook, and print key versions in the first cell so a saved output shows what produced it. - Fix random seeds for anything stochastic:
random.seed(42),numpy.random.default_rng(42),torch.manual_seed(42). GPU kernels can still be non-deterministic; note it when it matters. - Use relative paths from the project root (
Path("data/orders.csv")), neverC:\Users\...or/home/.... - Restart and run all (Kernel menu) before you commit, share or quote a number. It is the only check against hidden state.
# First cell of the notebook: record what produced these outputsimport sysimport numpy as npimport pandas as pd
print(f"Python {sys.version.split()[0]} | pandas {pd.__version__} | numpy {np.__version__}")rng = np.random.default_rng(42)Version control
Section titled “Version control”An .ipynb file stores outputs, execution counts and metadata alongside the code, so every run changes the file and diffs are full of base64 images. Three tools fix this:
- nbstripout removes outputs and execution counts.
nbstripout --installadds a Git filter to the repository so outputs are stripped when files are staged, while your working copy keeps them. In the test run, a notebook with seven outputs had zero afternbstripout notebook.ipynb. - nbdime gives notebook-aware diff and merge.
nbdiff a.ipynb b.ipynbshows cell-level changes,nbdiff-webshows them in the browser, andnbdime config-git --enablemakesgit diffuse it. - Jupytext pairs each notebook with a plain-text version (a
.pyfile in “percent” format or Markdown) that is easy to review.jupytext --set-formats ipynb,py:percent notebook.ipynbkeeps both in sync; you can commit only the.pyfile and regenerate the notebook.
python -m pip install nbstripout nbdime jupytextnbstripout --install # per repository: strip outputs on git addnbdime config-git --enable # notebook-aware git diff and mergejupytext --set-formats ipynb,py:percent regional_report.ipynbAlso add .ipynb_checkpoints/ to .gitignore.
Pitfall: stripping outputs from notebooks that are the deliverable. If a rendered report is the point (for example a notebook published as documentation), commit an executed HTML export or let CI execute it, rather than committing large outputs.
Parameterized runs with papermill
Section titled “Parameterized runs with papermill”papermill executes a notebook headless with injected parameters, which turns an exploratory notebook into a repeatable report. Tag one cell parameters (in JupyterLab: the property inspector, Common Tools > Cell Tags) holding the defaults. papermill inserts a new cell tagged injected-parameters after it with your values, runs everything, and saves the executed copy.
The report notebook used here has a parameters cell with region = "EU" and min_total = 0.0, then a cell that filters the orders and prints a summary:
papermill regional_report.ipynb out_us.ipynb -p region US -p min_total 50The executed copy out_us.ipynb prints:
US: 1 orders, 89.00and its cell tags are [], ['parameters'], ['injected-parameters'], [], so the run’s inputs are visible in the output file. From Python, papermill.execute_notebook("regional_report.ipynb", "out_us.ipynb", parameters={"region": "US"}) does the same, which makes it easy to loop over regions or months from a scheduler. If any cell raises, papermill stops and still writes the partial output notebook, so the failure is easy to inspect.
Converting and executing
Section titled “Converting and executing”nbconvert turns notebooks into other formats and can execute them first.
jupyter nbconvert --to html --execute regional_report.ipynb --output report_eu # run, then export HTMLjupyter nbconvert --to script regional_report.ipynb # .py file with the code cellsjupyter nbconvert --to webpdf regional_report.ipynb # PDF through a headless browser (needs nbconvert[webpdf])jupyter nbconvert --to pdf regional_report.ipynb # PDF through LaTeX (needs a TeX installation)jupyter nbconvert --to slides regional_report.ipynb --post serve # reveal.js slidesThe HTML export in the test run wrote report_eu.html (about 280 KB, most of it the embedded styles). Add --no-input to hide code cells in an export for non-technical readers.
Testing notebooks
Section titled “Testing notebooks”Two levels of testing are useful:
- The notebook runs. The cheapest test is executing it in CI.
pytest --nbval-lax notebook.ipynbruns every cell and fails if one raises an error; plain--nbvalalso compares outputs with the saved ones, which is only useful for deterministic notebooks. In the test run,pytest -q --nbval-lax regional_report.ipynbreported2 passed(one test item per code cell). - The logic is right. Test functions in the package the notebook imports with ordinary pytest. testbook can also call functions defined inside a notebook (
tb.ref("add")), but it has not had a release since 2021, so prefer moving logic into modules.
Pitfall: tests that need production data or credentials. Give the notebook a small fixture dataset through a papermill parameter (data_path) so CI can run it without secrets.
Generating notebooks
Section titled “Generating notebooks”nbformat reads and writes .ipynb files, and nbclient executes them, which is how the samples on these pages were produced:
import nbformatfrom nbformat.v4 import new_notebook, new_code_cell, new_markdown_cellfrom nbclient import NotebookClient
params = new_code_cell('region = "EU"\nmin_total = 0.0')params.metadata["tags"] = ["parameters"]
nb = new_notebook(cells=[ new_markdown_cell("# Regional revenue report"), params, new_code_cell( 'import pandas as pd\n' 'orders = pd.DataFrame({"region": ["EU", "US", "EU", "APAC"],\n' ' "total": [129.50, 89.00, 240.25, 56.75]})\n' 'subset = orders[(orders.region == region) & (orders.total >= min_total)]\n' 'print(f"{region}: {len(subset)} orders, {subset.total.sum():.2f}")'),])nb.metadata["kernelspec"] = {"name": "python3", "display_name": "Python 3", "language": "python"}NotebookClient(nb, timeout=60, kernel_name="python3").execute()print(nb.cells[2].outputs[0].text, end="")nbformat.write(nb, "regional_report.ipynb")EU: 2 orders, 369.75JupyterLab extensions
Section titled “JupyterLab extensions”JupyterLab 4 extensions are prebuilt and installed with pip into the environment that runs JupyterLab; restart JupyterLab afterwards and check them with jupyter labextension list. Several features that used to need extensions are now built in: the table of contents, the visual debugger (for kernels that support it, including ipykernel) and a property inspector for cell tags.
| Package | What it adds |
|---|---|
jupyterlab-git |
A Git panel: stage, commit, diff and push from the sidebar |
jupyterlab-execute-time |
Start time and duration under each executed cell |
jupyterlab-code-formatter |
Format cells with Black, Ruff or isort |
jupyterlab-spellchecker |
Spell checking in Markdown cells |
jupyter-resource-usage |
Memory and CPU use of the kernel in the status bar |
lckr-jupyterlab-variableinspector |
A panel listing variables, types and shapes |
python -m pip install jupyterlab-git jupyterlab-execute-time jupyter-resource-usagejupyter labextension listClassic Notebook extensions (jupyter_contrib_nbextensions, last released in 2022) only work with Notebook 6 and nbclassic, not with Notebook 7 or JupyterLab.
Sharing and serving
Section titled “Sharing and serving”- Voilà serves a notebook as a web app with the code hidden and widgets live:
pip install voila, thenvoila regional_report.ipynb. Each visitor gets their own kernel, so plan capacity accordingly. - JupyterHub runs a multi-user Jupyter service for a team or a class, with authentication and a separate server per user, on a single machine or on Kubernetes (Zero to JupyterHub).
- Remote servers. Start JupyterLab on the remote machine bound to localhost (
jupyter lab --no-browser --port 8888) and forward the port over SSH (ssh -N -L 8888:localhost:8888 analyst@gpu01.example.test). Do not bind to0.0.0.0on a public network: anyone with the token can run code as your user. - VS Code opens
.ipynbfiles natively with its Jupyter extension, including a variable viewer, debugging and the same kernels.
Secrets and performance
Section titled “Secrets and performance”Never type API keys into a cell: the value ends up in the saved output, the execution history and often the Git history. Read them from environment variables, a .env file excluded from Git (python-dotenv), or the platform’s secret store, such as Colab Secrets.
import osfrom dotenv import load_dotenv # pip install python-dotenv
load_dotenv() # reads .env if presentapi_key = os.environ["ORDERSPORTAL_API_KEY"] # fails loudly if missingFor speed, the usual Python advice applies more strongly in notebooks because cells are re-run often: use vectorized pandas or NumPy operations instead of Python loops, read large CSV files in chunks (pd.read_csv(path, chunksize=100_000)) or as Parquet, profile with %prun or %timeit before optimizing, and cache expensive intermediate results to disk so a kernel restart does not mean an hour of recomputation.
Interview questions
Section titled “Interview questions”1. How do you keep notebooks reviewable in Git?
Strip outputs on commit with nbstripout, use nbdime for notebook-aware diffs, and for heavily reviewed work pair the notebook with a Jupytext .py file so reviewers read plain code. Ignore .ipynb_checkpoints/.
2. What does papermill add over running jupyter nbconvert --execute?
Parameters: it injects values into a tagged cell, records them in the output notebook, and is callable from Python, which makes one notebook a reusable report for many inputs. nbconvert executes as-is and focuses on format conversion.
3. What is the minimum CI check for a repository of notebooks?
Execute each notebook on a small fixture dataset, for example with pytest --nbval-lax or papermill, in a pinned environment. It catches broken imports, renamed columns and hidden-state notebooks that only worked out of order.
4. When should code move out of a notebook? When it is reused across notebooks, needs unit tests, or runs on a schedule. Move it into a module or package, import it into the notebook, and keep the notebook for narrative and results.
5. Senior scenario: 40 analysts share notebooks on laptops with different package versions. What platform do you propose? A JupyterHub (often on Kubernetes) with a curated, versioned image per team, shared storage for data, and per-user servers sized to the work; Git with nbstripout and Jupytext for collaboration; papermill jobs for scheduled reports; and Voilà or exported HTML for consumers who should not run code. Upgrades roll out by publishing a new image version that users opt into, so a pandas upgrade never silently changes last quarter’s numbers.
Related pages
Section titled “Related pages”- Jupyter notebooks: kernels, magics and rich output — how notebooks work, the start of this section
- Google Colab — hosted notebooks with GPUs, Drive and secrets
- Python basic commands — venv, pip and test commands used here
- Python tools — linters, formatters and test runners