Skip to content

Jupyter Notebooks: Kernels, Magics and Rich Output

Jupyter is an open-source environment for interactive computing. A notebook is a document that mixes live code, its output (tables, charts, HTML), equations and narrative text, which makes it the standard tool for data exploration, machine-learning experiments, teaching and reproducible research. The name refers to three languages it supported early on: Julia, Python and R. Today more than a hundred languages have kernels.

This page is the start of the Jupyter section. It explains how notebooks work and covers the day-to-day features: kernels, magic commands, shell commands, rich output and widgets. Jupyter best practices and tools continues with reproducibility, version control, automation and extensions, and Google Colab covers the hosted notebook service.

Versions at the time of writing (November 2026): JupyterLab 4.6, Notebook 7.6, IPython 9.18, ipykernel 7.4 and ipywidgets 8.1. The samples use one invented dataset, orders from an OrdersPortal shop, and were executed with Python 3.13 through nbclient; the outputs shown are from that run.

Topic What it answers
How notebooks work Server, kernel, .ipynb files and cell types
Installing JupyterLab and Notebook Which interface to install and how to start it
Keyboard shortcuts Command mode and edit mode
Kernels Managing, adding and choosing kernels
Hidden state Why out-of-order execution causes bugs
Magic commands % and %% helpers for timing, files and the environment
Shell commands Running ! commands and passing Python values
Rich output HTML, Markdown, images, math and DataFrames
Interactive widgets Sliders, dropdowns and callbacks with ipywidgets
Plotting Matplotlib inline and interactive plotting libraries

A notebook file (.ipynb) is JSON: a list of cells with their source, their saved outputs and metadata such as the kernel name. When you open it, a Jupyter Server process serves the web interface and starts a kernel, a separate process that executes the code. The browser sends each cell to the kernel over a WebSocket and renders whatever comes back.

Browser (JupyterLab or Notebook) ⇄ Jupyter Server ⇄ Kernel (for example ipykernel running Python)

Three cell types exist: Code cells run in the kernel; Markdown cells hold formatted text, headings, lists and LaTeX math; Raw cells are passed through unchanged, mainly for conversion tools. Outputs are saved inside the file, which is why notebooks are convenient to share and awkward in version control.

There are two front ends, built on the same components since Notebook 7:

JupyterLab 4 Notebook 7
Layout IDE-style: tabs, split panes, file browser, terminals, consoles One document per browser tab, simple and focused
Extensions JupyterLab extensions (prebuilt, installed with pip) The same JupyterLab extensions
Start command jupyter lab jupyter notebook
Terminal window
python -m venv .venv && source .venv/bin/activate # or .venv\Scripts\Activate.ps1 on Windows
python -m pip install jupyterlab notebook pandas
jupyter lab # opens http://localhost:8888/lab

Install Jupyter into each project’s virtual environment (or register the environment as a kernel, below) so notebooks import the same package versions as the project.

Pitfalls:

  • Old extensions after upgrading to Notebook 7. Extensions written for Notebook 6 and earlier (nbextensions) do not run in Notebook 7. Look for a JupyterLab equivalent, or use the nbclassic package if you must keep the old interface for a while.
  • pip install into the wrong interpreter. Inside a notebook, use %pip install pandas rather than !pip install; the magic installs into the environment of the running kernel.

Notebooks have two modes. Edit mode (Enter, or click into a cell) types into the cell. Command mode (Esc) acts on whole cells.

Shortcut Action
Shift+Enter Run the cell and select the next one
Ctrl+Enter Run the cell and stay on it
Alt+Enter Run the cell and insert a new cell below
Esc / Enter Switch to command mode / edit mode
A / B (command mode) Insert a cell above / below
D, D (command mode) Delete the cell
M / Y (command mode) Change the cell to Markdown / Code
0, 0 (command mode) Restart the kernel
Ctrl+Shift+C (JupyterLab) Open the command palette

A kernel is the process that holds your variables. Each open notebook is connected to one kernel. From the Kernel menu you can Interrupt a long-running cell (like Ctrl+C), Restart to clear all state, or Restart Kernel and Run All Cells to prove the notebook runs top to bottom.

The most useful kernel operation is registering a project’s virtual environment so it appears in the kernel picker:

Terminal window
# inside the activated project environment
python -m pip install ipykernel
python -m ipykernel install --user --name ordersportal --display-name "Python (OrdersPortal)"
jupyter kernelspec list # shows installed kernels and their folders

Other languages install their own kernels: IRkernel for R, IJulia for Julia, and .NET Interactive for C# and F# (dotnet tool install -g Microsoft.dotnet-interactive, then dotnet interactive jupyter install).

Cells can run in any order, and the kernel keeps whatever the last executed cell assigned. The execution counter ([5]) shows the order cells actually ran in, not their position.

# Cell 1
discount = 0.10
# Cell 2
discount = 0.25
# Cell 3
print(f"Discount applied: {discount:.0%}")
# Run 1, 2, 3, then edit and re-run only cell 1 and cell 3:
# Cell 3 now prints 10%, but anyone running the saved notebook top to bottom gets 25%.

This is the most common source of notebooks that “worked yesterday”. Restart and run all before you share, commit or trust a result.

Magics are IPython commands, not Python. Line magics start with % and apply to one line; cell magics start with %%, must be the first line of a cell and apply to the whole cell. %lsmagic lists them all and %timeit? shows help for one.

The first cell builds the sample data used for the rest of the page:

import pandas as pd
orders = pd.DataFrame({
"order_id": ["ORD-1040", "ORD-1041", "ORD-1042", "ORD-1043"],
"region": ["EU", "US", "EU", "APAC"],
"total": [129.50, 89.00, 240.25, 56.75],
})
orders.groupby("region")["total"].sum()
region
APAC 56.75
EU 369.75
US 89.00
Name: total, dtype: float64
%timeit -n 1000 -r 5 orders["total"].sum()
17.5 μs ± 860 ns per loop (mean ± std. dev. of 5 runs, 1,000 loops each)

Your timings will differ; the point is the format: mean and spread over several runs.

%%capture captured
print("loading 4 orders")
print(captured.stdout.strip())
loading 4 orders

Frequently used magics:

Magic Purpose
%time, %%time Time one run of a statement or a cell
%timeit, %%timeit Benchmark over many runs
%pip, %conda Install packages into the kernel’s environment
%run script.py Run a Python file in the notebook’s namespace
%load file.py Paste a file’s contents into the cell
%%writefile module.py Write the cell to a file
%who, %whos, %reset List or clear variables (%who DataFrame lists only DataFrames)
%env NAME=value Show or set environment variables
%matplotlib inline / widget Static or interactive Matplotlib output (widget needs ipympl)
%%capture name Capture stdout, stderr and display output into a variable
%prun f() Profile a call with cProfile
%load_ext autoreload then %autoreload 2 Reload edited modules before each cell runs

%autoreload 2 is essential when you develop a package in an editor and exercise it from a notebook: edits to ordersportal/reports.py are picked up without restarting the kernel. It cannot update every object safely (for example instances created before a class changed), so restart when behavior looks inconsistent.

A line starting with ! runs in the system shell. The output can be captured into a Python list, and Python variables are substituted with braces.

files = !ls *.py
files
['build.py']
region = "EU"
!echo "Exporting orders for {region}"
Exporting orders for EU

Each ! line runs in a new shell, so !cd data has no lasting effect; use the %cd magic to change the notebook’s working directory. Shell commands also make a notebook platform-specific (ls is not available in Windows cmd), so prefer Python’s pathlib in notebooks other people will run.

The last expression of a cell is displayed with its richest representation: a DataFrame becomes an HTML table, a Matplotlib figure an image. To display several things from one cell, or something other than the last value, call display() with an object from IPython.display.

from IPython.display import display, Markdown
eu = orders.loc[orders.region == "EU", "total"].sum()
display(Markdown(f"**EU revenue:** {eu:,.2f} EUR"))

Rendered output: EU revenue: 369.75 EUR

Other display classes include HTML, Image(filename=...), JSON (a collapsible tree in JupyterLab), Math(r"\sum_i x_i") and Latex. DataFrames can be styled with orders.style.highlight_max(subset=["total"]).format({"total": "{:.2f}"}), and pd.set_option("display.max_columns", None) shows every column instead of an ellipsis.

Pitfall: huge outputs. Printing a 100,000-row DataFrame stores it in the .ipynb file, slows the browser and bloats commits. Show head() or a summary, and clear outputs before committing (see best practices).

ipywidgets adds sliders, dropdowns, buttons and other controls whose values are synchronized between the browser and the kernel. JupyterLab 4 and Notebook 7 support ipywidgets 8 after pip install ipywidgets; no extra enable step is needed.

import ipywidgets as widgets
from IPython.display import display
region = widgets.Dropdown(options=sorted(orders.region.unique()), value="EU", description="Region")
threshold = widgets.FloatSlider(value=50, min=0, max=250, step=10, description="Min total")
out = widgets.Output()
def refresh(change=None):
subset = orders[(orders.region == region.value) & (orders.total >= threshold.value)]
out.clear_output()
with out:
print(f"{region.value}: {len(subset)} orders, {subset.total.sum():.2f}")
region.observe(refresh, names="value")
threshold.observe(refresh, names="value")
refresh()
display(widgets.VBox([region, threshold, out]))

Moving the slider to 200 shows EU: 1 orders, 240.25; switching the region to US then shows US: 0 orders, 0.00. (The callback logic was run headless with ipywidgets 8.1.9 by setting the widget values from code.)

For quick experiments, @widgets.interact(min_total=(0, 250, 10)) above a function builds the controls from its arguments. Layout containers (HBox, VBox, Tab, Accordion) group controls, and tqdm.notebook.tqdm gives a progress bar for loops. Widgets need a live kernel: in a static HTML export or on GitHub they appear as a placeholder or not at all.

Matplotlib output appears inline by default in JupyterLab. For zooming and panning, install ipympl and use %matplotlib widget. Plotly, Altair and Bokeh produce interactive charts that render in JupyterLab without extra setup and stay interactive in HTML exports.

# fragment: needs matplotlib
import matplotlib.pyplot as plt
ax = orders.groupby("region")["total"].sum().plot(kind="bar", title="Revenue by region")
ax.set_ylabel("EUR")
plt.show()

1. What is the difference between the Jupyter Server and a kernel? The server serves the web interface, manages files and starts kernels; the kernel is a separate process that executes code and holds state. One server can run many kernels, in different languages, one per open notebook.

2. Why does a notebook work for the author and fail for a colleague? Usually hidden state: cells were run out of order, or a variable was defined in a cell that was later deleted. Other causes are packages installed into a different environment than the kernel’s, or absolute file paths. Restart and run all before sharing.

3. When would you use %pip install instead of !pip install? Always inside a notebook. %pip installs into the environment of the running kernel; !pip runs whichever pip is first on the shell’s PATH, which may belong to another interpreter.

4. How do you make a project’s virtual environment available in JupyterLab? Install ipykernel into that environment and run python -m ipykernel install --user --name <name>. The kernel then appears in the kernel picker of any Jupyter installation for that user.

5. Senior scenario: a data team’s notebooks have become the production reporting pipeline and break often. What do you change? Move reusable logic into a tested Python package imported by the notebooks (with %autoreload during development), parameterize the notebooks and run them headless with papermill in a scheduler, pin the environment, strip outputs in version control, and add a CI job that executes each notebook. Anything that must be reliable and not exploratory should eventually become a script or a job, with the notebook kept as the readable report.