Python Packages for AI and Machine Learning
What you’ll learn
Section titled “What you’ll learn”- The major Python AI/ML package categories and what problem each solves.
- When to use NumPy, pandas, scikit-learn, PyTorch, and hosted model SDKs.
- Minimal installation sets for common goals: data analysis, classical ML, LLM apps, and RAG pipelines.
- How to choose packages without over-engineering the stack.
Summary
Section titled “Summary”Python dominates AI and ML workflows largely because of its package ecosystem. The ecosystem spans numerical computing, data manipulation, classical machine learning, deep learning, hosted model APIs, and orchestration frameworks. Knowing which layer you need prevents the common mistake of pulling in a heavy orchestration framework when a direct SDK call is sufficient.
The package landscape by layer
Section titled “The package landscape by layer”| Layer | Purpose | Key packages |
|---|---|---|
| Numerical computing | Arrays, vectorized math, linear algebra | NumPy, SciPy |
| Data analysis | Tables, joins, cleaning, aggregation | pandas, Polars |
| Classical ML | Structured data models, evaluation, pipelines | scikit-learn, XGBoost, LightGBM |
| Deep learning | Neural networks, GPU training, inference | PyTorch, TensorFlow/Keras |
| Hosted model SDKs | Calling LLMs via API | openai, google-genai, anthropic |
| Orchestration | Chains, agents, multi-step pipelines | LangChain, LlamaIndex |
| Vector storage | Embedding search and RAG | Chroma, FAISS, pgvector, Weaviate |
| Evaluation | Testing and benchmarking LLM outputs | DeepEval, Ragas, Promptfoo |
NumPy: numerical foundation
Section titled “NumPy: numerical foundation”NumPy provides the ndarray — a fast, typed, multi-dimensional array — and the functions to operate on it. Almost every other scientific Python package (pandas, scikit-learn, PyTorch) uses NumPy arrays internally.
import numpy as np
# Create arraysa = np.array([1.0, 2.0, 3.0])b = np.zeros((3, 3)) # 3x3 matrix of zerosc = np.linspace(0, 1, 100) # 100 evenly spaced values
# Vectorized operations — no loop neededprices = np.array([10.0, 20.0, 30.0])discounted = prices * 0.9 # [9.0, 18.0, 27.0]print(discounted.mean()) # 18.0print(discounted.sum()) # 54.0
# Matrix multiplicationA = np.random.randn(3, 4)B = np.random.randn(4, 2)C = A @ B # shape (3, 2)Install: python -m pip install numpy
pandas: data analysis
Section titled “pandas: data analysis”pandas provides the DataFrame — a labeled, tabular data structure. Use it for loading CSVs, cleaning records, filtering, grouping, and preparing features for ML models.
import pandas as pd
# Load datadf = pd.read_csv("orders.csv")
# Exploreprint(df.shape) # (rows, columns)print(df.dtypes) # column typesprint(df.describe()) # summary statistics
# Filterlarge_orders = df[df["amount"] > 100.0]
# Group and aggregatemonthly = df.groupby("month")["amount"].agg(["sum", "count", "mean"])
# Cleandf["amount"] = pd.to_numeric(df["amount"], errors="coerce")df = df.dropna(subset=["amount"])
# Exportdf.to_csv("orders_clean.csv", index=False)Install: python -m pip install pandas
scikit-learn: classical machine learning
Section titled “scikit-learn: classical machine learning”scikit-learn covers the full classical ML workflow: preprocessing, feature engineering, model training, evaluation, and pipelines. Use it for structured/tabular data classification, regression, and clustering.
from sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScalerfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import classification_reportimport pandas as pd
df = pd.read_csv("orders.csv")X = df[["amount", "items_count", "days_since_last_order"]]y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()X_train_scaled = scaler.fit_transform(X_train)X_test_scaled = scaler.transform(X_test)
model = RandomForestClassifier(n_estimators=100, random_state=42)model.fit(X_train_scaled, y_train)
predictions = model.predict(X_test_scaled)print(classification_report(y_test, predictions))Install: python -m pip install scikit-learn
PyTorch: deep learning
Section titled “PyTorch: deep learning”PyTorch is the leading framework for neural network research and production training. Use it when you need to train custom models, work with embeddings, or run inference on local GPU hardware.
import torchimport torch.nn as nn
# Check GPU availabilitydevice = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Simple linear modelmodel = nn.Sequential( nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1), nn.Sigmoid(),).to(device)
# Forward passx = torch.randn(32, 10).to(device) # batch of 32 examplesoutput = model(x)print(output.shape) # torch.Size([32, 1])Install: python -m pip install torch (CPU-only). For GPU builds, use the official installation command which picks the right CUDA version.
Hosted model SDKs
Section titled “Hosted model SDKs”For applications that call a hosted LLM rather than training their own models, use the provider’s Python SDK directly.
# OpenAIfrom openai import OpenAIclient = OpenAI()resp = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Summarize this order: ORD-001, BensonCo, $149.99"}],)print(resp.choices[0].message.content)
# Anthropic Claudeimport anthropicclient = anthropic.Anthropic()msg = client.messages.create( model="claude-sonnet-5-5", max_tokens=200, messages=[{"role": "user", "content": "What is RAG?"}],)print(msg.content[0].text)
# Google Geminifrom google import genaiclient = genai.Client()resp = client.models.generate_content(model="gemini-2.5-flash", contents="What is RAG?")print(resp.text)Install: python -m pip install openai anthropic google-genai
LangChain: orchestration
Section titled “LangChain: orchestration”LangChain is an orchestration framework for multi-step LLM workflows: retrieval pipelines, agent loops, tool use, and prompt management. Use it when a single API call is not enough — but do not introduce it prematurely.
from langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom langchain_community.vectorstores import FAISSfrom langchain.chains import RetrievalQA
# Build a simple RAG pipelineembeddings = OpenAIEmbeddings()vectorstore = FAISS.load_local("./order_docs", embeddings)retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
llm = ChatOpenAI(model="gpt-4o")chain = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
result = chain.invoke({"query": "What is the return policy for electronics?"})print(result["result"])Install: python -m pip install langchain langchain-openai langchain-community faiss-cpu
Installation sets by goal
Section titled “Installation sets by goal”Data analysis:
python -m pip install numpy pandas matplotlib seabornClassical machine learning:
python -m pip install numpy pandas scikit-learn xgboostLLM application:
python -m pip install openai anthropic google-genai python-dotenvRAG pipeline:
python -m pip install langchain langchain-openai chromadb tiktokenDeep learning:
python -m pip install torch # see pytorch.org for GPU buildsHow to choose
Section titled “How to choose”- Start with the smallest stack. A direct SDK call (
openai.chat.completions.create) is simpler to debug, test, and maintain than a LangChain chain that wraps the same call. - Add orchestration when you have state, tools, or multi-step logic. If your workflow needs tool calling, loop retries, or memory across turns, LangChain or LlamaIndex adds value.
- Check Python version compatibility before standardizing on a package. Some older ML packages lag behind the latest Python releases.
- Pin versions in
requirements.txt. The ML ecosystem moves fast — an unpinned install can break between CI runs.
Related pages
Section titled “Related pages”- Connect to OpenAI from Python — SDK setup and first request
- Connect to Gemini from Python — Google GenAI SDK
- Python standard library — built-in modules before reaching for packages
- Python tools overview — pip, uv, poetry
Official references
Section titled “Official references”- NumPy documentation — array computing reference
- pandas documentation — DataFrame reference
- scikit-learn documentation — ML algorithms and pipelines
- PyTorch documentation — neural network framework
- LangChain documentation — orchestration framework