Real-World Regression

From one feature to many

In previous chapters, you predicted from one feature:

Now imagine three features:

• $x_1$: square footage
• $x_2$: number of bedrooms
• $x_3$: distance to downtown (km)

The model:

Each feature gets its own weight. Each weight tells you how much that feature contributes. If $w_1=200$, every additional square foot adds $200 to the predicted price. If $w_3=-50000$, every additional kilometer from downtown subtracts $50,000.

For n features, in general:

Vector notation

Writing all those weights is annoying. Vector notation packs it up.

Let x be a vector of features (an n-dimensional vector), w a vector of weights. Then:

The dot is a dot product: multiply corresponding components and sum.

So w · x + b is exactly what you wrote above, just shorter. You'll see this notation everywhere in ML papers. It scales the same way to 10 features, 100, 10,000.

The cost function generalizes too. With multiple features, MSE is still:

The only thing that changed: ${\hat{y}}_i$ is now $\mathbf{w}\cdot {\mathbf{x}}_i+b$, where ${\mathbf{x}}_i$ is the feature vector for example i.

Gradient descent with vectors

The update rule generalizes too. Now you have a gradient component for each weight:

Where $x_{ij}$ is "the j-th feature of the i-th example." Update each weight:

Same logic, more components. In code, this is just a numpy vector operation; no extra loop.

The scaling problem

Here's a subtle issue that bites every beginner.

In the housing example:

• Square footage ranges from about 500 to 5000
• Bedrooms range from 1 to 5
• Distance ranges from 0 to 30

These are wildly different ranges. Gradient descent's "step" affects every parameter by the same learning rate. If the learning rate is right for the bedroom weight, it's wrong for the square-footage weight. The cost surface becomes a long, narrow valley instead of a round bowl, and gradient descent zigzags painfully.

The fix: feature scaling. Standardize each feature before training.

Two common methods:

Min-max scaling. Squash to [0, 1].

Standardization (z-score). Mean 0, standard deviation 1.

where μ is the feature's mean and σ is its standard deviation.

Standardization is the more common choice in ML. After scaling, all features have similar ranges, the cost surface is a nice round bowl, and gradient descent rolls smoothly.

Multivariate linear regression on California housing

The full pipeline. Open Colab.

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt

# 1. Load the data.
data = fetch_california_housing(as_frame=True)
df = data.frame
print("Shape:", df.shape)        # (20640, 9)
print("Columns:", df.columns.tolist())

# Separate features (X) from target (y).
X = df.drop('MedHouseVal', axis=1).values
y = df['MedHouseVal'].values

# 2. Train/test split. 80/20.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 3. Scale features. Fit scaler on train ONLY, then transform both.
# This avoids "leaking" test data information into training.
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# 4. Train.
model = LinearRegression()
model.fit(X_train_scaled, y_train)

# 5. Evaluate.
y_pred_train = model.predict(X_train_scaled)
y_pred_test = model.predict(X_test_scaled)
print(f"Train MSE: {mean_squared_error(y_train, y_pred_train):.3f}")
print(f"Test MSE:  {mean_squared_error(y_test, y_pred_test):.3f}")

# 6. Inspect what the model learned.
feature_names = df.columns.tolist()[:-1]   # all except 'MedHouseVal'
for name, weight in zip(feature_names, model.coef_):
    print(f"  {name:12s}: weight = {weight:+.3f}")
print(f"  {'intercept':12s}: {model.intercept_:+.3f}")

What do these mean? The MSE on test is 0.556, which (since the target is in units of $100k, squared) means the typical prediction is off by about √0.556 × $100k ≈ $75k. Not amazing, but reasonable for such a simple model.

The largest absolute weights tell you what the model relies on most:

• MedInc (median income): big positive. Higher income areas → higher prices. Makes sense.
• Latitude and Longitude: big negative. The north and east have lower prices in California. The model has learned California geography.
• AveRooms: small negative. Surprising! More rooms per household, lower prices? This is probably because more rooms per household correlates with rural/lower-density areas, where prices are lower despite the size. This is the kind of weird thing a real model surfaces.

The scaling demo

To dramatize feature scaling, run the same model without scaling, using gradient descent.

# Without scaling: gradient descent struggles or diverges.
from sklearn.linear_model import SGDRegressor

# SGDRegressor uses gradient descent (with mini-batches).
# Without scaling, it'll struggle with this data.
print("Without scaling:")
model_unscaled = SGDRegressor(max_iter=1000, learning_rate='constant', eta0=0.001)
try:
    model_unscaled.fit(X_train, y_train)
    print(f"  Test MSE: {mean_squared_error(y_test, model_unscaled.predict(X_test)):.3f}")
except Exception as e:
    print(f"  Failed: {e}")

print("\nWith scaling:")
model_scaled = SGDRegressor(max_iter=1000, learning_rate='constant', eta0=0.001)
model_scaled.fit(X_train_scaled, y_train)
print(f"  Test MSE: {mean_squared_error(y_test, model_scaled.predict(X_test_scaled)):.3f}")

The unscaled version often produces astronomical MSE or NaN values. The scaled version converges to a reasonable result. Same algorithm, same data. The only difference is whether we standardized first. This is why feature scaling is the first thing every ML practitioner does on a new dataset.

More features aren't always better

Three things to watch for:

1. Useless features dilute signal. Adding "average outdoor temperature on the day the house was listed" gives the model another variable to figure out is unhelpful, occupying training capacity that could have gone to better features.

2. Highly correlated features confuse the model. Including "square feet" and "square meters" is the same information twice. The model becomes unstable: which weight should be big? Either one. Both at once becomes ambiguous. (Term: multicollinearity.)

3. Some features actively mislead. "Year built" trained on 1990 data: model thinks old houses are cheap. Apply to 2026 data, badly mispriced historic properties.

This is where feature engineering matters. Picking, transforming, and combining features is its own craft.

Vocabulary

Questions you might have