An XGBoost Property Valuation Postmortem: Leakage, Overfitting, and SHAP Surprises

I spent the last few months building a property valuation engine for Orlando. The goal was to beat a basic baseline (median price per square foot) using XGBoost. My v1 model looked good on paper until I looked under the hood. It had fatal flaws that the standard metrics could not surface.

This is a postmortem on how a high R-square (R²) fooled me, how Optuna forced me to rethink my hyperparameter space, and why hyper-local real estate data will eat you alive if you treat it like a Kaggle dataset.

The V1 Trainwreck and the Random Split Trap

To begin, I tried the standard way of building a prototype model: square footage, number of beds, number of baths, lot size, year built, ZIP, and a random 80/20 train-test split. The R² was 0.84, so I thought I had a model I could start wrapping into an API.

But I was wrong.

The random split caused time-series leakage. Real estate actually is not tabular at all; rather, it's a time-series problem in a tabular wrapper. Suppose there is one house on Street A that sold in month eight for $450,000 and the adjacent house sold in month 10 for $460,000. A random split happily lets the model see the second sale during training and predict the first. Except in production, the model has no access to future comps.

When I switched to a strict time-series split (train on 2023–2024, predict 2025), that 0.84 R² collapsed to a brutal 0.61. The v1 model wasn't learning value. It was memorizing neighborhood comps.

Why XGBoost over LightGBM or CatBoost

One of the reasons I chose XGBoost was because of the monotonic constraints that it offers.

Without restrictions, the model might pick up really goofy patterns. Suppose three huge houses are all grouped in the same cul-de-sac, and they all happened to be foreclosures and sell for really cheap, and thus the model may think a high square footage for the houses in that region decreases their price. This is signal, not noise.

With XGBoost, I can hard-code the rule: as square footage goes up, the predicted price cannot go down. It's an obvious rule that every real estate agent already operates by, but the model has no idea unless I tell it. A model that violates this looks dumb in front of stakeholders, no matter how clean its RMSE is.

Too Much Hand-Tuning: A Cautionary Tale

I spent two days hand-tuning max_depth and learning_rate based on gut feel — what usually works on similar problems. I leaned deep (max_depth of 7 or 8), assuming the model needed depth to capture interactions between property type and age.

Then, I set up an Optuna study with 100 trials. The best parameters came back unexpectedly tame: max_depth of four, with heavy L2 regularization.

What I had been doing was forcing the model to overfit. Orlando has so many micro-segments and tiny subdivisions that deep trees were just a clever way of letting XGBoost memorize specific HOA names and cul-de-sacs with two recorded sales. Shallow trees forced the model to generalize at the neighborhood level.

What Shap Revealed: The Orlando “Huh” Moments

When I pulled SHAP values for v2, a few local quirks made me re-check my code: The 2,500 square feet bathroom cliff. 

Once a house crossed roughly 2,500 square feet, having only two bathrooms became a strongly negative SHAP contributor. In the Orlando market, a big house with too few bathrooms reads as a layout problem, and the model picked that up.

The Disney Distance Curve: Families pay a premium to live near the theme parks — but only up to a point. The premium dies off past a certain radius, and at a certain radius, it applies a penalty to the noise and traffic. Some of the areas are: Windermere, Celebration, Lake Butler, etc

Target-Encoding Blowups on Small Neighborhoods: When I target-encoded high-cardinality neighborhood names, the model overfit on tiny gated communities. A luxury estate with only two recorded sales would have its "neighborhood mean" anchored on those two extreme prices, and predictions for that area went wild. I switched to m-estimate smoothing, which pulls neighborhoods with few sales back toward the county median until they accumulate enough data.

Walkability: The Feature I Was Sure Would Help, and Didn’t

Downtown Orlando, Winter Park, and sections of Winter Garden are truly walkable areas. I assumed a walkability score feature would be significant in these areas, while distance to amenities would be a critical feature in the car-centric suburbs.

I calculated the Haversine distance to the nearest OpenStreetMap point of interest (restaurants, schools, parks, public transportation) for each property and added these distances as features.

The Result: less than 1% reduction in error.

In an area like Lake Nona, which is literally designed for you to drive everywhere, walking distance to the nearest park doesn't affect what people pay. I was engineering a feature based on my intuition about how cities should work, not what the Orlando market actually rewards. Lesson: let the data tell you what matters, then engineer features — not the other way around.

Where V2 Still Got Scope

This model does not understand interior quality. It will provide two identical predictions for two 2000-square-foot houses, one with three beds and two baths, one acre lot, and the same zip, if one is still a 90s time warp with the original carpeting and single-paned windows, and the other just had new custom windows, kitchen, bath, and finishes in the past year; which represent a massive difference. The correction will either be NLP on the listing descriptions or an image embedding of the listing photos. This is the problem with v3.

Takeaways for v3

1. Never use random splits for real estate or any time-dependent target. The metrics you get back aren't real. Use a temporal split from day one.
2. Invest in regularization. With missing values and high-cardinality string features, your model will overfit hard unless you keep it on a short leash.
3. Read your SHAP plots before you trust the model. If a feature's contribution looks nonsensical, you almost certainly have leakage or an encoding problem upstream.