What predicts US residential real-estate returns?

~19 min read · generated 2026-05-23 · all data from free / public sources

1. Where might the model want to buy today?

Hover any zip for ZIP code, metro, ZHVI, predicted fwd_3y return, and decile rank. Drag to pan; scroll to zoom. Open the map full-screen ↗

Using the post-2012 LightGBM trained on 3-year forward returns, we scored every zip in the top-100 US metros at the most recent month with enough feature coverage to make a confident prediction. The table below is rolled up to one row per metro (CBSA), showing the metro's principal (largest-population) city, the best-scoring ZIP inside that metro, the mean predicted return across the metro's top-5 ZIPs, and how many of its ZIPs land in the top 10% of all predictions nationally. Hover over any feature for its description; click any column header to sort.

1a. ZIP-level detail (top 20)

The same model expanded back to per-ZIP rows for users who want to drill into specific neighborhoods rather than cities.

2. Personal-overlay shortlists (Phase 4)

Two parallel ZIP-level shortlists, both nationwide and both built on the latest snapshot (2023-12), one tuned for an owner-occupier and one for a cash-flow investor. Filter thresholds use the median across top-100-metro zips on the snapshot (after forward-filling slow-moving structural features like school student-teacher ratio and ACS demographics from the most recent non-null panel month — these features lag the price target by 1–2 years and don't change month-to-month). The underlying parquets keep all 30 rows; the display tables below dedupe by metro (CBSA) so you don't see the same city ten times.

2a. Owner-occupier (price-return focus)

Scored on the price-only fwd_3y model. Filters keep zips with positive expected price return AND low climate risk (wildfire + flood ≤ median), low recent disaster frequency (≤ 5 FEMA declarations in 10 years), strong schools (student-teacher ratio ≤ median), good health outcomes (premature death rate ≤ median, as a life-expectancy proxy), and majority owner-occupied (acs__owner_occupied_pct ≥ 0.5).

30 zips; 22 of 30 zips fall in Washington, DC. Dominant metros: Washington, DC (22), Toledo, OH (7), Madison, WI (1). Median ZHVI $722,904; median predicted fwd_3y price return +1.59%.

Owner-occupier filters skipped:

• crime__*_per_100k (NIBRS not in feature store — needs FBI CDE_API_KEY)

2b. Investor (total-return + rent-yield focus)

Scored on the total-return fwd_3y model (price appreciation + ZORI rent carry, trained in the Phase 2 total-return run). Filters keep zips in the top quartile of predicted total return AND with a gross rent yield ≥ 6%, a ZHVI in the cash-flow-friendly band ($80K–$400K), and ACS owner-occupancy share in [0.4, 0.7] so there's an active rental market without signalling deep distress.

30 zips; spread across 15 metros: Kansas City, MO (4), St. Louis, MO (4), Detroit, MI (3). Median ZHVI $89,010; median rent yield 13.74%; median predicted fwd_3y total return +27.90%.

2c. Overlap and threshold transparency

The two shortlists do not overlap on any ZIP. The owner-occupier profile up-weights low-risk, school-strong, low-disaster areas while the investor profile favors lower-priced, higher-yield areas; they end up pointing at different zips by construction.

Median thresholds used (over top-100-metro snapshot zips):

• wildfire: median = 52.592
• flood: median = 62.692
• school: median = 15.093
• premature_death: median = 7500.495

3. Macro context: is now a good time? And how does RE stack up against other hedges?

Everything above is about where to buy. The two questions below are about when and what else: does the model think the broad RE universe is cheap or expensive right now, and how has private RE actually fared against the other classic inflation hedges (stocks, gold, REITs, BTC) when you put them on the same axis?

3a. Is now a good time to buy?

For each month from 2014 through the latest in the panel, we score every top-100-metro ZIP with the headline post-2012 LightGBM and take the universe mean of the predicted fwd_3y annualized return. The model's prediction at month t can be compared to the realized universe-mean fwd_3y return three years later (dashed line) for all months where t+3y is observable.

The model's current (2026-04) universe-mean predicted fwd_3y annualized return is -1.35% (median of the per-ZIP distribution that month; 10,396 ZIPs scored, 80% prediction interval roughly [-2.61%, -0.14%]). The historical median across all scored months in the post-2012 panel is +5.95% with an interquartile range of [-1.35%, +6.35%]. The current prediction sits at the 25th percentile of history — the model is currently bearish vs the historical baseline (n = 160 months scored).

How wide is the model's uncertainty over time?

Three separate LightGBM models trained at the 10th, 50th, and 90th conditional quantiles produce an 80% prediction band around the median. When the band is narrow, the model is confident; when it widens, it is hedging.

3b. Real estate vs other inflation hedges

Real estate is rarely held in isolation in a real portfolio — the question "is now a good time to buy a house" is really "is real estate the best use of the marginal dollar right now". We put residential RE on the same chart as the S&P 500, broad equities (VTI), publicly-traded REITs (IYR / VNQ), gold, and Bitcoin, all in real terms (net of CPI). Residential RE is measured unlevered and net of nothing (rent, maintenance, property tax, transaction costs all ignored) — conservative for RE since the typical homeowner has both leverage and carrying costs.

Historical 3y / 5y annualized real returns (rolling, since 2000)

Trailing 3-year real returns ending the most recent month

Cumulative real return per asset, 2000-now (log scale)

What beat what. Across the full 2000-now history of monthly rolling windows, the median 3-year annualized real return ranking is dominated by Bitcoin (+65.6%, on a much shorter post-2014 history and with extreme tail risk) and then by gold and broad-equity / S&P 500 total return. REITs sit in the +4–6% range. The Zillow ZHVI top-100 mean — our private-RE benchmark, before leverage and before rent — comes in at +3.53% median 3y real, ranking #8/8. The trailing-3y leader as of the latest month is Bitcoin at +36.99% real.

4. Alternative ranking: yield-aware total return

Everything in Section 1 ranks zips by predicted price appreciation. An investor who cares about yield (rent flowing in, not just paper appreciation) wants a different ranking: a separate LightGBM trained on fwd_3y_total = price_return + ZORI rent yield carry. The leaderboard below shows the top 30 zips by that target. Caveat: ZORI covers only ~11% of zip-months, so this ranking's universe is much smaller (~2K zips vs ~10K for the price-only model).

The decision-different finding: only 11 of 50 zips appear in both this and the price-only top 50 — a 22% overlap. Price-only tilts toward sunbelt fringe (Tulsa, Kansas City, Memphis, New Orleans) at median rent yield ~12.5%. Total-return is dominated by rust-belt high-yield zips (Detroit ×7, Cleveland ×5, Dayton, Akron, Toledo, Birmingham) at median rent yield ~15.8%. Same model architecture, just yield-aware → very different shortlist.

5. Headline results

The table below picks the best-performing model for each horizon, reported on both temporal splits. The post-2012 block is the main result (larger, more recent test window); the pre-2008 block is a stress test through the 2008 housing crisis. A signal that shows up in both — same sign, same order-of-magnitude spread — is much harder to dismiss as overfitting to one regime.

Why this study exists

Residential real estate is the largest asset class most people will ever own, and the conventional wisdom for picking where and when to buy is dominated by qualitative heuristics (school districts, "up-and-coming neighborhoods", proximity to Whole Foods) or single-metric proxies (population growth, median income). This study asks: which of these signals actually predict forward returns, and by how much, separated by time horizon — and are there underweighted predictors that the textbook approach misses?

The framing is deliberately quantitative. We built a 3.73-million-row panel of every zip code in the top 100 US metros, monthly, 2000-2026, joined with 140+ candidate predictors organized into 11 thematic classes. We trained gradient-boosted models to predict 1-, 3-, and 5-year forward home-price returns on two backtest windows: pre-2008 (GFC stress test) and post-2012 (rate-shock + COVID).

The setup in one minute

What we predict

For each zip code at each month, we compute the forward home-price return (Zillow ZHVI index) over three horizons — 1 year, 3 years, 5 years — annualized for the latter two. These are the targets our models try to learn from upstream features.

The chart below illustrates what those forward windows look like for a single real ZIP. The diamond marks an example "prediction date"; the three shaded windows are the 1-, 3-, and 5-year forward periods we score the model against. Predict-only-on-what-you-knew- at-time-t, score-on-what-actually-happened-by-time-t+h is the discipline that makes the backtest honest.

How we predict it

For each (zip, month) row, we join a wide set of features that were knowable at that time (with explicit publication-lag adjustment per source — e.g., HMDA mortgage data isn't available until 9 months after the year it describes). We then train two families of models per horizon:

• Baselines: predict the global mean; predict the within-metro mean; OLS regression on Census demographics only ("hedonic"). These are what a real model has to beat.
• Real models: ElasticNet (regularized linear, interpretable) and LightGBM (gradient-boosted trees, captures interactions). Both report Spearman rank correlation and decile spread on out-of-sample test data; LightGBM additionally produces SHAP feature attributions.

How we validate

Two temporal splits, both reported in this run:

• pre_2008 split: train on data through 2007-12, test on 2008-2013. Stress-tests whether the model breaks across the 2008 housing crisis (the "GFC" — Global Financial Crisis).
• post_2012 split: train through 2017-12, test on 2018-2024. Stress-tests across the Fed's 2022 rate hike cycle (the "rate shock") and the COVID-era demand surge.

This run additionally reports spatial cross-validation (hold out entire CBSAs) and walk-forward CV (rolling 12-month test windows) on top of the two static splits — see Section 1 below.

6. Confidence: walk-forward and spatial robustness

A single train/test split can flatter or punish a model by accident. Two stronger CV regimes re-run the post-2012 LightGBM to bound how much of the headline number is structural vs. noise.

1a. Walk-forward CV (rolling-origin)

Train through year T−1, test on year T; roll the cutoff from 2018-12 through 2022-12. Five test years per horizon (fewer for fwd_5y, which would need observability beyond the panel's end). The schematic below shows the sliding window: each row is one fold, blue is the training range, green is the held-out test year.

For the headline fwd_3y horizon, mean Spearman ρ = +0.447 with std 0.233 across 5 folds — the +0.327 figure from the single static post-2012 split survives in expectation, but with meaningful year-to-year noise. 2020 is the variance driver, as expected.

1b. Spatial CV (whole-CBSA holdout)

5-fold CBSA holdout, with train+test both restricted to ≤ 2017-12 so the only stressor is the spatial axis (no temporal regime shift). Two variants: the raw model predicts the price return directly; the demeaned model predicts the within-metro residual — i.e., the model is given each ZIP's return after subtracting the average return of every other ZIP in the same metro that month. That isolates "can the model rank ZIPs inside a metro?" from the easier question "can the model tell expensive metros from cheap ones?".

The plot below shows the actual partition: every top-100 metro is assigned to exactly one of five folds (color). Bubble size = metro population.

Raw fwd_3y

Demeaned (within-metro residual) fwd_3y

The headline lesson. The raw model's Spearman ρ = +0.817 with decile spread +20.54% looks spectacular — but most of it is a level effect ("expensive zips stay expensive"). The demeaned model's Spearman ρ = +0.287 with decile spread +3.36% is the genuine cross-sectional alpha.

7. Did this actually work? — portfolio backtest

The most decision-relevant question. We replay history: at each month from 2018-01 through 2021-12, we run the model with only the data it could have seen at that time, take its top-decile picks (a "basket" of about 1,000 ZIPs), and look up what those ZIPs actually returned 3 years later. Compare to the universe mean (equal-weight all ZIPs) and to the bottom-decile basket. Alpha is the basket's excess annualized return over the universe mean.

The model adds skill. The price-only fwd_3y headline model's top decile beat the universe mean in 91.7% of months (48/48), earning roughly +2.23 percentage points per year alpha and a +3.95 pp/yr top-vs-bottom spread. The yield-aware fwd_3y_total model is in another class — 100.0% hit rate, +6.52 pp/yr alpha — but its scoring universe is much smaller (only ZIPs where Zillow publishes rent data).

2a. Per-cohort wealth: $1 in, $? out after 3 years

The same backtest expressed as terminal wealth. For each entry month, $1 invested in the top-decile basket, the universe, or the bottom-decile basket is compounded forward for the full 3-year hold.

2b. Calibration — is the model a good ranker, a good forecaster, or both?

A model can rank ZIPs correctly yet still be mis-calibrated — predicting +3% when realized averages +7%. We bin every test-set prediction into 20 quantile buckets and plot mean predicted vs mean realized in each bucket. Points on the dashed line are perfectly calibrated.

The cloud's slope is the rank quality (steeper = stronger ranking); the cloud's shift relative to the diagonal is the calibration error. A model can have a steep slope but sit far below the diagonal — useful for picking but biased in level. That pattern is exactly what we'd expect in a regime-shift test window like post-2012, and is why R² is not the headline metric.

8. Decile spread, every model × horizon × split

Each bar is one model on one horizon on one test window. Positive values mean the model's top-decile picks beat its bottom-decile picks; negative values mean it anti-ranked. The left panel is the pre-2008 split (smaller and noisier); the right is post-2012 (the main result).

3a. Realized return by predicted decile (headline horizon × split)

The chart above collapses each (model, horizon, split) into a single "top minus bottom" number. The one below zooms into the headline (post-2012 LightGBM, fwd_3y) and shows the realized return of every decile, not just the extremes. A monotone climb from decile 1 → decile 10 is the strong-form claim: the model is ordering ZIPs, not just separating the very best from the very worst.

LightGBM wins decisively on the medium and long horizons in the post-2012 split. On the 1-year horizon, the demographics-only hedonic baseline (a simple linear regression using only Census ACS variables) actually edges LightGBM — a useful sanity check that when a simple baseline ties a sophisticated model, the signal probably comes from the data more than the algorithm.

9. What did the model learn? — SHAP feature importance

For each horizon, we compute mean absolute SHAP value per feature on the training sample. SHAP decomposes each prediction into per-feature contributions. The bars below show the 15 features that the LightGBM model used most heavily.

Short horizon (1 year)

Medium horizon (3 years)

Long horizon (5 years)

4a. Per-ZIP attribution — why this ZIP, not that one?

The aggregate SHAP charts above answer "which features does the model rely on, on average?" The two waterfalls below answer the more decision-relevant version: for this specific ZIP, which features pushed its predicted return up or down, and by how much? Green bars push the prediction higher than the model's base rate; red bars push it lower.

Top-decile example

Bottom-decile example

The pattern that recurs across nearly every top-decile ZIP: a large positive contribution from the drawdown feature combined with a uniform negative contribution from the 10Y real rate. Bottom-decile ZIPs typically show the opposite.

Four findings worth flagging:

• Drawdown is the #1 short-horizon predictor. The derived "ZHVI drawdown vs trailing 60-month peak" feature carries the largest mean-abs SHAP at fwd_1y. Zips that have fallen most relative to their recent peak tend to snap back hardest — a clean cross-sectional mean-reversion signal.
• Real rates dominate medium/long by a wide margin. At fwd_3y, 10Y TIPS yield's mean-abs SHAP is more than 4× the next-largest feature; at fwd_5y, more than 3×. Medium and long-horizon ranking is mostly about macro conditions, not zip-specific properties.
• New political + behavioral features are earning their keep. County Republican vote share (MIT Election Lab) appears in the top 10 for every horizon — and at fwd_5y, both party vote shares appear, so the model is reading the political-identity gradient, not taking sides. Wikipedia metro-article pageviews ranks #3 at fwd_3y.
• Climate has receded to a short-horizon signal. FEMA disaster counts and NRI wildfire risk drop out of the top 10 for fwd_3y and fwd_5y under the expanded feature pool, while staying in fwd_1y — most likely capturing short-term storm-driven price dynamics.

10. Does the signal survive price stratification?

A common worry with any model that ranks zip codes is that it has secretly learned a price-level proxy: cheap zips mean-revert upward, expensive zips compound more slowly, and the "alpha" is just a roundabout way of buying low. To check that, we split the post-2012 test set into three ZHVI tiers and re-compute the LightGBM fwd_3y decile spread within each tier.

Interpretation. The decile spread is +6.35% for sub-$200K homes, +6.57% in the $200K–$500K middle, and +3.34% for $500K+ homes — all positive, all out-of-sample. The two cheaper tiers carry essentially the same spread, and the $500K+ tier roughly halves but stays positive. This is the pattern you'd expect if the model has real cross-sectional signal at every price point and luxury markets simply have flatter forward returns — not the pattern you'd expect if the model were secretly a price-mean-reversion proxy.

Caveats and what's not in this run

• HMDA mortgage records are in the feature store but didn't reach the trained model. Tract-year origination counts + institutional-share are loaded at ~12% panel coverage, but the data only spans 2020 / 2022 / 2023. The post-2012 split's train window ends at 2017-12 — entirely pre-HMDA — so prep's coverage filter drops every HMDA column from X.
• Four data sources still pending. FCC broadband, NOAA VIIRS satellite nightlights, Foursquare POIs, and FBI NIBRS crime aggregates (needs CDE_API_KEY) are wired into the assembler but blocked on credentials.
• Owner-occupier overlay is geographically narrow. 22 of 30 zips fall in the Washington, DC metro because the filter chain collapses to wealthy suburbs of one dominant metro.
• Total-return modeling now done but on a smaller universe. ZORI rent index covers only ~11% of zip-months, so fwd_3y_total trains on ~215K rows vs ~1.98M for price-only.
• ElasticNet regressed under the new QuantileTransformer scaling. The Wave-1 fix solved an RMSE blowup but flipped fwd_3y Spearman to negative. LightGBM is unaffected and remains the headline model.
• Pre-2008 coverage is thin. Realtor inventory (2016+), QCEW (2010+), ACS (2013+), BPS (2011+), IRS-SOI (2011+) all start post-GFC. The pre_2008 results in the headline table train on 27 features vs 41 post-2012.
• fwd_5y walk-forward CV has 3 folds, not 5 — the 2021-12 and 2022-12 cutoffs would need fwd_5y observability through 2026-12 / 2027-12, beyond the panel's end.

arbok · model-first, personal-overlay layered on top · all data free / public sources · code at src/arbok/