Abstract
Land use (i.e., buildings) and transportation infrastructure are tightly coupled systems, such that residential property prices play a critical role in transportation planning. In a similar manner, transportation infrastructure influences property prices, such that accurate forecasts of both systems are related research problems. The objective of this study is to examine these interactions and evaluate the effectiveness of machine learning methods in modeling residential real estate prices across different urban contexts. Specifically, we first examine the impact of land and transportation infrastructure on residential real estate prices using Extreme Gradient Boosting (XGBoost) and Random Forest (RF) machine learning methods, and compare results between two cities representing diverse geographic and socioeconomic contexts. Second, we investigate the application of machine learning methods on spatial data and provide a comparison of non-spatial and spatial cross-validation on the performance of machine learning methods. We use SHapley Additive exPlanations (SHAP) values to study the impact of land use and transportation infrastructure on real estate prices. The models are applied, and results are compared between the Rawalpindi and Islamabad Metropolitan Area in Pakistan and the City of Toronto in Canada. We find that, despite differences in demographics and economic development, the two cities exhibit similarities in the effect of transportation infrastructure and local amenities on dwelling prices. Proximity to the major central city cores (i.e., downtowns) increases sale price. The effect of transportation infrastructure is differentiated, with high quality transit (e.g., subway and BRT) increasing and conventional bus stop proximity decreasing sale price, respectively. We confirm the previous finding in other fields that non-spatial cross-validation over-estimates the prediction accuracy of machine learning algorithms on spatially referenced datasets. We find that the XGBoost model has slightly higher performance than the RF model. We recommend careful use of machine learning methods in the case of spatial data specifically in modeling of land prices.
Introduction
The interaction of land use and transportation systems has been studied by many researchers over the past few decades, resulting in the development of computational integrated land use and transportation simulation models ( 1 ). Residential property price is an important component of the integrated modeling paradigm. Price plays an important role in residential (re)location choice of households. Residential location has a profound impact on travel behavior, specifically the number and length of trips. Many studies have confirmed the impacts of the transportation system on residential property prices ( 2 , 3 ). We aim to address two research gaps in the existing literature with this study. First, the interaction of property price with the transportation system is mostly studied in the context of developed cities, while these intricacies are less studied for and compared with developing cities. We contrast results for a developed city (Toronto, Canada) with those found for developing cities (Rawalpindi and Islamabad, Pakistan). Second, most machine learning applications to property price modeling do not adequately treat the spatial nature of property price data.
Machine learning methods are used across quantitative research domains for their data-driven basis and flexibility. This flexibility provides strong predictive accuracy. However, there remain many open questions for machine learning applications in regional planning. One set of questions stems from the spatial nature of most planning problems. A core component of many machine learning algorithms and validation methods is k-fold cross-validation. Cross-validation is used to test the transferability of a model trained on one dataset to predict the same outcome variable for an independent sample drawn from the same population. Standard k-fold cross-validation partitions randomly drawn observations into k equally sized samples. The model is trained on k−1 samples and tested on the hold-out sample until all samples are used in both training and testing. In the case of spatial data, the independence assumption underlying k-folds cross-validation is violated because of spatial autocorrelation among the observations. Several studies in ecology and other fields find that ignoring spatial autocorrelation leads to over-prediction bias (4–6). Spatial k-folds addresses this bias by sampling data based on spatial clustering. Machine learning methods also have the advantage of representing non-linear relationships, which are harder to capture using classical regression techniques.
The objectives of this study are: 1) application of machine learning to model residential property prices, with a focus on investigating the impact of land use and transportation infrastructure on prices; 2) compare the impact of non-spatial and spatial cross-validation methods on model performance to guide accurate use of the model and interpretation of model results; 3) compare two machine learning methods Extreme Gradient Boosting (XGBoost) and Random Forest (RF); 4) explore non-linear relationships between property price and its predictors using SHapley Additive exPlanations (SHAP) values; and 5) compare the factors affecting property price between a developed city and a developing city.
In this study, we train XGBoost and RF models on residential price data for two metropolitan regions representing diverse conditions - Rawalpindi and Islamabad Metropolitan Area (RIMA) in Pakistan and the City of Toronto in Canada. A 5-fold cross-validation method is used, wherein spatial folds are created using k-means clustering. SHAP plots are used to identify and explain the most important price determinants, along with their non-linear relationships, and make comparisons between metropolitan areas. Machine learning methods can overcome some of the challenges in property price modeling when investigating the impact of transportation infrastructure. Some studies have found non-linear relationships between property prices and transportation infrastructure. For instance, Mohammad et al. found that the impact of distance to rail stations on property prices shows a non-linear behavior ( 7 ). Therefore, machine learning models are promising for investigating the impact of transportation infrastructure on property prices.
Study Area
The two metropolitan areas chosen for this study differ significantly in their real estate and transportation features. The Greater Toronto Area (GTA) is consistently among the 10 least affordable real estate markets in the world, while RIMA does not appear in the rankings because of its lower prices ( 8 ). In this study, we focus on the City of Toronto proper because of real estate data availability, excluding its surrounding suburbs from analysis. However, we provide some statistics for the larger GTA as a metropolitan area comparison to RIMA. Figure 1 shows the Rawalpindi District, Rawalpindi City, and the City of Islamabad, including major transportation infrastructure. In this study, our analysis focuses on the RIMA region, which comprises the cities of Rawalpindi and Islamabad. Figure 2 shows the City of Toronto and its surrounding area (i.e., GTA), including major transportation infrastructure. In 2016, the average dwelling in Toronto sold for 730,000 Canadian dollars (CAD) (351 U.S. Dollars [USD] per sqft), while a dwelling in RIMA cost about 273 USD per sqft (adjusting for purchasing power parity) ( 9 – 11 ). Table 1 provides additional comparison statistics. GTA has roughly twice the population of RIMA but at half the population density. With respect to transportation infrastructure, Toronto is a developed city with an extensive multi-modal regional transit system that includes subway, streetcar, bus, and commuter rail connections to GTA and beyond. RIMA introduced a bus rapid transit (BRT) network in 2015, which has since expanded to 84 km of service ( 12 ). RIMA has plans for an integrated transportation system, including adding more feeder routes in phase 1 of the project and anticipated enhanced transit service in phase 2, which make this study timely to study the impacts of such services on property prices ( 13 ).
Rawalpindi Islamabad Metropolitan Area (RIMA).
Greater Toronto Area (GTA).
*All dollar amounts are converted to 2016 international dollars.
The following section summarizes the literature relating to two relevant topics: machine learning applications to real estate price prediction, and cross-validation methods applied to spatial datasets. The data used in the empirical study are then provided for each study area. We describe the real estate price model and contrast non-spatial and spatial results. Real estate price model results are provided for each study area. Important variables are compared using SHAP plots. We then conclude with a discussion of the implications of model setup on results and avenues for future research.
Literature Review
Hedonics, as introduction by Rosen, provides a theoretical basis for decomposing prices as arising from the combination of their attributes ( 17 ). The hedonic price literature for land and property is also informed by bid-rent theory, which posits that households trade-off between property and transportation costs ( 18 ). We structure the literature review around three themes: 1) machine learning applications in real estate price hedonics, 2) the relationship between transportation and locational variables with property prices, and 3) the use of cross-validation methods in spatial models across diverse fields.
Machine Learning in Real Estate Price Modeling
Machine learning methods have seen increased use in land and property price prediction to address the limitations of classical regression models, including low prediction accuracy and multicollinearity ( 19 ). Property price exhibits a strong spatial association, such that any model assessment must include a spatial component. For example, Ho et al. compared three machine learning methods: support vector machine (SVM), RF, and gradient boosting machine ( 20 ). They found that SVM provides the highest prediction accuracy among these methods. However, as summarized below, prediction accuracy is strongly influenced by the inclusion of spatial dependency in the goodness-of-fit statistic. Depending on the modeling approach and data, it is conceivable that SVM was not the highest performing method in the Ho et al. study if one were to consider spatial dependency ( 20 ). Soltani et al. estimate artificial neural network (ANN) models for Tehran using non-spatial cross-validation for comparison with a linear regression model ( 21 ). Both Genc and Colak (22), and Zhang et al. (23) highlight the importance of integrating spatially explicit methods (e.g., geographically weighted regression) into hedonic price models to better understand localized positive and negative impacts of transit investments, urban form, and neighborhood characteristics on housing markets ( 22 , 23 ).
Louati et al. highlighted the use of property price as an economic indicator (
24
). They estimated various machine learning models for price forecasting in Riyadh, Saudi Arabia. Their model used a district indicator, land and floor area, latitude and longitude, and type indicator (e.g., land, house, flat). RF provided the strongest predictive accuracy based on median squared error and mean absolute error (MAE). Limsombunc et al. followed a similar approach in Christchurch, New Zealand, using ANN to predict residential property price based on property features and community amenities (
25
). Their price dataset was relatively small (i.e., only 200 observations) so statistical power was low. They based their goodness-of-fit analysis on the
Mathotaarachchi et al. found overfitting in property price predictions with XGBoost and RF when evaluated using standard cross-validation ( 29 ). Genc et al. applied various machine learning models to property prices in Trabzon, Turkey, from which they measured local spatial prediction using inverse distance weighted spatial interpolation ( 30 ). They found RF performed the best in predicting adjacent prices.
Overall, we found that existing property price machine learning models fail to adequately consider spatial dependence in their design. Many studies found overfitting using non-spatial cross-validation but did not conduct suitable adjustments, such as incorporating spatial lag dependence or spatial cross-validation.
Impact of Transportation Infrastructure on Real Estate
There are numerous studies that examine the relationship between transportation variables and real estate prices ( 2 , 7 , 23 , 24 , 31 – 35 ). Among the studies of transportation access and real estate prices using machine learning methods, most focus on prediction accuracy rather than variable interpretation. Soltani et al. used several standard algorithms (i.e., decision trees, RF, and gradient-boosted trees) to predict residential prices in Adelaide, Australia ( 32 ). They incorporated a spatial-temporal lag. However, the lag variable is invalid, as they apply it to the dependent variable (i.e., price) and the lack of a likelihood function precludes the isolation of the dependent variable terms to a single side of the equation. While they include a wide range of accessibility variables, interpretation is limited to parameter signs. Trindade Neves et al. used XGBoost to study residential prices in Lisbon, Portugal ( 36 ). They used SHAP plots to provide a more robust interpretation of results but used non-spatial cross-validation, the limits of which we outline below. Kang et al. used gradient-boosting machines with non-spatial cross-validation and used Uber-derived transportation variables to predict housing prices in the Boston, U.S., area ( 37 ). Testing several machine learning algorithms (i.e., long short-term memory, gated recurrent units, and recurrent neural networks), Afandizadeh et al. found that accessibility leads to lower residential prices, counter to the standard literature findings ( 38 ). They attribute this result to noise pollution and decreased privacy, though it is equally likely to be because of omitted variable bias or their accessibility specification. Chen et al. considered public transit effects on single-family housing prices in Austin, Texas. Based on a variety of machine learning models and SHAP plots, they found that vulnerable (i.e., low-income and minority) communities place more value on access to public transit ( 39 ). Bowes and Ihlanfeldt highlighted the competing effects of rail transportation stations; access and retail employment effects are generally positive, while the induction of crime tends to reduce real estate prices ( 35 ).
Focusing on the regions in our case study, Toronto has a long history of hedonic property price studies. A 1976 paper by Dewees contrasted the effect of a new subway line on property prices parallel and perpendicular to its alignment ( 40 ). He found that the major effect was on the rent surface perpendicular to the subway and that walking time was a greater factor than subway travel time. Haider and Miller used a standard spatial autoregressive model to examine the effects of transportation and location variables on property prices in Toronto ( 41 ). They found that, after controlling for dwelling and neighborhood features, locational and transportation factors were not strong predictors of property price. Distance to the central business district (CBD) and shopping centers were the only variables found to statistically influence property prices. Zhang et al. proposed a systematic framework for decomposing prices into land price and dwelling price ( 42 ). They found that neighborhood socioeconomic environment had a larger influence on prices than either dwelling attributes or transportation access. In the most recent study of Toronto real estate and transportation, Kasraian et al. focused on comparison of regional and local transit accessibility indicators ( 43 ). They used a gravity-based measure to represent regional accessibility and distance to commuter and subway rail stations to represent local accessibility. They found that regional accessibility plays an important role beyond local accessibility.
Contrasting the focus in Toronto, Naeem and Rana found that water availability and proper drainage are critical factors when purchasing a property in Islamabad, followed by proximity to public transport, workplaces, shopping centers, and airports ( 44 ). Rehman and Jamil found that rents and commuting costs are generally higher in Islamabad than Rawalpindi ( 45 ). Rawalpindi residents are more likely to use motorbikes or public transit, while wealthier Islamabad residents are more likely to own an automobile. Pakistani cities also face a higher rate of property disputes than Canadian cities. There were an estimated 5,500 unregistered or illegal housing developments in Pakistan as of 2018 ( 46 ). Land-related disputes account for 50%–70% of all court cases in Pakistan ( 47 ). Islamabad faces an inability to finance public infrastructure development and maintenance, increased commuting costs because of urban sprawl, and affordability issues ( 48 – 51 ).
Cross-Validation Methods
Cross-validation is a resampling technique used in modeling to assess the predictive accuracy of a model and to tune the hyperparameters of the machine learning models. Conventional k-folds cross-validation partitions the data into k equally sized samples, then trains the model on k−1 samples. The held out sample is then used to test the predictive accuracy of the model, with the process being repeated for each of the k samples. The implicit assumption made by k-folds cross-validation is that observations are independent; that is, there is no dependence structure among observations. This assumption is typically violated in spatially referenced datasets by Tobler’s First Law of Geography—everything is related to everything else, but near things are more related than distant things ( 52 ). There are several approaches to overcome this limitation of cross-validation. The first approach is to impose a spatial contiguity constraint on the samples (spatial k-folds). Figure 3 illustrates the results for a 3-fold cross-validation example. An important feature of the spatial k-folds approach is that samples may not be drawn from the same data support. A lack of common data support among samples will lead to extrapolation. There are two perspectives of the implications of spatial k-fold cross-validation. If the modeling goal is inferential then the potential lack of a common data support violates a central tenet of causal inference ( 53 ). Conversely, in purely predictive modeling tasks, training and testing the model on spatially distinct datasets acts as a stress test. A second approach to cross-validation, accounting for spatial dependence, is buffered leave-one-out (B-LOO). In this approach, a single observation is withheld from each spatial fold ( 5 ). The advantage of this approach over spatial k-folds is that B-LOO cross-validation maintains a constant data support across the training and testing datasets. However, We used spatial k-means rather than B-LOO to ensure roughly equal sample sizes and balanced spatial coverage across folds; B-LOO evaluates one test location at a time and can leave large areas out of the training set, producing unstable variance estimates ( 6 ).
Comparing non-spatial and spatial cross-validation (CV) ( 5 ).
An important question about cross-validation on spatial data is the magnitude of the error associated with ignoring spatial dependence. Several studies find that standard non-spatial cross-validation provides overoptimistic error estimates because of the problem of spatial dependence (i.e., autocorrelation) ( 6 , 54 – 60 ). Pohjankukka et al. estimated the effect size at 40% overprediction based on three empirical case studies ( 61 ). Ploton et al. trained a RF model on environmental variables. They found that non-spatial cross-validation methods overpredict outcomes ( 4 ). Furthermore, they highlighted the close link between geographical and environmental domains, such that removing training observations in the spatial neighborhood of a test observation may remove the environmental conditions found at the test location from the model’s calibration domain. This removal may then influence the inferential validity of the model. Zhang et al. highlight the importance of spatial dependence in property price modeling, stating “housing units are prone to form a spatially aggregated cluster, which represent the regional price of a neighborhood” ( 42 ).
We contribute to the literature by contrasting non-spatial and spatial cross-validation for machine learning-based prediction. Our specific application is property prices in two diverse metropolitan areas. Contrasting results between these areas helps to identify the spatial and contextual extent of cross-validation variation. We use an interpretation approach, SHAP, to consider land use and transportation variables effects on prices.
Data
Rawalpindi Islamabad Metropolitan Area (RIMA)
We use residential sale price data for the twin cities of Islamabad (capital city) and Rawalpindi, Pakistan, for the year 2019. The data comprised about 19,417 records, with about 53% from Islamabad and 47% from Rawalpindi. The data include location, price, area, and dwelling type information. This dataset is combined with land use and transportation-related variables, chiefly derived from OpenStreetMap layers. Land use variables include distance to the old and new airports, old Rawalpindi city center, and major markets (e.g., Sadar Bazar in Rawalpindi and the Blue Area Market in Islamabad). Figure 4 illustrates the spatial distribution of RIMA data points and the location of major facilities used in our study. The color gradient represents the variation in price values. The prices are higher in the Islamabad region than in Rawalpindi city. This is because Islamabad is one of the most developed cities in the country with high quality of living and a concentration of major businesses in the downtown area. Prices are higher near major economic hubs. Overall, the spatial distribution of the data appears to be well represented across the study area.
Rawalpindi Islamabad Metropolitan Area (RIMA) price distribution.
Transportation-related variables included distance to bus stops, highways, roads, motorway (i.e., freeway) interchanges, and railway stations. Bus stops are divided into two types: 1) metro bus stops where buses have physically separated dedicated lanes and 2) regular bus stops. Distance to highways is included via two dummy variable that indicate whether the dwelling is located within 1 or 2 km of the nearest highway, respectively. For primary and secondary roads, dummy variables are used to identify whether the dwelling is located within 500 m or 1 km of these types of road. Table 2 shows the descriptive statistics of all variables used in our model. The prices in Table 2 are converted to CAD for comparison with Toronto. For model training, dwelling floor area is converted to square meters and all distances are measured in meters. All continuous variables, including sale price, are log transformed as a normalization (Table 2).
Rawalpindi Islamabad Metropolitan Area (RIMA) Descriptive Statistics
Note: na = not applicable.
*Based on average exchange rate in 2019 (1 Pakistani rupee = 0.0089 Canadian dollars).
City of Toronto
For Toronto, we use real estate data for the City of Toronto only, but consider the transportation system within the larger GTA metropolitan area. Figure 5 shows the map of GTA outlining the boundaries of major cities (i.e., census subdivisions) in the metropolitan region. Major cities in the region include Toronto, Mississauga, Brampton, Vaughan, and Markham. These cities are the population and economic hubs in the GTA region and affect land prices in the City of Toronto. The Toronto data are chosen to approximately match variables available for RIMA. Residential sale price data for 2016 were available from the Toronto Real Estate Board for 32,000 transactions within the City of Toronto. The data include location, price, area, and dwelling type information, similar to that available for RIMA. Figure 6 shows the spatial distribution of prices throughout the City of Toronto. The color gradient represents the variation in price values. Similar to RIMA, the spatial distribution of the data appears to be well represented across the City of Toronto.
Major cities and airports in the Greater Toronto Area (GTA).
Toronto price distribution.
OpenStreetMap land use and transportation data provided comparable variables to those available for RIMA. Transit station (i.e., subway, streetcar, and bus) distances parallel transit proximity variables calculated for RIMA. Distances to airports (i.e., Lester B. Toronto, Pearson International, and Billy Bishop Toronto City—see Figure 5) and major interchanges are calculated in a similar manner. The RIMA model used proximity to markets. To approximate this amenity variable, we calculated the distance to the city center (i.e., downtown) of major cities in GTA (see Figure 5). Table 3 shows the descriptive statistics of all the variables used in this study. For both study areas, we excluded records outside of the study area, records with outlier values, and all buildings other than houses.
Toronto Descriptive Statistics
The data for RIMA and Toronto were collected at similar times. The 2016–2019 period is not affected by either the global financial crisis (2008–2009) or the COVID-19 pandemic (2020–2022). Given the research objectives of comparing cross-validation methods and transportation infrastructure effects on real estate prices, the 3 year difference should have minimal effect on interpretation. If our objective was to examine the effect of an exogenous shock across real estate markets, it would be necessary to collect panel data and introduce year fixed effects.
Methods of Analysis
We first assess the presence of spatial autocorrelation in prices using the Moran’s I test to determine whether spatial effects should be considered in the modeling framework ( 63 ). The Moran’s I statistic is given by Moran ( 62 ):
where
We then use XGBoost and RF models to measure variations in prediction accuracy based on non-spatial and spatial cross-validation frameworks. Both models are ensemble tree-based methods, such that spatial k-folds can be incorporated into the model training stage rather than prediction accuracy analysis only. The RF method belongs to the bagging class of ensemble learning algorithms that builds multiple decision trees and combines their outputs to improve prediction accuracy and reduce overfitting ( 63 ). The diversity among trees is created through two randomization mechanisms: 1) each tree is trained on a bootstrap sample from the training data and 2) a random subset of predictors is selected at each node split. The final prediction is obtained by aggregating the outputs from all trees—through averaging in regression or majority voting in classification tasks. The RF method has three hyperparameters:
The total number of decision trees (trees) in the ensemble model: more trees generally decrease the error but increase the computational time
The size of the random subset of predictors at each node split (mtry)
Node size (min_n) which represents the smallest allowable number of observations in a terminal node
For RF, iterative training via bagging generates a strong learner from multiple decision trees based on bootstrap sampling (which may incorporate spatial dependence). Roberts et al. and Valavi et al. recommend such a spatial blocking approach for RF training to mitigate autocorrelation errors and ensure independence between training and testing sets ( 6 , 64 ).
XGBoost is also a tree-based ensemble method, wherein trees are built in a sequential manner to reduce the error of former trees via a boosting approach ( 65 ). Whereas RF addresses variance through bagging, XGBoost focuses on bias reduction by boosting weaker learners. XGBoost is more prone to overfitting, requiring careful parameter tuning. In addition to (trees), (mtry), and (min_n), the hyperparameters of the XGBoost model are:
The maximum depth of a tree (tree_depth)
Learning rating that scales the contribution of each tree to prevent overfitting (learn_rate)
The minimum loss reduction required to create an additional split on a leaf node (loss_reduction)
Subsample ratio of the training data (sample_size)
In this study, we fix the number of trees to 1,000, because model performance does not improve beyond 1,000 trees and computational time increases substantially. The optimal values of all other hyperparameters are selected using a hyperparameter tuning process. The process involved generating a random grid of hyperparameters and testing the values using a cross-validation process. The optimal values of hyperparameters are selected based on the best performing model. In this study, we use a grid size of 20 and 5-fold cross-validation procedure. We use k-means clustering for spatial cross-validation and random sampling for non-spatial cross-validation. The model performance is evaluated using RMSE, MAE, mean absolute percentage error (MAPE), and R2. The optimal values of the hyperparameters are selected based on the RMSE value.
Model Development
The model development is comprised of two steps.
In the first step, for both spatial and non-spatial approaches, the data are divided into five samples. For the spatial approach, the data are divided using a spatial clustering approach. Specifically, we apply k-means clustering based on the spatial coordinates to create spatially contiguous folds for cross-validation. In contrast, the non-spatial approach involves random partitioning of the data. We develop five models for each approach (i.e., spatial and non-spatial) using both XGBoost and RF, resulting in a total of 20 models. The reason for developing five models for each approach is to tease out the variability in model performance because of differences in data partitioning. Each sample contains a training set and a testing set. Figures 7 and 8 illustrate how the data are divided into two spatially distinct regions for each study area. In contrast, Figures 9 and 10 show that training and testing sets are divided randomly for non-spatial cross-validation, with the model being essentially trained and tested on similar data. The rest of the spatial and non-spatial folds are provided in Appendix A.
Training and testing set 1 based on spatial clustering (Rawalpindi Islamabad Metropolitan Area [RIMA]).
Training and testing set 1 based on spatial clustering (Toronto).
Training and testing set 1 based on non-spatial clustering (Rawalpindi Islamabad Metropolitan Area [RIMA]).
Training and testing set 1 based on non-spatial clustering (Toronto).
In the second step, each model is trained on the training set using 5-fold cross validation (using k-means clustering with spatial coordinates in the case of spatial models and random sampling in the case of non-spatial models). The optimal values of the hyperparameters are selected during the 5-fold cross validation procedure.
One challenge with using machine learning methods for inferential analysis is parameter interpretation. SHAP plots are a useful tool for this purpose ( 66 ). SHAP is a game theoretic approach that assigns an importance value to each model feature. The appealing properties of SHAP are that it is 1) additive, such that feature contributions are independent and additive to the cumulative effect on the outcome, 2) locally accurate, such that SHAP provides local interpretation of model prediction for a given input, 3) robust to missing values, and 4) consistent under changes in model specification. SHAP parameters are used in this study to compare feature importance.
Results
Moran’s I statistics for Toronto and RIMA property prices were 0.70 (p-value
Hyperparameter Tuning
Table 4 presents the optimal hyperparameters of the XGBoost models developed for RIMA under both spatial and non-spatial approaches, while Table 5 provides the corresponding results for Toronto. The variability in hyperparameter values is higher in spatial models than non-spatial models. This further shows the impact of data variability among spatial folds compared with non-spatial folds. Similar results are observed in the case of RF models (see Tables 6 and 7). These parameters are used to develop the final models on the respective training sets, and model performance is assessed on the corresponding testing sets.
Rawalpindi Islamabad Metropolitan Area (RIMA) XGBoost Model Hyperparameter Values
Note: Min_n = node size (represents the smallest allowable number of observations in a terminal node); Mtry = the size of the random subset of predictors at each node split
Toronto XGBoost Model Hyperparameter Values
Note: Min_n = node size (represents the smallest allowable number of observations in a terminal node); Mtry = the size of the random subset of predictors at each node split
Rawalpindi Islamabad Metropolitan Area (RIMA) Random Forest Model Hyperparameter Values
Note: Min_n = node size (represents the smallest allowable number of observations in a terminal node); Mtry = the size of the random subset of predictors at each node split
Toronto Random Forest Model Hyperparameter Values
Note: Min_n = node size (represents the smallest allowable number of observations in a terminal node); Mtry = the size of the random subset of predictors at each node split.
Model Performance
This section summarizes the model performance results for the 10 XGBoost and 10 RF models, and five spatial and five non-spatial models, trained on data from each study area. The spatial models used k-means clustering using spatial coordinates to split the data for cross-validation, while the non-spatial model used random sampling for cross-validation.
Tables 8 and 9 shows model performance results of XGBoost and RF for the RIMA training data. Overall, the results show higher model performance for non-spatial models than spatial models. These results parallel prior findings in the literature that non-spatial models over-predict model accuracy because they do not consider spatial autocorrelation in the cross-validation process (
6
,
54
–
60
). According to the non-spatial model results, prediction performance is not influenced by how the data are divided into training and testing sets. Performance metrics are consistent across all five non-spatial models. However, this pattern does not hold true for the spatial models. For instance, for the XGBoost model (see Table 8), the mean
Rawalpindi Islamabad Metropolitan Area XGBoost Model Performance on Training Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Rawalpindi Islamabad Metropolitan Area Random Forest Model Performance on Training Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Tables 10 and 11 provide similar results for Toronto for the XGBoost and RF models, respectively. As with the RIMA model performance results, non-spatial models exhibit no variation in their results for either modeling method. In the same way, spatial performance results vary across models. These results strengthen the argument that non-spatial models consistently overpredict model performance and neglect inter-model variation arising from spatial heterogeneity. This conclusion is further supported by applying the models to two different cities.
Toronto XGBoost Model Performance on Training Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Toronto Random Forest Model Performance on Training Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Each model is trained on the training dataset using its corresponding optimal hyperparameters and subsequently applied to the testing dataset to evaluate predictive performance. Table 12 shows the XGBoost model’s performance and Table 13 shows the RF model’s performance on the testing data for RIMA. Overall, we observed that the performance of non-spatial models is higher than spatial models. The variation in model performance in spatial models is higher than non-spatial models, which shows that the five spatial models are different models while the five non-spatial models are almost the same model. The difference in model performance between training and testing data among spatial and non-spatial XGBoost (see Tables 8 and 12) and RF (see Tables 9 and 13) models also reinforce these findings. Specifically, the RF model performance on the training data and testing data of the first spatial model shows high model overfitting on the training data, while the XGBoost model performed better than the RF model. To further examine this performance difference, we plot a box plot comparing the distribution of prices across the two datasets. Figure 11 shows the box plot of price (log-transformed) for the first spatial fold. Figure 11 shows the difference in prices between the training and testing data indicating that the RF model did not perform well for out-of-sample prediction. The box plots for other folds are presented in Appendix B. The Toronto models show better performance among the training (Tables 10 and 11) and testing datasets (Tables 14 and 15). This difference suggests the Toronto real estate market is more clearly described by the chosen variables, while RIMA exhibits greater residual unexplained variations.
Rawalpindi Islamabad Metropolitan Area XGBoost Model Performance on Testing Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Rawalpindi Islamabad Metropolitan Area Random Forest Model Performance on Testing Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Toronto XGBoost Model Performance on Testing Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Toronto Random Forest Model Performance on Testing Data
Note: MAE = mean absolute error; MAPE = mean absolute percentage error; RMSE = root mean squared error.
Distribution of prices in Rawalpindi Islamabad Metropolitan Area (RIMA) spatial fold 1.
Because of the overfitting of the spatial RF models in the case of RIMA, we chose XGBoost models as the preferred models. We also conducted residual analysis on the XGBoost models using testing data for both RIMA and Toronto. Figures 12 and 13 show XGBoost model residual results for both spatial and non-spatial models for RIMA and Toronto, respectively. Results for both study areas confirm that non-spatial XGBoost outperforms spatial XGBoost under random cross-validation prediction accuracy. Residual magnitudes are substantially larger in RIMA than Toronto, reflecting a more heterogeneous and data-sparse market where distance-based features alone cannot fully capture hyper-local price determinants, such as gated-community premiums. Spatially, Toronto’s residuals exhibit clear neighborhood-level clustering, especially along the waterfront and outer suburbs, whereas RIMA’s spatial clustering is more concentrated in the dense Rawalpindi core.
Rawalpindi Islamabad Metropolitan Area (RIMA) XGBoost model residuals: (a) spatial XGBoost using five spatial cross-validation (CV) test folds and (b) non-spatial XGBoost using five non-spatial CV test folds.
Toronto XGBoost Model Residuals: (a) spatial XGBoost using five spatial cross-validation (CV) test folds and (b) non-spatial XGBoost using five non-spatial CV test folds.
Discussion
Impact of Land Use and Transportation on Property Prices in RIMA
In this section, we present SHAP plots only for the XGBoost models. SHAP beeswarm plots provide a rich summary of feature importance. Each dot represents a single data point, with its horizontal position showing the impact (i.e., SHAP value) of a feature on the model’s prediction, and the color representing the value of that feature. Features are ranked on the y-axis by their overall importance (i.e., mean absolute SHAP value). Figure 14 shows SHAP values for the first non-spatial model results for RIMA. (SHAP plots for the remaining four folds are included in Appendix C). The variables are arranged based on their importance from high to low. For instance, area is the most important variable, and proximity to a highway is the least important variable in predicting the property sale price. The color shows the value of a variable (or feature), which in the case of dummy variables only has two values. Figure 14 also shows non-linear relationships i.e., longer tails in one direction than the other between dependent and independent variables.
SHapley Additive exPlanations (SHAP) values for non-spatial model 1 (Rawalpindi Islamabad Metropolitan Area [RIMA]).
Floor area is the most important variable. As expected, a larger building area correlates with a higher sale price. Accessibility to transportation infrastructure also affects prices. For instance, houses located closer to motorway ramps have lower sale prices. This result is because motorways in Pakistan are typically for inter-city travel and are tolled. Motorway interchanges are mostly located at a city’s outskirts. However, we note that the non-spatial model exhibits a weaker effect than observed in the spatial model (see Figure 15). That is, using non-spatial cross-validation tends to miss the effect size (i.e., importance ranking) and the negative effect on property price captured by the spatial model.
SHapley Additive exPlanations (SHAP) values for spatial model 1 (Rawalpindi Islamabad Metropolitan Area [RIMA]).
Based on the spatial model, proximity to metro stations and regular bus stops has a negative impact on sale price, but the impact of metro stations is larger than that of regular bus stops. This difference could be because metro bus has fixed stops and connects all major commercial centers in Rawalpindi and Islamabad. The overall quality of the service (e.g., comfort and reliability) is far superior to the other bus service. As expected, sale prices are also higher near the major commercial centers, as well as both downtown Rawalpindi and Islamabad. These areas are major employment centers and have higher accessibility to services. Proximity to airports affects sale price, but the impact is unclear. The old airport is located closer to the Sadar market, while the new airport is very far from both the cities. Distance to highways has a weak effect on prices.
Impact of Land Use and Transportation on Property Prices in Toronto
Figure 16 shows SHAP values for the first non-spatial model for Toronto. The variables are arranged based on their importance from high to low. Floor area is the most important variable and proximity to the nearest streetcar station is the least important variable in predicting the property sale price. Figure 16 also shows non-linear relationships (i.e., longer tails in one direction than the other) between dependent and independent variables. Distance to the nearest motorway has a positive influence on property sale price, while distance to Toronto downtown has a negative effect, meaning that higher-priced properties tend to be located closer to Toronto downtown. Accessibility to transportation infrastructure also affects prices. Proximity to a subway station tends to be associated with higher prices, while proximity to bus stops tends to decrease price. This result is likely capturing the location of bus stops along major roadways that are viewed as undesirable, whereas subway stations tend to be located in desirable areas with high amenity concentrations (e.g., theaters and restaurants). Proximity to the major airport, Pearson International Airport, is also perceived as undesirable.
SHapley Additive exPlanations (SHAP) values for non-spatial model 1 (Toronto).
The strength of many features is weaker in the spatial model than the non-spatial model, as shown in Figure 17 wherein the SHAP values are less variable. The effect of distance to motorways is weaker in this model, while the effect of proximity to Toronto downtown remains strong.
SHapley Additive exPlanations (SHAP) values for spatial model 1 (Toronto).
Comparison of RIMA and Toronto Results
The Toronto market exhibits several similarities to that of RIMA, despite their geographic and socioeconomic differences. Both models found that floorspace area is the most important factor in determining sale price. Higher-order transit (e.g., BRT for RIMA and subway for Toronto) have strong positive effects on property price. Proximity to the major central city is desirable in both areas. Results for airport proximity differ between the two areas depending on whether one considers non-spatial or spatial model results. Distance from the new airport is desired in RIMA, to avoid the noise and associated traffic. This effect is also strong for Pearson Airport in GTA in the non-spatial model but much weaker when using spatial cross-validation. Proximity to Billy Bishop Airport arises as a desirable feature in the spatial Toronto model; however, it must be noted that this airport is relatively small and located on an island connected to downtown Toronto by a tunnel. Distance to non-Toronto city centers have fairly weak effects, similar to non-Islamabad markets in RIMA. With respect to roadway variables, RIMA has a stronger feature importance for motorways, whereas Toronto shows a stronger influence from subway and streetcar proximity. Higher-order transit (e.g., rail and metro bus) station proximity have stronger effects on sale price than local bus station proximity in both regions. Some of the differences in the impact of transportation infrastructure, specifically the public transit system, are because of cultural and planning differences between the two cities. For example, the spatial coverage of transit infrastructure in GTA is far higher than RIMA. The transit infrastructure in RIMA is not well connected, especially in Islamabad. However, modes such as rickshaws and motorcycles are common in RIMA but not GTA.
Our results align with prior research in both regions, finding that proximity to the main CBD tends to increase property price, aligned with the monocentric city model of Alonso and subsequent research in urban economics ( 18 , 67 ). Based on SHAP analysis, we recommend prioritizing high-capacity, dedicated right-of-way transit investments (i.e., BRT/subway expansion) to maximize positive property value impacts and fund transit through land value capture ( 43 ). Methodological findings also have implications for policy. Non-spatial models overestimate accuracy; thus, policy analysts should use spatial validation techniques in real estate appraisal and land use planning models to ensure reliable and generalizable predictions for investment decisions.
Conclusions and Future Work
Land use and transportation systems interact with each other in cities. Residential property prices are an important component of urban systems that are influenced by the integrated nature of land use and transportation. In this study, we examined the impact of land use and transportation systems on property prices using machine learning methods and compare the results between two diverse areas (i.e., Toronto in Canada and RIMA in Pakistan). We also investigated the differences between non-spatial and spatial cross-validation methods arising from the presence of spatial autocorrelation. We confirmed prior findings in ecology and other fields that non-spatial cross-validation leads to erroneous prediction accuracy results applying machine learning models in two diverse regions and comparing multiple model diagnostic statistics ( 6 ). These findings are important to highlight for transportation researchers as they increasingly apply machine learning methods developed for non-spatial uses in spatial contexts.
We found similar impacts of land use and transportation infrastructure for both Toronto and RIMA, despite their socioeconomic differences. We also found non-linear effects of these variables in predicting property prices for both regions using SHAP-based variable importance. These non-linear impacts are crucial to capture, especially for accurate policy analysis. For instance, such results are useful to test the impact of infrastructure investments by type of transit system such as bus, subway, or BRT. While we provide results for both developed and developing economy cities, robust generalization of our results will require applications across more cities. We also emphasize that our study focused on high-level associations rather than causal identification of a specific treatment effect. Causal machine learning is a growing field ( 68 ), but one we did not explore in this work ( 68 ).
Turning to the cross-validation question, we used a spatial blocking approach to cross-validation, which has some trade-offs associated with its application. This approach addresses the independence assumption of cross-validation, that hold-out testing samples are independent from the training sample, but may lead to the testing sample outcome values not existing within the same support as the training data. The interpretation of this assumption depends on whether one is interested in inference or extrapolation in predictor space. An alternative approach would be to block the data in the predictor space based on dissimilarity criteria; that is, spatially cluster observations to maximize the dissimilarity within each fold (i.e., similarity across folds) across predictor variables. To our knowledge, there are few examples in the literature of cross-validation using blocks defined in the predictor space in this way.
A second challenge with marrying machine learning and spatial econometrics is the lack of a likelihood function in most machine learning approaches. This difference precludes the consistent introduction of spatial lag or error terms. Flexible non-parametric statistical models, such as Bayesian additive regression trees, should be explored ( 69 ). They offer the advantage of maintaining a likelihood function for integration with standard spatial econometric specifications, while maintaining the flexibility of algorithmic machine learning models.
Other future work might include the use of panel data to study the dynamics of price changes. Such dynamics are important to understand how transit investments affect socio-spatial equity, a challenge identified in both Toronto and RIMA. Our transit variables also did not consider reliability or service quality variation across the two study regions. Such information is available for Toronto but may be more challenging to obtain for RIMA or similarly developed areas.
Supplemental Material
sj-pdf-1-trr-10.1177_03611981261457410 – Supplemental material for Residential Price Modeling Using Spatially Validated Machine Learning Methods: A Comparison Across Geographical Contexts
Supplemental material, sj-pdf-1-trr-10.1177_03611981261457410 for Residential Price Modeling Using Spatially Validated Machine Learning Methods: A Comparison Across Geographical Contexts by Usman Ahmed and Jason Hawkins in Transportation Research Record
Footnotes
Author Contributions
The authors confirm contribution to the paper as follows: study conception and design: U. Ahmed, J. Hawkins; data collection: U. Ahmed, J. Hawkins; analysis and interpretation of results: U. Ahmed, J. Hawkins; draft manuscript preparation: U. Ahmed, J. Hawkins. All authors reviewed the results and approved the final version of the manuscript.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
