Abstract
State Departments of Transportation (DOTs) are facing workforce shortages and a decline in experienced construction workers. One approach to address this problem is through the active adoption of modern technologies, particularly artificial intelligence (AI) tools. Multimodal large language models (LLMs) offer emerging capabilities, such as instruction following and question answering, making them suitable for numerous practical applications in civil engineering. A key area of interest is effectively communicating complex concepts to train engineers and construction workers; however, there remains a need for a structured approach to selecting appropriate models and evaluating LLMs’ knowledge for specialized engineering areas. This paper presents an evaluation of twenty-six different LLMs using over 100 automatically generated questions, revealing a wide range of accuracies in concrete pavement construction expertise. The proposed methodology uses carefully selected, domain-specific documents and best practices to automatically build a customized framework for assessing model performance in niche fields. It also demonstrates that techniques like the “Retry” method and the integration of domain-specific information through retrieval augmented generation (RAG) can significantly enhance LLM accuracy. The results indicate that open-source, small- and medium-sized models combined with RAG, which can run efficiently on laptop computers, could be deployed for future applications. These techniques show promise for model improvement without additional fine-tuning or retraining, significantly reducing the computational requirements for tailored engineering tasks.
Keywords
State Departments of Transportation (DOTs) are facing workforce shortages and quality of maintenance challenges in concrete pavement construction projects. The decrease in experienced construction workers and inspectors has led to productivity issues, prompting the need for innovative strategies to manage inspection workloads ( 1 , 2 ). Other problems, such as high turnover rates, labor shortages, and diversity issues, emphasize the need for skill development and retention programs ( 3 , 4 ). Strategies such as involving workers in decision-making, providing professional training, and creating a supportive work environment have been identified to address many of these concerns ( 4 ). Most importantly, young professionals in transportation engineering must undergo training to stay abreast of technological advancements, standards, and policies. Factors such as relevance, cost, and location play a significant role in determining adequate education and training programs ( 5 ), which are essential for improving career readiness and addressing skill gaps in civil engineering and transportation ( 6 ). Overall, the construction industry’s workforce shortages, training quality, and knowledge transfer challenges necessitate a strategic approach to workforce development and management.
To address workforce scarcity and enhance efficiency without compromising quality, DOTs have turned to finding new ways to streamline practices ( 7 ). These concerns can be addressed with modern technologies, particularly artificial intelligence (AI) tools. Construction projects have recently explored AI, demonstrating its potential to enhance various aspects such as safety culture, maintenance strategies, asset management, and decision-making processes. Automated inspection technologies, including unmanned vehicles, robotic dogs, and multimodal large language models (LLMs), are revolutionizing construction inspection processes ( 8 ). These technologies facilitate the rapid generation of high-quality inspection reports, real-time image transmission, and efficient data collection for safety inspections. Furthermore, LLMs, a subset of Generative AI (GenAI), have shown impressive capabilities for streamlining operations by analyzing data, predicting equipment failures, optimizing maintenance schedules, and improving risk management practices ( 9 ).
LLMs such as OpenAI’s Generative Pre-Trained Transformer (GPT), based on transformer architectures, have significantly affected natural language processing (NLP) and content generation applications ( 10 , 11 ). The advancement of these models offers emergent abilities, such as instruction following and question answering ( 12 , 13 ), suiting them for many practical applications in the realm of civil engineering as demonstrated by various studies.
For example, Smetana et al. utilize GPT-3.5, accompanied by other NLP and machine learning techniques, to analyze highway construction accidents using only textual narratives ( 14 ). The results of this study signify that LLMs should be explored in future studies, while specifically targeting areas of interest such as enhanced safety protocols, training programs, and site inspections to reduce inherent risks in construction work zones. Similarly, Aladağ assessed ChatGPT’s performance over risk management processes in construction projects ( 15 ). Through a questionnaire and expert evaluations, Aladağ found that ChatGPT has a moderate understanding of managing risks, with strengths in risk response and monitoring. These findings demonstrate the importance of combining human expertise with AI tools in any future applications, and also suggest the need for real-life case studies and historical data to enhance the models’ accuracy over time ( 15 ).
Other examples include Prieto et al. evaluating the performance of LLMs for project management ( 16 ), along with Yao and García de Soto seeking to enhance cyber risk identification in the construction industry through LLM utilization ( 17 ). Prieto et al. found benefits in using models like ChatGPT in project management, specifically tasking it to generate construction schedules based on provided information; however, limitations such as generic responses and task relevance surfaced ( 16 ). Yao and García de Soto trained a base model on a large text corpus related to cybersecurity and construction, finding that although the model has the potential to serve as a cybersecurity consultant, additional updates may enhance the model’s capabilities ( 17 ).
As confirmed by these recent studies, the versatility and effectiveness of these technologies in construction are evident in addressing diverse challenges and driving innovation in the industry ( 9 ), though they also indicate the necessity of improving performance for applications in civil engineering. Despite the potential benefits of GenAI models in construction, the industry is still in the early stages of adoption, requiring its users to establish formal policies and controls for optimal utilization. One area of particular interest is explaining complex concepts to train engineers and construction workers. Similar to most civil engineering industries, the concrete pavement construction industry incorporates domain knowledge into textual data (i.e., specifications and other training materials), offering the potential to apply LLMs to address the challenges of workforce development and employee retention.
Specialized training and models tailored to specific domains, such as concrete pavement construction, are crucial for effective information processing ( 18 ). LLMs have shown promise in improving language understanding, analysis, and common-sense reasoning across various fields ( 19 ). However, challenges like interpretability, fairness, and domain-specific knowledge integration persist, necessitating further research ( 20 , 21 ). Factual inaccuracies produced by these models highlight the need for robustness and ethical considerations in future development ( 22 ). Specialized LLMs like the ChatDoctor model have demonstrated superior performance in clinical information prediction and question-answering tasks compared with general-purpose models like GPT ( 11 , 18 , 23 ).
Essentially, language model application poses challenges and opportunities for domain adaptation. To trust the generated content from these models, they should be supplemented with additional information through methods such as fine-tuning and subsequently evaluated before practical use. Furthermore, evaluating LLMs for question-answering faces hurdles in assessment because of limited datasets and appropriate metrics, affecting performance evaluation in specific domains. Incorporating external knowledge into LLM context generation can enhance performance, but it may struggle with complex questions requiring information from multiple sources or modalities. To further emphasize the importance of incorporating domain-specific information into these models, Zhong et al. assessed language models’ understanding of construction management systems (CMS) texts ( 24 ). Through additional fine-tuning, their study enabled pre-trained models to better perform under various NLP tasks in CMS and related domains.
Transportation engineers should be aware of the capabilities of current LLMs before using the generated content in their applications. To be able to utilize these technologies for training, knowledge transfer, or simple question answering, further assessment of available models is vital. Currently, there exists a gap in the evaluation of LLMs’ knowledge of concrete pavement construction, leaving potential users unaware of their capabilities in this niche domain. This study seeks to close this gap by evaluating open-source models’ understanding of complex topics ranging from subgrade preparation to surface smoothness in efforts to provoke the usage of performant models in future tasks like employee training. It is assumed that many users default to leading models such as ChatGPT for their tasks; however, open-source models provide security when using sensitive data, as many models can run locally on a consumer-grade laptop.
Through a careful selection of documents representing the identified topics, we present a framework to develop similar evaluations in future studies. First, we foster the ability to automatically generate evaluation quizzes in the form of multiple-choice questions based on ground truth data extracted directly from reliable pavement construction documents. A total of twenty-six state-of-the-art language models are then prompted to answer the generated questions for further evaluation. From these results, we are able to identify how models tend to perform over the various topics and which models may be suitable for downstream tasks. Additionally, we present a technique to improve LLM performance by injecting context from new documents into the generated questions. This methodology procures more accurate responses from the models without the need for additional training, commonly referred to as fine-tuning. As further training is generally expensive, resource-intensive, and time-consuming, the ability to garner more accurate generated content with trusted information may be ideal for many practical civil engineering applications.
Methods
This study developed a three-stage approach to evaluate the expertise of modern LLMs in concrete pavement construction, as illustrated in Figure 1. In the first stage, multiple-choice questions are automatically generated using OpenAI’s GPT-3.5-Turbo. These questions are based on information from relevant documents, particularly best practices and guides for various aspects of rigid pavement construction. Identifying specific topics to be evaluated, such as subgrade preparation, is crucial in assessing LLM performance in different areas within the domain. The abundance of extracted textual information necessitates a manual review to ensure the questions used in the subsequent evaluation stage are relevant, accurate, and high quality.
Question generation and large language model (LLM) evaluation approach.
After completing the first stage, the second stage involves evaluating significant open-source LLMs. Various models from major corporations significant in the AI industry, like Microsoft, Google, and Meta, are selected based on their sizes and claimed performances. Each selected model is then prompted to answer the questions generated in the previous stage. The responses are analyzed to assess the models’ accuracy across the prescribed topics. To ensure concise answers, such as “c” over “The correct answer is c,” a retry prompting strategy is implemented. This strategy grants the ability to capture whether a model provides the correct answer, but requires additional parsing for extraction.
Finally, in the third stage, the incompetencies of the tested LLMs in certain topics are addressed by providing additional context from new documents through retrieval augmented generation (RAG). A Python script created by the authors vectorizes documents, ranks relevant pieces of information, and injects them into the prompt to guide the model in selecting the correct answer. This method allows LLMs to adapt to areas where their expertise is limited, resulting in more accurate responses to domain-specific questions.
Question Generation
The field of NLP has seen advancements in question generation, focusing on creating questions from text and improving question–answer (QA) pair models. These questions can be answer-aware or answer-agnostic, showcasing the various approaches used in this area ( 25 ). Automated systems leveraging these techniques efficiently create multiple-choice questions (MCQs), saving time and effort in the manual question generation process ( 26 , 27 ). As an example of its potential, Gilal et al. demonstrate a system named QURU, offering a user-friendly approach to MCQ creation by extracting stems, keywords, and distractors from text content, providing teachers with valuable tools for question generation ( 27 ). Other educational tools like EduQuiz have enhanced quiz generation processes, emphasizing the need for expert feedback in assessing QA pair quality and the benefits of automated QA pairs ( 26 , 28 ).
Similar to systems like QURU and EduQuiz, this section presents the methods employed to facilitate automatic MCQ generation from concrete pavement construction documents with a premise of creating evaluation quizzes to assess the multitude of LLMs presented in a later section. The overall process involves collecting relevant documents, converting them to quizzes with OpenAI’s GPT-3.5, and utilizing manual evaluation of the generated content. The latter manual judgment allows for the procurement of a reliable dataset of questions related to various topics in the domain.
Collection of Documents and Topic Selection
To obtain relevant question generation, several high-quality concrete pavement construction documents representing domain knowledge were selected. These documents provide complete, relevant, and substantial amounts of textual information related to the following five topics: subgrade preparation, base preparation, joint sawing/sealing, finishing and placement, and surface smoothness, also covering the following concepts. Although these topics do not encapsulate all facets of rigid pavement construction, they provide enough context to build durable, safe, and efficient pavements.
Topic 1: Subgrade Preparation ( 29 ) – stability, preparation, staking, grading equipment, lime/cement stabilization, proof rolling, traffic control, and moisture/environment
Topic 2: Base Preparation ( 29 ) – materials, stability, cement-treated base (CTB), mix/compaction, econocrete, and drainage layer
Topic 3: Joint Sawing/Sealing ( 30 ) – requirements, sawing process, and sealant types
Topic 4: Finishing and Placement ( 31 ) – paving methods, curing/hydration, machines/equipment, surface preparation, vibration/consolidation, environmental conditions, and curing compounds
Topic 5: Surface Smoothness ( 32 ) – cracking, irregularities/ripples, spring back/humps, and surface variance
Generating Questions
After parsing and extracting content from the selected topics, a total of 250 questions were automatically generated with GPT-3.5-Turbo using the prompt shown in Figure 2. The inference of modern language models is achieved using textual inputs, known as prompts, which involve forming questions, instructions, or statements that guide a model to generate certain responses. These prompts can contain a single example that the LLM can learn from to provide a similar response, known as one-shot learning ( 12 ).
Question generation prompt.
Within the prompt in Figure 2, the left-hand side shows the overarching instructions for GPT-3.5-Turbo to generate questions. The highlighted brackets represent inputs expected by the user. In this case, the injected field is “concrete pavement construction,” and the injected topic is a placeholder for one of the five previously described topics. The injected number of questions is used as an input to determine the quantity of questions to generate per extracted text, which can be adjusted based on user preferences (three for the extent of this study). Additionally, this prompt defines a set of rules carefully curated to guide the model in creating more representative and reliable questions. The one-shot example on the right-hand side ensures that the model produces a consistent output. The example text and answers are arbitrarily derived and serve no other purpose. This style of one-shot prompting is used throughout the study.
Manual Evaluation
The authors manually evaluated the initial set of automatically generated questions (250 total). Their assessment revealed that 101 seed questions were relevant, accurate, and high-quality. This resulted in twenty-one questions for Subgrade Preparation, eighteen for Base Preparation, six for Joint Sawing/Sealing, forty-four for Finishing and Placement, and twelve for Surface Smoothness. Sample questions for all five topics and the distribution of concepts are shown in Figure 3.
Sample questions and distributions for: (a) subgrade preparation, (b) base preparation, (c) joint sawing/sealing, (d) finishing and placement, and (e) surface smoothness.
LLM Evaluation
Several LLM models have been identified and evaluated in this study. The main focus was made on open-source models because of their free licensing, the ability to give users full control over data, security, and confidentiality, as well as the opportunity to run models locally without network connectivity. Some of these models have several configurations, as discussed below. Their performance was compared with two proprietary versions of OpenAI’s GPT selected because of their popularity. Table 1 summarizes the selected LLM configurations used in this study and their main characteristics, including the following:
Parameters: the number of trainable parameters within each model. A higher number of parameters means a more capable model of understanding and generating more nuanced text while requiring higher operational costs. Selected models were grouped into three categories: small (fewer than 7B parameters), medium (more than 7B but fewer than 13B), and large (more than 13B).
Weights size: the disk or memory space required to store the model’s weights, which contain LLMs’ numerical parameters.
Context: the maximum number of tokens (words or parts of words) the model can consider simultaneously while processing input. Typically, 1000 tokens correspond to 750 words in English, comprising 1.5 pages. Models with larger context windows can maintain longer narratives or handle more complex and nuanced interactions.
Quantization: the process of simplifying a large, complex model by reducing the precision of the numbers it uses, making it smaller and faster. This approach reduces memory and computational resource requirements, enabling model execution on laptops, but sacrifices some accuracy. “Supported” indicates whether the model can be quantized, and “Utilized” refers to whether an LLM has undergone quantization for evaluation.
Large Language Models Selected for Quiz Evaluation (*Local Model, Q4 Quantization)
Note: OpenAI does not disclose the number of parameters or the weights size in their documentation for GPT-3.5-Turbo and GPT-4o-mini models. NA = not available; na = not applicable.
These twenty-six LLM configurations were asked the same set of MCQs, with the same prompt as shown in Figure 4. This prompt contains the instructions for the model (left), and another one-shot example (right). The answer was considered to be correct only if the correct option was selected and only if it was given in proper format. This format, as specified in the one-shot example, is a single letter, with optional symbols such as parentheses or quotes (e.g., “c,”“‘c’,” or “(c)”). It is required to facilitate automatic evaluation, which can handle an arbitrary number of questions without the need for human intervention.
Multiple choice question answering prompt.
For each model configuration, to evaluate how well they performed, the following accuracy was computed:
where,
TP = The number of correctly predicted positive cases (correct answer),
TN = The number of correctly predicted negative (not applicable),
FP = The number of incorrectly predicted positive cases (wrong answer), and
FN = The number of incorrectly predicted negative cases (not applicable)
Response Parsing (Retry)
In the evaluation process described above, there were cases when LLMs identified the correct option but gave the answers in the wrong format. Because the models didn’t follow the instructions, those answers were classified as incorrect. To improve the overall quality of the evaluation, an additional technique, Retry, requesting the LLM to reformat its answer, was applied.
Consider a case when the LLM responded “The correct answer is (b) To keep the temperature down and apply the curing compound early.” Because the answer has a wrong format, the LLM was immediately prompted as shown in Figure 5 with {inject text} replaced with “The correct answer is (b) To keep the temperature down and apply the curing compound early.”
Response parsing (retry) prompt.
Domain Adaptation
The third stage of the evaluation process involved assessing the ability to improve LLM accuracy in responding to questions by providing it with additional information from sources independent from the document(s) used to generate text questions and using the RAG technique, as illustrated in Figure 6. RAG is a modern approach in QA systems that combines generative retrieval mechanisms to boost performance and accuracy by adapting to dynamic data and reducing the occurrence of incorrect answers ( 51 ). The approach implemented in this study involved vectorizing the information and searching through document(s) to find relevant pieces of information using a Python script based on the input query or prompt. This additional information is supplied to an LLM as a part of the prompt to enable the model to make a more informed decision. The advantage of RAG is that, although it doesn’t require costly modification of the LLM, it allows the model to be adapted to a specific domain by closing the knowledge gaps with relevant context. Recent studies demonstrated the superior performance of models like RAG in various tasks compared with traditional methods and expert humans ( 52 – 54 ).
Basic framework for information retrieval with new documents.
The questions related to concrete finishing and placement were selected and the evaluation was conducted on the LLM configurations that have shown the need for performance improvement from the previous stage. The meticulous selection of models for this adaptation included small- and medium-sized LLMs, as their quantized versions can be run locally while maintaining acceptable inference performance. This assessment was ultimately limited to three different, widely used model types. Specifically, a single model is chosen from those sharing the same base architecture (e.g., Llama, Mistral, Phi). Both full and quantized versions of the selected models were considered if they showed potential for improvement. Additionally, models that failed to adhere to the response format requirement in 100% of their responses—showing zero accuracy—were excluded for further evaluation.
Results
During the evaluation experiments conducted to determine the models’ performance in the five topics of concrete pavement construction, we discovered varying accuracy among the different LLMs. Table 2 summarizes the results of this evaluation. Although some LLMs consistently showed high accuracy across evaluations for all five topics, others of similar size categories performed poorly.
Model Performance for Topic Multiple-Choice Questions (*Local Model, Q4 Quantization)
Figure 7 shows performance (left) and spread (right) per topic for small, medium, and large LLMs over all MCQs. As expected, there is a notable positive correlation between the model size and accuracy for each evaluated topic of concrete pavement construction knowledge. This correlation can be attributed to the fact that the larger the general-purpose model, the fewer its limitations, and the knowledge gaps decrease with the increase in the LLM’s size. This observation confirms the findings for general-purpose LLM scaling laws ( 55 ). It should be noted, however, that two out of four large-size LLM models are proprietary: GPT-3.5-Turbo and GPT-4o-mini.
LLM evaluation of generated questions with (left) all scores and (right) spread per model size.
Concerning model size and performance, three small and six medium-size models showed zero accuracy for all topics. On inference, these nine models failed to adhere to the required format across all responses, as specified with a one-shot example in the prompt from Figure 4. A notable exception among small-size models was Phi-3 3.8B IT*, which performed similarly to larger models. The performance of the medium-size models varied from very poor to approaching that of large-size models.
An open-source model, Llama 3 70B IT, exhibited performance similar to or exceeding that of the proprietary models; however, its large size requires execution on a supercomputer or cluster with multiple GPUs, making it very expensive to use. Two other large-size models of the Mixtral family, full (Mixtral 8x7B IT v0.1) and quantized (Mixtral 8x7B IT v0.1*), performed worse than some smaller models. Similarly to the nine low-performant models that failed to comply with format requirements, these models typically struggled to respond in a fixed-format manner.
Furthermore, questioning the LLMs on the “Subgrade Preparation” and “Finishing and Placement” resulted in a higher percentage of correct answers compared with other topics. The average accuracy for all LLMs was greater than 40%, and some large- and medium-size models, as well as Phi-3 3.8B IT*, showed an accuracy of 80% and higher. Answers to other topics were less accurate, as can be observed from Table 2 and Figure 7.
The small- and medium-size models that demonstrated zero accuracy across all topics were excluded from further domain adaptation. Phi-3 3.8B IT* was selected from the small-size models to evaluate improvement techniques, as it was the only small-size LLM that showed potential for higher performance. Among the medium-sized model families, the following were selected with similar criteria: Llama-based (Llama 3 8B IT*, Llama 3 8B IT, Llama 3 8B ChatQA-1.5, Hermes 2 θ Llama-3 8B, Hermes 2 Pro Llama-3 8B, LLaMA Pro 8B IT) and Mistral-based (Mistral 7B OpenOrca, Mistral 7B IT v0.3*, Mistral 7B IT v0.3, Zephyr 7B beta). From the Llama-based models, Llama 3 8B IT* and Llama 3 8B IT were selected, and from the Mistral-based models, Mistral 7B IT v0.3* and Mistral 7B IT v0.3 were chosen. These four medium-size models allowed for potential improvement and comparison between full and quantized versions of the same LLM.
As a result, the techniques for increasing the domain accuracy of the LLMs were evaluated using the following five models:
Phi-3 3.8B IT* (Quantized)
Llama 3 8B IT
Llama 3 8B IT* (Quantized)
Mistral 7B IT v0.3
Mistral 7B IT v0.3* (Quantized)
The quantized versions of the models have a relatively small weight file size and can be run locally on a laptop, demanding lower computational requirements. Although their performance yielded accuracy above average, it was significantly lower than that of large models such as GPT-3.5-Turbo and Llama 3 70B, indicating the need for improvement.
Seven additional documents were selected to be utilized in the RAG technique, with an emphasis on concrete placement ( 56 – 58 ), slipform paving vibration ( 59 ), guidelines and standards for constructing rigid pavements ( 60 ), best practices for constructing smooth concrete pavements ( 61 ), and best practices for achieving durable and high-quality concrete pavements ( 62 ). Although there is no limit on the specific number of documents to be used in RAG, the inclusion of these documents already shows a significant improvement in the selected models’ understanding of concrete pavement finishing and placement. Adding any more documents should only improve accuracy further.
Figure 8 shows that the use of improved prompting, the Retry technique, resulted in a modest increase in accuracy for the Llama 3 8B IT* model and Mistral 7B IT v0.3 model. These results further indicate no change in accuracy observed for Phi-3 3.8B IT*, Llama 3 8B IT, and Mistral 7B IT v0.3* models. However, using both Retry and RAG techniques improved the accuracy for all five models. Figure 8 also shows that when Retry and RAG were used, each model exhibited an accuracy greater than 75%. Moreover, the accuracy of Phi-3.8B IT* exceeded 90%, demonstrating performance comparable to GPT-4o-mini. Also, two models, Llama 3 8B IT* (84%), and Mistral 7B IT v0.3* (89%), showed better accuracy than GPT-3.5-Turbo.
Open-source large language models (LLMs), full and quantized, with retrieval augmented generation (RAG) improvement over five selected models.
Figure 9 presents a comparative analysis of the five selected models in terms of context handling. Each chart illustrates the proportion of the forty-four finishing and placement questions where the models were enhanced with RAG context and Retry. The results show the following conditions over the baseline evaluation: no improvement (cases 1 and 2), improvement with context (case 3), or degradation with context (case 4). Case 1 represents the condition where both modes resulted in the correct answer. Case 2 represents the opposing condition where both modes resulted in the incorrect answer. Case 3 shows moderate improvement with RAG over baseline, indicating that the injected context directed the model to select the correct answer. Case 4 finally illustrates where the injected context caused the model to select the incorrect answer.
Distribution of retrieval augmented generation improvement cases over the five selected models.
For example, Phi-3 3.8B IT* answered thirty-two out of forty-four questions correctly, both with and without RAG context. It answered one question incorrectly in both scenarios, showing no improvement with additional context. The remaining eleven questions fell into two categories: in eight cases, the context improved the model’s performance, whereas in three cases, the context degraded its ability to select the correct answer.
Finally, Figure 10 further illustrates the four cases described, along with examples of questions for each case for the Phi-3 3.8B IT* model. In case 1, the model showed some understanding of concrete hydration, with or without additional context. Case 2 demonstrates a gap in knowledge regarding concrete mixture requirements for slipform paving. The additional context provided to the model alluded to the fact that the concrete must be stiff enough to mitigate edge slump, yet fluid enough to consolidate under vibration to minimize entrapped air. Although this context provided relevant information, the model proceeded to generate an incorrect response, further showing some of the model’s limited reasoning capabilities.
Examples conveying no improvement, improvement, and degradation with retrieval augmented generation (RAG).
The remaining cases, case 3 and case 4, illustrate implicit results from the RAG model adaptation. More specifically, in case 3, the LLM showed improvement in understanding slipform paving by effectively utilizing context that aided in the model’s ability to acquire external knowledge related to staking in slipform paving. Conversely, case 4 demonstrates how the context that discusses related paving practices—such as handling pavement edges, checking tools, and finishing techniques—can mislead the model by shifting focus to inspection processes, smoothing techniques, and corrections during paving rather than directly addressing dry areas on forms. This shift in context alters the model’s probability of generating a correct answer, causing the model to incorrectly answer this question even though Phi-3 3.8B IT* initially generated the correct answer in the baseline evaluation.
Discussion
The three-stage approach presented in this study, namely automatic question generation, LLM evaluation, and the provision of additional context, reveals unique insights about the current status of this technology when applied to the niche domain of concrete pavement construction. The five topics identified for this study, ranging from subgrade preparation to surface smoothness, present a foundation for further exploration and adaptation of models. Although the automatic question generation method utilized in this study provided adequate and representative questions to assess each of the designated topics, there remains a need for high-quality documents and manual evaluation from experienced engineers for the subsequent tasks to be performed sufficiently.
Table 2 and Figure 7 provide an overview of the performance of the selected LLMs for this study. Although larger models, such as the proprietary GPT-3.5-Turbo and GPT-4o mini models, can achieve high accuracy over the prescribed topics, sometimes exceeding 90%, other open-source options are easily accessible. Not only are these larger models expensive in terms of computational and monetary requirements, but they also lack control and security. Engineers and state agencies should be aware of smaller open models, especially with their ability to be prompted on a local device, which contrasts with proprietary models concerning level of control and data privacy.
Figure 7 also shows a notable positive correlation between model size and accuracy for each evaluated topic. This correlation can be attributed to the fact that the larger the general-purpose model, the less limited its capabilities are, and the knowledge gaps decrease with the LLM’s size increase. Some medium and large models (e.g., Llama 2 7B IT and Mixtral 8x7B IT) performed poorly compared with smaller models, with zero or low accuracy for most of the topics. This difference is possibly the result of the models’ capabilities and limitations in answering questions, proper response formatting, and the lack or absence of domain data in the training corpus.
However, even with poorer performance, these smaller models can be improved, as shown through improved prompting strategies and the third stage in the devised approach. The one-shot prompting technique used throughout the study guides the models to produce responses in a specific format. This strategy does not require much creativity and can be expanded to multiple examples to enforce stricter adherence to a desired output. Furthermore, if an LLM still did not produce the desired output, the Retry strategy implemented in this study enables a more inclusive evaluation of the selected models. Applying this may improve a full open-source model’s average accuracy by as much as 16% and quantized by approximately 7%, as shown in Figure 8.
These findings further emphasize the need to adapt LLMs to the concrete pavement construction domain and to encourage practitioners to use other models. The RAG technique combined with Retry may improve a full open-source model’s average accuracies by as much as 29% and the quantized model’s average accuracies by 25%, also depicted in Figure 8. RAG presents itself as an attractive method to improve performance in domain-specific tasks, accompanied by many advantages, including the omission of further model tuning. It is highly adaptive, such that any document can be used to supplement factual context to LLMs. This adaptability is important for state agencies, where different regions have contradictory specifications. Furthermore, it is relatively easy to update the RAG model with new documents, accounting for updates and changes to best practices.
As only a narrow variety of new documents were selected for the RAG strategy in this study, the improvements in LLM accuracies may be limited. Figures 9 and 10 illustrate several cases where the injected RAG context elicited either no improvement, improvement, or degradation in answer correctness. Such degradation, in spite of RAG improving many answers, can probably be attributed to the low relevance, accuracy, or quality of the provided context for some of the questions, which led the model to generate an incorrect response. In turn, the decrease in context quality is likely caused by a low number of independent documents, calling for the addition of more documents for further improvement.
It’s important to note that the quality, accuracy, and ability of the evaluated models to generate desirable responses are directly linked to their weight files, which store the learned parameters that govern how information is processed. LLMs continuously evolve, and developers frequently release updates introducing enhancements or optimizations to the weights, full and/or quantized, even for the same model version. Ongoing updates can significantly affect model performance, potentially enhancing specific aspects like natural language understanding while risking a decline in other areas, such as factual knowledge in particular domains.
In particular, during the course of this study, updates had a significant impact on the performance of Phi-3 3.8B IT*. The initial version of this Phi model (id: 64c1188f2485; accessed on June 5, 2024), and the updated version of the model (id: 4f2222927938; accessed on October 5, 2024) were both downloaded from the same repository online ( 63 ). This specific difference between both versions, supposedly pointing to the same model, led to a decline in accuracy on the “Finishing and Placement” topic from 79.5% in the initial version to 38.6% in the updated version. With additional context from RAG, the updated Phi-3 3.8B IT* achieved an accuracy of 63.6%, compared with 90.9% in the initial version. This variability can be problematic if consistent accuracy and performance are required, signaling researchers and practitioners to take full control over the model’s weight files, which provide the required results. As this study had been running Phi-3 3.8B IT locally, preserving a duplicate copy of all model parameters, the results of the earlier, more proficient model are still reproducible.
Conclusions
This study evaluated the expertise of twenty-six state-of-the-art general-purpose LLMs in concrete pavement construction. It performed assessments of automatically generated MCQs in subgrade preparation, base preparation, joint sawing/sealing, finishing and placement, and surface smoothness. The evaluation results showed a variation in accuracy, typically related to the size of the models. Their understanding of this domain may be limited, confirming that out-of-the-box LLMs may not be readily available for civil engineering applications without additional improvements.
We found that the evaluated models don’t always provide reliable accuracy in answering domain-specific questions. However, we also demonstrated how open-source LLMs’ expertise can be improved with advanced techniques, such as RAG, enabling the injection of relevant knowledge from industry documents such as guidelines, best practices, and reports into generated responses. The improved LLMs, both full and quantized, quickly become aware of new or updated domain knowledge (e.g., newly issued regulations or updated best practices), excluding an expensive long training process for updating that could take up to several months. This allows the extension of LLMs to new, previously uncovered topics, as well as improves and deepens current knowledge of the existing expertise in the industry.
Through an unexpected update of the Phi-3 3.8B IT* model during the course of this study, we found that with continuous developer updates, maintaining control over a model’s weight files is crucial for ensuring consistent accuracy, reproducibility, and reliable performance. Preventing unexpected disruptions caused by ongoing updates is especially important in the field of concrete pavement construction where factual information must be maintained. Preserving models in their as-evaluated state safeguards against unexpected performance changes, helping to maintain reliable results over time and prevent further disruptions.
It is important to note that the question set generated for this study, although systematically sampled from key industry documents, was not fully exhaustive. Certain subtopics were represented by only a small number of questions, and a more comprehensive benchmarking effort would require the inclusion of additional documents and broader question sets to ensure deeper coverage. Although MCQs offer a structured and interpretable means of evaluation, they also introduce limitations. Multiple-choice formats can oversimplify complex knowledge and may not fully capture the depth of understanding required in practice. This is important for civil engineering applications, where open-ended problem solving and nuanced reasoning are critical. As LLM research advances, there is a growing need for novel benchmarks that move beyond multiple-choice to assess open-ended, problem-solving, and conversational interactions, especially for niche technical domains.
The performance of retrieval methods is inherently sensitive to the selection, structure, and preprocessing of source documents. However, RAG offers major advantages, including low cost, rapid adaptability, and the ability to improve model reliability without extensive fine-tuning. By retrieving targeted, high-quality context, this approach can effectively mitigate hallucinations and enhance model accuracy, even in specialized domains like rigid pavement construction. Factors such as the number of documents retrieved, text parsing and chunking techniques, and the relevance of the retrieved content all influence the effectiveness of RAG. Ongoing research is needed to refine these processes, identify best practices for document selection, and optimize the balance between document volume and model performance.
Our findings also confirm that secure AI systems that are aware of concrete pavement construction knowledge and allow users to remain in control of their data can be built. The AI systems, in turn, may be widely and safely used by industry practitioners to gain, validate, and improve concrete pavement construction knowledge to be used in applications such as employee training. This study can be further exploited to develop AI-based solutions (e.g., AI assistants or chatbots) based on open-source LLMs. These solutions offer practitioners confidential, secure, and fully controlled data usage while maintaining accuracy close to the proprietary LLMs that do not offer such control. Agencies or private companies can thus use these tools to make the onboarding process more efficient, improve the training processes for their personnel, and comply with the latest industry standards and practices.
Footnotes
Author Contributions
Mason Smetana: Writing – review & editing, Writing – original draft, Formal analysis, Investigation, Methodology, Conceptualization, Validation, Visualization, Software, Data curation. Igor Sukharev: Writing – review & editing, Writing – original draft, Formal analysis, Investigation, Methodology, Conceptualization, Validation, Visualization, Software, Data curation. Lev Khazanovich: Writing – review and editing, Methodology, Conceptualization, Validation, Formal analysis, Investigation, Data curation, Supervision. All authors reviewed the results and approved the final version of the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors would like to acknowledge the Anthony Gill Chair and Impactful Resilient Infrastructure Science and Engineering (IRISE) Consortium for funding this project. IRISE partners: Golden Triangle Construction, Pennsylvania Turnpike Commission, Allegheny County, Pennsylvania Department of Transportation, Michael Baker International, Constructors Association of Western Pennsylvania, CDR Maguire Corporation and the Southwest Planning Commission of Pennsylvania. This research was also supported in part by the University of Pittsburgh Center for Research Computing and Data, RRID:SCR_022735, through the resources provided. Specifically, this work used the H2P cluster, which is supported by NSF award number OAC-2117681.
Disclaimer: Large Language Model (LLM) Usage
OpenAI’s ChatGPT was used to assist proofreading and revision of the manuscript. The authors of this paper acknowledge the limitations of LLMs such as potential biases, errors, and gaps in knowledge, as documented in the results of the study.
