Advancing Small Business Inclusion in Public Procurement: Evidence From U.S. Federal Government R&D Contracts

Abstract

To encourage small business inclusion in public procurement, the U.S. federal government has established the set-aside program that mandates government agencies to award a portion of their contracts to small businesses. We study whether and to what extent the performance of R&D contracts awarded through this program differs from those awarded through open competition. Analyzing a large dataset of federal R&D contracts, we find that despite restricting competition to small businesses, set-aside R&D contracts experience lower schedule and cost overrun than R&D contracts awarded through open competition. Furthermore, although set-aside R&D contracts experience lower schedule and cost overrun when they are awarded to more experienced contractor firms, this benefit arises primarily from a contractor firm's experience in executing R&D contracts across different agencies compared to the firm's experience with the same agency. Finally, set-aside R&D contracts awarded early in a fiscal year experience lower schedule and cost overrun than those awarded later. Post-hoc analysis examining the underlying dimensions of different-agency experience highlights the asymmetric effects of related-agency experience and unrelated-agency experience of contractor firms on the performance of set-aside R&D contracts awarded by the Department of Defense. While related-agency experience improves contract performance, unrelated-agency experience has a detrimental effect on contract performance. These findings demonstrate that small business inclusion policies may not necessarily compromise contract performance. Importantly, they emphasize the need for federal agencies and contracting officers to consider the underlying dimensions of contractor firm experience and contract award timing to improve contract performance and taxpayer money utilization.

Keywords

Small Business R&D Contracts Public Procurement Preference Programs Inclusive Supply Chains

Small business set-asides are of perennial interest to Congress because of their role in effectuating the long-standing declared policy of the Congress that the Government should aid, counsel, assist, and protect, insofar as is possible, the interests of small-business concerns in order to preserve free competitive enterprise…. (Congressional Research Service, 2016, p. 1)

1. Introduction

Small businesses represent the lifeblood of economies globally. In the U.S., small businesses make up more than 99.9% of all businesses, account for approximately 44% of all economic activity, and are responsible for creating two-thirds of net new jobs (Forbes, 2022; Small Business Administration, 2019). Given the impact of small businesses, governments around the world have established preference programs that encourage the inclusion of small businesses in public procurement (Athey et al., 2013; Calvo et al., 2019). As a part of these programs, small businesses receive preferential treatment in the awarding of a specific set of public contracts (Chatterji et al., 2013). Nearly 43% of the countries worldwide have some form of legislative provision that encourages the inclusion of small businesses in public procurement (World Bank, 2016). For example, the U.S. federal government's set-aside program mandates that at least 23% of federal government prime contracts be awarded to small businesses. Similarly, under the Kankouju program in Japan, more than half of public contracts are reserved for small and medium businesses annually.¹

Notwithstanding the welfare intent of preference programs in public procurement, much debate exists in prior research about the ambiguity surrounding their performance implications and the “extra cost that society is paying” for such programs (Nakabayashi, 2013, p. 43). Studies have found that although preference programs facilitate the creation of new businesses (Chatterji et al., 2013) and increase their participation in public contracts (Nakabayashi, 2013), reserving such contracts for specific groups of businesses can decrease revenues (Hyndman and Parmeter, 2015) and increase the cost of providing the goods and services to the public (Marion, 2009). Beyond these mixed findings, an evaluation of preference programs at the contract level remains sparse in the extant literature, which limits our understanding of the factors in the contracting environment that may influence the performance of contracts awarded through preference programs.

In this study, we evaluate the impact of preference programs by comparing the performance of contracts that are awarded preferentially to small businesses through the set-aside program of the U.S. federal government with that of similar contracts awarded through open competition. The set-aside program represents a relevant context for evaluating the impact of preference programs at the contract level for the following reasons. On the one hand, awarding a contract through the set-aside program requires a federal government agency (hereinafter, federal agency) to conduct a search only within small businesses, reducing the likelihood of finding a highly qualified firm for executing the contract. This potential for adverse selection is higher for R&D contracts² (compared to routine service contracts or general construction contracts) that involve knowledge-intensive tasks and greater uncertainty in the execution process (Azoulay et al., 2019; Crama et al., 2017). On the other hand, by excluding large businesses from participation, the set-aside program attempts to level the playing field in terms of competition, thereby encouraging more small businesses to compete for set-aside contracts and improving the likelihood of finding a highly qualified firm for executing a contract (Athey et al., 2013). These tradeoffs associated with the set-aside program and the uncertain nature of tasks in R&D contracts provide a relevant setting to test the effect of preference programs on contract performance. Our study, therefore, has the following objectives.

First, we examine whether the performance of R&D contracts awarded through the set-aside program differs from those awarded through open competition and, if so, in what way. Second, we examine how characteristics of the contracting environment—namely, contractor firm experience (i.e., the number of R&D contracts executed by a contractor firm for the federal government) and contract award timing (i.e., awarded early in a federal fiscal year)—interact with the set-aside status of an R&D contract to affect its performance. Given that contractor firms can differ in terms of their experience with federal agencies, we partition contractor firm experience into two underlying dimensions—same-agency experience (i.e., the number of R&D contracts executed by a contractor firm for the same federal agency) and different-agency experience (i.e., the number of R&D contracts executed by a contractor firm across different federal agencies). We examine whether and to what extent do these underlying dimensions of contractor firm experience affect the relationship between a contract's set-aside status and its performance. The performance measures of interest are schedule overrun and cost overrun, metrics that capture the delay and additional cost incurred in executing an R&D contract, respectively. We test these relationships using detailed execution-level data on 30,902 R&D contracts executed for 49 different agencies of the U.S. federal government.

The analysis results, consistent across a range of model specifications including an instrumental variable approach and matching, indicate that set-aside R&D contracts experience significantly lower levels of schedule overrun and cost overrun compared to similar R&D contracts awarded through open competition. However, the performance of set-aside R&D contracts depends on both the nature of contractor firm experience and the timing of awarding the contract. Specifically, with regard to the effects of contractor firm experience, we find that although set-aside R&D contracts experience lower levels of schedule and cost overrun when they are executed by contractor firms with greater experience, such benefits are derived primarily from a contractor firm's different-agency experience. That is, we do not observe any benefits associated with increasing levels of a contractor firm's same-agency experience. Additional post-hoc analysis finds that for set-aside R&D contracts awarded by the Department of Defense (DoD), the performance benefits of different-agency experience arise from a contractor firm's related-agency experience with NASA only. However, a contractor firm's unrelated-agency experience with agencies other than DoD and NASA may result in poorer performance for set-aside R&D contracts awarded by DoD. These results highlight asymmetries in the effects of contractor firm experience and suggest that differences in terms of not only the extent of experience, but more importantly, the type of experience, has a significant effect on the performance of set-aside R&D contracts. With regard to the effect of contract award timing, we find that set-aside R&D contracts awarded early in a federal fiscal year (i.e., between October and December) experience lower levels of schedule and cost overrun than those awarded later in the fiscal year. This suggests that the timing of awarding an R&D contract represents an important factor that affects a federal agency's ability to identify an appropriate contactor firm, thereby affecting the performance of set-aside R&D contracts.

Together, these results make the following contributions toward advancing research and practice. First, by focusing on small business preference programs, our study highlights the challenges associated with managing policy initiatives to encourage the inclusion of small businesses in public procurement and responds directly to calls for research that has the potential to influence the inclusion of underrepresented entities in supply chains (Deshpande and Swaminathan, 2020; Kalkanci et al., 2019). Second, although prior research on small business preference programs has highlighted the welfare implications of these policy initiatives (Athey et al., 2013; Chatterji et al., 2013), the present study departs from this trend by examining the contract-level performance implications of small business preference programs. Our results provide preliminary empirical evidence suggesting that an important objective of the set-aside program—relating to promoting the inclusion of small businesses in federal contracts—is being fulfilled without negatively affecting the performance of contracts that are being awarded through this program. The third contribution lies in identifying two characteristics of the contracting environment—the nature of contractor firm experience and the timing of awarding an R&D contract—which need to be considered by federal agencies and contracting officers while awarding set-aside R&D contracts. Specifically, our study demonstrates that set-aside R&D contracts perform better when they are executed by contractor firms with higher levels of different-agency experience and when these contracts are awarded early in a fiscal year. In summary, beyond demonstrating that awarding contracts through the set-aside program may not necessarily deteriorate contract performance, our study provides insights that have the potential to further improve the performance of contracts that are awarded preferentially to small businesses.

2. Prior Literature

Our study builds upon prior work at the intersection of two streams of research: the emerging management literature on public procurement (e.g., Bruce et al., 2019; Calvo et al., 2019; Coviello et al., 2018; Rathje and Katila, 2021) and recent studies in public economics that have examined the use of preference programs in procurement initiatives (e.g., Athey et al., 2013; Chatterji et al., 2013; Nakabayashi, 2013). We highlight some of the key studies across these streams and discuss how our study builds upon prior work.

Within the management literature, Coviello et al. (2018) use data from public procurement auctions in Italy to examine how different auction formats (which affect the level of discretion exercised by Italian authorities) impact procurement outcomes (e.g., number of bidders, types of winners, value of winning bids) of construction contracts awarded by the Italian government. Focusing on R&D initiatives, Rathje and Katila (2021) examine whether partnering with the U.S. federal government helps contractor firms to develop novel technologies, whereas Bruce et al. (2019) examine the drivers of contractual form choices between grants and cooperative agreements which heterogeneously determine the level of oversight exercised by the federal government. Relatedly, Calvo et al. (2019) examine how the level of oversight exercised by the U.S. federal government impacts the performance of public infrastructure projects executed by a contractor firm. While this recent stream of work sets the foundation for our study's focus on public procurement, our study also differs from this stream of work in three key areas.

First, we examine a relatively understudied feature of public procurement—i.e., the use of small business preference programs—and its impact on contract performance, after controlling for heterogeneity in the level of oversight exercised by the government. With the U.S. federal government having a mandate to award at least 23% of its annual contracting dollars through the set-aside program, an examination of the performance differences between set-aside contracts and open competition contracts represents a consequential line of inquiry for both researchers and policymakers alike. The set-aside program's role in encouraging the inclusion of small businesses in the federal procurement process represents a topic of particular interest from a congressional and executive branch standpoint and is closely regulated by the U.S. Small Business Administration with each agency being held accountable for noncompliance (Congressional Research Service, 2016). That is, from the federal government standpoint, the set-aside program remains a legislation of significant consequence, requiring a careful investigation of its performance consequences.

Second, in contrast to Calvo et al. (2019), who examine the moderating effects of a contractor firm's overall experience on the relationship between a contract's oversight level and performance, we take an important step forward in partitioning contractor firm experience into same-agency experience and different-agency experience and studying how these underlying dimensions of experience moderate the relationship between a contract's set-aside status and performance. Specifically, while the assessment of a contractor firm's overall experience may provide federal agencies with valuable information to reduce adverse selection issues associated with awarding set-aside R&D contracts, contractor firms may differ from one another in terms of the extent to which their experience is accumulated from working with a focal federal agency or with different federal agencies. The performance implications of such differences in contractor firm experience remain unclear, requiring greater attention.

Finally, we also distinguish our study from Calvo et al. (2019) by examining how the timing of awarding a contract moderates the relationship between the contract's set-aside status and performance. The federal government's fiscal year is a 12-month period that starts on October 1 of a calendar year and ends on September 30 of the next calendar year.³ While the start of a fiscal year typically involves fresh budgetary allocation and less pressure to meet end-of-fiscal-year targets, there is a substantive increase in the volume of tasks that federal agencies carry out as the fiscal year progresses (Congressional Research Service, 2008). As differences in the volume of tasks over a fiscal year may impact a federal agency's ability to identify the most qualified contractor firm (Boland and Godsell, 2021; Warren, 2014), we study how the timing of awarding an R&D contract interacts with its set-aside status to impact contract performance.

Beyond the management literature, prior research on preference programs in the public economics stream has reported mixed findings regarding their impact. For example, Blanchflower and Wainwright (2005) find that preference programs result in an increase in the value of contracts awarded to underrepresented small businesses, while Chatterji et al. (2013) find that preference programs facilitate the creation of new businesses. Similarly, Nakabayashi (2013) finds that participation of small and medium-sized businesses in public contracts would be reduced by at least 40% without preference programs. Notwithstanding such welfare benefits, Athey et al. (2013) find that the restricted nature of competition for public contracts under preference programs reduces auction revenues for government agencies. Relatedly, Marion (2009) finds that, by restricting competition, preference programs increase government expenditures for constructing highways, as they raise the prices at which highway construction contracts are awarded to contractor firms. Collectively, the prior research on preference programs suggests that although preference programs may enable governments to include more small businesses in executing public contracts, the efficiency implications of such programs remain debatable.

Further, these studies provide limited insights into the impact of such programs at the more granular contract level of the procurement process. A focus on the contract level is important as it can reduce the extent of unobserved heterogeneity impacting a transaction and enable a more precise assessment of the transaction performance. We address this gap by examining the performance differences across R&D contracts awarded through the set-aside program vs. those awarded through open competition. Furthermore, a better understanding of the impact of preference programs on contract performance can help to identify contingencies relating to the contracting environment that can affect contract execution and enable different stakeholders (e.g., contractor firms, federal agencies) to take appropriate actions to effectively manage contract performance. To that end, we study how the nature of contractor firm experience and the timing of awarding an R&D contract impact the performance of set-aside R&D contracts.

3. Hypotheses Development

3.1. Background: The Set-Aside Program

As noted earlier, the set-aside program mandates that each year at least 23% of the total value of all federal contracts should be preferentially awarded to firms that qualify as small businesses, as per the Small Business Administration's regulations.⁴ In terms of total dollar value, the set-aside program constitutes a substantial portion of the U.S. federal government's procurement expenditure, with close to $100 billion of contracts awarded through this program annually in recent years.⁵ The specific details on how contracts are to be awarded through the set-aside program are listed in the federal acquisition regulation (FAR), a 2320-page contracting handbook that provides detailed information on contracting rules and regulations for federal agencies to follow in the procurement process. In particular, the planned cost of a federal contract is an important determinant of whether the contract will be awarded through the set-aside program. As illustrated in Figure 1, while a federal contract with a planned cost between $3500 and $150,000 is automatically set aside for small businesses, contracts with a planned cost above $150,000 are set aside for small businesses only if there is an expectation for the Rule of Two to be met—that is, if at least two or more small businesses that are competitive in terms of market prices, quality, and delivery should be expected to submit an offer.⁶

Figure 1.

The process of awarding set-aside contracts by the U.S. Federal Government.

Additionally, if a contract with a planned cost between $3500 and $150,000 is set aside for small businesses and only one offer is received, the contracting officer must make an award to that firm. However, there are some exceptions. First, if contracts with a planned cost between $3500 and $150,000 are not set aside for small businesses, the contracting officer must officially record why such a decision was made. Second, if no offers are received for a set-aside contract, the contracting officer solicits offers through open competition. Given these exceptions, perfect compliance with the $150,000 threshold is typically not observed.

3.2. Implications for Contract Performance

We first consider whether and how the performance of set-aside R&D contracts differs from those awarded through open competition. On the one hand, for a set-aside contract, the mandate to exclude large businesses can limit the pool of eligible bidder firms, thereby reducing the breadth of search that can be carried out by contracting officers to identify the most qualified contractor firm to execute the contract. This limitation in the breadth of search can lower the likelihood of selecting a highly qualified contractor firm to execute a set-aside contract, thereby negatively impacting contract performance. That is, in comparison to a scenario wherein all possibilities can be evaluated, a limitation in the breadth of search may result in suboptimal performance outcomes (Gavetti et al., 2005; Levinthal, 1997). With regard to the selection of a contractor firm, as long as the entire pool of firms (i.e., both large firms and small firms) is not searched, there might always exist a possibility of selecting a contractor firm that is not the most qualified, thereby negatively impacting contract performance. Moreover, selecting the most qualified contractor firm from a limited pool of participating small businesses may be particularly difficult for R&D contracts as these contracts involve highly knowledge-intensive tasks and entail a greater level of uncertainty in the execution process (Crama et al., 2017). In sum, the limited breadth of search for set-aside contracts and the knowledge-intensive and uncertain nature of tasks in R&D contracts may lower the likelihood of selecting a highly qualified contractor firm for set-aside R&D contracts, negatively impacting contract performance.

On the other hand, by excluding large businesses from competition and providing a “level playing field” for small businesses to participate in public procurement, the set-aside program increases incentives for small businesses to compete for set-aside contracts. This, in turn, can encourage more small businesses to compete for set-aside contracts as such businesses now anticipate more homogenous competition and foresee a higher likelihood of winning ex-ante. To that end, prior research has found that limiting competition to small and medium-sized businesses increases the participation of these business types in public contracts and auctions (Athey et al., 2013; Hyndman and Parmeter, 2015; Nakabayashi, 2013). For example, Athey et al. (2013) find that limited competition increases small business participation such that there is a greater overall number of participants in set-aside auctions. To be successful in such a competitive marketplace for set-aside contracts, small businesses may have to push themselves harder and improve their capabilities (Ethiraj et al., 2005), which in turn may translate into better performance outcomes for set-aside contracts. Our conversations with federal contracting officers highlighted that although small businesses do not compete against large businesses for set-aside contracts, they still need to outperform several other small businesses that are aiming for the same set-aside contract. That is, the potential for increased participation by small businesses for set-aside contracts may provide federal agencies not only with more choices to select a highly qualified contractor firm, but also compel small businesses to enhance their capabilities, thereby improving contract performance. Taking the above arguments into consideration, we propose the following set of competing hypotheses regarding the performance of set-aside R&D contracts and open-competition R&D contracts.

Hypothesis 1A: R&D contracts awarded through the set-aside program have lower performance than those awarded through open competition.

Hypothesis 1B: R&D contracts awarded through the set-aside program have higher performance than those awarded through open competition.

3.3. The Role of Contractor Firm Experience

The overall experience of a contractor firm comprises two underlying dimensions: same-agency experience, which arises from executing R&D contracts for the same federal agency as the focal contract, and different-agency experience, which arises from executing R&D contracts for a different federal agency relative to the focal contract. The performance benefits arising from increasing the overall experience of a contractor firm, as documented in prior research (e.g., Alcacer and Oxley, 2014; Clark et al., 2013), suggest that both dimensions of contractor firm experience should contribute to better performance outcomes for set-aside contracts. However, due to specific mechanisms through which these impacts occur, the impact of same-agency experience and different-agency experience on the performance of set-aside contracts could vary.

When a set-aside contract is awarded to a contractor firm with increasing levels of same-agency experience, the contractor firm's familiarity and influence (Huckman and Pisano, 2006) over the focal agency may result in improved contract performance. First, under restricted competition (as mandated by the set-aside program), awarding a set-aside contract to a contractor firm with increasing levels of same-agency experience may improve contract performance because of the contractor firm's increased familiarity over the focal agency's resources (e.g., technological assets, project managers, etc.) and an accumulation of agency-specific knowledge. The specific regulations to successfully execute federal contracts vary across federal agencies, for example, contracting regulations for the Department of Defense (i.e., Defense Acquisition Regulation System)⁷ are substantively different from those for the Department of Health and Human Services (i.e., Health and Human Services Acquisition Regulation).⁸ Within a limited pool of bidder firms, a contractor firm with greater (vs. lower) levels of same-agency experience is more likely to accumulate knowledge about agency-specific contracting regulations, contributing to the improved performance of set-aside contracts. Second, the cultivation of relationships with key agency officials (for example, small business specialists, and acquisition officers) represents an important success factor in federal contracting and increasing levels of same-agency experience can help contractor firms to develop stronger connections with key officials within the focal agency (GovCon Chamber of Commerce, 2021; Goodloe, 2018). Within a limited pool of bidder firms, a contractor firm with greater (vs. lower) levels of same-agency experience is more likely to be able to leverage these agency-specific relationships and influence the focal agency about a set-aside contract's business and technical requirements, thereby executing the contract in a manner that is consistent with its requirements and performance expectations. Collectively, the above arguments suggest that, although the set-aside program may restrict the breadth of search for the most qualified contractor firm, awarding set-aside contracts to contractor firms with increasing levels of same-agency experience may result in improved contract execution due to the contractor firm's familiarity and influence over the focal agency's resources.

A contractor firm's different-agency experience may also help to improve the execution of set-aside contracts, albeit through a different set of mechanisms in comparison to a contractor firm's same-agency experience. First, when the pool of eligible bidder firms is restricted, awarding a set-aside contract to a contractor firm with increasing levels of different-agency experience may improve contract performance due to the contractor firm's ability to accelerate its learning process by combining knowledge from multiple sources (Schilling et al., 2003) and developing a more holistic understanding about executing R&D contracts. For instance, an infrastructure R&D contract from the Department of Transportation (DoT)⁹ may also need to adhere to environmental sustainability standards established by the Environmental Protection Agency (EPA). A contractor firm that has prior experience with the EPA can be expected to have a better understanding of such sustainability standards, resulting in a more holistic understanding of contract requirements and improved performance of set-aside R&D contracts awarded by the DoT. Second, within a limited pool of bidder firms, a contractor firm with greater (vs. lower) levels of different-agency experience is more likely to be able to identify and transfer best practices (Huckman and Staats, 2011) from other federal agencies to improve the performance of the set-aside contract awarded to it by the focal agency. Such opportunities to identify and transfer best practices are particularly relevant for successfully executing R&D contracts because producing novel solutions in a timely and cost-effective manner often requires borrowing ideas from disparate areas (in this case, from different agencies) rather than through an intensive focus on a single area (in this case, on the same agency). As an illustrative example, both the Department of Veteran Affairs (VA) and the Department of Homeland Security (DHS) rely on product prototype demonstrations to evaluate supplier capabilities (Office of Federal Procurement Policy, 2019, p. 12). Since both these agencies utilize product prototype demonstrations to evaluate supplier capabilities, contractor firms who have collaborated with the VA gain a deeper understanding of the intricacies involved in showcasing product prototypes effectively. This familiarity helps them to tailor their demonstration strategies more adeptly to meet the specific requirements of DHS contracts, ultimately leading to improved contract performance. In sum, although the set-aside program may restrict the breadth of search for the most qualified contractor firm, a contractor firm with greater levels of different-agency experience may have technical knowledge relevant to contract execution from a variety of sources, resulting in a faster rate of learning (Clark et al., 2013) and improved contract performance. Given these distinct yet beneficial impacts of a contractor firm's same-agency and different-agency experience, we propose the following hypotheses:

Hypothesis 2A: R&D contracts awarded through the set-aside program have higher performance as contractor firms’ same-agency experience increases.

Hypothesis 2B: R&D contracts awarded through the set-aside program have higher performance as contractor firms’ different-agency experience increases.

3.4. The Role of Contract Award Timing

As noted earlier, the federal government's fiscal year is a 12-month period which starts on October 1 of a calendar year and ends on September 30 of the next calendar year. At the beginning of a fiscal year, federal agencies start with fresh budgetary allocations and are at the early stages of implementing the current fiscal year's budget. However, as the fiscal year progresses, federal agencies face increased pressures to meet end-of-fiscal year deadlines, are tasked with obtaining Congressional funds for the next fiscal year, and start budgetary planning for the fiscal year after the next (Haughey, 2020; Congressional Research Service, 2008). In Figure 2, we provide an overview of the activities performed by federal agencies over a fiscal year.

Figure 2.

Timeline of activities carried out by federal agencies over a fiscal year.

In sum, at the start of a fiscal year, contracting personnel in federal agencies, particularly contracting officers, are less constrained for time as they have a lower volume of tasks to focus on (GAO, 2017). As a result, during this period of the fiscal year, contracting officers may have more time to evaluate individual offers carefully, provide greater clarifications on a contract's terms and requirements, and reduce potential adverse selection challenges associated with awarding set-aside contracts. In this regard, Warren (2014, p.395–396) notes that, while there has been a significant uptick in federal spending on procurement contracts (including R&D contracts) over the past decade, what has remained relatively unchanged is the number of contracting personnel who oversee a federal agency's contracting efforts. Thus, as the contracting environment becomes increasingly busy as the federal fiscal year progresses, contracting officers who inherently face significant constraints on their time are likely to spend lesser time in writing contracts, examining various contracting contingencies, or negotiate contract parameters. This additional time and effort may be particularly useful in identifying a highly qualified contractor firm for executing R&D contracts as these contracts are characterized by requirements that are difficult to understand (compared to routine service contracts or general construction contracts) and involve knowledge-intensive tasks (Giuffrida and Raiteri, 2021; Crama et al., 2017).

In addition, given that a federal agency's budget rules are distinct from its procurement rules, the uncertainty that typically prevails within an agency at the beginning of the federal fiscal year regarding the budget that will be available for contract spending limits discretion on the part of the contracting officers in the early part of the fiscal year as opposed to the later part of the fiscal year (Boland and Godsell, 2021; GAO, 2017). As a result, while the set-aside program may limit the pool of bidder firms to small businesses, lesser constraints on time and greater uncertainty regarding agency's budget at the start of a fiscal year is likely to motivate contracting officers to invest greater effort and increase the depth of their search within this limited pool of small businesses to identify the most qualified contractor firm for executing a set-aside contract. Taking these arguments into consideration, we propose the following hypothesis regarding the performance of set-aside R&D contracts awarded early in a fiscal year.

Hypothesis 3: Set-Aside R&D contracts awarded early in the fiscal year have higher performance than those awarded later in the fiscal year.

4. Research Design

4.1. Data

We collected detailed data on federal R&D contracts from USAspending.gov, the official source of U.S. federal government spending data, to test our hypotheses. This website was developed as an outcome of a congressional legislation, the Federal Funding Accountability and Transparency Act (FFATA) of 2006, which requires every federal contract, grant, loan, or other financial assistance award to be displayed on a publicly accessible and searchable website. The mission of this website is to “show the American public what the federal government spends every year and how it spends the money.”¹⁰ The website contains data on all R&D contracts awarded by the U.S. federal government from October 2007 to the current date.

Contract-level data on this website are organized into two distinct streams: award-level data and transaction-level data. Award-level data contain information on a contract's baseline characteristics (e.g., a contract's planned cost, planned completion date, award date, set-aside status), while transaction-level data contain information on any modifications made to a contract's baseline characteristics (e.g., changes made to a contract's planned cost or planned completion date). The transaction-level data also provide information on the actual dollar amount spent to execute a contract. We link the award-level data of each contract with its transaction-level data using the Procurement Instrument Identifier (PIID)—a unique alphanumeric code associated with federal contracts—to create an integrated dataset that contains the key variables used in our analysis. Although the website contains data on all R&D contracts awarded by the U.S. federal government from October 2007 onwards, the threshold value to determine set-aside status was increased from $100,000 to $150,000 in October 2010. Our study sample therefore comprises R&D contracts awarded by the federal government from October 2010 onwards. Specifically, we collected award- and transaction-level data on 46,763 R&D contracts awarded between October 2010 and March 2018 to construct our dataset. From this dataset, we excluded 50 contracts that belonged to a “bundled contract” package, because the performance of such contracts will depend on how other contracts in the bundle perform. Of the remaining 46,713 contracts, we restricted our sample to R&D contracts with a value between $3500 and $700,000, since contracts that do not fall within this range are subject to different mandates. We also excluded contracts which represented a one-time purchase or maintenance of R&D equipment. Therefore, our main analysis is based on a sample of 30,902 R&D contracts executed by 9927 contractor firms for 49 agencies of the federal government. Overall, nearly 57% of the R&D contracts in this sample were set-aside contracts and, on average, a contractor firm had been awarded 46 R&D contracts by the federal government in the past.

4.2. Dependent and Independent Variables

4.2.1 Schedule Overrun and Cost Overrun

The dependent variables of interest in this study are the schedule overrun and cost overrun for each R&D contract in the sample. To determine the schedule overrun and cost overrun measures for a contract, we first retrieved the planned estimates of contract duration and contract cost from the award-level data associated with each contract. Next, we obtain information on the actual duration and the actual cost of each contract from the transaction-level data to compute percent schedule overrun [(actual contract duration – planned contract duration)/planned contract duration] and percent cost overrun [(actual contract cost – planned contract cost)/planned contract cost]. While schedule overrun measures the delay experienced by a contract relative to its planned contract duration, cost overrun measures the additional cost incurred in executing a contract relative to its planned cost. Several OM studies have used similar metrics to measure the performance of federal contracts (Calvo et al., 2019; Coviello et al., 2018; Mishra et al., 2016).

4.2.2 Set-Aside

The award-level data in our sample provide information on whether an R&D contract was awarded through the set-aside program or through open competition. We construct a dummy variable, SA_i, representing the treatment status of a contract, which is coded as “1” for a set-aside R&D contract and “0” for an open competition R&D contract.

4.2.3 Same-Agency Experience

We use the Data Universal Numbering System (DUNS)—a unique identifier for each contractor firm—to count the number of R&D contracts awarded to a contractor firm and construct experience measures for each contractor firm in our sample. The same-agency experience of a contractor firm is measured as the total number of R&D contracts executed by a contractor firm in the past for the same federal agency as the focal contract.

4.2.4 Different-Agency Experience

The different-agency experience of a contractor firm is measured as the total number of R&D contracts executed by a contractor firm in the past for a different federal agency relative to the focal contract.

4.2.5 Early in Fiscal Year

As noted earlier, the federal fiscal year begins on October 1 of each calendar year and ends on September 30 of the next calendar year. For example, the 2013 federal fiscal year begins on October 1 of the 2012 calendar year and ends on September 30 of the 2013 calendar year. Early in Fiscal Year represents the quarter of a federal fiscal year in which an R&D contract was awarded. Specifically, it is a dummy variable coded as “1” if the contract was awarded in the first quarter of a federal fiscal year (October to December), and as “0” otherwise. About 16% of the contracts in our sample were awarded early in a fiscal year i.e., in the months of October to December.

4.3. Control Variables

Variations in the characteristics of R&D contracts and contractor firms executing the contracts can offer alternative explanations for performance differences. We therefore control for several such characteristics in our analysis. Table 1 describes the control variables¹¹ along with the dependent and independent variables, and Table A2 in the online appendix provides the pairwise correlations and descriptive statistics. Furthermore, Table A3 in the online appendix provides descriptive statistics of the contracts above and below the $150,000 threshold.

Table 1.
Description of dependent, independent, and control variables.

Dependent variable Variable description

Schedule Overrun The delay experienced by a contract relative to its planned duration.

Cost Overrun The additional cost incurred in executing a contract relative to its planned cost.

Independent variables

SA_i A dummy variable which is coded as “1” if an R&D contract was awarded through the set-aside program and “0” otherwise.

Same-Agency Experience The number of R&D contracts executed by a contractor firm in the past for the same federal agency as the focal contract.

Different-Agency Experience The number of R&D contracts executed by a contractor firm in the past for a different federal agency relative to the focal contract.

Early in Fiscal Year A dummy variable which is coded as “1” if an R&D contract is awarded in the first quarter of a federal fiscal year (October to December) and “0” otherwise.

Control variables

Task Type A dummy variable which is coded as “1” if an R&D contract is a development contract and “0” if it is a research contract.

Planned Contract Duration¹⁶ The difference between the initial completion date and start date of an R&D contract in number of days.

Planned Contract Cost The initial cost of the contract in USD.

Contractor Firm Size The average annual revenues of a contractor firm (in million USD) in the past three years (from the award year of an R&D contract).

Contract Pricing A dummy variable which is coded as “1” when contract pricing is time and materials (T&M) and “0” when contract pricing is fixed price (FP).

Competition Number of offers received for a contract in the contractor selection stage.

Oversight Level Contracts using simplified acquisition procedure (SAP) have low oversight levels, while contracts not using SAP have high oversight levels (FAR Part 13). We construct a dummy variable coded as “0” when SAP is used (low oversight) and “1” when SAP is not used (high oversight).

Small Business A dummy variable which is coded as “1” when the contractor firm is a small business and “0” otherwise.

Federal Agency The federal agency awarding the R&D contract. We created 48 dummy variables to represent the federal agencies awarding the R&D contracts.

Region of Contract Execution Denotes the geographical region (Northeast, Midwest, South, West and Outside Mainland US) in which the R&D contract is executed. We created four dummy variables to represent the region of contract execution in our analysis.

Contract Award Year The year in which an R&D contract was awarded. We created seven dummy variables to represent the different award years of the contracts in our sample (2010, 2011, 2012, 2013, 2014, 2015, 2016 and 2017).

R&D Contract Code A code representing the precise nature of R&D contract being executed that captures the area of R&D (e.g., agriculture, energy, medical) and the stage of R&D (e.g., basic research, applied research, engineering development). We created 433 dummy variables to represent the precise nature of R&D contracts in our sample.

Contractor Firm Cluster We created 573 dummy variables to represent the homogenous firm clusters (based on contractor firm size, headquarter location, ownership type, organization type, and profit motive) in our sample.

Dependent variable	Variable description
Schedule Overrun	The delay experienced by a contract relative to its planned duration.
Cost Overrun	The additional cost incurred in executing a contract relative to its planned cost.
Independent variables
SA_i	A dummy variable which is coded as “1” if an R&D contract was awarded through the set-aside program and “0” otherwise.
Same-Agency Experience	The number of R&D contracts executed by a contractor firm in the past for the same federal agency as the focal contract.
Different-Agency Experience	The number of R&D contracts executed by a contractor firm in the past for a different federal agency relative to the focal contract.
Early in Fiscal Year	A dummy variable which is coded as “1” if an R&D contract is awarded in the first quarter of a federal fiscal year (October to December) and “0” otherwise.
Control variables
Task Type	A dummy variable which is coded as “1” if an R&D contract is a development contract and “0” if it is a research contract.
Planned Contract Duration¹⁶	The difference between the initial completion date and start date of an R&D contract in number of days.
Planned Contract Cost	The initial cost of the contract in USD.
Contractor Firm Size	The average annual revenues of a contractor firm (in million USD) in the past three years (from the award year of an R&D contract).
Contract Pricing	A dummy variable which is coded as “1” when contract pricing is time and materials (T&M) and “0” when contract pricing is fixed price (FP).
Competition	Number of offers received for a contract in the contractor selection stage.
Oversight Level	Contracts using simplified acquisition procedure (SAP) have low oversight levels, while contracts not using SAP have high oversight levels (FAR Part 13). We construct a dummy variable coded as “0” when SAP is used (low oversight) and “1” when SAP is not used (high oversight).
Small Business	A dummy variable which is coded as “1” when the contractor firm is a small business and “0” otherwise.
Federal Agency	The federal agency awarding the R&D contract. We created 48 dummy variables to represent the federal agencies awarding the R&D contracts.
Region of Contract Execution	Denotes the geographical region (Northeast, Midwest, South, West and Outside Mainland US) in which the R&D contract is executed. We created four dummy variables to represent the region of contract execution in our analysis.
Contract Award Year	The year in which an R&D contract was awarded. We created seven dummy variables to represent the different award years of the contracts in our sample (2010, 2011, 2012, 2013, 2014, 2015, 2016 and 2017).
R&D Contract Code	A code representing the precise nature of R&D contract being executed that captures the area of R&D (e.g., agriculture, energy, medical) and the stage of R&D (e.g., basic research, applied research, engineering development). We created 433 dummy variables to represent the precise nature of R&D contracts in our sample.
Contractor Firm Cluster	We created 573 dummy variables to represent the homogenous firm clusters (based on contractor firm size, headquarter location, ownership type, organization type, and profit motive) in our sample.

4.4. Model Specification

Our base model to test the relationship between a contract's set-aside status and contract performance employs an instrumental variable two-stage least squares (IV-2SLS) approach with a rich set of control variables. While the inclusion of control variables helps to reduce the possibility of alternative explanations in our analysis from observed factors, the IV-2SLS estimation helps to reduce concerns of unobserved factors impacting the results. The IV-2SLS approach requires that the instruments used in the first stage model to predict a contract's set-aside status meet the following conditions, instrument relevance and instrument exogeneity (Angrist and Pischke, 2009). We define the instrumental variables below along with conceptual and empirical justification for their validity:

4.4.1 Same-Year Prior Set-Aside Contracts

This variable represents the total value of set-aside contracts awarded by an agency in a given year prior to awarding the focal contract. Each agency has an annual target for awarding contracts through the set-aside program—therefore, agencies that have awarded a lower value of set-aside contracts in a year (prior to awarding the focal contract) are more likely to award the focal contract through the set-aside program. However, this variable, capturing the total value of set-aside contracts awarded prior to the focal contract is not a characteristic of the focal contract and is unlikely to directly affect the performance of the focal contract, except for its influence on whether the focal contract is awarded through the set-aside program.

4.4.2 Below Threshold

This variable is coded as 1 if a contract's planned cost is below the $150,000 threshold and 0 if otherwise. As noted earlier, contracts with a planned cost less than or equal to $150,000 are more likely to be awarded through the set-aside program, while contracts with a planned cost of more than $150,000 are less likely to be awarded through the set-aside program (FAR Part 19). However, this variable would not directly affect the performance of the focal contract, except for its influence on whether the focal contract is awarded through the set-aside program. Prior research on public procurement has used similar thresholds to instrument for a contract's characteristics (Calvo et al., 2019; Coviello et al., 2018).

In addition, we perform statistical tests to check for the validity of the chosen instruments. The Lagrange Multiplier test for under-identification is statistically significant (Kleibergen-Paap rk statistic = 10.02, p ≤ 0.01) indicating the strong predictive power of the excluded instruments for a contract's set-aside status. Furthermore, in the first stage, the F-test of joint significance of the instruments (F-statistic = 12.20, p ≤ 0.01) is greater than the threshold value of 10 (Angrist and Pischke, 2009), indicating that the instruments are unlikely to be weakly correlated with a contract's set-aside status. Finally, the over-identification test continues to demonstrate statistical insignificance (Hansen J-statistic = 0.10, p = 0.76 for schedule overrun; Hansen J-statistic = 0.21, p = 0.64 for cost overrun) indicating the presence of at least one instrumental variable that satisfies the exogeneity assumption.

For a contract i, the first stage model of the IV-2SLS approach predicting its set-aside status is specified as follows:

S A_{i} = α + β Z_{i} + γ X_{i} + v_{i}

(1)

where SA_i is the set-aside status, Z_i represents the instrumental variables and X_i represents the control variables. Table A4 in the online appendix presents results from the first stage model estimating a contract's set-aside status. The predicted value,

{\hat{S A}}_{i}

from (1) is used in the second stage along with the control variables,

X_{i}

to estimate the dependent variables. The second-stage model is specified as follows:

Y_{i} = α + ρ {\hat{S A}}_{i} + γ X_{i} + ε_{i}

(2)

where ρ captures the treatment effect. To estimate the moderating effects of same-agency experience, different-agency experience and early in fiscal year, we use Z_i× Same-Agency Experience, Z_i× Different-Agency Experience and Z_i× Early in Fiscal Year as excluded instruments for SA_i× Same-Agency Experience, SA_i× Different-Agency Experience and SA_i× Early in Fiscal Year respectively.¹²

5. Analysis and Results

5.1. Main Analysis

Table 2 presents the results from equation (2) in a hierarchical manner separately for schedule overrun and cost overrun as the dependent variables. We use columns 1 and 5 to interpret the main effect of ${\hat{S A}}_{i}$ , and columns 4 and 8 to interpret the moderating effects for schedule overrun and cost overrun, respectively.

Table 2.
Main analysis: Effect of set-aside status on contract performance and moderating effects.

Dependent variable: schedule overrun Dependent variable: cost overrun

(1) (2) (3) (4) (5) (6) (7) (8)

Main effects

${\hat{SA}}_{i}$ −0.340 (0.083)* −0.362 (0.085)* −0.326 (0.082)* −0.345 (0.085)* −0.089 (0.016)* −0.094 (0.016)* −0.088 (0.016)* −0.092 (0.016)*

Same-Agency Exp. 0.005 (0.004) −0.009 (0.008) 0.005 (0.004) −0.013 (0.010) 0.000 (0.001) −0.003 (0.002)* 0.000 (0.001) −0.003 (0.002)

Different-Agency Exp. −0.002 (0.002) 0.013 (0.011) −0.002 (0.002) 0.014 (0.012) 0.000 (0.000) 0.004 (0.003) 0.000 (0.000) 0.004 (0.003)

Early in Fiscal Year −0.001 (0.010) −0.001 (0.010) 0.083 (0.034) 0.086 (0.034) 0.002 (0.002) 0.002 (0.002) 0.014 (0.007) 0.014 (0.007)**

Moderating effects

${\hat{SA}}_{i} \times$ Same-Agency Exp. 0.017 (0.018) 0.028 (0.019) 0.003 (0.003) 0.005 (0.003)

${\hat{SA}}_{i} \times$ Different-Agency Exp. −0.090 (0.048)* −0.090 (0.049)* −0.023 (0.011)** −0.022 (0.011)*

${\hat{SA}}_{i} \times$ Early in Fiscal Year −0.142 (0.048)* −0.148 (0.049)* −0.020 (0.010) −0.021 (0.010)

Control variables

Task Type (0: Res, 1: Dev) 0.322 (0.188)* 0.321 (0.189)* 0.325 (0.189)* 0.326 (0.191)* 0.055 (0.043) 0.055 (0.043) 0.056 (0.043) 0.056 (0.043)

Planned Contract Duration −0.116 (0.010)* −0.115 (0.010)* −0.116 (0.010)* −0.116 (0.010)* 0.004 (0.001)* 0.004 (0.001)* 0.004 (0.001)* 0.004 (0.001)*

Planned Contract Cost 0.074 (0.011)* 0.073 (0.011)* 0.074 (0.011)* 0.073 (0.011)* 0.005 (0.003) 0.005 (0.003) 0.005 (0.003) 0.005 (0.003)

Contractor Firm Size −0.007 (0.018) −0.011 (0.018) −0.007 (0.018) −0.010 (0.018) 0.004 (0.003) 0.003 (0.003) 0.004 (0.003) 0.003 (0.003)

Contract Pricing 0.103 (0.032)* 0.101 (0.032)* 0.100 (0.032)* 0.099 (0.032)* 0.031 (0.007)* 0.031 (0.008)* 0.030 (0.008)* 0.030 (0.008)*

Competition −0.011 (0.004)* −0.010 (0.004)* −0.011 (0.004)* −0.010 (0.004) −0.004 (0.001)* −0.004 (0.001)* −0.004 (0.001)* −0.004 (0.001)*

Oversight 0.073 (0.018)* 0.077 (0.019)* 0.075 (0.018)* 0.079 (0.019)* 0.007 (0.003)* 0.008 (0.003) 0.007 (0.003) 0.008 (0.003)

Small Business 0.148 (0.042)* 0.160 (0.043)* 0.152 (0.041)* 0.163 (0.042)* 0.034 (0.009)* 0.037 (0.009)* 0.035 (0.009)* 0.037 (0.009)***

Federal Agency FE Yes Yes Yes Yes Yes Yes Yes Yes

Region FE Yes Yes Yes Yes Yes Yes Yes Yes

Year FE Yes Yes Yes Yes Yes Yes Yes Yes

R&D Contract Code FE Yes Yes Yes Yes Yes Yes Yes Yes

Contractor Firm Cluster FE Yes Yes Yes Yes Yes Yes Yes Yes

Constant −0.372 (0.212)* −0.396 (0.215)* −0.408 (0.219)* −0.431 (0.223)* −0.076 (0.041)* −0.082 (0.041) −0.082 (0.041) −0.087 (0.042)**

N 30902 30902 30902 30902 30902 30902 30902 30902

	Dependent variable: schedule overrun	Dependent variable: cost overrun
Main effects
${\hat{SA}}_{i}$	−0.340 (0.083)***	−0.362 (0.085)***	−0.326 (0.082)***	−0.345 (0.085)***	−0.089 (0.016)***	−0.094 (0.016)***	−0.088 (0.016)***	−0.092 (0.016)***
Same-Agency Exp.	0.005 (0.004)	−0.009 (0.008)	0.005 (0.004)	−0.013 (0.010)	0.000 (0.001)	−0.003 (0.002)*	0.000 (0.001)	−0.003 (0.002)**
Different-Agency Exp.	−0.002 (0.002)	0.013 (0.011)	−0.002 (0.002)	0.014 (0.012)	0.000 (0.000)	0.004 (0.003)	0.000 (0.000)	0.004 (0.003)
Early in Fiscal Year	−0.001 (0.010)	−0.001 (0.010)	0.083 (0.034)**	0.086 (0.034)**	0.002 (0.002)	0.002 (0.002)	0.014 (0.007)**	0.014 (0.007)**
Moderating effects
${\hat{SA}}_{i} \times$ Same-Agency Exp.		0.017 (0.018)		0.028 (0.019)		0.003 (0.003)		0.005 (0.003)
${\hat{SA}}_{i} \times$ Different-Agency Exp.		−0.090 (0.048)*		−0.090 (0.049)*		−0.023 (0.011)**		−0.022 (0.011)*
${\hat{SA}}_{i} \times$ Early in Fiscal Year			−0.142 (0.048)***	−0.148 (0.049)***			−0.020 (0.010)**	−0.021 (0.010)**
Control variables
Task Type (0: Res, 1: Dev)	0.322 (0.188)*	0.321 (0.189)*	0.325 (0.189)*	0.326 (0.191)*	0.055 (0.043)	0.055 (0.043)	0.056 (0.043)	0.056 (0.043)
Planned Contract Duration	−0.116 (0.010)***	−0.115 (0.010)***	−0.116 (0.010)***	−0.116 (0.010)***	0.004 (0.001)***	0.004 (0.001)***	0.004 (0.001)***	0.004 (0.001)***
Planned Contract Cost	0.074 (0.011)***	0.073 (0.011)***	0.074 (0.011)***	0.073 (0.011)***	0.005 (0.003)**	0.005 (0.003)**	0.005 (0.003)**	0.005 (0.003)**
Contractor Firm Size	−0.007 (0.018)	−0.011 (0.018)	−0.007 (0.018)	−0.010 (0.018)	0.004 (0.003)	0.003 (0.003)	0.004 (0.003)	0.003 (0.003)
Contract Pricing	0.103 (0.032)***	0.101 (0.032)***	0.100 (0.032)***	0.099 (0.032)***	0.031 (0.007)***	0.031 (0.008)***	0.030 (0.008)***	0.030 (0.008)***
Competition	−0.011 (0.004)***	−0.010 (0.004)***	−0.011 (0.004)***	−0.010 (0.004)**	−0.004 (0.001)***	−0.004 (0.001)***	−0.004 (0.001)***	−0.004 (0.001)***
Oversight	0.073 (0.018)***	0.077 (0.019)***	0.075 (0.018)***	0.079 (0.019)***	0.007 (0.003)*	0.008 (0.003)**	0.007 (0.003)**	0.008 (0.003)**
Small Business	0.148 (0.042)***	0.160 (0.043)***	0.152 (0.041)***	0.163 (0.042)***	0.034 (0.009)***	0.037 (0.009)***	0.035 (0.009)***	0.037 (0.009)***
Federal Agency FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Region FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Year FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
R&D Contract Code FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Contractor Firm Cluster FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Constant	−0.372 (0.212)*	−0.396 (0.215)*	−0.408 (0.219)*	−0.431 (0.223)*	−0.076 (0.041)*	−0.082 (0.041)**	−0.082 (0.041)**	−0.087 (0.042)**
N	30902	30902	30902	30902	30902	30902	30902	30902

Note: Heteroscedasticity robust standard errors clustered by the agency in a given year are in parentheses.

* p ≤ 0.1, ** p ≤ 0.05, *** p ≤ 0.01.

Focusing on columns 1 and 5, we find that the coefficient of ${\hat{S A}}_{i}$ is negative and statistically significant for both schedule overrun (β=−0.340, p ≤ 0.01) and cost overrun (β=−0.089, p ≤ 0.01). This result indicates that with a one-unit increase in treatment probability (i.e., the probability that an R&D contract will be awarded through the set-aside program), the schedule (cost) overrun of R&D contracts is reduced by 34% (9%) in comparison to those awarded through open competition. That is, we find support for Hypothesis 1B but not Hypothesis 1A. To understand this result better, we compare the average competition (i.e., the average number of offers received) between set-aside R&D contracts and open-competition R&D contracts. We find that the average competition for set-aside R&D contracts (μ=133) is higher than the average competition for R&D contracts awarded through open competition (μ=34), and this difference is statistically significant (p ≤ 0.01). This difference in competition levels is consistent with findings in recent studies that have highlighted increased competition for contracts awarded through preference programs for underrepresented businesses (e.g., Athey et al., 2013; Hyndman and Parmeter, 2015). Specifically, as Athey et al. (2013, p.6) note, “a small business set-aside excludes big bidders, but increases the incentives for small bidders to participate because they anticipate less competition … the increase in small bidder participation can lead to a greater overall number of auction entrants.” This increased participation of small businesses may provide federal agencies with more choices to select a highly qualified contractor firm for set-aside contracts, thereby improving contract performance.

Regarding the moderating effects of same-agency and different-agency experience of contractor firms, we find that only the coefficient of ${\hat{S A}}_{i}$ × Different-Agency Experience is statistically significant for schedule overrun (β=−0.090, p ≤ 0.1) and cost overrun (β=−0.022, p ≤ 0.1). This result suggests that, in comparison to the experience accumulated while working for the same federal agency, the experience accumulated while working for a different federal agency is more beneficial in reducing the schedule overrun and cost overrun of set-aside R&D contracts. Therefore, we find support for Hypothesis 2B but not Hypothesis 2A. The lack of support for the moderating effect of same-agency experience suggests that the execution of federal R&D contracts benefits more from accumulating diverse knowledge from different agencies in comparison to the specialized knowledge accumulated by working for the same federal agency. Regarding the moderating effects of the timing of awarding a contract, from columns 4 and 8, we find that the coefficient of ${\hat{S A}}_{i}$ × Early in Fiscal Year is negative and statistically significant for schedule overrun (β=−0.148, p ≤ 0.01) and cost overrun (β=−0.021, p ≤ 0.05). This result indicates that an increase in treatment probability leads to a greater reduction in the schedule overrun and cost overrun values of contracts awarded early in the fiscal year, compared to those awarded later in the fiscal year. Hypothesis 3 is supported. To delve deeper into this finding, we compare the average number of set-aside contracts that are awarded early in a fiscal year (i.e., in the first quarter) with those awarded later in a fiscal year (i.e., in the second, third, and fourth quarters). As shown on the left-hand side of Figure 3,¹³ a lower number of set-aside contracts are awarded early in a fiscal year, suggesting that contracting officers are likely to experience a lower workload in this period in terms of evaluating responses to requests for proposals, thereby providing them with more time to carefully assess contract requirements and identify a well-qualified contractor firm to execute set-aside contracts in the earlier part of a fiscal year.¹⁴

Figure 3.

Average number of set-aside contracts awarded early vs. late in a fiscal year.

Finally, to enable an intuitive understanding of the moderating effects, we plot the average marginal effects of an R&D contract's set-aside status across various levels of different-agency experience and early in fiscal year for schedule overrun (Figure 4(a) and 4(b)) and cost overrun (Figure 4(c) and 4(d)). Figure 4(a) and 4(c) shows that increasing levels of different-agency experience reduces the schedule (cost) overrun associated with set-aside R&D contracts. Figure 4(b) and (4(d)) shows that set-aside R&D contracts awarded early in the fiscal year have a lower schedule (cost) overrun in comparison to those awarded later in the fiscal year.

Figure 4.

Average marginal effect plot for different-agency experience and early in fiscal year.

5.2. Robustness Checks: Estimation Using Matching and Additional Model Specifications

We perform several additional analyses to check for the robustness of our results. These additional model specifications include: (i) controlling for contractor firm's past performance, (ii) using propensity score matching (PSM) to generate a matched sample of contracts, (iii) controlling for “agency-year” time trends, and (iv) using an alternative measure for Early in Fiscal Year. To save space, we provide details of each model specification in the Online Appendix and present the results in columns 9 to 18 of Table 3. The results using these alternative model specifications continue to provide support for our main results, highlighting the robustness of our findings.

Table 3.
Robustness checks: estimation with additional controls, alternative measures and Propensity Score Matching (PSM).

Dependent variable: schedule overrun

(9) (10) (11) (12) (13)

Controlling for PSM using PSM using Controlling for Early in Fiscal

Past Performance Firm Revenues Firm Employees Agency × Year Year (Revised)

Main effects

${\hat{SA}}_{i}$ −0.398 (0.101)* −0.356 (0.096)* −0.349 (0.097)* −0.328 (0.086)* −0.318 (0.087)*

Same-Agency Exp. −0.014 (0.008) −0.011 (0.009) −0.015 (0.010) −0.015 (0.013) −0.010 (0.009)

Different-Agency Exp. 0.014 (0.012) 0.010 (0.010) 0.014 (0.013) 0.014 (0.012) 0.011 (0.010)

Early in Fiscal Year 0.100 (0.042) 0.059 (0.036) 0.067 (0.037)* 0.084 (0.035)** 0.052 (0.034)

Moderating effects

${\hat{SA}}_{i} \times$ Same-Agency Exp. 0.026 (0.018) 0.035 (0.018)* 0.032 (0.019) 0.030 (0.023) 0.022 (0.018)

${\hat{SA}}_{i} \times$ Different-Agency Exp. −0.102 (0.052) −0.102 (0.049) −0.098 (0.051)* −0.086 (0.049)* −0.078 (0.045)*

${\hat{SA}}_{i} \times$ Early in Fiscal Year −0.151 (0.056)* −0.111 (0.054) −0.129 (0.057) −0.146 (0.050)* −0.111 (0.050)**

Control variables

Task Type (0: Res, 1: Dev) 0.344 (0.264) 0.336 (0.191)* 0.331 (0.190)* 0.449 (0.196)** 0.321 (0.189)*

Planned Contract Duration −0.126 (0.011)* −0.115 (0.010)* −0.115 (0.010)* −0.116 (0.010)* −0.115 (0.010)*

Planned Contract Cost −0.033 (0.023) 0.076 (0.012)* 0.074 (0.012)* −0.010 (0.018) 0.072 (0.012)*

Contractor Firm Size 0.076 (0.012)* −0.013 (0.021) 0.004 (0.014) 0.073 (0.012)* −0.010 (0.018)

Contract Pricing 0.089 (0.039) 0.096 (0.035)* 0.098 (0.035)* 0.102 (0.032)* 0.103 (0.032)*

Competition −0.011 (0.005) −0.007 (0.004) −0.006 (0.004) −0.013 (0.004)* −0.012 (0.004)*

Oversight 0.096 (0.023)* 0.079 (0.020)* 0.078 (0.020)* 0.081 (0.019)* 0.076 (0.019)*

Small Business 0.188 (0.053)* 0.169 (0.048)* 0.170 (0.049)* 0.148 (0.043)* 0.158 (0.042)*

Average Past Performance 0.007 (0.005)

Federal Agency FE Yes Yes Yes Yes Yes

Region FE Yes Yes Yes Yes Yes

Year FE Yes Yes Yes Yes Yes

R&D Contract Code FE Yes Yes Yes Yes Yes

Contractor Firm Cluster FE Yes Yes Yes Yes Yes

Constant −0.502 (0.249) −0.423 (0.141)* −0.295 (0.210) −0.403 (0.215)* −0.429 (0.226)*

N 27211 24882 24852 30902 30902

Dependent variable: cost overrun

(14) (15) (16) (17) (18)

Controlling for PSM using PSM using Controlling for Early in Fiscal

Past Performance Firm Revenues Firm Employees Agency × Year Year (Revised)

Main effects

${\hat{SA}}_{i}$ −0.398 (0.101)* −0.101 (0.020)* −0.100 (0.020)* −0.328 (0.086)* −0.083 (0.017)*

Same-Agency Exp. −0.014 (0.008) −0.003 (0.001) −0.003 (0.002)* −0.015 (0.013) −0.003 (0.001)*

Different-Agency Exp. 0.014 (0.012) 0.003 (0.002) 0.005 (0.004) 0.014 (0.012) 0.003 (0.003)

Early in Fiscal Year 0.100 (0.042) 0.012 (0.007) 0.015 (0.007) 0.084 (0.035)** 0.017 (0.009)*

Moderating effects

${\hat{SA}}_{i} \times$ Same-Agency Exp. 0.026 (0.018) 0.007 (0.003) 0.005 (0.003) 0.030 (0.023) 0.005 (0.003)

${\hat{SA}}_{i} \times$ Different-Agency Exp. −0.102 (0.052) −0.020 (0.011)* −0.024 (0.012)** −0.086 (0.049)* −0.017 (0.010)*

${\hat{SA}}_{i} \times$ Early in Fiscal Year −0.151 (0.056)* −0.017 (0.011)† −0.022 (0.011) −0.146 (0.050)* −0.030 (0.013)

Control variables

Task Type (0: Res, 1: Dev) 0.344 (0.264) 0.055 (0.042) 0.054 (0.043) 0.449 (0.196) 0.055 (0.043)

Planned Contract Duration −0.126 (0.011)* 0.005 (0.001)*** 0.005 (0.002)* −0.116 (0.010)* 0.004 (0.001)*

Planned Contract Cost −0.033 (0.023) 0.006 (0.003) 0.005 (0.003) −0.010 (0.018) 0.005 (0.003)*

Contractor Firm Size 0.076 (0.012)* 0.005 (0.003) −0.004 (0.002) 0.073 (0.012)* 0.004 (0.003)

Contract Pricing 0.089 (0.039) 0.027 (0.009)* 0.029 (0.009)* 0.102 (0.032)* 0.031 (0.008)*

Competition −0.011 (0.005) −0.006 (0.001)* −0.005 (0.001)* −0.013 (0.004)* −0.004 (0.001)*

Oversight 0.096 (0.023)* 0.007 (0.004) 0.008 (0.004) 0.081 (0.019)* 0.007 (0.004)

Small Business 0.188 (0.053)* 0.043 (0.011)* 0.041 (0.011)* 0.148 (0.043)* 0.036 (0.009)***

Average Past Performance 0.007 (0.005)

Federal Agency FE Yes Yes Yes Yes Yes

Region FE Yes Yes Yes Yes Yes

Year FE Yes Yes Yes Yes Yes

R&D Contract Code FE Yes Yes Yes Yes Yes

Contractor Firm Cluster FE Yes Yes Yes Yes Yes

Constant −0.502 (0.249) −0.044 (0.020) −0.052 (0.039) −0.403 (0.215)* −0.091 (0.043)**

N 27211 24882 24852 30902 30902

	Dependent variable: schedule overrun
Main effects
${\hat{SA}}_{i}$	−0.398 (0.101)***	−0.356 (0.096)***	−0.349 (0.097)***	−0.328 (0.086)***	−0.318 (0.087)***
Same-Agency Exp.	−0.014 (0.008)	−0.011 (0.009)	−0.015 (0.010)	−0.015 (0.013)	−0.010 (0.009)
Different-Agency Exp.	0.014 (0.012)	0.010 (0.010)	0.014 (0.013)	0.014 (0.012)	0.011 (0.010)
Early in Fiscal Year	0.100 (0.042)**	0.059 (0.036)	0.067 (0.037)*	0.084 (0.035)**	0.052 (0.034)
Moderating effects
${\hat{SA}}_{i} \times$ Same-Agency Exp.	0.026 (0.018)	0.035 (0.018)*	0.032 (0.019)	0.030 (0.023)	0.022 (0.018)
${\hat{SA}}_{i} \times$ Different-Agency Exp.	−0.102 (0.052)**	−0.102 (0.049)**	−0.098 (0.051)*	−0.086 (0.049)*	−0.078 (0.045)*
${\hat{SA}}_{i} \times$ Early in Fiscal Year	−0.151 (0.056)***	−0.111 (0.054)**	−0.129 (0.057)**	−0.146 (0.050)***	−0.111 (0.050)**
Control variables
Task Type (0: Res, 1: Dev)	0.344 (0.264)	0.336 (0.191)*	0.331 (0.190)*	0.449 (0.196)**	0.321 (0.189)*
Planned Contract Duration	−0.126 (0.011)***	−0.115 (0.010)***	−0.115 (0.010)***	−0.116 (0.010)***	−0.115 (0.010)***
Planned Contract Cost	−0.033 (0.023)	0.076 (0.012)***	0.074 (0.012)***	−0.010 (0.018)	0.072 (0.012)***
Contractor Firm Size	0.076 (0.012)***	−0.013 (0.021)	0.004 (0.014)	0.073 (0.012)***	−0.010 (0.018)
Contract Pricing	0.089 (0.039)**	0.096 (0.035)***	0.098 (0.035)***	0.102 (0.032)***	0.103 (0.032)***
Competition	−0.011 (0.005)**	−0.007 (0.004)	−0.006 (0.004)	−0.013 (0.004)***	−0.012 (0.004)***
Oversight	0.096 (0.023)***	0.079 (0.020)***	0.078 (0.020)***	0.081 (0.019)***	0.076 (0.019)***
Small Business	0.188 (0.053)***	0.169 (0.048)***	0.170 (0.049)***	0.148 (0.043)***	0.158 (0.042)***
Average Past Performance	0.007 (0.005)
Federal Agency FE	Yes	Yes	Yes	Yes	Yes
Region FE	Yes	Yes	Yes	Yes	Yes
Year FE	Yes	Yes	Yes	Yes	Yes
R&D Contract Code FE	Yes	Yes	Yes	Yes	Yes
Contractor Firm Cluster FE	Yes	Yes	Yes	Yes	Yes
Constant	−0.502 (0.249)**	−0.423 (0.141)***	−0.295 (0.210)	−0.403 (0.215)*	−0.429 (0.226)*
N	27211	24882	24852	30902	30902
	Dependent variable: cost overrun
	(14)	(15)	(16)	(17)	(18)
	Controlling for	PSM using	PSM using	Controlling for	Early in Fiscal
	Past Performance	Firm Revenues	Firm Employees	Agency × Year	Year (Revised)
Main effects
${\hat{SA}}_{i}$	−0.398 (0.101)***	−0.101 (0.020)***	−0.100 (0.020)***	−0.328 (0.086)***	−0.083 (0.017)***
Same-Agency Exp.	−0.014 (0.008)	−0.003 (0.001)**	−0.003 (0.002)*	−0.015 (0.013)	−0.003 (0.001)*
Different-Agency Exp.	0.014 (0.012)	0.003 (0.002)	0.005 (0.004)	0.014 (0.012)	0.003 (0.003)
Early in Fiscal Year	0.100 (0.042)**	0.012 (0.007)	0.015 (0.007)**	0.084 (0.035)**	0.017 (0.009)*
Moderating effects
${\hat{SA}}_{i} \times$ Same-Agency Exp.	0.026 (0.018)	0.007 (0.003)**	0.005 (0.003)	0.030 (0.023)	0.005 (0.003)
${\hat{SA}}_{i} \times$ Different-Agency Exp.	−0.102 (0.052)**	−0.020 (0.011)*	−0.024 (0.012)**	−0.086 (0.049)*	−0.017 (0.010)*
${\hat{SA}}_{i} \times$ Early in Fiscal Year	−0.151 (0.056)***	−0.017 (0.011)†	−0.022 (0.011)**	−0.146 (0.050)***	−0.030 (0.013)**
Control variables
Task Type (0: Res, 1: Dev)	0.344 (0.264)	0.055 (0.042)	0.054 (0.043)	0.449 (0.196)**	0.055 (0.043)
Planned Contract Duration	−0.126 (0.011)***	0.005 (0.001)***	0.005 (0.002)*	−0.116 (0.010)***	0.004 (0.001)***
Planned Contract Cost	−0.033 (0.023)	0.006 (0.003)**	0.005 (0.003)**	−0.010 (0.018)	0.005 (0.003)*
Contractor Firm Size	0.076 (0.012)***	0.005 (0.003)	−0.004 (0.002)**	0.073 (0.012)***	0.004 (0.003)
Contract Pricing	0.089 (0.039)**	0.027 (0.009)***	0.029 (0.009)***	0.102 (0.032)***	0.031 (0.008)***
Competition	−0.011 (0.005)**	−0.006 (0.001)***	−0.005 (0.001)***	−0.013 (0.004)***	−0.004 (0.001)***
Oversight	0.096 (0.023)***	0.007 (0.004)**	0.008 (0.004)**	0.081 (0.019)***	0.007 (0.004)**
Small Business	0.188 (0.053)***	0.043 (0.011)***	0.041 (0.011)***	0.148 (0.043)***	0.036 (0.009)***
Average Past Performance	0.007 (0.005)
Federal Agency FE	Yes	Yes	Yes	Yes	Yes
Region FE	Yes	Yes	Yes	Yes	Yes
Year FE	Yes	Yes	Yes	Yes	Yes
R&D Contract Code FE	Yes	Yes	Yes	Yes	Yes
Contractor Firm Cluster FE	Yes	Yes	Yes	Yes	Yes
Constant	−0.502 (0.249)**	−0.044 (0.020)**	−0.052 (0.039)	−0.403 (0.215)*	−0.091 (0.043)**
N	27211	24882	24852	30902	30902

Note: Heteroscedasticity robust standard errors clustered by agency in a given year are in parentheses.

†p ≤ 0.2, * p ≤ 0.1, ** p ≤ 0.05, *** p ≤ 0.01.

5.3. Post-Hoc Analysis of Set-Aside R&D Contracts Awarded by the Department of Defense

Given the significant moderation effects of different-agency experience and the large proportion of DoD contracts in our sample (∼55% of the total contracts), we delve into what type of different-agency experience is beneficial for the performance of set-aside R&D contracts awarded by DoD. Specifically, for the sub-sample of DoD contracts, we partition different-agency experience into two underlying dimensions—related-agency experience (i.e., the number of R&D contracts executed by a contractor firm for NASA) and unrelated-agency experience (i.e., the number of R&D contracts executed by a contractor firm for agencies other than NASA and DoD). For DoD contracts, the experience acquired in executing NASA contracts is related as both DoD and NASA are categorized as mission agencies (Rathje and Katila, 2021; Ergas, 1987). Mission agencies like DoD and NASA support national sovereignty goals by investing in R&D initiatives that enhance national security and space travel. Their mission is to fund R&D initiatives for achieving “practice-oriented (aka mission goals)” (Rathje and Katila, 2021).

In re-estimating our main models, we use a sub-sample of DoD contracts but replace different-agency experience with related-agency experience and unrelated-agency experience. The sub-sample analysis results are presented in Table 4, and to save space, we discuss the results only for the moderating effects of related-agency experience and unrelated-agency experience.¹⁵ Focusing on columns 22 and 26 in Table 4, we find that the coefficient of ${\hat{S A}}_{i}$ × Related-Agency Experience is negative for both schedule overrun (β=−0.795, p ≤ 0.01) and cost overrun (β=−0.072, p ≤ 0.1); while the coefficient of ${\hat{S A}}_{i}$ × Unrelated-Agency Experience is positive for both schedule overrun (β=1.046, p ≤ 0.01) and cost overrun (β=0.202, p ≤ 0.1). These results suggest that for DoD contracts, the performance benefits of different-agency experience for set-aside contracts are derived primarily from a contractor firm's related-agency experience, and that a contractor firm's unrelated-agency experience may be detrimental to the performance of set-aside contracts. In explaining this result, we note that R&D contracts awarded by both NASA and DoD often involve technology-intensive tasks. The execution of R&D contracts for NASA can help contractor firms to develop relevant technical capabilities and expertise that can also be applied to the execution of R&D contracts for DoD. Suppose a contractor firm has prior experience in developing advanced imaging technology for NASA's earth observation satellites. This experience could potentially translate into capabilities that are relevant to the execution of DoD R&D contracts, such as the development of advanced surveillance technology for military aircrafts, radars, and drones. In a similar vein, contractor firms involved in the development of Mars Perseverance Rover may acquire expertise in robotics, development of autonomous systems and advanced materials, which can be relevant to the execution of DoD R&D contracts that involve the development of unmanned aerial vehicles, autonomous vehicles and advanced sensors.

Table 4.
Estimation using the Department of Defense (DoD) contracts only.

Dependent variable: schedule overrun Dependent variable: cost overrun

(19) (20) (21) (22) (23) (24) (25) (26)

Main effects

${\hat{SA}}_{i}$ −0.463 (0.084)* −0.511 (0.102)* −0.444 (0.084)* −0.487 (0.107)* −0.101 (0.023)* −0.096 (0.023)* −0.094 (0.023)* −0.090 (0.024)*

Same-Agency Exp. 0.010 (0.007) 0.012 (0.066) 0.011 (0.007) 0.035 (0.070) 0.004 (0.001)* −0.005 (0.026) 0.004 (0.002) −0.000 (0.026)

Related-Agency Exp. 0.035 (0.017) 0.691 (0.266)* 0.034 (0.017) 0.702 (0.280) 0.008 (0.003)*** 0.065 (0.037)* 0.008 (0.003)*** 0.065 (0.036)*

Unrelated-Agency Exp. −0.040 (0.021)* −0.225 (0.126)* −0.043 (0.021) −0.260 (0.130) −0.020 (0.006)* −0.050 (0.032) −0.021 (0.007)* −0.058 (0.034)*

Early in Fiscal Year −0.012 (0.007) −0.012 (0.006)* 0.092 (0.069) 0.109 (0.067) 0.003 (0.002) 0.002 (0.002) 0.037 (0.014)* 0.034 (0.013)*

Moderating effects

${\hat{SA}}_{i} \times$ Same-Agency Exp. −0.048 (0.099) −0.110 (0.184) 0.001 (0.032) −0.005 (0.033)

${\hat{SA}}_{i} \times$ Related-Agency Exp. −0.747 (0.270)* −0.795 (0.001)* −0.071 (0.041)* −0.072 (0.040)*

${\hat{SA}}_{i} \times$ Unrelated-Agency Exp. 0.933 (0.575)* 1.046 (0.003)*** 0.174 (0.101)* 0.202 (0.106)*

${\hat{SA}}_{i} \times$ Early in Fiscal Year −0.135 (0.081)* −0.158 (0.079) −0.045 (0.016)* −0.041 (0.015)*

Control variables

Task Type (0: Res, 1: Dev) 0.643 (0.257) 0.559 (0.277) 0.676 (0.264) 0.587 (0.285)** 0.138 (0.071)* 0.130 (0.065) 0.149 (0.076) 0.138 (0.069)

Planned Contract Duration −0.137 (0.017)* −0.138 (0.017)* −0.136 (0.017)* −0.136 (0.017)*** 0.004 (0.003) 0.004 (0.002) 0.004 (0.003)* 0.004 (0.003)*

Planned Contract Cost 0.065 (0.018)* 0.068 (0.018)* 0.065 (0.018)* 0.067 (0.018)* 0.007 (0.004)* 0.008 (0.004)* 0.007 (0.004)* 0.007 (0.004)*

Contractor Firm Size −0.015 (0.033) −0.012 (0.034) −0.015 (0.034) −0.013 (0.035) 0.002 (0.004) 0.003 (0.004) 0.002 (0.004) 0.003 (0.004)

Contract Pricing 0.107 (0.026)* 0.093 (0.029)* 0.105 (0.026)* 0.087 (0.030)* 0.037 (0.009)* 0.037 (0.007)* 0.037 (0.009)* 0.036 (0.007)*

Competition −0.016 (0.003)* −0.017 (0.003)* −0.016 (0.003)* −0.017 (0.003)* −0.007 (0.001)* −0.007 (0.001)* −0.007 (0.001)* −0.007 (0.001)*

Oversight 0.115 (0.037)* 0.116 (0.039)* 0.117 (0.036)* 0.119 (0.038)* 0.015 (0.008)* 0.016 (0.009)* 0.016 (0.008)* 0.016 (0.008)

Small Business 0.233 (0.059)* 0.238 (0.064)* 0.234 (0.059)* 0.240 (0.066)* 0.046 (0.015)* 0.047 (0.015)* 0.046 (0.014)* 0.047 (0.014)*

Region FE Yes Yes Yes Yes Yes Yes Yes Yes

Year FE Yes Yes Yes Yes Yes Yes Yes Yes

R&D Contract Code FE Yes Yes Yes Yes Yes Yes Yes Yes

Contractor Firm Type FE Yes Yes Yes Yes Yes Yes Yes Yes

Constant −0.381 (0.151) −0.230 (0.180) −0.442 (0.172)** −0.293 (0.203) −0.070 (0.042)* −0.059 (0.047) −0.090 (0.052)* −0.076 (0.056)

N 16870 16870 16870 16870 16870 16870 16870 16870

	Dependent variable: schedule overrun	Dependent variable: cost overrun
Main effects
${\hat{SA}}_{i}$	−0.463 (0.084)***	−0.511 (0.102)***	−0.444 (0.084)***	−0.487 (0.107)***	−0.101 (0.023)***	−0.096 (0.023)***	−0.094 (0.023)***	−0.090 (0.024)***
Same-Agency Exp.	0.010 (0.007)	0.012 (0.066)	0.011 (0.007)	0.035 (0.070)	0.004 (0.001)***	−0.005 (0.026)	0.004 (0.002)**	−0.000 (0.026)
Related-Agency Exp.	0.035 (0.017)**	0.691 (0.266)***	0.034 (0.017)**	0.702 (0.280)**	0.008 (0.003)***	0.065 (0.037)*	0.008 (0.003)***	0.065 (0.036)*
Unrelated-Agency Exp.	−0.040 (0.021)*	−0.225 (0.126)*	−0.043 (0.021)**	−0.260 (0.130)**	−0.020 (0.006)***	−0.050 (0.032)	−0.021 (0.007)***	−0.058 (0.034)*
Early in Fiscal Year	−0.012 (0.007)	−0.012 (0.006)*	0.092 (0.069)	0.109 (0.067)	0.003 (0.002)	0.002 (0.002)	0.037 (0.014)***	0.034 (0.013)***
Moderating effects
${\hat{SA}}_{i} \times$ Same-Agency Exp.		−0.048 (0.099)		−0.110 (0.184)		0.001 (0.032)		−0.005 (0.033)
${\hat{SA}}_{i} \times$ Related-Agency Exp.		−0.747 (0.270)***		−0.795 (0.001)***		−0.071 (0.041)*		−0.072 (0.040)*
${\hat{SA}}_{i} \times$ Unrelated-Agency Exp.		0.933 (0.575)*		1.046 (0.003)***		0.174 (0.101)*		0.202 (0.106)*
${\hat{SA}}_{i} \times$ Early in Fiscal Year			−0.135 (0.081)*	−0.158 (0.079)**			−0.045 (0.016)***	−0.041 (0.015)***
Control variables
Task Type (0: Res, 1: Dev)	0.643 (0.257)**	0.559 (0.277)**	0.676 (0.264)**	0.587 (0.285)**	0.138 (0.071)*	0.130 (0.065)**	0.149 (0.076)**	0.138 (0.069)**
Planned Contract Duration	−0.137 (0.017)***	−0.138 (0.017)***	−0.136 (0.017)***	−0.136 (0.017)***	0.004 (0.003)	0.004 (0.002)	0.004 (0.003)*	0.004 (0.003)*
Planned Contract Cost	0.065 (0.018)***	0.068 (0.018)***	0.065 (0.018)***	0.067 (0.018)***	0.007 (0.004)*	0.008 (0.004)*	0.007 (0.004)*	0.007 (0.004)*
Contractor Firm Size	−0.015 (0.033)	−0.012 (0.034)	−0.015 (0.034)	−0.013 (0.035)	0.002 (0.004)	0.003 (0.004)	0.002 (0.004)	0.003 (0.004)
Contract Pricing	0.107 (0.026)***	0.093 (0.029)***	0.105 (0.026)***	0.087 (0.030)***	0.037 (0.009)***	0.037 (0.007)***	0.037 (0.009)***	0.036 (0.007)***
Competition	−0.016 (0.003)***	−0.017 (0.003)***	−0.016 (0.003)***	−0.017 (0.003)***	−0.007 (0.001)***	−0.007 (0.001)***	−0.007 (0.001)***	−0.007 (0.001)***
Oversight	0.115 (0.037)***	0.116 (0.039)***	0.117 (0.036)***	0.119 (0.038)***	0.015 (0.008)*	0.016 (0.009)*	0.016 (0.008)*	0.016 (0.008)**
Small Business	0.233 (0.059)***	0.238 (0.064)***	0.234 (0.059)***	0.240 (0.066)***	0.046 (0.015)***	0.047 (0.015)***	0.046 (0.014)***	0.047 (0.014)***
Region FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Year FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
R&D Contract Code FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Contractor Firm Type FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Constant	−0.381 (0.151)**	−0.230 (0.180)	−0.442 (0.172)**	−0.293 (0.203)	−0.070 (0.042)*	−0.059 (0.047)	−0.090 (0.052)*	−0.076 (0.056)
N	16870	16870	16870	16870	16870	16870	16870	16870

Note: Heteroscedasticity-robust standard errors clustered by year are in parentheses.

* p ≤ 0.1, ** p ≤ 0.05, *** p ≤ 0.01.

Taken together, these examples and our results pertaining to the moderating effect of related-agency experience highlight an important aspect of successfully executing set-aside R&D contracts awarded by DoD. The execution of these contracts often entails the development of novel solutions in a timely and cost-efficient manner, and within this context, the acquisition of knowledge through related variation in experience (i.e., executing contracts for NASA) may be more effective in improving contract performance in comparison to repeatedly working for the same agency (i.e., executing contracts for DoD) or for non-mission agencies (i.e., executing contracts for agencies other than NASA and DoD).

6. Discussion

6.1. Summary and Contributions

The U.S. federal government awards nearly $500 billion in contracts annually of which at least 23% must be awarded to small businesses through the set-aside program. While this mandate encourages the inclusion of small businesses in the federal procurement process, our understanding about the performance implications of awarding contracts by restricting competition to small businesses is limited. This study takes a first step in this direction by empirically examining the performance differences between R&D contracts awarded through the set-aside program vs. through open competition.

The analysis results using detailed execution-level data on a large sample of federal R&D contracts find that set-aside contracts experience lower schedule overrun and cost overrun in comparison to open competition contracts. However, the performance of set-aside contracts depends on the nature of contractor firm experience (i.e., same-agency vs. different-agency experience) and the timing of awarding a contract (i.e., early vs. later in a fiscal year). With regard to the effect of contractor firm experience, we find that the benefits of more experience for set-aside contracts arise primarily from a contractor firm's different-agency experience. Surprisingly, we do not find any benefits associated with increasing levels of a contractor firm's same-agency experience. With regard to the effect of a contract's award timing, we find that set-aside R&D contracts awarded early in a fiscal year perform better than those awarded later in a fiscal year. Taken together, these findings make the following contributions to research and practice.

First, our study contributes to the operations management (OM) literature on public procurement by empirically examining the impact of small business preference programs on contract performance. While prior OM research on public procurement has examined the impact of government oversight on contract performance (Calvo et al., 2019), our study advances this research by examining a relatively understudied feature of public procurement—the use of preference programs—and its impact on contract performance, after controlling for the contract's oversight level. Although preference programs provide a level playing field for small businesses to compete for public contracts, they constitute a significant portion of the government's overall procurement expenditures funded through taxpayer money. It is, therefore, important for these contracts to achieve their objectives within the planned schedule and cost. By showing that awarding contracts based on considerations that promote the inclusion of small businesses may not necessarily deteriorate contract performance, we respond to recent calls for OM scholars to conduct research on inclusive supply chains (Deshpande and Swaminathan, 2020; Kalkanci et al., 2019).

Second, although prior research has highlighted the welfare implications of preference programs—for example, creation of new firms (Chatterji et al., 2013) and inclusion of underrepresented businesses in public procurement (Nakabayashi, 2013)—a distinctive feature of our study is in evaluating the impact of preference programs at the more granular contract level of the procurement process. Specifically, our finding that set-aside R&D contracts perform better than open-competition R&D contracts suggests that the key purpose of the set-aside program (i.e., to promote the inclusion of small businesses in federal contracts) is being fulfilled without compromising contract performance. Moreover, an examination of the contract-level implications of preference programs represents a consequential line of inquiry as it can help to identify important contingencies related to a contract's environment that affect a contract execution and that, therefore, need to be considered by stakeholders during the procurement process to improve the performance of future set-aside contracts.

Relatedly, the third contribution of our study lies in identifying two contingencies related to a contract's environment, the nature of contractor firm experience and the timing of awarding a contract, that interact with the contract's set-aside status to affect its performance. Specifically, we find that set-aside R&D contracts perform better when they are executed by firms with more different-agency experience and when these contracts are awarded early in the fiscal year. This finding regarding the moderating effect of contractor firm experience offers a better understanding of the differential effects of same-agency experience and different-agency experience on the performance of set-aside R&D contracts. Post-hoc analysis using set-aside R&D contracts awarded by DoD demonstrates that the benefits of different-agency experience come from a contractor firm's related-agency experience with NASA; however, a contractor firm's unrelated-agency experience with agencies other than DoD and NASA is associated with poorer performance for these contracts. Such an understanding can enable federal agencies to go beyond the overall experience of a contractor firm and develop more nuanced metrics of contractor firm experience, which in turn may facilitate the identification of a better-qualified contractor firm to execute a set-aside contract and improve contract performance. Our finding regarding the moderating effect of a contract's award timing emphasizes the need to pay greater attention to contracts awarded later in a fiscal year as such consideration has direct consequences for contract performance. Taken together, these results have implications for the federal government in that they highlight how heterogeneity in the nature of contractor firm experience and an R&D contract's award timing can affect the performance of set-aside R&D contracts.

6.2. Implications for Policymakers

From a policy standpoint, our research highlights the importance of studying the contract-level implications of small business preference programs in public procurement. Existing evaluations of the set-aside program by the federal government have focused largely on tracking the participation levels of small businesses and in identifying mechanisms for increasing their participation levels (Congressional Research Service, 2016). Our findings regarding the impact of the set-aside program on contract performance complement these existing evaluations by providing a holistic assessment of the overall impact of the set-aside program in the federal procurement process. In particular, our findings regarding the role of the nature of contractor firm experience and an R&D contract's award timing highlight important considerations for policymakers. While the FAR provides guidelines to federal agencies for identifying a highly qualified contractor firm (e.g., consideration of a contractor firm's past performance), these guidelines do not include a contractor firm's prior experience in executing federal contracts as a criterion for awarding R&D contracts through the set-aside program. Given the differential effects of a contractor firm's same-agency experience and different-agency experience on the performance of set-aside R&D contracts, an actionable recommendation for the federal government is to track these underlying dimensions of a contractor firm's experience for identifying a highly qualified contractor firm to execute a set-aside R&D contract.

From a contractor firm's standpoint, while there are additional intangible benefits of repeated contracting with the same agency (e.g., reduced uncertainty about agency's contracting process and greater trust between the two sides), the focus on different-agency experience can also encourage contractor firms to execute R&D contracts for different federal agencies within the same technical domain to augment their expertise. Within different-agency experience, executing contracts for NASA (i.e., a mission agency) may help contractor firms to improve their performance on set-aside R&D contracts awarded by DoD, while executing contracts for agencies other than DoD and NASA (i.e., non-mission agencies) may be detrimental to the performance of these contracts. Our finding that set-aside R&D contracts awarded early in a fiscal year experience lower schedule overrun and cost overrun than contracts awarded later in a fiscal year highlights the role of a contract's award timing in impacting contract performance. Such an understanding may direct greater attention from federal agencies to set-aside R&D contracts that are being awarded later in a fiscal year and encourage the agency personnel as well as contracting officers to invest greater effort and scrutiny into the contract award process (e.g., evaluations of the offers received, clarification on contract requirements, etc.) for these contracts.

6.3. Limitations and Directions for Future Research

Our study has limitations that serve as opportunities for future research. First, our sample is restricted to R&D contracts below $700,000, since contracts that fall above this range are subject to different mandates that may confound the effects of set-aside program. Future research should examine whether and how the performance of high-value R&D contracts awarded through the set-aside program differs from those awarded through open competition. Second, our study focuses solely on R&D contracts awarded by the U.S. federal government. Future research can inform the existing literature on preference programs by examining if our study findings can be generalized to other contract types (e.g., construction contracts, maintenance contracts) or to other countries (e.g., public procurement in Japan, Canada, and South Africa that have similar preference programs). Third, while our study evaluates the impact of the set-aside program on the schedule overrun and cost overrun of R&D contracts, it does not offer any insights into the long-term implications of preference programs for small businesses (e.g., long-term revenue growth, survival, and employment generated by small businesses). Future studies in this direction can provide a comprehensive understanding of the long-term implications of preference programs in public procurement. Furthermore, while the uncertainty associated with an R&D contract is implicitly considered in establishing its schedule and cost performance metrics (FAR Part 35), future research can extend our study findings by using measures other than schedule overrun and cost overrun for capturing contract performance. Additionally, as contractor firms may execute more than one contract at any given point in time, it is possible that the portfolio of contracts with a contractor firm influences the performance of set-aside contracts awarded to the firm. Future research can examine whether and what specific characteristics of the contract portfolio (for example, the degree of similarity among the contracts in the portfolio) influence the performance of set-aside contracts awarded to contractor firms. Finally, our study evaluates the performance implications of a specific legislative provision, the set-aside program, which encourages the participation of small businesses in public procurement. Future research can evaluate the impact of other legislative provisions (e.g., regarding sub-contracting to small businesses, acquisition of environmentally sustainable products) to further inform policy-making in public procurement.

Notwithstanding these limitations, our study provides consequential, theoretically grounded insights into how the execution of R&D contracts awarded through the set-aside program can be improved. The importance of preference programs in public procurement toward delivering policy mandates and creating more opportunities for underrepresented businesses cannot be overstated. While this line of inquiry has received less attention in academic literature, it has the potential to generate insights that have significant academic and practical relevance.

Supplemental Material

sj-pdf-1-pao-10.1177_10591478241270112 - Supplemental material for Advancing Small Business Inclusion in Public Procurement: Evidence From U.S. Federal Government R&D Contracts

Supplemental material, sj-pdf-1-pao-10.1177_10591478241270112 for Advancing Small Business Inclusion in Public Procurement: Evidence From U.S. Federal Government R&D Contracts by Dwaipayan Roy, Anant Mishra and Kingshuk K Sinha in Production and Operations Management

Footnotes

Acknowledgments

The authors gratefully acknowledge the developmental comments provided by Bradley Staats (department editor), the anonymous associate editor, and the two anonymous referees during the review process. They also thank Arzi Adbi, Kevin Linderman, Xiaotong Shen, Sarang Deo, and seminar participants at the 2018 ISB-POMS Workshop, the 2019 Applied Economics Workshop at the Carlson School of Management, Colorado State University, Eindhoven University of Technology, Erasmus University, Indian School of Business, University of Virginia, Michigan State University, Tilburg University, University of Notre Dame, and York University. The development of the initial versions of the paper has also benefited from discussions with thought leaders and staff associated with The Greg and Camille Baroni Center for Government Contracting at George Mason University.

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Dwaipayan Roy

Anant Mishra

Kingshuk K Sinha

Supplemental Material

Supplemental material for this article is available online ().

Notes

How to cite this article

Roy D, Mishra A and Sinha KK (2024) Advancing Small Business Inclusion in Public Procurement: Evidence From U.S. Federal Government R&D Contracts. Production and Operations Management 33(11): 2201–2220.

References

Alcacer

Oxley

(2014) Learning by supplying. Strategic Management Journal 35(2): 204–223.

Angrist

Pischke

(2009) Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton NJ: Princeton University Press.

Athey

Coey

Levin

(2013) Set-asides and subsidies in auctions. American Economic Journal: Microeconomics 5(1): 1–27.

Azoulay

Zivin

JSG

, et al. (2019) Public R&D investments and private-sector patenting: Evidence from NIH funding rules. The Review of Economic Studies 86(1): 117–152.

Blanchflower

Wainwright

(2005) An analysis of the impact of affirmative action programs on self-employment in the construction industry (No. w11793). National Bureau of Economic Research.

Boland

Godsell

(2021) Bureaucratic discretion and contracting outcomes. Accounting, Organizations, and Society 88: 101173.

Bruce

de Figueiredo

Silverman

(2019) Public contracting for private innovation: Government capabilities, decision rights and performance outcomes. Strategic Management Journal 40(4): 533–555.

Calvo

Cui

Serpa

(2019) Oversight and efficiency in public projects: A regression discontinuity analysis. Management Science 65(12): 5651–5675.

Chatterji

Chay

Fairlie

(2013) The impact of city contracting set-asides on black self-employment and employment. Journal of Labor Economics 32(3): 507–561.

10.

Clark

Huckman

Staats

(2013) Learning from customers: Individual and organizational effects in outsourced radiological services. Organization Science 24(5): 1539–1557.

11.

Congressional Research Service (2008) The Federal Fiscal Year. Report No. 98-325, June 17, 2008.

12.

Congressional Research Service (2016) Set-Asides for Small Businesses: Legal Requirements and Issues. Report R42981, Jan 28, 2016.

13.

Coviello

Guglielmo

Spagnolo

(2018) The effect of discretion on procurement performance. Management Science 64(2): 715–738.

14.

Crama

De Reyck

Taneri

(2017) Licensing contracts: Control rights, options, and timing. Management Science 63(4): 1131–1149.

15.

Deshpande

Swaminathan

(2020) Introduction to the special issue on responsible operations and supply chain management. Manufacturing & Service Operations Management 22(6): 1107–1111.

16.

Ergas

(1987) Does technology policy matter. Technology and Global Industry: Companies and Nations in the World Economy 191: 245.

17.

Ethiraj

Kale

Krishnan

, et al. (2005) Where do capabilities come from and how do they matter? A study in the software services industry. Strategic Management Journal 26(1): 25–45.

18.

FAR Part 13. Simpliied Acquisition Procedures. Available at: www.acquisition.gov/far/part-13 (accessed 31 July 2023).

19.

FAR Part 19. Small Business Programs. Available at: www.acquisition.gov/far/part-19 (accessed 31 July 2023).

20.

FAR Part 35. Research and Development Contracting. Available at: www.acquisition.gov/far/part-35 (accessed 31 July 2023).

21.

Forbes (2022) How Small Businesses Drive the American Economy. Available at: https://www.forbes.com/sites/forbesbusinesscouncil/2022/03/25/how-small-businesses-drive-the-american-economy/?sh=228f0c944169 (accessed 31 July 2023).

22.

GAO (2017) Budget Uncertainty and Disruptions Affect Timing of Agency Spending. Washington, DC: US Government Accountability Office.

23.

Gavetti

Levinthal

Rivkin

(2005) Strategy making in novel and complex worlds: The power of analogy. Strategic Management Journal 26(8): 691–712.

24.

Giuffrida

Raiteri

(2021) Buyers’ workload and R&D procurement outcomes: Evidence from the US Air Force Research Lab (No. 21-059). ZEW Discussion Papers.

25.

Goodloe

(2018) 4 Keys to Success in Government Contracting. Tampa Bay Small Business Development Council. Available at: https://sbdctampabay.com/4-keys-to-success-in-government-contracting/ (accessed 29 January 2024).

26.

GovCon Chamber of Commerce (2021) How Small Businesses Develop Agency Relationships for Government Contracting Sales. Available at: https://www.govconchamber.com/blog/how_small_businesses_develop_relationships_for_government_contracting (accessed 29 January 2024).

27.

Haughey

(2020) The Federal Budget Timeline and Process. Available at: https://fiscalnote.com/blog/14-steps-to-the-federal-budget-timeline (accessed 15 October 2021).

28.

Huckman

Pisano

(2006) The firm specificity of individual performance: Evidence from cardiac surgery. Management Science 52(4): 473–488.

29.

Huckman

Staats

(2011) Fluid tasks and fluid teams: The impact of diversity in experience and team familiarity on team performance. Manufacturing & Service Operations Management 13(3): 310–328.

30.

Hyndman

Parmeter

(2015) Efficiency or competition? A structural econometric analysis of Canada's AWS auction and the set-aside provision. Production and Operations Management 24(5): 821–839.

31.

Kalkanci

Rahmani

Toktay

(2019) The role of inclusive innovation in promoting social sustainability. Production and Operations Management 28(12): 2960–2982.

32.

Levinthal

(1997) Adaptation on rugged landscapes. Management Science 43(7): 934–950.

33.

Marion

(2009) How costly is affirmative action? Government contracting and California's proposition 209. The Review of Economics and Statistics 91(3): 503–522.

34.

Mishra

Das

Murray

(2016) Risk, process maturity, and project performance: An empirical analysis of US federal government technology projects. Production and Operations Management 25(2): 210–232.

35.

Nakabayashi

(2013) Small business set-asides in procurement auctions: An empirical analysis. Journal of Public Economics 100: 28–44.

36.

Office of Federal Procurement Policy (2019) Myth-Busting #4—Strengthening Engagement with Industry Partners through Innovation Business Practices. Executive Office of the President, Office of Management and Budget. Washington, DC, United States.

37.

Rathje

Katila

(2021) Enabling technologies and the role of private firms: A machine learning matching analysis. Strategy Science 6(1): 5–21.

38.

Schilling

Vidal

Ployhart

, et al. (2003) Learning by doing something else: Variation, relatedness, and the learning curve. Management Science 49(1): 39–56.

39.

Small Business Administration (2019) Small Businesses Generate 44 Percent of U.S. Economic Activity. Release No. 19-1 ADV.

40.

Warren

(2014) Contracting officer workload, incomplete contracting, and contractual terms. The RAND Journal of Economics 45(2): 395–421.

41.

Wooldridge

(2010) Econometric analysis of cross section and panel data. Cambridge, MA, USA: MIT Press.

42.

World Bank (2016) Benchmarking Public Procurement 2017: Assessing Public Procurement in Regulatory Systems in 180 Economies. Washington, DC: World Bank.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.28 MB

	Dependent variable: schedule overrun
	(9)	(10)	(11)	(12)	(13)
	Controlling for	PSM using	PSM using	Controlling for	Early in Fiscal
	Past Performance	Firm Revenues	Firm Employees	Agency × Year	Year (Revised)
Main effects
${\hat{SA}}_{i}$	−0.398 (0.101)***	−0.356 (0.096)***	−0.349 (0.097)***	−0.328 (0.086)***	−0.318 (0.087)***
Same-Agency Exp.	−0.014 (0.008)	−0.011 (0.009)	−0.015 (0.010)	−0.015 (0.013)	−0.010 (0.009)
Different-Agency Exp.	0.014 (0.012)	0.010 (0.010)	0.014 (0.013)	0.014 (0.012)	0.011 (0.010)
Early in Fiscal Year	0.100 (0.042)**	0.059 (0.036)	0.067 (0.037)*	0.084 (0.035)**	0.052 (0.034)
Moderating effects
${\hat{SA}}_{i} \times$ Same-Agency Exp.	0.026 (0.018)	0.035 (0.018)*	0.032 (0.019)	0.030 (0.023)	0.022 (0.018)
${\hat{SA}}_{i} \times$ Different-Agency Exp.	−0.102 (0.052)**	−0.102 (0.049)**	−0.098 (0.051)*	−0.086 (0.049)*	−0.078 (0.045)*
${\hat{SA}}_{i} \times$ Early in Fiscal Year	−0.151 (0.056)***	−0.111 (0.054)**	−0.129 (0.057)**	−0.146 (0.050)***	−0.111 (0.050)**
Control variables
Task Type (0: Res, 1: Dev)	0.344 (0.264)	0.336 (0.191)*	0.331 (0.190)*	0.449 (0.196)**	0.321 (0.189)*
Planned Contract Duration	−0.126 (0.011)***	−0.115 (0.010)***	−0.115 (0.010)***	−0.116 (0.010)***	−0.115 (0.010)***
Planned Contract Cost	−0.033 (0.023)	0.076 (0.012)***	0.074 (0.012)***	−0.010 (0.018)	0.072 (0.012)***
Contractor Firm Size	0.076 (0.012)***	−0.013 (0.021)	0.004 (0.014)	0.073 (0.012)***	−0.010 (0.018)
Contract Pricing	0.089 (0.039)**	0.096 (0.035)***	0.098 (0.035)***	0.102 (0.032)***	0.103 (0.032)***
Competition	−0.011 (0.005)**	−0.007 (0.004)	−0.006 (0.004)	−0.013 (0.004)***	−0.012 (0.004)***
Oversight	0.096 (0.023)***	0.079 (0.020)***	0.078 (0.020)***	0.081 (0.019)***	0.076 (0.019)***
Small Business	0.188 (0.053)***	0.169 (0.048)***	0.170 (0.049)***	0.148 (0.043)***	0.158 (0.042)***
Average Past Performance	0.007 (0.005)
Federal Agency FE	Yes	Yes	Yes	Yes	Yes
Region FE	Yes	Yes	Yes	Yes	Yes
Year FE	Yes	Yes	Yes	Yes	Yes
R&D Contract Code FE	Yes	Yes	Yes	Yes	Yes
Contractor Firm Cluster FE	Yes	Yes	Yes	Yes	Yes
Constant	−0.502 (0.249)**	−0.423 (0.141)***	−0.295 (0.210)	−0.403 (0.215)*	−0.429 (0.226)*
N	27211	24882	24852	30902	30902
	Dependent variable: cost overrun
	(14)	(15)	(16)	(17)	(18)
	Controlling for	PSM using	PSM using	Controlling for	Early in Fiscal
	Past Performance	Firm Revenues	Firm Employees	Agency × Year	Year (Revised)
Main effects
${\hat{SA}}_{i}$	−0.398 (0.101)***	−0.101 (0.020)***	−0.100 (0.020)***	−0.328 (0.086)***	−0.083 (0.017)***
Same-Agency Exp.	−0.014 (0.008)	−0.003 (0.001)**	−0.003 (0.002)*	−0.015 (0.013)	−0.003 (0.001)*
Different-Agency Exp.	0.014 (0.012)	0.003 (0.002)	0.005 (0.004)	0.014 (0.012)	0.003 (0.003)
Early in Fiscal Year	0.100 (0.042)**	0.012 (0.007)	0.015 (0.007)**	0.084 (0.035)**	0.017 (0.009)*
Moderating effects
${\hat{SA}}_{i} \times$ Same-Agency Exp.	0.026 (0.018)	0.007 (0.003)**	0.005 (0.003)	0.030 (0.023)	0.005 (0.003)
${\hat{SA}}_{i} \times$ Different-Agency Exp.	−0.102 (0.052)**	−0.020 (0.011)*	−0.024 (0.012)**	−0.086 (0.049)*	−0.017 (0.010)*
${\hat{SA}}_{i} \times$ Early in Fiscal Year	−0.151 (0.056)***	−0.017 (0.011)†	−0.022 (0.011)**	−0.146 (0.050)***	−0.030 (0.013)**
Control variables
Task Type (0: Res, 1: Dev)	0.344 (0.264)	0.055 (0.042)	0.054 (0.043)	0.449 (0.196)**	0.055 (0.043)
Planned Contract Duration	−0.126 (0.011)***	0.005 (0.001)***	0.005 (0.002)*	−0.116 (0.010)***	0.004 (0.001)***
Planned Contract Cost	−0.033 (0.023)	0.006 (0.003)**	0.005 (0.003)**	−0.010 (0.018)	0.005 (0.003)*
Contractor Firm Size	0.076 (0.012)***	0.005 (0.003)	−0.004 (0.002)**	0.073 (0.012)***	0.004 (0.003)
Contract Pricing	0.089 (0.039)**	0.027 (0.009)***	0.029 (0.009)***	0.102 (0.032)***	0.031 (0.008)***
Competition	−0.011 (0.005)**	−0.006 (0.001)***	−0.005 (0.001)***	−0.013 (0.004)***	−0.004 (0.001)***
Oversight	0.096 (0.023)***	0.007 (0.004)**	0.008 (0.004)**	0.081 (0.019)***	0.007 (0.004)**
Small Business	0.188 (0.053)***	0.043 (0.011)***	0.041 (0.011)***	0.148 (0.043)***	0.036 (0.009)***
Average Past Performance	0.007 (0.005)
Federal Agency FE	Yes	Yes	Yes	Yes	Yes
Region FE	Yes	Yes	Yes	Yes	Yes
Year FE	Yes	Yes	Yes	Yes	Yes
R&D Contract Code FE	Yes	Yes	Yes	Yes	Yes
Contractor Firm Cluster FE	Yes	Yes	Yes	Yes	Yes
Constant	−0.502 (0.249)**	−0.044 (0.020)**	−0.052 (0.039)	−0.403 (0.215)*	−0.091 (0.043)**
N	27211	24882	24852	30902	30902

	Dependent variable: schedule overrun				Dependent variable: cost overrun
	(19)	(20)	(21)	(22)	(23)	(24)	(25)	(26)
Main effects
${\hat{SA}}_{i}$	−0.463 (0.084)***	−0.511 (0.102)***	−0.444 (0.084)***	−0.487 (0.107)***	−0.101 (0.023)***	−0.096 (0.023)***	−0.094 (0.023)***	−0.090 (0.024)***
Same-Agency Exp.	0.010 (0.007)	0.012 (0.066)	0.011 (0.007)	0.035 (0.070)	0.004 (0.001)***	−0.005 (0.026)	0.004 (0.002)**	−0.000 (0.026)
Related-Agency Exp.	0.035 (0.017)**	0.691 (0.266)***	0.034 (0.017)**	0.702 (0.280)**	0.008 (0.003)***	0.065 (0.037)*	0.008 (0.003)***	0.065 (0.036)*
Unrelated-Agency Exp.	−0.040 (0.021)*	−0.225 (0.126)*	−0.043 (0.021)**	−0.260 (0.130)**	−0.020 (0.006)***	−0.050 (0.032)	−0.021 (0.007)***	−0.058 (0.034)*
Early in Fiscal Year	−0.012 (0.007)	−0.012 (0.006)*	0.092 (0.069)	0.109 (0.067)	0.003 (0.002)	0.002 (0.002)	0.037 (0.014)***	0.034 (0.013)***
Moderating effects
${\hat{SA}}_{i} \times$ Same-Agency Exp.		−0.048 (0.099)		−0.110 (0.184)		0.001 (0.032)		−0.005 (0.033)
${\hat{SA}}_{i} \times$ Related-Agency Exp.		−0.747 (0.270)***		−0.795 (0.001)***		−0.071 (0.041)*		−0.072 (0.040)*
${\hat{SA}}_{i} \times$ Unrelated-Agency Exp.		0.933 (0.575)*		1.046 (0.003)***		0.174 (0.101)*		0.202 (0.106)*
${\hat{SA}}_{i} \times$ Early in Fiscal Year			−0.135 (0.081)*	−0.158 (0.079)**			−0.045 (0.016)***	−0.041 (0.015)***
Control variables
Task Type (0: Res, 1: Dev)	0.643 (0.257)**	0.559 (0.277)**	0.676 (0.264)**	0.587 (0.285)**	0.138 (0.071)*	0.130 (0.065)**	0.149 (0.076)**	0.138 (0.069)**
Planned Contract Duration	−0.137 (0.017)***	−0.138 (0.017)***	−0.136 (0.017)***	−0.136 (0.017)***	0.004 (0.003)	0.004 (0.002)	0.004 (0.003)*	0.004 (0.003)*
Planned Contract Cost	0.065 (0.018)***	0.068 (0.018)***	0.065 (0.018)***	0.067 (0.018)***	0.007 (0.004)*	0.008 (0.004)*	0.007 (0.004)*	0.007 (0.004)*
Contractor Firm Size	−0.015 (0.033)	−0.012 (0.034)	−0.015 (0.034)	−0.013 (0.035)	0.002 (0.004)	0.003 (0.004)	0.002 (0.004)	0.003 (0.004)
Contract Pricing	0.107 (0.026)***	0.093 (0.029)***	0.105 (0.026)***	0.087 (0.030)***	0.037 (0.009)***	0.037 (0.007)***	0.037 (0.009)***	0.036 (0.007)***
Competition	−0.016 (0.003)***	−0.017 (0.003)***	−0.016 (0.003)***	−0.017 (0.003)***	−0.007 (0.001)***	−0.007 (0.001)***	−0.007 (0.001)***	−0.007 (0.001)***
Oversight	0.115 (0.037)***	0.116 (0.039)***	0.117 (0.036)***	0.119 (0.038)***	0.015 (0.008)*	0.016 (0.009)*	0.016 (0.008)*	0.016 (0.008)**
Small Business	0.233 (0.059)***	0.238 (0.064)***	0.234 (0.059)***	0.240 (0.066)***	0.046 (0.015)***	0.047 (0.015)***	0.046 (0.014)***	0.047 (0.014)***
Region FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Year FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
R&D Contract Code FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Contractor Firm Type FE	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Constant	−0.381 (0.151)**	−0.230 (0.180)	−0.442 (0.172)**	−0.293 (0.203)	−0.070 (0.042)*	−0.059 (0.047)	−0.090 (0.052)*	−0.076 (0.056)
N	16870	16870	16870	16870	16870	16870	16870	16870