“Run Forrest Run!”: Measuring the Impact of App-Enabled Performance and Social Feedback on Athletic and Usage Outcomes

2.1. Performance and Social Feedback Studies in Operations

The role of performance and social feedback in the general stimulation of task efficacy and productivity in various settings has interested OM and management researchers. Apart from the factory setting (Awaysheh et al., 2023; Schultz et al., 2003; Zhang et al., 2022), OM researchers have also examined the impact of feedback provision in the crowdsourcing settings (Chan et al., 2021; Jiang and Wang, 2020), digital platform settings (Zhu et al., 2024), knowledge transfer settings (Letmathe et al., 2012) and even the automotive driving context (Choudhary et al., 2021). Past works have found mixed results with the provision of feedback. For instance, (Gardner*, 2020) found that in the factory setting, feedback on work quality improves production yield while providing feedback about working speed worsens it. However, Zhang et al. (2022) discovered a different result in a similar setting, wherein feedback about quality did not significantly impact productivity, whereas feedback on quantity did. The disparity in results can come from the differences in the tasks performed, indicating the need for OM research to examine the effects of performance feedback in different contexts closely. Jung et al. (2021) summarized past works studying performance feedback in various operational settings. We add to this body of work by examining the role of performance feedback in the relatively understudied context of athlete development.

Social feedback generated by peers or one’s social network in work settings in the form of peer mentoring (Chun et al., 2022), social support (Yan, 2020, 2018) and peer feedback (Zhang et al., 2023) in non-work settings to improve outcomes is of interest in past works. We extend these works by exploring the digital, auditory, and contemporaneous delivery of feedback instead of conventional offline feedback delivery, which is less immediate and less scalable.

Recently, OM literature has begun to examine the value of contemporaneous feedback in end-user/individual settings like driving (Choudhary et al., 2021), focusing on the nuances of positive or negative feedback. We examine these effects in the unique non-work setting (i.e., physical training and exercising) to understand product operational efficiency towards the end-user instead of process efficiency, as seen in most work settings. This distinction matters as the motivation of workers in direct fiscal control of a firm is likely to differ from that of end-users of a digital app used for improving one’s physical performance. Lastly, we study the combinatorial efficacy of different feedback types. As Zhang et al. (2022) notes, the delivery of different forms of feedback may not necessarily lead to synergistic outcomes, so determining how these different forms will work together is a non-trivial question.

2.2. Motivational Processes in Athletic Development

Scholars in the field of athlete development often cite the importance of self-efficacy in motivating athletes to achieve performance goals. Feltz et al. (2008) provided an insightful and in-depth review of the role of self-efficacy in professional and non-professional sports. As Bandura (1977) notes, self-efficacy is not just about the ability to do a task but believing in oneself in executing it in a particular instance. Much of sports and athletics literature acknowledges the importance of self-efficacy (also called self-confidence or sport-confidence), even in seasoned athletes (Feltz et al., 2008). Self-efficacy may be influenced by both internal (e.g., mastery, demonstration of ability, and performance accomplishments in the past (Faulkner et al., 2011)) and external factors (e.g., verbal encouragement (from coaches and peers), social support (Gould et al., 1989), situational advantages).

In addition to self-efficacy, a review of the sports psychology literature reveals that self-enhancement is another influential motivational process that could spur athletic performance (Dejonge et al., 2019; Smith and Smoll, 1990). Self-enhancement involves the use of cognitive and/or behavioral processes to amplify the positive aspects of the self in one’s own eyes or in the eyes of others (Sommer, 2012). Self-enhancement is linked with a host of desirable outcomes, such as better psychological adjustment and well-being (Bonanno et al., 2005; Taylor et al., 2003), lower psychological distress after trauma (Gupta and Bonanno, 2010), and higher self-esteem and ego-resiliency (Paulhus et al., 2003), which are essential traits for athletic success. Within the sports context, social support through verbal (e.g., praises and cheering) and non-verbal (e.g., pat on the back) affirmations of the athletes are common sources of self-enhancement (DeFreese and Smith, 2013). With the possibility of delivering social support digitally, recent work has begun to examine the effectiveness of using social features on exercise apps to improve athletic performance (Tu et al., 2019).

Past work has shown that offline performance and social feedback can improve self-efficacy and self-enhancement in sports (Beattie et al., 2016). However, it remains unknown if online, real-time feedback would induce a similar impact on athletes. This link is not immediately apparent given that the online environment transmits fewer nonverbal and social cues, representing essential factors for the encouragement and persuasion process (Burgoon et al., 2002). Yet, there is growing interest in real-time strategies for providing feedback in sports (Van Hooren et al., 2020), as this would allow for various advantages, including scalability, remote coaching, and lowered costs. Early work utilizing small samples (Van Hooren et al., 2024) has shown that contemporary feedback via wearables and mobile devices has the potential to influence the bio-mechanics of how people run.

Next, we discuss the impact of performance and social feedback through the perspective of self-efficacy and self-enhancement motivation, respectively. Building on the insights of previous works like Zhang and Zhang (2022) and Yan (2020), we also discuss how and when heterogeneity may exist in this impact.

3.1. Enabling Self-Efficacy Motivation Via Performance Feedback

According to the self-efficacy theory, individuals’ assessment of personal capabilities is essential to behavioral regulation (Bandura, 1977). Self-efficacy motivation is found to maintain behavioral changes across various contexts, such as competitive sports, academic settings, and physical activities (e.g., Liu et al., 2020; Zhang and Zhang, 2022). Behaviors maintained through self-efficacy motivation have resulted in more adaptive outcomes, such as greater persistence in fitness training and improved psychological well-being. Under the predictions of the self-efficacy theory, the adoption and sustained usage of exercise apps can be motivated by maintaining individuals’ self-efficacy motivation. We argue that a link between in-exercise performance feedback and self-efficacy motivation is likely.

By informing users periodically of their time, distance, effort, or calorific expenditure during exercise activities, information on user progress brings attention to one’s fulfillment of intermediate goals (Amir and Ariely, 2008). By acknowledging users’ accomplishment of an intermediate milestone, performance feedback can bolster self-efficacy and even strengthen the regulation of habitual physical training (Aarts et al., 1997; Lally and Gardner, 2013). The strengthened self-efficacy that accompanies the positive reinforcement of making progress would not only induce initial app adoption but can also help to sustain usage behaviors over time (Tams, 2022). It does so by inducing the belief that the effort exerted in training (which comes with app usage) is a good decision and is associated with mastery experience (Bandura, 1977).

Similarly, under the nudging literature (Vlaev et al., 2016), performance feedback can be seen as a type of self-improvement technique that builds one’s determination to train and exercise harder. Such automated feedback on the running performance of users exemplifies the principles of increasing the ease and convenience of reminding users of achievements they have made through their exercise efforts (Sunstein, 2014), which likely motivates them towards completing their training goals and engaging in more future exercises. The positive link between future usage and self-efficacy motivation is also supported by the study of feedback designs in human–computer interaction literature, which found that making users aware of their intermediate achievement/progress can have the effect of spurring greater subsequent effort and instilling goal persistence (Baretta et al., 2019). In addition to inducing positive reinforcement, in-exercise performance feedback can alter perceived physical exertion (Tucker, 2009). Particularly relevant here is research from kinesiology, biomechanics, and sports sciences that has looked at distance-based performance feedback of the kind we study. Knowing how much has been run (via offline methods like feedback from coaches has been shown to influence the gait, pacing (Baggaley et al., 2017), vertical loading rate (Foster et al., 1994), and velocity (Faulkner et al., 2011) in the last spurt before stopping. This suggests that timely distance feedback may have advantages for seasoned runners in improving their running efficiency.

A unique aspect of performance feedback is that it is automated and deterministic and is routinely and systematically provided to users whenever they reach certain pre-determined physical activity milestones. Performance feedback received during the exercise may spur users to work harder if their perceived progress does not match their actual progress. This way, systematic feedback acknowledging user progress can invoke persistent and sustained app usage and training behaviors. Thus, performance feedback’s routine and predictable nature can cultivate user commitment toward long-term performance goals that might seem cognitively distant or difficult to achieve at first glance (Locke et al., 1988; Dorris et al., 2012). In sum, the in-exercise feature of providing performance feedback is likely to increase app usage (both short-term and long-term), which in turn enhances athletic performance.

3.2. Enabling Self-Enhancement Via Social Feedback

While self-efficacy motivation comes from one’s recognition of personal ability in performing the activity, self-enhancement motivation tends to be driven by social recognition of one’s performance. Originating from social psychology (Fiske, 2004) and social identity perspectives (Tajfel and Turner, 2004), the self-enhancement theory holds that people are motivated to strategically present themselves in ways that boost their ego in social settings (Shrauger, 1975). Sports psychologists have shown that individuals driven mainly by identity enhancement are predominately motivated by ego reinforcement to keep up with regular physical training (e.g., Dejonge et al., 2019). An in-exercise social feedback feature that facilitates user-to-user interactions by allowing connected users to leave real-time feedback on exercising users’ activities during their training will likely enhance the focal user’s sense of personal worth. Such a social feedback feature involves the dual processes of self-presentation and feedback generation, occurring repetitively in a cycle. The self-presentation process involves disclosing one’s activities based on the notions of self-concept, social roles, and the preferences of the intended audience (Kerner and Goodyear, 2017; Leary and Kowalski, 1990). Individuals are motivated to create a polished and glorified image for others (Foot, 2001).

Actual and anticipated reactions to one’s image influence subsequent self-presentation activities. In his seminal work on self-presentation theory, Goffman (1959) describes social life as a series of performances in which individuals engage in self-presentation activities to advance their ego. Specifically, Goffman explains that the audience’s anticipated or imagined reactions serve as feedback on whether self-presentation activities have severed the expected ego-boosting objectives. In the context of exercise apps, positive social feedback generated during the training session (e.g., in the form of praises and encouragements) serves as a source of ego enhancement, encouraging further episodes of self-presentation via the app feature. Evidence of such self-presentation concerns in influencing exercise behavior is present in traditional settings where offline social feedback is delivered (Hausenblas et al., 2004).

It is worth noting that the ego-building effect of self-presentation could vary with changing audience acceptance. Initial self-presentation typically enhances individuals’ worth since the audience would likely find it informative, as it often helps them reduce relational uncertainty and develop an understanding of the individuals. Evidence suggests that self-presentation is essential for gaining recognition in initial social interactions. For instance, in a study examining self-presentations on LinkedIn, Tifferet and Vilnai-Yavetz (2018) found that users exercised prudence when selecting photo portraits and displaying profile information that projected professionalism, personal uniqueness, and attractiveness, as these were crucial for managing the initial impression of potential recruiters. Similarly, in work settings, Brown et al. (2016) opined that individuals should pay special attention to initial email communications, which could powerfully influence co-workers’ preliminary impressions of individuals. Under this understanding, the initial use of the social feedback feature is likely accompanied with an increase in social feedback provided by other users, which would spur more app usage.

From a nudging perspective, social feedback nudges users to participate in exercising behaviors through the provision of verbal incentives from peers (Vlaev et al., 2016). This social feedback serves as an additional source of information that affirms users’ decisions and efforts to train (Sunstein, 2014). However, the audience might find repetitive self-presentation irrelevant and annoying in the long run. Evidence suggests that audiences typically find repetitive self-presentation trivial and, at times, irritating (e.g., Tu et al. (2018)). For instance, Krasnova et al. (2017) found that although Facebook friends were initially attracted to follow users’ status updates, friends could eventually become irritated by users’ persistent self-presentation in postings, as these updates make them feel inferior and depressed. Likewise, Vaterlaus et al. (2015) reported that frequent workout postings could be perceived as a form of showiness on online social networks, threatening users’ desired images. Furthermore, persistent sharing of users’ ability to complete physical exercises could irritate the audience because they could be belittled or shamed by their counterparts’ physical competence. Given the in-exercise nature of social feedback, it can sometimes be less convenient to provide such feedback, as the timings of the exercising user and their peers might not be aligned. As such, other users’ social feedback to the exercising user might decrease with time, either due to disinterest, annoyance, or simple inconvenience. A decline in social feedback over time would also mean that the extrinsic motivation to use the app decreases, leading to lower usage and exercise frequency. Accordingly, athletic performance will increase and fall with the level of social feedback received over time.

3.3. Interactions Between Self-Efficacy and Self-Enhancing Motivations

A prominent view is that social feedback via praises and encouragement can amplify the effect of self-efficacy (Olivera et al., 2008; Krasnova et al., 2015). Past research has offered several theoretical mechanisms to account for the reinforcement effects of social feedback. One such explanation is derived from cognitive evaluation theory, which suggests that public acknowledgments of one’s capabilities through social feedback can effectively complement one’s private recognition of one’s task capability, that is, self-efficacy. Not only do public affirmations create positive moods, make individuals feel good about themselves, and bolster their ego (Delin and Baumeister, 1994), but more importantly, the self-enhancement aspect of social feedback provides an external verification of individuals’ capabilities, which galvanizes their self-efficacy that had been derived through performance feedback. Like the offline support coaches and peers can offer from the sidelines to athletes building self-efficacy (Feltz et al., 2008), the same could happen with input from geographically distant but digitally connected others. Such reasons might lead companies and app developers to believe that the simultaneous offering of both types of feedback can produce complementary effects, with the thinking that social feedback from friends and other runners can reinforce self-efficacy. However, we should also note that our understanding of the effect of social feedback on self-efficacy motivation is far from conclusive. Past research suggests that the joint presence of self-efficacy and self-enhancement motivation do not always produce synergistic effects in terms of maintaining user behaviors (e.g., Cannon and Rucker, 2022). In some instances, the concurrent provision of performance and social feedback can bolster ego to the point of contentment or even overconfidence. The literature suggests that overconfidence is often accompanied by an upshift in positive affect, which signals that a goal or sub-goal has been partially attained, resulting in decreased effort in subsequent periods (Carver and Scheier, 1990). In particular, when performance feedback is consistent with the established anchors, individuals would consider it as additional confirmation of their abilities, which would galvanize self-efficacy and self-enhancement.

The selective accessibility model explains how social and performance feedback can jointly contribute to overconfidence. The model demonstrates that individuals draw on the most self-relevant information as the reference for ability evaluation (Mussweiler and Strack, 1999). The reference is then utilized to interpret other ability-relevant information through assimilation or contrast (Gerber et al., 2018). According to the model, individuals typically assimilate with additional favorable information but contrast themselves away from undesirable ones. In this study, praises and encouragements from friends and fellow users can be relatively more self-relevant to the running user than automated performance feedback provided by an app, given that the source of social feedback may use specific verbal motivational cues known to spur the recipient user on. Having established the belief that one has a good running ability through social feedback, users are likely to focus on corroborating performance feedback while focusing less on contradicting performance feedback. The selective assimilation and avoidance of information produce overconfident behavior. Finally, when delivering multiple forms of frequent feedback concurrently, especially via audio, the possibility of annoyance must be mentioned. As Van Hooren et al. (2020) points out, when only performance feedback is provided, it can be controlled, but real-time social feedback is determined by others, raising the possibility of it clashing with the timing of performance feedback, leading to irritation. With these various theoretical possibilities, it is unclear how the joint use of the two feedbacks might impact usage outcomes.

4.1. Study Context

To study our research questions, we collaborated with a national service organization to launch a field study involving 1,037 members of the uniformed service from October 2014 to April 2016. Upon reaching a set age in the country of our study context, all male able-bodied citizens, regardless of demographic features, are conscripted for national service, which consists of uniformed organizations like the military, police, or civil defense force. This conscription process is advantageous to our research objectives, ² as the mandatory enrollment of a nation’s male population produces a sample of subjects from all walks of life. This can reduce some of the concerns of sampling biases that are inherent in user studies where the opt-in process by subjects induces a selection effect. Other advantages of using a uniformed service organization as a study context include high treatment compliance, impartial periodic measurement of behavioral and physiological outcomes, and reduced cross-contamination between the study groups (details to follow).

After their basic training, these conscripted individuals are released from their full-time service and transition to part-time status, in which they return to their normal civilian lifestyles and are only required to perform reservist duties for a short period of time each year (no more than 40 days each year). By focusing on such part-time service, we get a relatively generalizable pool of subjects with activity levels and occupations comparable to the underlying young male population of the region.

To assess their level of physical fitness, servicemen in the organization undergo a compulsory physical fitness test twice each year. The Physical Fitness Test (abbreviated as PFT henceforth) systematically tracks exercise outcomes related to running and strength training. One of the assessments within the PFT is a timed 2.4 km run. Through our collaboration with the organization, we could observe the test run times of the subjects before and after the introduction of the study interventions. Specifically, we observed five of the latest test timings prior to the start of our study interventions and three test timings after the study interventions were initiated. Each participant took the PFT within a six-month window during the study period, allowing us to track the athletic performance and usage levels of subjects over an extended period of eighteen months post-intervention. ³

Specifically, the study outcomes of interest are the test run times and the app usage frequency (in the post-intervention period). These standardized fitness tests allowed us to arrive at impartial measurements of a subject’s physical performance over time. It is well-known that the very act of measuring and observing outcomes in study settings can influence the outcome itself (McCambridge et al., 2014). This was less of a concern in our study, as the test utilizes a pre-existing, uniform, incentive structure that financially rewards servicemen based on their performance, such that each serviceman would be inclined to perform as well as they possibly could when taking the physical test. Under organizational regulations, servicemen who do not pass this mandatory fitness test have to undergo remedial training sessions, which take additional time out of their daily lives. Also measured in these tests are physiological measures such as height, weight, and body mass index (BMI). The standardized nature of executing the tests and timekeeping also help to ensure that the run times of study subjects were measured in a highly consistent fashion. The recorded running time for the PFT (measured in seconds) is our primary outcome of interest.

4.2. Study Design

We used the popular application, https://www.pcmag.com/reviews/endomondo-for-iphoneEndomondo (retired in 2020), to deliver various app-based interventions to our study subjects. There are several reasons for choosing this particular app. First, it is a widely used exercise app that has features that are representative of other running apps in the market. Second, Endomondo allows for the delivery of progress and social feedback during exercise through its Audio Coach and Pep Talk features, allowing for these respective feedbacks to be delivered during an exercise episode. Furthermore, the application also allowed a LiveTracking feature that allows the live broadcasting of a focal user’s run to his/her social network. This enables peers to be aware of the user’s training efforts in real-time and provides opportunities for synchronous social feedback. Friends of exercising individuals are alerted under the live tracking feature, encouraging social feedback to be provided in time windows close to or during each individual’s exercises. Finally, it is possible to manipulate the availability of these features by turning them on and off, which allows for the creation of different intervention groups that have access to certain features. The reader can see the features of the app via the Internet Archive’s entries of the Google Playstore page of Endomondo https://web.archive.org/web/20141120183110/https://play.google.com/store/apps/details?id=com.endomondo.android here.

Under the organization’s hierarchy, each unit has ∼ 300 individuals and consists of six sub-units that are made up of ∼  50 soldiers each ( μ = 51.7 in our sample). All sub-units within a unit train on similar schedules and are under the command of a single commander who determines their training regimen. With the cooperation of these commanders of four units, we assign differential treatments to different sub-units within a unit. Specifically, we requested that four sub-units within a unit be assigned to one of the four main Endomondo treatment conditions (i.e., App-only, PF-only, SF-only, and Joint). We also requested that one sub-unit within each unit be assigned to a baseline condition with no treatment (No-App) and the last sub-unit of each unit to have a placebo-app treatment condition (see Figure EC.1.1 in the EC for a diagram of the assignment). The individuals in the placebo condition are only used as a robustness check (see Subsection 5.1.7) and are not considered in any of the main analyses of the article. Since members of the same sub-unit were geographically co-located in their training sessions, they were assigned to the same treatment condition to reduce the possibility of cross-contamination.

The five different conditions in our study are described in Table 1, a total of 1,037 individuals are considered in the main analyses (out of the 1,241 total individuals) across the four units. ⁴

Our study procedure was as follows. The servicemen within each sub-unit were briefed before the commencement of treatment. In the briefing, the servicemen in the treatment groups (control, PF, SF, and SF&PF) were asked to download the running app, Endomondo, and were instructed to turn on the feature(s) reflecting their manipulation group and to use the app in their personal training. Here, the treated users are asked to turn on the main study features (i.e., PF and SF) along with the basic features of Endomondo, while the users are asked to turn on only the basic features. To encourage app usage, all servicemen were given exercise armbands to hold their mobile devices for their runs.

Our study participants only performed collective training with other servicemen of their unit for 20 days over the span of the whole study period (18 months). As part-time servicemen who spent most of their time outside the uniformed organization, the subjects conduct individual training and exercises on days other than those spent on collective training in preparation for their fitness tests. Given that the subjects’ use of the Endomondo app took place largely during the time they were not physically co-located, contamination across the various conditions is very unlikely. The servicemen stayed in their sub-units throughout the post-treatment period (18 months), which further prevented contamination issues. At the end of each study wave, officers randomly checked the servicemen’s apps to see if they followed the instructions for turning on and using the assigned app feature(s). These were spot checks, conducted randomly, in which not all participants would be checked, and the subjects were not pre-informed of. These spot checks were helpful in understanding if the study contained non-compliant behavior. Subjects were free to use the app as frequently or infrequently as they saw fit for their own exercise sessions, much like users do in civilian settings. Receiving social feedback in a non-social study condition might be a concern for such a self-regulated intervention delivery. Based on the data on social interactions, we found no individual outside the social condition had any record of receiving social feedback.

We also administered a post-treatment survey (delivered over two weeks) a week after the final wave finished in three batches to randomly selected subjects from each study condition with the Endomondo app. This was done to minimize collusion behaviors in the responses within each sub-unit. Each subject was surveyed only once. By the end of the two weeks, all 825 participants who had used Endomondo were surveyed. The questions measured each user’s awareness of features available in his assigned condition. Table EC.1.2 in the EC contains the survey questions for each of these constructs. All subjects in the performance (PF) and in the social conditions (SF) successfully perceived their intended manipulations. Figure EC.1.2 in the EC shows the details of the overall timeline of each participant in the study.

Table EC.1.3 in the EC outlines the descriptive statistics for subjects in our sample across the manipulation conditions. The average serviceman was about 30 years old, had an average annual income of 33,010 US dollars, and held a diploma degree.

Using standardized mean difference tests, we tested for differences in outcome variables and covariates between groups in the pre-treatment period. We see that the pre-treatment test run times are not significantly different across the treatment groups, which is a helpful indication that the various study groups are statistically comparable. We also conducted T-tests between the control and different treatment groups to check for systematic differences on multiple dimensions, which could pose a potential to our claim of random assignment of treatment. The T-test results are presented in the EC in Table EC.1.4, no significant differences were found in any behavioral or demographic dimensions on measures prior to treatment. A summary of the descriptive statistics of the variables used in our analysis is presented inTable 2.

We adopt a difference-in-difference strategy in analyzing the effects of the various interventions. The treatment groups in our study refer to the groups that received the Endomondo app, and groups that were able to use the PF and SF features, in any combination. The baseline group does not use the Endomondo app. This design produces independent treatment groups, by which the interventions from one group do not affect another treatment group or the control group, that is, these app-based interventions are independent exogenous shocks applied to different groups of servicemen. Our data also consists of the test performance of servicemen in five periods before administering the various treatments, allowing us to contrast the within-self change in outcomes of the servicemen in each treatment group with that in outcomes of servicemen in the untreated group. Furthermore, the PF-only, SF-only, and joint treatment groups are independently stacked onto the app-only condition, we measure the difference-in-difference estimate of each intervention as follows:

l o g ( R u n T i m e i , t ) = β 1 ⋅ A p p i , t + β 2 ⋅ A p p i , t * P F i + β 3 ⋅ A p p i , t * S F i + β 4 ⋅ A p p i , t * P F i * S F i + η i + γ t + ε i , t .

(1)

A key assumption of any study using a differences-in-differences estimation strategy for identification is that the trends in the dependent variable are parallel between the treated and the control groups (Angrist and Pischke, 2008). The parallel trends assumption, is needed to ensure that the treated condition is not already showing deviations from the control group before treatment is administered, which would otherwise signify that other effects other than the assigned treatment are at work. The parallel trend assumption can only be validated for the pre-treatment period and is examined visually, as well as tested frequently using a relative time lags-leads model Autor (2003), which we present in Subsection 5.1.5. The per wave means of logged PFT runtimes of the treated (groups with the Endomondo application and solid-line) and that of the untreated (BaselineNoApp) are plotted over time in Figure EC.1.3 in the EC. As the figure shows, the pre-treatment model-free trends of the treated and the untreated individual means follow each other well before diverging in the post-treatment period. In the next section, we present the results of the formal estimations of the differences-in-differences model.

5.1. Impact of Performance and Social Feedback on Runtimes

Our primary model is as modeled in equation (1). All standard errors in regressions reported were clustered at the sub-unit level as sub-unit membership determined the treatment of individuals.

The strength of the social feedback treatment is likely contingent on the frequency with which subjects received messages from their peers on the app. To understand how this treatment intensity influences our analyses, we also consider a continuous measure of the frequency of social feedback received by an individual i in a wave t , S f F r q i , t , in place of S F i in equation (1). The resultant models from the same equation are shown in Table 3.

The coefficients of the difference-in-differences indicator, A p p i , t are interpreted as a percentage change in the performance test runtimes since our DV is logged. To assess the incremental improvement in users’ performance relative to their pre-treatment state, we calculated the average treatment effects of an individual being in each study condition. We do so by summing the coefficients of A p p and the corresponding interactions with treatment vectors in column (1) in Table 3. These were, a reduction of 4% ( β = − 0.04 , P v a l < 0.001 ) for PF, 3% ( β = − 0.03 , P v a l < 0.001 ) for SF, and 3.6% ( β = − 0.036 , P v a l < 0.001 ) ⁶ for the joint condition of SF and PF. Not only did the three conditions have differential impacts but more importantly, the joint usage of PF and SF produced non-additive impacts, this is further explored in the following subsections.

Next, we look at the results that involved the number of social feedback delivered S F f r q i , t as a regressor. Results of these analyses are presented in column (2) of Table 3. The signs and significance of the terms with SF frequency are qualitatively similar to those of their binary counterparts, though their magnitudes are much reduced. This is expected, given that this frequency regressor measures the impact arising from each additional social feedback received (per a wave of 6 months) and not the average effect of all social feedback received. For the interpretation of these results, we focus on the marginal impact of each additional social feedback received (above and beyond having the Endomondo app). Specifically, the receipt of each social feedback in the SF condition reduces the test runtime by 0.13% ( β = − 0.0013 , p v a l < 0.001 ) , while that received in the joint condition has no significant effect on average test run time above and beyond being in the performance condition.

5.1.3. Heterogeneity Across Users

While the above analyses have highlighted how different social feedback types can have varying impacts on users when used in conjunction with performance feedback, it is also possible these treatments had differential impacts on different types of subjects. We performed a moderation analysis based on a modified version of equation (1) to assess the potential for such heterogeneous effects. For each user, we noted the average running speed ( S p e e d ( 1 M ) ) across all recorded runs on the Endomondo app during their first month of usage. We used this as a measure of their inherent running ability, which is unlikely to be impacted by the app’s features within the first month of adoption. This measure moderated all treatment terms in the primary regression in (1). The results are presented in the EC in Table EC.1.7. The average marginal effects from these results are presented in Figure 1. We find that the performance feedback (PF) condition is effective in improving test outcomes across runners of varying initial running abilities, but its impact diminishes with increasing initial average running speeds. In fact, PF users with the fastest running speeds do not seem to differ from the control users in terms of their test outcomes. The other two conditions involving social feedback (SF) have a marginally increasing impact on the initial speed and are only effective for stronger runners. Interestingly, this analysis reveals that conditions involving social feedback tend to be effective only for stronger runners.

OPEN IN VIEWER

Figure 1.

Impact of treatment conditions as moderated by average running speed of the user in first month’s recorded exercises.

5.2. Impacts of Feedback Types on Usage Outcomes

After determining the impact of the various app features on physical runtimes, we now turn to a set of analyses related to the usage of the Endomondo app over time. Since this outcome is observable only in the treatment period, the subsequent analyses are restricted to the 18 months (three waves) when the treated subjects could use the Endomondo app. We first examine how usage frequency (i.e., the self-reported exercise episode frequency) fits into the relationship between the treatment conditions and runtime performance. This is done given the understanding that increased usage will likely provide more opportunities for users to be exposed to the different feedback instances commensurate to their treatment conditions.

5.2.1. Impact on Wave-Level App Usage

We first look at the model-free statistics of how app usage varies across each treatment condition. Figure EC.1.7 in the EC shows the app’s usage over time. While usage steadily increased in the PF & SF&PF conditions, it peaked in the second wave for SF.

To formally determine if the delivery of different kinds of feedback was associated with different frequencies of usage of the app, we performed regression analyses similar to our specifications in our earlier subsection. Specifically, we modified equation (1) by replacing the runtime dependent variable with the app usage frequency of a user i in a wave t . The new model is shown in (2).

U F i , t = β 1 ⋅ P F + β 2 ⋅ S F + β 3 ⋅ P F * S F + β 4 ⋅ l o g ( p a s t R u n t i m e i ) + β 5 ⋅ B M I ( h i s t ) i + ϵ i , t .

(2)

It should be noted that since there is no app usage for all users in the pre-treatment period, a fixed effects model cannot be applied here. To control for individual differences, we controlled for the log of the last, pre-treatment PFT runtimes and the three-year pre-treatment average BMI of each individual. Since variation in usage frequency is needed for the analysis, we only included the users who were endowed with the Endomondo app and their observations during the manipulation period for this set of analyses. The results of our analysis are presented in Table 6 in columns (1) and (2). Given that the dependent variable is of a count nature, our regressions are estimated with a Poisson specification.

OPEN IN VIEWER

Table 6.

Marginal impact of performance and social feedback on per-wave usage of Endomondo application.^a

	(1)	(2)
D V =	U F i , t	U F i , t
PF	0.409***	0.383***
	(0.033)	(0.032)
SF	0.224***
	(0.031)
PF*SF	−0.272***
	(0.061)
SFfrq		0.019***
		(0.002)
PF*SFfrq		−0.022***
		(0.004)
l o g ( p a s t R u n t i m e )	−0.469***	−0.462***
	(0.129)	(0.128)
BMI	−0.026	−0.031
	(0.018)	(0.018)
N	2,475	2,475
Wave FEs	Yes	Yes
AIC	37659.846	37628.313
BIC	37706.358	37674.825

Note: FEs = fixed effects; PF = performance feedback; SF = social feedback; BMI = body mass index; AIC: Akaike information criterion; BIC: Bayesian information criterion.

^aColumns 1 and 2 are estimated using Poisson specification.

Clustered standard errors in parentheses.

^* p < 0.05 , ^** p < 0.01 , ^*** p < 0.001 .

We found that both PF and SF conditions were significantly and positively associated with app usage frequency. While the PF condition saw nearly 50% ( β = 0.409 , P v a l < 0.001 ) more usage than those in the control app-only group; those in the SF group saw about 25% ( β = 0.224 , P v a l < 0.001 ) more usage than the control app-only group, and those in the joint condition had nearly 43% more usage than the control app-only group (calculated based on the addition of the three coefficients). Interestingly, this analysis also reveals a diminishing role played by social feedback when used jointly with performance feedback, as evidenced by the negative and significant interaction term of P F * S F . The direction and magnitudes in column(2) were consistent to those in column (1) when we consider social feedback frequency in place of the binary SF indicator.

To further explore how membership in a treatment group and usage levels vary over time, we modified equation (3) to replace the counts of feedback with the binary indicators PF&SF denoting the respective treatment groups (similar to column (1) in Table 3) and the interaction between the two. Weekly fixed effects were replaced by linear and quadratic terms to control for the time since the commencement of treatment. These terms interacted with all treatment vectors too. The model and its results are presented in the EC in Section EC.1.1.2. Here, we plot the predicted marginal effects of the same by different conditions to indicate how usage behavior manifested over time.

While the usage level in all treated conditions is higher than that in the app-only condition, we find heterogeneity over time. Specifically, the conditions delivering performance feedback show a sustained increase in app usage over time while the social feedback condition shows a U-shape (see Figure EC.1.8 in the EC), peaking earlier than others but eventually statistically indistinguishable from the no-feedback condition. This suggests that, in the long run, the delivery of social feedback alone might not be as effective as the otherconditions.

We next attempt to explain the mechanism of the observed main results. Specifically, we aim to provide reasons backed by empirical evidence to explain the lack of an additive effect between SF and PF. We perform a series of tests at finer levels to understand how receiving the different feedbacks might affect subsequent exercise proclivity and intensity within an exercise episode.

5.3.1. Impact on Subsequent Exercise Levels

First, we begin by understanding why the combination of PF and out-exercise SF can affect subsequent exercise frequency. Since social feedback received outside an exercise can only affect a user’s actions for future runs, it makes sense to look at how its receipt changes users’ tendency to exercise subsequently. To this point, we modeled the weekly usage frequency U s a g e F r e q i , w of user i in week w as a function of the feedback volumes received in the past week, namely S F f r q O u t i , w − 1 (out-exercise feedback frequency received the week before), P F f r q i , w − 1 (number of performance feedback messages received in the week before) and the interactions between the social feedback frequency and P F f r q i , w − 1 . We included individual fixed effects δ i and week fixed effects γ w . The model captures within-self differences between the behavior of the same person when feedback was received and when it was not received weekly. We also note that the app usage frequency will likely capture a partial view of the total exercise episodes performed by users, as it is possible for users not to use the app when they are exercising. That said, estimates in this analysis would be conservative and do not overestimate the treatmenteffects.

U s a g e F r e q i , w = β 1 ⋅ P F f r q i , w − 1 + β 2 ⋅ S F f r q O u t i , w − 1 + β 3 ⋅ S F f r q O u t i , w − 1 * P F f r q i , w − 1 + δ i + γ t + ϵ i , w .

(3)

The results from the estimation of equation (3) are reported in Table 7. Although the model is presented in a linear equation for simplicity, it is implemented as a Poisson regression, as the DV is a count of weekly exercise frequency.

OPEN IN VIEWER

Table 7.

Impact of last week’s feedback volumes on subsequent exercise reporting frequency.

D V = U s a g e F r e q ( w )	Poisson (1)
S F f r q O u t ( w − 1 )	0.051**
	(0.018)
P F f r q ( w − 1 )	0.049***
	(0.009)
S F f r q O u t ( w − 1 ) * P F f r q ( w − 1 )	−0.037*
	(0.018)
Intercept	0.475***
	(0.004)
N	61,050
User FEs	Yes
Week FEs	Yes
Pseudo R 2	0.158

Note: FEs = fixed effects; PF = performance feedback; SF = social feedback.

Clustered standard errors in parentheses.

^* p < 0.05 , ^** p < 0.01 , ^*** p < 0.001 .

The results in Table 7 indicate that the receipt of out-exercise feedback increases subsequent exercise levels ( β = 0.05 , p v a l < 0.01 ) and so did the receipt of performance feedback ( β = 0.05 , p v a l < 0.001 ) . Interestingly, the joint receipt of both types of social feedback with performance feedback was associated with a reduction in subsequent exercise frequency. This reinforces and explains our main findings from the wave-level analyses. Through this analysis, we learned that both out-exercise SF and PF improve test runtimes by motivating users to engage in more subsequent exercises. On the other hand, the combined receipt of these two motivators dampened the desire to exercise and decreased subsequent exercise frequency. This result suggests that the affirmation of users’ abilities via the receipt of both types of feedback simultaneously may cause users to be overconfident and engage in fewer subsequent exercises. Furthermore, out-exercise social feedback might be considered as redundant information since users already knew about their run times through the performance feedback provided during their exercise. Here, redundant social feedback may lead to user annoyance that could also reduce subsequent app usage and exercise frequency.

6.1. Theoretical Implications

In this work, we theorize that performance feedback enhances one’s self-efficacy while social feedback induces self-enhancement. This conceptualization helps explain various outcomes observed in the context of exercise apps. First, we found that the joint delivery of self-efficacy and self-enhancement motivations does not produce additive outcomes in our context. Based on our fine-grained tests, we found that SF, when offered in conjunction with PF, can, in fact, lead to overconfident behaviors like decreasing the likelihood of engaging in subsequent running exercises. The reduced participation rates in subsequent exercises are detrimental to users’ test performance, as stamina and strength are physical attributes that require constant maintenance. This may explain why we see the combination of PF and out-exercise SF associated with a non-additive effect. Another plausible mechanism is that the provision of self-enhancement motivators through social feedback could be perceived as redundant by the users who already received performance feedback that provided objective affirmations of their self-efficacy. Here, the additional offering of redundant information may prove to be annoying, leading users to reduce their subsequent usage of the app. More work should be performed to refine our understanding of the exact mechanism and uncover other scenarios in which the combination of different motivators can lead to non-complementary outcomes.

Second, within exercise apps, an important user characteristic to consider with respect to the use of different feedback types is the physical capability of the user. Self-efficacy motivation is shown to be useful across runners with varying physical capabilities. However, it is interesting to see that its effect on test outcomes diminishes with users’ physical capabilities, with the strongest runners not benefitting from the performance feedback feature. This could be due to a ceiling effect at work, wherein it is much harder to motivate athletes to make further improvements when they already achieved top-notch physical performance. In contrast, self-enhancement motivation works differently, in which the motivating effect of social feedback increases with users’ physical capabilities. Moreover, we learn that social feedback is only effective in improving the outcomes of relatively stronger runners. This result is in line with our finding that only positive social feedback is responsible for improving the running outcomes of users. Since praises and acknowledgments are typically awarded to good performance, only users with better running times during training would receive a boost in their self-enhancement and be encouraged to do even better.

Third, we attempt to explain the temporal impact of each feedback type on usage patterns. Performance feedback produces a steady but slow increase in usage frequency over time. Features tied to self-efficacy motivation may work best by allowing self-comparison over time, where measurements of gradual improvements can lead to self-sustaining behaviors of repeated use and engagement, as dedicated use in the prior period always generates rewards that motivate participation in the current.

On the other hand, social feedback produces a fast increase in usage levels but tapers off in effectiveness over time. Being a feature that mainly facilitates self-enhancement motivation, social feedback might cause the exercising user to develop a dependency on the encouragement of others to continue in their efforts in using the app and engage in exercises. Should social feedback be provided in an expected fashion, this feature could effectively induce sustained heightened usage levels. Yet, as seen in our analysis, the generation of such self-enhancement motivations is highly dependent on the willingness and availability of other social actors to provide encouragement and feedback, who are subject to attrition due to potential timing constraints, boredom, and lack of interest over time. Thus, users motivated by in-exercise social feedback might find it hard to keep up their usage levels as the non-deterministic feedback from friends tapers off with time.

6.2. Practical Implications

Our study has several takeaways for athletes, fitness coaches, sports organizations, and sports clinicians. First, our research empirically validated the effectiveness of in-exercise feedback features of exercise apps. With field evidence of the efficacy of such apps, coaches, and sports clinicians should prescribe fitness apps with such features to users to improve their training outcomes. In particular, our study findings show that the type of feedback that should be used depends on whether one wants to achieve short-term or long-term gains in athlete performance.

Second, our findings also speak about the effectiveness of a particular type of social feature in exercise apps. Despite in-exercise social feedback being effective in spurring initial usage and enhancing athletic performance, sports organizations should be cautious in using such feedback features, as our results show that the efficacy of social feedback can drop off over time. In particular, the efficacy of social feedback highly depends on the amount of feedback generated for the exercising user. To rely on an organic flow of social feedback among friends is not optimal, as the volume of social feedback generated is likely to drop over time due to disinterest and annoyance from the connected friends. Moreover, it is concerning that these social messages could dampen the motivation to exercise instead of building it up, especially when users expected others to recognize them verbally but did not receive the anticipated social feedback. Specifically, sports teams need to be aware that the positive impact of social feedback only materializes when the feedback is encouraging and positive in nature. Indeed, some fitness tech startups have begun to recognize this crucial point and have taken steps to provide “controlled” in-exercise social feedback through pre-recorded workout instructions and motivation provided by professional trainers (e.g., Aaptiv and Peloton). Early success from the approach adopted by these companies has ushered in non-trivial amounts of investment dollars.

Third, the finding of a non-additive effect between the two feedback features should be of direct interest to app companies and developers. Given that a large number of resources are involved in the development and launch of new app features, companies should carefully evaluate the inter-relationships between the new features to be developed and existing ones. While it might be tempting to keep up with the market by launching similar features competitors have, such a strategy might not always be beneficial, given that the feature may thwart the positive aspects of other existing features.

Finally, our study finds that the approach of “one size fits all” would not apply in the context of exercise apps, given that different users respond differently to different app features. Here, we saw that the efficacy of social features in motivating user behavior is contingent on the user’s capabilities to create positive images of themselves. Stronger runners benefited from the social feedback, while slower runners did not. Taken from the perspective of user motivation, our results are aligned with the findings of James et al. (2019), which finds users with different motivational needs would be best served by different features that provide the appropriate motivation. Furthermore, sports organizations should note that users may need different app features at different stages of their training regime to maximize their usage behaviors.

6.3. Limitations and Future Work

Like all field studies, our work is not free from limitations. First, our results are based on the features implemented in Endomondo, which may not resemble the feature implementations of other fitness apps. At the same time, conducting the study using members from a uniform group might mean that our results may not generalize to other populations. Future work could examine performance and social feedback in alternative apps and different contexts (e.g., with casual commercial users) to determine if consistent results are derived. Second, on a related note, we have focused on examining only two app features in a specific commercial implementation. Automatic goal setting, live social competing, run route recommendation based on friend’s activity, integration with live leaderboards, and inclusion of music in the auditory feedback are other novel forms of app features that are worth investigating. Third, like other field studies, subjects in ours chose when to report exercise behavior on the apps, hence we may observe only a fraction of their entire offline exercise activity. However, using an unrelated uniform outcome could alleviate concerns around this to an extent. Finally, it is also helpful to consider whether the effects of performance and social feedback would extend beyond running apps into other mHealth apps, such as diabetes tracking, sleeping tracking, and diet tracking apps. Despite these limitations, our work has unveiled the causal impact of two in-exercise app features on usage and exercise outcomes and their resulting interaction effects and long-term consequences. In doing so, we utilized the concepts of intrinsic and extrinsic motivations to characterize performance and social feedback features. Through this conceptualization, we hope our work can provide a deeper understanding of the effects of such app features on usage outcomes so that the design and use of such apps can be further enhanced.