Abstract
Traditional discrete-event simulation of large-scale networks at the packet level is computationally expensive. This article presents a fast rate-based transmission control protocol (RTCP) traffic model designed to reduce the time and space complexity for simulating network traffic whilst maintaining good accuracy. A distinct feature of the proposed model is that the transmission control protocol (TCP) congestion control behavior is represented using analytical models that describe the send rate at the traffic source as a function of the round-trip time and the packet loss rate at different phases of a TCP connection. Rather than modeling at the granularity of individual packets visiting the intermediate routers, the model approximates traffic flows as a series of rate windows, each consisting of a number of packets considered to possess the same arrival rate. The model calculates the queuing delays and the packet losses as these rate windows traverse the individual network queues along the flow path. The proposed RTCP model is able to achieve a performance advantage over other TCP models, by integrating analytical solutions and aggregating traffic using rate windows. Empirical results show that the RTCP model can correctly capture the overall TCP behavior and achieve a speedup of more than two orders of magnitude over the corresponding detailed packet-oriented simulation.
1. Introduction
As the size and complexity of the Internet increases, it becomes increasingly challenging to study the network behavior. Simulation can be useful for representing the detailed behavior of network applications in large-scale settings. Traditional packet-oriented simulation typically requires several simulation events to represent a packet visiting each router along its way from the traffic source to the destination (with at least two events to represent the packet arrival and departure at each router). Such detailed packet-level representation of traffic can incur substantial cost when simulating traffic for large-scale networks with millions of hosts and routers. 1 Furthermore, since the space complexity of a detailed packet-level simulation is proportional to the number of packets stored in the network queues, for networks that consist of links with a large delay bandwidth product, the memory cost can be exorbitant.
There are existing solutions for reducing the cost of simulating large-scale networks by representing the network traffic at a higher level of modeling abstraction through aggregation. For example, Ahn and Danzig 2 proposed to model the network traffic as “packet trains”, each consisting of a number of closely spaced packets. Guo et al. 3 also proposed a method to group the packets at a router within a time interval as a “chunk”. Fluid models4–9 represent network traffic as fluid flows rather than individual packets. Hybrid models aim at integrating fluid models into the detailed packet-level simulation.10–14 Their performance is adjustable by tuning the proportion of fluid traffic and packet-oriented traffic in the simulation.
Since the transmission control protocol (TCP) handles the brunt of today’s network traffic, a good aggregate traffic model must be able to accurately capture the TCP behavior. One type of solution6,8 simulates TCP by only capturing the long-term average behavior of TCP in order to improve computational efficiency. This approach inevitably results in a loss of accuracy as it overlooks the fine-grained time-varying features of TCP. The other type of solution7,9 maintains the detailed state transitions of TCP, in particular, the evolution of the TCP congestion window at the senders. However, in doing so it can greatly inflate the computational demand.
In this article, we present a fast rate-based model to approximate the TCP behaviors, which offers a new level of granularity for simulating TCP traffic. Our solution uses the packet aggregation idea, similar to those employed by Ahn and Danzig, 2 Guo et al., 3 and Nicol and Yan. 9 In particular, rather than modeling the network traffic packet by packet as in the traditional approach, we group the packets as chunks of variable length, called rate windows, which consist of a number of packets from the same TCP session and with a similar arrival rate. We describe each rate window by the packet arrival rate and duration, both of which can be adjusted according to the network condition as the chunk of packets visit the routers along the path from the source to the destination altogether. Different from the existing solutions, we apply analytical models to capture the congestion window behavior in response to the simulated network traffic conditions at different stages of TCP. That is, rather than modeling the complex state transitions of TCP, we apply mathematical models to derive the packet send rate as a function of the round-trip time (RTT) and the packet loss rate. The RTT and the packet loss rate are measured along the flow path as the rate windows are sent to visit the network queues and interact with other flows represented either as rate windows or individual packets. The latter, in particular, allows the model to seamlessly mix with a packet-oriented simulation.
We call our model the rate-based TCP (RTCP) model. Speedup is achieved because we are simulating the TCP traffic at a higher level of granularity. This comes at the cost of potential degradation of accuracy. Experiments show that the RTCP model is able to correctly capture the overall TCP behavior. Even for congested networks, our model can reduce the number of simulation events and accelerate simulation by more than two orders of magnitude. For uncongested network conditions, the RTCP model can improve execution time even more drastically.
The contribution of this article can be summarized as follows:
— We propose a fast rate-based TCP model to approximate the behavior of TCP traffic in network simulation at a new level of granularity.
— We apply analytical models to capture the approximate behavior of the TCP end systems to further speed up network simulation.
— We conduct extensive experiments to validate our model and show that it can lead to more than two orders of magnitude reduction both in the number of simulation events and the simulation time for congested networks.
The remainder of this article is organized as follows. In Section 2, we review the related work in traffic modeling. In Section 3, we describe an overview of the proposed rate-based TCP model. In Section 4, we provide a detailed description of the model for representing the TCP congestion control behavior at the end nodes; and in Section 5, we describe the interaction among the flows traversing the intermediate network queues. We conducted extensive experiments to analyze the performance of our model and the results are shown in Section 6. Finally, in Section 7, we present our conclusions and outline future research directions.
2. Related work
There exist several traffic generators, such as Surge, 15 Harpoon, 16 Tmix, 17 and Swing, 18 which can be used to either replay the packet trace or extend it to produce specified traffic load in real networks. They are fully fledged traffic generators and can certainly be used to generate packets for simulation. However, using these tools to produce the detailed packet sequence can incur substantial cost, making them computationally undesirable for large-scale network simulations. Abstract models, in contrast, solve this problem by modeling traffic at a higher level of abstraction. In general, abstract models can greatly reduce the computational demand. However, the reduction in computational complexity results in a loss of accuracy at various degrees.
Ahn and Danzig 2 proposed the idea of “packet trains”. Each packet train consists of a number of closely spaced packets. The time complexity of this approach depends on the size of the packet train. The modeling granularity can be adjusted to go between a packet-level simulation, where the train size is limited to contain only a single packet, and a conversation-level simulation, where the entire flow is modeled as a single packet train. It has been shown that the packet train model can achieve a significant reduction in the number of simulation events over the corresponding packet-oriented simulation, by as much as an order of magnitude if using a coarse granularity.
Guo et al. 3 proposed a time-stepping hybrid simulation (TSHS) framework. The packets that arrive within a time step are put together as a single “chunk”. Instead of handling individual packets, the routers process the whole chunk as it arrives. At the flow destination, packets within a chunk are individually acknowledged. Similar to the packet train approach, the time complexity of this approach can be adjusted, in this case, by choosing the length of the time step. Compared with the traditional packet-oriented simulation, the hybrid model has been shown to achieve moderate speedup when using sufficiently large time steps. It has been shown that TSHS can achieve over three times the speedup with a time step of 2 ms. 3
Although both the packet train and TSHS employ aggregation, many packet-level details remain in the models. For example, the packet train model still emits individual packets at the source and converts them into packet trains on the fly. TSHS carries the information of individual packets in the chunk so that the packets can be separately acknowledged at the destination. In contrast, fluid models4–9 eliminate the packet-level information altogether and only describe the flow-level traffic intensity.
There are two major categories of fluid models. The first category consists of event-driven models,4,7,9 where the traffic flows are broken down as segments each described using a piece-wise constant rate function. The flow rate may change when the TCP congestion window is changed at the sender due to the changing RTT or packet loss rate. As the flow traverses the network queues, the flow rate may also change as it interacts with other flows and competes for resources. The changes in the flow rate are represented by simulation events. Consequently, the efficiency of the fluid model depends on the network traffic condition. Experiments show that the number of simulation events can be reduced significantly by as much as three orders of magnitude over the packet-oriented simulation for large flow size and with little congestion. However, for heavily congested networks, the number of events increases, due to more interactions among flows in the congested network queues, which can “ripple” through the rest of the network.
The other category of fluid models consists of time-driven fluid models.5,6,8 These models aggregate the traffic flows from the same source to the same destination into a “fluid class” and use differential equations to describe the changes in the traffic intensity at the TCP senders and at the subsequent network queues. These differential equations can be solved numerically using the fixed time-stepped Runge–Kutta method. The speedup of the time-driven fluid models is primarily achieved by flow aggregation. Experiment results suggest that with heavy flow aggregation, that is, when each fluid class contains a large number of TCP flows, the fluid models can achieve a speedup of nearly three orders of magnitude. However, such performance gains can be reduced significantly if the flows are more spread out.
The fluid models face difficulty in modeling the packet-level details of the fluid flows. The time-driven fluid models 6 can only capture the average behavior of long-term TCP flows (during the TCP congestion avoidance mode). The models therefore cannot accurately represent the transient behavior of the TCP flows. The event-driven fluid models 9 address this problem by including complex logic to handle detailed TCP transactions in response to packet losses during the TCP slow start and congestion avoidance phases. This method, however, increases the computational demand.
Instead of flow-level aggregation, the aggregation of traffic in the gateway at access network level is proposed by Chen et al. 14 This model treats the access network as one single node. The traffic generator generates the amount of traffic for the entire access network and pushes the traffic to transmission controller. The transmission controller estimates the network condition according to the feedback of the ACKs and sends the packets to the destination through the network. This method achieves better performance than the traditional packet-oriented simulation through the aggregation of individual senders within one access network. However, the traffic transactions through the network are still represented packet by packet. Therefore, the improvement of its performance is limited.
Most aggregate models discussed above can be extended to hybrid models that can handle both fluid traffic and packet-oriented traffic. Hybrid models10–13 aim at capturing the interactions between fluid background traffic and foreground packet-oriented flows. Therefore, their focus is on the integration mechanism, not the statistical behavior of Internet traffic.
To efficiently model the TCP behavior, analytical models have been widely used. These analytical models19–24 provide an approximation of the TCP congestion control behavior. Some models only capture the steady-state behavior of TCP for long-lived flows;19,20,22 others only consider the connection establishment and slow-start phase of TCP for short-lived flows; 24 while others model TCP transfers for any given sizes of flows.21,23 To the best of the authors’ knowledge, these analytical models have not been used extensively in network simulation. Our rate-based TCP traffic model incorporates the analytical TCP models to reduce the computational complexity at the end nodes. In particular, to deal with the arbitrary sizes of TCP flows, we use the models proposed by Cardwell et al. 21 to predict the traffic intensity for both short- and long-lived TCP flows as a function of the RTT and the packet loss rate. For simplicity, we use the model proposed by Padhye et al. 22 to approximate the TCP send rate in steady state. This approximation has also been applied in Cardwell et al.’s model as a solution for capturing the TCP behavior in the congestion avoidance stage. The models we adopt in our model are well validated by using simulations, controlled Internet measurements, and comparisons with live traces. 25 Our solution also adopts the general packet aggregation idea from the packet train model, 2 TSHS, 3 and the event-driven fluid model. 9
3. An overview of the model
The essential aspect of TCP is its congestion control mechanism. Since TCP congestion control is performed at end nodes, we use an RTCP sender and an RTCP receiver to regulate the data transfer in simulation. It is important to note that our model is based on three assumptions. First, we assume that the time it takes to send all packets in a congestion window is smaller than the RTT. This assumption is normally true for today’s network with high bandwidth and long latency. Second, we assume that packet losses within a round (defined by the RTT) are independent of the losses in other rounds, while packet losses in the same round can be correlated. This assumption holds for FIFO drop-tail queues,26–28 but may not be true for RED queues 29 or for paths where packet loss is largely due to link errors rather than congestion. 21 Finally, the ACK losses are insignificant and therefore ignored after the initial handshake. This assumption is acceptable because the ACK packets are much smaller than the normal data packets, which makes them less likely to be dropped due to congestion. As expected, network paths are often more congested in the direction from data sender to receiver than in the reverse direction. 30
It is also important to note that our study here is limited to TCP Reno, which implements fast retransmit and fast recovery. In addition, we assume that the receiver implements delayed ACK: an ACK is sent for the successful delivery of two consecutive data segments. These limitations are simply because of the analytical models which we use for estimating the send rate for the TCP flows. Our model can be extended in principle by substituting the analytical models with those for other TCP variants, such as BIC and CUBIC.
We model three distinct phases of TCP. Connection establishment consists of the three-way handshake used by TCP to establish the connection between the sender and the receiver. Slow start is the exponential growth phase, during which TCP aggressively increases its congestion window by the same number of acknowledged segments. Congestion avoidance is the primary phase for congestion control, where TCP implements the additive increase and multiplicative decrease scheme. The congestion avoidance phase may also include periods of slow start, where TCP recovers from detected packet losses. However, we do not model slow start after retransmission timeouts because of the analytical models we adopt here. In addition, retransmission timeouts do not happen very often under low loss rate conditions. We do not model TCP termination phase because it does not play an important role in determining the throughput or latency of a data transfer. 21
We define four types of messages between the RTCP sender and the RTCP receiver:
— The START messages are used as probes, which accumulate the RTT and packet loss rate during the TCP connection establishment and slow start.
— The DATA messages carry the data over the TCP connection from the RTCP sender to the RTCP receiver. Each DATA message contains a “rate window”, which represents a chunk of packets considered to have the same arrival rate. A DATA message keeps track of the packet arrival rate, the duration, as well as the delivery ratio for the rate window, which can be adjusted as the message travels over the network from the sender to the receiver.
— The UPDATE messages are sent from the RTCP receiver or an intermediate router to inform the RTCP sender of the network condition so that the sender can adjust the TCP congestion window and send rate accordingly.
— An END message is sent from the RTCP receiver to the RTCP sender once the receiver has successfully received all the data. The END message will cause the sender to immediately stop the transmission.
To guarantee that the RTCP sender is aware of the network conditions, we do not allow START, UPDATE, nor END messages to be dropped in simulation. Both START and DATA messages are treated as regular packets, which means that they experience proper queuing delays in the network. UPDATE and END messages, however, are special packets which are delivered to the RTCP sender directly and are never queued.
To reduce the computational cost of the simulation, we model TCP data transfer in the unit of “rate windows”, where a number of consecutive packets from the same TCP session are lumped together and described using a constant arrival rate. The changes in rate of each rate window are determined by the network conditions (i.e. RTT and loss rate) that the previous rate window of the same flow has experienced. The RTT and loss rate are both rate-window specific. When loss happens, we do not keep track of which packet (or packets) are lost within a given rate window, instead we update the delivery ratio to indicate the loss. Each DATA message carries one rate window sent from the RTCP sender to the RTCP receiver. A rate window has the following attributes:
— Send time: the time at which the rate window is sent out from the RTCP sender. The send time will be copied to the UPDATE message on the way back to the RTCP sender, which uses the UPDATE message to calculate the RTT.
— Arrival rate: the rate at which packets belonging to the rate window are sent. At the beginning, the arrival rate is the same as the send rate of the TCP sender. In the subsequent queues, the arrival rate is the departure rate at the preceding queue.
— Duration: the length of the rate window in time. The product of the arrival rate and the duration is the total number of packets represented by the rate window, which remains unchanged as the rate window travels from the sender to the receiver.
— Delivery ratio: the proportion of data successfully delivered by the rate window so far. It is the number of bytes remaining in a rate window over the total number of bytes originally sent from the RTCP sender. At the beginning, the delivery ratio is one; and the delivery ratio may decrease as a result of packet losses at the intermediate routers.
Figure 1 shows the high-level interactions between the sender and the receiver. A TCP session starts when the RTCP sender sends a START message over the network to the RTCP receiver, which immediately sends it back (steps 1 and 2). This allows the sender to probe the network and calculate the expected duration of the handshake phase before a TCP connection can be established successfully. After that, the RTCP sender starts sending DATA messages, each representing one rate window with the packet arrival rate calculated from measured RTT and the packet loss probability. For each DATA message received, the RTCP receiver returns an UPDATE message carrying the necessary information for the RTCP sender to get the RTT and the packet loss probability (steps 3 and 4).

Interactions between RTCP sender and receiver.
Each intermediate network queue on the flow path from the sender to the receiver can modify the arrival rate, the duration, and delivery ratio of the DATA message, as the rate window interacts with other rate windows. If somehow the delivery ratio of a DATA message drops to zero, the entire rate window is considered lost, in which case an UPDATE message is immediately sent back to the RTCP sender so that the sender can be notified of the loss and adjust its send rate quickly (step 5). The RTCP receiver is responsible for determining whether a data transfer has completed (the data transfer size has been given to the receiver during the connection establishment), and if so, the RTCP receiver sends an END message to the sender to stop further transmissions (step 6).
The cost of the RTCP model depends on the size of the rate window. If the rate window is too small, we cannot achieve a significant performance improvement over the traditional packet-oriented simulation. If the rate window is too large, RTCP may not be able to properly respond to network congestions and cause significant errors in its estimation of the throughput. We choose to use the RTT as the duration of the rate window, which means that the size of a rate window is equal to the current congestion window size. This is a reasonable choice because TCP cannot react to network congestions unless the packet is delivered and acknowledged (or timeout happens). As an optimization, if we have prior knowledge that the network is under congested, that is, if the network has sufficient resources so that traffic experiences little loss, we can increase the size of the rate window for improved efficiency.
4. Determining send rate
We extend existing analytical models to represent the TCP congestion control behavior. In particular, during the connection establishment and slow start phases, we apply the model proposed by Cardwell et al. 21 to estimate the duration of the phases and the send rate. We use the model proposed by Padhye et al., 22 which is also a part of the model proposed by Cardwell et al., 21 to estimate the send rate during the congestion avoidance phase. Both models describe the TCP congestion window behavior as a function of the RTT and the packet loss rate. Both TCP timeout and triple duplicate ACKs are taken into account in these two models. In the following, we elaborate on the analytical models during the three TCP phases.
4.1. Connection establishment
Connection establishment is the first phase of TCP, which consists of the three-way handshake before a successful TCP connection can be established. The three-way handshake consists of zero or more failed attempts by the sender to transmit the TCP SYN message, followed by one successful delivery of the TCP SYN message, and then zero or more failed attempts to transmit the TCP SYN/ACK message by the receiver, followed by one successful delivery of the SYN/ACK message. The expected number of failed attempts depends on pf(t) and pr(t), which are defined to be the packet loss probability on the forwarding path (from the sender to the receiver) and on the reverse path (from the receiver to the sender) at time t, respectively. Typically, TCP will give up connection attempts after 4–6 failures. If pf(t) and pr(t) are low, most TCP handshakes will be successful before giving up. Using the analytical model by Cardwell et al., 21 we can estimate the duration of a successful connection establishment, Tce(t), as follows:
where π(t) is the measured RTT and ts is the length of the TCP timeout for the SYN message.
We set ts to be 3 seconds, which is a typical value. 31 The RTT, π(t), and the packet loss probabilities, pf(t) and pr(t), are collected during the exchange of the START messages between the RTCP sender and receiver. Once the expected duration of the connection establishment is determined, the RTCP sender waits for the duration before it enters the slow start phase, which we describe next.
4.2 Slow start
In the slow start mode, TCP quickly increases its congestion window until it detects a packet loss. From now on, we assume that packet loss only happens in the forwarding path from the sender to the receiver. Suppose that the flow size is ξ. If there is no packet loss, i.e. pf (t) = 0, we expect to send all ξ segments during the slow start phase. However, if pf (t) > 0, according to Cardwell et al., 21 the expected number of segments to be sent during the slow start phase, Lss(t), can be estimated as follows:
Here, we assume that the size of a segment (i.e., a TCP packet) is the maximum segment size (MSS).
If the TCP congestion window is not constrained by the maximum window size Wmax, the resulting TCP congestion window after sending Lss segments can be approximated by
where W1 is the initial value of the sender’s congestion window, and γ denotes the rate of exponential growth of the congestion window during the slow start. Typically, 1 ≤ W1≤ 3. We set γ to be 1.5 to capture the expected TCP behavior at slow start: the TCP sender increases its congestion window size by one MSS for each ACK packet received; and the TCP receiver that implements delayed ACK sends one ACK roughly for every two data packets. That is, during each RTT, the congestion window size increases by as much as half of the congestion window size during the previous round. Therefore, γ = 1.5.
To determine whether the TCP congestion window is constrained by the maximum send window size, we compare Wss(t) and Wmax. If Wss(t) > Wmax, TCP experiences two phases during the slow start: at first, the congestion window size increases from W1 to Wmax at the rate γ per RTT; after that, the congestion window remains at Wmax. If Wss(t) ≤Wmax, TCP would send all Lss(t) segments in the first phase.
The model proposed by Cardwell et al. 21 predicts the duration of the slow start phase, Tss(t), as follows:
We model the slow start phase by having the RTCP sender and receiver first exchange the START messages, from which we can use Equation (2) to determine Lss(t), the expected number of segments to be sent during the slow start phase. Then we use Equation (3) to determine Wss(t), the unconstrained congestion window size, and then use Equation (4) to determine Tss(t), the expected duration of the slow start phase. If it is determined that the TCP session is still in the slow start phase, i.e. if the number of segments sent by TCP is less than Lss(t), for each RTT π(t), the RTCP sender sends a DATA message carrying a rate window with the packet arrival rate Rss(t) = (Lss(t)/Tss(t)). Whenever the RTCP sender receives an UPDATE message, it recalibrates Lss(t), Tss(t) and Rss(t). When the number of sent segments has gone beyond the expected value, TCP switches to congestion avoidance.
Our model may not be able to capture the exact time of the transition from slow start to congestion avoidance. This is because the model handles packet transfers in the unit of rate windows. The model does not keep track of the exact packet positions within the rate windows, and therefore it is impossible to record the exact time the congestion avoidance model would start. The error is considered insignificant since it depends on the size of the rate window, which is the RTT.
4.3. Congestion avoidance
TCP enters steady state during the congestion avoidance phase. We use the model proposed by Padhye et al. 22 to estimate the send rate as a function of the RTT, the packet loss (whether it is due to duplicate ACKs or timeouts), and the current congestion window size. According to the model, the send rate can be determined as follows:
where R(t) is a rate determined by the RTT π(t), and the packet loss probability on the forwarding path pf (t) as follows:
where tO is the length of the timeout and b is the average number of packets that are acknowledged by each received ACK. We set tO = 0.2 s and b = 2.
Here Wm(t) is the current congestion window size. The model originally proposed by Padhye et al. 22 does not provide a way to calculate the current congestion window. We extend the model by incorporating TCP’s additive increase and multiplicative decrease behavior.
We calculate Wm in rounds. At first we set
The TCP send rate is controlled by a closed-loop system. A change in the measured RTT and packet loss rate will cause the RTCP sender to adjust its send rate. The RTCP receiver is responsible for sending the information back to the sender. During the connection establishment phase, the receiver immediately returns a START message after it receives a START message from the sender. During the data transfer (at the slow start and congestion avoidance phases), the receiver sends an UPDATE message for each received DATA message to the sender to report the current RTT and the packet loss rate. When the receiver determines that it has received all of the data, the receiver sends an END message to inform the sender to terminate further transmission.
5. Conducting flows at queues
Being able to capture the interaction among the traffic flows at the network routers both accurately and efficiently is essential to the success of the RTCP model. In this section we show how the network queues react to rate windows. Our model is restricted to FCFS queues with drop-tail behavior. We first consider a single flow at the queue and then extend the model to handle the interaction among multiple flows.
5.1 Single flow
We start by considering a drop-tail queue being fed by one and only one flow. Suppose that a rate window arrives at an empty queue with an arrival rate λ in , a duration δ in , and a delivery ratio τ in . In this case, λ in δ in is the total amount of data originally sent by the RTCP sender for the rate window (it is an invariant); λ in δ in τ in is the total amount of data actually arriving at the queue during the rate window; and λ in τ in is the effective arrival rate. Let µ be the link capacity, which is the service rate of the queue.
If the effective arrival rate λ in τ in is less than the service rate µ, there will be no accumulation at the queue, and the attributes of the rate window will not change: λ out = λin, δout = δ in , and τ out = τ in .
Otherwise, if λ in τ in ≥µ, the rate window will be served at the maximum capacity µ and the queue will start to accumulate a back log at a rate equal to the difference between the effective arrival rate, λ in τ in , and the service rate, µ. Suppose that the buffer size is B. The final queue length when the rate window completes its arrival at the queue can be calculated as
where
Since the product of the arrival rate and the duration should indicate the total amount of data for the rate window sent from the RTCP sender, the arrival rate of the output rate window can be easily adjusted:
The delivery ratio needs to be calculated to account for the losses due to possible buffer overflow. In case that (λ in τ in −µ)δ in ≤B, the rate window will not experience any loss and therefore τ out = τ in . Otherwise, there will be data loss due to buffer overflow:
The delivery ratio can be updated as follows:
5.2. Multiple flows
Now we consider the more interesting case when the queue consists of multiple flows. We define Wi (λ
i
, δ
i
, τi, si, ei) to be a rate window currently visiting the network queue, where λ
i
is the packet arrival rate, δ
i
is the duration, τ
i
is the delivery ratio, si is the start time, and ei is the end time (ei = si+δ
i
). In the algorithm, we maintain a priority queue Q to store all current rate windows visiting the network queue; we use the end time ei as the priority key. Initially, the priority queue is empty. We use A to store the aggregate effective arrival rate of all rate windows, i.e.
Algorithm 1 shows the algorithm that processes the simulation event representing the arrival of a new rate window Wf at time sf (which is the current simulation time). An example is given in Figure 2, where rate window f arrives at the queue at time sf . In this example, the previous rate window arrived at the queue is rate window 3. Assume that the algorithm correctly sets t0 = s3, q0 = q(s3), and

An example showing the queuing length changes at the start and end of the rate windows.
The algorithm starts by calculating the current queue length (lines 1–9). If there are rate windows in the priority queue Q that finish before the current time sf, the algorithm adjusts the queue length q0 (line 4) and the aggregate effective arrival rate A (line 5), to simulate the effect of these rate windows in the increasing order of the end time. The rate windows are also removed from the priority queue Q (line 6), since they will no longer be needed. After that, the algorithm updates the current queue length (line 8) and adds the effective arrival rate of the new rate window to the aggregate rate (line 9). In the example, rate window 2 is the only one that completes after s3 and before sf. Therefore, we update q0 from q(s3) to q(e2) and then from q(e2) to q(sf); the example assumes that the aggregate effective arrival rate at the two segments is larger than the service rate; that is why the queue length increases.
Next, the algorithm needs to compute the projected queue length at the end time of the newly arrived rate window. The queue length
To compute the projected values, the algorithm needs to scan into the simulated future. In order to protect the queue length, the current time, and the aggregate effective arrival rate from being overwritten, the algorithm first creates a set of shadow variables and copies the values from the original variables (line 10). The algorithm also uses a set S to temporarily store the rate windows removed from the priority queue; these rate windows need to be restored once the scan is finished (lines 22 and 29). The algorithm uses, which is initialized at line 11, to accumulate the amount of data loss due to buffer overflow.
If there are rate windows in the priority queue Q that finish before the end time of the newly arrived rate window, ef, the algorithm deals with them one by one in increasing end time order. The algorithm first adjusts the shadow queue length
In the example, rate window 3 ends before the newly arrived rate window ends. Therefore, the shadow queue length
The algorithm finally inserts the newly arrived rate window into the priority queue (line 30). In the meantime, it creates the output rate window with the updated attributes (line 31); the output rate window will be sent downstream to the next hop. The start time of the output rate window is calculated to be the start time of the input rate window sf, plus the time needed to flush all data enqueued at time sf (right before the input rate window arrives), and the propagation delay between this queue and the one downstream (line 32). The algorithm schedules an event to represent the arrival of the output rate window at the next queue (line 33).
We note that the algorithm calculates the projected queue length and data losses based only on the interaction between the newly arrived rate window and the existing rate windows in the queue. In particular, it assumes that future arrivals will not alter the prediction. This is obviously an approximation. If a rate window later arrives at a congested queue (resulting an aggregate arrival rate larger than the service rate), every flow in the queue needs to adjust its output rate for a fair share of the bandwidth. That is, one rate change may result in the update of all of the rate windows at this queue as well as all subsequent queues. This phenomenon is called the “ripple effect” and has been well documented.7,32
The ripple effect can significantly increase the computational demand. To avoid that, in our model, we do not allow the rate windows to be changed once they have been propagated to the downstream queues. We only make changes to the newly arrived rate windows and project the queuing effect based on the rate windows currently in the queue. Our solution is reasonable because our model is a closed-loop system: the queuing state is updated continuously as the RTCP sender adjusts its send rate at each round and sends a DATA message carrying a rate window and subsequently receives the UPDATE message carrying the network measurements. Even if the current rate window in a queue does not immediately adjust its rate to respond to the succeeding arrivals that overlap with it, the rate window in the next round will.
We need to be aware that such approximation may result in a rare condition where the rate window sent in the next round (from the same sender to the same receiver) catches up with the rate window in the previous round at a queue. This condition is caused by the over-estimation of the RTT in the previous round. Since the changes to the existing rate window is not propagated, it is possible that the sender sends the rate window for the next round with a start time earlier than the end time of the previous rate window. The successive rate windows of the same flow may overlap. To compensate for this error, we simply terminate the previous rate window in the queue as it carries stale information.
6. Experiments
We implemented the RTCP model in PRIME, 33 which is a parallel simulator designed for simulating large-scale network models. PRIME provides detailed models of 14 TCP congestion control algorithms, including RENO, BIC, and CUBIC, which have been validated carefully through extensive studies. 34 PRIME also supports real-time simulation, where simulated network protocols can interact with real network devices. The following experiments were conducted on a Linux workstation with a 2.3 GHz Intel Core2 Duo processor and 2 GB of RAM. All measurements shown are averages of 20 trials.
6.1 Dumbbell topology
Our first set of experiments aimed to provide a baseline comparison between RTCP and the detailed TCP models. We use a simple dumbbell network (shown in Figure 3), which consists of two routers and four hosts with two servers and two clients. The connection between the two routers forms a bottleneck link, configured with 10 Mb/s bandwidth and 64 ms delay. We set the bandwidths of all other links to be 1 Gbps, and set their delays to be 1 µs. All network interfaces use drop-tail queues with a buffer size of 70 KB (around 50 packets). Using this dumbbell model, we study RTCP’s accuracy and performance in terms of speedup and reduction in event count.

Dumbbell network with two flows.
First, we consider the scenario with a single flow. At the start of the simulation, we direct one TCP flow from S1 to C1. We observe the queue length and loss at R1; we also measure the overall throughput of the TCP flow. The results, shown in Table 1, indicate that RTCP can accurately capture the overall throughput of a detailed TCP flow. However, RTCP’s average loss probability and queue length differ from the detailed TCP model, although RTCP can capture the overall queue behavior, as is seen in Figure 4(a).
Statistics for the dumbbell network with one flow.

Comparison of the instantaneous queue size at R1.
We next study the scenario with two flows. At the start of the simulation, we direct one TCP flow from S1 to C1 and a competing flow from S2 to C2. The results are shown in Table 2. RTCP is able to capture the overall throughput and fairly divides the bandwidth between the two flows. RTCP is also able to capture the overall queue behavior, which is shown in Figure 4(b).
Statistics for the dumbbell network with two flows.
RTCP achieves a significant speed up in both scenarios. The number of events is reduced by a factor of 67 for a single flow and a factor of 43 for two flows. Likewise, the overall execution time is reduced by a factor of 91 and a factor of 53 for the single flow and two flow cases. The two-flow scenario sees a smaller speed up because RTCP must send more rate windows in response to the competition between the two flows.
6.2. Multiple clients
Our next experiment examines the behavior of RTCP in the case where multiple clients download from a single server. The network model is depicted in Figure 5. Each client (C x ) is connected with a link with delay set to be 1 +x*N. Here N is the delay increment, which we vary in the experiment between 0 and 5 ms. For example, if we set N = 2, the adjacent links connecting clients and the router R will differ by a delay of 5 ms. The buffer size for all interfaces is configured to be 50 packets. The bottleneck link between the server S and router R has a bandwidth of 45 Mb/s and a delay of 64 ms. At the start of the simulation, each client initiates a download of a large file from the server.

Multiple client network.
Owing to space limitations, we only show the results of the scenario when the delay increment is 0 in Table 3. To compare the model performance under all different scenarios as we change the delay increment, we show the results in Figures 6 and 7. Figure 6 shows the throughput achieved by each client using both the TCP and RTCP models after 100 seconds. Overall, RTCP and TCP match well. RTCP seems to achieve more balanced throughput than TCP (meaning it is more fair). As in the previous experiment, we also observed the average queue size at S for RTCP is consistently lower than that for TCP.
Statistics for a multiple client network with four flows (with 0 delay increment).

Throughput of the multiple client network.

Reduction in execution time and number of events for the multiple client network.
Figure 7 shows the reduction in the number of events and in the execution time for the different values of N. On average, RTCP reduces the number of events by a factor of 18 and reduces the execution time by a factor of 70.
6.3. Multiple bottleneck model
In this experiment, we evaluate the performance of RTCP using the so-called “parking lot” model, which contains multiple bottleneck links. The topology, shown in Figure 8, consists of four routers, four servers, and four clients. The bandwidths of all links are set to be 100 Mb/s. The delays of the links connecting routers and hosts are set to be 1 ms, and the delays of the links between the routers are set to be 5 ms. Downloads are initiated by the clients with a 5 second offset. That is, Flow 0 starts at time 0, Flow 1 at 5 s, Flow 2 at 10 s, and Flow 3 at 15 s. We initialize the flows at different times so that we can avoid the synchronization among the flows. The results are shown in Table 4. RTCP closely matches TCP’s behavior in terms of the average and aggregate throughput. In this case, RTCP is able to reduce the number of events by a factor of 143 and the execution time by a factor of 125.

Parking lot network.
Statistics for the parking lot network.
6.4 Large-scale topology
For our last experiment, we evaluate RTCP in a large network scenario. To achieve the necessary scale and realism for our experiments, here we use BRITE for the backbone network and use a campus network model for the network topology at the institutional level. We use BRITE 35 to generate the large network topology, which uses the statistics extracted from real network measurements. BRITE can produce random network models using a top-down method: it first generates a random network topology for the autonomous systems (ASs), then for each AS generates a router-level topology, and finally merges the AS-level and router-level topologies by connecting the routers belonging to different ASs. For the experiment, we use BRITE to generate a network topology consisting of 10 ASs, each with 10 routers. We then randomly choose two routers in each AS and attach a single campus network to each selected router. The campus network is a synthetic network defined by Nicol. 36 Each campus network has 13 LANs, 508 hosts with 504 clients, 4 servers, and 18 routers. The resulting topology consists of 10 ASs and 20 campus networks with a total of 460 routers and 10,160 hosts. To generate traffic, we randomly select the clients and have them download a 100 MB file from a randomly selected server. We schedule the clients to initiate downloads with an exponentially distributed inter-arrival time with a mean of 1 second. We run the simulation for 300 seconds and measure both the aggregate and average per-flow throughputs.
The results are shown in Table 5. The speedup achieved by RTCP is significant in this case. RTCP reduces the number of events by a factor of 60 and the execution time by a factor of 200. The aggregate throughput using RTCP, however, is 15% lower than that of TCP. Figure 9 is the Q–Q plot for comparing the empirical distributions of the throughput for individual flows between RTCP and TCP. The curved pattern suggests that the central quantiles are more closely spaced in TCP throughput than in RTCP throughput. We see that overall the flows match quite well, although RTCP seems to have underestimated the throughput at the low end of the range and overestimated the throughput at the high end of the range.
Statistics for a large network.

Q–Q plot of RTCP throughput versus TCP throughput for large network.
6.5. Discussion
RTCP is able to capture the behavior of TCP reasonably well. However, RTCP tends to underestimate throughput and the average queue size. We suspect that this is because the analytical models used by RTCP react to data losses faster than they should. Further, we observed that the analytical models seem to be less sensitive to amount of data loss. For small models, these errors cause negligible differences (less than 5.4%) in the overall throughput, which is confirmed by our experiments. For large models, however, the effect on the throughput is noticeable (14% difference). In the last experiment, RTCP significantly underestimates the throughput of a handful of flows that happen to contribute a large portion of the aggregate throughput. In addition to the inaccuracy introduced by the abstraction of the traffic at the “rate window” level, the deviation can also be explained by the analytical TCP models we adopt.
The model proposed by Cardwell et al. 21 has been evaluated through simulations, controlled Internet measurements and comparing with live traces under many different scenarios with different RTT, transfer size, MSS, and maximum congestion window size. The connection establishment model has been shown to have good accuracy as long as its assumption holds (i.e. the loss rate is below 0.5). The data transfer model that consists of the slow start model and the steady-state model presents variable performances of different lengths of data transmission under different loss conditions. Since the model proposed by Padhye et al. 22 is used as an approximate model to deal with the data transfer in the congestion avoidance model, the errors introduced here are mainly because of the two limitations of this approximation. First, in the approximate model, once the first loss is detected, the window size will be immediately adjusted to its steady-state value. However, in a real TCP implementation, when the sender detects a loss in the initial slow start phase, its instantaneous window size is often much larger than the steady-state average congestion window size. 21 Therefore, it takes a period of time for the sender to adjust its congestion window size from its value at that moment to its steady-state value. The lower the loss rate is, the longer the duration of this transition time will be. According to the results shown by Cardwell et al., 21 for high loss rates (i.e. 5% and higher), the sender exits slow start mode and immediately drops its window size to a small value that is close to the steady-state window value. So the error of the model in this case should be small. For low loss rates (i.e. 0.1% and below), it takes longer, which is usually three or more loss indications, to transit from the slow start mode to the steady-state phase. So the model in this case often overestimates the transmission latency. Consequently, the throughput will be underestimated. Second, Padhye et al.’s model does not model slow start after retransmission timeouts. For loss rates above 1%, the send rate of the steady-state phase is similar to that of the slow start phase after retransmission timeouts. So the error of the model should be small. On the other hand, for lower loss rates, although retransmission timeouts rarely happen, once they occur, the errors will be inevitable.
Regardless of these errors, both models we apply have proved their accuracy for most cases. In addition, the implementation of our RTCP model allows the analytical sending models to be easily replaced with other solutions for different targets. For example, we assume TCP RENO because Padhye et al.’s model 22 is restricted to modeling TCP RENO. However, by replacing with other TCP analytical models, our RTCP model can definitely provide solutions for modeling other TCP protocols.
Overall, for all cases, the estimated instantaneous queue size is similar between RTCP and TCP. More importantly, the inaccuracies are well compensated for by the reduction in the number of events and in the execution time when the packet-level details are not critical for the goal of the study. For large experiments, we can observe a speedup of over two orders of magnitude.
7. Conclusions and future work
In this article we have presented a traffic model which reduces the time and space complexity for simulating TCP traffic behavior with good accuracy. The proposed RTCP model introduces a new level of granularity for representing the network traffic. Rather than modeling at the granularity of individual packets, we have approximated the traffic flows as individual rate windows, each consisting of a number of packets with the same arrival rate. To model the detailed TCP behavior, we have used existing analytical models to calculate the send rate at the traffic source as a function of the measured RTT and packet loss probability. We have calculated the queuing delay and the loss probability as the rate window traverse individual network queues along the flow path. Experimental results are encouraging. They have shown that RTCP can speed up the simulation of large network traffic by a factor of 200, while still maintaining a reasonable level of accuracy. For all of the experiments we conducted, the RTCP model has been shown to achieve greater than a factor of 50 speed-up over the TCP model.
Although we only evaluate RTCP as a pure rate-based traffic model in this article, the model can actually be extended to deal with mixed traffic generated by packet-oriented simulations. Nicol and Yan 9 proposed a method for mixing packets and fluid flows. The method can also be used with RTCP.
In the following, we outline a simple method. We keep track of the recent packet arrival rate. Once a packet arrives, we can estimate the current queue length by using the information of the aggregate arrival rate of packets and rate windows. Suppose a new packet of size S arrives at the queue at time t. The queue length can be updated as follows:
where λ rate is the aggregate arrival rate of all rate windows, λ pkt is the recent packet arrival rate, tlast is arrival time of the previous packet, and qlast is the queue length at tlast. Whether we enqueue the packet will depend on the current queue length, the mixture of different types of flows, and the queue capacity. A similar approach has been use by Yan 37 for mixed packet and fluid flows. We are currently investigating this approach so that RTCP can be used to describe background traffic (which does not require very high fidelity).
Footnotes
Funding
This work is supported in part by the National Science Foundation (grant numbers CNS-0836408 and HRD-0833093), and a subcontract from the GENI Project Office at Raytheon BBN Technologies.
