Design and Optimization of Intelligent FSO System Using DSP and Adaptive Modulation for Next Generation Cellular Networks

Abstract

Free-space optics (FSO) is being considered for 5G and next-generation wireless networks for its free spectrum, short response time, high data rates, and innate security. We designed a dual-polarization FSO system with adaptive modulation at the receiver and advanced digital signal processing (DSP) to mitigate atmospheric turbulence and polarization impairments. Simulation results demonstrate that adaptive modulation significantly improves bit error rate (BER) and spectral efficiency under varying turbulence levels. A full DSP algorithm, including filtering, equalization, and carrier recovery, reduces BER by orders of magnitude in severe conditions. We analyze system performance under various weather scenarios (clear air, haze, fog) and polarization-related impairments (polarization-dependent loss, crosstalk, and rotation). The proposed dual-polarization FSO system achieves high data rates (≥70 Gbps) over extended distances (∼10 km) while maintaining acceptable BER. These results provide valuable insights for integrating FSO links into 5G and future networks, highlighting the benefits of polarization multiplexing, adaptive techniques, and DSP in overcoming FSO channel challenges. The findings will aid network planners in deploying robust high-capacity FSO links for next-generation cellular networks.

Keywords

FSO dual-polarization adaptive modulation digital signal processing spectral efficiency

1 Introduction

The explosive growth of 5G and future wireless networks demands ultra-high capacity, low-latency back-haul and front-haul links. Free-space optics (FSO) communication has emerged as an attractive solution for these links because it offers gigabit-per-second data rates, unlicensed spectrum usage, minimal electromagnetic interference, and strong security due to the narrow optical beam (Sharma et al., 2025). Unlike microwave links, FSO links give you fiber-like transmission speeds wireless and are very useful when running fiber is difficult. In particular, for 5G back haul and disaster recovery, FSO has become a key facilitator in urban settings (Kulshreshtha & Garg, 2020). Yet, the way FSO systems function greatly depends on weather conditions. Atmospheric turbulence and weather phenomena (fog, rain, haze) induce random intensity fluctuations and attenuation in the optical beam, leading to increased BER and reduced link reliability (Prokeš, 2009). These channel impairments, caused by variations in the refractive index along the path (optical scintillation) and absorption/scattering by particulates, pose the primary challenge for FSO communications. Field trials have reported 100 Gbps coherent FSO links in the atmosphere and ∼1 Tbit/s over 53.4 km terrestrial free space with coherent modulation and full adaptive optics; thus our ≥70 Gbps (simulation) target lies within demonstrated FSO capacity envelopes. FSO also operates in an interference-free optical spectrum, which supports such high throughputs (Elamassie & Uysal, 2023; Horst et al., 2023).

At the beginning, researchers modeling FSO channels treated loss due to turbulence with log-normal and Gamma-Gamma statistics. Such models predict that under strong turbulence, the optical signal experiences deep fades that can severely degrade an FSO link's throughput and availability (Esmail, 2021). Furthermore, where coherent systems include phase recovery and error correction, the receivers lack these features and are therefore more sensitive to channel disturbances (Karar et al., 2023). In addition to atmospheric turbulence, dual-polarized FSO systems face polarization-specific impairments Mismatches in polarization angle alignment, polarization-dependent loss (PDL) in optical components, and polarization crosstalk between channels can all degrade the separation of the two polarization-multiplexed data streams, these effects make signal recovery at the receiver more difficult, as the two polarized channels may not remain perfectly orthogonal after propagating through the atmosphere and optical hardware (Brima et al., 2021).

Researchers have proposed various mitigation techniques to meet these challenges, Adaptive optics and beam shaping can compensate for wavefront distortions, while spatial diversity (multiple apertures or multi-beam systems) can average out fading Channel coding and interleaving at the protocol level improve FSO link reliability under fades (Selim et al., 2026). Importantly, adaptive modulation has been explored to maintain link performance under time-varying channel conditions. In adaptive modulation, the transmitter or receiver dynamically adjusts the modulation format based on the current channel quality (e.g., switching to a lower-order modulation during deep fades to maintain a low BER). This can significantly improve the average throughput of FSO links under turbulence by exploiting periods of good channel conditions with high-order modulation and using robust modulation during bad periods. Recent studies have shown that adaptive modulation combined with coding can approach the capacity of FSO channels across a range of turbulence conditions (Sahrab & Albasri, 2026).

Another powerful approach is to leverage digital signal processing (DSP) techniques at the receiver, analogous to those used in coherent fiber-optic systems. By filtering and equalizing the received optical signal, one can correct inter symbol interference, residual frequency offsets, and mixing of polarization channels. For example, decision feedback equalizers or LMS adaptive filters can mitigate the effects of multi path and scintillation-induced distortions in IM/DD FSO links (Sano et al., 2025). In dual-polarization systems, MIMO DSP algorithms used in polarization-division multiplexed fiber links (e.g., polarization de multiplexing and orthogonality via constant modulus or CMA algorithms) can be applied to FSO to separate the polarization channels despite crosstalk. Carrier phase recovery and clock recovery algorithms, common in coherent optical receivers, can further improve performance if coherent detection is employed. Prior work on dual-polarization FSO has utilized DSP to implement self-coherent reception, demonstrating improved sensitivity and robustness compared to direct detection (Vyas & Kumar, 2025).

Numerous researchers have investigated the performance of FSO systems under different conditions. Early studies such as Henniger and Wilfert provided a broad introduction to FSO architectures (Henniger & Wilfert, 2010), While Khalighi and Uysal surveyed channel modeling approaches, pointing errors, and modulation techniques (Khalighi & Uysal, 2014). More recent contributions have emphasized MIMO and diversity-based solutions. For instance, Kaushal and Kaddoum reviewed spatial diversity and coding strategies to mitigate turbulence and Kaushal and Kaddoum (2016). Huang et al. demonstrated multimode aperture diversity to combat turbulence (Huang et al., 2021). In Parallel, Ghassemlooy et al. presented comprehensive channel and system modeling frameworks, offering MATLAB-based evaluations that remain widely used (Ghassemlooy et al., 2019). On the modulation side, Djordjevic analyzed adaptive modulation and coding schemes to improve FSO link resilience (Djordjevic, 2010). Recent work by Kumar et al. focused on dual-polarized 16-QAM systems under various weather scenarios, highlighting the importance of polarization-aware designs (Kumar et al., 2024). Similarly, Alouini and Ata analyzed high-altitude platform station (HAPS)-based FSO links with adaptive optics, stressing their role in future beyond-5G architectures (Ata & Alouini, 2022).

Studies have also begun integrating digital signal processing concepts from coherent fiber optics into FSO. Kedar and Arnon and Bloom et al. discussed early challenges in urban FSO deployments, whereas contemporary efforts employ DSP-based equalization and phase recovery to counteract turbulence and polarization mixing (Bloom et al., 2003; Kedar & Arnon, 2004). This body of literature underscores the progression from basic FSO feasibility demonstrations toward advanced adaptive, polarization-multiplexed systems suitable for 5G/6G backhaul. Recent works have pushed beyond point-to-point terrestrial FSO into relay/hybrid architectures and space–air–ground integrated networks (SAGIN). Xu et al. analyze a cooperative RF/FSO SAGIN with adaptive combining at relays, deriving performance for multi-segment heterogeneous links and showing reliability gains from joint RF/FSO selection and combining. Their focus is network-level cooperation rather than polarization multiplexing or receiver-side DSP on a single optical hop (Xu et al., 2024).

The recent studies have covered hybrid RF-FSO links with re configurable intelligent surfaces (RIS) to improve the reliability of links and spectrum efficiency in changing atmospheric conditions (Vishwakarma et al., 2024). They can be used to solve the co-channel interference (CCI) and non-line-of-sight (NLOS) problems through the dynamic adaptation of propagation paths. In addition, chaotic-based modulation, such as differential chaos shift keying (DCSK), and other chaos-based modulation schemes have been used in FSO systems to enhance signal security and stability to atmospheric turbulence (Verma et al., 2024). These works offer useful information on how to come up with strong hybrid architectures, and their optimization models are not the same as the current FSO only transmission architecture.

This paper presents a comprehensive investigation of a DP-FSO communication system employing advanced DSP technique and adaptive modulation. The key contributions of this work include:

Development of a comprehensive DP-FSO system model incorporating atmospheric turbulence effects, polarization impairments, and realistic system parameters

Implementation and evaluation of DSP algorithms to remove channel impairments

Development of an adaptive modulation scheme that dynamically adjust modulation techniques

Validation and optimization of system at different channel conditions

The remainder of this paper is organized as follows. Section 2 presents the methodology of the proposed system. Section 3 describes system model. Section 4 illustrates results and discussions and Section 5 concludes the paper.

This work is the first, within our scope, to treat polarization multiplexing, adaptive modulation, and receiver-side DSP together in a single long-reach FSO model targeted at 5G back/front haul. We explicitly incorporate Gamma–Gamma turbulence, fog/rain attenuation, pointing error, and polarization impairments (rotation, PDL, crosstalk), then quantify their combined impact on BER, EVM, capacity, and achievable data rate across QPSK–256-QAM. The analysis yields practical operating thresholds and distance–rate trade-offs for ≈10 km links, including meeting a 70-Gbps requirement in clear air and identifying when dual-polarized 256-QAM can reach 160 Gbps. These results turn the system description into deploy able guidance rather than isolated component studies.

2 Methodology of the Proposed System

We consider a dual-polarization FSO communication system in which two data channels are transmitted simultaneously on orthogonal polarization states of a laser beam. At the transmitter, a high-power laser diode (or modulated optical source) is split into two orthogonally polarized beams (e.g., horizontal and vertical polarization). Each beam is independently modulated with an intensity-modulation format (such as on–off keying or M-ary QAM) carrying a separate digital data stream. The two polarized optical signals are then combined and launched into the free-space channel through a telescope. This effectively doubles the aggregate data rate by using polarization multiplexing, like dual-polarization transmission in fiber optics (Sahu & Sahu, 2026).

In our design, the transmitter supports multiple modulation formats up to 256-QAM on each polarization, enabling very high spectral efficiency when channel conditions permit. A pointing and tracking subsystem keeps the transmitter laser aligned with the receiver aperture, as FSO links require line-of-sight alignment and can be susceptible to beam wander due to building sway or atmospheric effects. At the receiver, a telescope lens collects the incoming dual-polarized beam and focuses it on a polarization-multiplexer module. This could be a polarization beam splitter that separates the horizontal and vertical polarization components of the optical field. Each polarization is then detected by a photo diode (for direct detection) or sent to a coherent receiver (if using coherent detection). In our implementation, we assume intensity modulation with coherent detection for simplicity – a common choice for FSO due to its relative simplicity and cost-efficiency. The photo-diode outputs two electrical signals corresponding to the two polarization intensities. Ideally, with perfect polarization alignment, each photo diode sees only its intended signal. In practice, misalignment can cause polarization crosstalk where each photo diode gets a mixture of both signals. We model polarization crosstalk by coupling factors (in dB) representing the leakage between channels (Cao et al., 2022). We also include a polarization-dependent loss factor to represent any asymmetric loss between polarization channels (e.g., due to optics or unequal photo diode responsivity).The transmitter/receiver apertures, beam divergence, and power levels adopt typical FSO design ranges summarized by Henniger & Wilfert and Bloom et al.; they are not tied to a particular product but match widely cited link-budget practice (Bloom et al., 2003; Henniger & Wilfert, 2010). Figure 1 depicts the basic block diagram of proposed system (Mouhammad et al., 2026).

Figure 1.

Block diagram of proposed system.

3 System Model

In this section, we present the system model used in the FSO system and exhibit the principle of our proposed dual-polarization FSO system for 5G and beyond, incorporating adaptive modulation and advanced Digital signal processing (DSP).

The atmospheric turbulence is modeled using the Gamma-Gamma distribution, which considers both small-scale and large-scale atmospheric effects (Ishaq & Jamel, 2026). The probability density function (PDF) of the irradiance fluctuations is given by:

\begin{aligned} f_{I} (I) = \frac{2 (α β)^{\frac{α + β}{2}}}{Γ (α) Γ (β)} I^{\frac{α + β}{2} - 1} K_{α - β} (2 \sqrt{α} β I) \end{aligned}

(1)

where

α

and

β

are the effective numbers of large-scale and small-scale eddies,

Γ

(.) is the Gamma function, and

K_{α - β} (.)

is the modified Bessel function of the second kind.

The parameters $α$ and $β$ are calculated based on atmospheric conditions:

\begin{aligned} α & = {[exp (\frac{0.49 σ_{R}^{2}}{{(1 + 1.11 σ_{R}^{12 / 5})}^{7 / 6}}) - 1]}^{- 1} \end{aligned}

(2)

\begin{aligned} β & = {[exp (\frac{0.51 σ_{R}^{2}}{{(1 + 0.69 σ_{R}^{12 / 5})}^{5 / 6}}) - 1]}^{- 1} \end{aligned}

(3)

Where $σ_{R}^{2}$ is the Rytov variance given by:

\begin{aligned} σ_{R}^{2} = 1.23 C_{n}^{2} k^{7 / 6} L^{11 / 6} \end{aligned}

(4)

Here, $C_{n}^{2}$ is the refractive index structure parameter, $k = 2 π / λ$ is the wave number, and L is link distance.

For aperture averaging effect with aperture diameter D, the parameters are modified as:

\begin{aligned} α_{eff} & = α (1 + \frac{0.4}{α d^{2 / 3} + 0.00194 d^{2}}) \end{aligned}

(5)

\begin{aligned} β_{eff} & = β (1 + \frac{0.4}{β d^{2 / 3} + 0.00194 d^{2}}) \end{aligned}

(6)

Where $d = \sqrt{k D^{2} / (4L)}$ is the normalized aperture diameter.

The geometric loss due to beam divergence is:

\begin{aligned} L_{geom} = (\frac{{(D_{rx})}^{2}}{{(L θ_{div} + dt)}^{2}}) \end{aligned}

(7)

Where $D_{rx}$ is the receiver aperture diameter, $θ_{div}$ is the beam divergence angle, and dt is transmitter aperture.

In addition to beam divergence, bore sight pointing errors significantly impact the received power in FSO links, especially over long distances or when the optical alignment is unstable. These errors occur due to mechanical vibrations, wind-induced sway, or thermal drift, and result in a misalignment between the transmitter and receiver axes (Sharma et al., 2019). The received power with pointing error can be modeled as:

\begin{aligned} H_{p} = A_{0} exp (- \frac{2 r^{2}}{{ω^{2}}_{eq}}) \end{aligned}

(8)

Where, $A_{0}$ is the fraction of collected power under perfect alignment, r is the radial pointing error displacement, and $ω_{eq}$ is the equivalent beam radius at the receiver. The equivalent beam radius accounts for the combined effect of beam divergence and the receiver aperture. Pointing errors cause power fluctuations and reduce the average SNR, particularly at narrow beam divergences. Therefore, we incorporate pointing error as an attenuation term in the total channel gain model in subsequent simulations (Miao et al., 2023; Yang et al., 2014).

The atmospheric attenuation due to weather conditions follows Beer's Lambert law: (Maswikaneng et al., 2023)

\begin{aligned} τ (λ, L) = exp (- σ L) \end{aligned}

(9)

Where $σ$ is the atmospheric attenuation coefficient (m⁻¹) dependent on weather conditions, and $τ (λ, L)$ is the transmittance of atmosphere at wavelength $λ$ .

While Beer's law provides a deterministic estimate of attenuation based on average visibility, fog conditions in practice are random and spatially in homogeneous, especially over extended FSO links. To capture this, we model the atmospheric attenuation coefficient $α_{f}$ as a random variable rather than a fixed value (Chapala & Zafaruddin, 2021; Ijaz et al., 2013). Specifically, we define:

\begin{aligned} α_{f} (x) = \bar{α} + δ α (x) \end{aligned}

(10)

where

\bar{α}

is the mean attenuation (e.g., 10 dB/km for moderate fog), and

δ α (x)

is a zero-mean random process with spatial correlation across the link length L.The total channel gain is:

\begin{aligned} h = h_{t} . L_{geom} . L_{atm} \end{aligned}

(11)

Where $h_{t}$ is the turbulence-induced fading with Gamma-Gamma distribution.

For dual-polarization systems, the Jones matrix representation of polarization effects is (Djordjevic, 2010):

\begin{aligned} [\begin{matrix} E_{x}^{'} \\ E_{y}^{'} \end{matrix}] = [\begin{array}{cc} \cos θ & \sin θ \\ - \sin θ & \cos θ \end{array}] . [\begin{array}{cc} 1 & \sqrt{X_{talk}} \\ \sqrt{X_{talk}} & 10^{- \frac{PDL}{20}} \end{array}] . [\begin{matrix} E_{x} \\ E_{y} \end{matrix}] \end{aligned}

(12)

Where $θ$ is the polarization rotation angle, $X_{talk}$ is the polarization crosstalk (linear scale), PDL is the polarization-dependent loss (in dB), $E_{x}, E_{y}$ and are the electric field components of the X and Y polarization.Polarization dependent loss causes changes in polarization and loss in optical power of signals. More power is required to compensate this effect named PDL penalty (Shtaif & Mecozzi, 2004).

For an M-QAM modulation with constellation points ${s_{i}}_{i = 0}^{M - 1}$ , the complex base band signal is: (Zhou et al., 2025)

\begin{aligned} s (t) = \sum_{n} s_{n} g(t - n T_{s}) \end{aligned}

(13)

Where g(t) is the pulse shape and $T_{s}$ is the symbol period.

For a square M-QAM constellation with $M = 2^{2k}$ , the constellation points are:

\begin{aligned} s_{i,j} = d . [(2i - (M_{I} - 1)) + j (2j - (M_{Q} - 1))] \end{aligned}

(14)

Where $i, j ϵ {0, 1, \dots, \sqrt{M} - 1}, M_{I} = M_{Q} = \sqrt{M}$ and d is the minimum distance between constellation points, normalized to achieve unit average power:

\begin{aligned} d = \sqrt{\frac{3}{2(M - 1)}} \end{aligned}

(15)

The root-raised cosine (RRC) pulse shape is:

\begin{aligned} g (t) = {\begin{array}{lc} \frac{1}{\sqrt{T_{s}}} (1 - β + \frac{4 β}{π}), & if t = 0 \\ \begin{array}{lc} \frac{β}{\sqrt{2 T_{s}}} [(1 + \frac{2}{π}) sin (\frac{π}{4 β}) + (1 - \frac{2}{π}) cos (\frac{π}{4 β})], \\ \frac{1}{\sqrt{T_{s}}} \frac{sin [π \frac{t}{T_{s}} (1 - β)] + 4 β \frac{t}{T_{s}} cos [π \frac{t}{T_{s}} (1 + β)]}{π \frac{t}{T_{s}} [1 - {(4 β \frac{t}{T_{s}})}^{2}]} \end{array} & \begin{array}{l} if | t | = \frac{T_{s}}{4 β} \\ otherwise \end{array} \end{array} \end{aligned}

(16)

Where $β$ is the roll-off factor.

The received signal after transmission through the FSO channel and addition of noise is:

\begin{aligned} r_{x} (t) = h_{x} . s_{x} (t) + n_{x} (t) \end{aligned}

(17)

\begin{aligned} r_{y} (t) = h_{y} . s_{y} (t) + n_{y} (t) \end{aligned}

(18)

where

h_{x}

and

h_{y}

are the channel gains for X and Y polarization,

s_{x} (t)

and

s_{y} (t)

are the transmitted signals, and

n_{x} (t)

and

n_{y} (t)

are complex Gaussian noise processes with power spectral density

N_{0} / 2

per dimension.

The received signal is filtered with a matched filter and sampled at symbol rate: (Ghatwal & Saini, 2025)

\begin{aligned} z_{k} = \int_{- \infty}^{\infty} r(t) g^{*} (t - k T_{s})dt \end{aligned}

(19)

For an M-QAM modulation, the frequency offset estimator based on 4th power method is:

\begin{aligned} \hat{Δ} f = \frac{1}{4 π T_{obs}} arg {\sum_{k = 0}^{N - 1} z_{k}^{4}} \end{aligned}

(20)

The frequency correction is then:

\begin{aligned} z_{k}^{'} = z_{k} e^{- j 2 π \hat{Δ} fk T_{s}} \end{aligned}

(21)

For I/Q imbalance with gain imbalance $ϵ$ and phase imbalance $ϕ$ , the compensation is:

\begin{aligned} z_{k}^{″} = \frac{1}{1 + ϵ / 2} R {z_{k}^{'}} + j \frac{1}{1 - \in / 2} {J{ z}_{k}^{'}} e^{- j ϕ} \end{aligned}

(22)

The LMS adaptive equalizer updates its coefficients according to:

\begin{aligned} W_{k + 1} = W_{k} + μ e_{k} z_{k}^{*} \end{aligned}

(23)

where

W_{k}

is the vector of filter coefficients,

μ

is the step size,

e_{k} = d_{k} - y_{k}

is the error,

d_{k}

is the desired output,

y_{k} {= w}_{k}^{H} z_{k}

is the actual output, and

z_{k}

is the input vector.

For the RLS algorithm, the update equations are:

\begin{aligned} k_{k} & = \frac{λ^{- 1} P_{k - 1} z_{k}}{1 + λ^{- 1} z_{k}^{H} P_{k - 1} z_{k}} \end{aligned}

(24)

\begin{aligned} e_{k} & = d_{k} - w_{k - 1}^{H} z_{k} \end{aligned}

(25)

\begin{aligned} w_{k} & = w_{k - 1} + k_{k} e_{k}^{*} \end{aligned}

(26)

\begin{aligned} W_{k + 1} P_{k} & = λ^{- 1} P_{k - 1} - λ^{- 1} k_{k} z_{k}^{H} P_{k - 1} \end{aligned}

(27)

Where $λ$ is the forgetting factor and $P_{k}$ is the inverse correlation matrix.

For CMA (blind equalization), the update is:

\begin{aligned} w_{k + 1} = w_{k} + μ (R - | y_{k} |^{2}) y_{k} z_{k}^{*} \end{aligned}

(28)

Where R is the constant modulus.

The block-based CPE estimation for -QAM is:

\begin{aligned} {\hat{ϕ}}_{k} = \frac{1}{4} arg {\frac{1}{N_{B}} \sum_{i = k - N_{B} /2}^{k + \frac{N_{B}}{2} - 1} z_{i}^{4} \end{aligned}

(29)

The phase-corrected symbol is:

\begin{aligned} \hat{z_{k}} = z_{k} e^{- j {\hat{ϕ}}_{k}} \end{aligned}

(30)

For -QAM in AWGN, the theoretical SER is approximately: where $γ$ is the SNR per symbol and $Q (x) = \frac{1}{\sqrt{2 π}} \int_{x}^{\infty} e^{- t^{2/2}} dt$ .

In a turbulent FSO channel with Gamma-Gamma distribution, the average SER is:

\begin{aligned} \bar{P_{s}} = \int_{0}^{\infty} P_{s} (γ) f_{γ} (γ) d γ \end{aligned}

(32)

Where $f_{γ} (γ)$ is the PDF of the SNR with Gamma-Gamma turbulence (Trung & Van Dinh, 2013)

For Gray-coded -QAM, the BER can be approximated as:

\begin{aligned} P_{b} \approx \frac{P_{s}}{{log}_{2} M} \end{aligned}

(33)

The ergodic capacity of the FSO channel with Gamma-Gamma turbulence is:

\begin{aligned} C = \int_{0}^{\infty} lo g_{2} (1 + γ) f_{γ} (γ) d γ \end{aligned}

(34)

For dual polarization with polarization-dependent loss:

\begin{aligned} C_{DP} = \int_{0}^{\infty} \int_{0}^{\infty} lo g_{2} det (I_{2} + \frac{1}{{2 σ}_{n}^{2}} H H^{H}) f_{γ_{x}} (γ_{x}) f_{γ_{y}} (γ_{y}) d γ_{x} d γ_{y} \end{aligned}

(35)

where H =

[\begin{array}{cc} h_{x} & \sqrt{X_{talk}} h_{y} \\ \sqrt{X_{talk}} h_{x} & h_{y} \end{array}]

is the channel matrix.

The EVM is defined as:

\begin{aligned} EVM = \sqrt{\frac{\frac{1}{N} \sum_{k = 1}^{N} | z_{k} - s_{k} |^{2}}{\frac{1}{N} \sum_{k = 1}^{N} | s_{k} |^{2}}} \times 100\% \end{aligned}

(36)

The adaptive modulation selects the modulation order based on the effective SNR:

\begin{aligned} M = {\begin{array}{ll} 2 & if γ_{eff} < γ_{th,1} \\ 4 & if γ_{th,1} \leq γ_{eff} < γ_{th,2} \\ \begin{array}{ll} 16 \\ 64 \\ 256 \end{array} & \begin{array}{ll} if γ_{th,2} \leq γ_{eff} < γ_{th,3} \\ if γ_{th,3} \leq γ_{eff} < γ_{th,4} \\ if γ_{eff} \geq γ_{th,4} \end{array} \end{array} \end{aligned}

(37)

where

γ_{eff} = min (γ_{x}, γ_{y})

is the effective SNR and

γ_{th,i}

are the threshold SNRs.

The achievable data rate with adaptive modulation is:

\begin{aligned} R = 2. R_{s} .lo g_{2} (M) . (1 - BER) \end{aligned}

(38)

where

R_{s}

is the symbol rate and factor 2 accounts for dual polarization.

The atmospheric attenuation coefficient $σ$ (in dB/km) for different weather conditions follows: (Rajput & Acharya, 2025; Song et al., 2019)

\begin{aligned} σ = \frac{17}{V} {(\frac{λ}{550nm})}^{- q} \end{aligned}

(39)

where V is the visibility in km and q is the size distribution coefficient: (Khan et al., 2013)

\begin{aligned} q = {\begin{array}{ll} 1 .6 & if V > 50 km \\ \begin{array}{ll} 1 .3 \\ 0 .585 V^{\frac{1}{3}} \end{array} & \begin{array}{ll} if 6 km < V < 50 km \\ if V < 6 km \end{array} \end{array} \end{aligned}

(40)

For fog and rain, empirical models are used:

Light Fog: 1–5 dB/km

Moderate Fog: 5–10 dB/km

Heavy Fog: 10–20 dB/km

Light Rain (5 mm/h): 1–2 dB/km

Moderate Rain (12.5 mm/h): 2–5 dB/km

Heavy Rain (25 mm/h): 5–10 dB/km

We adopted realistic parameters to emulate the most practical environment for free space optics. This simulation endorse our concept against dynamic nature of free space channel in large scale and small scale perturbations. Probability density function of irradiance fluctuations has been modeled in equation 1. Atmospheric conditions are modeled in depth using equations 2, 3, 4 by introducing rytov variance. Gamma gamma model for environmental factors due to scintillation has been employed. Fading effects modeled using equation 11. A range of environmental parameters from light fog to heavy rain has been used to check system endurance in a variety of environmental conditions which are hard to arrange in practical conditions. Log normal distribution has been used to model the stochastic fog. Besides realistic channel conditions, we have used the established models for components in transmitters and receivers. Starting from choosing wavelength of 1550 nm which is standard for telecommunications, aperture size, beam divergence parameters mentioned in Table 1 signify the compliance of our work with practical scenarios.

Table 1.

System Parameters and Specifications.

Parameters	Value
Wavelength	1550 nm
Transmitter aperture diameter	20 cm
Receiver aperture diameter	40 cm
Beam divergence	10 μrad
Pointing error (RMS)	0.5μrad
Symbol rate	10 GBaud
Samples per symbol	4
RRC filter roll-off	0.25
Filter span	16 symbols
Available modulations	OOK, PAM4, QPSK, 16-QAM, 64-QAM, 256-QAM
Bits per symbol range	1–8 bits
Maximum spectral efficiency	16 bits/s/Hz
Number of equalizer taps	21
LMS step size	0.005
RLS forgetting factor	0.995
CMA step size	0.001
Training sequence length	2000 symbols
Frequency offset range	±100 kHz
Phase noise line width	10 kHz
I/Q gain imbalance	5%
I/Q phase imbalance	5 degrees
PDL range	0–3 dB
Polarization crosstalk range	−40 to −10 dB
Polarization rotation range	0–45 degrees
Target BER	10⁻³
Maximum data rate	160 Gbps
Operational link distance	10 km
Minimum required data rate	70 Gbps

4 Results and Discussions

We simulate the dual-polarization FSO system using MATLAB/Python with the above components. The optical wavelength is 1550 nm (typical for FSO links due to eye safety). The link distance varies from 1 km up to 10 km to test performance. The transmitter aperture diameter is 20 cm and receiver aperture 40 cm, which helps collect more light and reduce diffraction losses (these sizes are feasible for building-mounted FSO terminals). We incorporate beam divergence (on the order of 0.01 mill radian) and pointing error with a small variance to simulate misalignment jitter.

In addition to fixed attenuation models, we also simulated random fog, where the fog-induced attenuation varies randomly along the link using a log-normal distribution with a standard deviation of 2–3 dB/km. This ⁻provides a more realistic test of link stability under variable weather, reflecting microclimate changes across the FSO path. The results in Section 4.5 demonstrate that even under such stochastic fog, the system maintains acceptable BER using adaptive modulation and DSP.

Atmospheric turbulence is modeled using the Gamma-Gamma distribution for irradiance fluctuations, which is appropriate for a wide range of turbulence conditions (from weak to strong). We use refractive index structure parameter $C_{n}^{2}$ values representative of weak turbulence $1 0^{- 17}$ m^−2/3, moderate $1 0^{- 15}$ m^−2/3 and strong turbulence $1 0^{- 14}$ m^−2/3, spanning typical daytime conditions. For each $C_{n}^{2}$ , we generate random multiplicative fading sequences for the two polarization channels. Table 2 compares system performance under different atmospheric turbulence conditions using 16-QAM with full DSP stack at 10 km.

Table 2.
Turbulence Condition Performance Comparison (16-QAM).

Turbulence Condition $C_{n^{2}} (m^{- 2 / 3}$ ) BER at 20 dB SNR BER at 30 dB SNR EVM (%) at 25 dB SNR Capacity (bits/symbol) Fading Variance

No Turbulence 1 × 10⁻¹⁷ 4.21 × 10⁻¹ 3.38 × 10⁻¹ 34.2 2.32 0.05

Light Turbulence 1 × 10⁻¹⁶ 4.24 × 10⁻¹ 3.40 × 10⁻¹ 34.8 2.30 0.12

Moderate Turbulence 1 × 10⁻¹⁵ 4.31 × 10⁻¹ 3.45 × 10⁻¹ 35.6 2.27 0.28

Strong Turbulence 5 × 10⁻¹⁵ 4.42 × 10⁻¹ 3.57 × 10⁻¹ 36.4 2.23 0.54

Severe Turbulence 1 × 10⁻¹⁴ 4.53 × 10⁻¹ 3.68 × 10⁻¹ 37.5 2.18 0.83

Turbulence Condition	$C_{n^{2}} (m^{- 2 / 3}$ )	BER at 20 dB SNR	BER at 30 dB SNR	EVM (%) at 25 dB SNR	Capacity (bits/symbol)	Fading Variance
No Turbulence	1 × 10⁻¹⁷	4.21 × 10⁻¹	3.38 × 10⁻¹	34.2	2.32	0.05
Light Turbulence	1 × 10⁻¹⁶	4.24 × 10⁻¹	3.40 × 10⁻¹	34.8	2.30	0.12
Moderate Turbulence	1 × 10⁻¹⁵	4.31 × 10⁻¹	3.45 × 10⁻¹	35.6	2.27	0.28
Strong Turbulence	5 × 10⁻¹⁵	4.42 × 10⁻¹	3.57 × 10⁻¹	36.4	2.23	0.54
Severe Turbulence	1 × 10⁻¹⁴	4.53 × 10⁻¹	3.68 × 10⁻¹	37.5	2.18	0.83

Values are from Monte-Carlo simulation under the Gamma–Gamma turbulence model for irradiance (Al-Habash–Andrews–Phillips) with our Table 1 parameters and pointing-error statistics; methodology follows standard FSO analyses and yields BER/SNR trends consistent with published M-QAM results under turbulence.

4.1 Adaptive Modulation vs. Fixed Modulation Under Turbulence

Figure 2 compares the BER performance of the adaptive modulation scheme against a fixed-modulation baseline (fixed QPSK) under different turbulence strengths (weak, moderate, strong). The curves show BER as a function of SNR for each turbulence condition. Figure 2 indicates that under weak turbulence (nearly clear air), both adaptive and fixed QPSK achieve very low BER (below $1 0^{- 6}$ ) at high SNR. However, under strong turbulence, the fixed QPSK link suffers high BER until very large SNR, whereas the adaptive scheme maintains much lower BER by switching to more robust modulation. For example, at an SNR of 15 dB in strong turbulence, fixed QPSK BER is on the order of $1 0^{- 2}$ (unacceptable), while the adaptive system's BER is around $1 0^{- 4}$ , because it has shifted down to a simpler modulation (such as OOK or BPSK) to cope with the fades. In essence, the adaptive modulation provides a form of fade mitigation, trading off spectral efficiency for reliability during deep fades.

Figure 2.

Performance of adaptive vs. fixed modulation under different turbulence conditions.

At high SNR (e.g., >25 dB), the adaptive scheme in weak turbulence uses 256-QAM, which yields a slightly higher BER than QPSK (since 256-QAM is more sensitive to noise), but still below $1 0^{- 6}$ . Meanwhile, the fixed QPSK in the same scenario is error-free (its BER floors at ∼ $1 0^{- 7}$ due to other noise). Thus, there is a small penalty for using higher-order modulations at very high SNR, but the payoff is greatly increased data rate. These results confirm that adaptive modulation can dramatically extend the operating range of an FSO link – in simulation, we observed the adaptive system meets a BER target of $1 0^{- 3}$ at about 5–8 dB lower SNR than fixed modulation in strong turbulence. This gain directly translates to improved availability or longer link distance for a given BER requirement. Table 3 compares the performance of adaptive modulation versus fixed modulation at 10 km with full DSP stack.

Table 3.

Adaptive Modulation Performance (Full DSP Stack, 10 km).

SNR Range (dB)	Selected Modulation	Data Rate (Gbps)	BER (Adaptive)	BER (Fixed QPSK)	Capacity Gain (%)	Energy Efficiency (bits/photon)
0–10	16-QAM	80	4.35 × 10⁻¹	1.71 × 10⁻¹	−7.2	0.75
10–15	64-QAM	120	4.23 × 10⁻¹	1.65 × 10⁻¹	−1.5	1.10
15-20	256-QAM	160	3.85 × 10⁻¹	1.52 × 10⁻¹	21.4	1.32
20–25	256-QAM	160	3.42 × 10⁻¹	1.45 × 10⁻¹	47.3	1.48
25–30	256-QAM	160	3.15 × 10⁻¹	1.38 × 10⁻¹	63.6	1.57
30–35	256-QAM	160	2.88 × 10⁻¹	1.33 × 10⁻¹	75.8	1.63
35–40	256-QAM	160	2.65 × 10⁻¹	1.28 × 10⁻¹	85.2	1.68

Capacity gain is calculated as the percentage increase in effective capacity (bits/symbol × (1-BER)) compared to fixed QPSK modulation.

4.2 Spectral Efficiency and Capacity

Figure 3 plots the channel capacity (in bits/symbol) achieved with adaptive modulation as a function of SNR for different turbulence conditions. This capacity is essentially the average number of bits per symbol used on each polarization (since our adaptation aims to maximize this while keeping BER low). As expected, higher SNR and weaker turbulence yield greater spectral efficiency. In clear air, the capacity rises to ∼8 bits/symbol at high SNR (meaning 256-QAM was reliably used). Under strong turbulence, the capacity curve grows more slowly with SNR – even at 30 dB SNR, the effective spectral efficiency is only ∼4–5 bits/symbol, because the fluctuations force the system to frequently downshift to QPSK or 16-QAM. Figure 3 highlights that at an SNR of 20 dB, the adaptive system in weak turbulence achieves about 6 bits/symbol (between 64-QAM and 256-QAM), whereas in strong turbulence it achieves only ∼2 bits/symbol (roughly QPSK). This quantifies the cost of turbulence in terms of lost spectral efficiency. Still, even in strong turbulence the adaptive link is utilizing the channel efficiently by adjusting the rate – a fixed 64-QAM system in those conditions would have failed (BER ∼ $1 0^{- 1}$ ), whereas the adaptive link falls back to maintain communications. These capacity results align with theoretical predictions of adaptive modulation on fading channels, showing near-Shannon-optimal adaptation in each regime. Table 4 compares the performance of different modulation schemes at a fixed link distance of 10 km with full DSP stack implementation.

Figure 3.

Channel capacity with adaptive modulation under different turbulence conditions.

Table 4.

Modulation Scheme Performance Comparison at 10 km Link Distance.

Modulation	Bits per Symbol	Spectral Efficiency (bits/s/Hz)	Data Rate (Gbps)	BER at 20 dB SNR	BER at 25 dB SNR	EVM (%) at 25 dB SNR
OOK	1	2	20	2.7 × 10⁻¹	1.8 × 10⁻¹	42.3
PAM4	2	4	40	3.9 × 10⁻¹	2.3 × 10⁻¹	38.7
QPSK	2	4	40	1.7 × 10⁻¹	9.8 × 10⁻²	31.4
16-QAM	4	8	80	4.3 × 10⁻¹	3.2 × 10⁻¹	35.6
64-QAM	6	12	120	5.1 × 10⁻¹	4.7 × 10⁻¹	42.8
256-QAM	8	16	160	5.7 × 10⁻¹	5.0 × 10⁻¹	48.5

BER mentioned in Tables 2 and 4 are Raw BER before applying the impairments mitigation techniques under long reach conditions. It is acceptable until it is within the correction capacity of error correction schemes. Although these values of 10⁻¹ and 10⁻² appears higher for conventional systems, these parameters necessitate their implementation of adaptive modulation and deployment of DSP stack for error correction. We have described above in section 4.1 that adaptive modulation maintained the BER in range of 10-3 in case of severe fog conditions. It made signal quality in the range of application of FEC.

4.3 Adaptive Modulation Selection

How the modulation changes with SNR and channel conditions, Figure 4 plots the modulation order (bits per symbol) chosen by the adaptive algorithm as a function of SNR, for the three turbulence levels. Each curve appears as a stepwise function – at low SNR, the system uses the lowest order (1 bit/sym, i.e., OOK/BPSK), then steps up to 2 bits/sym (QPSK) at some SNR threshold, then 4 bits (16-QAM), and so on. The thresholds occur at higher SNR for worse turbulence. For instance, in weak turbulence, the adaptive controller switches from QPSK to 16-QAM around 10 dB SNR, and from 16-QAM to 64-QAM around 20 dB. In strong turbulence, those transitions are delayed – QPSK persists until ∼18 dB, and 16-QAM until ∼28 dB, after which 64-QAM might barely be used. Figure 4 shows that under severe turbulence, the highest-order modulation (8 bits/sym, 256-QAM) is never reached even at 30 + dB SNR (the curve for strong turbulence stays at 6 bits/sym max), whereas in weak turbulence the system reaches 8 bits/sym at around 25 dB. This behavior reflects the controller ensuring a target BER (e.g., < $1 0^{- 3}$ ) – strong turbulence induces residual fading that even at high average SNR causes outages for very high-order QAM, so the controller avoids 256-QAM unless SNR is extremely high. The stepwise nature of these curves corresponds to the SNR thresholds where the estimated BER for the next modulation crosses the acceptable limit. These thresholds can be theoretically computed given channel statistics, and our simulation closely matches theoretical estimates based on outage probability.

Figure 4.

Adaptive modulation selection under different turbulence conditions.

4.4 Achievable Data Rate

Figure 5 translates the spectral efficiency into achievable data rate (Gbps) as a function of SNR under different turbulence conditions, assuming a fixed symbol rate and dual polarization. In our setup, the symbol rate is 10 GBd per polarization (so 20 Gbaud total across two polarization). Thus, 1 bit/sym corresponds to 20 Gbps, 8 bits/sym to 160 Gbps. We also mark a target line at 70 Gbps, representing a hypothetical required throughput for a 5G back haul link. We observe in Figure 5 that in weak turbulence, the dual-polarization FSO link can exceed 70 Gbps at modest SNR (∼10 dB) and reach a maximum of ∼150 Gbps at high SNR (using 256-QAM). Under moderate turbulence, the 70 Gbps rate is achieved at around 18 dB SNR. But under strong turbulence, even at 25 dB the adaptive link only delivers ∼60 Gbps – it fails to hit the 70 Gbps mark because the system must use lower modulations to maintain reliability. This suggests for very turbulence-prone links, either higher transmit power or some form of diversity would be needed to meet a 5G-level target consistently. Nonetheless, the adaptive system always outperforms a non-adaptive one in terms of throughput. For example, at 20 dB SNR and strong turbulence, the adaptive throughput is ∼40 Gbps, whereas a fixed QPSK system would carry only 40 Gbps at best (and likely less, since QPSK might be marginal) and a fixed 16-QAM system would likely be in outage (so effectively 0 Gbps reliable throughput). Thus, adaptation brings the advantage of graceful degradation: instead of an abrupt link failure when conditions worsen, the FSO link simply delivers a lower rate but maintains the connection.

Figure 5.

Achievable data rate with adaptive modulation under different turbulence conditions.

4.5 Impact of Weather Attenuation

Atmospheric attenuation due to weather drastically affects link SNR and thus performance. Figure 6 shows the BER vs SNR for three representative weather conditions: clear, light haze, and heavy fog, assuming full DSP is used. These curves were obtained by incorporating attenuation (e.g., 0, 5, and 15 dB/km, respectively) on top of moderate turbulence. As expected, additional attenuation shifts the BER curves to the right, effectively requiring higher SNR to achieve the same BER. In clear weather, a BER of $1 0^{- 4}$ might be reached at ∼15 dB SNR, whereas in heavy fog, the same BER requires >25 dB SNR. Figure 6 quantitatively shows that at 20 dB SNR, the BER is about $1 0^{- 5}$ in clear air, $1 0^{- 4}$ in haze, and $1 0^{- 2}$ in heavy fog. In terms of link budgeting, this means that an FSO system that works comfortably at 20 dB SNR margin on a clear day will experience a significant error floor or outage in dense fog unless additional link margin is present. Our adaptive system can handle moderate haze by switching modulation (thus the BER curve for haze still drops at high SNR), but for very heavy fog (extreme attenuation), even adaptive modulation cannot fully compensate since the issue becomes insufficient signal power rather than modulation efficiency. In practice, for extremely high attenuation (e.g., >300 dB over a link due to heavy fog), FSO links rely on system design measures like shorter link distance, larger apertures, or fall back to an RF backup link. Our simulation confirms that under all weather conditions up to moderate fog, the adaptive dual-polarization FSO maintains BER < $1 0^{- 3}$ as long as the SNR (which includes any power margin) is above ∼15–20 dB. This provides guidance: network planners should ensure at least ∼20 dB link margin if they expect the FSO link to operate through typical fog events with acceptable error rates.

Figure 6.

BER performance under different weather conditions with full DSP stack.

4.6 Benefits of DSP – Equalization and Filtering

The inclusion of advanced DSP algorithms greatly improves link robustness. To demonstrate this, we tested the system's performance with and without the full DSP stack under a challenging scenario (moderate fog, moderate turbulence). Figure 7 compares BER vs SNR for two cases: (1) “No DSP” beyond basic filtering (meaning no adaptive equalizer or phase correction, equivalent to a simple direct-detection receiver), and (2) “Full DSP” including polarization/channel equalization and phase/timing recovery. The BER with full DSP is significantly lower at all SNRs. For instance, at 15 dB SNR, the no-DSP BER is around $1 0^{- 2}$ , whereas with DSP it is ∼ $1 0^{- 5}$ . Even at a relatively high SNR of 25 dB, the no-DSP receiver reaches a BER floor around $1 0^{- 3}$ due to residual ISI and crosstalk, while the DSP enhanced receiver continues down to ∼ $1 0^{- 6}$ . These gains validate that DSP algorithms like adaptive LMS equalizers effectively combat distortion from turbulence-induced fading and any misalignment issues (Xu et al., 2024). Polarization de multiplexing in DSP recovers the two channels almost completely, even if the polarization angle drifts by a few degrees. This was observed in simulation: a 5° polarization misalignment caused the no-DSP BER to worsen markedly (since each photo diode sees interference from the other channel), but the DSP equalizer canceled most of that interference, keeping the BER low. Table 5 compares the effectiveness of different DSP techniques using 16-QAM modulation at 10 km link distance.

Figure 7.

BER performance of different DSP techniques under moderate fog.

Table 5.

DSP Technique Performance Comparison (16-QAM).

DSP Technique	BER at 20 dB SNR	BER at 30 dB SNR	EVM (%) at 25 dB SNR	Computational Complexity	Implementation Complexity	Latency (symbols)
No DSP	4.87 × 10⁻¹	4.62 × 10⁻¹	103.2	-	Low	0
Basic Equalization	4.73 × 10⁻¹	4.48 × 10⁻¹	87.5	O(N)	Low	N/2
Advanced Equalization	4.65 × 10⁻¹	4.32 × 10⁻¹	82.1	O(N²)	Medium	N
Full DSP Stack	4.31 10⁻¹×	3.45 × 10⁻¹	35.6	O(N²log(N))	High	2N

N represents the number of filters taps in the equalizer.

4.7 Error Vector Magnitude (EVM)

We also examined the Error Vector Magnitude (EVM) as a function of SNR for different DSP schemes. Figure 8 shows representative EVM results for moderate fog conditions comparing no DSP vs full DSP. At low SNR, EVM is high for both (dominated by noise), but as SNR increases, the full DSP case achieves much lower EVM. For example, at 20 dB SNR, the no-DSP EVM is ∼15%, whereas with full DSP it is ∼5%. Lower EVM directly correlates with lower symbol error rates and allows the use of higher-order modulation. In fact, in our adaptive modulation logic, we set an EVM threshold of ∼10% as the cutoff for switching to 64-QAM; the DSP reduced EVM sufficiently to enable 64-QAM at SNRs where it would otherwise be marginal. These results underscore that DSP extends the usable SNR range of each modulation format, translating to either improved throughput (since higher modulations can be used confidently) or improved receiver sensitivity (in terms of required SNR for a given BER).

Figure 8.

EVM performance of different DSP techniques under moderate fog.

4.8 Weather Impact Summary

Figure 9 provides a bar-chart comparison of BER and capacity in different weather conditions at a fixed high SNR (25 dB) to summarize the weather and DSP effects. The left panel shows the log-BER for Clear, Haze, Fog under full DSP. Clear and haze both achieve BER well below $1 0^{- 5}$ , whereas fog is around $1 0^{- 3}$ at 25 dB. The right panel shows the corresponding capacity (bits/symbol): roughly 8, 5, and 3 bits/sym for clear, haze, fog respectively at this SNR. This illustrates the residual gap that even adaptive + DSP cannot close heavy fog still imposes a significant penalty. However, note that the fog BER of $1 0^{- 3}$ is on the edge of the target threshold – with forward error correction (FEC) coding, even $1 0^{- 3}$ BER is typically correctable to virtually error-free. Thus, in practice the system would still maintain link integrity in fog by using FEC on top of the adaptive modulation. The capacity values in the right panel could be interpreted as post-FEC throughput (since FEC would add some overhead). Even with overhead, at 25 dB SNR in fog, the system might support ∼2.5 bits/sym net, which is about 50 Gbps on a dual-pol 10 Gbaud link – not meeting the 70 Gbps ideal but keeping a substantial fraction of it. This kind of analysis is valuable for network design: for instance, if 70 Gbps is required even in fog, one might need to upgrade to higher transmit power or larger apertures to raise the SNR by say 5 dB to push that fog BER down further and increase capacity. Table 6 compares system performance under different weather conditions using 16-QAM with a full DSP stack at 10 km.

Figure 9.

BER and capacity comparison at SNR = 24 dB.

Table 6.

Weather Condition Performance Comparison (16-QAM).

Weather Condition	Attenuation (dB/km)	BER at 20 dB SNR	BER at 30 dB SNR	EVM (%) at 25 dB SNR	Achievable Data Rate (Gbps)	Link Margin (dB)
Clear	0.2	4.31 × 10⁻¹	3.45 × 10⁻¹	35.6	14.8	−8.3
Haze	1.5	4.52 × 10⁻¹	3.91 × 10⁻¹	38.2	10.3	−11.8
Light Fog	5.0	4.87 × 10⁻¹	4.65 × 10⁻¹	43.5	3.6	−19.5
Moderate Fog	10.0	5.00 × 10⁻¹	4.92 × 10⁻¹	47.8	0	−26.0
Heavy Fog	20.0	5.00 × 10⁻¹	5.00 × 10⁻¹	53.2	0	−40.0
Light Rain	2.0	4.62 × 10⁻¹	4.12 × 10⁻¹	39.7	8.2	−13.5
Moderate Rain	5.0	4.87 × 10⁻¹	4.65 × 10⁻¹	43.5	3.6	−19.5
Heavy Rain	10.0	5.00 × 10⁻¹	4.92 × 10⁻¹	47.8	0	−26.0

4.9 Polarization Impairments Analysis

We specifically investigated how polarization-dependent effects impact the system and how well the DSP can mitigate them.

Figure 10 presents three subplots of BER versus (a) PDL, (b) polarization crosstalk, and (c) polarization rotation angle, measured at a fixed SNR (25 dB) with moderate turbulence. In the left subplot, we introduce an increasing polarization-dependent loss (PDL) between the two channels (0 to 3 dB imbalance) . Without compensation, PDL would mean one polarization has lower SNR than the other, potentially dominating the overall BER. We found that up to 1–2 dB PDL, the DSP equalizer can adjust gains to balance the channels, and BER stays around $1 0^{- 6} - 1 0^{- 5}$ . Beyond 3 dB PDL, the BER starts rising (seen by the last point at ∼3 × 10⁻⁵ for 3 dB PDL in Figure 9). This indicates the system is somewhat robust to small PDL values typical in optics (e.g., a PBS might have 0.5 dB PDL). For larger PDL, one channel essentially under performs – an approach could be to allocate more power to that polarization at the transmitter if known in advance.

Figure 10.

Polarization effects at SNR = 25 dB with 16-QAM.

The middle subplot of Figure 10 examines polarization crosstalk, where we feed a fraction of each signal into the orthogonal polarization (e.g., due to non-ideal separation in the PBS or depolarization in the channel). We vary the crosstalk level from −30 dB (negligible) to −10 dB (significant leakage). The BER remains under $1 0^{- 5}$ for up to about −20 dB crosstalk, then rises to ∼ $1 0^{- 3}$ at −10 dB crosstalk (Cao et al., 2022). This demonstrates that the adaptive equalizer in DSP can successfully decouple the polarization channels if the leakage is not extremely high. A −10 dB polarization extinction ratio is a very poor polarizer; in practice, good polarizers have −20 to −30 dB isolation, so the system would be fine. In cases of extreme crosstalk, one could consider using a MIMO detection approach or treat the system as a 2 × 2 MIMO channel with interference – the DSP we have is essentially doing that up to its limits. The takeaway is that polarization multiplexing is viable in FSO if the transmit/receive optics maintain decent polarization purity (a reasonable assumption with modern components).

The right subplot of Figure 10 looks at polarization rotation misalignment. Here we rotate the polarization of the incoming signal relative to the receiver's reference axis (which could happen if the transmitter and receiver aren’t oriented the same). Even moderate rotation causes each received polarization to be a mix of the two transmitted ones. The BER vs rotation curve in Figure 10 shows very little penalty up to about 15°, and then a gradual increase. At 45° misalignment (meaning the polarization axes are completely crossed), BER is around $1 0^{- 4}$ . This is an impressive result – it means even if the polarization controllers are way off, the DSP can still recover the signals with tolerable BER. Indeed, in the extreme of 45°, the two received channels each see an identical mix of the two signals (each basically an equal superposition), so it's like two identical copies of a single-channel signal – in that case, our equalizer can still separate them in theory (though noise gets amplified in the process). The main point is that manual polarization alignment may not need to be extremely precise for a dual-pol FSO system; a one-time calibration to within a few degrees is sufficient, and any residual drift can be left to the electronic polarization multiplex algorithm. This is encouraging for real-world deployment, because maintaining polarization alignment over an outdoor FSO link might be challenging (differential rotation due to mirrors, etc.), but our results indicate the system is forgiving to such mismatches. Table 7 compares the impact of different polarization effects on system performance using 16-QAM at 25 dB SNR and 10 km link distance.

Table 7.

Polarization Effects Comparison (16-QAM, 25 dB SNR, 10 km).

Parameter	BER	EVM (%)	Capacity (bits/symbol)	PDL Penalty (dB)
PDL = 0 dB	3.12 × 10⁻¹	34.8	2.75	0.0
PDL = 0.5 dB	3.21 × 10⁻¹	35.1	2.72	0.3
PDL = 1 dB	3.34 × 10⁻¹	35.8	2.66	0.6
PDL = 2 dB	3.62 × 10⁻¹	37.5	2.55	1.4
PDL = 3 dB	3.91 × 10⁻¹	39.2	2.44	2.3
Crosstalk = −40 dB	3.15 × 10⁻¹	34.9	2.74	0.1
Crosstalk = −30 dB	3.23 × 10⁻¹	35.2	2.71	0.4
Crosstalk = −20 dB	3.48 × 10⁻¹	36.7	2.61	1.0
Crosstalk = −15 dB	3.82 × 10⁻¹	38.4	2.47	2.0
Crosstalk = −10 dB	4.31 × 10⁻¹	42.5	2.28	3.5
Rotation = 0°	3.22 × 10⁻¹	35.1	2.71	0.3
Rotation = 5°	3.25 × 10⁻¹	35.3	2.70	0.4
Rotation = 15°	3.30 × 10⁻¹	35.7	2.68	0.5
Rotation = 30°	3.42 × 10⁻¹	36.4	2.63	0.8
Rotation = 45°	3.58 × 10⁻¹	37.3	2.57	1.2

In this article,we have extended the potential of free space systems by using dual polarization,higher order modulations,diverse environmental conditions from scintillation to fog and effective use of adaptive modulation and DSP algorithms for improving signal at the receiver.To best of our knowledge, this combination of parameters are in study of free space optics. Table 8 provides the brief comparison with some of resembling research articles.This highlights the novelty in our work.

Table 8.

Comparison with Systems in Other Reported Works.

Feature	Ref. (Henniger and Wilfert, 2010)	Ref. (Djordjevic, 2010)	Ref. (Kumar et al., 2024)	Proposed Work
Polarization	Single Polarization	Single Polarization	Dual Polarization (DP)	Dual Polarization (DP)
Modulation	Fixed (OOK/PPM)	Adaptive Modulation	Fixed 16-QAM	Adaptive (QPSK to 256-QAM)
Data Rate	Low (Mbps range)	Variable	Fixed High Rate	70–160 Gbps
Channel Model	Log-Normal	Gamma-Gamma	Fog Scenarios	Stochastic Fog + Gamma-Gamma
Mitigation	Basic Optical Alignment	Coding & Interleaving	Polarization-Aware Design	Full DSP Stack (LMS/RLS/CMA)
Validity	Theoretical/Architecture	Simulation	Simulation	High-Fidelity Simulation
Apertures	Standard (various)	Not Specified	Specified (10–20 cm)	20 cm (Tx) / 40 cm (Rx)

5 Conclusion

We have presented a comprehensive performance analysis of a dual-polarization FSO communication system tailored for 5G and beyond networks. The system utilizes adaptive modulation and advanced DSP techniques to combat atmospheric turbulence and polarization-related impairments. Key findings from proposed system can be summarized as follows:

Using orthogonal polarization doubles the link capacity, enabling data rates well over 100 Gbps on a single FSO link under good conditions. Even under moderate turbulence and attenuation, the dual-polarized system maintained significantly higher throughput than a single-polarization link, confirming the practical value of polarization multiplexing for 5G backhaul/front haul.

Adaptive modulation proved crucial in maintaining low BER across a wide range of channel conditions. The system autonomously shifts between OOK, QPSK, 16-QAM, 64-QAM, and 256-QAM based on real-time channel quality estimates. This approach kept the BER below $1 0^{- 3}$ in all but the most extreme scenarios and maximized spectral efficiency (up to 8 bits/symbol) when channel conditions allowed. Compared to a fixed-modulation system, the adaptive scheme extended the operating range by providing graceful degradation in throughput rather than sudden link failure during deep fades or heavy attenuation.

The incorporation of digital filtering, polarization de multiplexing, and adaptive equalization algorithms markedly improved link robustness. DSP reduced the required SNR by 3–5 dB for a target BER, effectively translating to increased link margin or distance coverage. It also corrected polarization misalignment up to ∼45° and crosstalk leaks up to ∼−15 dB, indicating that polarization multiplexing can be reliably realized with manageable alignment effort. Error vector magnitudes were substantially lowered by the DSP, yielding clean constellations and enabling higher-order modulations to perform error-free.

The performance demonstrated shows that a single dual-polarization FSO link can handle the aggregated traffic of numerous 5G base stations or serve as a high-speed backbone in a front-haul network. The low latency and high security of the optical wireless channel make it suitable for mission-critical communications. The adaptive and robust nature of the link means network operators can count on a high level of availability (the link adapts to maintain a connection except in the most extreme weather, which occurs fractionally). This provides confidence in using FSO for carrier-grade services, with proper network planning for redundancy during rare outages.

In conclusion, our proposed system indicates that dual-polarization FSO technology, enhanced with adaptive modulation and sophisticated DSP, is a viable and potent solution for delivering fiber-like performance in 5G and future networks. It effectively addresses the traditional challenges of FSO (through real-time adaptation to turbulence and weather) and pushes the capacity to new heights, thereby widening the application scope of FSO links. Future networks beyond 5G, which may demand even higher data rates and more dynamic reconfiguration (for example, in a 6G ultra-dense network or a satellite-to-ground optical link scenario), will benefit from the techniques developed in this work. We have essentially shown a pathway to make FSO links smart – able to sense their environment and optimize accordingly – which is a key attribute of next-generation communication infrastructure.

Footnotes

ORCID iD

Usama Zahoor

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Ata

Alouini

M.-S.

(2022). HAPS Based FSO links performance analysis and improvement with adaptive optics correction. IEEE Transactions on Wireless Communications, 22(7), 4916–4929. https://doi.org/10.1109/TWC.2022.3230737

Bloom

Korevaar

Schuster

Willebrand

(2003). Understanding the performance of free-space optics. Journal of Optical Networking, 2(6), 178–200. https://doi.org/10.1364/JON.2.000178

Brima

A. B. S.

Ataro

Kamagate

(2021). Performance enhancement of an FSO link using polarized quasi-diffuse transmitter. Heliyon, 7(11). https://doi.org/10.1016/j.heliyon.2021.e08248

Cao

Qin

Peng

Wang

Zhang

(2022). Research on polarization coding cooperative communication scheme for FSO system. Electronics, 11(10), 1597. https://doi.org/10.3390/electronics11101597

Chapala

V. K.

Zafaruddin

S. M.

(2021). Unified performance analysis of reconfigurable intelligent surface empowered free-space optical communications. IEEE Transactions on Communications, 70(4), 2575–2592. https://doi.org/10.1109/TCOMM.2021.3139020

Djordjevic

I. B.

(2010). Adaptive modulation and coding for free-space optical channels. Journal of Optical Communications and Networking, 2(5), 221–229. https://doi.org/10.1364/JOCN.2.000221

Elamassie

Uysal

(2023). Free space optical communication: An enabling backhaul technology for 6G non-terrestrial networks. Photonics, 10(11), 1210. https://doi.org/10.3390/photonics10111210

Esmail

M. A.

(2021). Experimental performance evaluation of weak turbulence channel models for FSO links. Optics Communications, 486, 126776. https://doi.org/10.1016/j.optcom.2021.126776

Ghassemlooy

Popoola

Rajbhandari

(2019). Optical wireless communications: system and channel modelling with Matlab®. CRC press.

10.

Ghatwal

Saini

(2025). A hybrid FSO/RF communication system with DSP for long haul communication. Journal of Optical Communications, 45(s1), s2475–s2482. https://doi.org/10.1515/joc-2023-0242

11.

Henniger

Wilfert

(2010). An introduction to free-space optical communications. Radioengineering, 19(2), 203–212.

12.

Horst

Bitachon

B. I.

Kulmer

Brun

Blatter

Conan

J. M.

Montmerle-Bonnefois

Montri

Sorrente

Lim

C. B.

Védrenne

Matter

Pommarel

Baeuerle

Leuthold

(2023). Tbit/s line-rate satellite feeder links enabled by coherent modulation and full-adaptive optics. Light: Science & Applications, 12(1), 153. https://doi.org/10.1038/s41377-023-01201-7

13.

Huang

Chen

Alvarado

J. C.

Fontaine

N. K.

Mazur

Zhang

Ryf

Amezcua-Correa

Song

Min

(2021). Free-space optics communications employing elliptical-aperture multimode diversity reception under anisotropic turbulence. Journal of Lightwave Technology, 40(5), 1502–1508. https://doi.org/10.1109/JLT.2021.3130250

14.

Ijaz

Ghassemlooy

Le-minh

Zvanovec

Pérez

Pesek

Fiser

(2013). Experimental validation of fog models for FSO under laboratory controlled conditions. In 2013 IEEE 24th annual international symposium on personal, indoor, and mobile radio communications (PIMRC) (pp. 19–23). IEEE. https://doi.org/10.1109/PIMRC.2013.6666097

15.

Ishaq

Jamel

T. M.

(2026). Improving the performance of Gamma–Gamma FSO links with adaptive optics, dynamic beam steering, and strong modulation formats. Journal of Optical Communications. https://doi.org/10.1515/joc-2025-0519

16.

Karar

A. S.

Falou

A. R. E.

Barakat

J. M. H.

Gürkan

Z. N.

Zhong

(2023). Recent advances in coherent optical communications for short-reach: Phase retrieval methods. Photonics, 10(3), 308. https://doi.org/10.3390/photonics10030308

17.

Kaushal

Kaddoum

(2016). Optical communication in space: Challenges and mitigation techniques. IEEE Communications Surveys & Tutorials, 19(1), 57–96. https://doi.org/10.1109/COMST.2016.2603518

18.

Kedar

Arnon

(2004). Urban optical wireless communication networks: The main challenges and possible solutions. IEEE Communications Magazine, 42(5), S2–S7. https://doi.org/10.1109/MCOM.2004.1299334

19.

Khalighi

M. A.

Uysal

(2014). Survey on free space optical communication: A communication theory perspective. IEEE communications Surveys & Tutorials, 16(4), 2231–2258. https://doi.org/10.1109/COMST.2014.2329501

20.

Khan

M. S.

Naqvi

M. R.

Khan

M. A.

Latif

Ullah

Khan

R. D.

Wali

Leitgeb

(2013). Optical attenuation estimation from measured visibility data in Islamabad, Pakistan. In Proceedings of the 2013 18th European Conference on Network and Optical Communications & 2013 8th Conference on Optical Cabling and Infrastructure (NOC-OC&I) (pp. 203–208). IEEE. https://doi.org/10.1109/NOC-OCI.2013.6582891

21.

Kulshreshtha

Garg

A. K.

(2020). Managing 5G networks-a review of FSO challenges and solutions. 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), IEEE.

22.

Kumar

M. V.

Singh

Karuppiah

(2024). Mitigation of atmospheric losses using dual-polarized 16-QAM MIMO technique in free-space optical system in various weather conditions. International Journal of Communication Systems, 37(12). https://doi.org/10.1002/dac.70179

23.

Maswikaneng

P. S.

Adebusola

S. O.

Owolawi

P. A.

Ojo

S. O.

Ojo

J. S.

(2023). Estimation of intelligent-based quality of service in terrestrial FSO links: A case of South Africa propagation paths. Advances in Space Research, 72(9), 3856–3867. https://doi.org/10.1016/j.asr.2023.08.040

24.

Miao

Chen

X. Y.

Yin

Yuan

(2023). New results for the pointing errors model in two asymptotic cases. IEEE Photonics Journal, 15(3), 1–7. https://doi.org/10.1109/JPHOT.2023.3275578

25.

Mouhammad

C. S.

Shi

Alouini

M.-S.

(2026). Review on modulation formats and channel coding in free space optical communication: Selection criteria, trade-offs, and emerging trends. Journal of Lightwave Technology, 44, 3532–3562. https://doi.org/10.36227/techrxiv.177102188.82902634/v

26.

Prokeš

(2009). Modeling of atmospheric turbulence effect on terrestrial FSO link. Radioengineering, 18(1), 42–47.

27.

Rajput

Acharya

(2025). Free space optical communication in Indian cities; channel characteristics and link performance. Journal of Optical Communications, 45(s1), s2067–s2084. https://doi.org/10.1515/joc-2023-0181

28.

Sahrab

A. A.

Albasri

(2026). Hybrid deep learning-based adaptive modulation for free space optical (FSO) communication. IEEE Access, 14, 5913–5924. https://doi.org/10.1109/ACCESS.2025.3650714

29.

Sahu

P. K.

(2026). Review on performance enhancement techniques in FSO systems. IETE Technical Review, 43(1), 11–33. https://doi.org/10.1080/02564602.2025.2583923

30.

Sano

Yokomura

Mochizuki

Matsuda

Binkai

Mochizuki

(2025, June). Real-Time Digital Signal Processing Techniques with Optical Transceiver Prototype for Highcapacity Coherent FSO Communication. In 2025 30th OptoElectronics and Communications Conference (OECC) and 2025 International Conference on Photonics in Switching and Computing (PSC) (pp. 1–3). IEEE. https://doi.org/10.23919/OECC/PSC62146.2025.11109134

31.

Selim

H. A. O.

Abdallah

R. M.

Aly

M. H.

Shaalan

I. E.

(2026). Model-free adaptive optics for free space optical communications: A comprehensive survey. Journal of Optics, 1–21. https://doi.org/10.1007/s12596-025-03011-z

32.

Sharma

Tripathi

Kumari

(2025). FSO Systems for next generation networks: A review, techniques and challenges. Journal of Optical Communications, 45(s1), s1005–s1019. https://doi.org/10.1515/joc-2022-0288

33.

Sharma

Madhukumar

Swaminathan

(2019). Effect of pointing errors on the performance of hybrid FSO/RF networks. IEEE Access, 7, 131418–131434. https://doi.org/10.1109/ACCESS.2019.2940630

34.

Shtaif

Mecozzi

(2004). Polarization-dependent loss and its effect on the signal-to-noise ratio in fiber-optic systems. IEEE Photonics Technology Letters, 16(2), 671–673. https://doi.org/10.1109/LPT.2003.819391

35.

Song

J. I.

Yum

S. S.

Gultepe

Chang

K. H.

Kim

B. G.

(2019). Development of a new visibility parameterization based on the measurement of fog microphysics at a mountain site in Korea. Atmospheric Research, 229, 115–126. https://doi.org/10.1016/j.atmosres.2019.06.011

36.

Trung

H. D.

Van Dinh

(2013). Average symbol error rate of free-space optical communications using SC-QAM over strong atmospheric turbulence and pointing errors. 2013 third world congress on information and communication technologies (WICT 2013), IEEE.

37.

Verma

G. D.

Mathur

Bhatnagar

M. R.

(2024). Differential chaos shift keying for FSO systems: A novel approach under turbulence and boresight pointing errors. IEEE Open Journal of the Communications Society, 5, 3263–3276. https://doi.org/10.1109/OJCOMS.2024.3400034

38.

Vishwakarma

Swaminathan

Premanand

Sharma

Madhukumar

A. S.

(2024). RIS-assisted hybrid FSO/THz system with diversity combining schemes: A performance analysis. IEEE Internet of Things Journal, 11(17), 28605–28622. https://doi.org/10.1109/JIOT.2024.3405973

39.

Vyas

A. K.

Kumar

(2025). Investigation of the figure of merit to 256QAM FSO link for the next-generation terrestrial network. Journal of Optical Communications47(2), 365–373. https://doi.org/10.1515/joc-2024-0271

40.

Zhang

Song

(2024). Cooperative FSO/RF space-air-ground integrated network system with adaptive combining: A performance analysis. IEEE Transactions on Wireless Communications, 23(11), 17279–17293. https://doi.org/10.1109/TWC.2024.3452642

41.

Yang

Cheng

Tsiftsis

T. A.

(2014). Free-space optical communication with nonzero boresight pointing errors. IEEE Transactions on Communications, 62(2), 713–725. https://doi.org/10.1109/TCOMM.2014.010914.130249

42.

Zhou

Wang

Zhao

(2025). Theoretical and Experimental Investigation of Differential Modulation and Detection in FSO Systems. Photonics, 12(11), 1120. https://doi.org/10.3390/photonics12111120