The performance of a bidirectional silicon mode-division (de)multiplexer has been analyzed in the radio over fiber passive optical network system. The device uses two grating-assisted contra-directional coupler sections to demultiplex the transmitted fundamental, first-order, and second-order quasi-transverse electric (TE) modes. A bandwidth of 791 GHz and acceptable crosstalks of −25.73  dB (TE_0m), −10.30  dB (TE_1m), and −11.84  dB (TE_2m) have been achieved from the 2.5D finite-difference time-domain method at a wavelength of 1550 nm. The system uses parabolic index multimode fiber (MMF) and free-space optics channels alternatively. A 120-Gbps data rate has been obtained using differential quadrature phase-shift keying scheme and hybrid mode-division multiplexing techniques. The system maintains an acceptable bit error rate (≤10^−  12) over transmission distances of ≤5.3  km MMF and ≤1.9  km for free-space optics link with receiver sensitivities of ≥  −  26.2 and ≥  −  24.8  dBm, respectively.