Multi-user, low-loss, and cost-efficient characteristics are highly desired for widely deployed passive optical networks (PON), which are constrained by the upstream power combining loss induced by optical splitters in optical distribution networks (ODNs). We propose a multi-user low-upstream-loss PON utilizing graded-index multi-mode fiber (GI-MMF) and a compact ODN constructed by a multi-mode transformer (MMT) for the first time. Enabled by the MMT, the ODN achieves multi-mode multiplexing for low-loss combining in upstream and power splitting in downstream, simultaneously. An implementation approach for MMT using a single MPLC is also proposed, and a 44-mode MPLC-based MMT is designed for verification. Simulation results show that the device achieves a combining loss of lower than 1.13 dB and a modal crosstalk of lower than −19.5 dB in upstream and an excess splitting loss of less than 1 dB in downstream for all modes. The bandwidth characteristics and tolerance are also investigated through simulation.