A method for reducing the coherent Rayleigh noise (CRN) in a Rayleigh and Brillouin (RB) self-heterodyne detection Brillouin optical time domain reflectometer (BOTDR) system using phase shift keying (PSK) pulse encoding is proposed and verified experimentally. A phase modulator and an electro-optic modulator driven by the same arbitrary function generator perform PSK pulse encoding modulation on the signal, and the traces obtained by PSK encoded pulses with different encoding patterns are superposed and averaged to reduce the CRN. The experimental results show that, in the case of 10,000 times average to eliminate other random noise effects, after using PSK pulse encoding, the amplitude fluctuation and signal-to-noise ratio at the fiber end of the RB self-heterodyne detection BOTDR sensing system are reduced by 1.12 dB and improved by 3.01 dB, respectively. In addition, the measurement accuracy and measurement stability of the Brillouin frequency shift and temperature are effectively improved. The proposed method offers a robust solution for reducing the CRN in the RB self-heterodyne detection BOTDR sensing system, facilitating high-precision and long-distance sensing capabilities.