The extreme computational complexity in generating a hologram makes the interactive holographic display very difficult. An optimized layered rendering method for real-time interactive holographic display based on ray-tracing technique is presented in order to overcome this challenge and realize real-time interaction of three-dimensional scenes. Ray tracing is able to render more complex and photorealistic images for different illuminations and circumstances. Rays are launched in parallel and traced to obtain ray–object intersections, and then the complex wave field of the effective pixels in different layers is determined. The occlusions between different layers are also considered to effectively reduce the overall computation. A computer-generated hologram is fused with the multilayer-hologram-diffraction results using the layered fast Fourier transform algorithm. The experimental results demonstrate the effectiveness of the proposed method. The reconstructed holographic image with real depth cues is demonstrated by the experiments, and the optical reconstruction images can be interacted in real-time.