Pose-based action recognition has aroused increasing attention for its broad application prospects and excellent performance. Though the pose-based action recognition methods have been significantly advanced, pose-based action recognition remains a challenging task for various human action categories and subtle changes in human poses. To solve those problems, we propose pose-based multisource networks. First, human pose features are extracted from the raw video, followed by a filtration. Then, using a convolutional neural network (CNN) and long short-term memory (LSTM), the extracted pose sequence is fed into the proposed multisource networks. Subsequently, the CNN-based spatial model processes the relative position in each frame, and the LSTM-based temporal model is built to learn the temporal correlation of pose sequence. Afterward, the temporal model contains three sublevels to fully exploit the subtle information in the temporal domain. Finally, the experimental results verify the effectiveness of the proposed approach on SUB-JHMDB, MPII Cooking Activities, SYSU 3D Human-Object Interaction, and NTU RGB+D.