In a charge-coupled device-based visual tracking system (VTS), the control bandwidth is restricted by the image sensor’s time delay, which hinders a high tracking performance. Compared with the feedback control, a delay-compensation control method called the Smith predictor (SP) could improve the tracking performance by moving the delay outside the closed loop to relax the constraint on the controller. However, the performance improvement is limited because delay still exists in the system, resulting in a deviation between input and output. In addition, in the previous VTS with SP, the controller was commonly adjusted based on experience, which would easily lead to an overly conservative design. To solve these problems, an enhanced Smith predictor (ESP) by Kalman filter prediction is introduced, and the controller design criteria based on control stability is quantitatively analyzed. Based on the typical SP, an additional Kalman filter is added to the forward path to predict the current position based on the past position information, which further eliminates the effect of the delay. Then, according to the small gain theorem, the design principle of the controller when there is a model mismatch is given to release the control performance as much as possible. Experimental results confirm that the proposed ESP method has a stronger error suppression characteristic in a low frequency when compared with the SP method.