Abstract: Recently, there has been a significant development towards panoramic videos in virtual reality technology, which can provide high perceptual quality and immersive experience. Consequently, the study of providing low latency panoramic videos service has become a hot topic. In this paper, factors related to latency that affect the quality of experience of users in panoramic video transmission system are analyzed, and a low latency method for panoramic video transmission is proposed. By using inter-frame prediction techniques in random access configuration, transmission delay caused by switching viewport is reduced, and the user’s visual experience quality is also improved. Through theoretical analysis and simulation, the proposed method can reduce the time delay of viewport switching to 2.4 frames on average and reduce the video bit rate of user viewport switching by 20.23% on average for different video sequences, without affecting the quality of user experience. This method can reduce the latency and improve the bandwidth utilization, which provides a new technical approach for efficient transmission of panoramic videos.