Abstract: In order to improve the quality of in-focus reconstructed images, random phase pupil was used to replace the pinhole in optical scanning holography system. After theoretical analysis and experimental verification, the comparison of signal to noise ratio, correlation coefficient and frequency spectrum distribution between the original and reconstructed images in the optical scanning holography system and the system based on random phase pupil was gotten respectively. The results show that under the same experimental condition, the reconstructed image from optical scanning holography based on random phase encoding has higher signal-to-noise ratio and correlation coefficient. The change trend of the reconstructed image spectrum is similar to that of the original image.