Abstract: Traditional algorithm of in-plane displacement measurement by speckle photography is based on the power spectrum of Fourier transform directly. However, the measured minimum displacement is limited by speckle size. In order to solve this problem, a novel algorithm based on the phase angle of Fourier transform was proposed. Theoretical analysis showed that the minimum and maximum displacement measured by this new algorithm was determined by the photoelectric converter, such as charge-coupled device or complementary metal oxide semiconductor. The size of speckle patterns had no effect on displacement. The result shows that the minimum and maximum displacement measured by new method can achieve 2pixel spacings and the half of photo-sensitive area width of photoelectric conversion device respectively. The actual measurement is in agreement with the theoretical analysis.