Abstract: In order to achieve high-degree isolation in a transceiver system, an isolation optical system was put forward based on polarization and analyzed by means of Jones matrix. The feasibility of polarization isolation in a coaxial optical transceiver was verified. The space separation was used to suppress the echo into the receiving system. The influence of the wedge angle of polarization beam splitter on transmitting-receiving isolation was studied. The results show that, the transmitting-receiving isolation of 100dB is achieved with a certain wedge angle. The results will be of guiding significance for the design of transmitting-receiving coaxial optical systems.