Abstract: In order to quickly identify a variety of solder joint defect types and solve the problem of high false detection rate and missed detection rate of traditional welding abnormal image recognition algorithms, a deep learning algorithm based on an improved convolutional neural network was designed. The self-organizing map classification technology improves the data selection adaptability of the convolutional neural network. At the same time, it combines the adaptive moment estimation analysis to restrict the convergence conditions of the feature set in the welding abnormal image. In the experiment, five kinds of common welding anomaly images were randomly distributed into the training set, verification set, and test set in the form of a random distribution of equal proportions. They were tested by traditional recognition algorithms (canny algorithm and k-means algorithm) and this deep learning algorithm, respectively. The results show that, three methods have no false detection and no missed detection for bridge defects. Three methods meet the requirements for small ball defects, and the detection ability of the canny algorithm is the best. For partial ball defects, the false detection rates of three algorithms are 12.4%, 7.3%, and 1.4%, and the missed detection rates of three algorithms are 13.3%, 6.5%, and 1.1%, respectively. For virtual soldering and tin-less defects, the accuracy of this algorithm is about an order of magnitude higher than that of traditional algorithms. It can be seen that this algorithm has obvious advantages in identifying multiple types of solder joint defects.