-
为了实现快速焊点质量检测,需要对原始图像进行预处理,即图像增强、均衡化处理以及平滑补偿,从而突出待测图像特征,降低焊点异常提取的难度。由于相机对环境光强变化的动态适应性有限,故首先对照明变化导致的成像质量差异进行预处理, 利用图像增强技术调节原始图像亮度范围,本系统中采用了直方图均衡配合灰度拉伸算法完成图像灰度调节。算法步骤如下。
-
将原始图像中的灰度直方图均匀化分布,使每个灰度级对应的像素数一致,进而增加其动态范围。设图像灰度级为横坐标,灰度级像素出现频率为纵坐标,灰度级∈[0,L-1],则灰度直方图函数可表示为:
$ h({r_k}) = {n_k}, \left( {k \in \left[ {0, L1} \right]} \right) $
(1) 式中,rk为第k个灰度级,nk为灰度级rk的像素数, L表示图像的最大灰度值。
-
通过灰度拉伸算法对图像中每个点的灰度值进行拉伸处理。设f(x,y)为原始图像,其灰度级∈[l1,l2]。为了得到[l3,l4]范围内图像函数g(x,y),需要进行转换,转换函数可表示为
$ T = ({l_4}{l_3}){({l_2}{l_1})^{ - 1}}(r{l_1}) + {l_3} $
(2) 式中,r表示图像像素点的灰度值。由此可见,经过拉伸函数处理后,图像中所有像素点的灰度值均完成灰度变换,图像对比度得到提高,实现图像增强。
-
为了降低焊点图像的不均匀性,抑制杂散光等的影响,选取平滑算法优化图像质量,提高图像信噪比。采用中值滤波的方式对所有像素对应值进行排序处理,对所选图像区域所有像素点灰度值取中值Qmid,其图像矩阵可写为:
$ \mathit{\boldsymbol{g}}\left( {i, j} \right) = T{Q_{{\rm{mid}}}}\sum\limits_{\left( {i, j} \right) \in C} {\mathit{\boldsymbol{f}}(i, j)} $
(3) 式中,(i, j)为算法选择区域中滤波区域对应的像素位置,(i, j)在图像边界区域中,C表示图像区域。由于中心像素选取的灰度值是根据原始像素及其相邻像素的灰度值统计分类获得的,故本滤波算法可用于非线性图像处理。利用此中值滤波算法可以剔除误差较大的孤立像素,对图像全局影响不大,同时大大降低离散噪声对图像的影响。
-
深度学习(deep learning, DL) [14]是学习样本数据的内在规律及表示层次的一种数据处理方法,属于机器学习算法的一个方向,重点应用于图像识别方面。其中,深度学习算法中具有深度结构反馈运算特性的有卷积神经网络(convolutional neural network, CNN)[15-16],可对原始图像样本进行数据处理。其结构由输入层、卷积层、汇集层、连接层和逻辑回归层构成。系统通过卷积核实现卷积操作,指示符为长度、宽度及深度,其卷积神经网络传递函数的表达式有:
$ \begin{array}{l} \mathit{\boldsymbol{S}}\left( {i, j} \right) = \left( {\mathit{\boldsymbol{I}}*\mathit{\boldsymbol{K}}} \right)\left( {i, j} \right) = \\ \sum\limits_x {\sum\limits_y {\mathit{\boldsymbol{I}}\left( {x, y} \right)\mathit{\boldsymbol{K}}\left( {ix, jy} \right)} } \end{array} $
(4) 式中,I为图像矩阵,*表示卷积,K(i-x, j-y)为卷积核。x和y为i和j像素点对应的图像位置,x和y的集合为焊点区域。将相邻像素点与卷积核对应点相乘再求和,获得该点的像素值,该操作遍历每个像素点,从而得到卷积神经网络的前向传播。
-
卷积神经网络属于前馈式神经网络,训练样本不需要太大,最重要的是其对图像畸变、模式变化具有较高的容错能力,所以对大量焊接图像不同缺陷类型具有较好的兼容性。该网络结构由输入层、卷积层、降维层、连接层以及输出层构成。焊接图像进入输出层后与核函数进行卷积,由激励函数输出神经元,实现特征图像的获取并进入下一层。设第n卷积层的第j个特征为:
$ {\mathit{\boldsymbol{X}}_{j, n}} = f(\sum\limits_{i \in {\mathit{\boldsymbol{N}}_j}} {\left( {{\mathit{\boldsymbol{X}}_{i, n1}} \times {k_{i, j, n}}} \right)} + {b_{j, n}}) $
(5) 式中,f()为激励函数,Nj为上层特征图像集合,ki, j, n为卷积核权值,bj, n为和偏置。获取特征后导入降维层减小运算量。则第n降维层第j个特征图有:
$ {\mathit{\boldsymbol{Y}}_{j, n}} = f(\eta g\left( {{\mathit{\boldsymbol{Y}}_{j, n-1}}} \right) + {b_{j, n}}) $
(6) 式中,g()为降维函数,η为积偏置。
因为焊接异常图像特征本身具有多样性,每种缺陷虽然存在相似性,但尺寸面形仍具有较大差异,所以需要具有数据选择的自适应性,由此提出了通过对数据自组织映射实现对卷积神经网络的改进。
设输入数据样本为xi, p(p=1, 2…, m, i为分量),输出为Cij,则输出方程有:
$ {\mathit{\boldsymbol{C}}_{ij}} = \sigma \left( {\sum\limits_{i = 1}^n {{\omega _j}{\mathit{\boldsymbol{x}}_{i, p}}} } \right) $
(7) 式中,σ()为单调函数,ωj为第j个神经元的权值。利用赫布规则化简可得:
$ \left\{ {\begin{array}{*{20}{l}} {{{\left. {\frac{{{\rm{d}}{\omega _j}}}{{{\rm{d}}t}}} \right|}_{_{Cij = 1}}} = \eta \left( {{\mathit{\boldsymbol{C}}_{ij}}} \right)\left( {{\mathit{\boldsymbol{x}}_{i, p}} - {\omega _j}} \right)}\\ {{{\left. {\frac{{{\rm{d}}{\omega _j}}}{{{\rm{d}}t}}} \right|}_{_{Cij = 0}}} = 0} \end{array}} \right. $
(8) 式中,t是时间。
由此得到输出层相当于具有中心分布的样本数据,然后利用训练网络与学习网络配合实现数据分类的加速收敛。
-
在改进型卷积神经网络框架基础上构建焊接质量识别算法的模型,其中,损失函数利用Cross Entropy实现。在本算法中利用反馈数据自适应的方式完成,从而依据每次的测试结果进行参量调整。基于自组织映射的卷积神经网络算法参量设置与实现步骤如下:(1)在训练网络中设卷积神经网络的核函数、激活函数等,确定学习速率初值并构建邻域初值;(2)将标记样本导入训练网络,由(5)式和(6)式获取特征样本;(3)在邻域权值竞争基础上完成异常图像特征点调整,并将获得的新特征点加入样本标记;(4)将训练网络导入学习网络,迭代寻找最优值。
其工作流程如图 2所示。
-
实验中选用UNIQ-3000型CCD相机完成焊机图像采集,共采集5000张,其中包括焊接良好和焊接异常两大类,焊接异常中分为小球、桥连、虚焊、少锡、偏球5种类型。3000张焊接图像构成算法模型训练集,1000张焊接图像构成模型验证集,1000张焊接图像用于测试。基于深度学习的焊接异常识别算法在Ten-sorflow环境下采用Python编译实现。
-
为了检验对比基于深度学习的焊接异常图像识别算法的效果,对其漏检率与误检率量化分析。与传统的canny边缘提取算法进行对比,从而考察本算法在边缘特征信息保留方面的性能;同时,与k均值聚类算法进行对比,从而考察本算法在特性分类方面的性能。同时采用3种方法针对5种典型焊接异常进行识别检测,3种方法获得的处理结果如图 3所示。
由图像处理结果可知,canny算法的边界处理效果最好,清晰度高、对比度强,有利于识别桥连和小球类型的焊接异常(例如桥连缺陷S32和S33、小球缺陷S23),但焊点中的分布信息被明显削弱,造成对虚焊、偏球、少锡类型的焊接异常识别能力降低。k均值算法的边界处理效果虽然清晰度不如canny算法好,但采用反馈型阈值解析仍可以获得很好的边缘检测精度。另外,k均值算法获得的焊点图像仍保留了同灰度级的差异,可有效地识别偏球缺陷(例如偏球缺陷S12),但对虚焊和少锡的特征分类效果并不明显。本算法中边界处理效果与k均值算法相近,同时,焊点特征保留更为明显,不但有表征偏球缺陷的同灰度级差异,还有梯度变化信息,可以对虚焊和少锡问题进行阈值量化分析(例如虚焊缺陷S24,少锡缺陷S22)。对验证集的检测结果进行统计分析,获得3种方法漏检率与误检率数据分布如表 1所示。
Table 1. Comparison of missed detection rate and false detection rate of three methods
detection
methodaverage recognition accuracy/% weld bridge small ball partial ball virtual weld missing solder FDR MDR FDR MDR FDR MDR FDR MDR FDR MDR canny 0 0 0.1 0.1 12.4 13.3 16.5 17.5 14.3 21.4 k-means 0 0 0.7 0.2 7.3 6.5 15.7 16.2 15.5 19.6 DL 0 0 0.6 0.2 1.4 1.1 2.7 2.4 2.6 2.9 由测试结果可知,3种算法对桥连的判断都能达到100%识别,即误检率和漏检率均为0,分析认为桥连的图像范围大,易识别;相比而言,小球缺陷的检测对边界特征信息要求更高,canny算法的边界处理效果更好,其误检率和漏检率均低于k均值算法和本算法;对于偏球缺陷而言,其不但涉及边界分布还需考虑焊点灰度分布特性,canny算法的识别效果最差,k均值算法优于canny算法,本算法效果最好,误检率与漏检率相比canny算法减小了一个数量级;对于虚焊和少锡缺陷而言,都需要对焊点的灰度分布梯度进行解析,传统的两种算法测试效果相近,本算法识别精度有所下降,但总体水平仍优于前两种方法。实验结果表明,相比传统图像分割算法, 本算法可以更好地获得各种焊接缺陷图像,从而为焊接质量分析提供了更好的支撑。
-
算法验证后采用未知缺陷分布情况的焊接图片进行测试,将1000张测试图片分别输入3种算法后迭代运算,分类后完成数据统计分析,对误检率、漏检率和召回率进行统计。为了对图像测试数据进行综合评价,在1000张测试图像中等比例插入了5种不同缺陷形式的焊接图像。统计结果显示,canny算法在测试集中的误检率为7.8%、漏检率为9.7%、召回率为90.75%;k均值算法在测试集中的误检率为7.4%、漏检率为8.3%、召回率为92.15%;本算法在测试集中的误检率为1.6%、漏检率为1.7%、召回率为98.45%。由此测试集统计数据可知,3种算法的识别能力与验证集中各测试结果的加权平均值相近,说明基本可以表征各算法的实际测试效果,也验证了本算法具有更好的焊接异常检测识别性能。
改进型卷积神经网络焊点缺陷识别算法研究
Research on solder joint defect recognition algorithm based on improved convolutional neural network
-
摘要: 为了同时对多种焊点缺陷类型进行快速识别,解决现有焊接异常图像识别算法误检率与漏检率偏高的问题,设计了基于改进型卷积神经网络的深度学习算法。利用自组织映射分类技术,提高了卷积神经网络的数据选择自适应性,结合自适应矩估计分析, 约束了焊接异常图像中特征集合的收敛条件。实验中将5种常见焊接异常图像以等比例随机分布的形式放入训练集、验证集和测试集中,再分别用传统识别算法(canny算法和k均值算法)和该算法进行测试。结果表明,对于桥连缺陷,3种方法均无误检、无漏检;对于小球缺陷,3种方法均符合要求,而canny算法的检出能力最优;对于偏球缺陷, 3种算法的误检率分别是12.4%, 7.3%和与1.4%,漏检率分别是13.3%, 6.5%和1.1%;对于虚焊和少锡缺陷,该算法相比传统算法精度高约1个数量级。该算法在对多种焊点缺陷类型识别中具有明显优势。Abstract: In order to quickly identify a variety of solder joint defect types and solve the problem of high false detection rate and missed detection rate of traditional welding abnormal image recognition algorithms, a deep learning algorithm based on an improved convolutional neural network was designed. The self-organizing map classification technology improves the data selection adaptability of the convolutional neural network. At the same time, it combines the adaptive moment estimation analysis to restrict the convergence conditions of the feature set in the welding abnormal image. In the experiment, five kinds of common welding anomaly images were randomly distributed into the training set, verification set, and test set in the form of a random distribution of equal proportions. They were tested by traditional recognition algorithms (canny algorithm and k-means algorithm) and this deep learning algorithm, respectively. The results show that, three methods have no false detection and no missed detection for bridge defects. Three methods meet the requirements for small ball defects, and the detection ability of the canny algorithm is the best. For partial ball defects, the false detection rates of three algorithms are 12.4%, 7.3%, and 1.4%, and the missed detection rates of three algorithms are 13.3%, 6.5%, and 1.1%, respectively. For virtual soldering and tin-less defects, the accuracy of this algorithm is about an order of magnitude higher than that of traditional algorithms. It can be seen that this algorithm has obvious advantages in identifying multiple types of solder joint defects.
-
Key words:
- image processing /
- deep learning /
- convolutional neural network /
- gray gradient
-
Table 1. Comparison of missed detection rate and false detection rate of three methods
detection
methodaverage recognition accuracy/% weld bridge small ball partial ball virtual weld missing solder FDR MDR FDR MDR FDR MDR FDR MDR FDR MDR canny 0 0 0.1 0.1 12.4 13.3 16.5 17.5 14.3 21.4 k-means 0 0 0.7 0.2 7.3 6.5 15.7 16.2 15.5 19.6 DL 0 0 0.6 0.2 1.4 1.1 2.7 2.4 2.6 2.9 -
[1] SUN X, LI X G, LI J F, et al. Research progress of image super-re-solution restoration based on deep learning[J]. Acta Automatica Sinica, 2017, 43(5): 697-709. [2] FENG W, WANG Y D, ZHANG L. Deep feature extraction and cla-ssification recognition algorithm based on weighting and dimension reduction[J]. Laser Technology, 2018, 42(5): 666-672(in Chin-ese). [3] PENG F Q, CHEN Zh J, CHEN T, et al. Portable surface acoustic wave radio-frequency identification system[J]. Piezoelectrics & Acoustooptics, 2018, 40(3): 340-343(in Chinese). [4] MA J L. Research on art appraisal based on hyperspectral imaging technology[J]. Cultural Relics Protection and Archaeology, 2018, 30(3):15-19. [5] ZHU Zh M, LV Zh K, SONG R Ch, et al. PCB solder joint quality inspection based on machine vision[J]. Controlling Engineering of China, 2017, 24(4):721-727(in Chinese). [6] CHEN Y Q, HU G Q, HUANG D Q, et al. Research on positioning technology of welding spot on the silver net based on robot vision[J]. Aeronautical Manufacturing Technology, 2017, 60(9): 50-54(in Chinese). [7] SHAH H N M, SULAIMAN M, SHUKOR A Z, et al. Butt welding joints recognition and location identification by using local threshold-ing[J]. Robotics and Computer-Integrated Manufacturing, 2018, 51:181-188. doi: 10.1016/j.rcim.2017.12.007 [8] XIA Ch X, YANG Ch, ZHAO X, et al. Surface detection and analysis of PCB based on image processing[J]. Video Engineering, 2018, 42(8): 28-32(in Chinese). [9] SUN Sh B, YIN L H, YAN X L, et al. Defect recognition method of weld image based on texture feature[J]. Computer Applications and Software, 2018, 35(5):248-252. [10] KAZANTSEV I G, LEMAHIEU I, SALOV G I, et al. Statistical detection of defects in radiographic images in nondestructive testing[J]. Signal Processing, 2002, 82(5):791-801. doi: 10.1016/S0165-1684(02)00158-5 [11] MA L Y, YUAN T Sh. Quality inspection of solder joint based on extreme learning machine[J]. Computer Engineering and Applications, 2018, 54(12): 214-239(in Chinese). [12] FU L. Application of high temperature laser welding innovation technology to expand advanced high strength steel [J]. Aerospace Ma-nufacturing Technology, 2018(1): 70-72. [13] ZUO K Zh, ZOU Y Y, LI P F, et al. Iterative based identification method for boundary feature points of unequal thickness laser tailor welded seams[J]. Journal of Applied Lasers, 2018, 38(3): 114-119. [14] LI J Y, FANG L Y, HU D C, et al. A recognition and correction method of declining BGA X-ray image based on transformation matrix[J]. High Power Laser and Particle Beams, 2018, 30(10): 143-148(in Chinese). [15] LIU M J, LI L Y, GUO W B. Based on machine vision solder Joint detection algorithm research[J]. Chinese Journal of Electron Devices, 2017, 40(4): 1015-1020(in Chinese). [16] LEI H L, ZHANG B H. Crowd counting algorithm based on multi-model deep convolution network integration[J]. Laser Technology, 2019, 43(4): 476-481(in Chinese).