人机智能协同关键技术及其在智能制造中的应用

项目概要

以深度学习为基础的AI技术的难解释、鲁棒性差等缺点将人重新请回控制的闭环中，形成了“人机智能 协同”新型人机协同范式，成为自动驾驶、智能制造等领域不可替代的关键技术，亟需全新的概念框架和技术方法。依托项目团队及其合作者在人机混合智能系统自主性理论和方法方面的前期研究成果，及与联宝科技（联想集团子公司、合肥市第一大企业）的前期合作基础，本项目将前期理论成果应用于智能制造领域，解决人机智能协同在智能制造中的一系列关键共性技术问题，形成面向智能制造的人机智能协同系统性的技术，助力智能制造产业升级。

主要研究内容

• 利用“人在环上”人机智能协同有效提升智能制造系统性能的关键共性技术
• 利用“人在环内”人机智能协同有效提升智能制造系统性能的关键共性技术
• 人机智能协同用于智能制造提升生产效率和可靠性的关键共性技术

研究成果

Journal Articles

1. 基于动态信道切换的无线网络化控制系统的资源调度策略 郝小梅, and 赵云波 高技术通讯 2023 [Abs] [pdf]

   本文针对通信网络中存在竞争和非竞争信道的无线网络化控制系统，提出了一种基于估计 器的信道选择策略，在保证控制系统稳定性的同时尽可能地节约了宝贵的非竞争信道资源。在无 线网络化控制系统中，控制信号通过竞争信道传输时可能发生数据包丢失，导致执行器无法收到 实时的控制信号。而传感器端未知控制信号的实际传输情况，因而也无法得知每个时刻执行器所 使用的控制信号。针对这种情况，本文首先设计了估计器来估计执行器端上一时刻实际使用的控 制信号，再通过信道选择策略来约束执行器端使用控制信号的误差。然后，在所提信道选择策略 下设计控制器来保证控制系统稳定。最后，通过数值仿真验证了所提算法的有效性。

2. 基于优先级预测器的无线网络化控制系统的动态传输策略 闫文晓, and 赵云波 高技术通讯 2023 [Abs] [pdf]

   文针对无线通信网络中存在丢包的多包传输无线网络化控制系统，提出了一种基于预测 器的动态传输策略，在几乎不增加信道资源占用的情况下显著提升系统稳定性。在多包传输的无 线网络化控制系统中，由于通信资源的限制，传感器到控制器间的数据传输中出现丢包问题，影 响控制系统性能。针对这个问题，本文首先设计了优先级预测器来预测下一时刻每个传感器数据 对系统稳定性的影响，帮助系统决策每个传感器的发送优先级，再通过传输调节器对不同优先级 传感器补偿相应的随机退避时间上限，进而让优先级高的传感器在随机退避的方式下优先传输， 然后在此策略下设计控制器使系统稳定。最后通过数值仿真验证了本文策略的有效性。

3. 面向人机序贯决策实现共享控制下的仲裁优化 张倩倩, 赵云波, 吕文君, and 陈谋 中国科学：信息科学 2023 [doi] [pdf]
4. Estimation Based Approximating Control for Wireless Networked Control Systems Qipeng Liang, Qiaohui Zhu, Yu Kang, and Yun-Bo Zhao J. Univ. Sci. Tech. China 2021 [Abs] [doi] [pdf]

   The control design and system analysis of wireless networked control systems with unknown roundtrip delay characteristics are investigated. An estimation based approximating control strategy is proposed to stabilize the systems by using delay characteristics in a practically feasible way. The strategy first uses a delay transition probability estimator to obtain the delay characteristics estimation by measuring delay data online, and then uses an approximating controller to take advantage of the estimation. On this basis, a packet delay variation detector is designed, making the strategy adaptive to the variation of delay characteristics. The sufficient conditions to ensure the closed-loop system being mean-square uniformly ultimately bounded are given, with also the controller gain design method. The effectiveness of the proposed approach is verified numerically.

Conference Articles

1. Board-Level Functional Test Selection Method Based on Fault Tree Analysis Yaoyao Li, Kangcheng Wang, Yu Kang, Yunbo Zhao, and Peng Bai In 6th International Symposium on Autonomous Systems (ISAS2023) 2023 [Abs] [pdf]

   With the increasing complexity of the circuit board, the cost of board-level functional test ensuring the board quality becomes dramatically high. Data-driven-based test selection methods have been widely studied for test-cost reduction. However, existing test selection methods tend to overfit due to overlooking the root causes of faulty boards. To address this issue, a test selection method based on reliability analysis is proposed. A fault tree oriented to the board-level functional test is established for analyzing the reliability of the board and test items. The reliability analysis result is then effectively utilized to formulate a test selection method. Three indices are introduced to evaluate the test efficiency and the test quality. Experimental results demonstrate the effectiveness of the proposed method.

2. A Feature Engineering-based Method for PCB Solder Paste Position Offset Prediction Binkun Liu, Yunbo Zhao, Yu Kang, Yang Cao, Peng Bai, and Zhenyi Xu In 6th International Symposium on Autonomous Systems (ISAS2023) 2023 [Abs] [pdf]

   Solder paste printing position offset is a common type of defective printed circuit board (PCB) printing, and accurate position offset prediction helps to avoid the production of defects, thus improving efficiency. The existing methods mainly use the powerful nonlinear fitting ability of deep learning to learn the variation pattern of solder paste printing quality to achieve a good prediction. However, factories also focus on the interpretability of the model, and existing methods are difficult to give the basis for decisions, so there are still limitations in the practical application. To solve this problem, we propose a Support vector machine (SVM) approach, in which we manually design 14 statistical features based on the original data, then the resampling reduces the effect of data imbalance and achieves PCB pad-level offset prediction. Finally we verified on about one week of real solder paste printing production data and achieved good experimental results.

3. Defect Detection of Laptop Appearance Based on Improved Multi-Scale Normalizing Flows Jie Zhang, Yunbo Zhao, and Zerui Li In The 38th Youth Academic Annual Conference of Chinese Association of Automation 2023 [Abs] [pdf]

   In the laptop production process, timely detection of appearance defects is essential to ensure product quality. At present, there are many shortcomings in the manual visual inspection-based method on the laptops production line. In addition, due to the wide variety of laptop appearance defects and extreme differences in defect scales, existing defect detection algorithms perform poorly in the field of laptop appearance inspection. In response to the above problems, this paper proposes a defect detection algorithm based on improved multi-scale normalizing flows. First, the multi-level features extracted from the backbone network are fused by using the pyramid feature fusion module to obtain multi-scale features with rich semantic and spatial information. Then, the effective density estimation of the multi-scale features is achieved by fusing the normalizing flows of attention mechanisms. Finally, the defects are detected and localized based on the output likelihood values. The experimental results demonstrate the effectiveness of the proposed method in detecting and locating appearance defects.

4. A Robustness Benchmark for Prognostics and Health Management Binkun Liu, Yun-Bo Zhao , Yang Cao, Yu Kang, and Zhenyi Xu In 2022 41st Chin. Control Conf. CCC 2022 [Abs] [doi] [pdf]

   With the rise of intelligent manufacturing, prognostics and health management(PHM) has developed rapidly as an important part of intelligent manufacturing.Existing deep learning-based PHM methods are data-dependent. However, sensor data often contains noise and is redundant and high-dimensional, making it difficult for the PHM methods to learn a stable set of model parameters, so the methods are likely to be wrong when disturbed. However, the factory hopes that the PHM methods are robust enough to adapt to various disturbances, so it is necessary to perform robustness evaluation on the existing methods in advance for easy deployment. Although the existing robust theoretical analysis methods for neural networks can obtain tight robust boundaries, they consume a lot of computing resources and are difficult to scale to large neural networks. To slove this problem, We design a benchmark for robustness analysis of large deep learning PHM models, in which we test the model robustness using a variety of perturbations to simulate the actual production environment of the factory. Specifically, Gaussian noise is used to test the robustness of the model to background noise; random mask is used to test the robustness of the model to data loss. We hope that our robustness benchmark can serve as a reference for designing PHM models to improve the robustness of factory PHM models.

5. Anomaly Detection for Surface of Laptop Computer Based on Patchcore Gan Algorithm Huijuan Zhu, Yu Kang, Yun-Bo Zhao , Xiaohui Yan, and Junqiang Zhang In 2022 41st Chin. Control Conf. CCC 2022 [Abs] [doi] [pdf]

   Timely detection of notebook appearance defects is an important means to prevent products from being delivered to customers before leaving the factory.In industrial production, more emphasis is placed on fast and accurate detection methods, but the existing difficulties: 1. Defect samples are rare and difficult to obtain; 2. In high-resolution images, there are slight differences between abnormal samples and normal samples; 3. Slowly detection and insufficient accuracy.The existing methods mainly use a large amount of abnormal samples, so it is difficult to extend to the field of notebook appearance anomaly detection.To solve this problem, we designed a method that firstly uses unsupervised PatchCore which the algorithm was trained on normal samples and Defect GAN is used in test phase. To create a large number of verisimilitude abnormal samples and test these samples with PatchCore. On TKP-Surface datasets, the AUROC score of image-level anomaly detection achieves 96.1%, which meets the requirements of industrial applications.

Book Chapters

1. SMT Component Defection Reassessment Based on Siamese Network Chengkai Yu, Yun-Bo Zhao , and Zhenyi Xu In Methods and Applications for Modeling and Simulation of Complex Systems 2022 [Abs] [doi] [pdf]

   In the SMT process, after component placement, checking the quality of component placement on the PCB board is a basic requirement for quality control of the motherboard. In this paper, we propose a deep learning-based classification method to identify the quality of component placement. This is a comparison method and the novelty is that the siamese network is trained to extract the features of the standard placement component map and the placement component map to be inspected and output the probability of similarity between the two to determine the goodness of the image to be inspected. Compared to traditional hand-crafted features, features extracted using convolutional neural networks are more abstract and robust. In addition, during training, the concatenated network pairs the sample images to expand the amount of training data, increasing the robustness of the network and reducing the risk of overfitting. The experimental results show that this method has better results than the general model for the classification of placement component images.

patent

1. 锡膏印刷机参数调整数据处理软件V1.0 许镇义, 刘斌琨, 康宇, 曹洋, and 赵云波 2022 [pdf]
2. 用于PCB微小缺陷检测的单帧目标检测方法及存储介质 许镇义, 桂旺友, 曹洋, 康宇, and 赵云波 2022 [Abs] [pdf]

   本发明的一种用于PCB微小缺陷检测的单帧目标检测方法及存储介质,包括以下步骤：S1、获取PCB图像信息,并对图像进行数据预处理；S2、构建网络模型,将处理后的图像输入VGG-16特征提取网络,并对不同层次的特征进行融合,同时消除融合过程中所带来的负面影响；S3、对模型进行训练,并根据训练得到的结果评估性能。本发明利用注意机制来学习跨通道融合的特征之间的关系,并利用shuffle模块消除融合后的混叠效应。提出了非最大抑制方法,以减轻PCB图像的重叠效应。语义上升模块通过将不同层次的特征进行融合,不仅使低层次的特征具备丰富的语义信息,还能让高层次的特征具备更好的回归性,在目标分类与定位方面显著增强,能够更好地适应微小目标的检测。

3. 印刷电路板的微小缺陷检测方法及存储介质 许镇义, 余程凯, 曹洋, 康宇, and 赵云波 2022 [Abs] [pdf]

   本发明的一种印刷电路板的微小缺陷检测方法及存储介质,包括以下步骤,S1、获取PCB缺陷样本数据并进行数据预处理；S2、在PCB训练集的边界框上使用k-means聚类来找到符合要求的anchor尺度；S3、采用多尺度特征金字塔结构提取特征,对主干卷积网络中得到自底向上的特征图进行上采样,得到自顶向下的特征图,然后将其与自底向上的特征图逐元素相加得到最终的特征图；S4、通过计算损失,训练网络参数。本发明通过数据增强技术提供深度学习所需要的充足训练数据,利用k-means聚类设计合理的锚点尺度,再将特征金字塔与Faster R-CNN网络相结合,加强了不同层次特征图之间的关系,从而实现对PCB微小缺陷的检测。本发明提高检测效率,并且能够适应多种缺陷检测,适应性强。

4. 锡膏印刷机离线故障预测软件V1.0 赵云波, 刘斌琨, 曹洋, 康宇, and 许镇义 2022 [pdf]
5. 减震器故障检测方法、装置、设备及存储介质 赵云波, 刘斌琨, 康宇, 曹洋, and 许镇义

项目人员

赵云波 余程凯 刘斌琨 刘朝虎 张天浩 张子辰 张年坤 张杰 朱慧娟 李佳玉 李瑶瑶 桂旺友 梁秀华 董少杰 谢飞 赵昀昇 陈明 陈龙鑫 青凡迪 马树森 齐振宇