1 Introduction
In the industrial field, prognostics & health management (PHM) integrates research on defect mechanisms with lifecycle management of systems, enabling early defect diagnosis and lifespan prediction through comprehensive monitoring of manufacturing processes [1]. Particularly with the advancement of the industrial internet of things (IIoT), the increase in the types and amounts of data collected in industrial fields is contributing to the establishment of stable mass production systems and increased productivity through the application of data-driven approaches with artificial intelligence (AI) [2,3].
Despite the widespread adoption of PHM research across various industrial fields, several limitations constrain its practical application. One of the primary challenges is the issue of data collection necessary for developing PHM systems [4]. These systems are mainly data-driven and exhibit independence, being applicable only to the specific situations where the data was collected. Industrial fields are characterized by variable situations, such as equipment diversity, product design changes, and material differences.
Consequently, extensive data collection is required to tailor independent PHM systems for each specific situation [5]. In other words, PHM systems must achieve high performance across diverse variable situations for rapid model construction, even with limited amounts of training data. Therefore, to ensure the proper application of PHM models in diverse situations, a model capable of being aware of all variable situations occurring in the manufacturing process must be fundamentally designed [6]. Additionally, the pre-established situation awareness model must achieve high performance even with limited amounts of training data, as it will be retrained with data from various situations.
In this context, one of the representative manufacturing fields where PHM systems are essential is the automotive industry, particularly in the robotic spot-welding (RSW) process. RSW, an automated manufacturing process that uses industrial robots to join two or more sheets together based on the principle of Joule heating, is critical in the production of car bodies [7,8]. Here, precise and consistent weld quality is paramount to ensure the structural integrity and safety of the vehicles. Given the various situations such as material thickness, material type, and welding parameters, it is imperative to implement a robust PHM system to monitor and maintain weld quality. Kim et al. developed a system for predicting the diameter of spot-welded buttons for each welding material situation, crucial for welding quality, using an artificial neural network (ANN) algorithm. This system achieved high accuracy with a root mean square error (RMSE) of 0.06 mm between the actual and predicted button diameters [9]. Xia et al. created a novel online system for monitoring welded spot penetration in RSW process using electrode displacement signals [10]. In this context, high-performance models for diagnosing various quality defects (expulsion, undersized-weld, etc.) occurring in machine learning-based welding processes are being developed through data-driven approaches, utilizing collected time-series sensor data in a single material type or thickness situation [11–15].
However, as previously mentioned, variable situations arise in automotive industry for RSW, so it is essential to have a model aware of the specific situation before employing well-trained diagnostic models to evaluate quality for each situation. Park et al. developed predictive manufacturing situation awareness (MSAW) based on Multi-Entity Bayesian Networks (MEBN) to enhance industrial competitiveness by supporting manufacturing process. This model integrates various data sources to estimate current situations and predict future scenarios in manufacturing process [16]. Peng and Xu presented a novel situation awareness model framework for float glass manufacturing using denoising long short-term memory (DLSTM) autoencoders to extract neighborhood features and multi-scale isometric convolution networks (MICN) to capture local and global features over different time scales [17]. The proposed model trained by 9,800 training data demonstrates high prediction accuracy compared to Temporal convolutional network (TCN) and multi-scale temporal convolution network (MTCN). Xu et al. also proposed a novel unified adaptive deep classification framework for real-time situation awareness in industrial environments [18]. The proposed framework and model effectively detect unknown patterns in large-scale industrial dataset. These studies proposed high-performance situation awareness models through extensive industrial fields data. Using these situation awareness models, a framework can be established to diagnose processes and equipment with appropriate PHM models tailored to specific situations. However, there are limitations due to the need to collect and train on large amounts of new situation data. Therefore, methodology is needed to construct high-performance situation awareness model that can operate effectively even trained with relatively small amounts of data.
In this study, a robust situation awareness model was developed to recognize various situations before applying appropriate diagnosis model by data visualization technique and convolutional neural networks (CNN) with relatively small amount data. First, sensor data collected from the RSW system under two variable situations (changes in thickness and material type). The data were transformed into pattern image to visualize energy differences corresponding to variable situations. The physical information of the image for each situation was analyzed, and a CNN was applied to build model. Finally, to evaluate the performance of sensor data visualization, the CNN model based on visualized data compared with a CNN model trained on raw sensor data.
2 Data Collection and Analysis
2.1 Experiments and Data Collection
To facilitate experiments for variable situations, an experimental RSW testbed was established. As depicted in Fig. 1, this automated RSW testbed utilized an industrial 6-axis robot (HS180, Hyundai Robotics) and an AC servo-motor C-type welding gun (SRTC-Y003, Obara), which are key components in car body assembly automation systems. A Rogowski coil (CWT-M-60R, PEM) and a differential voltage probe (DP-30HS, PINTEK) were attached to the welding gun to gather current and voltage signals, crucial for obtaining electric energy information. Additionally, an accelerometer (353B03, PCB Piezotronics) and a laser displacement sensor (LK-G157, Keyence) were mounted along the electrode’s movement direction to track geometrical changes in spot-welds, correlating with electrode force and the physics of nugget formation.
To determine appropriate process parameters for each variable situations in RSW system, major process parameters that significantly impacting welding quality (welding current and welding force) are selected as searching process parameter, along with a destructive test based on the measurement of spot-weld nugget width. Generally, welding current is a process parameter that determines the amount of resistance heat generated between metals, while welding force is a closely related parameter influencing metal contact and the proper formation of the weld nugget [19,20]. The suitable process parameters were selected based on a criterion of nugget width exceeding 4√t (t: thickness of sheet material), in accordance with standards set by the American Welding Society (AWS) for resistance spot-welding [21].
For precise weld quality assessment, the cross-sectional measurement of the weld spot nugget width was employed. The preparation of the welded spot cross sections involved sequential processes, including wire electrical discharge machining (EDM) cutting, followed by surface polishing using #1000, #1500, and #2000 grit in sequence, and etching of the surface for 150 seconds. The cross-sectional weld nuggets were then measured using an optical microscope (BX53M, Olympus) and a laser scanning confocal microscope (VK-X1000, Keyence). Based on the quality assessment, appropriate process parameters were selected and listed in Table 1.
For the purpose of collecting sensor data under variable situations, a data acquisition (DAQ) system was constructed. Each data sample in this system comprised the four types of signals mentioned above. The hardware configuration of the DAQ system included all sensor types, an analog-to-digital (A/D) signal converter compatible with BNC-type sensor cables (NI 9232, National Instruments), and a workstation linked to the A/D converter. The DAQ software, a GUI-based program, was developed using the Python programming language and incorporated various data science libraries such as NumPy, Pandas, and Matplotlib, as well as PyQt, a prominent GUI development tool for Python. This DAQ system, operating at a sampling frequency of 12,800 Hz, facilitated the efficient collection of sensor data.
The variable situations were divided into two phases: Phase 1 involved two levels of thickness variation (GI 0.6 t and 1.0 t), and Phase 2 entailed two levels of material type variation (MS and GA) as shown in Fig. 2(a). The standard data for GI 0.8 t situation, comprised 600 samples for normal condition. For all situational changes, 240 samples for normal condition were collected, as depicted in Fig. 2(b).
2.2 Sensor Data Analysis
Prior to the sensor data visualization step, a profile of the collected raw sensor signals was analyzed. In Fig. 3, the raw signals of four sensors across varying thickness situations can be observed. The welding current followed the AC waveform applied by the system, and the voltage data exhibited variations in the waveform due to resistance changes occurring during the spot-welding process. For the acceleration signal, an increasing trend in vibration was noted at the beginning of the spot-welding process due to the rising welding force. Regarding the displacement changes between electrode tips, there was an increase in indentation depth-the impression marks on the material where the electrode tips meet due to the spot-welding process-during the formation of the spot-weld nugget. This increase is attributed to the liquefaction of the metal and the applied welding force, followed by a stable phase maintaining the shape of the nugget, and a decrease in displacement as the electrode tips move apart upon completion of the spot-welding process.
As shown in Fig. 3, these magnitude or trend in the raw sensor signals occur consistently regardless of thickness changes and exhibit similar trends irrespective of material type changes. Recent high-performance AI can be aware of energy differences in raw signals that are not discernible by humans and classify what variable situation occurred in manufacturing process. However, this requires a large amount of data collection in each variable situation. Therefore, it is necessary to apply a method that can effectively represent these energy differences. Such a method would allow the construction of effective classification models with relatively small amounts of data.
3 Data Visualization and Model Construction
3.1 Sensor Data Visualization on Variable Situations
To be aware of changes according to different situations, the collected sensor data were applied using a FEPI visualization technique. This sensor data visualization method utilizes physical features extracted from the raw signals to visualize energy differences between standard state and comparison state. In the RSW process, energy differences also arise according to variable situations, and these changes can be effectively perceived through the processed data using the FEPI method [22].
To facilitate sensor data visualization by FEPI method, it is essential to understand the energy behavior in RSW system. RSW fundamentally involves two main types of energy affecting spot-weld quality: electric and kinetic. Initially, the weld nugget between two specimens is formed and grows according to the principle of Joule heating. Thus, the supply of electric resistance energy predominantly dictates the welding quality [23]. Additionally, kinetic energy, particularly in the direction of the electrode force, offers crucial insights into nugget formation [24].
Dynamic Resistance (DR) curve represents the ratio of voltage change to current change and is a critical index in describing the nugget formation process [25]. The typical DR curve, two inflection points known as the α and β peaks are critical points associated with the formation of nugget. To effectively observe the energy behavior in the RSW process, it is crucial to separately analyze the energy dynamics in the initial and subsequent sections. Accordingly, four distinct types of energy patterns were identified for visualization in the FEPI data: electric and kinetic energy, both before and after the β peak.
The processing of raw sensor data into FEPI data for different variable situations based on the GI 0.8 t standard situation is visualized as shown in Fig. 4. All variable situations display results where the images differ from the standard GI 0.8 t data, indicating that changes in material thickness and type can be effectively recognized through sensor data visualization method, FEPI data.
In the GI 0.6 t situation, compared to the standard situation GI 0.8 t, a difference in energy was observed in the kinetic energy area before the formation of the welding nugget. While data exhibiting stable magnitude and variability like the standard situation was outputted, the GI 0.6 t situation resulted in a lower resolution image as shown in Fig. 4.
As shown in Fig. 5, for FEPI data, the two features selected for each axis form the top two input membership functions (MFs) based on the data distribution of the standard situation. The corresponding feature values of each situation create the output MFs visible in the middle, and the lower FEPI data in each area are outputted through the MFs of each magnitude and variability axis. Typically, when the energy in a new situation is either insufficient or excessive compared to the standard situation, the intersection points of the two lines occur at the sides rather than in the center, as can be seen in Fig. 5(a). However, in the case of GI 0.6 t, the energy output was formed lower while being stable state, compared to the average acceleration and standard deviation features, and the slope and difference features of displacement before nugget formation in GI 0.8 t. As a result, the image was converted into a lower resolution image, as shown in Fig. 5(b).
This is analyzed as being due to the reduction in thickness compared to the standard situation, resulting in lower welding force, in turn, leads to a decrease in the mechanical stress applied to the material during the welding process.
In the GI 1.0 t situation, differences compared to the standard situation occurred in three areas. Firstly, the kinetic energy after the nugget generation area showed a low magnitude. This is attributed to the less vibration occurring in the situation of higher rigidity under the same welding force for both 0.8 and 1.0 t. Therefore, it is analyzed that the low magnitude in the 1.0 t, as shown in Fig. 6, occurred due to the thicker material exhibiting reduced vibration.
Additionally, different visualized energy areas occurred in two regions of electric energy, as shown in Fig. 7. This result is analyzed as being caused by the significantly higher welding current compared to the standard situation.
Lastly, in the MS 0.8 t situation, a low magnitude was observed in the kinetic energy after the nugget generation area due to the significantly lower welding force. In the GA 0.8 t situation, an excessive magnitude pattern was observed in the kinetic energy area before nugget generation. This phenomenon was caused by excessively high feature values in the displacement and acceleration sensors. The weak spatter-like occurrence in GA steel, which happens as the surface’s thin iron and zinc alloy layer melts under welding current, is reflected as spike signals in the data. This energy difference, attributed to the characteristics of GA steel, is analyzed to be the cause of these observed spikes.
3.2 Situation Awareness Model Construction
Previously, the visualization technique for sensor data enabled the collection of FEPI data according to variable situations, and it was analyzed that FEPI data effectively determine situational changes based on physical behavior when compared with raw sensor data (time-series data). However, in situations like GI 0.6 t, where the image shape is the same yet limited by resolution differences, human discernment of variable situations is challenging. For assessing the validity of the sensor data visualization technique, the situation awareness model was constructed using CNN, as can be seen in Fig. 8.
To evaluate the performance of the FEPI data-based situation awareness model, models for changes in thickness situations, classifying three situations: GI 0.6 t, GI 0.8 t, and GI 1.0 t, as well as for material type changes, classifying MS 0.8 t, GI 0.8 t, and GA 0.8 t as three distinct situations, were developed using both FEPI and time-series data. The sensor raw signals are time-series data in a vector format, so a 1D-CNN was used to build the model. The 1D-CNN is tailored to leverage the advantages of CNN architectures on vector-shaped data. It is especially suitable for handling time-series data, where spatial relationships across a single dimension are significant [26]. Unlike 2D-CNN, which are commonly used for image data involving two spatial dimensions, 1D-CNN focuses on detecting patterns in sequences of data, making them ideal for sensor data analysis in manufacturing applications. The optimized hyperparameters for each model determined by grid search are shown in Table 2. To assess the validity of the data in situation awareness, the performance degradation trend was evaluated using 60 test data samples while reducing the training data from 180 to 20.
4 Model Evaluation
4.1 Material Thickness Situation Awareness Model Comparison
Generally, both types of models showed high situation awareness accuracy with sufficient training data, ranging from 180 to 100. As analyzed in sensor data visualization, FEPI data visualizes physical behavior according to situational changes, enabling the CNN model to effectively determine differences in the areas where situational changes occur, resulting in high performance. For time-series data, differences in process parameters and physical behavior changes due to thickness variable situations allow the CNN model to discern thickness changes with sufficient training data, similarly yielding high performance.
However, beyond 90 training data samples, the performance of the time-series data-based model significantly declined. This is analyzed to be due to the similarity in the trends of the raw sensor signals despite differences in magnitude according to situational changes, as observed in Fig. 3. Therefore, with fewer training data samples, the performance for the intermediate value, GI 0.8 t, was lower, as can be seen in Fig. 9(b).
The FEPI data-based material thickness situation awareness model maintained high accuracy even with 20 training data. The confusion matrix in Fig. 9(a) showed relatively lower performance in distinguishing between GI 0.8 t and 0.6 t situations, likely due to the presence of data with similar shapes but subtle intensity differences, as shown in Fig. 5, and the reduced training data. Although the performance of both data-based models declined with fewer training data, the FEPI data-based model maintained relatively high performance. This indicates that material thickness situation awareness based on FEPI data is feasible even with a limited amount of training data.
4.2 Material Type Situation Awareness Model Comparison
Similarly, the performance of the material type situation awareness models, based on both FEPI and time-series data, was evaluated. Like the thickness situation awareness models, the FEPI data-based model showed robust classification performance despite a reduction in training data. This demonstrates the effectiveness of FEPI data in discerning material types, as well as thickness awareness. Compared to the thickness awareness model, the material type awareness model not only had a high classification accuracy but also maintained high accuracy with a minimum of 20 training data. This is due to the different physical behaviors visualized in the kinetic energy areas according to the material type, as observed in Fig. 4. As shown in Fig. 10(a), a slight model performance decrease in classifying between GI 0.8 t and MS 0.8 t situations. This phenomenon is analyzed to occur due to the decrease in vibration energy in the MS 0.8 t FEPI data, as seen in Fig. 6, but also because there is data located in the center, similar to the GI 0.8 t pattern image.
For time-series data, while high classification accuracy was observed under conditions of sufficient training data, a significant performance decline was noted with reduced training data from 70 training data. The confusion matrix in Fig. 10(b) for the 20 training data showed reduced performance in distinguishing between GI 0.8 t and GA 0.8 t situations. This is analyzed to be due to the spike-shaped vibration signals caused by the zinc and iron alloy layer in GA steel being similar to the spike signals caused by spatter in the GI 0.8 t situation, and as seen in Table 1, this phenomenon is attributed to the similar process parameters. This similarity led to difficulties in differentiating between the two situations when training data was limited in time-series data-based CNN model.
5 Conclusions
In this study, situation awareness models were developed in material thickness and type variable situations using sensor data visualization technique and CNN through small amount of data. The contributions of this study are as follows:
Sensor data collected for discerning changes according to variable situations (Material thickness and type) in RSW process were applied using the effective energy difference visualization technique, FEPI method. This image processing method utilizes physical features extracted from the raw signals to visualize energy differences between the standard situation and the one being compared. As energy differences occur in the RSW process due to variable situations, FEPI data were processed for four scenarios, and the resulting pattern images in areas where energy differences occurred were analyzed based on raw sensor signals and domain knowledge to understand the physical behaviors causing them.
Additionally, to evaluate the performance of FEPI data, a comparison was made with models of the same structure trained on time-series data, confirming the high-performance classification capability of the FEPI data-based situation awareness model. This FEPI data-based model can accurately discern changes in thickness (GI 0.6 t, 0.8 t, 1.0 t) and material types (MS 0.8 t, GI 0.8 t, GA 0.8 t) occurring in the RSW process. Comparing the performance of the situation awareness models with reduced data, the FEPI-CNN showed a maximum accuracy drop of 6.67% and maintained high average accuracies of 99.08% and 97.71% across two variable situations when the training data was reduced from 180 to 20. In contrast, the Time-CNN experienced a maximum accuracy drop of 31.11% and had relatively lower average accuracies of 88.40% and 87.55%. This demonstrates that the FEPI-based sensor data visualization technique is effective in constructing robust situation awareness models, even with reduced training data. In this context, when new situations regarding material thickness or type need to be learned, it is predicted that the model’s performance can be extended to recognize more situations by training the initially developed situation awareness model with a relatively small amount of collected data.