Introduction of Human-in-the-Loop in Smart Manufacturing (H-SM)

Duck Bong Kim; Mahdi Sadeqi Bajestani; Ju Yeon Lee; Seung-Jun Shin; Sang Do Noh

doi:10.57062/ijpem-st.2024.00115

Int. J. Precis. Eng. Manuf.-Smart Tech. > Volume 2(2); 2024 > Article

Kim, Bajestani, Lee, Shin, and Noh: Introduction of Human-in-the-Loop in Smart Manufacturing (H-SM)

Short Communication

International Journal of Precision Engineering and Manufacturing-Smart Technology 2024;2(2):209-214.

Published online: July 1, 2024

DOI: https://doi.org/10.57062/ijpem-st.2024.00115

Introduction of Human-in-the-Loop in Smart Manufacturing (H-SM)

Duck Bong Kim¹ · Mahdi Sadeqi Bajestani² · Ju Yeon Lee³ · Seung-Jun Shin⁴ · Sang Do Noh⁵

¹Department of Manufacturing and Engineering Technology, Tennessee Technological University Cookeville, TN 38505, USA

²Department of Mechanical Engineering Tennessee Technological University Cookeville, TN 38505, USA

³Department of Mechanical System Design Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Seoul 01811, Republic of Korea

⁴School of Interdisciplinary Industrial Studies, Hanyang University, 222 Wangsimni-ro, Seongdong-gu, Seoul 04763, Republic of Korea

⁵Department of Systems Management Engineering Sungkyunkwan University, 2066, Seobu-ro, Jangan-gu, Suwon 16419, Republic of Korea

Duck Bong Kim, dkim@tntech.edu

Received June 13, 2024 Revised June 26, 2024 Accepted June 26, 2024

Abstract

This paper introduces the concept of human-in-the-loop in smart manufacturing (H-SM), which aims to involve humans more extensively by utilizing advanced technologies such as artificial intelligence, wearable devices, and immersive technologies. The ultimate objective of H-SM, based on the five pillars of intelligence, integration, interoperability, interaction, and immersion (5I), is to achieve human-centricity, sustainability, and resilience in order to realize Industry 5.0. The paper first discusses and classifies enhanced users and human digital modes (i.e., model, shadow, and twin) based on varying levels of physical and cognitive capabilities and experiences.

Keywords: Human-in-the-loop · Smart manufacturing · Industry 5.0 · Operator 5.0

1 Introduction

One of the main goals in smart manufacturing and Industry 4.0 is to realize autonomous manufacturing, meaning that it aims to minimize human involvement [1]. Due to the lessons learned from unexpected and manufacturing-unfriendly situations, such as COVID-19 [2], the human is being rethought as one of the primary components in manufacturing with the advent of human-centrism in Industry 5.0 [3]. To align the manufacturing paradigm with Industry 5.0 essentials, the concepts of Operator 5.0 and human cyber-physical system (HCPS) have been presented [4,5]. Its key technologies can include artificial intelligence, machine learning, wearable sensors and actuators, extended reality, and immersive technology [6].

Case studies in different manufacturing sectors have been well-discussed in a review paper [7]. Operator 5.0 has been viewed as an approach to enhance users’ physical and cognitive capabilities, mainly for shop operators, by utilizing key technologies. On the other hand, we rethink the human-centrism in terms of the involvement of diverse stakeholders in smart manufacturing, referred to here as ‘human-in-the-loop in smart manufacturing (H-SM)’. We introduce this concept, ultimately aiming for human-centrism, resilience, and sustainability, and classify and describe users with different physical and cognitive capabilities.

In addition, we first propose the concepts of the human digital model, digital shadow, and digital twin. Under the umbrella of Industry 5.0, the implementation of H-SM is expected to provide significant opportunities and advantages by addressing the following three essentials [5].

Human-centrism: Humans are involved diversely with different levels of physical and cognitive capabilities in manufacturing. For example, a disabled person with less experience can participate in manufacturing processes by taking advantage of advanced technologies (e.g., robotic exoskeletons and artificial intelligence).
Resilience: Manufacturing systems are evolved to detect unexpected disruptions and recover performance in a timely manner, even in manufacturing-unfriendly conditions.
Sustainability: Manufacturing processes become sustainable through reducing energy consumption, avoiding the emission of hazardous substances, and increasing resource efficiency. Remote manufacturing can minimize unnecessary trips for face-to-face visits, leading to the reduction of time, energy, and carbon footprint.

2 Human-in-the-loop in Smart Manufacturing

The conceptual architecture of the digital twin with human-in-the-loop in smart manufacturing was proposed, consisting of the cyber-physical system (CPS), avatar-user system (AUS), and collaborative decision-making engine [8]. Based on that, Fig. 1 shows the core components of H-SM, consisting of CPS and avatar-human system (AHS). Occurrences, such as innovations, pandemics, and natural disasters, are shown as inputs, and the three desired outcomes are human-centrism, sustainability, and resilience for Industry 5.0. As well-established in [8], the CPS comprises digital twins and physical entities. The digital twin (DT) is a digital replica of the physical entities and phenomena created through 2-Dimensional(2D)/3-Dimensional(3D) visualization, modeling, and simulation approaches, including but not limited to physics-based, data-driven, and physics-informed data-driven models [7]. In contrast, physical entities can be potential and actual physical assets of the process (e.g., manufacturing systems, sensors, actuators, and network devices).

The AHS comprises the enhanced user (EU) and human digital twin (HDT). In this paradigm, humans can be equipped with key enabling technologies such as wearables and artificial intelligence to endow them with more physical and cognitive capabilities, called the “enhanced user. For example, mixed reality and deep learning-based CPS systems have been proposed to provide more flexibility, customization, and capability for design and manufacturing in manufacturing [9]. Accordingly, the human digital twin is a digital representation of a user or enhanced user [10].

The AHS component will be discussed in the following subsections since they are core aspects of this paper. In addition, intelligence, integration, interoperability, interaction, and immersion, the pillars on which H-SM is established, will be discussed in Section 3.

2.1 Enhanced User

Each user has a different level of physical and cognitive capability and experience. By adopting key technologies for individually customized needs, the user experience can be enhanced to facilitate human work efficiently and effectively in different stages of manufacturing processes. For this, it is necessary to identify the roles, needs, and necessary technologies.

Table 1 shows the classification and descriptions of the enhanced user with respect to the level of physical and cognitive capabilities and experience. In this paper, the main focus is on humans with fewer physical and cognitive capabilities, so three categories are realized: “Disabled”, “Capability-lessening”, and “Normal”. Meanwhile, the level of experience can be classified into “Beginner”, “Intermediate”, and “Expert”. For example, the disabled beginner can actively take advantage of advancing technologies, which is a significant step toward practicing human-centrism and realizing sustainability. In such a way, different users in the other categories can be involved in a wide spectrum of applications.

In H-SM, resilience and sustainability are considered mainstreams. For example, if the facility shuts down and needs to be repaired in manufacturing-unfriendly situations (e.g., pandemic and natural disasters), a service engineer may not be able to visit the facility on-site. In this situation, H-SM can provide and support the necessary technologies and functionalities for remote inspection and maintenance. Additionally, CO₂ emissions and energy consumption can be reduced, contributing to sustainability goals [11].

2.2 Human Digital Twin

Currently, the advancement of technologies enables the digitalization of human users’ data, information, knowledge, wisdom, and purpose (DIKWP) in a 3D digital domain. This is the realization of the human digital twin, which prioritizes human needs in every aspect of manufacturing, spanning from health and safety to self-actualization and personal growth. An HDT is a fit-for-purpose digital counterpart of the human to reflect multi-dimensional information and realize the two-way interaction between the physical and digital worlds. In addition to the personality and characteristics of human beings, the HDT takes the interaction of humans and other resources (e.g., machines and environment) into consideration for depicting humans thoroughly [12].

Features and characteristics of HDT could be similar to other DT research, such as high fidelity, dynamics, identifiability, multi-scalability, multi-physics, real-time reflection, and intelligent prediction and decision-making. In addition, HDTs are expected to include many models that are associated with the properties belonging to interaction, such as models of personality, perception, cognitive performance, and emotion.

Well-aligned with the industry 5.0 paradigm, HDTs are required to be personalized based on complex human attributes including but not limited to physical, physiological, psychosocial, perceptual, emotional, and behavioral characteristics [13,14]. Fig. 2 shows the three different HDT modes: human digital model, shadow, and twin. Here, a human digital object stands for a virtualized human that mirrors its real counterpart in computers. Different HDT modes accompany different HCI (human-computer interface) counterparts, which can be cognitive, voice, or motion capture (MOCAP), giving feedback to different actuators such as wearable robots. The different modes are described as follows:

Human digital model: The DIKWP between the enhanced user and the human digital object is exchanged manually, so any changes in the state of the enhanced user are not reflected in the human digital object directly, and vice versa. For example, offline operational health and safety simulation involves exchanging offline data and information between the enhanced user and human digital object.

Human digital shadow: The DIKWP from the enhanced user is fed forward automatically to the digital object, but the opposite direction still remains manual. As a result, any change in the enhanced user can be seen in its human digital object, but not vice versa. For example, motion capture of the workforce to acquire the posture or physiological data for the user’s condition can be one of the cases where human digital shadows are applicable. This can provide actionable recommendations for upcoming tasks.

Human digital twin: The DIKWP is fed forward and backward automatically between the enhanced user and the human digital object. Therefore, changes in either the physical or digital object directly lead to changes in the other. For example, motion capture sensors, smart glasses, and wearable robots are used to support human digital twins, where physical and digital entities are fully integrated in an online manner. These can be used to propose empirical recommendations for upcoming and current tasks.

Based on these different types of human digital modes, we classify and describe the human digital model, shadow, and twin with respect to different levels of experience, as shown in Table 2. For example, MOCAP and wearables [15] (e.g., smart watches) support the human digital shadow because they can be used to acquire the user’s physical workload and physiological factors for job assignment of a production line [16]. If a user is equipped with smart glasses or wearable robot and the DIKWP is exchanged automatically, it can enable a human digital twin in the near future. In turn, the human digital twin will help achieve operational health and safety for disabled and capability-reducing users.

Similarly, human digital twins can be used in various applications such as task allocation, remote manufacturing, and virtual training. For example, the expert HDT can efficiently and effectively train and share the know-how and techniques to enhance users in a customized manner since a user can enter the extended reality (XR) environment and solely interact with the expert HDT.

3 Pillars of Human-in-the-loop in Smart Manufacturing

There are critical issues for the realization of the human-in-the-loop in smart manufacturing. Considering the current and foreseeable technical challenges in H-SM, the foundations of H-SM are based on five pillars: intelligence, integration, interoperability, interaction, and immersion.

Intelligence: It involves the use of automation, data collection, or any other type of technology to create optimal production conditions. Intelligence in H-SM provides four main features: autonomy, cognitive intelligence, collaborative intelligence, and decentralized intelligence. The combination of autonomy and decentralized intelligence enables objects to negotiate with their environment, observe it together, and make decisions based on their observations and information sharing regarding their status and needs. Cognitive and collaborative intelligence help realize human involvement throughout manufacturing processes by combining human cognition and artificial intelligence for decision-making.
Integration: H-SM has integration issues, including vertical/horizontal and end-to-end integrations, which have originated from traditional smart manufacturing [17,18]. A new integration issue arises with the proposed H-SM, which requires new standards. These new standards will be associated with multidisciplinary domains, including engineering, science, education, and sociology, among others.
Interoperability: Interoperability remains a critical bottleneck for implementing the proposed H-SM framework. One reason is that many manufacturers lack the infrastructure needed to adopt cloud-based standards such as open platform communication unified architecture (OPC-UA) [19], which is a data communication standard for collecting and exchanging industrial data and realizing the concept of “plug-and-play”. In addition, cybersecurity and privacy are ongoing issues that must be linked to interoperability standards because the abundance of personal data and immersive content is prone to cyber threats.
Interaction: Realtime interactions are compulsory to use the immersive experiences in H-SM effectively. For example, the interactions between photorealistic 3D-rendered objects and humans in an immersive environment (e.g., cave automatic virtual environment) must take place in real-time. However, implementing such real-time interaction is challenging because of the huge amount of data that must be collected and processed. In addition, since these interactions can occur in a wireless environment, the low data transmission rate is another issue. Advanced wireless techniques (e.g., 5 and 6 G) should be investigated and developed for H-SM. Also, timely and bidirectional information flows between CPS to AHS and EU to HDT are critical to realizing H-SM.
Immersion: XR and immersive technologies need to be utilized to provide a sense of realism to users in H-SM. These technologies will provide users with a more accurate perception of real manufacturing activities. To realize this, immersive modeling techniques and multimodal interfaces (e.g., visual, audio, and haptic) should be advanced [20]. Also, we should consider the user’s mental/physical health and socio-economic impacts.

4 Conclusion

This paper aims to define the H-SM concept for the wide spectrum of human involvements in manufacturing. This paper also classifies and describes the different levels of physical and cognitive capabilities and experiences in enhanced users and human digital twins. We believe that this concept will provide real industrial impacts in terms of human-centrism, sustainability, and resilience by managing a user’s involvement in the evolutionarily-complex and dynamically-changing manufacturing environment. For example, it can facilitate the involvement of users with less experience and fewer physical-cognitive capabilities in the manufacturing process and the extension of the work period for capability-reducing users. Finally, it can guide future research directions toward developing reference architectures, identifying key-enabling technologies, and implementing case studies from the H-SM perspective.

Declarations

Conflict of interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. This research was supported by Brain Pool program funded by the Ministry of Science and ICT through the National Research Foundation of Korea (RS-2023-00262501).

Fig. 1

Framework for human-in-the-loop in smart manufacturing (H-SM)

Fig. 2

DIKWP flow in human digital model, shadow, and twin

Table 1

Enhanced users with respect to physical and cognitive capabilities and experience

		Level of physical and cognitive capability
		Disabled	Capability-lessening	Normal
		- Users with an impairment causing activity and participation restrictions	- Users whose physical and cognitive capabilities are reducing due to age or illness	- Users who are neither disabled nor capability-lessening
Level of experience	Beginner	- They recently started a particular activity and learning/training is necessary - Their decisions are made from a small set of options
		- The progress is slow - Users should be able to control the pace of interaction	- They require more feedback and more opportunities for closure -The input procedures are required to be concise and brief	- Help is required to be always available - They are likely eager to learn
	Intermediate	- Someone who has more advanced knowledge/skill than a beginner but is not yet an expert - They need consistent structures, sufficient help facilities, and good documentation - They have enough basic skills to operate a system
		- They tend to follow the system rules strictly	- They are less likely to aim to become an expert	- They try to use new techniques to operate the system very effectively - The majority of users are in this class
	Expert	- Someone who has a broad and deep understanding/competence in a particular field - The number of users in this group is always fewer than other groups
		- The level of experience outweighs the disabilities - They can provide trainings and support knowledge transfer	- Systems trust them and ask them for advice - They use intuition based on sufficient experience	- They have a fast response time and require brief feedback - They are considered assets for the organization

Table 2

Classification and descriptions of a HDT with mode types and user experience

		Human digital mode
		Human digital model	Human digital shadow	Human digital twin
		-Non-real-time sensors and actuators -The physical and digital entities are not integrated	- Realtime sensors and non-real-time actuators - The HCI is limited to sensors - The physical and digital entities are one-way integrated	- Realtime sensors and actuators - The HCI includes sensors and actuators - The physical and digital entities are fully integrated
Level of experience/cognitive capability	Beginner	- Has limited knowledge/skill of a specific domain - Requires training and feedback
	Beginner	- Data is exchanged manually and offline	- One-way automatic data exchange (manually from EU to HDS and automatically in the opposite direction)	- Complete automatic data and information exchange
	Intermediate	- Has more knowledge/skill than a beginner but not yet an expert - Can train beginners
	Intermediate	- DIK is exchanged manually	- One-way automatic DIK exchange	- Complete automatic DIK exchange
	Expert	- Has broad and deep understanding/competence in a particular field - Needs few feedback and instructions
	Expert	- DIKWP is exchanged manually	- One-way automatic DIKWP exchange	- Complete automatic DIKWP exchange - The concept of CraftsAvatar [8] (Adapted from Ref. 8 on the basis of OA)

References

1. Kusiak, A. (2023). Smart manufacturing. Springer Handbook of Automation, In S. Y.. Nof (Ed.), Springer: 973–985.

2. Bragazzi, N. L., (2020). Digital technologies-enabled smart manufacturing and industry 4.0 in the post-COVID-19 era: lessons learnt from a pandemic. International Journal of Environmental Research and Public Health, 17(13), 4785.

3. Panagou, S., Neumann, W. P. & Fruggiero, F. (2024). A scoping review of human robot interaction research towards industry 5.0 human-centric workplaces. International Journal of Production Research, 62(3), 974–990.

4. Wang, B., Zheng, P., Yin, Y., Shih, A. & Wang, L. (2022). Toward human-centric smart manufacturing: A human-cyber-physical systems (HCPS) perspective. Journal of Manufacturing Systems, 63, 471–490.

5. Gladysz, B., Tran, T.-a., Romero, D., van Erp, T., Abonyi, J. & Ruppert, T. (2023). Current development on the Operator 4.0 and transition towards the Operator 5.0: A systematic literature review in light of industry 5.0. Journal of Manufacturing Systems, 70, 160–185.

6. de Giorgio, A., Monetti, F. M., Maffei, A., Romero, M. & Wang, L. (2023). Adopting extended reality? A systematic review of manufacturing training and teaching applications. Journal of Manufacturing Systems, 71, 645–663.

7. Haricha, K., Khiat, A., Issaoui, Y., Bahnasse, A. & Ouajji, H. (2023). Recent technological progress to empower smart manufacturing: Review and potential guidelines. IEEE Access, 11, 77929–77951.

8. Kim, D. B., Bajestani, M. S.Shao, G.Jones, A. & Noh, S. D. (2023). Conceptual architecture of digital twin with human-in-the-loop-based smart manufacturing. In: Proceedings of the ASME International Mechanical Engineering Congress and Exposition (IMECE); pp V003T03A076..

9. Malik, A., Lhachemi, H. & Shorten, R. (2023). A cyber-physical system to design 3D models using mixed reality technologies and deep learning for additive manufacturing. PLoS One, 18(7), doi.org/10.1371/journal.pone.0289207.

10. Wang, B., Zhou, H., Li, X., Yang, G., Zheng, P., Song, C., Yuan, Y., Wuest, T., Yang, H. & Wang, L. (2024). Human digital twin in the context of industry 5.0. Robotics and Computer-Integrated Manufacturing, 85, 102626.

11. Zhang, C., Wang, Z., Zhou, G., Chang, F., Ma, D., Jing, Y., Cheng, W., Ding, K. & Zhao, D. (2023). Towards new-generation human-centric smart manufacturing in industry 5.0: A systematic review. Advanced Engineering Informatics, 57, 102121..

12. Shengli, W., (2021). Is human digital twin possible? Computer Methods and Programs in Biomedicine Update, 1, 100014.

13. Miller, M. E., & Spatz, E. (2022). A unified view of a human digital twin. Human-Intelligent Systems Integration, 4(1), 23–33.

14. Gräßler, I.Maier, G. W.Steffen, E. & Roesmann, D. (2023). The digital twin of humans: an interdisciplinary concept of digital working environments in industry 4.0, Springer Nature.

15. Song, Y., (2023). Human digital twin, the development and impact on design. Journal of Computing and Information Science in Engineering, 23(6), 060819.

16. Kim, G.-Y., Yun, J., Lee, C., Lim, J., Kim, Y. & Noh, S. D. (2024). Data-driven analysis and human-centric assignment for manual assembly production lines. Computers & Industrial Engineering, 188, 109896.

17. Sheth, A., & Kusiak, A. (2022). Resiliency of smart manufacturing enterprises via information integration. Journal of Industrial Information Integration, 28, 100370.

18. Zhuang, C., Miao, T., Liu, J. & Xiong, H. (2021). The connotation of digital twin, and the construction and application method of shop-floor digital twin. Robotics and Computer-Integrated Manufacturing, 68, 102075.

19. Shin, S.-J., (2020). An OPC UA-compliant interface of data analytics models for interoperable manufacturing intelligence. IEEE Transactions on Industrial Informatics, 17(5), 3588–3598.

20. Lee, Y.-G., Park, H., Woo, W., Ryu, J., Kim, H. K., Baik, S. W., Ko, K. H., Choi, H. K., Hwang, S.-U., Kim, D. B., Kim, H. & Lee, K. H. (2010). Immersive modeling system (IMMS) for personal electronic products using a multi-modal interface. Computer-Aided Design, 42(5), 387–401.

Biography

Duck Bong Kim received his MS and PhD degrees at Gwangju Institute of Science and Technology (GIST), Republic of Korea, in 2006 and 2011. He worked as a guest researcher at National Institute of Standards and Technology (NIST) from 2011 to 2016. He has been a professor in the Department of Manufacturing and Engineering Technology at Tennessee Technological University since 2016. His research interests include metal additive manufacturing, smart manufacturing, and data analytics.

Biography

Mahdi Sadeqi Bajestani is a Ph.D. candidate in the Department of Mechanical Engineering at Tennessee Technological University, USA. He obtained his M. Tech. degree in Mechanical Engineering (Manufacturing and Production) from Ferdowsi University of Mashhad, Iran. His research interest is digital twin, cyber-physical systems, and smart manufacturing.

Biography

Ju Yeon Lee is an Assistant Professor in the Department of Mechanical System Design Engineering at Seoul National University of Science and Technology. She received her Ph.D. degree from the Sungkyunkwan University (South Korea) in 2011 and received her bachelor’s degree from the same university in 2005. She worked as a Principal Researcher at Korea Institute of Industrial Technology and as a guest researcher at National Institute of Standards and Technology. Her research interests include modeling & simulation, digital twin, manufacturing data analytics, sustainable manufacturing, and industry data standards.

Biography

Seung-Jun Shin is an Associate Professor in School of Interdisciplinary Industrial Studies at Hanyang University. He received his Ph.D. degree (2010) in Department of Industrial and Management Engineering at POSTECH. He worked as a senior engineer in Samsung Electronics, a senior consultant in Samsung SDS, a guest researcher at National Institute of Standards and Technology, and an assistant professor in Pukyong National University. His research interests include cyber-physical production systems, manufacturing data analytics, industrial standards, and environmentally-conscious manufacturing.

Biography

Sang Do Noh received his Ph.D. in mechanical design and production engineering from Seoul National University, Republic of Korea. He currently works as professor in Department of Industrial Engineering at Sungkyunkwan University, Republic of Korea. His major research areas are CAD/CAM/PLM, modeling & simulation of manufacturing system, smart manufacturing, smart factory, cyber-physical system, and digital twin.