CTNF 18/302,036 CTNF 99176 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 4, 11, and 18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 4 recites “the at least one equivalent component” (lines 2-3). There is insufficient antecedent basis for the limitations in this claim, and claims 1 and 3. For examination purposes claim 4 will be read as “at least one equivalent component of the second machine”. Claim 11 recites “the at least one equivalent component” (line 2). There is insufficient antecedent basis for the limitations in this claim, and claims 8 and 10. For examination purposes claim 11 will be read as “at least one equivalent component of the second machine”. Claim 18 recites “the at least one equivalent component” (lines 2-3). There is insufficient antecedent basis for the limitations in this claim, and claims 15 and 17. For examination purposes claim 18 will be read as “at least one equivalent component of the second machine”. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. (Visualisation of the Digital Twin data in manufacturing by using Augmented Reality, 52nd CIRP Conference on Manufacturing Systems, Procedia CIRP 81 (2019) 898–903, hereinafter “Zhu”) . Regarding claim 1 , Zhu discloses A computer-implemented method comprising: (page 898 / col. right / para. 2 – page 899 / col. left / para. 1, “Motivated by representing the Digital Twin data at a new level, this paper proposes a method to visualise the Digital Twin data by using AR technology in a real manufacturing environment”). identifying, by an augmented reality (AR) device, that a new action is to be performed on a first machine, the first machine being viewable by a user using the AR device, wherein the new action is configured to be performed by the user using at least one component of the first machine; (page 900, col. right, para. 1, “The operator can visualise and monitor the process of machines through HMDs. Furthermore, at the same time, they can also control the machines on the physical panels”; page 902, col. left, para. 4, “Figure 2 shows the architecture and workflow of the AR application. The physical part consists of the EMCO milling machine, cutting tools, work piece, and other DAQs, such as camera, RFID, and dynamometer”). Note that: (1) an HMD is an AR device for the operator (the user) to visualize the process as the action of a machine as a first machine; (2) the process (action) as a new action is controlled or performed by the operator (the user) by the physical panels as at least one component of the first machine; and (3) in the AR digital two application, the physical part corresponds to the EMCO milling machine as the first machine. obtaining, by the AR device, a digital representation of a second machine, the digital representation being associated with an equivalent action to the new action, wherein a first outcome of the new action and a second outcome of the equivalent action are determined to be equivalent; and (page 899, col. left, para. 2, “‘Digital twin is an integrated multi-physics, multi-scale, probabilistic simulation of a product or system that uses the best available physical models, sensor updates, etc., to mirror the life of its corresponding twin.’ Digital Twin consists of three components: a physical part, a virtual part, and the connections between the physical and virtual part … both physical and virtual parts exist in Digital Twin, the virtual part can keep ultra-high synchronisation and fidelity to reflect the physical part”; page 902 / col. left / para. 4 – page 902 / col. right / para. 1, “The virtual part consists of all the 3D models generated by CAD software, all the 3D models need to be converted to object file format and transferred into the Unity 3D game engine, which is used to build the AR application in this case. In addition, all the collected data from CNC machine and DAQ devices belong to virtual part. It includes axis position data, machine tool status, cutting tool data, historical tool path data, cutting force data, and safety volume data”; page 902, col. right, para. 3, “the control process can also be achieved through HoloLens. Gestures and voice commands can be used by users to control the virtual part of Digital Twin, and then the command will affect the counterpart in the physical world. The control process is the critical process that allows the operator to easily and intuitively interact and manage the Digital Twin data””). Note that: (1) a digital twin can be built by including the first machine as a physical part, and a virtual part, and their connections; (2) the virtual part is the digital representation (3D models and all the collected data from CNC machine) of a virtual machine as a second machine corresponding to the first machine in terms of 3D structures and functionalities; (3) the second machine (virtual machine corresponding to the physical machine) in the virtual part can be controlled, and the counterpart (the first machine) in the physical world can be affect accordingly to perform a process as an equivalent action associated with the new action; and (4) since the digital twins are implemented in the machine control application to perform equivalent processes (actions) with the same inputs by the physical panels and the corresponding virtual panels, it obvious to one having ordinary skill in the art that the outcome of the new action and a second outcome (response and of the equivalent action should be determined to be equivalent as the same functionalities are required for digital twins. displaying, by the AR device, the equivalent action of the digital representation of the second machine overlaid on the first machine such that the equivalent action of the digital representation is viewable to the user. (page 901, col. left, paras. 3-5, “Figure 1 shows the framework of visualising the Digital Twin data by using AR. Physical part represents the physical objects in the real world, it can be a part, a product, a machine … However, apart from this, all the real-time data collected from sensors, data acquisition devices, machines, and inputs from humans belong to the virtual part. Moreover, to achieve a comprehensive visualisation of a Digital Twin, all the historical data stored in the server are important to be treated as the virtual part … In the virtual part, a cube is designed to align with the binary marker in the physical part. When the cube is perfectly aligned with the marker through the user’s view, the calibration process is accurately done, the virtual world now is perfectly aligning with the physical world, which means all the 3D models in virtual part can be overlaid onto their counterparts in the physical part”; page 902, col. right, Fig. 3: “Captures of the HoloLens view taken from different angles and distances during machining simulation”; page 903, col. left, para. 1, “Figure 3 shows some views from HoloLens during the machine process. It can be seen from the captures that AR through HoloLens can provide an intuitive and comprehensive view of the Digital Twin data under manufacturing environment”). Note that: (1) through digital twins, the physical machine (the first machine) and the virtual machine (the second machine) and the corresponding data (all the real-time data collected from sensors, data acquisition devices, machines, and inputs from humans) can be overlaid to each other; and (2) the corresponding equivalent action of the digital representation from the second machine is viewable to the user with the AR device’s display while the new action from the first machine is viewable at the same time. Before the effective filing date of the claimed invention, it would have been obvious to learn and use the teachings on the digital twins for part, a product, a machine or even an entire factory, AR visualization of Digital Twin, AR applications for machine tools and CNCs, and AR devices, as taught by Zhu. Although the procedure that a first outcome of the new action and a second outcome of the equivalent action are determined to be equivalent are not explicitly described by Zhu using the application’s terms regarding the equivalence, it is obvious for one having ordinary skills in the art to understand performing the corresponding operations, calculations, and determinations. The motivation would have been “The developed application allows the operator to monitor and control the machine tool at the same time, but also enables to interact and manage the Digital Twin data simultaneously, which provides an intuitive and consistent human machine interface to improve the efficiency during the machining process.” (Zhu, page 898, Abstract). The suggestion for doing so would allow to determine a first outcome of the new action and a second outcome of the equivalent action to be equivalent. Therefore, it would have been obvious to use the teachings by Zhu. Regarding claim 2 , Zhu discloses The computer-implemented method of claim 1, wherein at least one equivalent component of the second machine is mapped to the at least one component of the first machine. (page 901, col. left, para. 5, “In the virtual part, a cube is designed to align with the binary marker in the physical part. When the cube is perfectly aligned with the marker through the user’s view, the calibration process is accurately done, the virtual world now is perfectly aligning with the physical world, which means all the 3D models in virtual part can be overlaid onto their counterparts in the physical part”). Note that: after the calibration by aligning a cube perfectly with the binary marker in the physical part through the user’s view, all components (3D models) of the second machine can be aligned or mapped with the corresponding components of the first machine. Regarding claim 3 , Zhu discloses The computer-implemented method of claim 1, wherein the new action comprises a first plurality of procedures performed on the first machine and the equivalent action comprises a second plurality of procedures performed on the second machine, at least one of the second plurality of procedures being different from the first plurality of procedures. (page 901, col. right, para. 3, “The control process allows the user to interact with both physical part and virtual part of Digital Twin. After getting intuitive and comprehensive visualised data from the augmented process, users can take advantage of this useful information to make decisions and control the physical part directly through the AR device. Through users’ commands and inputs into the control process, which will establish a closed-loop control to directly improve and update the Digital Twin data, and then the modified and improved the Digital Twin data will keep displaying onto the AR device”; page 899, col. left, para. 2, “Self-evolution: Digital Twin can collect and update data in real time, the virtual part can continually self-improve by comparing virtual part with a physical part in parallel”). Note that: (1) after getting intuitive and comprehensive visualised data from the augmented process (overlaying the digital presentation of the second machine on the display of the first machine), the user can make decisions and control the physical part (the first machine), resulting in one or more different sub-processes, sub-actions, procedures of the new action as a first plurality of procedures performed on the first machine; (2) through users’ commands and inputs into the new action, the different sub-processes, sub-actions, procedures can improve and update the Digital Twin data as a second plurality of procedures performed on the second machine; and (3) the different procedure(s) between the first plurality of procedures and the second plurality of procedures can optimize or improve the design and operation performance of the first machine and the second machine of the digital twin through Self-evolution. Regarding claim 4 , Zhu discloses The computer-implemented method of claim 3, wherein performing the second plurality of procedures of the equivalent action using the at least one equivalent component is displayed along with a view of the at least one component. (page 901, col. left, para. 5, “In the virtual part, a cube is designed to align with the binary marker in the physical part. When the cube is perfectly aligned with the marker through the user’s view, the calibration process is accurately done, the virtual world now is perfectly aligning with the physical world, which means all the 3D models in virtual part can be overlaid onto their counterparts in the physical part”). Note that: (1) after the calibration by aligning a cube perfectly with the binary marker in the physical part through the user’s view, all components (3D models) of the second machine can be aligned or mapped with the corresponding components of the first machine; and (2) it is obvious to one having ordinary skill in the art that the aligned and overlaid components of the digital representation used in performing the second plurality of procedures by the second machine are viewable along with the corresponding components of the first machine. Regarding claim 6 , Zhu discloses The computer-implemented method of claim 1, wherein: the first machine is physically viewable through the AR device; (page 902, col. left, para. 3, “This AR application is set up on an EMCO 3-axis milling machine … All the data transmission and connection between CNC machine, host server and HoloLens are based on TCP/IP. Furthermore, the server on the host PC can provide data to HoloLens through the Wi-Fi connection”). Note that: (1) Microsoft’s HoloLens is used in the AR application; and (2) it is known that HoloLens sees through by using a see-through, holographic lens system (waveguides) that layer digital images on top of the real world, rather than using cameras to display a video feed of the surroundings, which means that the first machine is physically viewable through the AR device. the second machine is virtually displayed by the AR device in a same field-of-view as the at least one component of the first machine. (page 901, col. left, para. 5, “In the virtual part, a cube is designed to align with the binary marker in the physical part. When the cube is perfectly aligned with the marker through the user’s view, the calibration process is accurately done, the virtual world now is perfectly aligning with the physical world, which means all the 3D models in virtual part can be overlaid onto their counterparts in the physical part”). Note that: (1) after the calibration by aligning a cube perfectly with the binary marker in the physical part through the user’s view, all components (3D models) of the second machine can be aligned or mapped with the corresponding components of the first machine; and (2) the Hololens as the AR device can display all virtual components of the second machine overlaid with that of the first machine in the field of view . 07-21-aia AIA Claim s 8-11, 13, 15-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu in view of Peng et al. (US 20200007914 A1, hereinafter “Peng”) . Claim 8 reciting “A system of an augmented reality (AR) device comprising: a memory having computer readable instructions; and a computer for executing the computer readable instructions, the computer readable instructions controlling the computer to perform operations comprising:” is corresponding to the method of claim 1. Zhu discloses an augmented reality (AR) device (page 900, col. right, para. 1, “The operator can visualise and monitor the process of machines through HMDs. Furthermore, at the same time, they can also control the machines on the physical panels”). Therefore, claim 8 is rejected for the same rationale for claim 1. However, Zhu fails to disclose, but in the same art of computer graphics, Peng discloses A system of an augmented reality (AR) device comprising: a memory having computer readable instructions; and a computer for executing the computer readable instructions, the computer readable instructions controlling the computer to perform operations comprising: (Peng, para. [0006], “The electronic device includes at least one processor and a computer readable storage. The computer readable storage is coupled to the at least one processor and stores at least one computer executable instruction thereon which, when executed by the at least one processor, causes the at least one processor to:”). Note that: the electronic device can be regarded as a system. Zhu and Peng, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply the electronic device includes at least one processor and a computer readable storage, as taught by Peng into Zhu. The motivation would have been “The electronic device includes at least one processor and a computer readable storage. The computer readable storage is coupled to the at least one processor and stores at least one computer executable instruction thereon which, when executed by the at least one processor, causes the at least one processor to:” (Peng, para. [0006]). The suggestion for doing so would allow to use an electronic data processing device to perform computer graphics operations. Therefore, it would have been obvious to combine Zhu and Peng. Claims 9-11 and 13 are corresponding to the method of claims 2-4 and 6, respectively. Therefore, claims 9-11 and 13 are rejected for the same rationale for claims 2-4 and 6, respectively. Claim 15 reciting “A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform operations comprising:” is corresponding to the method of claim 1. Therefore, claim 15 is rejected for the same rationale for claim 1. However, Zhu fails to disclose, but in the same art of computer graphics, Peng discloses A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform operations comprising: (Peng, para. [0007], “a non-transitory computer readable storage medium is provided. The non-transitory computer readable storage medium is configured to store a computer program which, when executed by a processor, causes the processor to carry out following actions”). Zhu and Peng are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply non-transitory computer readable media comprising computer readable instructions which, when executed, configure a data processing system to perform operations, as taught by Peng into Zhu. The motivation would have been “a non- transitory computer readable storage medium is provided. The non-transitory computer readable storage medium is configured to store a computer program which, when executed by a processor, causes the processor to carry out following actions” (Peng, para. [0007]). The suggestion for doing so would allow to use one or more non-transitory computer readable media to perform computer graphics operations. Therefore, it would have been obvious to combine Zhu and Peng. Claims 16-18 and 20 and are corresponding to the method of claims 2-4 and 6, respectively. Therefore, claims 16-18 and 20 are rejected for the same rationale for claims 2-4 and 6, respectively . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim s 5, 7, 12, 14, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Regarding dependent claim 5, 12, and 19 , in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “in response to a field-of-view excluding the at least one component of the first machine, a guide is displayed to orient the field-of-view of the user to the at least one component on the first machine such that both the equivalent action of the digital representation and the at least one component are viewable on the AR device”. Regarding dependent claims 7 and 14 , in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “identifying that the new action is to be performed on the first machine is in response to the user inquiring how to perform the new action on the first machine in relation to the second machine; feedback from the user is received regarding a plurality of procedures of the at least one equivalent action being unsuccessful to assist the user with performing the new action; and updates to the plurality of procedures of the at least one equivalent action are made in response to the feedback”. Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Webel et al. (An augmented reality training platform for assembly and maintenance skills, Robotics and Autonomous Systems 61 (2013) 398-403) teaches “Trainig of this type can be supported by Augmented Reality, a powerful industrial training technology that directly links instructions on how to perform the service tasks to the machine parts that require processing”. Wang et al. (An Augmented Reality Based System for Remote Collaborative Maintenance Instruction of Complex Products, 2014 IEEE International Conference on Automation Science and Engineering (CASE), Taipei, Taiwan, August 18-22, 2014) teaches “a framework of augmented reality based collaborative maintenance real-time instruction platform under network environment is proposed to achieve efficiently guidance from remote expert”. Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /KEE M TUNG/ Supervisory Patent Examiner, Art Unit 2611 /Biao Chen/ Patent Examiner, Art Unit 2611 Application/Control Number: 18/302,036 Page 2 Art Unit: 2611 Application/Control Number: 18/302,036 Page 3 Art Unit: 2611 Application/Control Number: 18/302,036 Page 4 Art Unit: 2611 Application/Control Number: 18/302,036 Page 5 Art Unit: 2611 Application/Control Number: 18/302,036 Page 6 Art Unit: 2611 Application/Control Number: 18/302,036 Page 7 Art Unit: 2611 Application/Control Number: 18/302,036 Page 8 Art Unit: 2611 Application/Control Number: 18/302,036 Page 9 Art Unit: 2611 Application/Control Number: 18/302,036 Page 10 Art Unit: 2611 Application/Control Number: 18/302,036 Page 11 Art Unit: 2611 Application/Control Number: 18/302,036 Page 12 Art Unit: 2611 Application/Control Number: 18/302,036 Page 13 Art Unit: 2611
Prosecution Insights