DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9, 11-14, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 20180267496 A1) in view of Atohira (US 20180121578 A1) in further view of Hazan (US 20230311324 A1)
Regarding claim 9, Wang teaches A control system for controlling a control target ([0010] A method and system is disclosed in which an AR application enables the improvement of an automation of tasks related to development and testing of a control application program of a system controller. [0019] FIG. 2 shows an example configuration of an AR environment according to embodiments of the disclosure), the control system comprising:
a control device including a processor configured to execute one or more programs to cause: ([0026] A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device)
a programmable logic controller (PLC) engine to cyclically execute a program including a sequence instruction ([0011] The controller 110, such as a PLC, may have multiple application programs 120 stored in a memory, each program for performing a particular control task in an automation arrangement.), a robot control engine to control a robot (Fig. 1 [0014] In an embodiment, control sensors associated with a robotic unit 1 may produce real unit 1 data 155. For example, motion sensors may provide signals to analyze maximum speed thresholds for robotic components. If robotic unit 2 is not yet installed, virtual unit 2 data 147 may be simulated for display on the AR device 115 and for interaction with the control application 120.), and an image processing engine to execute image processing on an image from a camera; and ([0012] and real vision system data 159 related to visual sensing devices. [0013] Virtual vision system data 149 may be based on simulated perceptions of a digital twin)
a support device including a processor configured to execute one or more programs to: ([0026] A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device)
construct, according to the user selection, one or more simulation modules that simulate at least a part of the control target, the robot, and the camera, ([0019] Server 251 may include one or more processors to execute virtual modules 252 to generate digital twins for simulation of hardware counterparts in section 210. [0013] Virtual vision system data 149 may be based on simulated perceptions of a digital twin [0014] robotic unit 2 is not yet installed, virtual unit 2 data 147 may be simulated for display on the AR device 115 and for interaction with the control application 120. [0020] The virtual gantry 221 and virtual gripper 224 may be simulated in virtual modules 252 to move a virtual workpiece 222 along a path 223 on a real table 211 for a collision avoidance application program. A visual sensor 212, such as a camera, may provide input to the controller 110 to provide an AR input that represents the optics for the real components, such as table 211 and a real object 214 [0021] The user 101 may initially setup the control application using initial parameters to allow training of the control application. )
associate, when the constructed simulation module is configured to simulate at least a part of the control target, the constructed simulation module with the PLC engine, ([0013] As shown in FIG. 1, virtual product/workpiece data 141 may include any signals or information related to a simulation of the product or workpiece as produced by the AR device 115 or a separate AR simulator (not shown). For example, virtual product/workpiece data 141 may include reactive motion responsive to the simulated environment, which may be translated into a visual representation 160 of the virtual product/workpiece for presentation in AR device 115.)
associate, when the constructed simulation module is configured to simulate the robot, the constructed simulation module with the robot control engine, and ([0020] A virtual gantry 221 and virtual gripper 224 is shown, representing the AR simulation that may be rendered and displayed on the AR device 115. The virtual gantry 221 and virtual gripper 224 may be simulated in virtual modules 252 to move a virtual workpiece 222 along a path 223 on a real table 211 for a collision avoidance application program)
associate, when the constructed simulation module is configured to simulate the camera, the constructed simulation module with the image processing engine. ([0020] The visual data from visual sensor 212 may also be provided to the AR device 115 via a wired or wireless connection for an AR display of the simulated operation in section 210, which enables remote viewing of the simulated operation.)
Wang does not expressly disclose but Atohira discloses receive selection of a real device or a virtual device for being associated with each of the PLC engine, the robot control engine, and the image processing engine, ([0052] The user operates the data input section 20 so as to select the robot model 102 and the conveyer model 104 from the list displayed on the display 22. The data input section 20 sends the input data input by the user to the CPU 12 via the I/O interface 18 [0053] In response to the received input data, the CPU 12 reads out the robot model 102 and the conveyer model 104 from the pluralities of types of robot models and conveyer models stored in the system memory 14, and arranges the robot model 102 and the conveyer model 104 in the virtual space 100. Then, the CPU 12 generates image data of the virtual space 100, and displays it on the display 22 as an image of the virtual space 100 as illustrated in FIG. 3 [0071] Below, a case is described in which the user selects the robot model 102. In this case, the data input section 20 receives the input operation by the user, and sends the input data indicating the selection of the robot model 102 to the CPU 12 via the I/O interface 18. In response to the received input data, the CPU 12 determines the robot model 102 as the replication subject.)
associate, when the real device is selected, specific information of the selected real device with the corresponding the PLC engine, the robot control engine, or the image processing engine, ([0052] The user operates the data input section 20 so as to select the robot model 102 and the conveyer model 104 from the list displayed on the display 22. The data input section 20 sends the input data input by the user to the CPU 12 via the I/O interface 18 [0053] In response to the received input data, the CPU 12 reads out the robot model 102 and the conveyer model 104 from the pluralities of types of robot models and conveyer models stored in the system memory 14, and arranges the robot model 102 and the conveyer model 104 in the virtual space 100. Then, the CPU 12 generates image data of the virtual space 100, and displays it on the display 22 as an image of the virtual space 100 as illustrated in FIG. 3 [0071] Below, a case is described in which the user selects the robot model 102. In this case, the data input section 20 receives the input operation by the user, and sends the input data indicating the selection of the robot model 102 to the CPU 12 via the I/O interface 18. In response to the received input data, the CPU 12 determines the robot model 102 as the replication subject.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Wang with the teachings of Atohira with a reasonable expectation of success by carrying out a simulation to simulatively operate the component model and the symmetric component model in the virtual space as taught by Atohira ([0015]).
Wang does not expressly disclose but Hazan discloses provide the one or more constructed simulation modules to the control device, and ([0027] providing access to a virtual simulation system capable of simulating a plurality of robotic tasks of the plurality of robots interacting with their corresponding object sets within the virtual environment. )
wherein the processor of the control device ([0056] The data processing system 100 illustrated can include a processor 102 ) is further configured to execute the one or more programs ([0019] Software applications for industrial virtual simulation are platforms able to simulate the robotic tasks of the cell robots by running their robotic programs within the virtual cell environment, preferably including all cell's signals and PLCs) to cause a simulator to simulate at least a part of the control target ([0075] Analyzing the simulated objects with their corresponding collected data), the robot ([0110] the virtual simulation system receives virtual data inputs on the virtual environment including its plurality of virtual robots and corresponding plurality of virtual object sets), and the camera according to the one or more constructed simulation modules. ([0027] The method further includes defining a given virtual camera point of view suitable for dynamically displaying the given robot and the object set of the given robotic relation group.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Wang with the teachings of Hazan with a reasonable expectation of success by automatically determine sample robotic motion data for dynamically and efficiently displaying a robotic task of a specific robot within a robotic environment as taught by Hazan ([0039]).
Regarding claim 11, Wang teaches The control system according to claim 9,
wherein the processor of the control device is further configured to execute the one or more programs to cause: ([0026] A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device)
a visualizer to visualize a state of the control target in a virtual space based on processing results of the PLC engine, the robot control engine, and the image processing engine. ([0011] In an embodiment, a user 101 may wear an Augmented Reality (AR) device 115, such as Microsoft HoloLens, while operating a graphical user interface (GUI) device 105 to develop a control application program 120 for a controller 110.)
Regarding claim 12, Wang teaches The control system according to claim 11, wherein
the simulation module is realized by using a result of collision detection in the virtual space visualized by the visualizer. ([0024] a virtual robotic unit may be displayed on the AR device 115 while performing various programmed movements, allowing the user to monitor for any potential hazards to a worker in the working environment, such as a possible collision between the worker and a workpiece or the robotic unit.)
Regarding claim 13, Wang teaches The control system according to claim 9, wherein
the simulation module is realized by using an emulator configured to simulate a behavior of a device and/or equipment included in the control target. ([0016] In such case, the AR device 115 may provide display of one or more simulated structures or objects within the work environment, such as products or workpieces or obstacles during testing and commissioning of the automation installation.)
Regarding claim 14, Wang teaches The control system according to claim 9, wherein
the simulation module is realized by using a simulator configured to simulate a physical motion of an object included in the control target. ([0016] The AR device 115 may also display a simulated path or movement of a real component as a visual prediction displayed to the user 101 prior to the actual path or movement. )
Regarding claim 21, Wang teaches The control system according to claim 9, wherein
the PLC engine cyclically executes a motion control instruction for controlling one or more motors that drive a second robot, and ([0022] The above simulation provides an example of training the controller application modules to learn various possible paths for a robotic unit, such as a gantry and/or gripper, for which control instructions are to be executed for motion control in conjunction with feedback from various sensor inputs.)
the robot control engine sequentially generates an instruction for controlling the robot according to a robot program. ([0023] The path planning module 322 may include instructions and coding related to operating the gantry 221 and gripper 224 used to move a workpiece 222 along a path 223 within a workspace.)
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 20180267496 A1) in view of Atohira (US 20180121578 A1) in further view of Hazan (US 20230311324 A1) in further view of Kanaya (US 20170293502 A1).
Regarding claim 22, Wang does not expressly disclose but Kanaya discloses The control system according to claim 9, wherein the processor is further configured to execute the one or more programs to cause: ([0008] The control device includes: a hardware resource, including one or more processors and storage devices;)
a hypervisor to manage common hardware resources, wherein ([0031] The hardware resource allocated to the real-time OS and the general purpose OS is managed by a hypervisor.)
the hypervisor provides an execution environment for a real-time operating system (OS) and a general- purpose OS, ([0031] The hardware resource allocated to the real-time OS and the general purpose OS is managed by a hypervisor.)
the PLC engine is running on the real-time OS, and ([0051] The real-time OS 114 executes a designated task based on designated priority, thereby providing an environment capable of completing a designated process within a predetermined time. Specifically, the real-time OS 114 includes a scheduler, an input/output (I/O) process module, a network processing module and so on.)
the robot control engine and the image processing engine are running on the general- purpose OS. ([0052] The general purpose OS 116 is executed using the hardware resource of the control device 100 and operates independently of the real-time OS 114. The general purpose OS 116 is an OS executed by a general personal computer or the like, and examples thereof include Windows®, Mac OS®, Unix®, Linux®, Android®, MS-DOS®, and any other arbitrary OS derived from Unix.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Wang with the teachings of Kanaya with a reasonable expectation of success by providing a more suitable configuration for a control device adopting a virtualization technique as taught by Kanaya ([0021]).
Response to Arguments
Applicants arguments filed 7/18/2025 have been fully considered as follows:
Applicant argues that the 35 USC 103 rejections to the claims should not be maintained in view of “As agreed in the interview of July 10, 2025, the cited references fail to disclose the amended features of claim 9. Therefore, claim 9 is patentable over the cited art of record.” However, a new ground of rejection is above in view of the amendments.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH TRAN whose telephone number is (313)446-6642. The examiner can normally be reached 8am-5pm M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at (571) 272-6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.A.T./Examiner, Art Unit 3656
/KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights