DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 12, “the monitoring module.” needs to be changed to “a monitoring module.”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 7, 9-12, 16-17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Rovaglio, U.S. Patent Application Publication No. 2009/0319058 A1 (hereinafter: ‘058).
As mapped below, ‘058 discloses each and every feature of claim 1; accordingly, implementing the claimed system in view of ‘058 would have been a routine design choice to one of ordinary skill in the art at the time the invention was made, rendering the subject matter of claim 1 obvious.
As per claim 1, ‘058 discloses an industrial process virtual evaluation system the industrial process being intended to be implemented in an industrial facility, the industrial process virtual evaluation system comprising at least one computer platform (Interpreted to correspond to a “virtual test bench” for a factory or plant process, that uses at least one computer to operate; See ‘058; [0003] and [0051]) configured to:
dynamically compute in a dynamic process simulator operating parameters of the industrial process, based on input parameters (e.g., Interpreted to correspond to the computer using input settings to run a real time simulation for the factory or plant process; See ‘058; [0053] - [0054]);
generate in an immersive simulator an immersive three-dimensional representation of the industrial facility comprising a plurality of simulated controls configured to define the input parameters (e.g., Interpreted to correspond to the computer generating a VR style 3D view of the plant or factory, where the operator can “see” many virtual controls for the plant or factory; See ‘058; [0056] and [0060]);
a display and/or controller configured to be operated by a real operator and/or by a robot, to navigate in the immersive three-dimensional representation, and to actuate at least one of the plurality of simulated control (e.g., Interpreted to correspond to a screen and/or a control device that an operator and/or a robot can use to virtually move around inside the plant or factory, and interact with the virtual controls; See ‘058; [0016] and [0057]);
wherein the at least one computer platform is further configured to:
interconnect in an interface the dynamic computing of operating parameters of the industrial process and the generating of the an immersive three-dimensional representation of the industrial facility so as to allow the operator and/or the robot to perform in real time, in the immersive three-dimensional representation a virtual implementation of the industrial process comprising actions performed in the immersive three-dimensional representation, the actions performed in the immersive three-dimensional representation including the actuation of the at least one simulated control (e.g., Interpreted to correspond to the computer interfacing a dynamic process simulator with the 3D VR plant/factory so that the operator/robot can, in real time, run through a virtual version of the industrial process inside the 3D plant/factory by performing actions in the 3D world, such as by using the virtual controls; See ‘058; [0055], [0061] and [0090]);
receive from the dynamic computing of operating parameters of the industrial process, operating parameters of the industrial process as a function of the input parameters, resulting from the action performed by the operator and/or the robot in the immersive three-dimensional representation (e.g., Interpreted to correspond to the computer receiving back process values based on the inputs that come from the actions that the operator or robot perform inside the 3D plant or factory; See ‘058; [0058] and [0061]);
record monitoring data including operating parameters of the process and/or actions of the operator and/or of the robot as a function of time (e.g., Interpreted to correspond to the system logging (a) the performed actions of the operator/human over time or (b) the process parameter values over time; See ‘058; [0022], [0045] and [0094]);
process the monitoring data and provide a rendering of a virtual implementation of the industrial process, to conduct or optimize the industrial process (e.g., See Interpreted to correspond to the computer analyzing the logged data and producing an output of the analysis showing how the process played out in the virtual plant or factory, the output being a report, a video or other type of display); See ‘058; [0076] – [0077] and [0121]).
As per claim 2, ‘058 further discloses that the interface is configured to dynamically receive at least one input parameter of the input parameters resulting from a simulated control performed by the operator and/or the robot in the immersive three-dimensional representation generated by the immersive simulator the dynamic process simulator being configured to load the at least one input parameter received in the interface, so as to perform a dynamic simulation of the process, and to obtain at least one operating parameter of the process, on the basis of the at least one input parameter resulting from a simulated control performed by the operator and/or the robot in the immersive three-dimensional representation the interface being configured to dynamically receive the at least one operating parameter of the process resulting from the dynamic simulation of the process (e.g., Interpreted to correspond to the VR interface being connected to the process simulator in real time, and that when an operator or robot changes a virtual control, the change becomes a simulator input, whereby the simulator calculates new process values and sends them back to the immersive 3D view (e.g., See ‘058; [0014], [0015] and [0061]).
As per claim 3, ‘058 further discloses that the at least one computer platform in configured to automatically produce a report on a virtual implementation of the process, including a time-dependent list of the monitoring data (e.g., Interpreted to correspond to the system automatically creating a report about a virtual run of the process, the report including a time ordered listing of the logged actions and/or process values; See ‘058; [0045], [0048] and [0094]).
As per claim 5, ‘058 further discloses that the display and/or controller comprises sensor measuring positions and orientations of the operator and/or of the robot in the immersive three-dimensional the report on a virtual implementation of the process comprising a list of orientation position data from a viewpoint of the operator and/or of the robot as a function of in the immersive three-dimensional representation during a virtual implementation of the process, the viewpoint of the operator and/or of the robot being obtained from measurements taken by the sensor measuring position and orientation of the operator and/or of the robot in the immersive three-dimensional representation (e.g., Interpreted to correspond to the system report also showing where the operator or robot was and which way they were facing in the 3D plant or factory over time by using trackers that follow positions and directions during the virtual run; See ‘058; [0073] and [0094]).
As per claim 7, ‘058 further discloses that the report on a virtual implementation of the process comprises recorded values of the operating parameters of the process (e.g., Interpreted to correspond to the report also including the recorded process value that were simulated and logged during the session; See ‘058; [0045], [0048] and [0094).
As per claim 9, ‘058 further discloses that the immersive simulator configured to represent, in the immersive three-dimensional representation, at least one information relating to the process and intended for the operator, coming from the dynamic process simulator (e.g., Interpreted to correspond to the 3D VR plant or factory showing process data (sensor readings or status info) from the process simulator so the operator can see that data inside the immersive 3D view; See ‘058; [0018] – [0019], [0058] and [0125]).
As per claim 10, ‘058 further discloses that the immersive simulator is configured to represent, in the immersive three-dimensional representation, a process monitoring tool displaying the process information (e.g. Interpreted to correspond to that while inside the 3D VR plant or factory, the operator can see a process monitoring screen which displays process information coming from the simulator; See ‘058; [0018]; [0090] and [0125]).
As per claim 11, ‘058 discloses a method of virtual execution of an industrial process, the industrial process being intended to be implemented in an industrial facility, the method comprising:
providing a virtual evaluation system according to claim 1 (e.g., the rational as set forth above with respect to rejection of claim 1 is incorporated by reference herein. The details of a virtual execution of an industrial facility are amply evident in the rejection of claim 1);
navigation, via the display and/or controller of a real operator and/or of a robot in the immersive three-dimensional representation generated by the immersive simulator, so as to perform actions in the immersive three-dimensional representation, the actions including the actuation of at least one simulated control defining input parameters for the dynamic process simulator (e.g., Interpreted to correspond to the operator or robot using the screen and controls to move around inside the 3D plant or factory and interact within it, by pressing buttons or using knobs that present as if real controls, the virtual actions becoming inputs to the simulator; See ‘058; [0015], [0056] – [0057] and [0061]);
obtaining real-time operating parameters of the process, via the dynamic process simulator, according to the input parameters resulting from the actions performed by the real operator and/or the robot, received by the interface and loaded by the dynamic process simulator (e.g., Interpreted to correspond to the simulator calculating in real time updated process values based on the simulator inputs from the operator or robots’ actions; See ‘058; [0014], [0053] – [0054]);
recording, of monitoring data including operating parameters of the process and/or actions of the operator and/or of the robot as a function of time (e.g., Interpreted to correspond to the system time stamping the logs during the virtual run, recording what the operator/robot did or the process values changing over time; See ‘058; [0022], [0045] and [0094]);
processing, of the monitoring data obtained for providing a rendering of a virtual implementation of the process, intended for the conduct or the optimization of the industrial process (e.g., Interpreted to correspond to the computer studying the logged data and producing an output that shows how the process behaved; See ‘058; [0018] – [0019], [0021] and [0125]).
As per claim 12, ‘058 further discloses that the processing of the monitoring data comprises the automatic production of a report on a virtual implementation of the process, including a time-dependent list of monitoring data collected by a monitoring module (e.g., Interpreted to correspond to generating a report of the virtual run, over time, of the monitored data collected during the session; See ‘058; [0045], [ 0048] and [0094]).
As per claim 16, ‘058 further discloses that the immersive simulator represents, in the immersive three-dimensional representation, a process monitoring tool, displaying information on the process to the operator (e.g., Interpreted to correspond to the VR simulator showing a process monitoring screen inside the 3D plant or factory view so the operator can see the information in the immersive environment; See ‘058; [0018], [0058] and [0090]).
As per claim 17, ‘058 further discloses preparing a procedure from at least one virtual implementation of an industrial process, and operating a real industrial process in an industrial facility according to the procedure (e.g., Interpreted to correspond to using the virtual run to plan, then executing the run in a real plant or factory; See ‘058; [0070] and [0121]).
As per claim 19, ‘058 further discloses that the immersive simulator is configured to load the at least one operating parameter of the process resulting from the dynamic simulation of the process, and to modify the immersive three-dimensional representation in response to the at least one operating parameter of the process resulting from the dynamic simulation of the process (e.g., Interpreted to correspond to using the simulated calculated process values to control the 3D view of the plant or factory by updating what the operator sees based on the updating of the values as they are updated in real time; See ‘058; [0018] – [0019], [0058] and [0125]).
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Rovaglio (‘058), as applied to claims 3 and 12, respectively, from above, in view of Dutta, U.S. Patent Application Publication No. 2018/0357922 A1 (hereinafter: ‘922).
As per claim 4, ‘058 does not specially disclose that the virtual evaluation system further includes a speech recognition application, the report on a virtual implementation of the process including a time-dependent word-by-word rendering of words pronounced by the operator during a virtual implementation of the process, as obtained using the speech recognition application.
In analogous art, ‘922 discloses this feature (e.g., Interpreted to correspond to listening to what the operator says, turning that into text, and storing the transcript as part of the logged run report; See ‘922; [0020] – [0023] and [0029]).
With respect to claim 13, the rational as applied with respect to the rejection of claim 4, from above, is incorporated by reference herein and applied.
It would be obvious to one of ordinary skill in the art at the time the invention was made to have incorporated the teachings of ‘922 into ‘058 for the purpose of fully recording operator behavior, including spoken words, in the virtual plant or factory, to generate more accurate run reports and support more reliable evaluation of trainee performance.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Rovaglio (‘058), as applied to claim 3, from above, in view of Dudehula, U.S. Patent Application Publication No. 2018/0356878 A1 (hereinafter: ‘878).
As per claim 6, ‘058 does not specifically disclose that the report on a virtual implementation of the process includes a list of pictures or videos from a viewpoint of the operator, as a function of time, in the immersive three-dimensional representation during a virtual implementation of the process.
In analogous art, ‘878 discloses this feature (e.g., Interpreted to correspond to the report including pictures or video clips taken from inside the 3D plant or factory view, showing what the operator viewed during the virtual session run; See ‘878; [0017] and [0026] – [0028]).
It would be obvious to one of ordinary skill in the art at the time the invention was made to have incorporated the teachings of ‘878 into ‘058 for the purpose of capturing and replaying what the operator experienced in the virtual plant or factory, so that review and future training could incorporate the viewpoints into instructions to provide optimized training.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Rovaglio (‘058), as applied to claim 11, from above, and further in view of XIONG et al., U.S. Patent Application Publication No. 2018/0169862 A1 (hereinafter: ‘862).
As per claim 14, ‘058 does not specifically disclose that the navigation comprises the operation of the display and/or controller by a robot, the robot comprising a central controller containing a list of actions of a real or a virtual process to be performed.
In analogous art, ‘862 discloses these features (e.g., Interpreted to correspond to the navigation and actions on the 3D plant or factory are carried out by a robot that uses a controller having a list of actions that define the real process; See ‘862; [00015] and [0046]).
It would be obvious to one of ordinary skill in the art at the time the invention was made to have incorporated the teachings of ‘862 into ‘058 for the purpose of automating repeated operator actions, improving consistency, and reducing human effort in running virtual plant procedures and increasing safety.
Allowable Subject Matter
Claims 8, 15 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
As per claim 8, the prior art of record fails to teach or adequately disclose the features of claim 8 that are interpreted to correspond to a robot using a stored list of actions to drive the 3D plant or factory, wherein if real action execution is temporarily blocked, the robot stops and a modified action list is then tried in the simulator and the results of the simulator are evaluated to determine which results are acceptable, in combination with the other claimed features and or limitations.
As per claim 15, the prior art of record fails to teach or adequately suggest that during the implementation of the real or a virtual process, the robot stops the actions the robot has to perform in the presence of conditions temporarily preventing the continuation of the real or a virtual implementation of the process, the method comprising the launch, by the robot or by a driver of the robot, of a virtual implementation of a continuation of the process, on the basis of at least one modified action in the action list, and establishing, from the monitoring data collected during a virtual implementation of the continuation of the process, evaluation criteria for a real or a virtual continuation of the process, containing at least one modified action (e.g., Interpreted to correspond to a feature whereby if something blocks the robot, then it pauses, tests potential actions from a list in a virtual mode, and then uses the virtual run results to decide how to continue), in combination with the other claimed features and or limitations as claimed.
As per claim 18, the prior art of record fails to teach or adequately suggest a plurality of reports on virtual implementations of the process being obtained by a plurality of implementations of the virtual execution method, under different execution conditions of the virtual execution method, the procedure then being prepared using one of the plurality of reports on a virtual implementation of the process (e.g., Interpreted to correspond to running several instances of the virtual run under different conditions, generating the respective reports, and then choosing specific reports to use as the basis for the operations of the real plant or factory), in combination with the other claimed features and or limitations as claimed.
References Cited but Not Relied Upon
The following references were considered but were not relied upon with respect to any prior art rejections:
(1) US 2017/0148214 A1, which discloses VR headsets that allows workers to be in virtual industrial worksites using virtual equipment and dealing with virtual hazards, while sensors track performance and stress for training purposes;
(2) US 2018/0131907 A1, which discloses an industrial visualization system that generates VR/AR 3D plant views to wearable devices, allowing users to remotely view and interact with automation systems in the facility;
(3) US 2018/0210436 A1, which discloses a digit twin that combines statis plant models and dynamic simulation/fault models to monitor industrial processes and predict or compare plant performance;
(4) US 8,977,527 B2, which discloses automatically building process simulation models from libraries to test distributed control system and support operator training;
(5) US 10,108,763 B2, which discloses simulating an automated plant using sub models with calculation algorithm and determining execution order and timing to analyze the plant model; and
(6) US 2016/0328883 A1, which discloses using digital twins and sensor data to generate AR overlays with virtual controls and live data on real equipment, thereby allowing remote control and diagnostics.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RONALD D HARTMAN JR whose telephone number is (571)272-3684. The examiner can normally be reached M-F 8:30 - 4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mohammad Ali can be reached at (571) 272-4105. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RONALD D HARTMAN JR/Primary Patent Examiner, Art Unit 2119 December 2, 2025
/RDH/