DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chinnakannan et al1 (“Chinnakannan”) in view of Rovaglio et al2 (“Rovaglio”).
Regarding claim 1, Chinnakannan teaches a method comprising:
providing a first digital twin model of a first physical object (see Chinnakannan, column 3, lines 63-67 through column 4, lines 1-27 teaching “a digital twin service 104 that implements building a digital twin that can be used to simulate behavior and changes of a physical system. In the example embodiment, the provider network includes a runtime environment 106 a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104” where “the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” where for example “model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” such that here any physical object such as “physical component 118” or the collection of objects comprising “physical system 112” may be modeled as a “digital twin” and can be considered to be of a first physical object );
establishing a virtual reality environment for simulating operation of the first digital twin model (see Chinnakannan, column 3, lines 63-67 through column 4, lines 1-27 teaching establishing of a virtual reality environment such as the “runtime environment 106a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104“ where “Each of the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” and “for example model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” where this runtime environment is a virtual reality environment that includes the properties of the digital twins and the environment that affects the digital twins being simulated in the virtual reality environment);
operating the first digital twin model in the virtual reality environment (see Chinnakannan, column 3, lines 63-67 through column 4, lines 1-27 teaching establishing of a virtual reality environment such as the “runtime environment 106a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104“ where “Each of the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” and “for example model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” such that the virtual reality runtime environment operates the digital twin model in the environment to “simulate the behavior of a corresponding physical component” and may do so according to how the model has been built and instantiated to be simulated in the virtual reality environment );
receiving, by the first digital twin model, operational parameters from the virtual reality environment to determine operational effectiveness of the first digital twin model in operating within the virtual reality environment (see Chinnakannan, column 3, lines 63-67 through column 4, lines 1-27 teaching establishing of a virtual reality environment such as the “runtime environment 106a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104“ where “Each of the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” and “for example model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” such that here the digital twin model receives operational parameters such as “wind speed” and can determine operational effectiveness such as “rpm of a blade assembly due to wind speed” when taking such parameters into effect in combination with the other parameters of the digital twin); and
enabling a user to visualize the operation of the first digital twin model and change a viewing angle of the first digital twin model in the virtual reality environment (see Chinnakannan, column 8, lines 12-67 through column 9, lines 3 teaching “As shown in FIGS. 3 and 4 , in some embodiments, the initiating/performing of the simulation may include generating a 2D or 3D visualization of the physical system based on definitions of the models that are executed. The runtime environment may send the visualization to the remote network of the client (e.g., to a display of the management device). In some embodiments, some or all of the telemetry data received by the simulation is overlaid onto the visualization. For example, rpm values may be overlaid onto and/or near a visualization of the blade assembly” such that this 3D visualization is one enabling a user to visualize the operation of the first digital twin in the VR environment in which it is being simulated; see also column 10, lines 4-67 teaching “an “observability” use case, the digital twin of the physical system enables a user to visualize and manage physical system assets with an enhanced look and feel compared to the real physical system” and a “3D static visual rendering of the physical system” and “an “advanced observability” use case may enhance the above use case and enables a user to observe the physical system rendering along with contextualized information, location specific information, and/or processed information from other the IoT service and/or other services of the provider network (e.g., data received from the client network and formatted/transformed/etc.). This use case may use real time 3D model rendering and augmented reality/extended reality (AR/XR) technologies as desired/specified” and “a “system behavioral” use case may enable a user to answer any number of what-if questions through different types (physics, machine learning/AI, transforms/metrics) of simulations of physical system components applied on the digital twins (models) of the system components (e.g., by using one or more behavior functions)” and “there are certain characteristics of the physical system and criteria (e.g., requirements) for representing a corresponding digital twin of the system” which include “a digital twin may require visualization of each component that makes the physical system identified above (e.g., spatial and time co-ordinates for rendering these components); 4) a digital twin may require digital representation of each component of the physical system with high-fidelity that enables the substitution of the physical system with the digital twin for any/all use cases (a high-fidelity digital twin of a physical system may require high-fidelity for only a subset of components present in the physical system, while one or more of the rest of the components may be represented with lower and different grades of fidelity; a different use case may require different degrees of fidelity between/for the components that make up the physical system)” such that here again these use cases enable a user to visualize the operation of the first digital twin model in the virtual reality environment as they can observe a 3D rendering of the digital twin of the asset operating and different information about the digital twin as well; see also column 16, lines 6-15 teaching that the runtime environment “may render the twin instance of the physical system using the 3D models of the different physical components” and “may perform XR rendering and/or hot spotting as a graphical overlay on the visual rendering of a physical system” such that again a 3D model is presented in a virtual reality environment as the user is able to visualize the 3D digital twin models).
Chinnakannan teaches all that is required as explained above but fails to explicitly teach enabling a user to change a viewing angle of the first digital twin model in the virtual reality environment. As explained above, Chinnakannan does teach generation of a 3D model that can be visualized in real-time by a user in a virtual reality environment where the digital twin is operating (see Chinnakannan, column 10, lines 4-24 teaching “enables a user to observe the physical system rendering along with contextualized information, location specific information, and/or processed information from other the IoT service and/or other services of the provider network” where “This use case may use real time 3D model rendering and augmented reality/extended reality (AR/XR) technologies as desired/specified”). However, Chinnakannan does not explicitly detail all of the types of viewing and others interactions that would be made available by offering of such 3D digital twin for rendering to a virtual reality enabled visualization device and thus does not specifically teach that when visualizing such real time 3D models in a virtual reality environment that the user is enabled to change a viewing angle of the digital twin model in the virtual reality environment. Thus Chinnakannan stands as a base device upon which the claimed invention can be seen as an improvement comprising such ability to specifically change a viewing angle of the first digital twin model in the virtual reality environment which would allow a user to more fully immerse themselves and understand the digital twins and virtual environment.
In the same field of endeavor relating to providing simulations of digital twin models of equipment and control systems for viewing by a user, Rovaglio teaches providing a virtual reality environment that includes 3D modeling and display of simulations of the control environment and control devices (see Rovaglio, paragraphs 0015-0022 teaching “virtual reality environment generates, as a function of that operational status and one or more physical aspects of the control environment, a three-dimensional (“3D”) display of the control environment. The virtual reality environment is responsive to user interaction with one or more input devices to generate the 3D display so as to permit the user to interact with at least one of the control devices and the controlled equipment at least as represented by the 3D display of the control environment” and “virtual reality environment, according to aspects of the invention, displays the control environment as the participant would see it in actuality, i.e., as if he/she were really there. As a result, systems according to the invention a user to experience, via the 3D display of the control environment, interaction with and travel through (e.g., a “walk-through”) the control environment—which maybe actual or simulated. They can be used to monitor and control operational parameters of the environment” and “altered representational displays can, likewise, be based on such simulated and/or actual operational data, e.g., representing the inner workings and/or “behind the scenes” operation of the control devices, the controlled equipment (and the materials they process)” and teaching “ability to generate 3D displays of the control environment that not only represent the behavior of the controlled equipment and/or control devices, but do so in a manner that is in agreement with the actual or simulated condition of the control environment” and as in paragraphs 0056-0057 “Virtual reality environment 64 provides a three-dimensional or other immersive display of the environment 5, including not only the physical layout of that environment but also a reproduction of the operation of the control system 10 and apparatus at the facility controlled by it and, more particularly, by way of example, controlled equipment 22-32, the materials or other things processed by them, the processes by which they are processed, and/or the elements of the control system 10, including, for example, control devices 36-46. Simply put, in some embodiments, the virtual reality environment 64 provides an immersive display of the environment 5 that permits the trainee and/or other user (e.g., maintenance personnel, emergency first responders, and so forth) to experience walkthroughs and other interactions with the environment as if he/she were really there” and “That display can be conveyed to participants 72—particularly, for example, trainees—via stereoscopic headgear of the type used for VR display”) and teaches not only enabling a user to visualize the operation of the first digital twin model in the virtual reality environment as explained above but also teaches enabling the user to change a viewing angle of the first digital twin model in the virtual reality environment (see Rovaglio, paragraph 0016 teaching “virtual reality environment, according to aspects of the invention, displays the control environment as the participant would see it in actuality, i.e., as if he/she were really there. As a result, systems according to the invention a user to experience, via the 3D display of the control environment, interaction with and travel through (e.g., a “walk-through”) the control environment—which maybe actual or simulated” where one of ordinary skill in the art recognizes that such 3D display a virtual reality environment that includes the ability for the user to “walk-through” the virtual environment when they travel through the environment changes the viewing angle of the digital twins of the control environment as the user is able to walk through and utilize their VR display such that for example if a user moved forward on a path in relation to a digital twin in the VR environment then even such movement of the viewpoint forward would change the viewing angle of the digital twin in the VR environment with respect to the previous position and view and as in paragraphs 0055-0061 “Virtual reality environment 64 provides a three-dimensional or other immersive display of the environment 5, including not only the physical layout of that environment but also a reproduction of the operation of the control system 10 and apparatus at the facility controlled by it and, more particularly, by way of example, controlled equipment 22-32, the materials or other things processed by them, the processes by which they are processed, and/or the elements of the control system 10, including, for example, control devices 36-46. Simply put, in some embodiments, the virtual reality environment 64 provides an immersive display of the environment 5 that permits the trainee and/or other user (e.g., maintenance personnel, emergency first responders, and so forth) to experience walkthroughs and other interactions with the environment as if he/she were really there” and “That display can be conveyed to participants 72—particularly, for example, trainees—via stereoscopic headgear of the type used for VR display” and “Virtual reality environment 64 accepts input from simulation engine 62 representing the status of the environment 5, including, of processing stations 20 a-20 d, controlled equipment 22-32” such that again here such an “immersive display” allowing “to experience walkthroughs and other interactions with the environment as if he/she were really there” which are presented in “headgear of the type used for VR display” is recognized by one of ordinary skill in the art as teaching the ability of the user to change the viewing angle of digital twins in the virtual reality environment as such VR displays that display such immersive environments include as one of their most crucial features the ability to change the viewing angle of the visualization in connection with movement of such VR display; see further paragraph 0090 teaching “purpose of the VR Engine is “to run” an environment that is photorealistic and richly detailed, with realistic “look and feel” objects and illuminations, but running in synchronism with the simulation engine 62 and, thus, for example, it can create the virtual (or simulated) environment at the operator-selected speeds (e.g., real-time or faster). A real time program gives the possibility to let the user move and interact freely within the environment, without being bound to prefixed paths or animations as it happens with a more conventional non-real time rendering” and paragraph 0127 teaching “altering displays of the equipment/devices to give the user the impression of seeing into, through and/or around them” such that again this described ability to move and interact freely within the virtual reality environment without being bound to pre-fixed paths or animations teaches changing the viewing angle as the user moves and interacts freely in the environment and there is furthermore the ability to change a viewing angle of the digital twin models in the virtual reality environment for “seeing…around them” as in order to see around something like an object this means the viewing angle of the object is changed so that you can see around it which could be done by altering the position of the user moving freely within the virtual environment or changing the viewing angle by showing another such angle to a viewer ). Thus the prior art is found to contain a comparable device that has been improved in the same way as the claimed invention as Rovaglio also provides a 3D VR environment simulating digital twins of objects operating in the VR environment and allows the user to navigate and change their viewing angle of the digital twins in the immersive VR environment.
Therefore it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify Chinnakannan to apply the known improvement technique of Rovaglio in the same way to the base technique of Chinnakannan as doing so would be no more than use of a known technique to improve similar devices in the same way where the result of such a modification would have been predictable to one of ordinary skill in the art. Chinnakannan already teaches to provide a 3D display of a VR environment in a runtime environment and already suggests to provide VR models and provide a graphical view of the twins in real time and to use real time 3D model rendering and augmented reality/extended reality (both forms of VR). Thus the predictable result of the combination would be the addition of the ability of the user to specifically move through and interact and change the viewing angle of the 3D model rendering of the VR environment as in Rovaglio. Furthermore one of ordinary skill in the art before the effective filing date of the invention would have been motivated to modify Chinnakannan with the teachings of Rovaglio above to take advantage of displaying “the control environment as the participant would see it in actuality, i.e., as if he/she were really there” in order “to monitor and control operational parameters of the environment and to support training of plant or other of personnel, e.g., first responders or other safety personnel” for example as suggested by Rovaglio (see Rovaglio, paragraph 0016).
Regarding claim 2, Chinnakannan as modified teaches all that is required as applied to claim 1 above and further teaches providing a second digital twin model of a second physical object (see Chinnakannan, column 3, lines 63-67 through column 4, lines 1-27 teaching “a digital twin service 104 that implements building a digital twin that can be used to simulate behavior and changes of a physical system. In the example embodiment, the provider network includes a runtime environment 106 a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104” where “the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” where for example “model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” such that here each “component” can be considered a digital twin model where for example a turbine blade digital twin component could be a first digital twin model and a gearbox component model 116b is a second digital twin model provided of a second physical object, where column 6, lines 59-67 through column 7, lines 1-56 further teaches “Based on the relationship (provided by the user for the model definition) between the model 116 a and the model 116 b, the model 116 a may send the generated output (e.g., rpm data and/or torque data) to the model 116 b of a gear box of the wind turbine. Based on the behavior simulation for the model 116 b (provided by the user for the model definition) the model 116 b may process the output of the model 116 a to generate another output for the gear box (e.g., rpm). In a similar way, the output from model 116 b may be sent to model 116 c of a generator, which processes the output (e.g., rpm) to generate another output/result (e.g., power)” such that again various second digital twin model of second physical objects could be provided).
Regarding claim 3, Chinnakannan as modified teaches all that is required as applied to claim 2 above and further teaches operating the second digital twin model in the virtual reality environment along with the first digital twin model (see column 3, lines 63-67 through column 4, lines 1-27 teaching “a digital twin service 104 that implements building a digital twin that can be used to simulate behavior and changes of a physical system. In the example embodiment, the provider network includes a runtime environment 106 a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104” where “the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” where for example “model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” such that here each “component” can be considered a digital twin model where for example a turbine blade digital twin component could be a first digital twin model and a gearbox component model 116b is a second digital twin model provided of a second physical object, where column 6, lines 59-67 through column 7, lines 1-56 further teaches “Based on the relationship (provided by the user for the model definition) between the model 116 a and the model 116 b, the model 116 a may send the generated output (e.g., rpm data and/or torque data) to the model 116 b of a gear box of the wind turbine. Based on the behavior simulation for the model 116 b (provided by the user for the model definition) the model 116 b may process the output of the model 116 a to generate another output for the gear box (e.g., rpm). In a similar way, the output from model 116 b may be sent to model 116 c of a generator, which processes the output (e.g., rpm) to generate another output/result (e.g., power)” such that here the second digital twin model is operated in the virtual reality environment along with the first component such that the turbine blade and gearbox operate in the virtual environment).
Regarding claim 4, Chinnakannan as modified teaches all that is required as applied to claim 3 above and further teaches enabling interaction between the first digital twin model and the second digital twin model in the virtual reality environment (see Chinnakannan, column 3, lines 63-67 through column 4, lines 1-27 teaching “a digital twin service 104 that implements building a digital twin that can be used to simulate behavior and changes of a physical system. In the example embodiment, the provider network includes a runtime environment 106 a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104” where “the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” where for example “model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” such that here each “component” can be considered a digital twin model where for example a turbine blade digital twin component could be a first digital twin model and a gearbox component model 116b is a second digital twin model provided of a second physical object, where column 6, lines 59-67 through column 7, lines 1-56 further teaches “Based on the relationship (provided by the user for the model definition) between the model 116 a and the model 116 b, the model 116 a may send the generated output (e.g., rpm data and/or torque data) to the model 116 b of a gear box of the wind turbine. Based on the behavior simulation for the model 116 b (provided by the user for the model definition) the model 116 b may process the output of the model 116 a to generate another output for the gear box (e.g., rpm). In a similar way, the output from model 116 b may be sent to model 116 c of a generator, which processes the output (e.g., rpm) to generate another output/result (e.g., power)” such that here interaction is enabled between the models where the first model may send data to the second model which affects the operation of the second model in the environment, or the second model could have values changed which changes how the output data from the first model is processed which is another form of interaction).
Regarding claim 5, Chinnakannan as modified teaches all that is required as applied to claim 3 above and further teaches enabling operation of the first digital twin model in the virtual reality environment to affect operation of the second digital twin model in the virtual reality environment (see Chinnakannan, column 3, lines 63-67 through column 4, lines 1-27 teaching “a digital twin service 104 that implements building a digital twin that can be used to simulate behavior and changes of a physical system. In the example embodiment, the provider network includes a runtime environment 106 a of a compute service 108 that initiates/implements a digital twin 110 of a physical system (e.g., simulation of the physical system 112 at the client network/site 114 a) by instantiating and/or executing models 116 a-116 e that are built by the digital twin service 104” where “the executing models 116 and/or the runtime environment 106 a may simulate the behavior of a corresponding physical component 118 of the physical system 112, which may include simulating any internal behavior of the component and/or any number of interactions (inputs and/or outputs) between the component and any number of other components that may be processed by the component or other components” where for example “model 116 a simulates the behavior of component 118 a, receives an input to the component (e.g., environmental data such as wind speed), and provides an output of component 118 a that affects component 118 b (e.g., rpm of a blade assembly due to wind speed)” such that here each “component” can be considered a digital twin model where for example a turbine blade digital twin component could be a first digital twin model and a gearbox component model 116b is a second digital twin model provided of a second physical object, where column 6, lines 59-67 through column 7, lines 1-56 further teaches “Based on the relationship (provided by the user for the model definition) between the model 116 a and the model 116 b, the model 116 a may send the generated output (e.g., rpm data and/or torque data) to the model 116 b of a gear box of the wind turbine. Based on the behavior simulation for the model 116 b (provided by the user for the model definition) the model 116 b may process the output of the model 116 a to generate another output for the gear box (e.g., rpm). In a similar way, the output from model 116 b may be sent to model 116 c of a generator, which processes the output (e.g., rpm) to generate another output/result (e.g., power)” such that here the operating of the first model is based on its parameters which affects its output which is fed to the second model such that this changes the output of the second model ).
Regarding claim 6, Chinnakannan as modified teaches all that is required as applied to claim 1 above and further teaches wherein enabling a user to visualize comprises enabling a user to visualize on a virtual reality device (see Chinnakannan, column 8, lines 12-67 through column 9, lines 3 teaching “As shown in FIGS. 3 and 4 , in some embodiments, the initiating/performing of the simulation may include generating a 2D or 3D visualization of the physical system based on definitions of the models that are executed. The runtime environment may send the visualization to the remote network of the client (e.g., to a display of the management device). In some embodiments, some or all of the telemetry data received by the simulation is overlaid onto the visualization. For example, rpm values may be overlaid onto and/or near a visualization of the blade assembly” such that this 3D visualization is one enabling a user to visualize the operation of the first digital twin in the VR environment in which it is being simulated; see also column 10, lines 4-67 teaching “an “observability” use case, the digital twin of the physical system enables a user to visualize and manage physical system assets with an enhanced look and feel compared to the real physical system” and a “3D static visual rendering of the physical system” and “an “advanced observability” use case may enhance the above use case and enables a user to observe the physical system rendering along with contextualized information, location specific information, and/or processed information from other the IoT service and/or other services of the provider network (e.g., data received from the client network and formatted/transformed/etc.). This use case may use real time 3D model rendering and augmented reality/extended reality (AR/XR) technologies as desired/specified” and “a “system behavioral” use case may enable a user to answer any number of what-if questions through different types (physics, machine learning/AI, transforms/metrics) of simulations of physical system components applied on the digital twins (models) of the system components (e.g., by using one or more behavior functions)” and “there are certain characteristics of the physical system and criteria (e.g., requirements) for representing a corresponding digital twin of the system” which include “a digital twin may require visualization of each component that makes the physical system identified above (e.g., spatial and time co-ordinates for rendering these components); 4) a digital twin may require digital representation of each component of the physical system with high-fidelity that enables the substitution of the physical system with the digital twin for any/all use cases (a high-fidelity digital twin of a physical system may require high-fidelity for only a subset of components present in the physical system, while one or more of the rest of the components may be represented with lower and different grades of fidelity; a different use case may require different degrees of fidelity between/for the components that make up the physical system)” such that here these are all instances of visualization of the digital twin in a virtual reality environment such that the device used to view such visualizations is a virtual reality device as it is a device used for visualizing a virtual reality, where for example Chinnakannan, column 8, lines 12-24, teaches “runtime environment may send the visualization to the remote network of the client (e.g., to a display of the management device)” such that this display is a virtual reality device as it displays the virtual reality visualization).
Regarding claim 7, Chinnakannan as modified teaches all that is required as applied to claim 1 above and further teaches enabling a user to control the operational parameters in the virtual reality environment (see Chinnakannan, column 7, lines 3-56 teaching “During the execution of the models of the simulation of the physical system, a user may provide input (e.g., via the management device 134 and the interface 122) that indicates one or more changes to the simulation. The simulation modifier 140 may send the change(s) and/or the indication of the change(s) to the runtime environment. The runtime environment may modify the simulation to implement the indicated change(s) (e.g., forward the telemetry data to the appropriate models executing in the runtime environment and/or make changes to the models themselves) The runtime environment/model(s) may generate one or more results based on the modification(s) to the simulation” and “For example, the change may be a much higher wind speed than runtime environment is currently receiving as telemetry data. The runtime environment may generate one or more results based on the modification to the simulation. For example, a wind speed of 40 mph may be provided based on user input. A model of the blade assembly may receive the new wind speed indicating 40 mph. The model of the blade assembly may process the new wind speed (e.g., based on the behavior simulation for the model) and generate output data that indicates an increased rpm of the blade assembly. The new rpm may result in a new value for the power output of the generator. The runtime environment may send the new value for the power output (and/or the increased rpm and/or any other results of the change(s)) to one or more destinations (e.g., to the management device 134 and/or to the digital twin service or other destination within the provider network)” such that here the user can change any parameter of the environment of the digital twin or component or attribute of the digital twin in the virtual reality environment).
Regarding claims 8-14, the instant claims correspond to a “computer program product comprising a computer-readable storage medium having computer-usable program code embodied therein, the computer-usable program code configured to perform the following when executed by at least one processor” perform the same functions of the method as in claims 1-7, respectively. Note that Applicant’s Specification at paragraph 0017 explicitly disavows such a storage medium as any type of transitory signal as “computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through a fiber optic cable, electrical signals communicated through a wire, and/or other transmission media” and only provides examples of non-transitory type storage media. Chinnakannan as modified teaches such a computer program product and storage medium and processor (see Chinnakannan, column 24, lines 24-60 teaching “methods may be implemented by a computer system (e.g., a computer system as in FIG. 13 ) that includes one or more processors executing program instructions stored on a computer-readable storage medium coupled to the processors. The program instructions may implement the functionality described herein (e.g., the functionality of the digital twin service, compute service, models, and any other components that implement the techniques described herein). The various methods as illustrated in the figures and described herein represent example embodiments of methods. The order of any method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc”) which performs the method as explained above in the rejections of claims 1-7, respectively. In light of this, the limitations of claims 8-14 correspond to the limitations of claims 1-7, respectively; thus they are rejected on the same grounds as claims 1-7, respectively.
Regarding claims 15-20, the instant claims recite an apparatus in the form of a “system comprising: at least one processor; at least one memory device operably coupled to the at least one processor and storing instructions for execution on the at least one processor, the instructions causing the processor to” perform the method as in claims 1-5 and 7, respectively. Chinnakannan as modified teaches such an apparatus (see Chinnakannan, column 24, lines 24-60 teaching “methods may be implemented by a computer system (e.g., a computer system as in FIG. 13 ) that includes one or more processors executing program instructions stored on a computer-readable storage medium coupled to the processors. The program instructions may implement the functionality described herein (e.g., the functionality of the digital twin service, compute service, models, and any other components that implement the techniques described herein). The various methods as illustrated in the figures and described herein represent example embodiments of methods. The order of any method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc”) which performs the method where the functions of the method correspond to those as addressed in the rejections of claims 1-5 and 7, respectively. In light of this, the limitations of claims 15-20 correspond to the limitations of claims 1-5 and 7, respectively; thus they are rejected on the same grounds as claims 1-5 and 7, respectively.
Response to Arguments
Applicant’s arguments, see “REMARKS”, filed 5/6/2026, with respect to the rejection(s) of claim(s) 1-20 under 35 U.S.C. 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Chinnakannan and Rovaglio.
Applicant argues with respect to claim 1 and the other independent claims that Chinnakannan fails to teach or suggest the newly added claim limitation relating to changing a viewing angle of a digital twin in a virtual reality environment. The Examiner agrees that Chinnakannan fails to explicitly teach such a limitation, and thus the Rovaglio prior art is provided to show that such visualizing of a virtual reality environment and objects represented therein including changing of a viewing angle of such objects in the VR environment is known in the art and renders the claims obvious in combination with Chinnakannan. Applicant also argues that “a runtime environment is not inherently or necessarily a virtual reality environment as that term is generally understood. A runtime environment need not provide the capabilities of a virtual reality environment.” However, regardless of whether this is true or not, it should be noted that it was never asserted by the Examiner that a runtime environment inherently or necessarily is a virtual reality environment as that term is generally understood. Rather, in Chinnakannan, the runtime environment does not necessarily provide a VR environment in which a user can visualize the environment, but the runtime environment is disclosed as including an embodiment in which such a virtual reality environment is displayed, including in the portions cited by Applicant where there an example use case of the runtime environment “may use real time 3D model rendering and augmented reality/extended reality (AR/XR) technologies as desired/specified” and as pointed about above and taught in column 14, “twin service collects data from the data sources and overlays the data with the 3D/AR/VR models to provide a graphical view of the twins in real time” and as in column 16, “twin runtime may render the twin instance of the physical system using the 3D models of the different physical components” and “digital twin monitor may perform XR rendering and/or hot spotting as a graphical overlay on the visual rendering of a physical system.” Thus clearly it is taught and suggested to provide a visualization of the digital twin models rendered in a virtual reality environment such as this visual rendering of a physical system such that the runtime environment is able to provide such a VR environment for visualization by a user. As explained above though, Chinnakannan does not specifically teach how such a rendering is able to be interacted with by a user and does not specifically teach the ability of the user to specifically move freely or interact to change views of the digital twin operating in the VR environment. Thus Rovaglio teaches such a limitation in combination with Chinnakannan as explained above. Thus the claims stand rejected as fully explained above.
The examiner notes that providing the ability for a user to change the viewing angle of an object they are viewing in a 3D virtual reality environment in which they are navigating is extremely old and well-known in the art of VR display and viewing as one of the main purposes of providing such an environment to a VR viewing device is to allow a user to freely view the environment and experience the immersion in the VR environment through such changes of the viewing angle to view the world around the user (see pertinent art section below for example).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
see Webb et al (US PGPUB No. 20180308379) teaching that it is known to create digital representations of objects for display in a virtual reality environment where the object’s behavior is being simulated and that viewing objects in a virtual reality environment includes viewing of objects in the environment from different viewing angles (see Webb, abstract “systems for creating digital representations of real world objects that are connected back to real world objects, creating a digital double. In some aspects a method includes the actions of receiving sensor data from a real world object; receiving data representing a digital version of the real world object; and performing a virtual reality simulation displaying (i) a representation of at least some of the sensor data, and (ii) the digital version of the real world object” and paragraph 0003 teaching “virtual reality technology can be used to enable engineers from different disciplines to view and test current product designs—sometimes prior to the availability of any physical prototypes. Within a virtual reality environment, products can be viewed from multiple angles” and paragraph 0052 teaching “functionalities include a manipulation functionality that allows a viewer of a digital representation of an object generated by a converted CAD file to rotate and zoom in”).
see Schmirler et al (US PGPUB No. 20180131907) teaching providing a VR environment for viewing of digital twins operating in a virtual reality environment including the ability for the user to view and interact with the virtual reality environment including through changing the viewing angle of objects in the virtual environment (see Abstract, “industrial visualization system generates and delivers virtual reality (VR) and augmented reality (AR) presentations of industrial facilities to wearable appliances to facilitate remote or enhanced interaction with automation systems within the facility. VR presentations can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility. The system can selectively render… a first-person view that renders the facility as a full-scale rendition that simulates the user's presence on the plant floor” and paragraph 0050 teaching “a system that generates and delivers augmented reality (AR) or virtual reality (VR) presentations (referred to collectively herein as “VR/AR presentations”) to a user via a wearable computer or other client device. VR/AR presentations generated by the system can comprise three-dimensional (3D) holographic views of a plant facility or a location within a plant facility (e.g., a work area, a production line, etc.). The holographic views can be delivered to a wearable visualization computer, which renders the 3D view as a function of the user's current location and/or orientation” and “an internal view that renders a realistic presentation of the factory floor area from the point of view of a person standing within the environment” and as in paragraphs 0087-0088 “In general, the VR/AR presentations rendered by wearable appliances 206 provide surrounded virtual renderings that encompass the user's entire field of view, and transition their line of sight or perspective as the user's location and orientation change” and “Rendering component 308 can support… first-person views of the area that simulate the user's presence within the industrial area by rendering a full-scale view of the area” where “VR/AR presentation system 302 can stream up-to-date VR/AR presentation data 604 to wearable appliance 206 to ensure that the view—including the user's angle of perspective and live industrial data values—remains current” as in the example in paragraph 0092 teaching “if the user's wearable appliance 206 is currently presenting the view depicted in FIG. 8, and the user moves forward and slightly to the left, the rendering component 308 will transition to the view depicted in FIG. 9. In general, the external view generated by VR/AR presentation system 302 renders the industrial area as a virtual scale model of the area, and allows the user to move around and interact with the scaled version of the area. As the user moves around, toward, or away from the virtual scaled industrial area, the wearable appliance 206 streams updated location and orientation data 606 to the presentation system 302, which updates the presentation data 604 substantially continuously to simulate the effect of walking around a scale model of the production area”).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SCOTT E SONNERS whose telephone number is (571)270-7504. The examiner can normally be reached Mon-Friday 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SCOTT E SONNERS/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
1 US Patent No. 12511454
2 US PGPUB No. 20090319058
Prosecution Insights