DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-25 are rejected under 35 U.S.C. 103 as being unpatentable over Schmirler (US 20200336706 A1) in view of Ha (US 20210201584 A1)
Regarding claim 1, Schmirler teaches An automation system comprising: ([0002] The subject matter disclosed herein relates generally to industrial automation systems, and, more particularly, to visualization of industrial data)
augmented reality glasses wearable by a user and configured to present visual information to the user; ([0054] FIG. 2 is a conceptual diagram illustrating presentation of augmented or virtual reality presentations 204 to a wearable appliance 206 or computing device worn by a user.)
executable program logic communicatively coupled to the database system and the augmented reality glasses([0110] the user can send a request to present system 302 (e.g., via a gesture or verbal command recognizable to the wearable appliance) for additional information about the control cabinet, including electrical schematics or line diagrams for the cabinet, ladder logic programming associated with an industrial controller mounted within the cabinet, diagnostic data for any of the devices, etc.)
wherein the executable program logic is configured to access the database system using the spatial position data to identify equipment located within a proximity of the current location of the user; ([0090] the location and orientation component 410 of wearable appliance 206 can be configured to determine a current geographical location of the appliance 206. In some embodiments, location and orientation component 410 can leverage global positioning system (GPS) technology to determine the user's absolute location, or may be configured to exchange data with positioning sensors located within the plant facility in order to determine the user's relative location within the plant.)
wherein the executable program logic is configured to transmit the display data to the augmented reality glasses to cause the augmented reality glasses to present the display data to the user, (Fig. 8-11 [0092] For example, if the user's wearable appliance 206 is currently presenting the view depicted in FIG. 8, and the user moves forward and slightly to the left, the rendering component 308 will transition to the view depicted in FIG. 9. In general, the external view generated by VR/AR presentation system 302 renders the industrial area as a virtual scale model of the area, and allows the user to move around and interact with the scaled version of the area. As the user moves around, toward, or away from the virtual scaled industrial area, the wearable appliance 206 streams updated location and orientation data 606 to the presentation system 302, which updates the presentation data 604 substantially continuously to simulate the effect of walking around a scale model of the production area.)
Schmirler does not expressly disclose but Ha discloses a database system configured to store, for each of a plurality of equipment of an industrial plant that are spatially distributed within the industrial plant, equipment-associated data and spatial location data corresponding to the equipment, the database system further configured to maintain spatial position data representing a current location of the user within the industrial plant; and ([0066] Furthermore, the virtual content database 530 may further include a three-dimensional virtual image of modeling the virtual object constructing the field twin model and a virtual content acquired by converting field sensor data for the virtual object into information.)
wherein the executable program logic is configured to receive the spatial position data from the database system; ([0083] the server system 500 may precisely calculate real-time location information (identification information and location of the field user) of the field user through positioning sensor data received the positioning sensor system 410 through at least one method of a Time Difference of Arrival (TDoA) method for calculating the location of the tag based on a radio signal reach time of the tag received from time-synchronized anchors or a Two-Way Ranging (TWR) method for calculating a Round Trip Time (RTT) and converting the calculated RTT into a distance.)
wherein the executable program logic is configured to retrieve, from the database system, the equipment-associated data corresponding to the identified equipment; ([0120] For example, referring to FIG. 4, an augmented reality application 111 of the computing device 101 may perform image recognition for a real object RO and a marker MK in the real object in the surrounding physical space 10 )
wherein the executable program logic is configured to generate display data based on the retrieved equipment-associated data; and ([0120] and control a virtual content VC1 to be displayed on the view field of the user corresponding to the recognized marker MK)
wherein movement of the user within the industrial plant causes the database system to update the spatial position data and causes the executable program logic to update the accessed equipment-associated data based on the updated spatial position data. ([0232] As such, the monitoring application 311 or 611 may generate and provide a field twin model that updates the real object information and the field user information to the virtual space matched to the actual field physical space 10 in real time and provide a monitoring function to monitor an overall field and check respective detailed information by the remote manager)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Schmirler with the teachings of Ha with a reasonable expectation of success by implementing a work field by a digital twin and monitoring a field based on the implemented digital twin and supports work to field workers as taught by Ha ([0002]).
Regarding claim 2, Schmirler teaches The automation system of claim 1, wherein the automation system further comprises a positioning system configured to generate the spatial position data representing the current location of the user within the industrial plant. ([0090] the location and orientation component 410 of wearable appliance 206 can be configured to determine a current geographical location of the appliance 206. In some embodiments, location and orientation component 410 can leverage global positioning system (GPS) technology to determine the user's absolute location, or may be configured to exchange data with positioning sensors located within the plant facility in order to determine the user's relative location within the plant.)
Regarding claim 3, Schmirler teaches The automation system of claim 2, wherein the positioning system includes a spatial mapping mesh covering at least a portion of the industrial plant, an indoor positioning system, a cellular positioning system, or a GPS receiver configured to determine the spatial position data. ([0090] the location and orientation component 410 of wearable appliance 206 can be configured to determine a current geographical location of the appliance 206. In some embodiments, location and orientation component 410 can leverage global positioning system (GPS) technology to determine the user's absolute location, or may be configured to exchange data with positioning sensors located within the plant facility in order to determine the user's relative location within the plant.)
Regarding claim 4, Schmirler teaches The automation system of claim 3, wherein the spatial mapping mesh comprises spatial anchors,
wherein each spatial anchor relates a coordinate system associated with the augmented reality glasses for display of the display data to a spatial coordinate system associated with the spatial location data corresponding to one or more of the plurality of equipment stored in the database system, and wherein the executable program logic is configured to display the display data corresponding to equipment that is located within the proximity of current location of the user via the augmented reality glasses at predefined positions relative to the spatial anchors. .([0138] one or more of the video capture devices 1414 may be a time-of-flight (TOF) optical scanner or sensor, which generates distance information (e.g., point cloud or depth map information) for objects and surfaces within the scanner's field of view. In such embodiments, monitoring component 316 can be configured to correlate object recognition results with the distance information, and generate a notification directed to a wearable appliance or an industrial controller in response to determining that a result of this correlation satisfies a defined criterion.[0167] VR/AR presentation system 302 can generate this documentation in a three-dimensional format. In some scenarios, the system 302 can collect the data required to generate these plant mappings from multiple wearable appliances 206 associated with multiple users distributed throughout the plant environment, and update the plant documentation as more data is received.)
Schmirler does not expressly disclose but Ha discloses wherein the spatial anchors are interconnected, ([0082] A plurality of anchors arranged at predetermined intervals in the physical space may receive the radio signal of the tag and transmit positioning sensor data for determining the location (e.g., the location of the field user) of the tag to the server system 500 or directly calculate the tag location.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Schmirler with the teachings of Ha with a reasonable expectation of success by implementing a work field by a digital twin and monitoring a field based on the implemented digital twin and supports work to field workers as taught by Ha ([0002]).
Regarding claim 5, Schmirler teaches The automation system of claim 1, wherein the automation system is configured to create an avatar for at least one remote user, the remote user being remote to the industrial plant, wherein the executable program logic is configured to display the avatar in a proximity of the current location of the user wearing the augmented reality glasses via the augmented reality glasses, and wherein the automation system is configured to provide a bidirectional visual and/or acoustic communication channel between the user wearing the augmented reality glasses and the remote user. (Fig. 8 [0050] The system can render a scaled down view of the factory floor area, which affords the user an external overview of the area. This external view can include real-time avatars representing human operators, superimposed production statistics and status data, and other information. [0090] Rendering component 308 can also render human icons 808 a and 808 b representing human operators present on in the production area. Returning briefly to FIG. 7, in some embodiments the locations and orientations of the human icons 808 a and 808 b within the VR/AR presentation can be determined based on location and orientation data 606 received by VR/AR presentation system 302 from the wearable appliances 206 associated with each user)
Regarding claim 7, Schmirler teaches The automation system of claim 5, wherein the automation system comprises a robot with a camera and/or a microphone, the robot being located in the industrial plant, ([0044] FIG. 1 is a block diagram of an example industrial control environment 100 [0045] Industrial devices 120 may include both input devices that provide data relating to the controlled industrial systems to the industrial controllers 118, and output devices that respond to control signals generated by the industrial controllers 118 to control aspects of the industrial systems. Example input devices can include telemetry devices (e.g., temperature sensors, flow meters, level sensors, pressure sensors, etc.), manual operator control devices (e.g., push buttons, selector switches, etc.), safety monitoring devices (e.g., safety mats, safety pull cords, light curtains, etc.), and other such devices. Output devices may include motor drives, pneumatic actuators, signaling devices, robot control inputs, valves, and the like [0051] The presentation system 302 can also be configured to work in conjunction with video capture devices (e.g., 360-degree cameras, webcams, swivel-based IP cameras, etc.) installed at one or more locations within the plant environment.)
wherein the automation system comprises a robot-control module configured to control a movement and/or orientation of the robot such that a position and/or the orientation of the robot reflects a position and/or orientation of the avatar within the augmented reality displayed via the augmented reality glasses, and ([0105] the user may speak a request for a current status of a particular asset (e.g., an industrial robot, a production line, a motor, a stamping press, etc.), which is received by the user's wearable appliance 402 and relayed to the VR/AR presentation system 302.)
wherein the robot-control module is configured to automatically update the position and/or orientation of the avatar and the robot in accordance with a change in a position and/or orientation of the remote user or in response to a control command submitted by the remote user. ([0105] The presentation system 302 can translate the spoken request into a query for the desired information about the specified asset, retrieve the relevant subset of plant data 610, and render the requested information as a VR/AR presentation on the user's wearable appliance 206)
Regarding claim 8, Schmirler teaches The automation system of claim 1, the database system being a graph database system, a spatial database system and/or a streaming database system. ([0005] the augmented reality representation from a virtual view of the industrial facility to a video presentation; and streaming, by the system to the wearable device as the video presentation, a subset of the video data received from a video capture device, of the video capture devices, corresponding to the camera icon.)
Regarding claim 9, Schmirler teaches The automation system of claim 1, further comprising: a distributed control system coupled to the database system, the distributed control system comprising:
a memory configured to store one or more process parameters in association with one or more of the equipment and to store process control software, and
a processor configured to execute the process control software for automatically controlling the production process. ([0197] With reference to FIG. 23, an example environment 2310 for implementing various aspects of the aforementioned subject matter includes a computer 2312. The computer 2312 includes a processing unit 2314, a system memory 2316, and a system bus 2318. The system bus 2318 couples system components including, but not limited to, the system memory 2316 to the processing unit 2314. The processing unit 2314 can be any of various available processors. Multi-core microprocessors and other multiprocessor architectures also can be employed as the processing unit 2314.)
Regarding claim 10, Schmirler teaches The automation system of claim 9, further comprising: one or more sensors coupled to at least one equipment, the one or more sensors configured to measure the one or more process parameters of the at least one equipment, the one or more process parameters indicating a current state of the production process of the industrial plant and/or a current state or mode of operation of the equipment; and ([0045] ndustrial devices 120 may include both input devices that provide data relating to the controlled industrial systems to the industrial controllers 118, and output devices that respond to control signals generated by the industrial controllers 118 to control aspects of the industrial systems. Example input devices can include telemetry devices (e.g., temperature sensors, flow meters, level sensors, pressure sensors, etc.), manual operator control devices (e.g., push buttons, selector switches, etc.), safety monitoring devices (e.g., safety mats, safety pull cords, light curtains, etc.), and other such devices. Output devices may include motor drives, pneumatic actuators, signaling devices, robot control inputs, valves, and the like.)
one or more actuators coupled to the at least one equipment, the one or more actuators configured to control the one or more process parameters or configured to be operated in accordance with one or more control parameters, wherein the distributed control system is coupled to the one or more sensors and the one or more actuators, and wherein the processor is configured to execute the process control software for automatically controlling the production process based on measurement signals received from the one or more sensors and control signals sent to the one or more actuators. ([0194] This can include substantially any type of control, communications module, computer, Input/Output (I/O) device, sensor, actuator, instrumentation, and human machine interface (HMI) that communicate via the network, which includes control, automation, and/or public networks. The PLC or automation controller can also communicate to and control various other devices such as standard or safety-rated I/O modules including analog, digital, programmed/intelligent I/O modules, other programmable controllers, communications modules, sensors, actuators, output devices, and the like.)
Regarding claim 11, Schmirler teaches The automation system of claim 9, further comprising:
a user database coupled to the distributed control system, the user database configured to store user information associated with the production process, wherein the user information comprises user IDs, user roles and/or user privileges. ([0154] Concurrently, device interface component 314 collects user data 1704 that can be used to confirm that the user is performing the steps recommended by the workflow delivered to the user's wearable appliance 206. User data 1704 can include, for example, the user's identity and location relative to the automation system or components thereof. The user's location can be used to confirm that the user is at the appropriate location to perform the workflow step currently awaiting completion (e.g., in front of the appropriate control panel, HMI, machine station, or mechanical/electrical component). User data 1704 can also include data indicating the user's interactions with devices associated with the automation system)
Regarding claim 12, Schmirler teaches The automation system of claim 11, wherein the executable program logic is configured to receive user credentials from the user and to authenticate the user, ([0059] VR/AR presentation system 302 can include…an authentication component 306) wherein a user ID of the user wearing the augmented reality glasses is associated with one or more of the user roles and each user role of the one or more user roles includes one or more of the user privileges, and([0061] Authentication component 306 may also determine a defined role associated with the user identification information, and grant a level of control privilege commensurate with the user's role.)
wherein the executable program logic is further configured to select and display content of the display data for presentation based upon the one or more user roles and/or the one or more user privileges associated with the user ID. ([0062] rendering component 308 can generate presentations based on an identity of an industrial device, automation system, control cabinet, or machine received from the wearable appliance, such that available information about devices, machines, or control cabinets within the user's line of sight is displayed on the appliance. The rendering component 308 can also select the VR/AR presentation in accordance with the user's control privileges (determined by the authentication component 306). The selected presentation can then be sent to the wearable appliance the client interface component 304.)
Regarding claim 13, Schmirler teaches The automation system of claim 12, wherein the one or more user roles includes one or more of a maintenance role and an engineering role, and the one or more user privileges define access to one or more specific equipment of the plurality of equipment. ([0065] Monitoring component 316 can be configured to monitor selected subsets of data collected by device interface component 314 according to defined monitoring rules, and to deliver notifications and/or workflow recommendations in response to detecting a maintenance or performance issue based on a result of the monitoring [0074] Example user roles that can determine how VR and AR data is presented to a user can include, but are not limited to, line operators, maintenance personnel, plant managers, plant engineers, or other roles.)
Regarding claim 14, Schmirler teaches The automation system of claim 1, wherein the equipment- associated data comprises one or more task instructions associated with the equipment. ([0071] The one or more processors 420 can perform one or more of the functions described herein with reference to the systems and/or methods disclosed. Memory 422 can be a computer-readable storage medium storing computer-executable instructions and/or information for performing the functions described herein with reference to the systems and/or methods disclosed.)
Regarding claim 15, Schmirler teaches The automation system of claim 1, wherein one or more equipment of the plurality of equipment are one or more machines. ([0045] Industrial devices 120 may include both input devices that provide data relating to the controlled industrial systems to the industrial controllers 118, and output devices that respond to control signals generated by the industrial controllers 118 to control aspects of the industrial systems. Example input devices can include telemetry devices (e.g., temperature sensors, flow meters, level sensors, pressure sensors, etc.), manual operator control devices (e.g., push buttons, selector switches, etc.), safety monitoring devices (e.g., safety mats, safety pull cords, light curtains, etc.), and other such devices. Output devices may include motor drives, pneumatic actuators, signaling devices, robot control inputs, valves, and the like)
Regarding claim 16, Schmirler teaches The automation system of claim 1, wherein the executable program logic is configured to perform a query of the database system based upon the data for retrieving, from the database system, the equipment- associated data corresponding to equipment that are located within the proximity of the current location of the user. ([0105] In addition to presenting asset data to the user in response to determining that the user's location and orientation places the asset within the user's line of sight, some embodiments of VR/AR presentation system can also process natural language spoken queries requesting specified information about an industrial asset, regardless of whether the user is currently viewing the asset. For example, the user may speak a request for a current status of a particular asset (e.g., an industrial robot, a production line, a motor, a stamping press, etc.), which is received by the user's wearable appliance 402 and relayed to the VR/AR presentation system 302. The presentation system 302 can translate the spoken request into a query for the desired information about the specified asset, retrieve the relevant subset of plant data 610, and render the requested information as a VR/AR presentation on the user's wearable appliance 206.)
Regarding claim 17, Schmirler teaches The automation system of claim 1, wherein the database system is accessible via a subscription-based streaming service, and wherein the executable program logic is configured to receive, from the database system via the subscription-based streaming service, the equipment-associated data corresponding to equipment that are located within the proximity of the current location of the user based on the spatial position data. ([0134] In response to receiving the request, rendering component 308 retrieves the stored video data corresponding to the identified video capture device 1414 and the indicated date and time, and can begin streaming the retrieved video data to the wearable appliance 206. Similar to the live video feeds, the user can interact with the historical video feed by moving his or her head to the left or right to change the perspective or line of site.)
Regarding claim 18, Schmirler teaches The automation system of claim 17, wherein the database system is configured to provide, in response to the query of the executable program logic, selectively the equipment-associated data corresponding to equipment that are located proximate to the current location of the user wearing the augmented reality glasses, whereby movement of the user relative to the equipment triggers the executable program logic to submit a new query to the database system comprising updated spatial position data representing the current location of the user; ([0105] In addition to presenting asset data to the user in response to determining that the user's location and orientation places the asset within the user's line of sight, some embodiments of VR/AR presentation system can also process natural language spoken queries requesting specified information about an industrial asset, regardless of whether the user is currently viewing the asset. For example, the user may speak a request for a current status of a particular asset (e.g., an industrial robot, a production line, a motor, a stamping press, etc.), which is received by the user's wearable appliance 402 and relayed to the VR/AR presentation system 302. The presentation system 302 can translate the spoken request into a query for the desired information about the specified asset, retrieve the relevant subset of plant data 610, and render the requested information as a VR/AR presentation on the user's wearable appliance 206.) or
wherein the database system is configured to provide, via the subscription-based streaming service, selectively the equipment-associated data corresponding to equipment that are located proximate to the current location of the user wearing the augmented reality glasses, whereby movement of the user relative to the equipment triggers the executable program logic to update the subscription-based streaming service with updated spatial position data representing the current location of the user.
Regarding claim 19, Schmirler teaches The automation system of claim 1, wherein the executable program logic is configured for:
receiving coordinates of visual objects graphically representing the equipment- associated data corresponding to the identified equipment proximate to the user wearing the augmented reality glasses; ([0055] In response to various conditions, such as the user's determined role, location, line of sight, or other information, the system can generate and deliver augmented or virtual reality presentations to the user's wearable appliance 206)
determining the user's line of sight in a real-world coordinate system; ([0055] The VR/AR presentation system can customize the presentations 204 based on a user's current context, line of sight, type of client device being used by the user (e.g., wearable computer, handheld device, etc.),
receiving, from the database system, coordinates of real-world objects which are proximate to the current location of the user wearing the augmented reality glasses; ([0090] the location and orientation component 410 of wearable appliance 206 can be configured to determine a current geographical location of the appliance 206. In some embodiments, location and orientation component 410 can leverage global positioning system (GPS) technology to determine the user's absolute location, or may be configured to exchange data with positioning sensors located within the plant facility in order to determine the user's relative location within the plant.)
determining that one or more of the visual objects graphically representing the equipment-associated data corresponding to the identified equipment are positioned along the user's line of sight and are farther away from the user than at least one of the real-world objects which is also positioned along the user's line of sight; and ([0056] as a user is viewing an automation system, machine, or industrial device through a wearable computer (or as a substantially real-time video image rendered on the user's client device), the VR/AR presentation system can monitor the wearable computer to determine the user's location relative to the automation system, the user's current line of sight or field of view, and/or other contextual information indicative of the user's relationship to the automation system)
preventing display of the one or more visual objects via the augmented reality glasses based on the one or more visual objects being positioned along the user's line of sight and being farther away from the user than the at least one real-world object. ([0103] In some embodiments, rendering component 308 can be configured to display a minimal amount of information about the cabinet 1102 (or other machine or industrial device) in response to determining that the cabinet 1102 is within the user's current line of sight, and display additional information about the cabinet in response to a user gesture or verbal command (e.g., a natural language spoken command) indicating a request for more detailed data)
Regarding claim 20, Schmirler teaches The automation system of claim 1, wherein the database system is a graph database system, the graph database system including a graph database having a plurality of interconnected nodes, wherein each node represents one or more objects of the industrial plant and includes spatial information associated with the one or more objects, and wherein the one or more objects are selected from the group consisting of a sensor, an actuator, raw material, and an equipment of the plurality of equipment, the equipment being one of a machine and a robot. ([0005] the augmented reality representation from a virtual view of the industrial facility to a video presentation; and streaming, by the system to the wearable device as the video presentation, a subset of the video data received from a video capture device, of the video capture devices, corresponding to the camera icon. [0203] Computer 2312 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 2344. The remote computer(s) 2344 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device or other common network node and the like, and typically includes many or all of the elements described relative to computer 2312. )
Regarding claim 21, Schmirler teaches The automation system of claim 20, wherein one or more of the nodes is associated with at least one respective container, each container being an isolated runtime environment instance comprising software for monitoring and/or operating at least one of the one or more objects represented by the node, and wherein the software in each container comprises a position reporting module configured to monitor a position of at least one of the one or more objects of the respective node and to update the spatial information associated with the at least one object of the respective node in the database system. ([0005] the augmented reality representation from a virtual view of the industrial facility to a video presentation; and streaming, by the system to the wearable device as the video presentation, a subset of the video data received from a video capture device, of the video capture devices, corresponding to the camera icon. [0179] the perspective or angle of view of the scaled facility will change in coordination with the user's position and orientation to simulate walking around a physical scale model of the facility. [0203] Computer 2312 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 2344. The remote computer(s) 2344 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device or other common network node and the like, and typically includes many or all of the elements described relative to computer 2312. )
Regarding claim 22, Schmirler teaches The automation system of claim 21, wherein the spatial information associated with the at least one object of the respective node includes one or more spatial coordinates of the at least one object, the one or more spatial coordinates corresponding to one or more points on a bounding surface of the object and/or inside the bounding surface of the object. ([0138] ne or more of the video capture devices 1414 may be a time-of-flight (TOF) optical scanner or sensor, which generates distance information (e.g., point cloud or depth map information) for objects and surfaces within the scanner's field of view. In such embodiments, monitoring component 316 can be configured to correlate object recognition results with the distance information, and generate a notification directed to a wearable appliance or an industrial controller in response to determining that a result of this correlation satisfies a defined criterion.)
Regarding claim 23, Schmirler teaches The automation system of claim 1, further comprising a network communicatively coupling the database system to the augmented reality glasses. ([0072] One or both of office network 108 or plant network 116 may also have access to external networks 514 such as the Internet (e.g., via firewall device 516).)
Regarding claim 24, Schmirler teaches A method of using an automation system, the method comprising: ([0002] The subject matter disclosed herein relates generally to industrial automation systems, and, more particularly, to visualization of industrial data)
providing augmented reality glasses wearable by a user and configured to present visual information to the user; ([0054] FIG. 2 is a conceptual diagram illustrating presentation of augmented or virtual reality presentations 204 to a wearable appliance 206 or computing device worn by a user.)
receiving, by executable program logic communicatively coupled to the database system and the augmented reality glasses, the spatial position data from the database system; ([0110] the user can send a request to present system 302 (e.g., via a gesture or verbal command recognizable to the wearable appliance) for additional information about the control cabinet, including electrical schematics or line diagrams for the cabinet, ladder logic programming associated with an industrial controller mounted within the cabinet, diagnostic data for any of the devices, etc.)
accessing, by the executable program logic, the database system using the spatial position data to identify equipment located within a proximity of the current location of the user; ([0090] the location and orientation component 410 of wearable appliance 206 can be configured to determine a current geographical location of the appliance 206. In some embodiments, location and orientation component 410 can leverage global positioning system (GPS) technology to determine the user's absolute location, or may be configured to exchange data with positioning sensors located within the plant facility in order to determine the user's relative location within the plant.)
transmitting, by the executable program logic, the display data to the augmented reality glasses to cause the augmented reality glasses to present the display data to the user; and (Fig. 8-11 [0092] For example, if the user's wearable appliance 206 is currently presenting the view depicted in FIG. 8, and the user moves forward and slightly to the left, the rendering component 308 will transition to the view depicted in FIG. 9. In general, the external view generated by VR/AR presentation system 302 renders the industrial area as a virtual scale model of the area, and allows the user to move around and interact with the scaled version of the area. As the user moves around, toward, or away from the virtual scaled industrial area, the wearable appliance 206 streams updated location and orientation data 606 to the presentation system 302, which updates the presentation data 604 substantially continuously to simulate the effect of walking around a scale model of the production area.)
Schmirler does not expressly disclose but Ha discloses storing, by a database system, for each of a plurality of equipment of an industrial plant that are spatially distributed within the industrial plant, equipment-associated data and spatial location data corresponding to the equipment; ([0066] Furthermore, the virtual content database 530 may further include a three-dimensional virtual image of modeling the virtual object constructing the field twin model and a virtual content acquired by converting field sensor data for the virtual object into information.)
maintaining, by the database system, spatial position data representing a current location of the user within the industrial plant; ([0083] the server system 500 may precisely calculate real-time location information (identification information and location of the field user) of the field user through positioning sensor data received the positioning sensor system 410 through at least one method of a Time Difference of Arrival (TDoA) method for calculating the location of the tag based on a radio signal reach time of the tag received from time-synchronized anchors or a Two-Way Ranging (TWR) method for calculating a Round Trip Time (RTT) and converting the calculated RTT into a distance.)
retrieving, by the executable program logic from the database system, the equipment- associated data corresponding to the identified equipment; ([0120] For example, referring to FIG. 4, an augmented reality application 111 of the computing device 101 may perform image recognition for a real object RO and a marker MK in the real object in the surrounding physical space 10 )
generating, by the executable program logic, display data based on the retrieved equipment-associated data; ([0120] and control a virtual content VC1 to be displayed on the view field of the user corresponding to the recognized marker MK)
updating, in response to movement of the user within the industrial plant, the spatial position data by the database system and updating, by the executable program logic, the accessed equipment-associated data based on the updated spatial position data. ([0232] As such, the monitoring application 311 or 611 may generate and provide a field twin model that updates the real object information and the field user information to the virtual space matched to the actual field physical space 10 in real time and provide a monitoring function to monitor an overall field and check respective detailed information by the remote manager)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Schmirler with the teachings of Ha with a reasonable expectation of success by implementing a work field by a digital twin and monitoring a field based on the implemented digital twin and supports work to field workers as taught by Ha ([0002]).
Regarding claim 25, Schmirler teaches The automation system of claim 1, wherein the database system is configured to receive the spatial position data representing the current location of the user ([0090] the location and orientation component 410 of wearable appliance 206 can be configured to determine a current geographical location of the appliance 206. In some embodiments, location and orientation component 410 can leverage global positioning system (GPS) technology to determine the user's absolute location, or may be configured to exchange data with positioning sensors located within the plant facility in order to determine the user's relative location within the plant.)
Schmirler does not expressly disclose but Ha discloses and to selectively provide the equipment-associated data based on a comparison of the spatial position data of the user with the spatial location data corresponding to the equipment. ([0066] Furthermore, the virtual content database 530 may further include a three-dimensional virtual image of modeling the virtual object constructing the field twin model and a virtual content acquired by converting field sensor data for the virtual object into information.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Schmirler with the teachings of Ha with a reasonable expectation of success by implementing a work field by a digital twin and monitoring a field based on the implemented digital twin and supports work to field workers as taught by Ha ([0002]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Schmirler (US 20200336706 A1) in view of Ha (US 20210201584 A1) in further view of Miller (US 20150302657 A1).
Regarding claim 6, Schmirler teaches The automation system of claim 5, wherein the augmented reality glasses comprise an acoustic output interface, wherein the automation system is configured to provide a bidirectional acoustic communication channel between the user wearing the augmented reality glasses and the avatar, and wherein the executable program logic is configured to control the acoustic output interface within the coordinate system associated with the augmented reality glasses for display of the display data. (Fig. 8-10 [0090] Rendering component 308 can also render human icons 808 a and 808 b representing human operators present on in the production area. Returning briefly to FIG. 7, in some embodiments the locations and orientations of the human icons 808 a and 808 b within the VR/AR presentation can be determined based on location and orientation data 606 received by VR/AR presentation system 302 from the wearable appliances 206 associated with each user [0158] Since the wearable appliances 206 support audio communication, the users can exchange verbal communication via the wearable appliances 206 while sharing views in order to facilitate coordination of activities between the users.)
Schmirler does not expressly disclose but Miller discloses such that the volume of the voice of the avatar output via the acoustic output interface to the user wearing the augmented reality glasses negatively correlates with a distance of the user wearing the glasses and the avatar([0158] the one or more parameters pertains to an intensity of the sound. [0227] As illustrated in FIG. 3, the audio subsystem 106 may take a variety of forms. For instance, the audio subsystem 106 may take the form of a simple two speaker 2 channel stereo system, or a more complex multiple speaker system (5.1, 7.1, 12.1 channels). In some implementations, the audio subsystem 106 may be operable to produce a three-dimensional sound field. [0576] Voice may be passed through to appear to be emanating from the avatar.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Schmirler with the teachings of Miller with a reasonable expectation of success by facilitating virtual and/or augmented reality interaction for one or more users as taught by Miller ([0007]).
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Schmirler (US 20200336706 A1) in view of Ha (US 20210201584 A1) in further view of Mehrotra (US 20210149889 A1).
Regarding claim 26, Schmirler does not expressly disclose but Mehrotra discloses The automation system of claim 1, wherein the executable program logic is configured to generate, for display by the augmented reality glasses, a first graphical user interface when the user is assigned a first role and the identified equipment is within the proximity of the current location of the user, ([0093] Returning to FIG. 9, the button 164C may cause the operator device 54 to generate a GUI (e.g., an application GUI) that outlines tools to be used when completing the AR UX workflow instructions. For example, FIG. 11 illustrates an EN application GUI 132G used by the operator device 54 to display a tool listing 186 corresponding to AR UX workflow instructions for the virtual product 166. Returning to FIG. 9, the button 164D may cause the operator device 54 to generate a GUI that guides the operator 50 to perform the desired experience. The button 164D may be similar to a “continue” button or operation, where the operator device 54 waits to begin showing an AR UX workflow visualization of a first instruction until provided indication that the device to be operated on is in a no-power state or is otherwise ready to be operated on) and a second graphical user interface when the user is assigned a second role and the identified equipment is within the proximity of the current location of the user ([0094] FIG. 12 illustrates an EN application GUI 132H used by the operator device 54 to display the AR UX workflow visualization for the virtual product 166. The EN application GUI 132H may include a selector 198 (e.g., a drop-down menu) that presents a list of operations associated with the AR UX workflow instructions to the operator 50 when selected. The list of operations indicated via the selector 198 may correspond to a list of operations specifically illustrated (e.g., highlighted, called-out) via the AR UX workflow visualizations. To start the rendering of AR UX workflow visualizations corresponding to the AR UX workflow instructions, the operator 50 may interact with a button 200. In response to receiving an input at the button 200, the operator device 54 may render a first AR UX workflow visualization), wherein the first graphical user interface and the second graphical user interface display different information or enable different actions for maintaining and/or controlling the identified equipment. (Fig. 11 remove front screen, Fig. 12 lower fan replacement)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention to modify Schmirler with the teachings of Mehrotra with a reasonable expectation of success by displaying or modifying visualizations associated with an industrial automation device or an industrial system based on detected user input as taught by Mehrotra ([0001]).
Response to Arguments
Applicants arguments filed 6/2/2026 have been fully considered as follows:
Applicant argues that the 35 USC 112 interpretation to the claims should not be maintained in view of “Applicants respectfully submit that the claims, as amended, recite sufficient structure and should not be interpreted under 35 U.S.C. §112(f).” This argument is persuasive. Therefore, the rejection is not maintained.
Applicant argues that the 35 USC 103 rejections to the claims should not be maintained in view of “the applied references cited in the Office Action, Schmirler et al. (US 2020/0336706 Al) and Miller (US 2015/0302657 Al), alone or in combination, fail to disclose, teach, or suggest the claimed system architecture, including the database-centered use of spatial position data to identify proximate equipment, retrieve corresponding equipment-associated data, and update the accessed equipment-associated data based on movement of the user within the industrial plant, as recited in amended claims 1 and 24, or the database-based comparison and role-specific graphical user interface features recited in new claims 25 and 26, which provide the advantages discussed above”. However, in view of the amendment a new ground of rejection is above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH TRAN whose telephone number is (313)446-6642. The examiner can normally be reached 8am-5pm M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at (571) 272-6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.A.T./Examiner, Art Unit 3656 /KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656
Prosecution Insights