Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
Drawings have been reviewed and accepted.
Specification
The specification filed on 09/16/24 has been entered. Specification has been reviewed and accepted.
Information Disclosure Statement
The information disclosure statement (IDS) submitted filed on 11/13/24 has been received. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4, 9, and 14-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Schmirler et al. (US20180131907, herein Schmirler).
Regarding claim 1, Schmirler teaches An automation system for controlling an automated production process of an industrial plant, the industrial plant comprising a plurality of equipment for performing the production process ([0043] Industrial controllers and their associated I/O devices are central to the operation of modern automation systems. These controllers interact with field devices on the plant floor to control automated processes relating to such objectives as product manufacture, material handling, batch processing, supervisory control, and other such applications), the plurality of equipment being spatially distributed in the industrial plant (Fig. 1, Fig. 8, [0056] determine current status information for devices and/or machines that make up the automation system), each equipment of the plurality of equipment associated with one or more corresponding virtual objects ([0051] the presentation system can enhance or augment this live view with superimposed operational or status data positioned on or near the view representations of relevant machines or devices, thereby yielding an augmented reality view of the environment), the automation system comprising: an augmented reality system, comprising: augmented reality glasses; and an executable program logic coupled to the augmented reality glasses (Fig. 7, [0005] rendering, by the system on a wearable appliance based on the industrial data, an augmented reality representation of the industrial facility, wherein the rendering comprises rendering camera icons at locations on the augmented reality representation corresponding to the locations within the industrial facility), the executable program logic configured to: receive a spatial position of a user wearing the augmented reality glasses from a positioning system ([0050] The holographic views can be delivered to a wearable visualization computer, which renders the 3D view as a function of the user's current location and/or orientation), display, via the augmented reality glasses, one or more virtual objects associated with one or more equipment proximate to the spatial position of the user, wherein at least one of the one or more-virtual objects is linked to at least one respective secondary virtual object ([0106] determines a suitable subset of relevant information to display on the user's wearable appliance—as well as a location within the presentation at which the data is to be displayed—based on the user's current location and orientation information (as obtained via location and orientation data 606), as well as a determination of the user's current virtual location within the simulated plant environment. For example, if the user is determined to be at a virtual location in front of a virtual control cabinet, and is oriented such that the control cabinet is within the user's line of sight, rendering component 308 will retrieve and render information relevant to the cabinet and its associated devices at a location within the presentation on or near the virtual cabinet. In some embodiments, rendering component 308 can determine which industrial assets are within the viewer's line of sight based on a correlation of the user's current location and orientation (as determined from the location and orientation data 606) with known locations of the industrial assets making up the industrial facility, which may be defined by plant model(s) 524, [0098] In FIGS. 8-10, asset information icon 810 a has been placed near a control cabinet 902, and another asset information icon 810 b has been placed near machine 904), the at least one respective secondary virtual object stored in a database system ([0072] The example industrial environment depicted in FIG. 5 includes one or more industrial controllers 504, HMIs 506, motor drives 518, industrial safety systems 520, databases 508 (e.g., data historians, employee databases, inventory databases, etc.)), receive, from the augmented reality glasses, a signal indicating a hands-free selection, by the user, of a virtual object linked to a respective secondary virtual object, and display, via the augmented reality glasses, the respective secondary virtual object and/or trigger an action associated with the selected virtual object ([0115] the user's line of sight is directed to the industrial controller, a control cabinet in which the controller resides, or a machine being controlled by the industrial controller 1204, the user can perform a gesture or speak a verbal command recognizable by the wearable appliance 206 indicating that the user has selected the industrial controller 1204 (or its associated machine) as a target for a control instruction. In response to the gesture or verbal command, wearable appliance 206 sends the identity of the target device to the VR/AR presentation system as selection data 1202. In some embodiments, selection of the industrial controller 1204 or its associated machine can cause rendering component 308 to render on the VR/AR presentation a predefined list of available commands that can be issued for the selected machine. Example commands can include, for example, machine start/stop commands, switch setting adjustments, setpoint adjustments, alarm reset commands, or other such comments, [0128] thereby providing the user with a preview of the video available at that location. While viewing the VR/AR presentation, the user can select one of the camera icons 806 using an appropriate selection gesture or verbal command in order to transition from the current external view of the production area to a live interactive video presentation rendered on the wearable appliance 206. With reference to FIG. 14A, in response to the selection gesture or verbal command, system interface component 404 of wearable appliance 206 sends camera selection data 1402 to the VR/AR presentation system 302).
Regarding claim 4, Schmirler teaches The automation system of claim 1, wherein the executable program logic is further configured to: dynamically assign a colour to each virtual object of the at least one virtual object linked to the at least one respective secondary virtual object, and display each virtual object of the at least one virtual object linked to the at least one respective secondary virtual object with the dynamically assigned colour ([0085] For each industrial asset, the plant model can define physical dimensions and colors for the asset, as well as any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.). The plant models 524 also define the physical relationships between the industrial assets, including relative positions and orientations of the assets on the plant floor, conduit or plumbing that runs between the assets, and other physical definitions, [0086] generate the presentation such that items of the plant data 610 are overlaid on or near graphical representations of the industrial assets to which the items of data relate)
Regarding claim 9, Schmirler teaches The automation system of claim 4, wherein the hands-free selection comprises: a voice selection of the virtual object, and wherein the augmented reality glasses and/or the positioning system comprises a voice detection unit, the unit configured to: detect the voice selection: and convert the voice selection into the signal indicating the hands-free selection, by the user, of the virtual object ([0095] If the location corresponds to a selectable icon (such as the operator information icon 804), the icon will be selected and an appropriate action performed. Icons can also be selected using verbal commands in some embodiments, [0098] . These asset information icons 810 can be selected using similar gesture or verbal recognition techniques used to select operator information icons 804. Selection of an asset information icon 810 can cause rendering component 308 to render an information window on or near the asset corresponding to the icon 810, and populate this information window with information relevant to the asset, [0105] the user may speak a request for a current status of a particular asset (e.g., an industrial robot, a production line, a motor, a stamping press, etc.), which is received by the user's wearable appliance 402 and relayed to the VR/AR presentation system 302. The presentation system 302 can translate the spoken request into a query for the desired information about the specified asset, retrieve the relevant subset of plant data 610, and render the requested information as a VR/AR presentation on the user's wearable appliance 206).
Regarding claim 14, Schmirler teaches The automation system of claim 1, wherein the executable logic is configured to: display, via the augmented reality glasses, the respective secondary virtual object by replacing the displayed virtual object linked to the secondary virtual object with the secondary virtual object ([0128] While viewing the VR/AR presentation, the user can select one of the camera icons 806 using an appropriate selection gesture or verbal command in order to transition from the current external view of the production area to a live interactive video presentation rendered on the wearable appliance 206. With reference to FIG. 14A, in response to the selection gesture or verbal command, system interface component 404 of wearable appliance 206 sends camera selection data 1402 to the VR/AR presentation system 302. The camera selection data 1402 indicates the identity of the video capture device 1414 selected by the user. As shown in FIG. 14B, in response to receipt of the selection data 1402, rendering component 308 transitions the user's current VR/AR presentation view to a live video presentation 1408 supplied by the video capture device 1414 corresponding to the selected camera icon 806. The video presentation 1408 is generated based on camera-specific video data 1410 retrieved from video storage 1404, which is the subset of received video data 1412 corresponding to the selected video capture device 1414. In some implementations, video and audio from the selected video capture device 1414 may be sent by the presentation system 302 to the wearable appliance 206 via two separate communication channels or networks in order to satisfy bandwidth requirements on a given network, [0174] At 1912, a determination is made as to whether selection input is received from the wearable appliance identifying a selected camera icon of the one or more camera icons. If no such selection input is received (NO at step 1912), the methodology returns to step 1902 and steps 1902-1912 repeat. Alternatively, if such selection input is received (YES at step 1912), the methodology proceeds to step 1914, where the rendering of the VR/AR presentation on the wearable appliance is zoomed from a current virtual view of the industrial facility toward the camera icon identified by the selection input received at step 1912, [0129] the camera that is within the user's line of sight can be changed when the user moves his or her head, simulating the real-world view the user would experience if the user were at the camera location, [0175] the methodology proceeds to step 1918, were the rendering on the wearable appliance is changed from the VR/AR presentation to a video stream presentation comprising a subset of the video data received at step 1910 from the 360-degree video camera corresponding to the camera icon selected at step 1912).
Regarding claim 15, Schmirler teaches The automation system of claim 1, wherein at least one of the virtual objects is a graphical representation of a respective one of the equipment associated with the corresponding virtual object, the graphical representation comprises: an image-based illustration, and/or a video-based illustration of the one equipment ([0085] a plant model for a given industrial area (e.g., a production area, a workcell, an assembly line, etc.) can define graphical representations of the industrial assets—including machines, conveyors, control cabinets, and/or industrial devices—located within that area, as well as the physical relationships between these industrial assets. For each industrial asset, the plant model can define physical dimensions and colors for the asset, as well as any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.). The plant models 524 also define the physical relationships between the industrial assets, including relative positions and orientations of the assets on the plant floor, conduit or plumbing that runs between the assets, and other physical definitions, [0098] Rendering component 308 can also superimpose asset information icons 810 (e.g., 810 a and 810 b) on or near representations of industrial assets for which additional information is available. In FIGS. 8-10, asset information icon 810 a has been placed near a control cabinet 902, and another asset information icon 810 b has been placed near machine 904. These asset information icons 810 can be selected using similar gesture or verbal recognition techniques used to select operator information icons 804, [0128] instead of or in addition to the embedded camera shape, the spherical camera icons 806 may contain a still or live video image obtained for the corresponding video capture device 1414, thereby providing the user with a preview of the video available at that location).
Regarding claim 16, Schmirler teaches The automation system of claim 1, wherein the one or more equipment proximate to the spatial position of the user comprise: equipment in the field of view of the user wearing the AR glasses ([0055] In response to various conditions, such as the user's determined role, location, line of sight, or other information, the system can generate and deliver augmented or virtual reality presentations to the user's wearable appliance 206. Data used to populate the presentations 204 can be obtained by the VR/AR presentation system from the relevant industrial devices and delivered as part of the VR/AR presentations 204, [0056] the VR/AR presentation system can monitor the wearable computer to determine the user's location relative to the automation system, the user's current line of sight or field of view, and/or other contextual information indicative of the user's relationship to the automation system. Based on the determined identity of the automation system currently being viewed by the user, the VR/AR presentation system can determine current status information for devices and/or machines that make up the automation system, or for a process being carried out by the automation system, [0122] in some embodiments the identity of the device or system can be determined by VR/AR presentation system 302 based on the user's location relative to the device or system, as well as the determined orientation of the user's wearable appliance 206. For example, based on location and orientation data 606, presentation system 302 can infer the user's current line of sight. By cross-referencing this location and orientation information with known location information for devices and systems within the plant environment, presentation system 302 can infer which devices or systems are currently within the user's field of view).
Regarding claim 17, Schmirler teaches The automation system of claim 1, wherein the virtual objects represent data enabling the user to operate and/or maintain the plurality of equipment ([0056] if the user's current view encompasses a real or virtualized motor-driven conveyor and a motor drive that controls the motor, the presentation system may superimpose a current operating status of the motor drive (e.g., a current speed, a fault condition, an operating mode, etc.) near the image or view of the motor drive as perceived by the user. If the user is currently viewing a die-cast furnace, the presentation system may superimpose a current furnace temperature near the view of the furnace, [0111] a bank of motor drives may generate and store a variety of operational and diagnostic data (e.g., motor speed data, motor current data, alarm data, etc.). When a user is viewing the bank of motor drives (or a virtual representation of the bank of motor drives) via the wearable appliance, the user may request—via a gesture or verbal command recognizable to the wearable appliance 206—a view that identifies which of the motor drives requires a fan replacement (e.g., based on a corresponding alarm that is active on the drives) .
Regarding claim 18, Schmirler teaches A method for controlling an automation system for controlling an automated production process of an industrial plant, the industrial plant comprising a plurality of equipment for performing the production process ([0043] Industrial controllers and their associated I/O devices are central to the operation of modern automation systems. These controllers interact with field devices on the plant floor to control automated processes relating to such objectives as product manufacture, material handling, batch processing, supervisory control, and other such applications), the plurality of equipment being spatially distributed in the industrial plant (Fig. 1, Fig. 8, [0056] determine current status information for devices and/or machines that make up the automation system), each equipment of the plurality of equipment associated with one or more corresponding virtual objects ([0051] the presentation system can enhance or augment this live view with superimposed operational or status data positioned on or near the view representations of relevant machines or devices, thereby yielding an augmented reality view of the environment), the automation system comprising; an augmented reality system comprising:) augmented reality glasses, and an executable program logic coupled to the augmented reality glasses (Fig. 7, [0005] rendering, by the system on a wearable appliance based on the industrial data, an augmented reality representation of the industrial facility, wherein the rendering comprises rendering camera icons at locations on the augmented reality representation corresponding to the locations within the industrial facility), the method comprising: receiving, by the executable program logic, a spatial position of a user wearing the augmented reality glasses from a positioning system ([0050] The holographic views can be delivered to a wearable visualization computer, which renders the 3D view as a function of the user's current location and/or orientation), displaying, via the augmented reality glasses, one or more virtual objects associated with one or more equipment proximate to the spatial position of the user, wherein at least one of the one or more-virtual objects are linked to at least one respective secondary virtual object ([0106] determines a suitable subset of relevant information to display on the user's wearable appliance—as well as a location within the presentation at which the data is to be displayed—based on the user's current location and orientation information (as obtained via location and orientation data 606), as well as a determination of the user's current virtual location within the simulated plant environment. For example, if the user is determined to be at a virtual location in front of a virtual control cabinet, and is oriented such that the control cabinet is within the user's line of sight, rendering component 308 will retrieve and render information relevant to the cabinet and its associated devices at a location within the presentation on or near the virtual cabinet. In some embodiments, rendering component 308 can determine which industrial assets are within the viewer's line of sight based on a correlation of the user's current location and orientation (as determined from the location and orientation data 606) with known locations of the industrial assets making up the industrial facility, which may be defined by plant model(s) 524, [0098] In FIGS. 8-10, asset information icon 810 a has been placed near a control cabinet 902, and another asset information icon 810 b has been placed near machine 904), the at least one respective secondary virtual object stored in a database system ([0072] The example industrial environment depicted in FIG. 5 includes one or more industrial controllers 504, HMIs 506, motor drives 518, industrial safety systems 520, databases 508 (e.g., data historians, employee databases, inventory databases, etc.)), receiving, from the augmented reality glasses, a signal indicating a hands-free selection, by the user, of a virtual object linked to a respective secondary virtual object, and displaying, via the augmented reality glasses, the respective secondary virtual object, and/or triggering an action associated with the selected virtual object ([0115] the user's line of sight is directed to the industrial controller, a control cabinet in which the controller resides, or a machine being controlled by the industrial controller 1204, the user can perform a gesture or speak a verbal command recognizable by the wearable appliance 206 indicating that the user has selected the industrial controller 1204 (or its associated machine) as a target for a control instruction. In response to the gesture or verbal command, wearable appliance 206 sends the identity of the target device to the VR/AR presentation system as selection data 1202. In some embodiments, selection of the industrial controller 1204 or its associated machine can cause rendering component 308 to render on the VR/AR presentation a predefined list of available commands that can be issued for the selected machine. Example commands can include, for example, machine start/stop commands, switch setting adjustments, setpoint adjustments, alarm reset commands, or other such comments, [0128] thereby providing the user with a preview of the video available at that location. While viewing the VR/AR presentation, the user can select one of the camera icons 806 using an appropriate selection gesture or verbal command in order to transition from the current external view of the production area to a live interactive video presentation rendered on the wearable appliance 206. With reference to FIG. 14A, in response to the selection gesture or verbal command, system interface component 404 of wearable appliance 206 sends camera selection data 1402 to the VR/AR presentation system 302).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 5-8, 12, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Schmirler et al. (US20180131907A1, herein Schmirler), in view of Weber et al. (US20160195924, herein Weber).
Regarding claim 2, Schmirler teaches The automation system of claim 1, wherein the executable program logic is further configured to: determine a number of the at least one virtual object linked to the at least one respective secondary virtual object and associated with one or more equipment proximate to the spatial position of the user ([0056] Based on the determined identity of the automation system currently being viewed by the user, the VR/AR presentation system can determine current status information for devices and/or machines that make up the automation system, or for a process being carried out by the automation system. The VR/AR presentation system can then generate augmented reality or virtual reality presentations and deliver these presentations to the user's wearable appliance; e.g., as graphical or text-based indicators overlaid on the user's field of view, such that each indicator is positioned near the machine or device to which the indicator pertains, [0099] The VR/AR presentation of the production area also includes a number of camera icons 806 (e.g., 806 a-806 d) that allow the user to switch the presentation to a live or historical video feed, as will be described in more detail below), dynamically assign a distinct colour from a predefined colour set of colours to each virtual object linked to respective secondary virtual objects and display each virtual object linked to respective secondary virtual objects with the dynamically assigned distinct colour ([0085] For each industrial asset, the plant model can define physical dimensions and colors for the asset, as well as any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.). The plant models 524 also define the physical relationships between the industrial assets, including relative positions and orientations of the assets on the plant floor, conduit or plumbing that runs between the assets, and other physical definitions, [0086] A rendering engine supported by rendering component 308 is configured to generate an interactive VR/AR presentation of the industrial area based on the industrial asset rendering definitions specified in the plant models. Rendering component 308 populates this virtual reality presentation with selected subsets of collected plant data 610 (as well as production or operational statistics calculated by reporting component 310 based on subsets of the plant data 610), and client interface component 304 delivers the resulting aggregate VR/AR presentation to wearable appliance 206 as VR/AR presentation data 604)
Schmirler does not teach and if the number of virtual objects linked to respective secondary virtual objects is greater than a number of colours of the predefined colour set, then display only those virtual objects linked to respective secondary virtual objects that are assigned distinct colours
Weber teaches and if the number of virtual objects linked to respective secondary virtual objects is greater than a number of colours of the predefined colour set, then display only those virtual objects linked to respective secondary virtual objects that are assigned distinct colours ([0124] if all visual identifiers (e.g. all colours in this example) in the predefined set have already been assigned, analysis is done to gather a neighbour set of activatables at step 101. In particular, for each visual identifier (i) in the predefined set the activatable neighbour (i,a) with visual identifier i closest to the current activatable a is located and stored in the neighbour set. The farthest activatable neighbour in the neighbour set from the unassigned current activatable a is then identified at step 102 and the visual identifier of that farthest activatable neighbour is assigned to the unassigned current activatable a, before the algorithm returns to decision 96 to assign visual identifiers to the next unassigned activatables, if any, [0132] the predetermined activation zone may be configured to be smaller than the assignment zone used in the assignment step, such that the activation is not likely to capture two activatables having the same assigned colour. However, if two or more activatables having the same assigned colour do happen to fall within the activation zone, then the activatable closest to the last gaze position sample is selected to be associated with its corresponding activated confirm button)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Schmirler’s teaching of virtual reality for industrial automation using a plant model defining physical dimensions and colors of assets with Weber’s teaching of a gaze controlled interface having predefined assigned colors . The combined teaching provides an expected result of virtual reality for industrial automation using a plant model defining physical dimensions and having predefined assigned colors. Therefore, one of ordinary skill in the art would be motivated to improve the usability by allowing the user to distinguish and select data more accurately.
Regarding claim 5, the combination of Schmirler and Weber teach The automation system of claim 2, wherein each virtual object of the at least one virtual object linked to the at least one secondary virtual object represents data of a data type, and wherein the executable program logic is configured to: dynamically assign the colour to each virtual object of the at least one virtual object linked to the at least one respective secondary virtual object based on the data type and/or equipment associated with each virtual object (Schmirler, [0085] For each industrial asset, the plant model can define physical dimensions and colors for the asset, as well as any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.). The plant models 524 also define the physical relationships between the industrial assets, including relative positions and orientations of the assets on the plant floor, conduit or plumbing that runs between the assets, and other physical definitions, [0086] generate the presentation such that items of the plant data 610 are overlaid on or near graphical representations of the industrial assets to which the items of data relate, [0102] during programming and configuration of an industrial controller using the configuration application, the user may define a data tag for storing a water pressure value (e.g., Tank 1 Water Pressure) measured by a pressure meter associated with a tank (e.g., Tank 1). Plant models 524 may define information about Tank 1 for the purposes of rendering VR or AR presentations for the tank, including but not limited to a location of the tank within the plant facility, a graphical representation of the tank, any animations associated with the tank, etc. After defining the Tank 1 Water Pressure data tag, the configuration application can also be used to specify that the data tag's value is to be associated with the Tank 1 entity. This association definition will be stored in one or more of the plant models 524, and based on this definition rendering component 308 will render the Tank 1 Water Pressure data value on a user's wearable appliance 206 in response to determining that the appliance's location and orientation places Tank 1 (or a graphical representation of Tank 1 in the case of VR presentations) within the user's line of sight).
Regarding claim 6, the combination of Schmirler and Weber teach The automation system of claim 5, wherein the data type is selected from the group consisting of: text data, image data, video data, audio data and alarm data (Schmirler, [0056] The VR/AR presentation system can then generate augmented reality or virtual reality presentations and deliver these presentations to the user's wearable appliance; e.g., as graphical or text-based indicators overlaid on the user's field of view, such that each indicator is positioned near the machine or device to which the indicator pertains. For example, if the user's current view encompasses a real or virtualized motor-driven conveyor and a motor drive that controls the motor, the presentation system may superimpose a current operating status of the motor drive (e.g., a current speed, a fault condition, an operating mode, etc.) near the image or view of the motor drive as perceived by the user, [0123] the system 302 can collect video (or audio-video) data from one or more cameras distributed throughout the plant environment, and integrate selected sets of this video data with a VR/AR presentation. When 360-degree cameras are used, such embodiments can provide users at remote locations with an interactive live video feed of the plant facility, simulating the user's physical presence on the plant floor. FIG. 14A is a diagram illustrating an example configuration that incorporates video information into VR/AR presentations generated by the VR/AR presentation system, [0180] alarm information generated by an industrial controller or HMI can be rendered on the VR/AR representation on or near the industrial assets to which the alarms relate).
Regarding claim 7, the combination of Schmirler and Weber teach The automaton system of claim 5, wherein each secondary virtual object of the at least one secondary virtual object linked to the at least one virtual object comprises data of the data type represented by the at least one virtual object (Schmirler, [0098] These asset information icons 810 can be selected using similar gesture or verbal recognition techniques used to select operator information icons 804. Selection of an asset information icon 810 can cause rendering component 308 to render an information window on or near the asset corresponding to the icon 810, and populate this information window with information relevant to the asset. Some or all of the rendered asset information can be derived from relevant subsets of plant data 610, or can be derived from stored data about the asset registered previously with the VR/AR presentation system 302. In the case of control cabinets, rendered information can include, but is not limited to, a name of the cabinet, a machine or industrial process controlled by the control cabinet, identification of devices mounted within the cabinet (e.g., industrial controllers, I/O modules, motor drives, contactors, etc.), statuses of devices mounted within the cabinet, or other such information. In the case of machines, information that can be rendered in response to selection of asset information icon 810 b can include, but is not limited to, a current operating mode of the machine (e.g. automatic, manual, semi-manual, etc.), a fault status for the machine, an operating speed or production rate, a total number of logged work hours (or work hours since a most recent maintenance activity), a date on which maintenance was most recently performed on the machine, information regarding maintenance actions performed on the machine, or other such information).
Regarding claim 8, Schmirler teaches The automation system of claim 4,
Schmirler does not teach wherein the executable program logic is configured to: dynamically assign one of a primary colour and/or a secondary colour and/or a tertiary colour to each virtual object of the at least one virtual object linked to the at least one respective secondary virtual object, wherein each virtual object is assigned a different colour
Weber teaches wherein the executable program logic is configured to: dynamically assign one of a primary colour and/or a secondary colour and/or a tertiary colour to each virtual object of the at least one virtual object linked to the at least one respective secondary virtual object, wherein each virtual object is assigned a different colour ([0049] the button visual identifiers are colour-based, such as different colours associated with each confirm button. In such embodiments, each element visual identifier corresponds to one of the colours of the confirm buttons. In another embodiment, the button visual identifiers are pattern-based, such as different patterns associated with each confirm button..
Regarding claim 12, Schmirler teaches The automation system of claim 4,
Schmirler does not teaches wherein the hands-free selection comprises: a gaze selection of the virtual object
Weber teaches wherein the hands-free selection comprises: a gaze selection of the virtual object ([0008] modifying the display of the electronic visual work to present the assigned element visual identifiers for at least a portion of the displayed activatable elements; receiving a gaze signal representing the user's gaze position on the display screen from an eye gaze tracker; sensing the user's gaze dwelling on or near an activatable element of the displayed electronic visual work based on the gaze signal).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Schmirler’s teaching of virtual reality for industrial automation using a plant model defining physical dimensions and colors of assets with Weber’s teaching of a gaze controlled interface. The combined teaching provides an expected result of virtual reality for industrial automation allowing a gaze controlled interface. Therefore, one of ordinary skill in the art would be motivated to improve the usability of hands free selection in the system.
Regarding claim 13, Schmirler teaches The automation system of claim 1, wherein the augmented reality glasses and/or the positioning system comprises one or more directional sensors configured to ([0004] generate augmented reality presentation data that renders a augmented reality presentation of the industrial facility on a wearable appliance, Fig. 13):
Schmirler does not teach detect a line of sight of the augmented reality glasses or an eye orientation of the user wearing the augmented reality glasses, and determine a first length of time the line of sight of the augmented reality glasses is fixed on the virtual object or a second length of time the eye orientation of the user wearing the augmented reality glasses is fixed on the virtual object, and if either the first length of time or the second length of time is greater than a predefined maximum gaze duration, generate the signal indicating the hands-free selection, by the user, of the virtual object.
Weber teaches detect a line of sight of the augmented reality glasses or an eye orientation of the user wearing the augmented reality glasses, and determine a first length of time the line of sight of the augmented reality glasses is fixed on the virtual object or a second length of time the eye orientation of the user wearing the augmented reality glasses is fixed on the virtual object, and if either the first length of time or the second length of time is greater than a predefined maximum gaze duration, generate the signal indicating the hands-free selection, by the user, of the virtual object ([0029] the step of sensing the user's gaze dwelling on or near an activatable element based on the gaze signal comprises sensing whether the user's gaze has being substantially stationary for a time interval corresponding to a first time threshold. In one form, the user's gaze is considered to be substantially stationary if for the time interval corresponding to the first time threshold each successive gaze signal sample representing the user's gaze position is within a predetermined distance of the previous sample defined by a predetermined distance threshold or parameter, [0032] sensing the user's gaze dwelling on an activated confirm button comprises sensing the user's gaze as being substantially stationary on a confirm button for a time interval corresponding to a second time threshold, [0045] receiving a gaze signal representing the user's gaze position on the display screen from an eye gaze tracker; sensing the user's gaze dwelling in a gaze area on or near an activatable element of the displayed electronic visual work based on the gaze signal and a first time threshold; activating at least the confirm button having the button visual identifier corresponding to the element visual identifier of the closest activatable element to the gaze area being dwelled on; sensing the user's gaze dwelling in a gaze area on an activated confirm button based on the gaze signal and a second time threshold; and activating the activatable element in response to the activated confirm button being dwelled on. ).
Regarding claim 19, Schmirler teaches The method of claim 18, further comprising: determining, by the executable program logic, a number of the at least one virtual object linked to the at least one respective secondary virtual object and associated with one or more equipment proximate to the spatial position of the user, ([0056] Based on the determined identity of the automation system currently being viewed by the user, the VR/AR presentation system can determine current status information for devices and/or machines that make up the automation system, or for a process being carried out by the automation system. The VR/AR presentation system can then generate augmented reality or virtual reality presentations and deliver these presentations to the user's wearable appliance; e.g., as graphical or text-based indicators overlaid on the user's field of view, such that each indicator is positioned near the machine or device to which the indicator pertains, [0099] The VR/AR presentation of the production area also includes a number of camera icons 806 (e.g., 806 a-806 d) that allow the user to switch the presentation to a live or historical video feed, as will be described in more detail below) dynamically assigning a distinct colour from a predefined colour set of colours to each virtual object linked to respective secondary virtual objects and displaying each virtual object linked to respective secondary virtual objects with the dynamically assigned distinct colour ([0085] For each industrial asset, the plant model can define physical dimensions and colors for the asset, as well as any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.). The plant models 524 also define the physical relationships between the industrial assets, including relative positions and orientations of the assets on the plant floor, conduit or plumbing that runs between the assets, and other physical definitions, [0086] A rendering engine supported by rendering component 308 is configured to generate an interactive VR/AR presentation of the industrial area based on the industrial asset rendering definitions specified in the plant models. Rendering component 308 populates this virtual reality presentation with selected subsets of collected plant data 610 (as well as production or operational statistics calculated by reporting component 310 based on subsets of the plant data 610), and client interface component 304 delivers the resulting aggregate VR/AR presentation to wearable appliance 206 as VR/AR presentation data 604),
Schmirler does not teach and if the number of virtual objects linked to respective secondary virtual objects is greater than a number of colours of the predefined colour set, then displaying only those virtual objects linked to respective secondary virtual objects that are assigned distinct colours.
Weber teaches and if the number of virtual objects linked to respective secondary virtual objects is greater than a number of colours of the predefined colour set, then displaying only those virtual objects linked to respective secondary virtual objects that are assigned distinct colours ([0124] if all visual identifiers (e.g. all colours in this example) in the predefined set have already been assigned, analysis is done to gather a neighbour set of activatables at step 101. In particular, for each visual identifier (i) in the predefined set the activatable neighbour (i,a) with visual identifier i closest to the current activatable a is located and stored in the neighbour set. The farthest activatable neighbour in the neighbour set from the unassigned current activatable a is then identified at step 102 and the visual identifier of that farthest activatable neighbour is assigned to the unassigned current activatable a, before the algorithm returns to decision 96 to assign visual identifiers to the next unassigned activatables, if any, [0132] the predetermined activation zone may be configured to be smaller than the assignment zone used in the assignment step, such that the activation is not likely to capture two activatables having the same assigned colour. However, if two or more activatables having the same assigned colour do happen to fall within the activation zone, then the activatable closest to the last gaze position sample is selected to be associated with its corresponding activated confirm button).
Claim(s) 3 is rejected under 35 U.S.C. 103 as being unpatentable over Schmirler et al. (US20180131907A1, herein Schmirler), in view of Weber et al. (US20160195924, herein Weber), and in further view of Goldberg et al. (US20190235641, herein Goldberg).
Regarding claim 3, the combination of Schmirler and Weber teach The automation system of claim 2,
Weber further teaches wherein if the number of virtual objects linked to respective secondary virtual objects is greater than the number of colours of the predefined colour set, the executable program logic is further configured to: ([0124] if all visual identifiers (e.g. all colours in this example) in the predefined set have already been assigned, analysis is done to gather a neighbour set of activatables at step 101. In particular, for each visual identifier (i) in the predefined set the activatable neighbour (i,a) with visual identifier i closest to the current activatable a is located and stored in the neighbour set. The farthest activatable neighbour in the neighbour set from the unassigned current activatable a is then identified at step 102 and the visual identifier of that farthest activatable neighbour is assigned to the unassigned current activatable a, before the algorithm returns to decision 96 to assign visual identifiers to the next unassigned activatables, if any, [0132] the predetermined activation zone may be configured to be smaller than the assignment zone used in the assignment step, such that the activation is not likely to capture two activatables having the same assigned colour. However, if two or more activatables having the same assigned colour do happen to fall within the activation zone, then the activatable closest to the last gaze position sample is selected to be associated with its corresponding activated confirm button) wherein dynamically assign a distinct colour further comprises: dynamically assign a distinct colour from the predefined colour set of colours to each virtual object of a second number of virtual objects…, the second number equal to the number of colours ([0049] the button visual identifiers are colour-based, such as different colours associated with each confirm button. In such embodiments, each element visual identifier corresponds to one of the colours of the confirm buttons, [0098] hyperlinks assigned red would be activatable by the red confirm button, hyperlinks assigned blue would be activatable by the blue confirm button, hyperlinks assigned green would be activatable by the green confirm button, and so on, in the context of a colour-based visual association. )
The combination of Schmirler and Weber do not teach … based on display priority …dynamically determine distances from the user to each of the one or more equipment proximate to the spatial position of the user and associated with the at least one virtual object linked to the at least one respective secondary virtual object, and assign a display priority to each of the at least one virtual objects linked to the at least one respective secondary virtual object based on the determined distances,
Goldberg teaches … based on display priority …dynamically determine distances from the user to each of the one or more equipment proximate to the spatial position of the user and associated with the at least one virtual object linked to the at least one respective secondary virtual object, and assign a display priority to each of the at least one virtual objects linked to the at least one respective secondary virtual object based on the determined distances, ([0076] some implementations may identify multiple controllable devices, which can then be presented to the user in a list. The list may be ordered based on distance to the controllable device, [0033] device identification engine 124 to identify the device at which the user is aiming. For example, the device identification engine 124 may identify the controllable device based on the direction the user is aiming and a location of the computing device 102 as determined using the positioning system 126. The device identification engine 124 may project a ray from the determined location in the direction in a representation of the physical space and then determine whether the ray identifies any controllable devices in the representations.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Schmirler’s teaching of virtual reality for industrial automation with Goldberg’s teaching distance-based ordering of controllable devices. The combined teaching provides an expected result of virtual reality for industrial automation using distance-based ordering of controllable devices. Therefore, one of ordinary skill in the art would be motivated to improve the usability by reducing display clutter.
Claim(s) 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Schmirler et al. (US20180131907A1, herein Schmirler), in view of Alexander et al. (US20180124895A1, herein Alexander).
Regarding claim 10, Schmirler teaches The automation system of claim 9,
Schmirler does not teach wherein the voice selection comprises: a colour selection
Alexander teaches wherein the voice selection comprises: a colour selection ([0052] a voice-activated household speaker serves as an input element to the lighting system, wherein the voice-activated household speaker receives verbal commands from a user (e.g. ‘change the first controllable zone to the color yellow’) and communicates the verbal commands to a control module associated with the lighting system).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Schmirler’s teaching of virtual reality for industrial automation using a plant model defining physical dimensions and colors of assets with Alexander’s teaching of a voice selection of color. The combined teaching provides an expected result of virtual reality for industrial automation using a voice selection of color. Therefore, one of ordinary skill in the art would be motivated to improve the usability of the user for hands free control.
Regarding claim 11, Schmirler teaches The automation system of claim 9,
Schmirler does not teach wherein the voice selection is expressed in English.
Alexander teaches wherein the voice selection is expressed in English 0052] a voice-activated household speaker serves as an input element to the lighting system, wherein the voice-activated household speaker receives verbal commands from a user (e.g. ‘change the first controllable zone to the color yellow’) and communicates the verbal commands to a control module associated with the lighting system).
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Schmirler et al. (US20180131907A1, herein Schmirler), in view of Os et al. (US20100312547A1, herein Os)
Regarding claim 20, Schmirler teaches The automation system of claim 1,wherein the triggered action is one or more actions selected from the group consisting of: sending a command to the one of the equipment associated with the selected virtual object, the command causing the equipment to start, stop or change its operation; activating a command interpreter for voice commands, whereby the command interpreter is configured to process and interpret voice commands for controlling the equipment associated with the selected virtual object in accordance with a list of commands specified in the selected virtual object ([0113] VR/AR presentations on wearable appliance 206, some embodiments of the VR/AR presentation system 302 can allow control instructions to be originated from wearable appliance 206 and delivered to a target control device (e.g., an industrial controller, a motor drive, a human-machine interface terminal, or other such control device). In scenarios in which the wearer of wearable appliance 206 is physically located on the plant floor, such control instructions can be delivered directly from the wearable appliance 206 to the control device in some embodiments. FIG. 12 is a diagram illustrating data flows between presentation system 302, wearable appliance 206, and an industrial controller 1204 in a scenario in which wearable appliance is configured to direct control instructions 1206 to the industrial controller 1204, [0115] Example commands can include, for example, machine start/stop commands, switch setting adjustments, setpoint adjustments, alarm reset commands, or other such comments. The user can select from among the list of predefined commands using a suitable recognizable gesture or verbal command, and the wearable appliance 206 can issue the selected command to the industrial controller 1204. For example, if the command is a binary instruction—such as an alarm reset command, a start command, or a stop command—the wearable appliance 206 can direct a momentary or latching ON command to the appropriate register of the industrial controller's data table via the plant network (e.g., a CIP network on which the controller resides)) ;
Schmirler does not teach and disabling at least one other command interpreter configured to process and interpret voice commands in accordance with a list of commands specified in a non-selected virtual object.
Os teaches and disabling at least one other command interpreter configured to process and interpret voice commands in accordance with a list of commands specified in a non-selected virtual object ([0034] the contextual voice command option can be active as a default, and the user can deactivate the option by providing a predetermined triggering input. For example, the device can exit the listening mode based on a voice input, such as “exit listening mode” or a physical input, such as a touch or a button press. Also, the device can enter and exit the listening mode using a combination of voice and physical inputs. For example, the user can touch a button to initiate entry into or exit from the listening mode and a follow-up voice command, such as “confirm exit or entry” to finalize the entry into or exit from the listening mode).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Schmirler’s teaching of virtual reality for industrial automation using voice commands with Os’s teaching of deactivating the voice *command. The combined teaching provides an expected result of virtual reality for industrial automation using voice commands that are able to be deactivated. Therefore, one of ordinary skill in the art would be motivated to reduce erroneous commands, improving system accuracy.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Simil (US20170277166) discloses a system to increase operator awareness for process control application providing operators team real time.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to YVONNE T FOLLANSBEE whose telephone number is (571)272-0634. The examiner can normally be reached on Monday - Friday 1pm - 9pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Robert, Fennema can be reached on (571) 272-2748 The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppairmy.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786- 9199 (IN USA OR CANADA) or 571-272-1000.
/YVONNE T FOLLANSBEE/
Examiner, Art Unit 2117
/ROBERT E FENNEMA/Supervisory Patent Examiner, Art Unit 2117
Prosecution Insights