DETAILED ACTION
This office action is in response to applicant’s submission filed on June 29, 2026. Claims 9, 14-20, and 25 were previously canceled. Claims 1-8, 10-13, and 21-24, and 26-29 are pending and are rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on June 29, 2026 has been entered.
Response to Amendment
This communication is in response to the amendment filed on June 29, 2026. The Examiner has acknowledged the amended claims 1, 6, 11, 13 and 21. Claims 9, 14-20, and 25 were previously canceled. Claims 1-8, 10-13, and 21-24, and 26-29 are pending and are rejected.
Response to Arguments
Applicant’s Arguments (Remarks) filed June 29, 2026 have been fully considered, but are moot.
Applicant’s arguments with respect to claim(s) 1, 6, and 21 been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-13, 21-24, and 26-29 are rejected under 35 U.S.C. 103 as being unpatentable over US 2021/0149359 A1 to Mehrotra et al. (hereinafter, “Mehrotra”) in view of US 2016/0328282 A1 to Rogati et al. (hereinafter, “Rogati”).
Regarding claim 1, Mehrotra discloses: An industrial automation component, comprising:
an electronic display configured to present first image data comprising a first machine-readable indicia representative of an identity of the industrial automation component (“electronic display device for presenting AR UX workflow visualizations” [0060]; “The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0072] [Examiner notes that the second text involves displaying machine-readable indicia and identifying an industrial automation component using that code which the system can retrieve information about that component via the indicia]); and
a processor communicatively coupled to the electronic display, wherein the processor is configured to (“processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005]):
acquire second image data associated with a secondary device via an image sensor disposed on the industrial automation component (“receive an input to the image data from the second device” [0005]; “the operator device 54 may, at block 226, receive updated image data of the industrial automation device 20 from the image sensor 66” [0104]),
wherein the second image data comprises a second machine-readable indicia rendered on a display of the secondary device, the second machine-readable indicia representative of an additional identity of the secondary device and an operator; determine a device identifier and an operator identifier associated with identifying the secondary device and identifying the operator, respectively based on the second image data (“The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50… The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0071-0072]; “The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like” [0040]; “The workflow dataset may cause the first device and the second device to display one or more visualizations representative of the one or more instructions, the one or more virtual objects, or both on image data depicted on a first display and on a second display associated with the first device and the second device, respectively. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device, where the input may include one or more additional visualizations, and causes the processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005] [Examiner notes that the first device is the industrial automation device 20 and the second device is the operator device 54. This text 0005 says that both devices and both displays of the devices can display and give input to the image data to receive image data that has these visualizations of indicia] [Examiner also notes that the device identifier is stored within the profile which is known after processing the image data. Image data is captured by an image sensor and that image data can contain machine-readable indicia. The indicia is used to identify devices – including multiple different devices. The system will be able to recognize and differentiate between devices and the user can capture different images, each with different device identifiers. Examiner also wants to note that the profile also contains the operator identifier. Examiner also notes that the operator identifier already implicitly shows an operator]);
identify authentication data based on a comparison between the device identifier and a stored list of device identifiers, wherein the authentication data comprises access credentials associated with one or more operators performing one or more operations on the industrial automation component for each device identifier of the stored list of device identifiers (“Additionally or alternatively, the computing device of the operator may be associated with a profile. The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like. The profile may be used by the AR UX system to pre-filter a query of the workflow database, such that a subset of workflows relevant to the industrial automation system and/or operator are searched by the AR UX system against the parameters. In some embodiments, the operator may input the parameters of the query into the computing device using drop-down menus, radio buttons, or the like, from a defined set of options. Thus, in some cases, the AR UX system may instruct the device to present a subset of options to the operator based at least in part on the profile corresponding to the operator. In this way, the operator is unable to select options that the profile of the operator is ineligible to select. The subset of options for parameters presented to the operator may be based on the profile. The profile may define permissions of the operator, such as operations or equipment that the operator is eligible to adjust. The profile may also include an indication of portions of the industrial automation system that the operator works within or devices that are included within the industrial automation system to filter irrelevant industrial automation devices out from the query. Profile filtering of the workflow database may improve a speed of the query used by the AR UX system when querying the workflow database based on parameters by reducing a subset of workflows that the AR UX system searches with the query” [0041]; “In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071] [Examiner notes that after the device identifier is identified, the profile itself (where the identifier resides) is then used to filter which devices can has the correct authentication permissions to interact with the device. Examiner also wants to note that each operator is tied to specific devices of the system, supporting the notion of access being associated with device identifiers, and the system also delivers device-specific operations when the device is identified which implies that the AR UX system matches the device ID to stored operations]);
authenticate the secondary device based on the authentication data and the operator identifier (“The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94” [0071] [Examiner wants to note that the image data and information from the operator device is used to match with a profile. Since the AR UX system matches the incoming data with profile(s) in the profile data – which includes the operator identifier – it is effectively authenticating both the secondary device and the operator]);
generate third image data in response to authenticating the secondary device, wherein the third image data comprises a third machine-readable indicia representative of information relating to an error code corresponding to a part of portion of computer code associated with the industrial automation component and the operator identifier performing one or more respective operations on the industrial automation component; and present the third image data via the electronic display (Fig. 6; “At block 116, the operator device 54 may use the input identifying the product to generate and send a query to the AR UX system 82. The AR UX system 82 may receive the query and use information of the query (e.g., input parameters of product type, product, desired experience) to search the AR UX system database 98. The query may result in the AR UX system 82 identifying a subset of AR UX workflow datasets that are relevant to the input received by the operator device 54. AR UX workflows may be associated with data, including image data, used by the operator device 54 to present visualizations of steps related to a procedure to perform various types of tasks. Visualizations associated with a respective AR UX workflow dataset may be rendered via a display of the operator device 54 to generate a graphical user interface and to provide overlaid images that communicate safety information, tools, lock-out/tag-out (e.g., electrical isolation) information, procedure steps, or the like to the operator 50 preparing to operate or actively operating on the industrial automation device 20” [0078]; “In some cases, the AR UX system 82 may receive an indication of the AR UX workflow dataset to transmit to the operator device 54 from the operator device 54 as opposed to query parameters. The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20. For example, the links, barcodes, or the like may include a code acquired via a scanner, a barcode acquired via the scanner, a digital link to a website. Furthermore, in some cases, the AR UX system 82 may access maintenance schedules or the like for the industrial automation device, and may automatically retrieve the corresponding AR UX workflow dataset to an upcoming or overdue maintenance procedure for transmission to the operator device 54. The above-described features and more are discussed below with reference to interactions between the operator device 54 and the AR UX system 82, such as to select and provide data associated with an AR UX workflow dataset” [0072] [Examiner notes that this text shows scanning of a link, QR code, etc. (machine-readable indicia) that gives information by retrieving the “workflow datasets” and “maintenance schedules”. The maintenance schedules and workflow datasets come from the device’s internal software which is seen as the device’s software. The indicia here connects to software-generated information and the information is a workflow/maintenance dataset from the software so scanning the indicia retrieves data that comes from specific software coming from the device]; “In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071] [Examiner notes that the first text shows the system rendering visualization and images on the operator device’s display which is the generating third image data; it also shows it is in response to authenticating since the system flow implies that the operator device must be authenticated and associated with a valid operator before queries and rendering can happen. The visualizations are tied to actions the operator is taking. The second text is added to show more support that the image data is being generated in response to authenticating the device as the scanning action serves as a form of authentication/identity confirmation. The system only responds with relevant workflow data after receiving and validating that input from the operator device which is reasonably interprets as a form of authentication as it is using the device identity/access permissions. The third text is used to show that the image data comprises information relating to the operator via the profile association]; “An AR UX workflow may include steps associated with performing various maintenance procedures or industrial operation activities by an operator via a computing device associated with the operator. That is, the computing device (e.g., mobile phone) may present a live image data stream (e.g., images captured in real-time from an image sensor of the device, video feed) of an industrial automation device and overlay instructions or information related to the industrial automation device on the image data stream. In some cases, the AR UX system may present one or more virtual objects that are overlaid into the image data stream. The virtual objects may be manipulated or animated as part of the AR UX workflow to show portions (e.g., steps, sequential operations) associated with the AR UX workflow. For example, the interactive AR UX system may guide an operator through a maintenance procedure by displaying or modifying one or more virtual objects in the visualization associated with the AR environment presented to the user” [0036]; “In an example embodiment, the AR UX system may guide the operator using a combination of text and image cues presented on the computing device of the operator. In this way, while performing a maintenance activity on an example industrial automation device, such as powering off a component, the operator may receive step-by-step instructions according to the AR UX workflow regarding how to perform the maintenance activity on the industrial automation device (e.g., for the particular model and version of the drive). The AR UX system may present a visualization of the industrial automation device (e.g., virtual object) and manipulate the visualization of the industrial automation device to guide the operator through performance of the maintenance activity. For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off. In some cases, the AR UX system may identify a time that an operation for a current instruction was completed by the operator and, in response to the identification, the AR UX system may automatically advance to a subsequent instruction” [0038]; “The maintenance history may be referenced by other industrial automation devices 20, control systems, or the like to determine whether the industrial automation device 20 is due for servicing (e.g., mechanical or electrical maintenance work) or the like” [0109] [Examiner notes that error codes are used to show a user what is not properly functioning in order to properly get to the root of the problem. This text is describing that by giving a text status and location via display with images cues in order for the user to see what is wrong and fix it. This shows the system’s ability to detect a problem, whether it is connection-based or external, and potentially flag it. Examiner also notes that the connected piece is that the access to the AR UX workflow dataset corresponding to the industrial automation device 20 happens through links, barcodes, or the like may include a code acquired via a scanner, a barcode acquired via the scanner, a digital link to a website. Furthermore, in some cases, the AR UX system 82 may access maintenance schedules. It is through this access (by scanning) to the maintenance schedules that gives information by retrieving the “workflow datasets” and “maintenance schedules” which the error codes are linked to as the maintenance schedules and procedures are representative of a specific error code]),
wherein the secondary device is configured to: receive fourth image data representative of the third machine-readable indicia (“The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0072]);
retrieve a set of instructions for resolving the error code based on the fourth image data and the authentication data (“In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071]; “In an example embodiment, the AR UX system may guide the operator using a combination of text and image cues presented on the computing device of the operator. In this way, while performing a maintenance activity on an example industrial automation device, such as powering off a component, the operator may receive step-by-step instructions according to the AR UX workflow regarding how to perform the maintenance activity on the industrial automation device (e.g., for the particular model and version of the drive). The AR UX system may present a visualization of the industrial automation device (e.g., virtual object) and manipulate the visualization of the industrial automation device to guide the operator through performance of the maintenance activity. For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off. In some cases, the AR UX system may identify a time that an operation for a current instruction was completed by the operator and, in response to the identification, the AR UX system may automatically advance to a subsequent instruction” [0038]; “Additionally or alternatively, the computing device of the operator may be associated with a profile. The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like. The profile may be used by the AR UX system to pre-filter a query of the workflow database, such that a subset of workflows relevant to the industrial automation system and/or operator are searched by the AR UX system against the parameters. In some embodiments, the operator may input the parameters of the query into the computing device using drop-down menus, radio buttons, or the like, from a defined set of options. Thus, in some cases, the AR UX system may instruct the device to present a subset of options to the operator based at least in part on the profile corresponding to the operator. In this way, the operator is unable to select options that the profile of the operator is ineligible to select. The subset of options for parameters presented to the operator may be based on the profile. The profile may define permissions of the operator, such as operations or equipment that the operator is eligible to adjust. The profile may also include an indication of portions of the industrial automation system that the operator works within or devices that are included within the industrial automation system to filter irrelevant industrial automation devices out from the query. Profile filtering of the workflow database may improve a speed of the query used by the AR UX system when querying the workflow database based on parameters by reducing a subset of workflows that the AR UX system searches with the query. Moreover, profile filtering and other features described herein may reduce the amount of processing power employed by the AR UX system, thereby improving the operation of the AR UX system” [0041] [Examiner notes that these 3 texts shows an operator capturing an image of the industrial automation device, which the AR UX system uses to identify the specific device (based on image data). The AR UX system retrieves and presents step-by-step workflow instructions to the operator for the identified device (retrieving a set of instructions). The operator’s profile defines identity and permissions, ensuring that only authorized operators can access relevant workflows or instructions. Queries are also filtered based on the profile to restrict access to permitted options (based on authentication data)]); and
present the set of instructions The workflow dataset may cause the first device and the second device to display one or more visualizations representative of the one or more instructions, the one or more virtual objects, or both on image data depicted on a first display and on a second display associated with the first device and the second device, respectively. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device, where the input may include one or more additional visualizations, and causes the processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005]).
Mehrotra does not disclose: present the set of instructions comprising software update instructions corresponding to the part of portion of computer code corresponding to the error code via an additional electronic display of the secondary device.
However, Rogati discloses: present the set of instructions comprising software update instructions corresponding to the part of portion of computer code corresponding to the error code via an additional electronic display of the secondary device (“As one example, a software glitch, error, bug or the like may manifest itself as a hardware failure that triggers generation of data representing a hardware-related device error code. Diagnosis at the stage 304 may include accessing the failure diagnosis database using the received device error code as a look-up or search key, for example, and determine that identical or similar devices with the same device error code were not replaced because a root cause of the device error code was related to a software issue rather than a hardware issue, and the software issue may be fixed by a software update instead of device replacement. As another example, diagnosis at the stage 304 may include accessing the failure diagnosis database and determining that the device error code received is associated with a low battery reserve condition on previously diagnosed devices (e.g., the device error code was due to low battery power instead of an actual hardware failure). Accordingly, instead of replacing the device, a message may be communicated to the device (e.g., 110 or 150) instructing a user (e.g., customer 101) to charge the battery of their device using a wall charger or the like” [0037] [Examiner notes that it would have been obvious to a POSITA to modify Mehrotra, which teaches presenting instructions in response to an error code, in view of Rogati, which teaches an error code may correspond to a software-related issue that can be resolved through a software update. Specially this reference teaches using the error code to diagnose a root cause and determining that the issue may be corrected by updating software rather than fixing hardware. Incorporating this teaching would have resulted in presenting software update instructions to the error code since providing update based remediation is a known extension of error-based system wherein such updated inherently correspond to the portion of software associate with said error condition]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mehrotra with the added structure of Rogati in order for the system to cover software/code issues on top of the technical/hardware system of Mehrotra.
Regarding claim 2, a combination of Mehrotra-Rogati discloses the system of claim 1.
Mehrotra further discloses: in response to the comparison being indicative of the device identifier not being present in the stored list of device identifiers, generate a notification comprising an additional error code; and transmit the notification to the secondary device (“An AR UX workflow may include steps associated with performing various maintenance procedures or industrial operation activities by an operator via a computing device associated with the operator. That is, the computing device (e.g., mobile phone) may present a live image data stream (e.g., images captured in real-time from an image sensor of the device, video feed) of an industrial automation device and overlay instructions or information related to the industrial automation device on the image data stream. In some cases, the AR UX system may present one or more virtual objects that are overlaid into the image data stream. The virtual objects may be manipulated or animated as part of the AR UX workflow to show portions (e.g., steps, sequential operations) associated with the AR UX workflow. For example, the interactive AR UX system may guide an operator through a maintenance procedure by displaying or modifying one or more virtual objects in the visualization associated with the AR environment presented to the user” [0036]; “In an example embodiment, the AR UX system may guide the operator using a combination of text and image cues presented on the computing device of the operator. In this way, while performing a maintenance activity on an example industrial automation device, such as powering off a component, the operator may receive step-by-step instructions according to the AR UX workflow regarding how to perform the maintenance activity on the industrial automation device (e.g., for the particular model and version of the drive). The AR UX system may present a visualization of the industrial automation device (e.g., virtual object) and manipulate the visualization of the industrial automation device to guide the operator through performance of the maintenance activity. For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off. In some cases, the AR UX system may identify a time that an operation for a current instruction was completed by the operator and, in response to the identification, the AR UX system may automatically advance to a subsequent instruction” [0038]; “The maintenance history may be referenced by other industrial automation devices 20, control systems, or the like to determine whether the industrial automation device 20 is due for servicing (e.g., mechanical or electrical maintenance work) or the like” [0109] [Examiner notes that error codes are used to show a user what is not properly functioning in order to properly get to the root of the problem. This text is describing that by giving a text status and location via display with images cues in order for the user to see what is wrong and fix it. This shows the system’s ability to detect a problem, whether it is connection-based or external, and potentially flag it]).
Claim 3 recites substantially the same limitation as claim 2, for implementing the corresponding method, therefore it is rejected under the same rationale.
Claim 4 recites substantially the same limitation as claim 2, for implementing the corresponding method, therefore it is rejected under the same rationale.
Regarding claim 5, a combination of Mehrotra-Rogati discloses the system of claim 1.
Mehrotra further discloses: wherein the first machine-readable indicia comprises a Quick Response (QR) code, a barcode, or a combination thereof (“barcode (e.g., a QR code, a matrix barcode)” [0085]).
Regarding claim 6, Mehrotra discloses: A method, comprising:
receiving connection data associated with an industrial automation component, wherein the connection data comprises a list of devices coupled to the industrial automation component (“The AR UX system 82 may also perform additional filtering before querying the AR UX workflow database 92. The additional filtering may reduce a number of workflows to be searched via the query. The AR UX system 82 may reference stored profiles to determine a subset of AR UX workflow datasets that are relevant to be queried. For example, a profile may include information about a permission or skill level of the operator 50, such that if the operator 50 is not trained on an industrial automation device 20 or is otherwise unpermitted to adjust operation of the industrial automation device 20, the AR UX system 82 may not query AR UX workflow datasets related to the industrial automation device 20 since those AR UX workflow datasets may be irrelevant to the operator 50. The AR UX system 82 may also filter AR UX workflow datasets based on profiles for the industrial automation system 10. In this way, a profile for the industrial automation system 10 may store indications of related equipment (e.g., part of an overall system), model numbers of the related equipment, operational preferences for the related equipment, or the like for the industrial automation system 10” [0066]; “The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94)” [0071] [Examiner notes that the connection data can be defined in the applicant’s specification as “a description of the coupled devices (e.g., model number, device type, respective connection port)” in paragraph 0064 which here, is included within the profile]);
generating first image data based on the connection data, wherein the first image data comprises a first machine-readable indicia having the connection data embedded therein (“In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data). In some cases, the AR UX system 82 may receive an indication of the AR UX workflow dataset to transmit to the operator device 54 from the operator device 54 as opposed to query parameters. The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20. For example, the links, barcodes, or the like may include a code acquired via a scanner, a barcode acquired via the scanner, a digital link to a website” [0071-0072]);
receiving updated connection data, wherein the updated connection data comprises an updated list of devices coupled to the industrial automation component (“update a profile corresponding to the operator device 54” [0109]),
wherein the updated list of devices comprises at least one additional device coupled to the industrial automation component, at least one device disconnected from the industrial automation component, or a combination thereof (“For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off” [0038]);
generating second image data based on the updated connection data, wherein the second image data comprises a second machine-readable indicia having the updated connection data embedded therein (“In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data). In some cases, the AR UX system 82 may receive an indication of the AR UX workflow dataset to transmit to the operator device 54 from the operator device 54 as opposed to query parameters. The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20. For example, the links, barcodes, or the like may include a code acquired via a scanner, a barcode acquired via the scanner, a digital link to a website” [0071-0072]);
displaying the second image data via an electronic display of the industrial automation component (“electronic display device for presenting AR UX workflow visualizations” [0060]);
receiving authentication data associated with the industrial automation component, wherein the authentication data comprises a list of device identifiers permitted to access data associated with the industrial automation component (“Additionally or alternatively, the computing device of the operator may be associated with a profile. The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like. The profile may be used by the AR UX system to pre-filter a query of the workflow database, such that a subset of workflows relevant to the industrial automation system and/or operator are searched by the AR UX system against the parameters. In some embodiments, the operator may input the parameters of the query into the computing device using drop-down menus, radio buttons, or the like, from a defined set of options. Thus, in some cases, the AR UX system may instruct the device to present a subset of options to the operator based at least in part on the profile corresponding to the operator. In this way, the operator is unable to select options that the profile of the operator is ineligible to select. The subset of options for parameters presented to the operator may be based on the profile. The profile may define permissions of the operator, such as operations or equipment that the operator is eligible to adjust. The profile may also include an indication of portions of the industrial automation system that the operator works within or devices that are included within the industrial automation system to filter irrelevant industrial automation devices out from the query. Profile filtering of the workflow database may improve a speed of the query used by the AR UX system when querying the workflow database based on parameters by reducing a subset of workflows that the AR UX system searches with the query” [0041]; “In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071]);
acquiring third image data associated with a secondary device via an image sensor of the industrial automation component, the third image data comprising third machine-readable indicia rendered on a display of the secondary device, the third machine-readable indicia representative of an identity of the secondary device and an operator; determining a device identifier and an operator identifier based on the third image data (“The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50… The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0071-0072]; “The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like” [0040]; “The workflow dataset may cause the first device and the second device to display one or more visualizations representative of the one or more instructions, the one or more virtual objects, or both on image data depicted on a first display and on a second display associated with the first device and the second device, respectively. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device, where the input may include one or more additional visualizations, and causes the processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005]; Fig. 6; “At block 116, the operator device 54 may use the input identifying the product to generate and send a query to the AR UX system 82. The AR UX system 82 may receive the query and use information of the query (e.g., input parameters of product type, product, desired experience) to search the AR UX system database 98. The query may result in the AR UX system 82 identifying a subset of AR UX workflow datasets that are relevant to the input received by the operator device 54. AR UX workflows may be associated with data, including image data, used by the operator device 54 to present visualizations of steps related to a procedure to perform various types of tasks. Visualizations associated with a respective AR UX workflow dataset may be rendered via a display of the operator device 54 to generate a graphical user interface and to provide overlaid images that communicate safety information, tools, lock-out/tag-out (e.g., electrical isolation) information, procedure steps, or the like to the operator 50 preparing to operate or actively operating on the industrial automation device 20” [0078]; “An AR UX workflow may include steps associated with performing various maintenance procedures or industrial operation activities by an operator via a computing device associated with the operator. That is, the computing device (e.g., mobile phone) may present a live image data stream (e.g., images captured in real-time from an image sensor of the device, video feed) of an industrial automation device and overlay instructions or information related to the industrial automation device on the image data stream. In some cases, the AR UX system may present one or more virtual objects that are overlaid into the image data stream. The virtual objects may be manipulated or animated as part of the AR UX workflow to show portions (e.g., steps, sequential operations) associated with the AR UX workflow. For example, the interactive AR UX system may guide an operator through a maintenance procedure by displaying or modifying one or more virtual objects in the visualization associated with the AR environment presented to the user” [0036]; “In an example embodiment, the AR UX system may guide the operator using a combination of text and image cues presented on the computing device of the operator. In this way, while performing a maintenance activity on an example industrial automation device, such as powering off a component, the operator may receive step-by-step instructions according to the AR UX workflow regarding how to perform the maintenance activity on the industrial automation device (e.g., for the particular model and version of the drive). The AR UX system may present a visualization of the industrial automation device (e.g., virtual object) and manipulate the visualization of the industrial automation device to guide the operator through performance of the maintenance activity. For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off. In some cases, the AR UX system may identify a time that an operation for a current instruction was completed by the operator and, in response to the identification, the AR UX system may automatically advance to a subsequent instruction” [0038]);
authenticating the secondary device based on the authentication data and the operator identifier (“The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94” [0071] [Examiner wants to note that the image data and information from the operator device is used to match with a profile. Since the AR UX system matches the incoming data with profile(s) in the profile data – which includes the operator identifier – it is effectively authenticating both the secondary device and the operator]);
based on the device identifier matching at least one device identifier in the list of device identifiers (“The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071]),
generating fourth image data, wherein the fourth image data comprises: a fourth machine-readable indicia representative of information relating to an error code corresponding to a part or portion of computer code associated with the industrial automation component (“At block 116, the operator device 54 may use the input identifying the product to generate and send a query to the AR UX system 82. The AR UX system 82 may receive the query and use information of the query (e.g., input parameters of product type, product, desired experience) to search the AR UX system database 98. The query may result in the AR UX system 82 identifying a subset of AR UX workflow datasets that are relevant to the input received by the operator device 54. AR UX workflows may be associated with data, including image data, used by the operator device 54 to present visualizations of steps related to a procedure to perform various types of tasks. Visualizations associated with a respective AR UX workflow dataset may be rendered via a display of the operator device 54 to generate a graphical user interface and to provide overlaid images that communicate safety information, tools, lock-out/tag-out (e.g., electrical isolation) information, procedure steps, or the like to the operator 50 preparing to operate or actively operating on the industrial automation device 20” [0078]; “In some cases, the AR UX system 82 may receive an indication of the AR UX workflow dataset to transmit to the operator device 54 from the operator device 54 as opposed to query parameters. The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20. For example, the links, barcodes, or the like may include a code acquired via a scanner, a barcode acquired via the scanner, a digital link to a website. Furthermore, in some cases, the AR UX system 82 may access maintenance schedules or the like for the industrial automation device, and may automatically retrieve the corresponding AR UX workflow dataset to an upcoming or overdue maintenance procedure for transmission to the operator device 54. The above-described features and more are discussed below with reference to interactions between the operator device 54 and the AR UX system 82, such as to select and provide data associated with an AR UX workflow dataset” [0072] [Examiner notes that this text shows scanning of a link, QR code, etc. (machine-readable indicia) that gives information by retrieving the “workflow datasets” and “maintenance schedules”. The maintenance schedules and workflow datasets come from the device’s internal software which is seen as the device’s software. The indicia here connects to software-generated information and the information is a workflow/maintenance dataset from the software so scanning the indicia retrieves data that comes from specific software coming from the device]),
wherein the fourth machine-readable indicia comprises access credentials embedded therein (“In some embodiments, the operator device 54 and/or the AR UX system 82 may receive inputs identifying a product based on a location of the operator device 54, such as a location determined via the location sensor 68, a location determined using location anchors, or the like. The operator device 54 and/or the AR UX system 82 may also receive an input identifying the product based at least in part on the operator device 54 scanning a code, a barcode, following a link, or the like to the identifying information. In some embodiments, the industrial automation device 20 may transmit its identifying information to the operator device 54, such as in response to the operator device 54 being within a defined proximity to the industrial automation device 20 and/or in response to the operator device 54 requesting the identifying information from the industrial automation device 20” [0076]; “The CSR device 316 may display the image data displayed by the operator device 54 on a first portion of its display 70 and may display a locally maintained AR UX workflow visualization on a second portion of its display. When the operator device 54 updates its displayed image data, the CSR device 316 may mirror the update and also update the first portion of its display 70. In some cases, manipulations to the second portion of the display 70 of the CSR device 316 may be used to generate notifications, alerts, overlaid images, or the like to be presented via the display 70 of the operator device 54. In this way, the AR UX system 82 may receive an input from the CSR device 316 corresponding to a manipulation of the second portion of the display 70 of the CSR device 316. The AR UX system 82 may generate an adjusted AR UX workflow dataset based at least in part on the input from the CSR device 316 and may transmit the adjusted AR UX workflow dataset to the operator device 54 and to the CSR device 316. The operator device 54 may update its display 70 with a representation of the manipulation made to the second portion of the display 70 of the CSR device 316 in response to receiving the adjusted AR UX workflow dataset. In some cases, the CSR device 316 may update the first portion of its display 70 with the representation of the manipulation in response to receiving the adjusted AR UX workflow dataset from the AR UX system 82, such that presentation of the AR UX workflow visualizations is mirrored between the display 70 of the operator device 54 and the display 70 of the CSR device 316” [0123]); and
information relating to the operator identifier performing one or more respective operations on the industrial automation component (“At block 116, the operator device 54 may use the input identifying the product to generate and send a query to the AR UX system 82. The AR UX system 82 may receive the query and use information of the query (e.g., input parameters of product type, product, desired experience) to search the AR UX system database 98. The query may result in the AR UX system 82 identifying a subset of AR UX workflow datasets that are relevant to the input received by the operator device 54. AR UX workflows may be associated with data, including image data, used by the operator device 54 to present visualizations of steps related to a procedure to perform various types of tasks. Visualizations associated with a respective AR UX workflow dataset may be rendered via a display of the operator device 54 to generate a graphical user interface and to provide overlaid images that communicate safety information, tools, lock-out/tag-out (e.g., electrical isolation) information, procedure steps, or the like to the operator 50 preparing to operate or actively operating on the industrial automation device 20” [0078]; “The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like” [0040] [Examiner notes that these texts are used to show that the image data comprises information relating to the operator and what operations they are performing via the profile association]); and
displaying the fourth image data via the electronic display of the industrial automation component (“electronic display device for presenting AR UX workflow visualizations” [0060]),
wherein the secondary device is configured to: receive fifth image data representative of the fourth machine-readable indicia (“The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0072]);
retrieve a set of instructions for resolving the error code based on the fifth image data and the authentication data (“In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071]; “In an example embodiment, the AR UX system may guide the operator using a combination of text and image cues presented on the computing device of the operator. In this way, while performing a maintenance activity on an example industrial automation device, such as powering off a component, the operator may receive step-by-step instructions according to the AR UX workflow regarding how to perform the maintenance activity on the industrial automation device (e.g., for the particular model and version of the drive). The AR UX system may present a visualization of the industrial automation device (e.g., virtual object) and manipulate the visualization of the industrial automation device to guide the operator through performance of the maintenance activity. For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off. In some cases, the AR UX system may identify a time that an operation for a current instruction was completed by the operator and, in response to the identification, the AR UX system may automatically advance to a subsequent instruction” [0038]; “Additionally or alternatively, the computing device of the operator may be associated with a profile. The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like. The profile may be used by the AR UX system to pre-filter a query of the workflow database, such that a subset of workflows relevant to the industrial automation system and/or operator are searched by the AR UX system against the parameters. In some embodiments, the operator may input the parameters of the query into the computing device using drop-down menus, radio buttons, or the like, from a defined set of options. Thus, in some cases, the AR UX system may instruct the device to present a subset of options to the operator based at least in part on the profile corresponding to the operator. In this way, the operator is unable to select options that the profile of the operator is ineligible to select. The subset of options for parameters presented to the operator may be based on the profile. The profile may define permissions of the operator, such as operations or equipment that the operator is eligible to adjust. The profile may also include an indication of portions of the industrial automation system that the operator works within or devices that are included within the industrial automation system to filter irrelevant industrial automation devices out from the query. Profile filtering of the workflow database may improve a speed of the query used by the AR UX system when querying the workflow database based on parameters by reducing a subset of workflows that the AR UX system searches with the query. Moreover, profile filtering and other features described herein may reduce the amount of processing power employed by the AR UX system, thereby improving the operation of the AR UX system” [0041] [Examiner notes that these 3 texts shows an operator capturing an image of the industrial automation device, which the AR UX system uses to identify the specific device (based on image data). The AR UX system retrieves and presents step-by-step workflow instructions to the operator for the identified device (retrieving a set of instructions). The operator’s profile defines identity and permissions, ensuring that only authorized operators can access relevant workflows or instructions. Queries are also filtered based on the profile to restrict access to permitted options (based on authentication data)]); and
present the set of instructions The workflow dataset may cause the first device and the second device to display one or more visualizations representative of the one or more instructions, the one or more virtual objects, or both on image data depicted on a first display and on a second display associated with the first device and the second device, respectively. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device, where the input may include one or more additional visualizations, and causes the processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005]).
Mehrotra does not disclose: present the set of instructions comprising software update instructions corresponding to the part of portion of computer code corresponding to the error code via an additional electronic display of the secondary device.
However, Rogati discloses: present the set of instructions comprising software update instructions corresponding to the part of portion of computer code corresponding to the error code via an additional electronic display of the secondary device (“As one example, a software glitch, error, bug or the like may manifest itself as a hardware failure that triggers generation of data representing a hardware-related device error code. Diagnosis at the stage 304 may include accessing the failure diagnosis database using the received device error code as a look-up or search key, for example, and determine that identical or similar devices with the same device error code were not replaced because a root cause of the device error code was related to a software issue rather than a hardware issue, and the software issue may be fixed by a software update instead of device replacement. As another example, diagnosis at the stage 304 may include accessing the failure diagnosis database and determining that the device error code received is associated with a low battery reserve condition on previously diagnosed devices (e.g., the device error code was due to low battery power instead of an actual hardware failure). Accordingly, instead of replacing the device, a message may be communicated to the device (e.g., 110 or 150) instructing a user (e.g., customer 101) to charge the battery of their device using a wall charger or the like” [0037] [Examiner notes that it would have been obvious to a POSITA to modify Mehrotra, which teaches presenting instructions in response to an error code, in view of Rogati, which teaches an error code may correspond to a software-related issue that can be resolved through a software update. Specially this reference teaches using the error code to diagnose a root cause and determining that the issue may be corrected by updating software rather than fixing hardware. Incorporating this teaching would have resulted in presenting software update instructions to the error code since providing update based remediation is a known extension of error-based system wherein such updated inherently correspond to the portion of software associate with said error condition]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mehrotra with the added structure of Rogati in order for the system to cover software/code issues on top of the technical/hardware system of Mehrotra.
Regarding claim 7, a combination of Mehrotra-Rogati discloses the system of claim 6.
Mehrotra further discloses: receiving an indication of the at least one additional device being coupled to the industrial automation component (“The operator device 54 and the CSR device 316 may be communicatively coupled via the network 90. The operator device 54, in response to the menu button 138E receiving an input, may transmit the ongoing AR UX workflow dataset to the CSR device 316 (e.g., such as transmitting an actively rendered portion of the AR UX workflow dataset to the CSR device 316). In some embodiments, the operator device 54 may notify the AR UX system 82 associated with the menu button 138E receiving the input and cause the AR UX system 82 to provision an AR UX workflow visualization to both the CSR device 316 and the operator device 54 and/or to provision the ongoing AR UX workflow visualization to the CSR device 316 on behalf of the operator device 54” [0116]);
establishing a connection between the at least one additional device and the industrial automation component based on a communication protocol (“Indeed, other industrial communication network protocol, such as ETHERNET/IP®, CONTROLNET®, DEVICENET®, and the like, may also be used. In any case, the network 90 may permit the exchange of data in accordance with a protocol. For example, the network 90 may permit the transmission of the AR UX workflow dataset from the AR UX system 82 to the operator device 54” [0063]); and
storing an additional device identifier associated with the at least one additional device and the communication protocol (“Furthermore, in some embodiments, the computing device may receive one or more inputs from the operator specifying parameters for a query of the AR UX workflow database. For example, the inputs may specify parameters associated with a model type of the industrial automation device, a type of device associated with the industrial automation device, an environment that the industrial automation device is operated within” [0040]; “FIG. 2 is a block diagram of an operator device 54 that may be employed in any suitable industrial automation system 10 to access or interface with other industrial automation device 20 and/or computing devices. For example, the operator device 54 may include a communication component 56, a processor 58, a memory 60, a storage 62, input/output (I/O) ports 64, an image sensor 66 (e.g., a camera), a location sensor 68, a display 70, additional sensors (e.g., vibration sensors, temperature sensors), and the like. The communication component 56 may be a wireless or wired communication component that may facilitate communication between the industrial automation device 20, cloud-based computing systems, and other communication capable devices” [0050]; “The I/O ports 64 may be interfaces that couple to other peripheral components such as input devices (e.g., keyboard, mouse), sensors, input/output (I/O) modules, and the like. I/O modules may enable the computing device or other operator devices 54 to communicate with the industrial automation device 20 or other devices in the industrial automation system 10 via the I/O modules” [0054]; “In this way, a profile for the industrial automation system 10 may store indications of related equipment (e.g., part of an overall system), model numbers of the related equipment, operational preferences for the related equipment, or the like for the industrial automation system 10. Thus, when any operator device 54 of the industrial automation system 10 requests an AR UX workflow, the AR UX system 82 may query AR UX workflow datasets associated with equipment related or operations performed by the industrial automation system 10 without querying AR UX workflow datasets associated with equipment unrelated or operations not performed by the industrial automation system 10. Profiles may be stored in a profile database 94 accessible by the AR UX system 82” [0066] [Examiner notes that the device itself has information (identifier) stored including the communication component (where all communications with device is controlled including the communication protocol) and the I/O ports. It also has a profile that includes more specific information including the filtering aspect to make sure the device communicates with "alike" equipment]).
Regarding claim 8, a combination of Mehrotra-Rogati discloses the system of claim 7.
Mehrotra further discloses: wherein the updated connection data comprises the communication protocol, the additional device identifier, or a combination thereof (“Indeed, other industrial communication network protocol, such as ETHERNET/IP®, CONTROLNET®, DEVICENET®, and the like, may also be used. In any case, the network 90 may permit the exchange of data in accordance with a protocol. For example, the network 90 may permit the transmission of the AR UX workflow dataset from the AR UX system 82 to the operator device 54” [0063]).
Regarding claim 10, a combination of Mehrotra-Rogati discloses the system of claim 6.
Mehrotra further discloses: wherein the access credentials permit the secondary device to access additional data associated with the industrial automation component via an external network (“To obtain the image data, the operator device 54 may communicate with the AR UX system 82 through a network 90 and retrieve the image data of the AR UX workflow dataset based on input data. The network 90 may include any wired or wireless network that may be implemented as a local area network (LAN), a wide area network (WAN), and the like. It should be noted that any suitable network may be employed in the embodiments described herein. Indeed, other industrial communication network protocol, such as ETHERNET/IP®, CONTROLNET®, DEVICENET®, and the like, may also be used. In any case, the network 90 may permit the exchange of data in accordance with a protocol. For example, the network 90 may permit the transmission of the AR UX workflow dataset from the AR UX system 82 to the operator device 54” [0063]).
Regarding claim 11, a combination of Mehrotra-Rogati discloses the system of claim 6.
Mehrotra further discloses: updating the first image data to include the fourth machine-readable indicia (“The CSR device 316 may display the image data displayed by the operator device 54 on a first portion of its display 70 and may display a locally maintained AR UX workflow visualization on a second portion of its display. When the operator device 54 updates its displayed image data, the CSR device 316 may mirror the update and also update the first portion of its display 70. In some cases, manipulations to the second portion of the display 70 of the CSR device 316 may be used to generate notifications, alerts, overlaid images, or the like to be presented via the display 70 of the operator device 54. In this way, the AR UX system 82 may receive an input from the CSR device 316 corresponding to a manipulation of the second portion of the display 70 of the CSR device 316. The AR UX system 82 may generate an adjusted AR UX workflow dataset based at least in part on the input from the CSR device 316 and may transmit the adjusted AR UX workflow dataset to the operator device 54 and to the CSR device 316. The operator device 54 may update its display 70 with a representation of the manipulation made to the second portion of the display 70 of the CSR device 316 in response to receiving the adjusted AR UX workflow dataset. In some cases, the CSR device 316 may update the first portion of its display 70 with the representation of the manipulation in response to receiving the adjusted AR UX workflow dataset from the AR UX system 82, such that presentation of the AR UX workflow visualizations is mirrored between the display 70 of the operator device 54 and the display 70 of the CSR device 316” [0123]).
Claim 12 recites substantially the same limitation as claim 11, for implementing the corresponding method, therefore it is rejected under the same rationale.
Regarding claim 13, a combination of Mehrotra-Rogati discloses the system of claim 6.
Mehrotra further discloses: receiving a signal indicative of the secondary device scanning the first machine-readable indicia (“In some cases, the AR UX system 82 may receive an indication of the AR UX workflow dataset to transmit to the operator device 54 from the operator device 54 as opposed to query parameters. The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0072]); and
in response to receiving the signal, generating fifth image data, wherein the fifth image data comprises a fifth machine-readable indicia ([recites substantially the same limitation as claim 11, for implementing the corresponding method, therefore it is rejected under the same rationale]).
Regarding claim 21, Mehrotra discloses: A non-transitory computer-readable medium comprising instructions that, when executed by processing circuitry of an industrial automation component, cause the processing circuitry to:
acquire first image data associated with a secondary device via an image sensor disposed on the industrial automation component (“receive an input to the image data from the second device” [0005]; “the operator device 54 may, at block 226, receive updated image data of the industrial automation device 20 from the image sensor 66” [0104]),
wherein the first image data comprises a first machine-readable indicia rendered on a display of the secondary device, the second machine-readable indicia representative of an identity of the secondary device and an operator; determine a device identifier and an operator identifier associated with identifying the secondary device and identifying the operator, respectively, based on the first image data (“The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50… The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0071-0072]; “The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like” [0040] “The workflow dataset may cause the first device and the second device to display one or more visualizations representative of the one or more instructions, the one or more virtual objects, or both on image data depicted on a first display and on a second display associated with the first device and the second device, respectively. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device, where the input may include one or more additional visualizations, and causes the processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005] [Examiner notes that the device identifier is stored within the profile which is known after processing the image data. Image data is captured by an image sensor and that image data can contain machine-readable indicia. The indicia is used to identify devices – including multiple different devices. The system will be able to recognize and differentiate between devices and the user can capture different images, each with different device identifiers. Examiner also wants to note that the profile also contains the operator identifier. Examiner also notes that the operator identifier already implicitly shows an operator]);
identify authentication data based on a comparison between the device identifier and a stored list of device identifiers, wherein the authentication data comprises access credentials associated with one or more operators performing one or more operations on the industrial automation component for each device identifier of the stored list of device identifiers (“Additionally or alternatively, the computing device of the operator may be associated with a profile. The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like. The profile may be used by the AR UX system to pre-filter a query of the workflow database, such that a subset of workflows relevant to the industrial automation system and/or operator are searched by the AR UX system against the parameters. In some embodiments, the operator may input the parameters of the query into the computing device using drop-down menus, radio buttons, or the like, from a defined set of options. Thus, in some cases, the AR UX system may instruct the device to present a subset of options to the operator based at least in part on the profile corresponding to the operator. In this way, the operator is unable to select options that the profile of the operator is ineligible to select. The subset of options for parameters presented to the operator may be based on the profile. The profile may define permissions of the operator, such as operations or equipment that the operator is eligible to adjust. The profile may also include an indication of portions of the industrial automation system that the operator works within or devices that are included within the industrial automation system to filter irrelevant industrial automation devices out from the query. Profile filtering of the workflow database may improve a speed of the query used by the AR UX system when querying the workflow database based on parameters by reducing a subset of workflows that the AR UX system searches with the query” [0041]; “In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071] [Examiner notes that after the device identifier is identified, the profile itself (where the identifier resides) is then used to filter which devices can has the correct authentication permissions to interact with the device. Examiner also wants to note that each operator is tied to specific devices of the system, supporting the notion of access being associated with device identifiers, and the system also delivers device-specific operations when the device is identified which implies that the AR UX system matches the device ID to stored operations]);
authenticate the secondary device based on the authentication data and the operator identifier (“The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94” [0071] [Examiner wants to note that the image data and information from the operator device is used to match with a profile. Since the AR UX system matches the incoming data with profile(s) in the profile data – which includes the operator identifier – it is effectively authenticating both the secondary device and the operator]);
generate second image data in response to authenticating the secondary device, wherein the second image data comprises a second machine-readable indicia representative of information relating to an error code corresponding to a part of portion of computer code associated with the industrial automation component and the operator identifier performing one or more respective operations on the industrial automation component; and present the second image data via the electronic display of the industrial automation component (Fig. 6; “At block 116, the operator device 54 may use the input identifying the product to generate and send a query to the AR UX system 82. The AR UX system 82 may receive the query and use information of the query (e.g., input parameters of product type, product, desired experience) to search the AR UX system database 98. The query may result in the AR UX system 82 identifying a subset of AR UX workflow datasets that are relevant to the input received by the operator device 54. AR UX workflows may be associated with data, including image data, used by the operator device 54 to present visualizations of steps related to a procedure to perform various types of tasks. Visualizations associated with a respective AR UX workflow dataset may be rendered via a display of the operator device 54 to generate a graphical user interface and to provide overlaid images that communicate safety information, tools, lock-out/tag-out (e.g., electrical isolation) information, procedure steps, or the like to the operator 50 preparing to operate or actively operating on the industrial automation device 20” [0078]; “In some cases, the AR UX system 82 may receive an indication of the AR UX workflow dataset to transmit to the operator device 54 from the operator device 54 as opposed to query parameters. The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20. For example, the links, barcodes, or the like may include a code acquired via a scanner, a barcode acquired via the scanner, a digital link to a website. Furthermore, in some cases, the AR UX system 82 may access maintenance schedules or the like for the industrial automation device, and may automatically retrieve the corresponding AR UX workflow dataset to an upcoming or overdue maintenance procedure for transmission to the operator device 54. The above-described features and more are discussed below with reference to interactions between the operator device 54 and the AR UX system 82, such as to select and provide data associated with an AR UX workflow dataset” [0072] [Examiner notes that this text shows scanning of a link, QR code, etc. (machine-readable indicia) that gives information by retrieving the “workflow datasets” and “maintenance schedules”. The maintenance schedules and workflow datasets come from the device’s internal software which is seen as the device’s software. The indicia here connects to software-generated information and the information is a workflow/maintenance dataset from the software so scanning the indicia retrieves data that comes from specific software coming from the device]; “In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071] [Examiner notes that the first text shows the system rendering visualization and images on the operator device’s display which is the generating third image data; it also shows it is in response to authenticating since the system flow implies that the operator device must be authenticated and associated with a valid operator before queries and rendering can happen. The visualizations are tied to actions the operator is taking. The second text is added to show more support that the image data is being generated in response to authenticating the device as the scanning action serves as a form of authentication/identity confirmation. The system only responds with relevant workflow data after receiving and validating that input from the operator device which is reasonably interprets as a form of authentication as it is using the device identity/access permissions. The third text is used to show that the image data comprises information relating to the operator via the profile association]),
wherein the secondary device is configured to: receive third image data representative of the second machine-readable indicia (“The operator device 54, for example, may scan a link, a barcode (e.g., a QR code, a matrix barcode), or a radio frequency identification tag (RFID tag), or the like, presented on the industrial automation device 20 to retrieve the AR UX workflow dataset corresponding to the industrial automation device 20” [0072]);
retrieve a set of instructions for resolving the error code based on the third image data and the authentication data (“In addition to improving the query of the AR UX system database 98, the location of the operator device 54 may be used by the AR UX system 82 to identify industrial automation devices 20 and/or other features in an image captured by the image sensor 66 of the operator device 54. In this way, the AR UX system 82 may use the captured image data received from the operator device 54 to identify devices (e.g., industrial automation devices 20, other suitable identifiable devices within an industrial automation system 10) in the captured image data. When the AR UX system 82 identifies the devices in the captured image data, the AR UX system 82 may send information associated with the identified devices to the operator device 54. For example, the operator 50 may desire to view AR UX workflow instructions for an industrial automation device 20 and may initiate the query by capturing a photograph of the industrial automation device 20. The operator device 54 may transmit the image data corresponding to the photograph to the AR UX system 82. The AR UX system 82 may receive the image data and use image processing operations to identify and match the image data of the industrial automation device 20 to known industrial automation devices 20 of the particular industrial automation system 10 (e.g., identified in the profiles stored in the profile database 94). Once a match is determined, the AR UX system 82 may transmit information corresponding to the now-identified industrial automation device 20 to the operator device 54 for reference by the operator 50. For example, in response to matching the image data of the industrial automation device 20 to data stored in the AR UX system database 98, the AR UX system 82 may transmit information to the operator device 54, such as AR UX workflow dataset corresponding to the industrial automation device 20. In this way, the AR UX system 82 and/or the operator device 54 may receive image data from the image sensor 66 of a likeness of the industrial automation device 20, may compare the image data to stored image data (e.g., image data stored in AR UX database 98) corresponding to industrial automation products of the industrial automation system 10, and may determine the industrial automation device 20 to be one of the industrial automation products based at least in part on the image data matching the stored image data (e.g., matching based on an amount of matching or substantially similar image data being greater than a threshold amount of matching data)” [0071]; “In an example embodiment, the AR UX system may guide the operator using a combination of text and image cues presented on the computing device of the operator. In this way, while performing a maintenance activity on an example industrial automation device, such as powering off a component, the operator may receive step-by-step instructions according to the AR UX workflow regarding how to perform the maintenance activity on the industrial automation device (e.g., for the particular model and version of the drive). The AR UX system may present a visualization of the industrial automation device (e.g., virtual object) and manipulate the visualization of the industrial automation device to guide the operator through performance of the maintenance activity. For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off. In some cases, the AR UX system may identify a time that an operation for a current instruction was completed by the operator and, in response to the identification, the AR UX system may automatically advance to a subsequent instruction” [0038]; “Additionally or alternatively, the computing device of the operator may be associated with a profile. The profile may include parameters that correspond to or define an identity of the operator, an identity of the industrial automation system (e.g., a company that owns or operates the industrial automation system), or the like. The profile may be used by the AR UX system to pre-filter a query of the workflow database, such that a subset of workflows relevant to the industrial automation system and/or operator are searched by the AR UX system against the parameters. In some embodiments, the operator may input the parameters of the query into the computing device using drop-down menus, radio buttons, or the like, from a defined set of options. Thus, in some cases, the AR UX system may instruct the device to present a subset of options to the operator based at least in part on the profile corresponding to the operator. In this way, the operator is unable to select options that the profile of the operator is ineligible to select. The subset of options for parameters presented to the operator may be based on the profile. The profile may define permissions of the operator, such as operations or equipment that the operator is eligible to adjust. The profile may also include an indication of portions of the industrial automation system that the operator works within or devices that are included within the industrial automation system to filter irrelevant industrial automation devices out from the query. Profile filtering of the workflow database may improve a speed of the query used by the AR UX system when querying the workflow database based on parameters by reducing a subset of workflows that the AR UX system searches with the query. Moreover, profile filtering and other features described herein may reduce the amount of processing power employed by the AR UX system, thereby improving the operation of the AR UX system” [0041] [Examiner notes that these 3 texts shows an operator capturing an image of the industrial automation device, which the AR UX system uses to identify the specific device (based on image data). The AR UX system retrieves and presents step-by-step workflow instructions to the operator for the identified device (retrieving a set of instructions). The operator’s profile defines identity and permissions, ensuring that only authorized operators can access relevant workflows or instructions. Queries are also filtered based on the profile to restrict access to permitted options (based on authentication data)]); and
present the set of instructions display of the secondary device (“The workflow dataset may cause the first device and the second device to display one or more visualizations representative of the one or more instructions, the one or more virtual objects, or both on image data depicted on a first display and on a second display associated with the first device and the second device, respectively. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device, where the input may include one or more additional visualizations, and causes the processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005]).
Mehrotra does not disclose: present the set of instructions comprising software update instructions corresponding to the part of portion of computer code corresponding to the error code via an additional electronic display of the secondary device.
However, Rogati discloses: present the set of instructions comprising software update instructions corresponding to the part of portion of computer code corresponding to the error code via an additional electronic display of the secondary device (“As one example, a software glitch, error, bug or the like may manifest itself as a hardware failure that triggers generation of data representing a hardware-related device error code. Diagnosis at the stage 304 may include accessing the failure diagnosis database using the received device error code as a look-up or search key, for example, and determine that identical or similar devices with the same device error code were not replaced because a root cause of the device error code was related to a software issue rather than a hardware issue, and the software issue may be fixed by a software update instead of device replacement. As another example, diagnosis at the stage 304 may include accessing the failure diagnosis database and determining that the device error code received is associated with a low battery reserve condition on previously diagnosed devices (e.g., the device error code was due to low battery power instead of an actual hardware failure). Accordingly, instead of replacing the device, a message may be communicated to the device (e.g., 110 or 150) instructing a user (e.g., customer 101) to charge the battery of their device using a wall charger or the like” [0037] [Examiner notes that it would have been obvious to a POSITA to modify Mehrotra, which teaches presenting instructions in response to an error code, in view of Rogati, which teaches an error code may correspond to a software-related issue that can be resolved through a software update. Specially this reference teaches using the error code to diagnose a root cause and determining that the issue may be corrected by updating software rather than fixing hardware. Incorporating this teaching would have resulted in presenting software update instructions to the error code since providing update based remediation is a known extension of error-based system wherein such updated inherently correspond to the portion of software associate with said error condition]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mehrotra with the added structure of Rogati in order for the system to cover software/code issues on top of the technical/hardware system of Mehrotra.
Regarding claim 22, a combination of Mehrotra-Rogati discloses the system of claim 21.
Mehrotra further discloses: in response to the comparison being indicative of the device identifier not being present in the stored list of device identifiers, generate a notification comprising an additional error code; and transmit the notification to the secondary device (“An AR UX workflow may include steps associated with performing various maintenance procedures or industrial operation activities by an operator via a computing device associated with the operator. That is, the computing device (e.g., mobile phone) may present a live image data stream (e.g., images captured in real-time from an image sensor of the device, video feed) of an industrial automation device and overlay instructions or information related to the industrial automation device on the image data stream. In some cases, the AR UX system may present one or more virtual objects that are overlaid into the image data stream. The virtual objects may be manipulated or animated as part of the AR UX workflow to show portions (e.g., steps, sequential operations) associated with the AR UX workflow. For example, the interactive AR UX system may guide an operator through a maintenance procedure by displaying or modifying one or more virtual objects in the visualization associated with the AR environment presented to the user” [0036]; “In an example embodiment, the AR UX system may guide the operator using a combination of text and image cues presented on the computing device of the operator. In this way, while performing a maintenance activity on an example industrial automation device, such as powering off a component, the operator may receive step-by-step instructions according to the AR UX workflow regarding how to perform the maintenance activity on the industrial automation device (e.g., for the particular model and version of the drive). The AR UX system may present a visualization of the industrial automation device (e.g., virtual object) and manipulate the visualization of the industrial automation device to guide the operator through performance of the maintenance activity. For example, the AR UX system may sequentially show the operator a first location to check a status, a second location to disconnect a first switch, a third location to disconnect a second switch, and a fourth location to verify an electrical status of the drive used to verify the drive is powered off. In some cases, the AR UX system may identify a time that an operation for a current instruction was completed by the operator and, in response to the identification, the AR UX system may automatically advance to a subsequent instruction” [0038]; “The maintenance history may be referenced by other industrial automation devices 20, control systems, or the like to determine whether the industrial automation device 20 is due for servicing (e.g., mechanical or electrical maintenance work) or the like” [0109] [Examiner notes that error codes are used to show a user what is not properly functioning in order to properly get to the root of the problem. This text is describing that by giving a text status and location via display with images cues in order for the user to see what is wrong and fix it. This shows the system’s ability to detect a problem, whether it is connection-based or external, and potentially flag it]).
Claim 23 recites substantially the same limitation as claim 2, therefore it is rejected under the same rationale.
Claim 24 recites substantially the same limitation as claim 2, therefore it is rejected under the same rationale.
Regarding claim 26, a combination of Mehrotra-Rogati discloses the system of claim 21.
Mehrotra further discloses: determine a user access level based on the device identifier, the operator identifier, or both (“For example, a profile may include information about a permission or skill level of the operator 50, such that if the operator 50 is not trained on an industrial automation device 20 or is otherwise unpermitted to adjust operation of the industrial automation device 20, the AR UX system 82 may not query AR UX workflow datasets related to the industrial automation device 20 since those AR UX workflow datasets may be irrelevant to the operator 50” [0066]).
Regarding claim 27, a combination of Mehrotra-Rogati discloses the system of claim 1.
Mehrotra further discloses: wherein the secondary device is configured to send the set of instructions to the industrial automation component, and wherein the set of instructions is configured to adjust an operation of the industrial automation component (“When additional AR UX workflow data is available, the operator device may, at block 232, render and animate the subsequent AR UX workflow instruction. The additional AR UX workflow data may correspond to a second operation instructing a second adjustment to the industrial automation device 20 to be performed after the first operation instructing a first adjustment to the industrial automation device 20. This process of checking for and retrieving subsequent portions of the AR UX workflow dataset permits the operator device 54 to progress through the AR UX workflow instructions, and thus progress through providing instructions for the corresponding procedure (e.g., experience type) to the operator 50” [0106] [Examiner notes that these AR UX workflow instructions are interpreted as the set of instructions that the secondary device sends for execution, resulting in the adjustment of the industrial automation component]).
Regarding claim 28, a combination of Mehrotra-Rogati discloses the system of claim 1.
Mehrotra further discloses: wherein the secondary device is configured to transmit the set of instructions The workflow dataset may cause the first device and the second device to display one or more visualizations representative of the one or more instructions, the one or more virtual objects, or both on image data depicted on a first display and on a second display associated with the first device and the second device, respectively. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device, where the input may include one or more additional visualizations, and causes the processor to transmit the one or more additional visualizations to the first device for display via the first display” [0005]).
Mehrotra does not disclose: wherein the secondary device is configured to transmit the set of instructions comprising the software update instructions to the industrial automation component, wherein a computing system of the industrial automation component is configured to update a software application based on the software update instructions.
However, Rogati discloses: wherein the secondary device is configured to transmit the set of instructions comprising the software update instructions to the industrial automation component, wherein a computing system of the industrial automation component is configured to update a software application based on the software update instructions (“As one example, a software glitch, error, bug or the like may manifest itself as a hardware failure that triggers generation of data representing a hardware-related device error code. Diagnosis at the stage 304 may include accessing the failure diagnosis database using the received device error code as a look-up or search key, for example, and determine that identical or similar devices with the same device error code were not replaced because a root cause of the device error code was related to a software issue rather than a hardware issue, and the software issue may be fixed by a software update instead of device replacement. As another example, diagnosis at the stage 304 may include accessing the failure diagnosis database and determining that the device error code received is associated with a low battery reserve condition on previously diagnosed devices (e.g., the device error code was due to low battery power instead of an actual hardware failure). Accordingly, instead of replacing the device, a message may be communicated to the device (e.g., 110 or 150) instructing a user (e.g., customer 101) to charge the battery of their device using a wall charger or the like” [0037] [Examiner notes that it would have been obvious to a POSITA to modify Mehrotra, which teaches presenting instructions in response to an error code, in view of Rogati, which teaches an error code may correspond to a software-related issue that can be resolved through a software update. Specially this reference teaches using the error code to diagnose a root cause and determining that the issue may be corrected by updating software rather than fixing hardware. Incorporating this teaching would have resulted in presenting software update instructions to the error code since providing update based remediation is a known extension of error-based system wherein such updated inherently correspond to the portion of software associate with said error condition]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mehrotra with the added structure of Rogati in order for the system to cover software/code issues on top of the technical/hardware system of Mehrotra.
Regarding claim 29, a combination of Mehrotra-Rogati discloses the system of claim 21.
Mehrotra further discloses: wherein the secondary device is configured to transmit an additional instruction to the industrial automation component configured to enable the industrial automation component to receive
Mehrotra does not disclose: wherein the secondary device is configured to transmit an additional instruction to the industrial automation component configured to enable the industrial automation component to receive the software update instructions.
However, Rogati discloses: wherein the secondary device is configured to transmit an additional instruction to the industrial automation component configured to enable the industrial automation component to receive the software update instructions (“As one example, a software glitch, error, bug or the like may manifest itself as a hardware failure that triggers generation of data representing a hardware-related device error code. Diagnosis at the stage 304 may include accessing the failure diagnosis database using the received device error code as a look-up or search key, for example, and determine that identical or similar devices with the same device error code were not replaced because a root cause of the device error code was related to a software issue rather than a hardware issue, and the software issue may be fixed by a software update instead of device replacement. As another example, diagnosis at the stage 304 may include accessing the failure diagnosis database and determining that the device error code received is associated with a low battery reserve condition on previously diagnosed devices (e.g., the device error code was due to low battery power instead of an actual hardware failure). Accordingly, instead of replacing the device, a message may be communicated to the device (e.g., 110 or 150) instructing a user (e.g., customer 101) to charge the battery of their device using a wall charger or the like” [0037] [Examiner notes that it would have been obvious to a POSITA to modify Mehrotra, which teaches presenting instructions in response to an error code, in view of Rogati, which teaches an error code may correspond to a software-related issue that can be resolved through a software update. Specially this reference teaches using the error code to diagnose a root cause and determining that the issue may be corrected by updating software rather than fixing hardware. Incorporating this teaching would have resulted in presenting software update instructions to the error code since providing update based remediation is a known extension of error-based system wherein such updated inherently correspond to the portion of software associate with said error condition]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mehrotra with the added structure of Rogati in order for the system to cover software/code issues on top of the technical/hardware system of Mehrotra.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
Homan et al. (US 2022/0224698 A1) teaches a method, performed by at least one processor, comprising generating a first set of images representative of configuration data associated with an industrial automation system. An electronic display may display at least one image of the first set of images. The processor may detect a change in the configuration data and retrieve updated configuration data from the industrial automation system. The processor may encode the updated configuration data into a second set of images and display at least one additional image of the second set of images on the electronic display.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARON MATTHEWOS WORKU whose telephone number is (703)756-1761. The examiner can normally be reached Monday - Friday, 9:30am - 6:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Linglan Edwards can be reached on 571-270-5440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARON MATTHEWOS WORKU/Examiner, Art Unit 2408 /LINGLAN EDWARDS/Supervisory Patent Examiner, Art Unit 2408
Prosecution Insights