CTNF 18/860,769 CTNF 83016 DETAILED ACTION This action is in response to communications: Preliminary-Amendment filed October 28, 2024. Claims 1-19 are pending in this case. Claims 1-7, 9-12, 14, and 16-19 have been newly amended. No claims have been newly added or cancelled. This action is made Non-Final. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Priority 02-26 AIA Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. Information Disclosure Statement The information disclosure statements (IDS) submitted on November 19, 2024 and December 11, 2024 were filed after the filing date of the application on October 28, 2024. The submissions are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner. Drawings 06-37 AIA The drawings were received on October 28, 2024 . These drawings are accepted . Claim Objections 07-29-01 AIA Claim s 1 and 19 are objected to because of the following informalities: Claims 1 and 19 similarly recite, “…augmented reality , AR, glasses…” but is suggested to recite, “…augmented reality (AR) glasses…” with “AR” in parentheses, representing an acronym . Appropriate correction is required. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1-9, 14, 15, 18, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mehrotra et al. (US 2021/0149889) . As to claim 1, Mehrotra et al. disclose an automation system for controlling an automated production process of an industrial plant ( Figure 1, [0044] notes example of an industrial automation system 10 employed by a food manufacturer, but may include any industry including automotive, mining, hydrocarbon production, manufacturing, and the like ), the industrial plant comprising a plurality of equipment for performing the automated production process ( [0045]-[0048] notes exemplary food manufacturer may include equipment such as silos 12, tanks 14, mixer 18, industrial automation devices 20, depositor 22, conveyor 24, oven 26, cooling tunnel 28, tray loader 30, tray wrapper 32, case packer 34, palletizer 36, and shrink wrapper 38, where [0049] further notes industrial automation devices 20 used to control operations of each of the equipment in the industrial automation system 10 ), the plurality of equipment being spatially distributed in the industrial plant ( Figure 1 illustrates equipment spatially distributed in the food manufacturer ), the automation system ( e.g. industrial automation system 10 employing an operator device 54, further illustrated in Figure 2, further implemented in Figure 3 ) comprising: an augmented reality system ( Figure 3, system comprising each of the components below further including augmented reality (AR) user experience (UX) system 82 ) comprising: a database system ( e.g. AR UX system 82 with AR UX system database 98 ) configured to store data associated with respective equipment of the plurality of equipment ( e.g. AR UX system database 98 comprises AR UX workflow database 92, profile database 94, and location anchor database 96, where [0066] notes AR UX workflow database 92 stores AR UX workflow datasets corresponding to different types of equipment, [0067] notes profile database 94 stores profiles, e.g. information about a permission or skill level of an operator 50 and/or indications of related equipment, model numbers of the related equipment, operational preferences for the related equipment, and [0070], [0071] notes location anchor database 96 stores location anchors regarding location, e.g. proximity, of equipment ), the database system comprising a task manager configured to manage processes with respect to one or more equipment ( [0060] notes AR UX system 82 may include a processor 58 for performing operations and executing instructions stored in a memory 60, such that processor 58 of the AR UX system 82 may retrieve an AR UX workflow dataset and initiate transmission of the AR UX workflow dataset to the operator device 54, thus considered the AR UX system 82 may be the task manager as described ), the data comprising a list of processes managed by the task manager ( e.g. AR UX workflow datasets managed by AR UX system 82 ), each process comprising one or more tasks to be performed with respect to the one or more equipment ( [0059] notes an AR UX workflow dataset may include data corresponding to virtual objects as part of a series or sequence of sequential visualizations to communicate sequentially presented instructions (e.g. AR UX workflow instructions) overlaid on real-time obtained and displayed images to communicated an operation or procedure to the operator 50 via dynamically generated instructions ); augmented reality, AR, glasses ( e.g. operator device 54 as smart glasses ) configured to display the data ( e.g. to display AR UX workflow dataset as visualizations via display 70 )( [0050] notes operator device 54 may include a display 70, where [0057] notes display 70 may depict visualizations, where [0061] further notes operator device 54 may use the AR UX workflow dataset to render a visualization 84 that includes a virtual representation of an industrial automation device 86 in an augmented reality (AR) environment as an AR UX workflow visualization, the operator device 54 may take the form of smart glasses for presenting AR UX workflow visualizations to the operator 50 ); and an executable program logic ( e.g. processor 58 ) coupled to the AR glasses ( e.g. part of operator device 54 ) and the database system ( e.g. in communication with AR UX system 82 comprising AR UX system database 98 )( [0052], [0054] notes processor 58 executes processor executable code and/or software applications, e.g. stored in memory 60 or storage 62 ), wherein the executable program logic is configured to ( e.g. Figure 4 ): receive, from the database system ( e.g. receive, from AR UX system database 98 ), the list of processes for display to a user of the AR glasses ( e.g. the AR UX workflow dataset for display to operator 50 of operator device 54 ), at least one of the processes assigned to the user of the AR glasses ( e.g. at least an AR UX workflow dataset assigned to the user ), receive a selection, by the user of the AR glasses ( e.g. receive a selection, by the operator 50 of operator device 54 ), of a displayed process assigned to the user ( e.g. of a displayed workflow dataset assigned to the operator 50 )( step 118, [0081] notes operator device 54 may receive AR UX workflow datasets from the AR UX system 82, e.g. as multiple AR UX workflow dataset options, where [0065] further notes AR UX system 82 may transmit an indication of available AR UX workflow datasets to the operator device 54, such that a selection of an AR UX workflow dataset may be made based on the available AR UX workflows, e.g. AR UX system 82 may receive the request for an AR UX workflow from the operator device 54, determine available AR UX workflow datasets for presentation on the operator device 54, generate indications of the available AR UX workflows, transmit the indications of the available AR UX workflow datasets to the operator device 54, and receive a selection of the available AR UX workflow datasets via the operator device 54 selecting one of the indications, where [0067] further notes AR UX system 82 may reference stored profiles to determine a subset of AR UX workflow datasets that are relevant to be queried, e.g. a profile may include information about a permission or skill level of the operator 50, such that if the operator 50 is not trained on an industrial automation device 20 or is otherwise unpermitted to adjust operation of the industrial automation device 20, the AR UX system 82 may not query AR UX workflow datasets related to the industrial automation device 20 since those AR UX workflow datasets may be irrelevant to the operator 50, thus may be considered to processes “assigned” to the operator 50 ), send the selected process to the database system ( e.g. send the selected AR UX workflow dataset to the AR UX system database 98 of the AR UX system 82 )( e.g. as noted above, [0065] notes AR UX system 82 may receive a selection of the available AR UX workflow datasets via the operator device 54 selecting one of the indications ), and receive, from the database system ( e.g. receive from the AR UX system database 98 of the AR UX system 82 ), additional data corresponding to the selected process for display to the user of the AR glasses ( e.g. data corresponding to the selected AR UX workflow dataset for display to the operator 50 of the operator device 54 )( step 120, [0082] notes the operator device 54 may display an AR UX workflow visualization 84 via the display 70, e.g. overlaying the AR UX workflow dataset over image data, where [0067] further notes selecting the AR UX workflow dataset may notify the AR UX system 82 to initiate a download of data corresponding to the selected AR UX workflow dataset to the operator device 54, the AR UX system 82 then transmits at least a portion of data corresponding to the selected available AR UX workflow dataset to the operator device 54, where [0070] notes each of databases 92, 94, and 96 of AR UX system database 98 accessed, e.g. queried, by the AR UX system 82 when performing operations to provide the operator device 54 with an AR UX workflow dataset ). As noted above, Mehrotra et al. discloses each of augmented reality (AR) user experience (UX) system 82, AR UX system database 98, operator devices 54 in communication via network 90 such that the operations as described above may be performed, e.g. providing AR UX workflow datasets to the operator device 54 utilizing AR UX system 82 and AR UX system database 98, thus obvious that one of ordinary skill in the art at the time of the invention to incorporate these components as an “augmented reality system,” yielding predictable results, without changing the scope of the invention. As to claim 2, Mehrotra et al. disclose each process in the displayed list of processes comprises one or more selected from the group consisting of a user assignment, a process line and/or process unit assignment, a description of the process ( e.g. Figures 5-12 and 14-22 illustrate various examples of experience navigator (EN) application graphical user interfaces (GUI) 132 of the operator device 54 comprising visualizations, e.g. AR UX workflow datasets ), a process target time, a process accept time, a process finish time, a process status ( e.g. [0077], [0078] notes AR UX system 82 may determine and reference a stored maintenance pattern to identify a time at which a maintenance operation is to be performed ) and a user-selectable task link, the task link enabling display in the AR glasses of a task of the one or more tasks ( Figure 6, [0085] notes the indications corresponding to the AR UX workflow datasets may include selectable text describing the AR UX workflow dataset, selectable images describing the AR UX workflow dataset, or the like )( e.g. as noted in claim 1, [0059] notes an AR UX workflow dataset may include data corresponding to virtual objects as part of a series or sequence of sequential visualizations to communicate sequentially presented instructions (e.g. AR UX workflow instructions) overlaid on real-time obtained and displayed images to communicated an operation or procedure to the operator 50 via dynamically generated instructions, where Figures 5-12 and 14-22 illustrate various examples of experience navigator (EN) application graphical user interfaces (GUI) 132 of the operator device 54 comprising visualizations, e.g. AR UX workflow datasets ). As to claim 3, Mehrotra et al. disclose the executable program logic is configured to receive the selection of the displayed process assigned to the user by receiving a selection of the user-selectable task link and send the selected process to the database system by sending the selected task link to the database system, wherein the additional data comprises a first task of the one or more tasks associated with the selected task link ( e.g. as noted in claim 1, indications of the available AR UX workflow datasets are sent to the operator device 54 where a selection of an AR UX workflow dataset made by the operator 50 of the operator device 54 via the operator device 54 and sent to the AR UX system database 98, e.g. to initiate a download of the selected AR UX workflow dataset, where Figure 6, [0085] notes the indications corresponding to the AR UX workflow datasets may include selectable text describing the AR UX workflow dataset, selectable images describing the AR UX workflow dataset, or the like ). As to claim 4, Mehrotra et al. disclose the first task displayed to the user comprises one or more selected from the group consisting of a description of the first task, a detailed description of the first task ( Figures 5-12 and 14-22 illustrate various examples of experience navigator (EN) application graphical user interfaces (GUI) 132 of the operator device 54 comprising visualizations, e.g. AR UX workflow datasets ), a user-selectable video link of the first task, one or more photos of the one or more equipment on which the first task is to be performed, a user- selectable link for uploading a photo ( e.g. Figure 8 illustrate prompt for operator 50 to point camera of operator device 54 to capture photograph/video ) and a user-selectable task completion link confirming completion of the first task by the user ( Figure 13, see claim 5 below, where Figure 6, [0085] further notes the indications corresponding to the AR UX workflow datasets may include selectable text describing the AR UX workflow dataset, selectable images describing the AR UX workflow dataset, or the like )( e.g. as noted in claim 1, AR UX workflow dataset may include data corresponding to virtual objects as part of a series or sequence of sequential visualizations to communicate sequentially presented instructions (e.g. AR UX workflow instructions) overlaid on real-time obtained and displayed images to communicated an operation or procedure to the operator 50 via dynamically generated instructions, where Figures 5-12 and 14-22 illustrate various examples of experience navigator (EN) application graphical user interfaces (GUI) 132 of the operator device 54 comprising visualizations, e.g. AR UX workflow datasets ). As to claim 5, Mehrotra et al. disclose upon selection, by the user, of the user-selectable task completion link ( e.g. upon operator 50 selecting selectable text or image to confirm completion of task ), the executable logic is further configured to: receive, from the database system, a next task for display to the user of the AR glasses ( e.g. receive, from the AR UX system database 98 of AR UX system 82, a subsequent portion of the AR UX workflow dataset for display on the operator device 54 ), and upon reception of a signal indicating a selection by the user of the task completion link confirming completion of the next task by the user ( e.g. upon confirming completion of subsequent portion of the AR UX workflow dataset ), iteratively receive-each subsequent next task upon reception of a signal indicating completion of a previous sent task until either all of the one or more tasks corresponding to the process selected by the user are complete or the user selects a user-selectable link indicating a pause to further task completion of remaining tasks of the one or more tasks in the process selected by the user ( e.g. iteratively receive all subsequent portions of the AR UX workflow dataset until all portions have been completed )( Figure 13, step 226, [0105] notes operator device 54 may receive updated image data of the industrial automation device 20 from the image sensor 66, and at step 228, use the updated image data to verify whether the operation instruct\ed via the animation was suitably performed in the real-world and thus whether the AR UX workflow instruction was completed, step 230, [0106] notes when the operator device 54 determines that the operation instructed at step 224 is completed, the operator device 54 may determine whether another portion to the AR UX workflow dataset is available for presentation via the display 70, the operator device 54 may reference the memory 60 or the storage 62 to retrieve any additional AR UX workflow data (e.g. AR image data) and/or may request additional AR UX workflow data (e.g. AR image data corresponding to a next portion of the AR UX workflow dataset) from the AR UX system 82, step 232, [0107] notes when additional AR UX workflow data is available, the operator device may render and animate the subsequent AR UX workflow instruction, [0110] notes in response to determining that each portion of the AR UX workflow instructions has been performed correctly by the operator 50, the operator device 54 may proceed to end presentation of the AR UX workflow instructions ). As to claim 6, Mehrotra et al. disclose when all of the one or more tasks corresponding to the process selected by the user are complete or when the user selects a user-selectable link indicating a pause to further task completion ( e.g. when all portions of the AR UX workflow dataset have been completed ), the task manager ( e.g. AR UX system 82 ) is further configured to: update the status of the process selected by the user, wherein the status is one of process complete or process in progress ( e.g. update profile corresponding to the operator device 54 ); and store, in the database system, the updated status of the process ( e.g. store the updated profiles in the AR UX system database 98 )( Figure 13, step 234, [0110] notes in response to determining that each portion of the AR UX workflow instructions was correctly performed by the operator 50 to the industrial automation device 20, the operator device 54 may proceed to end presentation of the AR UX workflow instructions and update a profile corresponding to the operator device 54, where the profile updated by the operator device 54 may indicate a most recent maintenance operation performed by the operator 50 corresponding to the operator device 54, e.g. updating a maintenance history corresponding to the industrial automation device 20, where [0067] notes profiles stored in profile database 94 of AR UX system database 98 ). As to claim 7, Mehrotra et al. disclose upon selection, by the user, of the user-selectable task link for uploading a photo, the AR glasses are further configured to take a photograph or video ( e.g. operator 50 may be prompted to capture photograph/video ), wherein initiating the taking of the photograph or video is based upon an action of the user wearing the AR glasses ( Figure 8, [0090] notes operator 50 prompted to virtually place the virtual product on a real-world surface, e.g. via image sensor 66 of operator device 54, Figure 13, [0105] further notes receiving updated image data and using to verify whether the operation instructed was completed ). As to claim 8, Mehrotra et al. disclose the action of the user comprises a hands-free action ( [0075] notes operator 50 may make requests via voice commands using operator device 54, thus denoting the operator device 54 capable of receiving voice commands, thus would render obvious other actions may be received via voice commands ). As to claim 9, Mehrotra et al. disclose the hands-free action by the user comprises at least one of: an orientation of the AR glasses, an orientation of eyes of the user , or a voice command ( as noted in claim 8, [0075] notes operator 50 may make requests via voice commands using the operator device 54 ). As to claim 14, Mehrotra et al. disclose the database system further comprises a user database ( e.g. profile database 94 ), the user database configured to store user information associated with the production process on the plant, wherein the user information comprises user IDs, user roles and/or user privileges ( e.g. as noted in claim 1, [0067] notes profile database 94 stores profiles, which may include information about permission or skill level of the operator 50, such that if the operator 50 is not trained on an industrial automation device 20 or is otherwise unpermitted to adjust operation of the industrial automation device 20, the AR UX system 82 may not query AR UX workflow datasets related to the industrial automation device 20 since those AR UX workflow datasets may be irrelevant to the operator ). As to claim 15, Mehrotra et al. disclose the executable program logic is configured to: receive user credentials from the user ( e.g. receive credentials from operator 50 of operator device 54 ); authenticate the user ( e.g. AR UX system 82 authenticates operator 50 to know to grant or deny permissions ); and send the list of processes to the AR glasses for display to a user of the AR glasses if and only if the user is authenticated ( e.g. send only AR UX workflow datasets relevant to the authenticated operator 54 )( e.g. as noted in claim 14, [0067] notes profile database 94 stores profiles, which may include information about permission or skill level of the operator 50, such that if the operator 50 is not trained on an industrial automation device 20 or is otherwise unpermitted to adjust operation of the industrial automation device 20, the AR UX system 82 may not query AR UX workflow datasets related to the industrial automation device 20 since those AR UX workflow datasets may be irrelevant to the operator, thus would render obvious that the operator must enter his/her credentials in order for the AR UX system 82 to distinguish one operator from another operator in order to know whether to grant or deny permission of operation of the industrial automation device ). As to claim 18, Mehrotra et al. disclose the task manager ( e.g. AR UX system 82 ) is configured to assign processes of the list of processes to one or more users and dynamically update parameters associated with each process ( [0060] notes AR UX system 82 may include a processor 58 for performing operations and executing instructions stored in a memory 60, such that processor 58 of the AR UX system 82 may retrieve an AR UX workflow dataset and initiate transmission of the AR UX workflow dataset to the operator device 54, [0110] notes updating profiles upon completion of an AR UX workflow dataset, where [0067] notes profiles stored in profile database 94 of AR UX system database 98, and [0121] further notes AR UX system 82 may update a portion of the AR UX workflow dataset ). As to claim 19, Mehrotra et al. disclose a method for controlling an automated production process of an industrial plant as performed by the automation system of claim 1. Please see the rejection and rationale of claim 1 . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim s 10-13, 16, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. 13-03-01 AIA The following is a statement of reasons for the indication of allowable subject matter: Regarding dependent claim 10, the prior art of record fails to teach or suggest the limitations of the claim as recited as a whole. Dependent claims 11-13 are indicated allowable for directly or indirectly depending upon indicated allowable claim 10. Regarding dependent claim 16, the prior art of record fails to teach or suggest the limitations of the claim as recited as a whole. Dependent claim 17 is indicated allowable for depending upon indicated allowable claim 16 . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Michalscheck et al. (US 10,950,051) disclose a system and method of presenting augmented reality in an industrial automation system. Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACINTA M CRAWFORD whose telephone number is (571)270-1539. The examiner can normally be reached 8:30a.m. to 4:30p.m. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /JACINTA M CRAWFORD/Primary Examiner, Art Unit 2617 Application/Control Number: 18/860,769 Page 2 Art Unit: 2617 Application/Control Number: 18/860,769 Page 3 Art Unit: 2617 Application/Control Number: 18/860,769 Page 4 Art Unit: 2617 Application/Control Number: 18/860,769 Page 5 Art Unit: 2617 Application/Control Number: 18/860,769 Page 6 Art Unit: 2617 Application/Control Number: 18/860,769 Page 7 Art Unit: 2617 Application/Control Number: 18/860,769 Page 8 Art Unit: 2617 Application/Control Number: 18/860,769 Page 9 Art Unit: 2617 Application/Control Number: 18/860,769 Page 10 Art Unit: 2617 Application/Control Number: 18/860,769 Page 11 Art Unit: 2617 Application/Control Number: 18/860,769 Page 12 Art Unit: 2617 Application/Control Number: 18/860,769 Page 13 Art Unit: 2617 Application/Control Number: 18/860,769 Page 14 Art Unit: 2617 Application/Control Number: 18/860,769 Page 15 Art Unit: 2617
Prosecution Insights