Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 9 is objected to because of the following informalities: “model comprises models includes” should read “models include”. Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-5, 9, and 11-24 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Neblett (US 20250245486 A1).
Regarding claim 1, Neblett teaches a non-transitory computer-readable storage medium including instructions that, when executed by a head-wearable device of an extended-reality system (par. 0060: “FIG. 9 shows a user 900 equipped with an augmented-reality device 902 in a work environment 904. Augmented reality device 902 may be an example of computing device 300. In this example, augmented reality device 902 is configured as a tablet computer, but alternatively may take the form of any suitable device with a camera and display, such as a phone or wearable computing device (e.g., head-mounted display device).”), cause the head-wearable device to perform:
obtaining a user input from a user wearing the head-wearable device comprising an initiation of an artificially intelligent assistant (par. 0023: “Input can be taken via user input in the form of text input and in-context progress made within a digital work instruction application, which is then transmitted to an AI assistant that interfaces with an LLM.”);
in response to the initiation of the artificially intelligent assistant, capturing contextual data including image data containing a target object (par. 0061: “Computer vision may be used for object recognition and/or image & object classification in a work environment, enabling work in AR.”);
determining, based on the user input, a user intent for textual translation (par. 0023: “Responses from the AI assistant are provided in a digital work instruction application and are displayed to users via text within the application, virtual object translation, and highlighting object capabilities where applicable.”);
determining, based on the contextual data, a contextual cue that indicates text on the target object (par. 0063: “The AI assistant receives contextual information based on the visual input. The digital work environment may then output text, visuals, imagery, etc., similar to text prompts and direct AI assistant prompts.”);
providing a portion of the contextual data related to the target object, the user intent, and the contextual cue to the artificially intelligent assistant (par. 0062: “AI assistant 1004 retrieves the appropriate information, provides a text response to the user at 1024, and highlights the wiring junction, as shown at 1026.”);
receiving a response from the artificially intelligent assistant, wherein the response is generated using a machine-learning model and comprises a translation of the text on the target object (par. 0035: “Accordingly, the graphical model may utilize the statistical features, previously trained machine learning models, and/or acoustical models to define transition probabilities between states represented in the graphical model.”); and
causing the head-wearable device to present the response (par. 0023, as above).
Regarding claim 2, Neblett teaches the non-transitory computer readable storage medium of claim 1, wherein the response is one or more of a textual response, an audible response, and a visual response (par. 0063: “The digital work environment may then output text, visuals, imagery, etc., similar to text prompts and direct AI assistant prompts.”).
Regarding claim 3, Neblett teaches the non-transitory computer readable storage medium of claim 1, wherein the response includes identification of the target object and a follow-up action associated with the target object to be performed by the head-wearable device (par. 0061: “Computer vision may be used for object recognition and/or image & object classification in a work environment, enabling work in AR.”).
Regarding claim 4, Neblett teaches the non-transitory computer readable storage medium of claim 1, wherein the portion of the contextual data is formed by compressing the contextual data (par. 0091: “When user data is collected, users or other stakeholders may designate how the data is to be used and/or stored. All potentially sensitive data optionally may be encrypted and/or, when feasible anonymized, to further protect user privacy. Users may designate portions of data, metadata, or statistics/results of processing data for release to other parties, e.g., for further processing.”).
Regarding claim 5, Neblett teaches the non-transitory computer readable storage medium of claim 1, wherein determining, based on the contextual data, the contextual cue comprises:
determining a region of interest within the image data, the region of interest identifying a portion of the image data including the target object (par. 0064: “Metadata, in particular, CAD metadata can be retrieved from the LLM using the AI assistant. If the user is training a camera at a portion of the build environment, the AI assistant may recognize what is relevant to the next step in the work instruction and may then tell the digital work environment to highlight a specific part.”); and
cropping the image data based on the region of interest to form cropped image data (par. 0037: “The storage device may comprise one or more local storage devices and/or one or more network accessible storage devices. In some examples, images are extracted from the technical documents. The extracted images can also be used to train the large language models. In some examples, the stored technical documents include CAD engineering and/or drafting files as well as associated metadata.”).
Regarding claim 9, Neblett teaches the non-transitory computer readable storage medium of claim 1, wherein the plurality of machine-learning model comprises models includes one or more of an on-device machine-learning model of the head-wearable device configured to determine the translation of the text on the target object and a remote machine-learning model (par. 0023: “Responses from the AI assistant are provided in a digital work instruction application and are displayed to users via text within the application, virtual object translation, and highlighting object capabilities where applicable.”).
Claim 11 is substantially similar to claim 1, except that it teaches a device rather than a storage medium. As such, it is rejected on a similar basis to claim 1.
Claim 12 is substantially similar to claim 2, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 2.
Claim 13 is substantially similar to claim 3, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 3.
Claim 14 is substantially similar to claim 4, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 4.
Claim 15 is substantially similar to claim 5, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 5.
Claim 16 is substantially similar to claim 1, except that it teaches a device rather than a storage medium. As such, it is rejected on a similar basis to claim 1.
Claim 17 is substantially similar to claim 2, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 2.
Claim 18 is substantially similar to claim 3, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 3.
Claim 19 is substantially similar to claim 4, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 4.
Claim 20 is substantially similar to claim 5, except that it depends from claim 11 rather than claim 1. As such, it is rejected on a similar basis to claim 5.
Regarding claim 21, Neblett teaches the non-transitory computer readable storage medium of claim 1, wherein:
the user input further includes an open-ended user request to identify target words in a first language and the open-ended user request is performed by the user in a second language, distinct from the first language (par. 0023: “which is then transmitted to an AI assistant that interfaces with an LLM. Responses from the AI assistant are provided in a digital work instruction application and are displayed to users via text within the application, virtual object translation, and highlighting object capabilities where applicable.”); and
the translated text comprises translating the text from the first language into the second language (par. 0023, as above).
Regarding claim 22, Neblett teaches the non-transitory computer readable storage medium of claim 1, wherein the image data comprises a field of view of the user (par. 0061: “Computer vision may be used for object recognition and/or image & object classification in a work environment, enabling work in AR.”).
Regarding claim 23, Neblett teaches the non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed by the head-wearable device, further cause the head-wearable device to perform:
obtaining another user input, distinct from the user input, from the user comprising another initiation of the artificially intelligent assistant (par. 0079: “In some examples, an updated contextual response to a prompt related to a second work protocol is generated at the artificially intelligent assistant.”);
in response to the other initiation of the artificially intelligent assistant, capturing other contextual data including other image data containing another target object, distinct from the target object (par. 0065: “The AI assistant may have the same capability as the user to click on the AR interface, give contextual hints to objects, giving human-like interfaces and capabilities to the AI assistant.”);
while the other contextual data is being captured:
determining, based on the other user input, another user intent for textual translation (par. 0089: “Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition.”;
determining, based on the contextual data, another contextual cue that indicates text on the other target object (par. 0039: “At 220, method 200 comprises extracting text and metadata from the technical documents. In some examples, the technical documents may comprise contextual links to other technical documents, via interlinking or cross-linking.”);
providing a first portion of the other contextual data related to the other target object, the other user intent, and the other contextual cue to the artificially intelligent assistant (par. 0080: “In some examples, a work protocol may be redundantly performed on one build, or multiple adjacent builds may be using the same or similar work protocols. As such, if an issue arises during the execution of one work protocol, the LLM may be retrained, and contextual hints presented to other technicians performing the same work protocol.”);
providing a second portion of the other contextual data related to the other target object, distinct from the first portion, to the artificially intelligent assistant (par. 0080, as above);
receiving another response from the artificially intelligent assistant, wherein the response is generated using the machine-learning model and comprises a translation of the text on the other target object (par. 0079: “In some examples, an updated contextual response to a prompt related to a second work protocol is generated at the artificially intelligent assistant. For example, changes in an adjacent build may impact the response for a current build. “);
in accordance with receiving the other response, ceasing to capture the other contextual data (par. 0077: “Completion of a step of a work protocol may influence whether initiation of other steps from the same or other work protocols is compliant or not compliant.”); and
causing the head-wearable device to present the other response (par. 0088: “When included, display subsystem 1730 may be used to present a visual representation of data held by storage subsystem 1720. This visual representation may take the form of a graphical user interface (GUI).”).
Regarding claim 24, Neblett teaches the non-transitory computer-readable storage medium of claim 1, wherein the artificially intelligent assistant comprises a system with a set of task-specific machine learning models (par. 0086: “Aspects of logic subsystem 1710 and storage subsystem 1720 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include program- and application-specific integrated circuits (PASIC/ASICs), program-and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.”).
Response to Arguments
Applicant’s arguments, see Remarks, filed 06/02/2026, with respect to the rejection(s) of claim(s) 1-20 under Browy (US 20200210127 A1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Neblett (US 20250245486 A1).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/ Examiner, Art Unit 2613
/XIAO M WU/ Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights