DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are currently pending in the present application, with claims 1, 15, and 18 being independent. Claims 1, 2, 4, 6, 14-16, 18, and 19 have been amended.
Response to Arguments
Applicant’s arguments, see pages 7 and 8, filed 27 April 2026, with respect to the 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejection of claims 1-20, along with accompanying amendments received on the same date, have been fully considered and are not persuasive. The 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejection of claims 1-20 has been maintained.
With respect to claims 1, 15, and 18, Given the broadest reasonable interpretation of claims 1, 15, and 18 in light of the disclosure and/or the plain and ordinary meaning of the words themselves, the scope of the claimed limitations remains unclear. For instance, it is not immediately clear as to:
What is meant by determining a procedure including steps performable with an augmented reality device. How is the procedure determined? It would seem from the corresponding disclosure that the procedure is associated with the physical environment. Paragraph 15 sets forth “the AR system enables creation and viewing of procedures associated with a physical environment. Here, a procedure may comprise a sequence of tasks set forth in a step-by-step manner. For each step of a procedure, the AR system may present an AR-based user interface that enables the viewer to view the physical environment supplemented with one or more virtual objects that provide instructional guidance for facilitating the procedure step”. Paragraph 19 sets forth “In an office environment, the AR system may be used to provide procedures or information relating to the use of devices such as copying machines, a coffee maker, or a printer. In a construction environment, the AR system may be employed to precisely indicate construction tasks....” Paragraph 55 sets forth “The AR device 520 obtains 802 a procedure for performing with AR-based guidance”. That is, it appears from the corresponding disclosure, that the procedure is obtained, as opposed to determined. The disclosure does not appear to describe determining a procedure.
What constitutes the sequence of events associated with carrying out the procedure. Is the sequence of events related to the user performing the steps in the procedure? The steps of the procedure? It would seem from the corresponding disclosure that the procedure should comprise the sequence of tasks and then the AR device tracks the user carrying out the sequence of events. As it stands, it is unclear as to how sequence of events relates to carrying out the procedure.
How are the anomalies, which are deviations from expected performance of the procedure, detected in the sequence of events? What is meant by expected performance of the procedure? Is related to a expected performance metric, such as how well the user performs the sequence of events? Is it a deviation of the user performing the sequence in events when compared to the expected steps in the sequence of events? Is it related to the outcome of the procedure? As currently claimed it is unclear as to what constitutes the anomaly and how that anomaly is detected.
Accordingly, the 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejection of claims 1, 15, and 18 and claims depending thereon have been maintained.
With respect to claims 2, 4, 6, 16, and 19, applicant’s amendments have clarified the scope of the claimed limitation. Accordingly, the 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejection of claims 2, 4, 16, and 19 and claims depending thereon have been withdrawn.
Applicant’s arguments, see pages 7 and 8, filed 27 April 2026, with respect to the 35 U.S.C. 101 rejection of claims 1-20, along with accompanying amendments received on the same date, have been fully considered and are not persuasive. The 35 U.S.C. 101 rejection of claims 1-20 has been maintained.
As drafted, the claimed limitations are mental process that can be performed by a human mind. For instance, the limitation “determining a procedure including steps performable with an augmented reality device, wherein at least a subset of the steps are associated with one or more virtual objects presented in an augmented reality view of the augmented reality device that provide guidance associated with the steps”, is a process that, under its broadest reasonable interpretation, is simply choosing a procedure to be performed. The subsequent steps merely tracks the user’s interactions to see if they are following the procedure, and outputting something if the user deviates from the procedure list. Displaying the steps/procedure on an augmented reality device, is merely (extra solution activity). The process is a mental process. Thus, the claim, as drafted, falls at least within the “Mental Processes” grouping of abstract ideas. It is noted, that a similar argument could be made for the claim to additionally fall within one or more groups, such as “Organizing Human Activity”.
This judicial exception is not integrated into a practical application because the method, is recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using a generic computer component. The additional elements (e.g., an augmented reality device, a machine, which could be the augmented reality device), do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Accordingly, claims 1, 15, and 18 are directed to an abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using an augmented reality device to perform the claimed limitations amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, claims 1, 15, and 18 are not patent eligible.
Similar mapping and rationale can be performed for each of the dependent claims. For instance, with respect to claims 12-14, a notification is generated and/or sent. The dependent claims do not cure the deficiency noted with respect to claims from which they depend. Accordingly, claims 1-20 as currently drafted are not patent eligible.
With respect to applicant’s argument that the technical process performed by the AR system that monitors procedure performance through sensor data and takes automated corrective action on physical equipment, the claims do not recite that automative corrective performance, rather they merely output an automated action of a machine associated with the procedure. The automated action could merely be an indication the user performed the step wrong and the machine could be the AR system itself. While as claimed, the claim is not integrated into a practical application, one potential way forward would be to integrate into the claims the features, such as the automative corrective action on the physical equipment, monitoring procedure performance, etc) into the claim.
With respect to applicant’s argument that tracking state data derived from image data and motion data captured by the augmented reality device involves specific sensor-based data processing tied to the AR device’s cameras and motion sensors, the claims do not require what the applicant is arguing, rather the claims simply require the end product tracked user interactions and tracked state data, which could be presented to a someone to show if the user performed the sequence of events correctly.
Applicant’s final argument, that the presentation of virtual objects in an AR view that provides guidance is not merely displaying data on a screen because it involves spatially integrating virtual content into a real-world view through an augmented reality device. By definition, augmented reality is the overlaying (spatial integration) of virtual content into the real-world environment. The screen with an AR device would be something being overlaid onto the real-world environment. The claim does not require a specific ordering of the virtual objects, nor does it concretely tie the guidance steps to specific objects of the scene, rather it simply recites broad language, namely that at least a subset of the steps are associated with one or more virtual objects presented in an AR view that provide guidance associated with the steps. The subset of steps could be a single step, and the virtual objects providing guidance could be any notation on the screen, including listing the virtual object.
Applicant’s arguments, see pages 10 and 11, filed 27 April 2026, with respect to the 35 U.S.C. 102(a)(1)/(a)(2) rejection of claims 1, 7-10, 12-15, and 18, along with accompanying amendments received on the same date, have been fully considered and are not persuasive. The 35 U.S.C. 102(a)(1)/(a)(2) rejection of claims 1, 7-10, 12-15, and 18 has been maintained.
With respect to applicant’s argument that Neeter does not disclose invoking an automated action of a machine associated with the procedure, let alone in response to detected anomalies, the broadest reasonable interpretation of the claimed limitation is the automated action of a machine associated with the procedure includes alerting the user of the error at the AR device or another device. As noted by the applicant in their remarks on pages 10 and 11, Neeter describes at “step 316, the error detection engine can cause the augmented reality device 124 or another device located at the real-world physical location to alert the user of the error and/or provide the user with feedback to correct the error”. Accordingly, Neeter teaches the broadest reasonable interpretation of the amended claims.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. The originally filed disclosure does not appear to support determining a procedure as currently claimed. Paragraph 15 sets forth “the AR system enables creation and viewing of procedures associated with a physical environment. Here, a procedure may comprise a sequence of tasks set forth in a step-by-step manner. For each step of a procedure, the AR system may present an AR-based user interface that enables the viewer to view the physical environment supplemented with one or more virtual objects that provide instructional guidance for facilitating the procedure step”. Paragraph 19 sets forth “In an office environment, the AR system may be used to provide procedures or information relating to the use of devices such as copying machines, a coffee maker, or a printer. In a construction environment, the AR system may be employed to precisely indicate construction tasks....” Paragraph 55 sets forth “The AR device 520 obtains 802 a procedure for performing with AR-based guidance”. Paragraph 11 sets forth “Event data may furthermore be aggregated over time and from multiple devices to generate various insights into performance differences between different users, facilities, or enterprises, to identify procedures or steps that can be improved, or to generate various recommendations for improving overall operations”. That is, it appears from the corresponding disclosure, the procedure is obtained, as opposed to determined according to some criteria. The examiner respectfully requests as to where support can be found for determining a procedure including steps performable with an augmented reality device.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Given the broadest reasonable interpretation of claims 1, 15, and 18 in light of the disclosure and/or the plain and ordinary meaning of the words themselves, the scope of the claimed limitations is unclear. For instance, it is not immediately clear as to:
What is meant by determining a procedure including steps performable with an augmented reality device. How is the procedure determined? It would seem from the corresponding disclosure that the procedure is associated with the physical environment. Paragraph 15 sets forth “the AR system enables creation and viewing of procedures associated with a physical environment. Here, a procedure may comprise a sequence of tasks set forth in a step-by-step manner. For each step of a procedure, the AR system may present an AR-based user interface that enables the viewer to view the physical environment supplemented with one or more virtual objects that provide instructional guidance for facilitating the procedure step”. Paragraph 19 sets forth “In an office environment, the AR system may be used to provide procedures or information relating to the use of devices such as copying machines, a coffee maker, or a printer. In a construction environment, the AR system may be employed to precisely indicate construction tasks....” Paragraph 55 sets forth “The AR device 520 obtains 802 a procedure for performing with AR-based guidance”. That is, it appears from the corresponding disclosure, that the procedure is obtained, as opposed to determined. The disclosure does not appear to describe determining a procedure.
What constitutes the sequence of events associated with carrying out the procedure. Is the sequence of events related to the user performing the steps in the procedure? The steps of the procedure? It would seem from the corresponding disclosure that the procedure should comprise the sequence of tasks and then the AR device tracks the user carrying out the sequence of events. As it stands, it is unclear as to how sequence of events relates to carrying out the procedure.
How are the anomalies, which are deviations from expected performance of the procedure, detected in the sequence of events? What is meant by expected performance of the procedure? Is related to a expected performance metric, such as how well the user performs the sequence of events? Is it a deviation of the user performing the sequence in events when compared to the expected steps in the sequence of events? Is it related to the outcome of the procedure? As currently claimed it is unclear as to what constitutes the anomaly and how that anomaly is detected.
How the output includes invoking an automated action of a machine associated with the procedure? Is the procedure for a machine?
The examiner respectfully requests the applicant clarify the scope of the claimed invention. Claims depending thereon do not cure all of the noted deficiencies and are therefore also rejected using substantially similar rationale as to that for the claims from which they depend.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) rendering a screen on a display comprising: selecting at least one resolution for rendering the screen based on a set of parameters associated with the screen; and rendering the screen with the at least one selected resolution on the display.
As drafted, the claimed limitations are mental process that can be performed by a human mind. For instance, the limitation “determining a procedure including steps performable with an augmented reality device, wherein at least a subset of the steps are associated with one or more virtual objects presented in an augmented reality view of the augmented reality device that provide guidance associated with the steps”, is a process that, under its broadest reasonable interpretation, is simply choosing a procedure to be performed. The subsequent steps merely track the user’s interactions to see if they are following the procedure, and outputting something if the user deviates from the procedure list. Displaying the steps/procedure on an augmented reality device, is merely (extra solution activity). The process is a mental process. Thus, the claim, as drafted, falls at least within the “Mental Processes” grouping of abstract ideas. It is noted, that a similar argument could be made for the claim to additionally fall within one or more groups, such as “Organizing Human Activity”.
This judicial exception is not integrated into a practical application because the method, is recited at a high-level of generality, such that it amounts to no more than mere instructions to apply the exception using a generic computer component. The additional elements (e.g., an augmented reality device, a machine, which could be the augmented reality device), do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Accordingly, claims 1, 15, and 18 are directed to an abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, with respect to integration of the abstract idea into a practical application, the additional elements of using an augmented reality device to perform the claimed limitations amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Thus, claims 1, 15, and 18 are not patent eligible.
Similar mapping and rationale can be performed for each of the dependent claims. For instance, with respect to claims 12-14, a notification is generated and/or sent. The dependent claims do not cure the deficiency noted with respect to claims from which they depend. Accordingly, claims 1-20 as currently drafted are not patent eligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 7-10, 12-15, and 18 is/are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Neeter (US PG Publication 2021/0019215).
Regarding claim 1, Neeter teaches a method for detecting anomalies associated with performance of procedures guided by an augmented reality system (see for instance, paragraph 64 and figs. 1, 3A, and 3B), the method comprising:
determining a procedure including steps performable with an augmented reality device, wherein at least a subset of the steps are associated with one or more virtual objects presented in an augmented reality view of the augmented reality device that provide guidance associated with the steps (At step 310 a user interface can be rendered on one or more displays of the augmented reality device...to guide the user on the procedure or routine to be performed in the real-world physical scene...the set of tasks can include one or more activities and actions to be performed by the user, where the action to be performed by the user can include...one or more interactions between the user and one or more virtual and/or actual objects, see for instance, paragraph 66);
tracking, by the augmented reality device, a sequence of events associated with carrying out the procedure, the sequence of events including tracked user interactions with the augmented reality device and tracked state data derived from image data and motion data captured by the augmented reality device (At step 312, the system received data from the augmented reality environment indicating that the user is undertaking a first activity in the set of tasks by performing one or more actions, see for instance, paragraph 66. If the error detection engine determines that the user of the augmented reality device deviated or is deviating from actions to be performed for the first activity, as defined by the set of tasks, see for instance, paragraph 66. Object recognition can be used to identify virtual objects and/or actual objects in the augmented reality environment based on images of the virtual objects and/or actual objects and attributes of the identified virtual and/or actual objects can be extracted from the identified virtual and/or actual objects, see for instance, paragraph 31. For example, a virtual object or an actual object in the augmented reality environment can be a thermometer or gauge, and a temperature of the environment can be determined by identifying the object as thermometer using one or more of the sensors (e.g., via machine vision) to extract the temperature from the image(s) of the thermometer, see for instance, paragraph 31. The event recognition engine can be executed to interface with the environment engine and the task management engine to process the feedback from one or more sensors, see for instance, paragraph 52.);
detecting one or more anomalies in the sequence of events associated with carrying out the procedure, the one or more anomalies including deviations from expected performance of the procedure (“At step 314, the event recognition engine identifies the actions and the error detection engine uses the trained machine learning models to determine whether the one or more actions are correct and are occurring at the correct time in the routine based on the order sequence defined by the set of tasks. As an example, the first activity in the routine can be to locate a control panel”, see for instance, paragraph 66); and
generating one or more outputs indicative of the one or more anomalies, wherein generating the one or more outputs include invoking, based on the one or more anomalies, an automated action of a machine associated with the procedure (The user can be alerted of the error, see for instance, paragraphs 66, 67 and 70 and fig. 3. If the error detection engine determines that the user of the augmented reality device 124 deviated or is deviating from the actions to be performed for the first activity, as defined by the set of tasks, at step 316, the error detection engine can cause the augment reality device 124 or another device located at the real-world physical location to alert the user of the error and/or provide the user with feedback to correct the error via the augment reality device 124 or the other device, see for instance, paragraph 66).
Regarding claim 6, Neeter teaches the method of claim 1 and further teaches wherein the tracked user interactions comprises selection of a control element of the augmented reality device presented in the augmented reality view of an environment in association with performance of one or more steps of the procedure (see for instance, paragraph 70 and fig. 3).
Regarding claim 7, Neeter teaches the method of claim 1 and further teaches wherein the tracked state data includes at least one of: a position, a velocity, an acceleration, an orientation, an angular velocity, and an angular acceleration associated with the augmented reality device, a set of feature points detected from image analysis of images captured by the augmented reality device, and a detected environment map detected from the images captured by the augmented reality device (see for instance, paragraphs 66-71 and fig. 3).
Regarding claim 8, Neeter teaches the method of claim 1 and further teaches wherein tracking the sequence of events comprises at least one of: capturing one or more user events triggered in response to a user input; and capturing one or more polling events triggered based on a polling mechanism (see for instance, paragraphs 66-71 and fig. 3).
Regarding claim 9, Neeter teaches the method of claim 1 and further teaches wherein detecting the one or more anomalies includes: applying a set of anomaly detection rules associated with the procedure to the sequence of events (see for instance, paragraphs 66 and 70).
Regarding claim 10, Neeter teaches the method of claim 1, and further teaches wherein detecting the one or more anomalies includes: applying a classification model associated with the procedure to the sequence of events to detect when the sequence deviates with statistical significance from historical event sequences associated with procedure (see for instance, paragraphs 63, 66, and 69-71).
Regarding claim 12, Neeter teaches the method of claim 1 and further teaches wherein generating the one or more outputs comprises: generating a notification in a user interface of the augmented reality device indicative of the one or more anomalies (see for instance, paragraph 66-70).
Regarding claim 13, Neeter teaches the method of claim 1 and further teaches wherein generating the one or more outputs comprises: sending a notification indicative of the one or more anomalies to an augmented reality server to an administrative client (see for instance, paragraph 66-70).
Regarding claim 14, Neeter teaches The method of claim 1 and further teaches wherein generating the one or more outputs comprises: generating a notification to an augmented reality server coupled to the augmented reality device, the augmented reality server being configured to invoke the automated action of the machine associated with the procedure, the action comprising at least one of: a calibration action, a reset action, a shutdown action, and a safety action (see for instance, paragraph 66-70).
Regarding claims 15 and 18, claim 16 is the computer-readable storage medium claim and claim 18 is the device claim of the method claim 1 and are accordingly rejected using substantially similar rationale as to that set forth with respect to claim 1. In addition, Neeter teaches a non-transitory computer-readable storage medium storing instructions for detecting anomalies associated with performance of procedures guided by an augmented reality system, the instructions when executed by a processor causing the processor to perform steps (see for instance, claim 21 and paragraphs 6 and 79) and an augmented reality device comprising: one or more cameras for capturing image data; one or more motion sensors for capturing motion data; one or more processors; and a non-transitory computer-readable storage medium storing instructions for detecting anomalies associated with performance of procedures guided by an augmented reality system, the instructions when executed by the one or more processors causing the one or more processors to perform steps (see for instance, paragraphs 49-52 and 79-85).
Allowable Subject Matter
Claims 2-6, 11, 16, 17, 19, and 20 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Manuri teaches a state validation system for AR based maintenance procedures. Manuri further teaches tracking and identifying elements that should be changed during a procedure, see for instance, ‘Tracking’ and ‘Workflow’.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J COBB whose telephone number is (571)270-3875. The examiner can normally be reached Monday - Friday, 11am - 7pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 571-272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J COBB/Primary Examiner, Art Unit 2615
Prosecution Insights