DETAILED ACTION
This Final Office Action is in response Applicant communication filed on
2/18/2026. In Applicant’s amendment, claims 1-20 were amended.
Claims 1-20 are currently pending and have been rejected as follows.
Response to Amendments
Applicant’s amendments necessitated new grounds of rejection under 35 USC 103.
Response to Arguments
Applicant's prior art arguments have been fully considered but they are not persuasive to overcome the rejection.
Applicant argues on p. 12-13 that neither reference teaches detecting objects and relationships using computer vision models. This argument is moot in light of the newly cited Tremblay reference combined with Wright and Kupcsik.
Applicant argues on p. 13 that neither reference teaches a symbolic task state generated from detected objects and relationships. This argument is moot in light of the newly cited Tremblay reference combined with Wright and Kupcsik.
Applicant argues on p. 14 that there is no proper motivation to combine Wright’s AR system with Kupcsik’s robot manipulation planning. Examiner respectfully disagrees. Wright’s AR system aims to assist a human user in performing multi-step physical tasks in an AR environment by providing task instructions in AR overlays. Kupcsik teaches a method of representing a task in a symbolic planning manner using objects, predicates, relations, initial state, goal state, and actions with effects and preconditions, to generate a sequence of actions. Both references are about multi-step task planning and execution involving objects and ordered actions for completing a goal. Kupcsik’s focus on the task planning is complementary to improving Wright’s AR guided task completion system.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-12, and 14-19 are rejected under 35 USC 103 as being unpatentable over the teachings of
Wright et al., US 20210035333 A1, hereinafter Wright in view of
Tremblay et al., US 20190228495 A1, hereinafter Tremblay, in view of
Kupcsik et al., US 20200398427 A1, hereinafter Kupcsik. As per,
Claims 1, 8, 15
Wright teaches
An extended reality system comprising: a head-mounted device comprising a display and one or more cameras; one or more processors for: /
A computer-implemented method comprising: /
A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform the following operations: (Wright [0005]; [0006] “An AR device may include input components. The input components may capture data about the physical environment and/or data from an operator. An input component may include a digital image capturing device, such as a digital camera, The digital camera may be utilized to capture digital images of the physical environment which may be analyzed with computer vision;” [0007] “The AR device may include a display. The display may include an optical projection system, a spatial augmented reality digital projector, a monitor, a handheld device, a wearable display, display eyeglasses, display contacts, a virtual retinal display, an eye tap, a head-up display (HUD), etc. The AR device may render digital information onto and/or through the display to be perceived by the operator of the device.” Note the wearable device capture and displaying images/content of the user’s environment)
obtaining image data corresponding to a field of view of a user of the head-mounted device, the image data captured by the one or more cameras; (Wright [0014] “the AR device 104 may include a digital camera that captures video and/or stills of a portion of the physical environment 102 and/or the objects 103-1 . . . 103-N present in the field of view of the camera.” Note the live video captured)
detecting, using one or more computer vision models on the image data, objects and […] for performing a task; (Wright [0011] “extract data from objects in a digital image of a physical environment, utilize the extracted data to identify information about a task to be performed;” [0020]-[0021] noting the extraction of data from objects and the environment to identify objects and provide additional data beyond simply what an object is; [0020] “Extracting data from the objects 103-1 . . . 103-N and/or the physical environment 102 may include acquiring, processing, and analyzing digital images of the objects 103-1 . . . 103-N and/or the physical environment 102 and extracting from those images data existing in the physical environment 102 and/or objects 103-1 . . . 103-N present in the physical environment 102. Extracting the data from the objects 103-1 . . . 103-N and/or the physical environment 102 may include utilizing a computer vision system implementing a neural net and/or deep learning based image and feature analysis and classification to achieve a digitized visual understanding of the contents of the digital image” Note the computer vision system for understanding the contents of the captured images from the cameras; [0027] “The information manager 106 may utilize the extracted data to identify information about the objects 103-1 . . . 103-N, the physical environment 102, and/or a task to be performed by the operator at the physical environment 102.” Note the extracted information utilized to identify information about the objects, environment, and a task to be performed)
[…];
[…];
[…];
[…];
[…];
rendering, on the display, virtual content in an extended-reality environment representative of instructions or recommendations to the user for performing at least some of the sequence of actions based on the plan. (Wright [0010] “the AR device may provide the operator with instructions for a task that he is already intimately familiar with and/or with instructions for a task that are outside his familiarity level;” [0011] “methods that may extract data from objects in a digital image of a physical environment, utilize the extracted data to identify information about a task to be performed by an operator at the physical environment, and select, based on a characteristic of the operator, a portion of the identified information about the task to include in a visual overlay to be displayed to the operator at the physical environment;” [0060] “The visual overlay that is displayed to the operator may be continuously updated as the performance of the maintenance task by the operator proceeds” corresponding to the rendering in response to executing the sequence of actions)
Wright does not explicitly teach, Tremblay however in the analogous art of task coordination teaches
[…] relationships between the objects […]; (Tremblay [0029] “a perception network 302 can be a deep neural network that accepts the demonstration data captured of the performance, such as may include image, distance, and other data … The perception network can process the demonstration data to generate a set of observations or “percepts” about the task. As mentioned, this can include relationships among the objects;” [0043] “After objects have been detected, their relationships can be inferred. This is accomplished via a fully connected neural network” noting the use of a neural-network to detect objects and their relationships)
prior to performing the task: generating label assignments for the detected objects and relationships; (Tremblay [0031] “The relationship inference network can utilize those coordinates, for example, to infer a probability map over the possible relationships. This can include relationships such as on top of, to the left of, in front of, halfway in front of, etc.;” [0043] “the output is a symbol from a set of relationships, such as the set {ABOVE, LEFT, NONE}. “ note the symbolic outputs for detected relationships corresponding to label assignments for the objects and relationships)
generating a symbolic task state using the label assignments; and; (Tremblay [0033] “the output of the relationship inference network is an array with a number of rows and columns corresponding to the number of objects, in order to determine the relative relationships between any pair of the objects. Within each cell can be a string of binary digit;” [0034] “The execution network can accept as input a version of the plan, which can correspond to a table of values (e.g., 0s and 1s) indicating the relationships between the various objects. Another input can be another table indicating the state that comes from the same perception network; [0035] “The learning in various embodiments thus is at the symbolic level” note the perception outputs generated into symbolic state representations based on object relationships)
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify Wright’s task sequence planning to include inferring relationships between objects in view of Tremblay in an effort to accurately represent data for tasks to be performed (see Tremblay ¶ [0026] & MPEP 2143G).
Wright / Tremblay do not explicitly teach, Kupcsik however in the analogous art of task coordination teaches
obtaining a task goal state representing a target configuration of the objects and relationship label assignments for task completion; (Kupcsik [0075] “The PDDL definition P has been constructed, which can be directly fed into any compatible PDDL planner” note the PDDL and feeding the current state and goal into the planner; [0052]-[0057] “The Planning Domain Definition Language (PDDL) is the standard classic planning language. Formally, the language consists of the following key ingredients: [0053] Objects, everything of interest in the world; [0054] Predicates, object properties and relations; [0055] Initial States, set of grounded predicates as the initial states; [0056] A goal Specification, the goal states; and [0057] Actions, how predicates are changed by an action and also the preconditions on the actions.”)
providing, using a domain specific planning language, the symbolic task state and the task goal state to a planner; (Kupcsik [0014] “constructing a PDDL model, wherein objects, initial state and goal specification define a problem instance, while predicates and actions define a domain of a given manipulation, wherein particularly the symbolic abstraction of the manipulations skills uses a classical PDDL planning language;” [0049] “to construct a PDDL model … where Objects, Initial State and Goal Specification define a problem instance P.sub.p, while Predicates and Actions define the domain of a problem P.sub.D;” [0050] “Once the domain and problem files are specified, a PDDL … planner has to find a sequence of actions to fulfill the given goal specification, starting from the initial state” note the use of a domain specific planning language to provide the symbolic task state and goal state to a planner)
generating, using the planner, a plan comprising a sequence of actions to achieve the task goal state, wherein the planner encodes how the actions impact the detected objects and relationships; and (Kupcsik [0050] “Once the domain and problem files are specified, a PDDL (Planning Domain Definition Language.) planner has to find a sequence of actions to fulfill the given goal specification, starting from the initial state” noting the planner finding a sequence of actions to complete the goal; [0052]-[0057] “The Planning Domain Definition Language (PDDL) is the standard classic planning language. Formally, the language consists of the following key ingredients: Objects, everything of interest in the world; Predicates, object properties and relations; Initial States, set of grounded predicates as the initial states; A goal Specification, the goal states; and Actions, how predicates are changed by an action and also the preconditions on the actions.” Note the action effects on objects and relations)
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify Wright’s task sequence planning and Tremblay’s network based relationship inferences to include a planning domain definition language for generating a sequence to achieve a goal task state in view of Kupcsik in an effort to provide high-level reasoning to understand how to incorporate actions into a sequence to achieve a goal state (see Kupcsik ¶ [0070] & MPEP 2143G).
Claims 2, 9, 16
Wright teaches
wherein: the one or more processors are further configured for receiving a request by the user for assistance in performing the task; (Wright [0063] “the information manager 106 may provide additional information in the visual overlay and/or may prompt an operator to decide whether he would like additional information and/or to be connected with an external information source such as a help line”)
Wright / Tremblay do not explicitly teach, Kupcsik however in the analogous art of task coordination teaches
the task comprises a temporal planning problem; and the plan is generated by solving the temporal planning problem for a sequence of actions and a duration of the actions. (Kupcsik [0075] “The PDDL definition P has been constructed, which can be directly fed into any compatible PDDL planner. Different optimization techniques can be enforced during the planning, e.g., minimizing the total length of the plan” noting the planner optimizing for the metric of the length of the plan)
The motivations/rationales to combine Wright / Tremblay with Kupcsik persists.
Claims 3, 10, 17
Wright / Tremblay do not explicitly teach, Kupcsik however in the analogous art of task coordination teaches
wherein the one or more processors are further configured for identifying a planning model for the task from a corpus of planning models for various tasks, wherein: the planning model for the task is expressed with the domain specific planning language, and the planning model encodes the actions for the task, and the sequence of actions is a temporal ordering of the actions that solve the temporal planning problem. (Kupcsik [0050] “Once the domain and problem files are specified, a PDDL (Planning Domain Definition Language.) planner has to find a sequence of actions to fulfill the given goal specification, starting from the initial state” noting the contemplation of multiple domains; [0075] “The PDDL definition P has been constructed, which can be directly fed into any compatible PDDL planner. Different optimization techniques can be enforced during the planning, e.g., minimizing the total length of the plan” noting the planner optimizing for time)
The motivations/rationales to combine Wright / Tremblay with Kupcsik persists.
Claims 4, 11, 18
Wright teaches
wherein detecting the objects and the relationships between the objects comprises extracting object features from the image data, and locating a presence of the objects with a bounding box and assigning labels to types or classes of the located objects and relationships between the located objects based on the extracted object features, and wherein the labels for the located objects and the relationships between the located objects are a set of state variables that are propositional in nature for the symbolic task state as observed by the user, and generating the symbolic task state comprises describing an association of the objects and the relationships between the objects with the labels as logical statements. (Wright [0020] “Extracting the data from the objects 103-1 . . . 103-N and/or the physical environment 102 may include utilizing a computer vision system implementing a neural net and/or deep learning based image and feature analysis and classification to achieve a digitized visual understanding of the contents of the digital image” noting the computer vision system to detect the objects and relationships; [0048] “The virtual labels may include virtual labels that include the environmental conditions and/or the physical condition of the particular objects” note the virtual labels)
Claims 5, 12, 19
Wright teaches
wherein the rendering comprises: executing at least some of the sequence of actions in the plan, wherein the executing comprises determining virtual content data to be used for rendering the virtual content based on the sequence of actions, and determining the virtual content data comprises mapping the actions to respective action spaces and determining the virtual content data associated with the respective action spaces; and (Wright [0060] “The information about the maintenance task that is included in successive visual overlays communicated during the performance of the maintenance task subsequent to a first visual overlay;” [0061] “the information manager may select a second portion of the identified information about the task and/or additional information about the task, captured since the last visual overall iteration of the visual overlay was sent, to include in the subsequent successive visual overlay based on the data extracted from objects 103-1 . . . 103-N, portions of the physical environment 102, and/or actions or feedback form the operator since a prior iteration of a visual overlay was communicated to the AR device 104” note the successive visual overlays based on the actions, data extracted from objects and the physical environment)
rendering the virtual content in the extended-reality environment displayed to the user based on the virtual content data. (Wright [0061] “the information manager 106 may continue to adapt the identified information that is included in the subsequent successive visual overlays, based on data from objects 103-1 . . . 103-N, portions of the physical environment 102, and/or the actions or feedback of the operator that occur during the performance of the maintenance task”)
Claims 7, 14
Wright teaches
wherein the one or more processors are further configured for obtaining additional data, including: (i) data regarding activity of the user in the extended-reality environment, (ii) data from external systems, or (iii) both, and wherein the symbolic task state is based on the additional data. (Wright [0061] “the information manager 106 may continue to adapt the identified information that is included in the subsequent successive visual overlays, based on data from objects 103-1 . . . 103-N, portions of the physical environment 102, and/or the actions or feedback of the operator that occur during the performance of the maintenance task” note the actions or feedback of the operator; [0062] “a feedback loop may be created whereby the system 100 continuously or semi-continuously extracts data from objects 103-1 . . . 103-N, portions of the physical environment 102, and an operator utilizing the AR device 102 in performing the maintenance task in the physical environment 102 in digital images;” [0095] “The digital image may be a video feed” note the video regarding the operator activity)
Claims 6, 13, and 20 are rejected under 35 USC 103 as being unpatentable over the teachings of
Wright in view of Tremblay in view of Kupcsik in view of
Duarte De Oliveira et al., US 20210373664 A1, hereinafter Duarte. As per,
Claims 6, 13, 20
Wright / Tremblay / Kupcsik do not explicitly teach, Duarte however in the analogous art of task coordination teaches
further comprising a plurality of head-mounted devices including the head-mounted device, wherein each of the plurality of head-mounted devices comprises a display and one or more cameras, and: (Duarte [0030] “where a manufacturing task involving one or more objects is performed by a plurality of workers (e.g., a team), the system may comprise one or a plurality of augmented-reality display systems (e.g., one for each worker) and may be configured to display respective assistance information to each worker in a controlled spatial relationship to the one or more objects.” Note the plurality of workers involved in the same task each with their own AR device)
the image data is obtained from each of the plurality of head-mounted devices; and (Duarte [0025] “The augmented-reality display system may comprise a sensor for determining a respective position and/or orientation of each of the one or more objects in space—e.g., one or more cameras.”)
the virtual content is rendered in the extended-reality environment for each respective user of the plurality of different users, wherein the virtual content rendered for each respect user is specific to the actions allocated for the respective user from the sequence of actions. (Duarte [0030] “The assistance information for a worker may be determined independently for that worker, or it may depend on the guidance setting of at least one other worker of the plurality of workers. The displayed assistance information may be unique to each worker, or may differ between at least two of the workers”)
Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to modify Wright’s task sequence planning, Tremblay’s network based relationship inferences, and Kupcsik’s optimization to include multiple devices working on the same task in view of Duarte in an effort to improve worker productivity at the task (see Duarte ¶ [0018] & MPEP 2143G).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US 20160124501 A1; WO 2019/021058 A2; Cheng et al., Towards Efficient Human-Robot Collaboration With Robust Plan Recognition and Trajectory Prediction, 2020.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED EL-BATHY whose telephone number is (571)270-5847. The examiner can normally be reached on M-F 8AM-4:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, PATRICIA MUNSON can be reached on (571) 270-5396. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED N EL-BATHY/Primary Examiner, Art Unit 3624
Prosecution Insights