DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 12/03/2025 and 05/14/2026 is/are being considered by the examiner.
Response to Amendment
The Amendment filed 03/30/2026 has been entered. Claims 1-20 remain pending in this application.
Response to Arguments
Applicant’s arguments, see pg. 8-11, filed 03/30/2026, with respect to the rejection(s) of claim(s) 1-20 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Man et al. ("Scenario-Aware Recurrent Transformer for Goal-Directed Video Captioning", 03/04/2022), hereinafter referred to as Man.
With respect to the 35 U.S.C. 103 rejection of claims 1-6, 8-16, and 18-20 under Gadre et al. ("CoWs on PASTURE: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation", 12/14/2022), hereinafter referred to as Gadre, in view of Campos Macias et al. (US Patent Application Publication No. 2023/0048578), hereinafter referred to as Campos Macias, and claims 7 and 17 under Gadre, in view of Campos Macias, and further in view of Huang et al. ("Visual Language Maps for Robot Navigation", 03/08/2023), hereinafter referred to as Huang, neither Gadre, Campos Macias, nor Huang disclose “converting the detected one or more activities into a natural language narration via a generative artificial intelligence (AI) model configured to convert one or more images or video into the natural language narration”. However, Man discloses “converting the detected one or more activities into a natural language narration via a generative artificial intelligence (AI) model configured to convert one or more images or video into the natural language narration” in Fig. 1 and Fig. 2, as well as in the introduction: ““To remedy this, we propose a novel, simple but effective scenario-aware recurrent transformer (SART) model to perform video captioning, as shown in Figure 1.” This inclusion of Man’s teaching of the scenario-aware recurrent transformer (SART) to perform video captioning would have been obvious to combine along with Gadre’s disclosure of language-driven zero-shot object navigation and Campos Macias’ disclosure of a method and system of real-time prediction of human movement within shared workspaces with robots. Utilizing a generative model allows for advantages such as better workflow efficiency, more contextual depth, and expanded accessibility. They are also more cost effective and time effective, as there’s no waiting for a human to manually describe or narrate a video. This description provided by a generative model would also help to better automate systems as a whole, as the system would be able to provide live descriptions of the video in order to provide to the next step or module within the system.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8-16, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gadre et al. ("CoWs on PASTURE: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation", 12/14/2022), hereinafter referred to as Gadre, in view of Campos Macias et al. (US Patent Application Publication No. 2023/0048578), hereinafter referred to as Campos Macias, and further in view of Man et al. ("Scenario-Aware Recurrent Transformer for Goal-Directed Video Captioning", 03/04/2022), hereinafter referred to as Man.
Regarding claim 1, Gadre discloses a method performed by at least one processor, the method comprising: receiving data from one or more sensors corresponding to a first time period ("The agent receives observations and sensor readings I_t (e.g., RGB-D images)," Gadre pg. 3);
extracting one or more items from a map corresponding to the space ("As a CoW moves, it updates a top-down map of the world created using RGB-D observations and pose estimates," Gadre pg. 3);
determining a relevancy score for each of the extracted one or more items based on an output of a language model that receives as input the combination of the narration with the sequence text ("These models provide an interface where one specifies—in text—the arbitrary objects they wish to classify, detect, or segment. For example, CLIP open-vocabulary classifiers compute similarity scores between an input image and a set of user-specified captions," Gadre pg. 1-2 and Gadre Figure 7 shows the scores in a relevance visualization);
correlating the extracted one or more items with one or more locations on the map based on the determined relevancy for each of the extracted one or more items (Gadre Figure 3(b) shows back-projected objected relevance scores provide object goal targets when a CoW has found an object);
and outputting a control signal for controlling a movement of one more devices based on the correlation of the extracted one or more items with the one or more locations on the map ("When a CoW’s confidence exceeds a threshold, it plans to the location of the goal and issues the STOP action," Gadre pg. 3).
However, Gadre fails to disclose converting the detected one or more activities into a natural language narration via a generative artificial intelligence (AI) model configured to convert one or more images or video into the natural language narration; detecting, from the received data, one or more activities of a person in a space; combining the natural language narration with sequence text that corresponds to an action that is sequential to the detected one or more activities.
Campos Macias teaches a method and system of real-time prediction of human movement within shared workspaces with robots.
Campos Macias teaches detecting, from the received data, one or more activities of a person in a space ("As will be discussed in more detail below, the prediction system 200 may include an environment detector 210 that may determine the collaborator's current kinematic state (e.g., current positions, velocities, accelerations, etc. of the joints of the limbs/arms of the collaborator), determine the expected goal of the collaborator's arm/limb movement, and determine the locations, states, and semantics of objects in the environment," Campos Macias para [0025]);
combining the natural language narration with sequence text that corresponds to an action that is sequential to the detected one or more activities ("As will be explained in more detail below, the environment detector 210 may record (e.g., in a memory) and observe a sequence of actual kinematic states over time, which may be used by the short-horizon predictor 230 and/or by the geometric planner 220 to improve trajectory predictions for future points in time," Campos Macias para [0027]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Campos Macias’ method of detecting humans in the shared space. This would allow for the autonomous robots working within that space to operate more safely, efficiently, and collaboratively. They would be able to navigate better and not risk the safety and health of the humans also working within the space.
Man teaches a scenario-aware recurrent transformer for goal-directed video captioning.
Man teaches converting the detected one or more activities into a natural language narration via a generative artificial intelligence (AI) model configured to convert one or more images or video into the natural language narration (“To remedy this, we propose a novel, simple but effective scenario-aware recurrent transformer (SART) model to perform video captioning, as shown in Figure 1”, Man Introduction pg. 2 and Man pg. 3 Fig. 1 and Fig. 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Man’s teaching of the scenario-aware recurrent transformer (SART) to perform video captioning. Utilizing a generative model allows for advantages such as better workflow efficiency, more contextual depth, and expanded accessibility. They are also more cost effective and time effective, as there’s no waiting for a human to manually describe or narrate a video. This description provided by a generative model would also help to better automate systems as a whole.
Regarding claim 2, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 1. Gadre further discloses wherein the correlating the extracted one or more items with the one or more locations on the map further comprises: aggregating the relevancy score of each of the one or more items belonging to a same partition in the map (Gadre Figure 3(b) shows back-projected objected relevance scores provide object goal targets when a CoW has found an object);
wherein a first partition in the map having a higher aggregated relevancy score than a second partition in the map represents a higher likelihood that will be located in the first partition than the second partition during the second time period (Gadre Figure 3(b) shows back-projected objected relevance scores provide object goal targets when a CoW has found an object).
However, Gadre fails to disclose the person.
Campos Macias teaches the person ("As will be discussed in more detail below, the prediction system 200 may include an environment detector 210 that may determine the collaborator's current kinematic state (e.g., current positions, velocities, accelerations, etc. of the joints of the limbs/arms of the collaborator), determine the expected goal of the collaborator's arm/limb movement, and determine the locations, states, and semantics of objects in the environment," Campos Macias para [0025]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Campos Macias’ method of detecting humans in the shared space. This would allow for the autonomous robots working within that space to operate more safely, efficiently, and collaboratively. They would be able to navigate better and not risk the safety and health of the humans also working within the space.
Regarding claim 3, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 2. Gadre further discloses wherein the outputting the control signal for controlling the movement of the one or more devices is based on the aggregated relevancy score of each partition in the map ("As the CoW roams, it keeps track of its confidence about the target object’s location using an object localization module (Sec. 4.3) and its top-down map. When a CoW’s confidence exceeds a threshold, it plans to the location of the goal and issues the STOP action," Gadre pg. 3).
Regarding claim 4, Gadre, in view of Campos Macias, discloses all of the limitations of claim 1. However, Gadre fails to disclose wherein the correlating the extracted one or more items with the one or more locations on the map further comprises: determining, for each path from the person’s current location to the one or more locations on the map, a likelihood that a respective path will be traversed.
Campos Macias teaches wherein the correlating the extracted one or more items with the one or more locations on the map further comprises: determining, for each path from the person’s current location to the one or more locations on the map, a likelihood that a respective path will be traversed ("The geometric planner 220 may take as inputs the estimated goals, the previously observed and forecasted trajectories for the collaborator, and/or the semantic map of the workspace. The geometric planner 220 may use this received information to plot a path between a current state of the collaborator to the estimated goal(s) of the collaborator," Campos Macias para [0031] and "To predict the likely arm/limb movements of a collaborator, multiple possible trajectories may be generated toward each of the collaborator's potential goals, and a likelihood for each possible trajectory may be determined based on comparing the possible trajectory to the observed trajectory," Campos Macias para [0024]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Campos Macias’ method of detecting humans in the shared space. This would allow for the autonomous robots working within that space to operate more safely, efficiently, and collaboratively. They would be able to navigate better and not risk the safety and health of the humans also working within the space.
Regarding claim 5, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 1. Gadre further discloses wherein the map includes, for each extracted item, a semantic label ("We use names shown in Fig. 4 as instance labels, which are minimal descriptions to identify each object," Gadre pg. 4) and a bounding box ("Objects are considered small if their 3D bounding box diagonal is below a threshold," Gadre pg. 4).
However, Gadre does not disclose a position.
Campos Macias teaches a position ("The environment detector 210 may represent the location of obstacles in the environment using an occupancy map," Campos Macias para [0029]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Campos Macias’ method of using a map with positions of objects. This would allow for the autonomous robots working within that space to operate more safely, efficiently, and collaboratively, as they would be informed the locations of objects via the map. They would be able to navigate better and not risk the safety of the humans and objects within the space.
Regarding claim 6, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 1. Gadre further discloses wherein each of the extracted one or more items are located at a distance from that is less than or equal to a distance threshold ("To determine [on top of] relations, we use THOR metadata and to determine [nearness] we use a distance threshold between pairs of objects," Gadre pg. 5).
However, Gadre does not disclose the person.
Campos Macias teaches the person ("As will be discussed in more detail below, the prediction system 200 may include an environment detector 210 that may determine the collaborator's current kinematic state (e.g., current positions, velocities, accelerations, etc. of the joints of the limbs/arms of the collaborator), determine the expected goal of the collaborator's arm/limb movement, and determine the locations, states, and semantics of objects in the environment," Campos Macias para [0025]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Campos Macias’ method of detecting humans in the shared space. This would allow for the autonomous robots working within that space to operate more safely, efficiently, and collaboratively. They would be able to navigate better and not risk the safety and health of the humans also working within the space.
Regarding claim 8, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 1. Gadre further discloses wherein the one or more devices comprises an autonomous robot (Gadre Figure 2 shows the CoW that explores a space and finds an object on its own).
Regarding claim 9, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 8. However, Gadre fails to disclose wherein the outputting the control signal for controlling the movement of the one or more devices further comprises: causing the autonomous robot to avoid the person during the second time period.
Campos Macias teaches wherein the outputting the control signal for controlling the movement of the one or more devices further comprises: causing the autonomous robot to avoid the person during the second time period ("The determined likelihoods may be used by the robot in its motion controller, so when the robot is controlling its joint positions, it may adapt its movement to avoid collisions with the collaborator, and in particular, collisions with the collaborator's arm/limb movements," Campos Macias para [0025]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Campos Macias’ method of detecting humans and avoiding them in the shared space. This would allow for the autonomous robots working within that space to operate more safely, efficiently, and collaboratively. They would be able to navigate better and not risk the safety and health of the humans also working within the space.
Regarding claim 10, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 8. However, Gadre fails to disclose wherein the outputting the control signal for controlling the movement of the one or more devices further comprises: causing the autonomous robot to assist the person during the second time period.
Campos Macias teaches wherein the outputting the control signal for controlling the movement of the one or more devices further comprises: causing the autonomous robot to assist the person during the second time period ("In particular, when the collaborator may use arms/limbs as part of the collaboration (e.g., to lift, move, and/or deliver an object to a robot), it may be helpful to accurately and quickly predict how the arm will move through various kinematic states from its current position to the target position so that the robot's own movements may be controlled to work safely and in tandem with the movements of the arms/limbs of the collaborator," Campos Macias para [0020]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Campos Macias’ method of detecting humans and assisting them in the shared space. This would allow for the autonomous robots working within that space to operate more safely, efficiently, and collaboratively. They would be able to navigate better and not risk the safety and health of the humans also working within the space.
As to claims 11-16 and 18-19, system claims 11-16 and 18-19 and method claims 1-6 and 8-9 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly, claims 11-16 and 18-19 are similarly rejected under the same rationale as applied above with respect to the method claims.
As to claim 20, computer-readable medium (CRM) claim 20 and method claim 1 are related as method and CRM of using same, with each claimed element’s function corresponding to the method step. Accordingly, claim 20 is similarly rejected under the same rationale as applied above with respect to the method claim.
Claim(s) 7 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gadre, in view of Campos Macias, further in view of Man, and further in view of Huang et al. ("Visual Language Maps for Robot Navigation", 03/08/2023), hereinafter referred to as Huang.
Regarding claim 7, Gadre, in view of Campos Macias, and further in view of Man, discloses all of the limitations of claim 1. However, Gadre fails to disclose wherein the language model is a large language model (LLM) trained on a text corpus. Huang teaches using visual language maps for robot navigation.
Huang teaches wherein the language model is a large language model (LLM) trained on a text corpus ("Specifically, when combined with large language models (LLMs), VLMaps can be used to (i) translate natural language commands into a sequence of open-vocabulary navigation goals," Huang Abstract pg. 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gadre’s method of language-driven object navigation by including Huang’s method of using a large-language model (LLM) for robot navigation. LLMs would allow for direct human speech and thereby natural language to be interpreted by the robot, allowing for the robot to navigate with much more accuracy and efficiency to targets that require semantical understanding.
As to claim 17, system claim 17 and method claim 7 are related as method and system of using same, with each claimed element’s function corresponding to the method step. Accordingly, claim 17 is similarly rejected under the same rationale as applied above with respect to the method claim.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADAM MICHAEL WEAVER whose telephone number is (571)272-7062. The examiner can normally be reached Monday-Friday, 8AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at (571) 272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADAM MICHAEL WEAVER/ Examiner, Art Unit 2658
/RICHEMOND DORVIL/ Supervisory Patent Examiner, Art Unit 2658
Prosecution Insights