Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1 – 7, 9 – 12 and 21 – 23 remain pending.
Claims 1, 3 – 5, 9, 10 and 12 are Amended
Claims 8, 13 – 20 have been canceled.
Response to Arguments
Applicant's arguments filed May 15, 2026 with respect to claims 1 – 7, 9 – 12 and 21 – 23 have been fully considered but are moot because the new grounds of rejection do not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 – 3 and 5 – 7 are rejected under 35 U.S.C 103 as being unpatentable over Cupersmith et al. US Patent Publication No. US-20210304559-A1 (hereinafter Cupersmith) in view of Chen Patent Application Publication No. WO-2024059179-A1 (hereinafter Chen).
Regarding claim 1, Cupersmith discloses about a robotic system comprising: a robot operable in an environment, the environment comprising a plurality of objects, the plurality of objects including a first object and a second object (Cupersmith in [0038] discloses, “a robot management system server is configured to manage a fleet of service robots that are deployed within a casino property ... dynamic data captured by on-board sensors for position determination and pathing, as well as on-board sensors for object detection and avoidance”. Furthermore, Cupersmith in [0114] discloses about detecting plurality of objects, “robot 300 to detect collisions between the robot 300 and other stationary or moving objects as the robot 300 moves through the operations venue or to detect other impacts to the robot 300”); an object recognition subsystem communicatively coupled to the robot (Cupersmith in [0038] discloses, “dynamic data captured by on-board sensors for position determination and pathing, as well as on-board sensors for object detection and avoidance” wherein ‘on-board’ sensors with robot implies coupled to the robot); wherein the object recognition subsystem comprises at least one processor and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing processor-executable instructions and/or data that, when executed by the at least one processor, cause the robotic system to perform a method for recognizing objects in the environment (Cupersmith in [0040] discloses, “The service robots are configured with various hardware components that enable their intended operations, also referred to herein as the “service role” of the robot. For example, the robots may include hardware and sensors that enable movement of the robot and navigation throughout the venue (e.g., position determination, obstacle avoidance) ... The service robots also include central processing components that include one or more central processing units (“CPUs”), volatile and non-volatile memory, and a rechargeable power supply system configured to provide power for components of the robot”. Furthermore, Cupersmith in [0038] discloses about service robots detecting objects).
Cupersmith doesn’t disclose about the following limitation as further recited in the claim.
Chen discloses an interface to a large language model (LLM) communicatively coupled to the object recognition system (Chen in [0021] discloses, “a large language model (LLM) can be utilized for the planning, and the determined object descriptor(s) can be processed, using the LLM and along with the FF NL instruction”), the method which includes: identifying the first object (Chen in [0033] discloses, “Further, FIG. 1B2 illustrates that the candidate object descriptor "pear" has been determined to be relevant to the region of interest 184A”); determining that a degree of confidence in identifying the first object exceeds a threshold; assigning a first label to the first object; identifying the second object; determining that a degree of confidence in identifying the second object fails to exceed the threshold (Chen in [0032] discloses about identifying first and second object, “For instance, "apple", "pear", and "banana" of the object descriptors 107 can be generated based on being determined to be members of a "fruit" class, and "fruit" being included in the FF NL instruction 105. Also, for instance, "sink" and "brush" can be generated based on prompting a large language model (LLM) using the FF NL instruction 105”. Furthermore, Chen in [0044] discloses about threshold for any object descriptor implies to first and second object having a threshold, “Any object descriptors, whose text embedding is not close to any (or at least a threshold quantity of) of the region embeddings, as indicated by the comparison (e.g., corresponding measure(s) fail to satisfy threshold(s)), can be excluded from a subset of the superset of object descriptors. Any object descriptors, whose text embedding is close to any (or at least a threshold quantity of) the region embeddings, can be included in a subset of the superset of object descriptors”); sending, by the interface, a query to the LLM, the query comprising the first label and a request to suggest a plurality of other objects that are likely to be in the same environment as the first object; receiving, by the interface, a response from the LLM, the response in reply to the query, the response comprising a list of labels for the plurality of objects likely to be in the same environment as the first object a second label; matching the second object to a second label in the list of labels in the response from the LLM; and assigning the second label to the second object (Chen in [0032] discloses about “potentially relevant” object equates to the list of object likely to be in the same environment, “even though the FF NL instruction 105 doesn't mention "sink", "brush", or any synonyms, prompting the LLM and analyzing resulting LLM output can still result in those object descriptors being determined to be potentially relevant to the task of the FF NL instruction 105” wherein second label could be sink or brush).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Chen into the system of Cupersmith because it would allow the system to label a new object meaningfully even when there is no perfect match of the object in the database.
Summary of Citations (Chen)
Paragraph [0032]; “For instance, "apple", "pear", and "banana" of the object descriptors 107 can be generated based on being determined to be members of a "fruit" class, and "fruit" being included in the FF NL instruction 105. Also, for instance, "sink" and "brush" can be generated based on prompting a large language model (LLM) using the FF NL instruction 105. Notably, even though the FF NL instruction 105 doesn't mention "sink", "brush", or any synonyms, prompting the LLM and analyzing resulting LLM output can still result in those object descriptors being determined to be potentially relevant to the task of the FF NL instruction 105”.
Paragraph [0033]; “Further, FIG. 1B2 illustrates that the candidate object descriptor "pear" has been determined to be relevant to the region of interest 184A”.
Paragraph [0044]; “Any object descriptors, whose text embedding is not close to any (or at least a threshold quantity of) of the region embeddings, as indicated by the comparison (e.g., corresponding measure(s) fail to satisfy threshold(s)), can be excluded from a subset of the superset of object descriptors. Any object descriptors, whose text embedding is close to any (or at least a threshold quantity of) the region embeddings, can be included in a subset of the superset of object descriptors”.
Paragraph [0064]; “The system can process all or portions of the FF NL input, using the LLM, in generating object descriptor(s) at sub-block 454B. As one example of sub-block 454B, if the FF NL instruction is "light up the room", the system can prompt the LLM, based on the FF NL instruction, to generate LLM output that indicates object descriptor(s) that include "switch". It is noted that the LLM output can indicate "switch" despite the FF NL instruction not including that term or any synonyms of that term”.
Regarding claims 2 and 3, the combination of Cupersmith and Chen as a whole teaches claim 1, and Chen teaches claims 2 and 3 for the same grounds of rejection from the Non-Final Office Action of 12/17/2025.
Regarding claim 5, Chenin the combination discloses he robotic system of claim 3, wherein the determining a degree of confidence in the identifying of the first object exceeds a threshold includes determining a probability (Chen in [0044] discloses about threshold for any object descriptor implies to first object having a threshold, “Any object descriptors, whose text embedding is not close to any (or at least a threshold quantity of) of the region embeddings, as indicated by the comparison (e.g., corresponding measure(s) fail to satisfy threshold(s)), can be excluded from a subset of the superset of object descriptors”).
Regarding claim 6, the combination of Cupersmith and Chen as a whole teaches claim 1, and Chen teaches claim 6 for the same grounds of rejection from the Non-Final Office Action of 12/17/2025.
Regarding claim 7, the combination of Cupersmith and Chen as a whole teaches claim 1, and Cupersmith teaches claim 7 for the same grounds of rejection from the Non-Final Office Action of 12/17/2025.
Claims 4 and 9 – 12 are rejected under 35 U.S.C 103 as being unpatentable over Cupersmith et al. in view of Chen and further in view of Fairfield US Patent Publication No. US-10761542-B1 (hereinafter Fairfield).
Regarding claim 4, Chen in the combination discloses the robotic system of claim 3, wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement (Chen in [0064] discloses, “At sub-block 454B, the system prompts an LLM, based on the FF NL instruction, to generate object descriptor(s)”), and the request to suggest the plurality of other objects that are likely to be in the same environment as the first object (Chen in [0032] discloses about “potentially relevant” object equates to the list of object likely to be in the same environment, “even though the FF NL instruction 105 doesn't mention "sink", "brush", or any synonyms, prompting the LLM and analyzing resulting LLM output can still result in those object descriptors being determined to be potentially relevant to the task of the FF NL instruction 105” wherein second label could be sink or brush).
Cupersmith and Chen in the combination doesn’t disclose about the following limitation as further recited in the claim.
Fairfield discloses about the natural language statement comprising the natural language label assigned to the first object (Fairfield in [Column – 27, Line 32 -36] discloses, “the computing system or other computing entity may generate a natural-language question, statement, and/or any other alertness data information based on a result of the vehicle's object detection of the object”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Fairfield into the system of Cupersmith in view of Chen because it will improve the relevance and accuracy of LLM response. Without the label the response would be ambiguous.
Regarding claim 9, Cupersmith in the combination discloses the robotic system of claim 1.
Cupersmith and Chen in the combination doesn’t disclose about the following limitation as further recited in the claim.
Fairfield discloses about the assigning the second label to the second object includes updating the degree of confidence in the identifying of the second object (Fairfield in [Column – 16, Line 66 – 67 & Column – 17, Line 1 – 8] discloses about asking for human intervention if the confidence score is below threshold. Additionally, [Column – 21, Line 33 – 36] discloses about confirming the identification implies to updating the degree of confidence for identifying the object).
Summary of Citations (Fairfield)
[Column – 16, Line 66 – 67 & Column – 17, Line 1 – 8]; “the natural-language question may be based on the preliminary identification of the object, so as to ask the human operator to confirm whether the preliminary identification is correct. In either case, the natural-language question may not include the correct identity of the object in some scenarios. For instance, if the vehicle has threshold low confidence that the object is a traffic signal with a green light, even though the object in reality is a traffic signal with a red light, the natural-language question may read “Is the light in this traffic signal green?””
[Column – 21, Line 33 -36]; “In the example depicted in FIG. 4E, the human operator may indicate a natural-language question 420 to identify the object identified as the temporary stop sign 404. Additionally, when an identification is confirmed”.
Regarding claim 10, Chen in the combination discloses the robotic system of claim 1, wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement (Chen in [0064] discloses, “At sub-block 454B, the system prompts an LLM, based on the FF NL instruction, to generate object descriptor(s)”).
Cupersmith and Chen in the combination doesn’t disclose about the following limitation as further recited in the claim.
Fairfield discloses about the natural language statement comprising the first label (Fairfield in [Column – 27, Line 32 -36] discloses, “the computing system or other computing entity may generate a natural-language question, statement, and/or any other alertness data information based on a result of the vehicle's object detection of the object”) and the request to suggest the plurality of other objects that are likely to be in the same environment as the first object (Fairfield in [Column – 14, Line 34 – 38] discloses, “the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, street signs, other vehicles, indicator signals on other vehicles, and other various objects detected in the captured environment data”).
Summary of Citations (Chen)
Paragraph [0064]; “At sub-block 454B, the system prompts an LLM, based on the FF NL instruction, to generate object descriptor(s)”.
Summary of Citations (Fairfield)
[Column – 14, Line 34 – 38]; “the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, street signs, other vehicles, indicator signals on other vehicles, and other various objects detected in the captured environment data”.
[Column – 27, Line 32 -36]; “the computing system or other computing entity may generate a natural-language question, statement, and/or any other alertness data information based on a result of the vehicle's object detection of the object”.
Regarding claim 11, the combination of Cupersmith, Chen and Fairfield as a whole teaches claim 1, and Chen teaches claim 11 for the same grounds of rejection from the Non-Final Office Action of 12/17/2025.
Regarding claim 12, Chen in the combination discloses the robotic system of claim 1, wherein the receiving, by the interface, a response from the LLM includes: receiving a natural language statement, the natural language statement comprising the list of labels for the plurality of objects likely to be in the same environment as the first object a natural language label; and parsing the natural language statement to extract the list of labels for the plurality of objects likely to be in the same environment as the first object natural language label (In [0063] Chen discloses that the system can extract (parse) object label, “the system can extract noun(s) and/or adjective(s) from the FF NL instruction directly. For instance, if the NL instruction is "give me some first-aid items", "first-aid items" can be extracted”. Furthermore, In [0064] Chen discloses about extracting the object label switch from the output of a LLM, “if the FF NL instruction is "light up the room", the system can prompt the LLM, based on the FF NL instruction, to generate LLM output that indicates object descriptor(s) that include "switch". It is noted that the LLM output can indicate "switch" despite the FF NL instruction not including that term or any synonyms of that term” wherein switch could be an object that is detected to be in the same environment as light. Furthermore, Chen in [0032] discloses about “potentially relevant” object equates to the list of object likely to be in the same environment, “even though the FF NL instruction 105 doesn't mention "sink", "brush", or any synonyms, prompting the LLM and analyzing resulting LLM output can still result in those object descriptors being determined to be potentially relevant to the task of the FF NL instruction 105”).
Claim 21 is rejected under 35 U.S.C 103 as being unpatentable over Cupersmith in view of Chen and further in view of Bostick US Patent Publication No. US-9717607-B1 (hereinafter Bostick).
Regarding claim 21, Chen in the combination discloses the robotic system of claim 2.
Cupersmith and Chen in the combination doesn’t disclose about the following limitation as further recited in the claim.
Bostick discloses the robot includes haptic sensors, and wherein: the detecting, by the sensor data processor, the presence of the first object and the second object includes establishing respective haptic profiles of the first object and the second object; identifying the first object includes identifying the first object based at least in part on the haptic profile of the first object; and identifying the second object includes identifying the second object based at least in part on the haptic profile of the second object (Bostick in [Column – 2, Line – 35 – 38] discloses, “Eye focus, facial expression, body movement, biometrics, and local sensor haptic feedback are used to determine the object and task intended by a user of a prosthetic device”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Bostick into the system of Cupersmith in view of Chen because it would improve the object recognition of the system since some object are hard to identify visually but easier to identify by touch.
Summary of Citations (Bostick)
[Column – 2, Line – 35 – 38]; “Eye focus, facial expression, body movement, biometrics, and local sensor haptic feedback are used to determine the object and task intended by a user of a prosthetic device”.
Claim 22 and 23 are rejected under 35 U.S.C 103 as being unpatentable over Cupersmith in view of Chen and further in view of Yoshida Patent Application Publication No. WO-2021125019-A1 (hereinafter Yoshida).
Regarding claim 22, Chen in the combination discloses the robotic system of claim 1.
Cupersmith and Chen in the combination doesn’t disclose about the following limitation as further recited in the claim.
Yoshida discloses the request to suggest the plurality of other objects that are likely to be in the same environment as the first object (Yoshida in [0037] discloses, “In the present specification, association data is data in which a plurality of regions and a plurality of labels in an environment are associated with each other”) includes a request to rank the list of labels according to an estimate of the degree of confidence that each label matches the second object (Yoshida in [0039] discloses, “the generation unit 317 generates a label list in which one or more labels pertaining to requests for labels identifying the recognition unsuitable objects are arranged in a predetermined order, for example, in the order of recommending that they correspond preferentially to the user ... As a result of the comparison, the generation unit 317 may select, from the output labels, a label overlapping between the specific label and the output label (hereinafter referred to as an overlapping label). Furthermore, the generation unit 317 of the present embodiment may generate the label list by arranging a plurality of overlapping labels in descending order of the degree of match using the degree of match corresponding to the output label”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Yoshida into the system of Cupersmith in view of Chen because it would make the identification and labelling process more efficient and accurate.
Summary of Citations (Yoshida)
Paragraph [0037]; “In the present specification, association data is data in which a plurality of regions and a plurality of labels in an environment are associated with each other”.
Paragraph [0039]; “the generation unit 317 generates a label list in which one or more labels pertaining to requests for labels identifying the recognition unsuitable objects are arranged in a predetermined order, for example, in the order of recommending that they correspond preferentially to the user ... As a result of the comparison, the generation unit 317 may select, from the output labels, a label overlapping between the specific label and the output label (hereinafter referred to as an overlapping label). Furthermore, the generation unit 317 of the present embodiment may generate the label list by arranging a plurality of overlapping labels in descending order of the degree of match using the degree of match corresponding to the output label”.
Regarding claim 23, Chen in the combination discloses the robotic system of claim 1.
Cupersmith and Chen in the combination doesn’t disclose about the following limitation as further recited in the claim.
Yoshida discloses matching the second object to a second label in the list of labels in the response from the LLM (Yoshida in [0039] discloses, “the generation unit 317 may compare the plurality of labels output from the object recognition DNN (hereinafter referred to as output labels) with the identified plurality of labels (hereinafter referred to as specific labels) for objects unsuitable for recognition. As a result of the comparison, the generation unit 317 may select, from the output labels, a label overlapping between the specific label and the output label (hereinafter referred to as an overlapping label)”) includes increasing the degree of confidence in identifying the second object to exceed the threshold (Yoshida in [0035] discloses, “In a case that the degree of match is equal to or less than the threshold value, the determination unit 313 determines that the recognition of the task target object is not appropriate. Hereinafter, a task target object determined by the determination unit 313 that recognition of the task target object is not appropriate is referred to as a recognition unsuitable object”).
Summary of Citations (Yoshida)
Paragraph [0039]; “the generation unit 317 may compare the plurality of labels output from the object recognition DNN (hereinafter referred to as output labels) with the identified plurality of labels (hereinafter referred to as specific labels) for objects unsuitable for recognition. As a result of the comparison, the generation unit 317 may select, from the output labels, a label overlapping between the specific label and the output label (hereinafter referred to as an overlapping label)”.
Paragraph [0035]; “In a case that the degree of match is equal to or less than the threshold value, the determination unit 313 determines that the recognition of the task target object is not appropriate. Hereinafter, a task target object determined by the determination unit 313 that recognition of the task target object is not appropriate is referred to as a recognition unsuitable object”.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZAID MUHAMMAD SALEH whose telephone number is (703)756-1684. The examiner can normally be reached M-F 8 am - 5 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached on (571)272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/ZAID MUHAMMAD SALEH/
Examiner, Art Unit 2668
06/22/2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights