DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Status of Claims
Claims 1-8 are currently pending and are being hereby examined herein.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy of DE102023210950.6, filed on 3 November 2023, was received on 24 February 2025.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 6 December 2024 has been considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f), except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“Control device configured to” in Claim 7
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The structure of a “control device” is a computer, or equivalents thereof (see specification page 17).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f), applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f).
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function. Such claim limitation(s) is/are:
“comprising the following steps” in Claim 1
“to perform the following steps” in Claim 8
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f), it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Objections
The claims are objected to because of the following informalities:
Claims 1, 7, and 8: “agen” should be “[[agen]] agent”.
Claim 2: “further training” should be “further comprising training”.
Claim 3: an Oxford comma should be added.
Appropriate corrections are required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 2 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Claims 2 recites “textual state descriptions”. Claim 1 introduced “converting the state information about the agent into a textual state description”. One of ordinary skill in the art would not know if the “textual state descriptions” of Claim 2 are different from or the same as the recitation in Claim 1. For the purposes of compact prosecution, the examiner will assume either of these could be true. Appropriate corrections are required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-6 are directed to a method. Claim 7 is directed to an apparatus. Claim 8 is directed to a non-transitory computer readable medium. Therefore Claims 1-8 are directed to a statutory category.
Step 2A Prong One: the claimed inventions are directed to abstract ideas. The abstract ideas, specifically mental processes, in Claim 1 (the independent method claim) are as follows:
collecting state information about an agent located in an environment of the robot apparatus
converting the state information about the agent into a textual state description
generating a task plan for the robot apparatus taking into account the prediction
Claims 7 and 8 recite similar abstract ideas. Claim 6 recites an additional abstract idea: generating a scene graph from the collected state information and from further information about the environment of the robot apparatus and generating the textual state description from the scene graph. These are abstract ideas because they can easily be completed in the human mind. For example, a human can think a text description of the environment of a robot.
The additional elements in Claims 1-8 are as follows:
A method for controlling a robot apparatus (Claim 1)
feeding the textual state description to a large language model for generating a prediction of a behavior of the agen and/or of a future state of the agent and/or of the environment (Claim 1, similar limitations in Claims 7 and 8)
controlling the robot apparatus according to the task plan (Claim 1, similar limitations in Claims 7 and 8)
training the large language model for predicting agent behavior from textual state descriptions (Claim 2)
generating the task plan by feeding a textual goal description, a textual environment description and a textual description of a state of the robot apparatus to the large language model or a further large language model (Claim 3)
generating the task plan by feeding the prediction and a textual goal description, a textual environment description, and a textual description of the state of the robot apparatus, to a further large language model (Claim 4)
training the further large language model for generating task plans (Claim 5)
A control device configured to control a robot apparatus (Claim 7)
A non-transitory computer-readable medium on which are stored commands for controlling a robot apparatus, the commands, when executed by a processor, causing the processor to perform (Claim 8)
Step 2A Prong Two: the additional elements, individually and in combination, fail to integrate the abstract idea into a practical application. Additional elements a, b, d, e, f, and g merely link the judicial exception to a technological environment (see MPEP 2106.05(h)). Additional elements h and i merely apply the abstract idea to one or more generic computing components (see MPEP 2106.05(f)). Additional limitation c is merely insignificant extra solution activity (see MPEP 2105.05(g)).
Step 2B: the additional elements, individually and in combination, fail to amount to significantly more than the judicial exception because the office takes Official Notice they are well-understood, routine, and convention activity previously known to the industry, specified at a high-level of generality (See MPEP 2106.05(d)), or else they are insignificant post-solution activity in the form of mere data outputting (additional element c) (see numerous court decisions pertaining to observations, evaluations, judgements, and opinions, such as the findings from Electric Power Group where it was found that collecting information, analyzing it, and outputting certain results of the collection and analysis was not significantly more than the judicial exception – see MPEP 2106.05(d)(II)).
To overcome the rejections under 35 U.S.C. 101, Applicant should claim a discrete control step. As currently claimed, the task plan may be to do nothing and controlling a robot to follow a task plan that includes nothing is not a discrete control step. As one possible option, Applicant could amend all independent claims to claim controlling arms of the robot apparatus and/or wheels of the robot apparatus to follow a motion trajectory based on the task plan (as supported by page 11 of the specification).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3 and 6-8 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by arXiv article GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning (Graule and Isler, hereinafter, Graule).
Regarding Claim 1, Graule discloses A method for controlling a robot apparatus, comprising the following steps:
collecting state information about an agent located in an environment of the robot apparatus; converting the state information about the agent into a textual state description (see at least page 3 column 2 and Fig. 2: “System overview. Our system receives observations on past human activity from connected home appliances, cameras, and other sensors (a), and relies on a narrator module that translates these signals into a natural language narration of past human activities”; “a list of items available in the environment”);
feeding the textual state description to a large language model for generating a prediction of a behavior of the agen and/or of a future state of the agent and/or of the environment (see at least page 1 column 2 and Fig. 2: “We present an approach that addresses these challenges by leveraging large language models (LLMs) to infer a human’s probable next actions”; “anticipate the objects with which a human is likely to interact next”; “narration is passed into the symbolic-geometric reasoning module (c). This module leverages an LLM to predict future actions from the narration of past activities”);
generating a task plan for the robot apparatus taking into account the prediction; and controlling the robot apparatus according to the task plan (see at least page 1 column 2 and Fig. 2: “experiments in which a robot aims to achieve full room coverage while minimizing disruptions to humans in the environment”; “a robot can use these geometrically-grounded predictions to minimize disturbances when planning its tasks”).
Regarding Claim 2, Graule discloses all the limitations of Claim 1. Furthermore, Graule further discloses training the large language model for predicting agent behavior from textual state descriptions (see at least page 1 column 2 and page 3 column 2: “LLMs are trained on extensive text corpora depicting various human behaviors”; “models trained specifically to detect human activities”).
Regarding Claim 3, Graule discloses all the limitations of Claim 1. Furthermore, Graule further discloses generating the task plan by feeding a textual goal description, a textual environment description and a textual description of a state of the robot apparatus to the large language model or a further large language model (see at least Fig. 1, Fig. 2, and TABLE 1: narration and binding sequences are provided to GG-LLM, robot adjusts tasks based on result).
Regarding Claim 6, Graule discloses all the limitations of Claim 1. Furthermore, Graule further discloses generating a scene graph from the collected state information and from further information about the environment of the robot apparatus and generating the textual state description from the scene graph (see at least page 3 column 2 and Fig. 1: “The symbolic reasoning submodule projects with which objects the user is likely interacting next. It takes as input the language narration of the history object Nh, as well as a list of items available in the environment (which can be extracted from the semantic map M). It assigns a relevancy score si to each of the items in M”).
Regarding Claim 7, this claim is substantially similar to Claim 1, and rejected for the same reasons as Claim 1.
Regarding Claim 8, this claim is substantially similar to Claim 1, and rejected for the same reasons as Claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Graule in view of U.S. Pub. No. 2024/0351218 (Porter et al., hereinafter, Porter).
Regarding Claim 4, Graule discloses all the limitations of Claim 1. Graule discloses additional information (for generating the task plan) is the prediction. Graule does not explicitly disclose generating the task plan by feeding the prediction and a textual goal description, a textual environment description, and a textual description of the state of the robot apparatus, to a further large language model.
Porter, in the same field of robot controls, and therefore analogous art, teaches generating the task plan by feeding additional information and a textual goal description, a textual environment description, and a textual description of the state of the robot apparatus, to a further large language model (see at least [0014], [0017], [0019]-[0020], [0027], and [0029]: “an LLM may be used to generate a series of navigation tasks and/or environmental interaction tasks”; “the LLM 120 may be instructed that if generating navigation tasks for the cobot 130, the LLM 120 is to provide responses as a sequence of discrete tasks”; “the user 140 may provide a baseline of syntax and/or grammar to the LLM 120 to identify the format and style in which the LLM 120 is to provide responses or navigation tasks when responding to queries”; “the cobot 130 may identify a workbench, a storage shelf, what is on the storage shelf, a refrigerator, a medicine cabinet, or other information in constructing a list of known locations. In some embodiments, the cobot 130 may provide this information to the LLM 120 such that as the LLM 120 generates tasks for the cobot 130 to perform”; “In some embodiments, the LLM 120 and/or the cobot 130 may interact with one or more dynamic object navigation techniques to facilitate operation of the cobot 130 in an environment with people which move about dynamically. For example, when the LLM 120 generates a set of discrete tasks, the LLM 120 may include consideration of dynamically moving objects such that the cobot 130 may navigate or move in a manner that accounts for moving people or other dynamically moving objects. In these and other embodiments, the auditing may include verification of navigation tasks, speeds, etc. to confirm that the navigation tasks take into account people moving in or near where the cobot 130 is operating”; “the LLM 120 may generally track what is available such that the LLM 120 may generate discrete tasks and/or computer-readable code for the cobot 130 to navigate to, retrieve, and deliver an item even without ever having previously scanned the room or identified, but instead being based on the previous scanning or database provided to the LLM 120 in other ways”).
It would have been obvious, before the effective filing date of the invention, with a reasonable expectation of success, to one having ordinary skill in the art, to substitute the task plan generation using an LLM of Porter for the task plan generation of Graule. Graule is directed to a robotic vacuum or similar (see at least Graule Fig. 1), and one of ordinary skill in the art would be motivated by the need for alternate complex task planning (i.e., using the LLM), as disclosed in Porter, for a robot instead designed for different tasks (see at least Porter FIG. 1: cobot has arms for doing different tasks than Graule).
Regarding Claim 5, the Graule and Porter combination teaches all the limitations of Claim 4. Furthermore, Porter further teaches (with the same motivation to combine as Claim 4) training the further large language model for generating task plans (see at least [0028]: “the LLM 120 may take into account local context as well as a larger, more global context. For example, with respect to the local context, the LLM 120 may account for specific instructions given either previously in a previous session of interaction, or in a current session of interaction. For example, the LLM 120 may have been instructed in a previous session that it is unsafe to transport batteries and water together. If a user asks the cobot 130, “Get me a drink of water and some more batteries,” the LLM 120 may recognize that such a combination would create an unsafe condition if undertaken in a single trip and so may generate a set of discrete navigation tasks that makes two trips, one for water and one for batteries. In these and other embodiments, the cobot 130 may provide a response indicating that it will make two trips as transporting water and batteries together is unsafe”).
Additional Relevant Art
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure and may be found on the accompanying PTO-892 Notice of References Cited:
U.S. Pub. No. 2024/0253224 which teaches robot control methods leveraging large language models.
U.S. Pub. No. 2025/0094731 which is similar in subject matter to the primary reference “Graule”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDRA ROBYN MORFORD whose telephone number is (571)272-6109. The examiner can normally be reached Monday - Friday 8:00 AM - 4:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached at (571) 272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.R.M./Examiner, Art Unit 3658
/THOMAS E WORDEN/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights