DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/04/2026 was filed and is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Interpretation
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
"query module is configured to" in claim 1 described in paragraph [0095] and implemented on hardware disclosed in paragraphs [0022, 0025-0034, 0057].
"query parsing module…to validate" in claim 1 described in paragraph [0100-0104] and implemented on hardware disclosed in paragraphs [0022, 0025-0034, 0057].
"guardrail module to validate" in claim 1 described in paragraph [0095] and implemented on hardware disclosed in paragraphs [0022, 0025-0034, 0057].
“scene understanding module configured to" in claim 1 described in paragraph [0091] and implemented on hardware disclosed in paragraphs [0022, 0025-0034, 0057].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brienza (“Multi-agent Planning using Visual Language Models”, 2024), in view of OpenAI (“GPT-4 Technical Report”, 2024).
Regarding claims 1, 6, 11, Brienza teaches An alert generating system, comprising:
a user interface configured to receive a user query about a scene (Brienza, pg 2, column 1, first full paragraph, reproduced below:
PNG
media_image1.png
368
774
media_image1.png
Greyscale
. “picture of the scene” and “a textual query” are being interpreted as involving a user query. These inputs are being interpreted as using a user interface to accomplish this as they do not appear out of thin air);
a query module (Brienza, see nearest image above, “picture of the scene” and “a textual query” are being interpreted as involving a user query) comprising … wherein the query module is configured to receive the user query from the user interface (Brienza, see pg 2, column 1 image above, “picture of the scene” and “a textual query” are being interpreted as involving a user query. These inputs are being interpreted as using a user interface to accomplish this as they do not appear out of thin air)
a scene understanding module configured to process the scene to generate a frame document that comprises information about an object in the scene;
a reasoning engine configured to carry out a rule-based check (Brienza, pg 3, column 2, first full paragraph:
PNG
media_image2.png
300
762
media_image2.png
Greyscale
. “The SKM Agent identifies object classes within the image and establishes the scene’s ontology”, which is being interpreted as involving a rule-based check) and a VLLM reasoning check (Brienza, see nearest image above, “The GKM Agent grounds these objects, providing short descriptions that include their relationships with surrounding objects”. Which is being interpreted as involving VLLM reasoning check) based on the validated user query (Brienza, see pg 2 image above, “picture of the scene” and “a textual query” are being interpreted as involving a user query. These inputs are being interpreted as being validated, as the non-limiting example of GPT-4, PHOSITA would know GPT-4 has guardrails to validate user query, illustrative reference here: https://www.searchenginejournal.com/research-gpt-4-jailbreak-easily-defeats-safety-guardrails/498386/ ) and the frame document (Brienza, see nearest image above, “scene description” and “relationships between objects” are being interpreted as non-limiting examples involving “frame document”);
the reasoning engine configured to use at least one of rule-based reasoning (Brienza, see nearest image above, “The SKM [Semantic-Knowledge Miner, cited on pg 3] Agent identifies object classes within the image and establishes the scene’s ontology”, which is being interpreted as involving a rule-based) and VLLM reasoning (Brienza, see nearest image above, “The GKM [Grounded-Knowledge Miner, cited on pg 3] Agent grounds these objects, providing short descriptions that include their relationships with surrounding objects”. Which is being interpreted as involving VLLM reasoning) to generate an alert (Brienza, see nearest image above, “The P Agent [Planner Agent, cited on pg 2] then generates a plan using the information from the SKM and GKM Agents”. The plan is being interpreted as a non-limiting example involving an alert) based on the validated user query () and the frame document (Brienza, see nearest image above, “scene description” and “relationships between objects” are being interpreted as non-limiting examples involving “frame document”);
the reasoning engine further configured to transmit an alert packet (Brienza, see nearest image above, “The P Agent [Planner Agent, cited on pg 2] then generates a plan using the information from the SKM and GKM Agents”. The plan is being interpreted as a non-limiting example involving an alert) to a computing device (Brienza, pg 2, Figure 2, “Output plan” being sent to for Evaluation is being interpreted as a non-limiting example of “to a computing device”); and
wherein the alert packet causes the computing device to generate a visual notification of the alert (Brienza, pg 2, Figure 2, “Output plan” being sent to for Evaluation is being interpreted as a non-limiting example of “to a computing device”. The output plan example is being interpreted as a non-limiting example of a visual notification, as you can see the text).
However, Brienza does not appear to explicitly teach validated user query. Although, Brienza does teach GPT-4, which has text parsing and user query validation built-in, as shown in the next reference.
Pertaining to the same field of endeavor, OpenAI teaches
a query parsing module (OpenAI, pg 12-13, Section Model-Assisted Safety Pipeline, reproduced below:
PNG
media_image3.png
679
924
media_image3.png
Greyscale
. “Then on the set of safety-relevant training prompts, which request harmful content such as illicit advice, we can reward GPT-4 for refusing these requests”. Which shows the text is parsed to look for harmful content. For a visual example of text parsing, see Figure 1 at the following illustrative reference: https://aclanthology.org/2024.acl-long.384/.)
and to use the query parsing module to validate the user query to generate a validated user query (OpenAI, see nearest image above, “we can reward GPT-4 for not refusing requests on a subset of prompts guaranteed to be safe and answerable”. Which is being interpreted as involving validating the user query);
Brienza and OpenAI are considered to be analogous art because they are directed to large language models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for multi-agent vision language model scene understanding (as taught by Brienza) to include validated user queries (as taught by OpenAI) because the combination provides an improvement for model output safety (OpenAI, pg 12-13, Section Model-Assisted Safety Pipeline). Further, Brienza cited GPT-4, and the OpenAI technical report is on GPT-4, the validate user query is inherent to GPT-4.
Regarding claims 7, Brienza teaches The alert generating system of claim 6, wherein the query module further comprises a guardrail module (OpenAI, see nearest image below, “Model-Assisted Safety Pipeline” is being interpreted as involving a guardrail module), and wherein the guardrail module is configured to filter prohibited user queries (OpenAI, pg 12-13, Section Model-Assisted Safety Pipeline, reproduced below:
PNG
media_image3.png
679
924
media_image3.png
Greyscale
. “Then on the set of safety-relevant training prompts, which request harmful content such as illicit advice, we can reward GPT-4 for refusing these requests”. “Refusing these requests” are being interpreted as involving “filter prohibited user queries”).
Regarding claim 8, Brienza teaches The alert generating system of claim 7, wherein the prohibited user queries are filtered based on a prohibited word or a prohibited context (OpenAI, pg 12-13, Section Model-Assisted Safety Pipeline, reproduced below:
PNG
media_image3.png
679
924
media_image3.png
Greyscale
. “Then on the set of safety-relevant training prompts, which request harmful content such as illicit advice, we can reward GPT-4 for refusing these requests”. “Refusing these requests” are being interpreted as involving “filtered based on a prohibited word or a prohibited context”. Look at Table 6 and Table 7 for non-limiting examples of prohibited context).
Regarding claim 9, Brienza teaches The alert generating system of claim 6, wherein the query parsing module is configured to determine a set of models that are necessary (Brienza, pg 3, column 2, lines 1-2: “GPT-4V is used for agents that process images, while GPT-4 is used for the planning agent [1].” GPT-4V and GPT-4 are being interpreted as non-limiting examples “a set of models” that are determined as necessary) to process the user query (Brienza, pg 2, column 1, first full paragraph: “All we need is a picture of the scene that captures the most relevant aspects of the environment, and a textual query on the goal we want to achieve.”).
Regarding claim 10, Brienza teaches The alert generating system of claim 6, wherein the query parsing module is further configured to select a set of models (Brienza, see nearest image below, “SKM Agent” and “GKM Agent” are being interpreted as non-limiting examples of “a set of models” that were selected) for the scene understanding module to use to generate the frame document (Brienza, pg 3, column 2, first full paragraph:
PNG
media_image2.png
300
762
media_image2.png
Greyscale
. “scene description” and “relationships between objects” are being interpreted as non-limiting examples involving “frame document”).
Claims 2 is rejected using the same rationale and motivation as applied to claim 7 discussed above.
Claims 3 and 12 is rejected using the same rationale and motivation as applied to claim 7 discussed above.
Claims 4 and 13 is rejected using the same rationale and motivation as applied to claim 8 discussed above.
Claim 5 and 14 is rejected using the same rationale and motivation as applied to claim 10 discussed above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Wang et al ("Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs”, 2024) disclose multi-agent Visual LLMs that output an alert (plan).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY B DUONG whose telephone number is (571)272-1358. The examiner can normally be reached Monday - Thursday 10a-9p (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.B.D./Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667