DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of US Provisional Application No. 63624632, filed 01/24/2024 but claims 1-20 contain subject matter not disclosed in the provisional application, thus the effective filing date of 01/14/2025 has been used.
If applicant intends to have these claims afforded the benefit of the earlier filing date, applicant may: (1) amend the claim limitation(s) to avoid it/them containing subject matter not in the provisional application; or (2) present a sufficient showing that the claim limitation(s) recite(s) material from specific sections of the provisional application.
Information Disclosure Statement
The IDS(s) dated 10/24/2025 has/have been considered and placed in the application file.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 11-14, and 19-20 is/are rejected under 35 U.S.C. 103 as obvious over Siebel et al (US 20240202225 A1, hereafter referred to as Siebel).
Claim 1
Regarding Claim 1, Siebel teaches A computer-implemented method for sequence generation using hierarchical machine-learned agents, comprising: obtaining, by one or more computing devices, a first machine-learned sequence processing model configured to use a plurality of first tools (Siebel in ¶¶23, 28-29 discloses an orchestrator using a large language model to process input and select from available agents and tools), wherein at least one first tool of the plurality of first tools is a second machine-learned sequence processing model configured to use one or more second tools (Siebel in ¶¶23, 64, 70-71 discloses callable agent modules that include LLMs and have their own tools. The proposed configuration exposes the learned decision model as the callable worker resource);
obtaining, by the one or more computing devices, an input context (Siebel in ¶¶28-29, 148 discloses receiving a query and supplying its preprocessed context to the learned orchestrator);
selecting, by the one or more computing devices using the first machine-learned sequence processing model based at least in part on the input context, a first tool of the plurality of first tools (Siebel in ¶¶28-29 discloses using the orchestrator model to interpret input and select the appropriate agent), wherein the first tool selected is the second machine-learned sequence processing model (Siebel in ¶¶23, 64, 70 discloses selection of an agent that may include an LLM. The proposed configuration makes its decision model the selected callable worker);
selecting, by the one or more computing devices using the second machine-learned sequence processing model, at least one second tool of the one or more second tools (Siebel in ¶¶28-30, 71, 228 discloses independent agent tool designation, model determined choice sequences and learned tool instruction generation. The proposed configuration assigns the tool choice operation to the callable worker decision model);
generating, by the one or more computing devices using the at least one second tool of the one or more second tools, a first output (Siebel in ¶¶30, 64, 149 discloses a selected tool retrieving records that are returned to the worker and used in producing the response).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the tool designation performed by the selected agent of Siebel with the large language model of that agent because Siebel teaches that the agent’s large language model provides reasoning functionality to accomplish the agent’s prescribed set of tasks (¶70), and that domain trained agent models provide accurate, context specific retrieval (¶24), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would have the orchestrator’s language model select, based on the input, the agent’s large language model as the tool for the delegated task and have that model choose the lower level tool that generates the output, with a reasonable expectation of success because Siebel’s agent models are already trained to determine instructions for a tool in the format the tool is configured to receive (¶228) and the selected tool’s output is already returned for determining the result (¶64).
A single reference rejection under 35 U.S.C. §103 is applied because Siebel’s agents may include other machine learning models or functions in addition to, or instead of, a large language model (¶70). Therefore, Siebel does not expressly disclose the claimed selection “using the second machine learned sequence processing model.” The modification only specifies which of Siebel’s own agent components makes Siebel’s own tool designation, and Siebel itself supplies the reason and the basis for expecting success, so no additional reference is needed for this difference. See MPEP §§2131, 2143, 2144(I).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 2
Regarding Claim 2, Siebel teaches The computer-implemented method of claim 1, wherein the plurality of first tools comprises a plurality of respective machine-learned agents, wherein each respective machine-learned agent of the plurality of respective machine-learned agents is a machine-learned sequence processing model configured to use one or more respective tools usable by the respective machine-learned agent (Siebel in ¶¶23, 30, 70-71, 228 discloses LLM equipped agent operations with respective tools, model based choices, and tool instruction generation).
Claim 11
Regarding Claim 11, Siebel teaches The method of claim 2, wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents is configured for performing multi-hop tasks using one or more decomposition tools configured to decompose an input to the at least one respective machine-learned agent (Siebel in ¶¶28, 34, 62, 70, 141, 148 discloses multi-hop planning, subquery decomposition and callable agents at further layers).
Claim 12
Regarding Claim 12, Siebel teaches The method of claim 1, wherein the at least one second tool comprises a third machine-learned sequence processing model (Siebel in ¶¶64, 70-71 discloses retrieval agent 506-1 selecting a further retriever agent 506-2 or 506-3, which may include its own LLM).
Claim 13
Regarding Claim 13, Siebel teaches The method of claim 12, wherein the third machine-learned sequence processing model is configured to use one or more third tools (Siebel in ¶¶64, 70-71, 149 discloses unstructured retriever agent 506-2 using its learned decision model and retrieval tool 508-1); and
generating the first output comprises:
selecting, by the one or more computing devices using the third machine-learned sequence processing model, at least one third tool of the one or more third tools (Siebel in ¶¶71, 149, 228 discloses model based agent choices, selection of the unstructured retrieval tool and learned tool instruction generation); and
generating, by the one or more computing devices using the at least one third tool of the one or more third tools, a second output (Siebel in ¶149 discloses retrieval tool 508-1 retrieving records used by the unstructured agent LLM on the selected answer generation branch).
Claim 14
Regarding Claim 14, Siebel teaches The method of claim 1, wherein at least one of the plurality of first tools and one or more second tools comprises a caption generator (Siebel in ¶¶26, 29, 34, 148 discloses image processing tool 234 generating a natural language summary of an image, with tools accessible directly or through agents).
Claim 19
Regarding Claim 19, Siebel teaches A computing system comprising one or more processors and one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising: obtaining a first machine-learned sequence processing model configured to use a plurality of first tools (Siebel in ¶¶23, 28-29 discloses an orchestrator using a large language model to process input and select from available agents and tools), wherein at least one first tool of the plurality of first tools is a second machine-learned sequence processing model configured to use one or more second tools (Siebel in ¶¶23, 64, 70-71 discloses callable agent modules that include LLMs and have their own tools. The proposed configuration exposes the learned decision model as the callable worker resource);
obtaining an input context (Siebel in ¶¶28-29, 148 discloses receiving a query and supplying its preprocessed context to the learned orchestrator);
selecting, using the first machine-learned sequence processing model based at least in part on the input context, a first tool of the plurality of first tools (Siebel in ¶¶28-29 discloses using the orchestrator model to interpret input and select the appropriate agent), wherein the first tool selected is the second machine-learned sequence processing model (Siebel in ¶¶23, 64, 70 discloses selection of an agent that may include an LLM. The proposed configuration makes its decision model the selected callable worker);
selecting, using the second machine-learned sequence processing model, at least one second tool of the one or more second tools (Siebel in ¶¶28-30, 71, 228 discloses independent agent tool designation, model determined choice sequences and learned tool instruction generation. The proposed configuration assigns the tool choice operation to the callable worker decision model);
generating, using the at least one second tool of the one or more second tools, a first output (Siebel in ¶¶30, 64, 149 discloses a selected tool retrieving records that are returned to the worker and used in producing the response).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the tool designation performed by the selected agent of Siebel with the large language model of that agent because Siebel teaches that the agent’s large language model provides reasoning functionality to accomplish the agent’s prescribed set of tasks (¶70), and that domain trained agent models provide accurate, context specific retrieval (¶24), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would have the orchestrator’s language model select, based on the input, the agent’s large language model as the tool for the delegated task and have that model choose the lower level tool that generates the output, with a reasonable expectation of success because Siebel’s agent models are already trained to determine instructions for a tool in the format the tool is configured to receive (¶228) and the selected tool’s output is already returned for determining the result (¶64).
A single reference rejection under 35 U.S.C. §103 is applied because Siebel’s agents may include other machine learning models or functions in addition to, or instead of, a large language model (¶70). Therefore, Siebel does not expressly disclose the claimed selection “using the second machine learned sequence processing model.” The modification only specifies which of Siebel’s own agent components makes Siebel’s own tool designation, and Siebel itself supplies the reason and the basis for expecting success, so no additional reference is needed for this difference. See MPEP §§2131, 2143, 2144(I).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 20
Regarding Claim 20, Siebel teaches One or more non-transitory computer-readable media storing instructions that are executable by a computing system to perform operations, the operations comprising: obtaining a first machine-learned sequence processing model configured to use a plurality of first tools (Siebel in ¶¶23, 28-29 discloses an orchestrator using a large language model to process input and select from available agents and tools), wherein at least one first tool of the plurality of first tools is a second machine-learned sequence processing model configured to use one or more second tools (Siebel in ¶¶23, 64, 70-71 discloses callable agent modules that include LLMs and have their own tools. The proposed configuration exposes the learned decision model as the callable worker resource);
obtaining an input context (Siebel in ¶¶28-29, 148 discloses receiving a query and supplying its preprocessed context to the learned orchestrator);
selecting, using the first machine-learned sequence processing model based at least in part on the input context, a first tool of the plurality of first tools (Siebel in ¶¶28-29 discloses using the orchestrator model to interpret input and select the appropriate agent), wherein the first tool selected is the second machine-learned sequence processing model (Siebel in ¶¶23, 64, 70 discloses selection of an agent that may include an LLM. The proposed configuration makes its decision model the selected callable worker);
selecting, using the second machine-learned sequence processing model, at least one second tool of the one or more second tools (Siebel in ¶¶28-30, 71, 228 discloses independent agent tool designation, model determined choice sequences and learned tool instruction generation. The proposed configuration assigns the tool choice operation to the callable worker decision model);
generating, using the at least one second tool of the one or more second tools, a first output (Siebel in ¶¶30, 64, 149 discloses a selected tool retrieving records that are returned to the worker and used in producing the response).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the tool designation performed by the selected agent of Siebel with the large language model of that agent because Siebel teaches that the agent’s large language model provides reasoning functionality to accomplish the agent’s prescribed set of tasks (¶70), and that domain trained agent models provide accurate, context specific retrieval (¶24), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would have the orchestrator’s language model select, based on the input, the agent’s large language model as the tool for the delegated task and have that model choose the lower level tool that generates the output, with a reasonable expectation of success because Siebel’s agent models are already trained to determine instructions for a tool in the format the tool is configured to receive (¶228) and the selected tool’s output is already returned for determining the result (¶64).
A single reference rejection under 35 U.S.C. §103 is applied because Siebel’s agents may include other machine learning models or functions in addition to, or instead of, a large language model (¶70). Therefore, Siebel does not expressly disclose the claimed selection “using the second machine learned sequence processing model.” The modification only specifies which of Siebel’s own agent components makes Siebel’s own tool designation, and Siebel itself supplies the reason and the basis for expecting success, so no additional reference is needed for this difference. See MPEP §§2131, 2143, 2144(I).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 3-4 and 9 is/are rejected under 35 U.S.C. 103 as obvious over Siebel et al (US 20240202225 A1, hereafter referred to as Siebel) in view of Yang et al (Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., ... & Wang, L. (2023). Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, hereafter referred to as Yang).
Claim 3
Regarding Claim 3, Siebel teaches The computer-implemented method of claim 2.
Siebel does not explicitly teach all of wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents of the plurality of respective machine-learned agents is configured for visual question answering based on text retrieved using a text retrieval tool usable by the at least one respective machine-learned agent.
However, Yang teaches wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents of the plurality of respective machine-learned agents is configured for visual question answering based on text retrieved using a text retrieval tool usable by the at least one respective machine-learned agent (Yang in §§3-4.1, Fig.3 discloses a prompted ChatGPT worker invoking vision experts and Bing search and answering a visual question from the returned text search information).
Yang is analogous art because Yang, like Siebel, is in the field of tool augmented language model reasoning and is reasonably pertinent to the problem of answering image questions that require external textual knowledge.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify one learned workers visual processing implementation of Siebel with the vision expert and Bing search question answering arrangement of Yang because Yang teaches that this arrangement provides visual understanding supplemented by external textual knowledge without additional multimodal training (§§3-4.1, Fig.3), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let the selected learned worker answer image related questions using retrieved text, with a reasonable expectation of success because Yang’s prompted LLM uses image references, callable experts and text observations compatible with Siebel’s agent/tool instructions and returned information interfaces.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 4
Regarding Claim 4, Siebel in view of Yang teaches The computer-implemented method of claim 3, wherein the at least one respective machine-learned agent is configured to:
determine, using one or more third tools configured to name one or more entities depicted in one or more images, a name of a first entity depicted in an input image (Yang in Fig. 3 discloses Celebrity Recognition returning the depicted names);
retrieve, using the text retrieval tool based on the name of the first entity, the text (Yang in Fig. 3 discloses Bing search using the recognized name and receiving text results); and
output, based at least in part on the text, an answer to an input question (Yang in Fig. 3 discloses answering that Kobe Bryant won 5 championship rings based on the retrieved information).
Claim 9
Regarding Claim 9, Siebel teaches The computer-implemented method of claim 2.
Siebel does not explicitly teach all of wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents is configured for reasoning based on optical character recognition.
However, Yang teaches wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents is configured for reasoning based on optical character recognition (Yang in §§3-4.1, 4.3 discloses an OCR expert returning scene text and prompted reasoning from those text observations).
Yang is analogous art because Yang, like Siebel, is in the field of tool augmented language model reasoning and is reasonably pertinent to the problem of reasoning from text depicted in images.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify one selected learned workers image-text processing of Siebel with the OCR expert and text observation reasoning process of Yang because Yang teaches that the expert to text interfaces provides visual reasoning from text usable by the language model (§§3-4.1, 4.3), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let the worker reason from recognized characters in an image, with a reasonable expectation of success because the OCR expert consumes an image reference and returns text compatible with the existing LLM input and tool result interface.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 5-8 and 15-18 is/are rejected under 35 U.S.C. 103 as obvious over Siebel et al (US 20240202225 A1, hereafter referred to as Siebel) in view of Gupta et al (Gupta, T., & Kembhavi, A. (2023, June). Visual programming: Compositional visual reasoning without training. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 14953-14962). IEEE, hereafter referred to as Gupta).
Claim 5
Regarding Claim 5, Siebel teaches The computer-implemented method of claim 2.
Siebel does not explicitly teach all of wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents is configured for counting a number of objects in an image.
However, Gupta teaches (Gupta in §4.1 discloses a generated visual program locating objects and counting their bounding boxes).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of computing a visual count through localized object detections.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify one selected learned workers visual task program of Siebel with the localization and counting modules of Gupta because Gupta teaches that the visual program provides computational answers through reusable localization and arithmetic operations (§4.1), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let that learned worker count objects depicted in an image, with a reasonable expectation of success because the LLM generated module instructions and returned box/count values fits Siebel’s callable worker/tool interfaces.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 6
Regarding Claim 6, Siebel teaches The computer-implemented method of claim 2.
Siebel does not explicitly teach all of wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents is configured for answering a question about a particular portion of an image indicated by the input context.
However, Gupta teaches (Gupta in §4.1 discloses localization and cropping of the region indicated by a spatially specified question, followed by VQA on that patch).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of answering a question about an indicated image region.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify one selected learned workers image question processing of Siebel with the context directed crop and patch question answering program of Gupta because Gupta teaches that the regional decomposition provides simpler localized questions and more interpretable compositional answers (§4.1), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let the worker answer about the image portion identified by the input question, with a reasonable expectation of success because the question, detected coordinates, crop, and VQA answer are defined inputs and outputs of the callable modules.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 7
Regarding Claim 7, Siebel teaches The computer-implemented method of claim 2.
Siebel does not explicitly teach all of wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents is configured for multi-image question answering.
However, Gupta teaches (Gupta in §4.2 discloses multi agent reasoning by decomposing an image pair question into image level questions and combining the answers).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of combining evidence from multiple images.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify one selected learned workers visual reasoning of Siebel with the multi-image decomposition and answer composition program of Gupta because Gupta teaches that the composition of single image modules provides multi image reasoning without a dedicated multi image model or expensive multi image training data (§4.2), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let the worker answer a question from multiple image level results, with a reasonable expectation of success because Gupta already passes the image level answers to logic/arithmetic modules through the same generated program interface.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 8
Regarding Claim 8, Siebel teaches The computer-implemented method of claim 2.
Siebel does not explicitly teach all of wherein at least one respective machine-learned agent of the plurality of respective machine-learned agents is configured for spatial reasoning.
However, Gupta teaches (Gupta in §4.1 discloses localization, left/right region selection and expression evaluation for spatial relataionship).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of resolving spatial relationships in an image question.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify one selected learned workers spatial task processing of Siebel with the localization, spatial crop and expression evaluation program of Gupta because Gupta teaches that the localized program provides compositional visual answers through simpler spatial computations (§4.1), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let the worker reason about relative locations of depicted objects, with a reasonable expectation of success because the modules expose coordinates, selected regions and answer values through callable inputs and outputs.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 15
Regarding Claim 15, Siebel teaches The method of claim 1.
Siebel does not explicitly teach all of wherein at least one of the plurality of first tools and one or more second tools comprises an image cropping tool.
However, Gupta teaches wherein at least one of the plurality of first tools and one or more second tools comprises an image cropping tool (Gupta in §§3, 4.1 discloses callable image cropping routines driven by localized boxes or spatial regions).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of supplying the appropriate image region to a selected computation.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the dispatcher and selected workers available toolsets of Siebel with the callable image cropping module of Gupta because Gupta teaches that regional cropping provides simpler localized inputs for compositional visual question answering (§4.1), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would make the crop resource available in both tool tiers, with a reasonable expectation of success because the crop module accepts an image and coordinates and returns an image, which Siebel’s direct and agent coordinated tool interfaces can carry.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 16
Regarding Claim 16, Siebel teaches The method of claim 1, further comprising:
generating, by the one or more computing devices using the second machine-learned sequence processing model, one or more instructions for the at least one second tool (Siebel in ¶228 discloses agent machine learned models generating recipient compatible tool instructions);.
Siebel does not explicitly teach all of wherein the one or more instructions comprise at least one variable name; and the first output is generated based at least in part on data associated with the at least one variable name.
However, Gupta teaches wherein the one or more instructions comprise at least one variable name (Gupta in §3, Code 1 discloses LLM generated module steps containing input and output variable names); and
the first output is generated based at least in part on data associated with the at least one variable name (Gupta in Code 1 discloses looking up state[var_name] and computing the module output from the associated value).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of passing computed values by name in generated tool instructions.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the selected learned workers generated tool instruction format of Siebel with the variable named input argument and state lookup syntax of Gupta because Gupta teaches that the input variable lookup provides use of intermediate types values in subsequent module computations (§3, Code 1), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let a second model instruction contain a variable name and let the selected second tool compute its output from the bound value, with a reasonable expectation of success because Siebel trains agent models to generate recipient compatible instructions, while Gupta supplies the parse/lookup/computation procedure for those named arguments.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 17
Regarding Claim 17, Siebel teaches The method of claim 1, further comprising:
storing, on one or more non-transitory computer-readable media in one or more locations associated with a variable name, the first output (Siebel in ¶¶203-205 discloses memory supporting the proposed controller);
generating, by the one or more computing devices using the second machine-learned sequence processing model, one or more instructions configured to use an additional tool of the one or more second tools (Siebel in ¶¶71, 228 discloses the learned worker determining a sequence of choices and generating instructions for different tools).
Siebel does not explicitly teach all of further comprising: storing, on one or more non-transitory computer-readable media in one or more locations associated with a variable name, the first output; …; and generating, by the one or more computing devices using the additional tool, a second output; wherein the one or more instructions comprise the variable name; and the second output is generated based at least in part on data associated with the variable name.
However, Gupta teaches further comprising:
storing, on one or more non-transitory computer-readable media in one or more locations associated with a variable name, the first output (Gupta in §3, Code 1 discloses binding a computed output with state[output_var_name] = output);
…; and
generating, by the one or more computing devices using the additional tool, a second output (Gupta in Code 1 discloses executing the next named module and computing its output from the resolved input);
wherein the one or more instructions comprise the variable name (Gupta in §3, Code 1 discloses generated module instructions that identify input variables); and
the second output is generated based at least in part on data associated with the variable name (Gupta in Code 1 discloses resolving the input variable to its stored value before computing the next module output).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of reliably carrying intermediate data into later tool calls.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the controller with the selected learned worker of Siebel with the variable keyed intermediate output storage and value resolution mechanism of Gupta because Gupta teaches that the name/value state dictionary provides reuse of arbitrary intermediate outputs as inputs to later modules (§3, Code 1), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would store the first tool output under a name and supplying its value to an additional tool called by the workers instructions, with a reasonable expectation of success because the inherited worker generates recipient tool instructions and decision sequences, Gupta resolves variable inputs before computation, and Siebel supplies program/working memory.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 18
Regarding Claim 18, Siebel teaches The method of claim 1, further comprising:
storing, on one or more non-transitory computer-readable media in a location associated with a variable name, the input context (Siebel in ¶¶28-29, 203-205 discloses the received original input and program/working memory used by the proposed name context controller state); and
generating, by the one or more computing devices using the first machine-learned sequence processing model, one or more instructions for the second machine-learned sequence processing model (Siebel in ¶¶28-29, 227 discloses the first model generating recipient compatible instructions for the selected agent);
wherein the one or more instructions comprise the variable name (Siebel in ¶¶28-29, 227 discloses first model input dependent agent instructions, adapted to use the designated original context argument name); and
the at least one second tool of the one or more second tools is selected based at least in part on data associated with the variable name (Siebel in ¶¶28-30, 71, 228 discloses input dependent assigned agent choices and model choice sequences)..
Siebel does not explicitly teach all of further comprising: storing, on one or more non-transitory computer-readable media in a location associated with a variable name, the input context; and …; wherein the one or more instructions comprise the variable name; and the at least one second tool of the one or more second tools is selected based at least in part on data associated with the variable name.
However, Gupta teaches further comprising:
storing, on one or more non-transitory computer-readable media in a location associated with a variable name, the input context (Gupta in §3, Code 1 discloses initializing program state with input values and resolving variable names); and
…;
wherein the one or more instructions comprise the variable name (Gupta in §3, Code 1 discloses generated instructions containing input variable names); and
the at least one second tool of the one or more second tools is selected based at least in part on data associated with the variable name (Gupta in §3, Code 1 discloses initial input name/value state and lookup of state[var_name] before the receiving computation).
Gupta is analogous art because Gupta, like Siebel, is in the field of language model directed task processing and is reasonably pertinent to the problem of passing the original input context by name without losing its role in worker tool selection.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the first model to worker argument binding of Siebel with the initialized variable keyed input state and name to value resolution of Gupta because Gupta teaches that the initialized state and variable lookup provides named access to initial input values and reuse of arbitrary typed values in generated instructions (§3, Code 1), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel would let the first model pass the original stored context by name and let the worker select its tool from the resolved context, with a reasonable expectation of success because Siebel already generates input dependent agent instructions and learned worker choices, while Gupta resolves an argument name to its value before the receiving computation.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as obvious over Siebel et al (US 20240202225 A1, hereafter referred to as Siebel) and Yang et al (Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., ... & Wang, L. (2023). Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, hereafter referred to as Yang), further in view of Borisyuk et al (Borisyuk, F., Gordo, A., & Sivakumar, V. (2018, July). Rosetta: Large scale system for text detection and recognition in images. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 71-79), hereafter referred to as Borisyuk).
Claim 10
Regarding Claim 10, Siebel in view of Yang teaches The computer-implemented method of claim 9.
Siebel in view of Yang does not explicitly teach all of wherein the at least one respective machine-learned agent is configured to: identify, using one or more third tools based on an input image, one or more regions of the input image that comprise one or more natural language characters; and read, using one or more fourth tools configured to perform optical character recognition, the one or more natural language characters.
However, Borisyuk teaches wherein the at least one respective machine-learned agent is configured to:
identify, using one or more third tools based on an input image, one or more regions of the input image that comprise one or more natural language characters (Borisyuk in §§3-3.1, Fig. 2 discloses a trained detector identifying rectangular text regions of the input image); and
read, using one or more fourth tools configured to perform optical character recognition, the one or more natural language characters (Borisyuk in §§3-3.1, Fig. 2 discloses a separate CNN/CTC recognizer reading and transcribing the characters in those detected regions).
Borisyuk is analogous art because Borisyuk, like Siebel in view of Yang, is in the field of machine extraction of text from images and is reasonably pertinent to the problem of localizing text and recognizing its characters for subsequent agent reasoning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the OCR implementation available to the learned worker of Siebel in view of Yang with the separate text region detector and text recognizer of Borisyuk because Borisyuk teaches that the two stage architecture provides independently updatable stages, parallel word recognition and support for different recognition languages (§3), and one of ordinary skill would have recognized that incorporating this feature into the method of Siebel in view of Yang would provide distinct tools for locating text and reading its characters before OCR based reasoning, with a reasonable expectation of success because the detector’s image regions are inputs to the recognizer whose text outputs preserves the image-in/text-out boundary of Yang’s expert.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUSTIN P CASCAIS whose telephone number is (703) 756-5576. The examiner can normally be reached Monday-Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mr. O'Neal Mistry can be reached on (313) 446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.P.C./Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Date: 10/7/2026
