DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In Claim 1, Lines 7-8, "a threshold complexity value" was previously referenced in line 6 and does not appear to find antecedent basis in the preceding term. Thus, it is unclear if this further instance should find antecedent basis in the earlier term or if a new instance of the threshold is being introduced. For claim interpretation in the interest of compact prosecution, "a threshold complexity value" in lines 7-8 will be construed as --the threshold complexity value--. Claims 18 and 20 contain similar antecedent basis issues, and thus, have been rejected under similar rationale.
In Claim 1, Line 15, "the model input" lacks antecedent basis and it is unclear if the model input finds reference in the term "input" found in line 14 or represents a new term that should not be introduced with a definite article. For claim interpretation in the interest of compact prosecution, "input" in line 14 will be construed as --model input-- similar to the language found in alternate embodiment claims 18 and 20.
In Claim 5, Lines 1-2 appears to reference a summarizing step not present in any of parent claims 1, 3, or 4 (i.e., "prior to summarizing, by the computing system, intermediate data into a model input." Thus, this limitation lacks antecedent basis and it is unclear what method step of any parent claims is being referenced. For claim interpretation, this limitation will be construed as --prior to generating, by the computing system, model input based on the intermediate data--.
Claims 8-9, Line 1 refer to "a respective processing step" when parent claim 7 already features this limitation in line 1. Thus, it is unclear whether these respective steps of claims 8-9 are references the respective step in claim 7 or represent some different step. For claim interpretation, "a respective processing step" in claim 8 will be construed as --a second respective processing step-- and in claim 9 will be construed as --a third respective processing step--.
The remaining dependent claims inherit the indefinite subject matter of their respective parent claims, and thus, are also rejected under 35 U.S.C. 112(b) for being indefinite by virtue of their dependency.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed towards a judicial exception in the form of an abstract idea without significantly more.
Independent Claims 1, 18, and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims regard a process that, as drafted under its broadest reasonable interpretation, covers performance of the limitations as a mental process, but for the recitation of generic computer components.
In regards to the process of these independent claims, the claimed functionality could be practiced as a mental process in the following manner:
receiving, (a human can receive a multimodal query through a collection of human analysis- reading text, mentally processing speech after listening, and vision);
determining(a human can mentally assign a complexity level (e.g., from 1-5) and can mentally decide that the query is too difficult to answer all at once in relation to a level threshold (e.g., 3 or above) by referring to a table-based model on paper or remembered);
in response to determining that the multimodal input query exceeds a threshold complexity value: generating, by the computing system, a plurality of processing steps for responding to the multimodal input query, wherein the processing steps include executing at least one subquery based on the multimodal input query (a human can mentally evaluate a query and break it into a series of steps or tasks to be carried out in a mental planning process);
performing, (a human can follow the series of tasks remembered or written on paper and carry out these tasks to yield an intermediate data result by using pen and paper, e.g., looking up information by checking documentation, a calendar, pricing, etc. and gathering the data); and
generating, (gathering data can result in leads or a need for further information where a human can mentally assess the intermediate data and render a judgement as to further search input/efforts);
processing,
transmitting, (a human can transmit the answer for display using pen and paper).
This judicial exception is not integrated into a practical application. Outside of the identified abstract idea, the claimed invention only includes computer systems/processors/non-transitory computer-readable media which amount to no more than mere instructions to implement an otherwise abstract idea using generic computer components. Moreover, the intended output on a display of a user computing device is extra-solutional activity again relying upon generic computer components. Note that the computer is not being improved as a tool in this process, it is only be used for its standard purpose of executing a program to carry out a method/functionality.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The above identified additional generic computer components are no more than mere instructions to apply the exception using generic computer components that are well-known, routine, and conventional as is evidenced by Bancorp Services v. Sun Life (Fed. Cir. 2012) and Alice Corp. v. CLS Bank (2014).
Accordingly, at least independent Claims 1, 18, and 20 are not patent eligible under 35 U.S.C. 101.
The remaining dependent claims fail to add patent eligible subject matter to their respective parent claims:
Claims 2 and 19 narrow the format types of the input query as been text or speech, both of which are capable of being practically understood by a human.
Claims 3-4 narrow the steps as a list and relating towards computing instructions, both of which are capable of being understood by a human via reading and the mental understanding of language and computer operations.
Claim 5 relates to a mental process under the BRI in that a human can decide that not enough information has been obtained to render an answer, mentally decide on additional steps, carry out those steps, and continuing the process until enough information has been gathered to provide an answer.
Claim 6 narrows steps with processes that can be performed by a human by examining visual information, data gathering, answering questions, and combining data mentally or on paper.
Claim 7 attempts to add a visual machine learned model to the method, however, use of the model is passive (i.e., only providing and receiving is claimed) where a human can manually enter an image by placing paper on a scanner or a type a text reference to an image and read the list of detected objects.
Claim 8 relates to a human mentally deciding upon a simple question based upon the image objects, manually typing in a search, and reading the results.
Claim 9 relates to a human manually combining the search results mentally or using a pen and paper to derive a new question.
Claim 10 regards a human manually typing in a query, reading a complexity score, and mentally comparing the score to a threshold number to render a mental judgement.
Claim 11 adds details to the machine-learned model that is passive under the BRI in a step that can be practically be performed by a human.
Claim 12 adds details to the query classification model as being a large vision language model. The presence of this model does not preclude human operation of a step that is recited at a high level of abstractness and merely automates the process. Moreover, these types of models are well-known as evidenced by Liébana de la Barrera, et al. (U.S. PG Publication: 2026/0017372 A1- LVMs are currently available to the public, Paragraph 0024) and Chen, et al. (U.S. PG Publication: 2025/0131027 A1- LVLMs are "conventional", Paragraphs 0058 and 0062).
Claim 13 narrows a type of output that a human can supply in a natural language by, e.g., writing on paper with pen.
Claim 14 relates to an input type that can be mentally understood by a human in the form of a citation.
Claim 15 narrows model output to a citation that can be provided by a human using pen and paper.
Claim 16 relates to a human providing a page of results including the pertinent answer.
Claim 17 relates to multimodal results that can be provided by a human on paper containing text and/or images and speaking.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 10-14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mayande, et al. (U.S. PG Publication: 2026/0024011 A1) in view of Lee, et al. (U.S. PG Publication: 2025/0200358 A1).
With respect to Claim 1, Mayande discloses:
A computer-implemented method for responding to a multimodal input query with a large-language model using a multi-step reasoning process, the method comprising:
receiving, by a computing system with one or more processors (computer system having one or more processors, Paragraph 0079; note that this citation also applies to repetition of the computing system through the recited process steps), the multimodal input query, the multimodal input query including image content (acceptance of a user input query that "may be of...multiple modalities," Paragraphs 0020 and 0073 along with Fig. 7, Element 702; query modalities include “images”, Paragraphs 0016 and 0043);
determining, by the computing system, that the multimodal input query exceeds a threshold complexity ("user query is first evaluated...to be either a simple query or a complex query" where "determination of whether the query is simple or complex may be carried out with any appropriate technique, such as using a suitably configured ML system" and "AI model 118 may include a module or component that is specifically trained or configured to evaluate queries for complexity," Paragraphs 0045-0046; note that the determination is based upon a sufficient classification to avoid a "simple query" classification and thus meets a classification threshold along with the consideration of "parameters" to qualify as complex);
in response to determining that the multimodal input query exceeds a threshold complexity the multimodal input query (for determination of a sufficiently complex user query, a "clear and thorough plan-of-action" is generated, Paragraphs 0056, 0058-0060, and 0062; the plan of action may include subqueries, for example, by "break down" of a user input query into partial queries or "simple queries", Paragraphs 0020 and 0046-0047);
performing, by the computing system, the plurality of processing steps to generate intermediate data (the plan-of-action/execution order is carried out to generate intermediate data that is ultimately used to construct a reply to the user query, Paragraphs 0064-0065, 0067, 0070-0071, and 0077-0078); and
generating, by the computing system, a model input (see preceding claim construction in the 35 U.S.C. 112(b) rejection) based on the intermediate data (see that intermediate data/outputs may be utilized as inputs/prompts to another query handling model or used as context, Paragraphs 0065-0068);
processing, by the computing system, the model input with a query response model to generate a model output based on the model input (the plan-of-action/execution order is carried out to generate model outputs to construct a “coherent reply” to the user query, Paragraphs 0064-0065, 0067, 0070-0071, and 0077-0078); and
transmitting, by the computing system, the model output for display at a user computing device (user device receives a communicated/transmitted response, Paragraphs 0042 and 0087; response modalities in the form of text/images using a display, Paragraphs 0016, 0048, and 0071; Fig. 6, Element 614).
While Mayande teaches that "determination of whether the query is simple or complex may be carried out with any appropriate technique,” Mayande does not specifically teach the comparison of a query to a complexity threshold value. Lee, however, recites an orchestrator estimating a complexity metric for an input query and comparing such metric value to a "defined threshold" to identify whether a input query is complex (Paragraph 0029 and 0049).
Mayande and Lee are analogous art because they are from a similar field of endeavor in handing queries of generative AI models. Thus, it would have been obvious to one of ordinary skill in the art to utilize the threshold-based complexity assessment taught by Lee in the complex/simple query classification decision taught by Mayande to provide a predictable result of an algorithm that can be used to separate query types into categories based upon an assessment and that qualifies as “any appropriate technique” in yielding a final decision in the teachings of Mayande.
With respect to Claim 2, Mayande further discloses:
The computer-implemented method of claim 1, wherein the multimodal input query includes textual content or speech content ("multi-mode input" comprising “text” or "sound", Paragraph 0043; queries including a user "asking" for a request, Paragraphs 0044-0048; thus, since a modality used in multi-mode can include sound and can include a query or a user asking about something, the teachings imply that the sound may include speech; note that Lee explicitly calls out “speech” in an input query in paragraph 0031).
With respect to Claim 3, Mayande further discloses:
The computer-implemented method of claim 1, wherein the plurality of processing steps are represented as a list of processing steps ("plan-of-action" as a series of ordered steps to be executed, Paragraphs 0058-0059, 0064, and 0076).
With respect to Claim 4, Mayande further discloses:
The computer-implemented method of claim 3, wherein each processing step is represented as a computing instruction (plan-of-action includes functional instructions and readable datasets for using computer models, Paragraphs 0058, 0062, and 0066).
With respect to Claim 10, Mayande and Lee further discloses:
The computer-implemented method of claim 1, wherein determining, by the computing system, that the multimodal input query exceeds a threshold complexity value for query complexity using a query classification model further comprises: providing, by the computing system, the multimodal input query as input to the query classification model (Mayande- teaches submitting a query to an AI model that carries out "evaluations" of complexity with "any appropriate technique," Paragraph 0046);
receiving, by the computing system, a complexity score for the multimodal input query as output from the query classification model (Mayande- AI model result decides complexity using the algorithm, Paragraph 0046 while Lee teaches an evaluation of a complexity as a complexity metric value, Paragraph 0029); and
comparing, by the computing system, the complexity score for the multimodal input query to the threshold complexity value (Lee- assessment of the complexity metric against a "defined threshold", Paragraph 0029).
With respect to Claim 11, Mayande and Lee further discloses:
The computer-implemented method of claim 10, wherein the query classification model is a machine-learned model trained to take a multimodal input query as input and output a complexity score (Mayande- “AI model” or “ML” system for producing an evaluation of complexity via “any algorithm,” Paragraph 0046; Lee discloses an algorithm in the form of the complexity score/metric, Paragraph 0029).
With respect to Claim 12, Mayande further discloses:
The computer-implemented method of claim 1, wherein the query classification model is a large vision language model (use of an "AI model" or "ML system" to evaluate complexity or ML system, Paragraph 0046; note that Mayande discloses that such models/systems include generative LLMs capable of handling input images, Paragraphs 0016-0020 and 0043, accordingly such AI models that may be used in the complexity assessment map to the claimed large vision language model used for the assessment).
With respect to Claim 13, Mayande further discloses:
The computer-implemented method of claim 1, wherein the model output comprises a natural language response to the multimodal input query (providing coherent natural language/text-based modality responses in the form of an answer or description, Paragraphs 0004, 0016, 0048, 0071, and 0078).
With respect to Claim 14, Mayande further discloses:
The computer-implemented method of claim 1, wherein the model input includes citation data (input to the model incorporating a citation of a "specific data source" to process a model input for the query, Paragraphs 0049-0051, 0062, 0064, and 0076).
With respect to Claim 16, Mayande further discloses:
The computer-implemented method of claim 1, wherein the model output is displayed on a page of search results (query may include a search for a particular result, answer, or data analysis presented on a display screen/page to the user that constitutes the page of search results, Paragraph 0038, 0045, 0048, 0071, and 0080).
With respect to Claim 17, Mayande further discloses:
The computer-implemented method of claim 16, wherein the search results are multimodal (an answer/reply may include "modalities" such as text accompanied by graphics, Paragraphs 0034, 0048, and 0071).
Claim 18 is directed towards an alternative system embodiment comprising one or more processors and one or more non-transitory computer-readable media that store instructions for carrying out the process of claim 1, and thus, is rejected under similar rationale. Moreover, Mayande teaches method implementation as a system comprising one or more processors and program instructions stored on a non-transitory computer-readable medium (Paragraphs 0081 and 0085-0086).
With respect to Claim 19, Mayande further discloses:
The computing system of claim 18, wherein the multimodal input query includes textual content or speech content ("multi-mode input" comprising “text” or "sound", Paragraph 0043; queries including a user "asking" for a request, Paragraphs 0044-0048; thus, since a modality used in multi-mode can include sound and can include a query or a user asking about something, the teachings imply that the sound may include speech; note that Lee explicitly calls out “speech” in an input query in paragraph 0031).
Claim 20 is directed towards an alternative embodiment comprising one or more non-transitory computer-readable media that store computer-executable instructions for carrying out the process of claim 1, and thus, is rejected under similar rationale. Moreover, Mayande teaches method implementation as a program instruction stored on a non-transitory computer-readable medium (Paragraphs 0081 and 0085-0086).
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Mayande, et al. in view of Lee, et al. and further in view of Siebel, et al. (U.S. PG Publication: 2024/0370709 A1).
With respect to Claim 5, Mayande in view of Lee teaches the method for evaluating query complexity followed by developing and carrying out a plan-of-action for response generation involving the generation of intermediate data as applied to Claim 1. Mayande in view of Lee does not teach the process for determining whether the intermediate data is sufficient to respond to the multimodal input query and if not having an orchestration model determine additional processing steps until the intermediate data is determined to be sufficient to respond to the user query as set forth in claim 5. Siebel, however, discloses:
determining, by the computing system, whether the intermediate data is sufficient to respond to the multimodal input query (when executing a "plan for answering the query," the intermediate result is observed and it is determined whether enough information is obtained to submit a final answer or if more information is needed, Paragraph 0194); and
responsive to determining that the intermediate data is not sufficient to respond to the multimodal query: generating, by an orchestration model, additional processing steps for responding to the multimodal input query (if insufficient/not enough information, an orchestrator decides upon "another prescribed set of tasks," Paragraph 0194);
performing, by the computing system, the additional processing steps to generate updated intermediate data (additional steps tasks are carried out the generate an answer, Paragraph 0194); and
continuing to generate and perform additional processing steps until the updated intermediate data is determined to be sufficient to respond to the multimodal input query ("the process can continue until the orchestrator...has enough information to answer," Paragraph 0194).
Mayande, Lee, and Seibel are analogous art because they are from a similar field of endeavor in machine learning model response generation. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to add the intermediate response observation taught by Seibel to the response generation process taught by Mayande in view of Lee to provide a predictable result in the form of increasing machine learning/AI model processing efficiency and decreasing latency by allowing answer generation once a sufficient amount of information has been obtained.
With respect to Claim 6, Mayande further discloses:
The computer-implemented method of claim 5, wherein the one or more processing steps comprise one or more of: object recognition, data retrieval, subquery execution, and data synthesization (processing steps to be carried out include data retrieval from a data source (Paragraphs 0049-0050, 0054, and 0064), execution of subqueries/portions of queries (Paragraphs 0046, 0065-0066, and 0077), and data synthesizing/answer generation (Paragraphs 0065 and 0077-0078)).
Claims 7-9 are rejected under 35 U.S.C. 103 as being unpatentable over Mayande, et al. in view of Lee, et al. in view of Siebel, et al. and further in view of Elbadrashiny, et al. (U.S. Patent: 12,307,299).
With respect to Claim 7, Mayande in view of Lee and further in view of Siebel discloses the method for evaluating query complexity followed by developing and carrying out a plan-of-action comprising various steps for response generation involving the generation of intermediate data that is observed for final answer generation as applied to Claim 6. Mayande in view of Lee and further in view of Siebel does not teach that a respective step includes objection rejection as set forth in claim 7. Elbadrashiny, however, discloses
wherein a respective processing step includes object recognition and the method further comprises (determining a sequence of tasks to be performed to generate a response to a user query using AI models, Col. 2, Lines 34-51; tasks include objection recognition within an image, Col. 11, Line 66- Col. 12, Line 9; Col. 19, Lines 17-30):
providing, by the computing system, the image content to an object recognition model, wherein the object recognition model is a visual machine-learned model trained to take an image as input (AI/machine learning model that is configured to process visual images to identify objects, Col. 11, Line 50- Col. 12, Line 9; Col. 19, Lines 17-30); and
receiving, by the computing system, a list of detected objects as output from the object recognition model (listing of objects received from the object recognition AI model that may be used for an additional task such as the generation of a description of such objects, Col. 11, Line 66- Col. 12, Line 9; Col. 19, Lines 17-30).
Mayande, Lee, Seibel, and Elbadrashiny are analogous art because they are from a similar field of endeavor in machine learning model response generation. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to add the computer vision-based object recognition taught by Elbadrashiny to process the image modality queries taught by Mayande in view of Lee in view of Seibel in order to provide a predictable result of providing a model capable of processing the image inputs of Mayande (see Mayande, Paragraph 0016) to identify portions that pertain to an input query for response generation.
With respect to Claim 8, Mayande and further discloses:
The computer-implemented method of claim 7, wherein a respective processing step includes subquery execution (Paragraphs 0046, 0065-0066, and 0077), and the method further comprises: generating, by the computing system, at least one subquery based on the multimodal input query and at least one object in the list of detected objects (Elbadrashiny teaches a sequence of tasks performed by AI models where one of the models as per the claim 7 rejection includes the generation of a list of detected objections; Mayande, then, teaches that the generation of additional subqueries using the prior output as a prompt and/or context, Paragraphs Paragraphs 0046, 0064-0068);
providing, by the computing system, the at least one subquery to a search system (providing the sub/simple query to a search system and retrieving a response/model output, Paragraphs 0064-0068 and 0077); and
receiving, by the computing system, one or more search results to the at least one subquery from the search system (receiving the subquery response/output for generating an answer to a query, Paragraphs 0064-0068 and 0077).
With respect to Claim 9, Mayande further discloses:
The computer-implemented method of claim 8, wherein a respective processing step includes a synthesization (data synthesizing/answer generation, Paragraphs 0065 and 0077-0078) and the method further comprises: aggregating, by the computing system, the one or more search results to generate combined search data (contents of an output buffer including a plurality of search results can be aggregated based upon dependency and/or context to generate combined search data, Paragraphs 0065-0066); and
generating, by the computing system, a respective subquery using data from the combined search result data (combined search data is input as a sub prompt/query in the process that ultimately achieves a response to an input multimodal query, Paragraphs 0065-0066).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Mayande, et al. in view of Lee, et al. and further in view of Kirk (U.S. PG Publication: 2025/0124024 A1).
With respect to Claim 15, Mayande in view of Lee teaches the method for evaluating query complexity followed by developing and carrying out a plan-of-action for response generation that includes the identification of data sources in query processing as applied to Claim 1 (see also the rejection of claim 14). Mayande in view of Lee does not specifically teach that the model output comprises citation data for data generated by the response generation model. Kirk, however, discloses an AI model in the form of an LLM generates a response along with a source citation for the information that is transmitted to a user interface for display (Abstract; Paragraphs 0036, 0038, and 0040).
Mayande, Lee, and Kirk are analogous art because they are from a similar field of endeavor in machine learning model response generation. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date to add the source identification output for display taught by Kirk to the answer generation/output taught by Mayande in view of Lee to provide a predictable result in the form of increasing trustworthiness of the AI model (Kirk, Paragraph 0036).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Torok, et al. (U.S. PG Publication: 2026/0004786 A1)- teaches complex task processing by breaking down a question into smaller parts (Paragraphs 0063-0071).
Jeong, et al. ("Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity," June 2024)- teaches a query complexity classifier language model to develop a multi-step approach for question answering by an LLM (Sections 3.1-3.2, Pages 7039-7040).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES S WOZNIAK whose telephone number is (571)272-7632. The examiner can normally be reached 7-3, off alternate Fridays.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant may use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JAMES S. WOZNIAK
Primary Examiner
Art Unit 2655
/JAMES S WOZNIAK/Primary Examiner, Art Unit 2655
Prosecution Insights