DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In response to Applicant’s claims filed on October 22, 2025, claims 1-30 are now pending for examination in the application.
Priority
Applicant’s claim for benefit of prior-filed provisional application 63/506,924 dated 06/08/2023 under 35. U.S.C. 119(e) or under 35 U.S.C. 120, 121, or 365(c) is acknowledged.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Claims 21, 25 contain limitations invoking 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph as detailed in the following:
Claim 21:
"a language model planner block..."
"a language model reasoner block..."
"a working memory..."
Claim 25:
"a language model reasoner block..."
has been interpreted under 35 U.S.C. 112 (f), or pre-AIA 35 U.S.C. 112 sixth paragraph, because it uses generic placeholder(s) “block” and “memory” coupled with functional languages without reciting sufficient structure to achieve the function and equivalents thereof. Furthermore, the generic placeholder is not preceded by a structural modifier.
Since the claim limitation(s) invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, claims 21 and 25 have been interpreted to cover the corresponding structure described in the specification that achieves the claimed function, and equivalents thereof.
A review of the specification shows that the following appears to be the corresponding structure described in the specification for the 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph limitation: NONE. The specification fails to show the corresponding structures of the components.
If applicant wishes to provide further explanation or dispute the examiner’s interpretation of the corresponding structure, applicant must identify the corresponding structure with reference to the specification by page and line number, and to the drawing, if any, by reference characters in response to this Office action.
If applicant does not intend to have the claim limitation(s) treated under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112 , sixth paragraph, applicant may amend the claim(s) so that it/they will clearly not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, or present a sufficient showing that the claim recites/recite sufficient structure, material, or acts for performing the claimed function to preclude application of 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
For more information, see MPEP § 2173 et seq. and Supplementary Examination Guidelines for Determining Compliance With 35 U.S.C. 112 and for Treatment of Related Issues in Patent Applications, 76 FR 7162, 7167 (Feb. 9, 2011).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 21-30 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 21 and 25 invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. Claims 11-14 and 18-19 are interpreted under 35 U.S.C. 112(f) (see above). Therefore Claim(s) 11-14 and 18-19 contain placeholders that require corresponding structure(s). It is unclear whether the recited structure, material, or acts in these claims are sufficient for performing the claimed function because the Specification is unclear about the corresponding structure(s) and the logic or algorithms necessary for performing the claimed functions.
Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Dependent claims 22-24, 26-30 are also rejected for inheriting the deficiencies of the independent claims from which they depend on.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-30 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claims are directed to an abstract idea without significantly more.
Claim 1-30 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than judicial exception. The eligibility analysis in support of these findings is provided below, on Claim Rejections - 35 USC 101 accordance with the "2019 Revised Patent Subject Matter Eligibility Guidance" (published on 1/7/2019 in Fed, Register, Vol. 84, No. 4 at pgs. 50-57, hereinafter referred to as the "2019 PEG").
Step 1. in accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), it is first noted the claim system (claims 1-10, 21-30) and method (claims 11-20) are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
Step 2A. In accordance with Step 2A, prong one of the 2019 PEG, it is noted that the independent claims recite an abstract idea falling within the Mental Processes enumerated groupings of abstract ideas set forth in the 2019 PEG. Examiner is of the position that independent claims 1, 8, and 15 are directed towards the Mental Process Grouping of Abstract Ideas.
Independent claims 1 recites the following limitations directed towards a Mental Processes:
processing the input data with a machine-learned model to generate first planning data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate planning data),
processing the input data and the first output data with the machine-learned model to generate second planning data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate planning data),
processing the input data and the second output data with the machine-learned model to generate response data, wherein the response data is descriptive of a response to the query (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate response data);
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 1:
one or more processors (i.e., as a generic component performing a generic computer function); and
one or more non-transitory computer-readable media (i.e., as a generic component performing a generic computer function) that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (recites insignificant extra solution activity that amounts to obtaining data);
transmitting, based on the first planning data, the input data to the first data processing tool to retrieve first output data (recites insignificant extra solution activity that amounts to transmitting data);
transmitting, based on the second planning data, data to the second data processing tool to retrieve second output data (recites insignificant extra solution activity that amounts to transmitting data).
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Independent claims 11 recites the following limitations directed towards a Mental Processes:
processing, by the computing system, the input data with a machine-learned model to generate first planning data, wherein the first planning data is descriptive of instructions
processing, by the computing system, the first output data with the machine-learned model to generate second planning data, wherein the second planning data is descriptive of instructions to provide a particular portion of the image data to a second data processing tool (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate second planning data);
providing, by the computing system, the particular portion of the image data to the second data processing tool to
processing, by the computing system, the text data and the second output data with the machine-learned model to generate response data, wherein the response data is descriptive of a response to the query (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate response data).
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 11:
obtaining, by a computing system comprising one or more processors, input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (recites insignificant extra solution activity that amounts to obtaining data);
processing, by the computing system, the input data with a machine-learned model
providing, by the computing system, the image data to the first data processing tool to receive first output data,
processing, by the computing system, the first output data with the machine-learned model to generate second planning data, wherein the second planning data is descriptive of instructions to provide a particular portion of the image data to a second data processing tool (recites insignificant extra solution activity that amounts to providing a portion of image data);
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Independent claims 17 recites the following limitations directed towards a Mental Processes:
processing the input data with a machine-learned model to generate first planning data,
processing the first output data with the machine-learned model to generate second planning data
processing the text data and the second output data with the machine-learned model to generate response data, wherein the response data is descriptive of a response to the query (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate response data).
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 17:
obtaining input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (recites insignificant extra solution activity that amounts to obtaining data);
providing the image data to the first data processing tool (recites insignificant extra solution activity that amounts to providing image data);
receiving first output data from the first data processing tool based on the image data (recites insignificant extra solution activity that amounts to mere data gathering);
processing the first output data with the machine-learned model to generate second planning data, wherein the second planning data is descriptive of instructions to provide a particular portion of the image data to a second data processing tool (recites insignificant extra solution activity that amounts to providing data);
providing the particular portion of the image data to the second data processing tool (recites insignificant extra solution activity that amounts to providing data);
receiving second output data from the second data processing tool based on the particular portion of the image data (recites insignificant extra solution activity that amounts to mere data gathering); and
processing the text data and the second output data with the machine-learned model to generate response data (recites insignificant extra solution activity that amounts to machine learning model), wherein the response data is descriptive of a response to the query.
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Independent claims 21 recites the following limitations directed towards a Mental Processes:
processing the input data with an artificial intelligence system to generate response data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate response data), wherein the response data is descriptive of a response to the query, wherein the artificial intelligence system comprises:
a language model planner block, wherein the language model planner block was conditioned to determine one or more data processing tools to utilize for processing data associated with responding to the query (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate response data);
a language model reasoner block, wherein the language model reasoner block was conditioned to process outputs of the one or more data processing tools to determine information associated with responding to the query (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determine information).
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 21:
one or more processors (i.e., as a generic component performing a generic computer function); and
one or more non-transitory computer-readable media (i.e., as a generic component performing a generic computer function) that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (recites insignificant extra solution activity that amounts to obtaining input data);
a working memory, wherein the working memory stores acquired information obtained and generated with the artificial intelligence system (recites insignificant extra solution activity that amounts to storing data); and
providing the response data as an output (recites insignificant extra solution activity that amounts to providing response data).
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Therefore, independent claims 1, 11, 17, and 21 are rejected under 35 U.S.C. 101.
With respect to claim(s) 2:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the first data processing tool comprises an object detection model (recites insignificant extra solution activity that amounts to providing input data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 3:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the first output data comprises one or more bounding boxes associated with one or more objects in the image data (recites insignificant extra solution activity that amounts to providing input data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 4:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the first output data comprises one or more segmented portions of one or more images of the image data and caption data associated with the one or more segmented portions (recites insignificant extra solution activity that amounts to providing input data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 5:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the caption data is descriptive of an object classification associated with one or more objects detected in the one or more segmented portions of one or more images (recites insignificant extra solution activity that amounts to providing input data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 6:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the second data processing tool comprises a search engine (recites insignificant extra solution activity that amounts to transmitting data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 7:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the second planning data comprises a model- generated query, and wherein the model-generated query is transmitted to the second data processing tool to retrieve the second output data (recites insignificant extra solution activity that amounts to transmitting data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 8:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the model-generated query is generated based on the input data and the first output data, wherein the model-generated query is descriptive of the query of the input data modified based on the first output data (recites insignificant extra solution activity that amounts to transmitting data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 9:
Step 2A, prong one of the 2019 PEG:
wherein the response data comprises a natural language text string that is responsive to the query of the input data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate response data).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 10:
Step 2A, prong one of the 2019 PEG:
processing the input data and the second output data with the machine-learned model to generate third planning data,
processing the input data and the third output data with the machine-learned model to generate the response data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate response data).
Step 2A Prong Two Analysis:
processing the input data and the second output data with the machine-learned model to data to a third data processing tool (recites insignificant extra solution activity that amounts to transmitting data);
transmitting, based on the third planning data, data to the third data processing tool (recites insignificant extra solution activity that amounts to transmitting data) to retrieve third output data (recites insignificant extra solution activity that amounts to retrieve data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 12:
Step 2A, prong one of the 2019 PEG:
Step 2A Prong Two Analysis:
wherein the machine-learned model was conditioned on a training dataset comprising a plurality of training examples, wherein each training example of the plurality of training examples comprises a training input, a training output, and a training rationale, wherein the training rationale is descriptive of a sequence of processing instances
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 13:
Step 2A, prong one of the 2019 PEG:
wherein the machine-learned model was conditioned on a training dataset comprising sequence data, wherein sequence data is descriptive of a sequence of actions for generating a training response in response to obtaining a particular type of input (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a training response).
Step 2A Prong Two Analysis:
wherein the machine-learned model was conditioned on a training dataset comprising sequence data, wherein sequence data is descriptive of a sequence of actions for generating a training response in response to obtaining a particular type of input (recites insignificant extra solution activity that amounts to machine learning modeling).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 14:
Step 2A, prong one of the 2019 PEG:
wherein the machine-learned model was conditioned on human input data descriptive of actions a human selected as being particular actions to perform when a particular input type is received (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to select an action).
Step 2A Prong Two Analysis:
wherein the machine-learned model was conditioned on human input data descriptive of actions a human selected as being particular actions to perform when a particular input type is received (recites insignificant extra solution activity that amounts to mere data gathering).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 15:
Step 2A, prong one of the 2019 PEG:
wherein the human input data was obtained via a user interface that provides a plurality of selectable options for a user (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to select options).
Step 2A Prong Two Analysis:
wherein the human input data was obtained via a user interface that provides a plurality of selectable options for a user (recites insignificant extra solution activity that amounts to obtaining data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 16:
Step 2A, prong one of the 2019 PEG:
wherein the plurality of selectable options comprise a plurality of external tools to call and a final output option (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to select options).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 18:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the query is descriptive of a question associated with a particular object depicted in the image data (recites insignificant extra solution activity that amounts to obtaining data).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 19:
Step 2A, prong one of the 2019 PEG:
detects a plurality of objects depicted in the image data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to detect objects);
generates a plurality of bounding boxes associated with the plurality of objects (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate bounding boxes);
generates a plurality of image segments based on the plurality of bounding boxes (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate image segments);
classifies each of the plurality of objects in the plurality of image segments to generate a plurality of object classifications (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to classifying objects); and
generates the first output data, wherein the first output data comprises the plurality of image segments and the plurality of object classifications (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate output data).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 20:
Step 2A, prong one of the 2019 PEG:
processes one or more of the plurality of image segments with the image search engine to determine one or more web resources associated with the one or more of the plurality of image segments, wherein the one or more of the plurality of image segments are selected by the machine-learned model based on the input data and the plurality of object classifications (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determine web resources); and
generates the second output data based on the one or more web resources (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate output data).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 22:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the artificial intelligence system comprises one or more machine-learned models conditioned on user behavior data (recites insignificant extra solution activity that amounts to machine learning modeling).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 23:
Step 2A, prong one of the 2019 PEG:
wherein the user behavior data is processed to generate a transition graph that is descriptive of a determined sequence of decisions made by users when performing a particular information seeking task (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a transition graph).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 24:
Step 2A, prong one of the 2019 PEG:
wherein the transition graph is descriptive of a plurality of distinct states and indicates a particular set of actions available at each state of the plurality of distinct states (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a transition graph).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 25:
Step 2A, prong one of the 2019 PEG:
Examiner is of the position the dependent claim is directed toward additional elements.
Step 2A Prong Two Analysis:
wherein the user behavior data was utilized to condition the language model planner block and the language model reasoner block for particular context-based processing (recites insignificant extra solution activity that amounts to machine learning modeling).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 26:
Step 2A, prong one of the 2019 PEG:
wherein the one or more data processing tools comprise at least one of a computer vision tool, a web search tool, or an image search tool (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determining a data modeling tool).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 27:
Step 2A, prong one of the 2019 PEG:
wherein the computer vision tool comprises at least one of an object detection model, an optical character recognition model, an image captioning model, or a visual question-and-answer model (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determining a data modeling tool).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 28:
Step 2A, prong one of the 2019 PEG:
wherein the web search tool retrieves open world knowledge from web resources (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determining a data modeling tool).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 29:
Step 2A, prong one of the 2019 PEG:
wherein the image search tool identifies relevant information from metadata associated with visually similar images (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to determining a data modeling tool).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 30:
Step 2A, prong one of the 2019 PEG:
wherein the language model planner block was conditioned to generate a processing tool query to provide to the one or more data processing tools based on the input data (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using computer as a tool to generate a query).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no
additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 21-30 is/are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Moon et al. (US Pub. No. 20200410012).
As to claim 21, Moon et al. teaches a computing system, the system comprising:
one or more processors (Paragraphs 227 and 228 discloses a processor); and
one or more non-transitory computer-readable media (Paragraph 234 discloses a computer-readable non-transitory storage medium or media) that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (Paragraph 34 discloses the user may provide input via different modalities. As an example and not by way of limitation, the modalities may include audio, text, image, video, motion, orientation, etc and Paragraph 70 discloses a query from the user);
processing the input data with an artificial intelligence system to generate response data, wherein the response data is descriptive of a response to the query (Paragraph 8 discloses generate a response based on the initial memory slots and episodic memories corresponding to the selected candidate nodes), wherein the artificial intelligence system comprises:
a language model planner block, wherein the language model planner block was conditioned to determine one or more data processing tools to utilize for processing data associated with responding to the query (Paragraph 99 discloses query encoder then encodes the input query with a language model, and the memory encoder encodes each memory slot for both structural and semantic properties of each memory and Paragraph 9 discloses implementing multiple sub-module nets as these sub-module nets can be combined in a variety of ways to deal with various types of questions);
a language model reasoner block, wherein the language model reasoner block was conditioned to process outputs of the one or more data processing tools to determine information associated with responding to the query (Paragraph 94 discloses evaluate multiple answer candidates with transitive reasoning); and
a working memory, wherein the working memory stores acquired information obtained and generated with the artificial intelligence system (Paragraph 46 discloses long-short term memory (LSTM) networks); and
providing the response data as an output (Paragraph 67 discloses response manager 325 may perform different tasks including storing/updating the dialog state 326 retrieved from data store 310 and generating responses 327).
As to clam 22, Moon et al. discloses the system of claim 21, wherein the artificial intelligence system comprises one or more machine-learned models conditioned on user behavior data (Paragraph 8 discloses the assistant system may select, by one or more machine-learning models based on the initial memory slots, one or more candidate nodes from the memory graph and Paragraph 43 discloses A user profile may include, for example, biographic information, demographic information, behavioral information, social information, or other types of descriptive information, such as work experience, educational history, hobbies or preferences, interests, affinities, or location).
As to clam 23, Moon et al. discloses the system of claim 22, wherein the user behavior data is processed to generate a transition graph that is descriptive of a determined sequence of decisions made by users when performing a particular information seeking task (Paragraph 78 discloses To bootstrap a large-scale dataset collection for Episodic Memory QA, the embodiments disclosed herein first build a synthetic memory graph generator, which creates multiple episodic memory graph nodes).
As to clam 24, Moon et al. discloses the system of claim 23, wherein the transition graph is descriptive of a plurality of distinct states and indicates a particular set of actions available at each state of the plurality of distinct states (Paragraph 44 discloses assistant system 140 may enable the user to interact with it with multi-modal user input (such as voice, text, image, video, motion) in stateful and multi-turn conversations to get assistance
As to clam 25, Moon et al. discloses the system of claim 22, wherein the user behavior data was utilized to condition the language model planner block and the language model reasoner block for particular context-based processing (Paragraph 102 discloses contextual representation of memories (m.sub.c), the embodiments disclosed herein compute attention-weighted sum of textual representation of neighboring nodes and attributes (connected via r.sub.j∈R), using the same language model as the query encoder).
As to clam 26, Moon et al. discloses the system of claim 21, wherein the one or more data processing tools comprise at least one of a computer vision tool, a web search tool, or an image search tool (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object).
As to clam 27, Moon et al. discloses the system of claim 26, wherein the computer vision tool comprises at least one of an object detection model, an optical character recognition model, an image captioning model, or a visual question-and-answer model (Paragraph 153 discloses visual QA systems).
As to clam 28, Moon et al. discloses the system of claim 26, wherein the web search tool retrieves open world knowledge from web resources (Paragraph 43 discloses the social-networking system 160 may include a variety of servers, sub-systems, programs, modules, logs, and data stores. In particular embodiments, the social-networking system 160 may include one or more of the following: a web server, action logger, API-request server, relevance-and-ranking engine, content-object classifier, notification controller, action log, third-party-content-object-exposure log, inference module, authorization/privacy server, search module, advertisement-targeting module, user-interface module, user-profile store, connection store, third-party content store, or location store).
As to clam 29, Moon et al. discloses the system of claim 26, wherein the image search tool identifies relevant information from metadata associated with visually similar images (Paragraph 76 discloses 1) the ability of querying a personal database to answer a user query (memory recall QA), 2) surfacing photos most relevant to the dialog, and 3) identifying other memories to surface that are relevant to conversational contexts, resulting in increased engagement and coherent interactions).
As to clam 30, Moon et al. discloses the system of claim 21, wherein the language model planner block was conditioned to generate a processing tool query to provide to the one or more data processing tools based on the input data (Paragraph 94 discloses output of the query encoder 806 and the memory encoder 808 may be processed by a modality attention module 812).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Moon et al. (US Pub. No. 20200410012) in view of Peng et al. (US Pub. No. 20230177821).
With respect to claim 1, Moon et al. teaches a computing system for visual information seeking, the system comprising:
one or more processors (Paragraph 227 discloses a processor); and
one or more non-transitory computer-readable media (Paragraph 230 discloses removable or non-removable (or fixed) media) that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (Paragraph 34 discloses the user may provide input via different modalities. As an example and not by way of limitation, the modalities may include audio, text, image, video, motion, orientation, etc and Pargraph 70 discloses a query from the user);
transmitting, based on the first planning data, the input data to the first data processing tool to retrieve first output data (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes). Moon et al. does not disclose processing the input data with a machine-learned model…; processing the input data and the first output data with the machine-learned model…; transmitting, based on the second planning data, data…; …generate response data.
However, Peng et al. discloses processing the input data with a machine-learned model (Paragraph 156 discloses running a machine learning model algorithm) to generate first planning data, wherein the first planning data is descriptive of instructions to provide the input data to a first data processing tool (Paragraph 55 discloses the neural network model 170, by executing text information extraction 150 and image information extraction 160, respective features of an text and an image for being input into a neural network can be obtained, while at downstream of the neural network model, the neural network model 170 may be trained according to different tasks in a target 190 or a document image understanding result may be obtained);
processing the input data and the first output data with the machine-learned model to generate second planning data, wherein the second planning data is descriptive of instructions to transmit data to a second data processing tool (Paragraph 55 discloses the neural network model 170, by executing text information extraction 150 and image information extraction 160, respective features of an text and an image for being input into a neural network can be obtained, while at downstream of the neural network model, the neural network model 170 may be trained according to different tasks in a target 190 or a document image understanding result may be obtained and Paragraph 60 discloses target 190 is a task that may be executed by utilizing the representation feature 188 output by the neural network model, and includes a fine-grained image-text matching task 192, a mask language model 194, fine-tuning 196, and a downstream task 198 for document image understanding);
transmitting, based on the second planning data, data to the second data processing tool to retrieve second output data (Paragraph 60 target 190 is a task that may be executed by utilizing the representation feature 188 output by the neural network model, and includes a fine-grained image-text matching task 192, a mask language model 194, fine-tuning 196, and a downstream task 198 for document image understanding); and
processing the input data and the second output data with the machine-learned model to generate response data, wherein the response data is descriptive of a response to the query (Paragraph 137 discloses determining a question and expected answer (i.e., ground truth label) related to the sample document image, generating the at least one text comprehensive feature corresponding to the question, then inputting feature and the text comprehensive feature and image comprehensive feature corresponding to the text in the document into the neural network model together, e.g., simultaneously, predicting the answer to the question based on the text representation feature).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify over Moon et al.’s question answering with Peng et al.’s document image understanding. This would have facilitated visual question answering.
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 2, Moon et al. teaches the system of claim 1, wherein the first data processing tool comprises an object detection model (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 3, Moon et al. teaches the system of claim 2, wherein the first output data comprises one or more bounding boxes associated with one or more objects in the image data (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 4, Moon et al. teaches the system of claim 2, wherein the first output data comprises one or more segmented portions of one or more images of the image data and caption data associated with the one or more segmented portions (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information and Paragraph 205 discloses a function i may map an object e to a vector (e) based on one or more n-grams associated with object e).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 5, Moon et al. teaches the system of claim 4, wherein the caption data is descriptive of an object classification associated with one or more objects detected in the one or more segmented portions of one or more images (Paragraph 43 discloses sub-systems, programs, modules, logs, and data stores. In particular embodiments, the social-networking system 160 may include one or more of the following: a web server, action logger, API-request server, relevance-and-ranking engine, content-object classifier).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 6, Moon et al. teaches the system of claim 1, wherein the second data processing tool comprises a search engine (Paragraph 92 discloses a search engine).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 7, Moon et al. teaches the system of claim 6, wherein the second planning data comprises a model- generated query, and wherein the model-generated query is transmitted to the second data processing tool to retrieve the second output data (Paragraph 55 discloses a straightforward backend query using deterministic filters to retrieve the candidate entities from a structured data store. The generation may be alternatively based on a machine-learning model that is trained based on the user profile, entity attributes, and relevance between users and entities).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 8, Moon et al. teaches the system of claim 7, wherein the model-generated query is generated based on the input data and the first output data, wherein the model-generated query is descriptive of the query of the input data modified based on the first output data (Paragraph 55 discloses a straightforward backend query using deterministic filters to retrieve the candidate entities from a structured data store. The generation may be alternatively based on a machine-learning model that is trained based on the user profile, entity attributes, and relevance between users and entities).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 9, Moon et al. teaches the system of claim 1, wherein the response data comprises a natural language text string that is responsive to the query of the input data (Paragraph 66 discloses the NLG 271 may use different language models and/or language templates to generate natural language outputs. The generation of natural language outputs may be application specific. The generation of natural language outputs may be also personalized for each user. The CU composer 270 may also determine a modality of the generated communication content).
The Moon et al. reference as modified by Peng et al. teaches all the limitations of claim 1. With respect to claim 10, Moon et al. teaches the system of claim 1, wherein the operations further comprise:
processing the input data and the second output data with the machine-learned model to generate third planning data, wherein the third planning data is descriptive of instructions to transmit data to a third data processing tool (Paragraph 66 discloses the NLG 271 may use different language models and/or language templates to generate natural language outputs. The generation of natural language outputs may be application specific. The generation of natural language outputs may be also personalized for each user. The CU composer 270 may also determine a modality of the generated communication content);
transmitting, based on the third planning data, data to the third data processing tool to retrieve third output data (Paragraph 66 discloses the NLG 271 may use different language models and/or language templates to generate natural language outputs. The generation of natural language outputs may be application specific. The generation of natural language outputs may be also personalized for each user. The CU composer 270 may also determine a modality of the generated communication content); and
processing the input data and the third output data with the machine-learned model to generate the response data (Paragraph 66 discloses the NLG 271 may use different language models and/or language templates to generate natural language outputs. The generation of natural language outputs may be application specific. The generation of natural language outputs may be also personalized for each user. The CU composer 270 may also determine a modality of the generated communication content).
As to claim 11, Moon et al. teaches a computer-implemented method for responding to visual prompts, the method comprising:
obtaining, by a computing system comprising one or more processors, input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (Paragraph 34 discloses the user may provide input via different modalities. As an example and not by way of limitation, the modalities may include audio, text, image, video, motion, orientation, etc and Pargraph 70 discloses a query from the user);
providing, by the computing system, the image data to the first data processing tool to receive first output data, wherein the first output data is generated with the first data processing tool based on the image data (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information). Moon et al. does not disclose processing the input data with a machine-learned model…; processing the input data and the first output data with the machine-learned model…; transmitting, based on the second planning data, data…; …generate response data.
However, Peng et al. discloses processing, by the computing system, the input data with a machine-learned model to generate first planning data, wherein the first planning data is descriptive of instructions to provide the image data to a first data processing tool (Paragraph 156 discloses running a machine learning model algorithm) to generate first planning data, wherein the first planning data is descriptive of instructions to provide the input data to a first data processing tool (Paragraph 55 discloses the neural network model 170, by executing text information extraction 150 and image information extraction 160, respective features of an text and an image for being input into a neural network can be obtained, while at downstream of the neural network model, the neural network model 170 may be trained according to different tasks in a target 190 or a document image understanding result may be obtained);
processing, by the computing system, the text data and the second output data with the machine-learned model to generate response data, wherein the response data is descriptive of a response to the query (Paragraph 67 discloses response manager 325 may perform different tasks including storing/updating the dialog state 326 retrieved from data store 310 and generating responses 327. Moon et al. does not disclose processing the input data with a machine-learned model…; providing, by the computing system, the particular portion of the image data to the second data processing tool...
However, Peng et al. discloses processing, by the computing system, the first output data with the machine-learned model to generate second planning data, wherein the second planning data is descriptive of instructions to provide a particular portion of the image data to a second data processing tool (Paragraph 55 discloses the neural network model 170, by executing text information extraction 150 and image information extraction 160, respective features of an text and an image for being input into a neural network can be obtained, while at downstream of the neural network model, the neural network model 170 may be trained according to different tasks in a target 190 or a document image understanding result may be obtained and Paragraph 60 discloses target 190 is a task that may be executed by utilizing the representation feature 188 output by the neural network model, and includes a fine-grained image-text matching task 192, a mask language model 194, fine-tuning 196, and a downstream task 198 for document image understanding);
providing, by the computing system, the particular portion of the image data to the second data processing tool to receive second output data, wherein the second output data is generated with the second data processing tool based on the particular portion of the image data (Paragraph 139 discloses tpositions, shapes, and sizes of the plurality of image regions in a fine-tuned task may be similar to or the same as positions, shapes, and sizes of the plurality of first image regions in the pre-training task above, so as to enhance the learning of the model for multimodal image-text information related to a specific region).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify over Moon et al.’s question answering with Peng et al.’s document image understanding. This would have facilitated visual question answering.
As to claim 12, Moon et al. teaches the method of claim 11, wherein the machine-learned model was conditioned on a training dataset comprising a plurality of training examples, wherein each training example of the plurality of training examples comprises a training input, a training output, and a training rationale, wherein the training rationale is descriptive of a sequence of processing instances and tool calls for determining the output data (Paragraph 204 discloses machine learning model may have been trained using a sequence of training data).
As to claim 13, Moon et al. teaches the method of claim 11, wherein the machine-learned model was conditioned on a training dataset comprising sequence data, wherein sequence data is descriptive of a sequence of actions for generating a training response in response to obtaining a particular type of input (Paragraph 211 discloses an ANN may be trained using training data. As an example and not by way of limitation, training data may comprise inputs to the ANN 1200 and an expected output).
As to claim 14, Moon et al. teaches the method of claim 11, wherein the machine-learned model was conditioned on human input data descriptive of actions a human selected as being particular actions to perform when a particular input type is received (Paragraph 120 discloses It is observed that the model performs better when the MGN component of the model is pre-trained with ground-truth paths. The embodiments disclosed herein thus first train the MGN network with the same training split (without answer labels), and then train the entire model with module networks, fully end-to-end supervised with custom-character=custom-character.sub.walk+custom-character.sub.module+custom-character.sub.ans and Paragraph 38 discloses the social-networking system 160 may provide users with the ability to take actions on various types of items or objects, supported by the social-networking system 160).
As to claim 15, Moon et al. teaches the method of claim 14, wherein the human input data was obtained via a user interface that provides a plurality of selectable options for a user (Paragraph 75 discloses a natural interface for retrieving and browsing through one's memories. Traditionally, the interfaces to access these memories are either (i) keyword-based search systems which demand specific keyword combinations to identify and retrieve the correct memories, or (ii) catalog based browsing systems that allow scrolling through memories across a single dimension, most commonly, time of creation).
As to claim 16, Moon et al. teaches the method of claim 15, wherein the plurality of selectable options comprise a plurality of external tools to call and a final output option (Paragraph 156 discloses the assistant system 140 may select, by one or more machine-learning models based on the initial memory slots, one or more candidate nodes from the memory graph based on a determination of relevance between the initial memory slots and each of the plurality of nodes in the memory graph).
As to claim 17, Moon et al. teaches one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises image data and text data, wherein the text data comprises a query associated with the image data (Paragraph 34 discloses the user may provide input via different modalities. As an example and not by way of limitation, the modalities may include audio, text, image, video, motion, orientation, etc and Pargraph 70 discloses a query from the user);
Pargraph 70 discloses a query from the user);
providing the image data to the first data processing tool (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information);
receiving first output data from the first data processing tool based on the image data (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information). Moon et al. does not disclose processing the input data with a machine-learned model…; processing the first output data with the machine-learned model to generate second planning data …; providing the particular portion of the image data to the second data processing tool …; receiving second output data from the second data processing tool based on the particular portion of the image data …; …generate response data.
processing the input data with a machine-learned model (Paragraph 156 discloses running a machine learning model algorithm) to generate first planning data, wherein the first planning data is descriptive of instructions to provide the image data to a first data processing tool (Paragraph 55 discloses the neural network model 170, by executing text information extraction 150 and image information extraction 160, respective features of an text and an image for being input into a neural network can be obtained, while at downstream of the neural network model, the neural network model 170 may be trained according to different tasks in a target 190 or a document image understanding result may be obtained);
processing the first output data with the machine-learned model to generate second planning data, wherein the second planning data is descriptive of instructions to provide a particular portion of the image data to a second data processing tool (Paragraph 55 discloses the neural network model 170, by executing text information extraction 150 and image information extraction 160, respective features of an text and an image for being input into a neural network can be obtained, while at downstream of the neural network model, the neural network model 170 may be trained according to different tasks in a target 190 or a document image understanding result may be obtained and Paragraph 60 discloses target 190 is a task that may be executed by utilizing the representation feature 188 output by the neural network model, and includes a fine-grained image-text matching task 192, a mask language model 194, fine-tuning 196, and a downstream task 198 for document image understanding);
providing the particular portion of the image data to the second data processing tool (Paragraph 60 target 190 is a task that may be executed by utilizing the representation feature 188 output by the neural network model, and includes a fine-grained image-text matching task 192, a mask language model 194, fine-tuning 196, and a downstream task 198 for document image understanding);
receiving second output data from the second data processing tool based on the particular portion of the image data (Paragraph 156 discloses running a machine learning model algorithm) to generate first planning data, wherein the first planning data is descriptive of instructions to provide the input data to a first data processing tool (Paragraph 55 discloses the neural network model 170, by executing text information extraction 150 and image information extraction 160, respective features of an text and an image for being input into a neural network can be obtained, while at downstream of the neural network model, the neural network model 170 may be trained according to different tasks in a target 190 or a document image understanding result may be obtained); and
processing the text data and the second output data with the machine-learned model to generate response data, wherein the response data is descriptive of a response to the query (Paragraph 137 discloses determining a question and expected answer (i.e., ground truth label) related to the sample document image, generating the at least one text comprehensive feature corresponding to the question, then inputting feature and the text comprehensive feature and image comprehensive feature corresponding to the text in the document into the neural network model together, e.g., simultaneously, predicting the answer to the question based on the text representation feature).
Therefore, it would have been obvious at the time the invention was made to a person having ordinary skill in the art to modify over Moon et al.’s question answering with Peng et al.’s document image understanding. This would have facilitated visual question answering.
As to claim 18, Moon et al. teaches the one or more non-transitory computer-readable media of claim 17, wherein the query is descriptive of a question associated with a particular object depicted in the image data (Paragraph 73 discloses main QA model takes these graph traversal paths and expanded memory slots as input and predicts correct answers via multiple module networks (e.g. COUNT, CHOOSE, etc.). Examples of such queries include “Where did we go after we had brunch with Jon?”, “How many times did I go to jazz concerts last year?”, etc. For episodic memory QA, a machine has to understand the contexts of a question and navigate multiple MG episode nodes as well as KG nodes to gather comprehensive information to match the query requirement. In other words, the query may be associated with a context and selecting the one or more candidate nodes may be further based on the context associated with the query and 205 discloses an object may be represented in the vector space 1100 as a vector referred to as a feature vector or an object embedding. As an example and not by way of limitation, objects e.sub.1 and e.sub.2 may be mapped to vectors custom-character and custom-character in the vector space 1100, respectively, by applying a function custom-character, such that custom-character=custom-character(e.sub.1) and custom-character=custom-character(e.sub.2). In particular embodiments, an object may be mapped to a vector based on one or more properties, attributes, or features of the object, relationships of the object with other objects, or any other suitable information associated with the object. As an example and not by way of limitation, a function may map objects to vectors by feature extraction, which may start from an initial set of measured data and build derived values (e.g., features). As an example and not by way of limitation, an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information).
As to claim 19, Moon et al. teaches the one or more non-transitory computer-readable media of claim 18, wherein the first data processing tool:
detects a plurality of objects depicted in the image data (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information);
generates a plurality of bounding boxes associated with the plurality of objects (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information); generates a plurality of image segments based on the plurality of bounding boxes; classifies each of the plurality of objects in the plurality of image segments to generate a plurality of object classifications; and
generates the first output data, wherein the first output data comprises the plurality of image segments and the plurality of object classifications.
As to claim 20, Moon et al. teaches the one or more non-transitory computer-readable media of claim 19, wherein the second data processing tool comprises an image search engine, wherein the second data processing tool:
processes one or more of the plurality of image segments with the image search engine to determine one or more web resources associated with the one or more of the plurality of image segments, wherein the one or more of the plurality of image segments are selected by the machine-learned model based on the input data and the plurality of object classifications (Paragraph 205 discloses an object comprising a video or an image may be mapped to a vector by using an algorithm to detect or isolate various desired portions or shapes of the object. Features used to calculate the vector may be based on information obtained from edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transformation, edge direction, changing intensity, autocorrelation, motion detection, optical flow, thresholding, blob extraction, template matching, Hough transformation (e.g., lines, circles, ellipses, arbitrary shapes), or any other suitable information); and
generates the second output data based on the one or more web resources (Paragraph 82 discloses The UI enables natural language QA queries through text or voice input. This UI can be deployed in a desktop, web or mobile application, or on a connected home device like smart TVs).
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US PG-PUB 20210081728 is directed to CONTEXTUAL GROUNDING OF NATURAL LANGUAGE PHRASES IN IMAGES: [0035-0036] Access to such a system is achieved through—for example—a computer browser that shows an input field for a user to type a query w.r.t. an image and renders retrieval results in an image. Accordingly, input to the system is a pair of textual query(ies) and image(s). The query is parsed into tokens and applied (fed) into an object detector to locate salient regions as visual object candidates for subsequent grounding. The contextual grounding module accepts both entity embeddings and visual object representations as input and scores their correspondences in probabilities. Finally, the object corresponding to the query language entity with the highest probability score is retrieved and visualized in a bounding box to the user.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS E ALLEN whose telephone number is (571)270-3562. The examiner can normally be reached Monday through Thursday 830-630.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached at (571) 270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.E.A/Examiner, Art Unit 2154
/SYED H HASAN/Primary Examiner, Art Unit 2154
