DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1-9 are objected to because of the following informalities:
Claim 1 line 2; claim 8 line 3; claim 9 line 5 “the basis of a correlation” should be “a basis of a correlation” because there is lack of antecedent basis for such limitation.
Claim 1 line 8-9; claim 8 line 9; claim 9 line 11 “the basis of a correlation of two different types of vectors” should be “a basis of a correlation of two different types of vectors” because there is lack of antecedent basis for such limitation.
Claim 1 line 11-13; claim 8 line 12-14; claim 9 line 14-16 “generating a second vector on the basis of the correlation of the two different types of vectors obtained from the corrected vector based on the information of the second modal” should be “generating a second vector on a basis of a correlation of two different types of vectors obtained from the corrected vector based on the information of the second modal” because there is lack of antecedent basis for a basis of a correlation corresponding to the second modal.
Claim 1 line 14-17; claim 8 line 15-18; claim 9 line 18 “generating a third vector in which the first vector and the second vector are aggregated on the basis of the correlation of the two different types of vectors obtained from a combined vector that includes a predetermined vector, the generated first vector, and the generated second vector” should be “generating a third vector in which the first vector and the second vector are aggregated on a basis of a correlation of two different types of vectors obtained from a combined vector that includes a predetermined vector, the generated first vector, and the generated second vector” because there is lack of antecedent basis for a basis of a correlation corresponding to a combined vector.
Claims 2-7 line 1 “The output method according” should be “The computer-implemented output method according” because claim 1 line 1 antecedently recites “A computer-implemented output method”. Furthermore, claim 1 line 18 also recites a step of outputting the generated third vector, so when reciting “The output method” can also be corresponding to such outputting step.
Claim 2 line 3-4 “the basis of an inner product” should be “a basis of an inner product” because there is lack of antecedent basis for such limitation.
Claim 2 line 16 “the basis of an inner product of the two different types of vectors” should be “a basis of an inner product of the two different types of vectors” because there is lack of antecedent basis for such limitation.
Claim 2 line 22-23 “the basis of an inner product of the two different types of vectors obtained from … the second modal” should be “a basis of an inner product of the two different types of vectors obtained from … the second modal” because there is lack of antecedent basis for the basis of an inner product corresponding to the second modal.
Claim 2 line 27-28 “the basis of an inner product of the two different types of vectors obtained from the combined vector” should be “a basis of an inner product of the two different types of vectors obtained from the combined vector” because there is lack of antecedent basis for the basis of an inner product corresponding to the combined vector.
Claim 3 line 1; claim 4 line 2; claim 7 line 1 “the computer” should be “a computer” because there is lack of antecedent basis for such limitation.
Dependent claims are also objected for inheriting the same deficiencies in which claims they depend on.
Appropriate correction is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 4 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 4 line 7-9 recites “correcting the set vector based on the information of the first modal on the basis of the correlation between the set vector based on the information of the first modal and the set vector based on the information of the second modal”. There is lack of antecedent basis for “the set vector”. For examination purposes, Examiner interprets as “correcting the new vector based on the information of the first modal on the basis of the correlation between the new vector based on the information of the first modal and the new vector based on the information of the second modal”.
Claim 4 line 10-13 “correcting the set vector based on the information of the second modal on the basis of the correlation between the set vector based on the information of the first modal and the set vector based on the information of the second modal”. There is lack of antecedent basis for “the set vector”. For examination purposes, Examiner interprets as “correcting the new vector based on the information of the second modal on the basis of the correlation between the new vector based on the information of the first modal and the new vector based on the information of the second modal”.
Claim 4 line 14-19 recites “generating the first vector on the basis of the correlation of two different types of vectors obtained from the corrected vector based on the information of the first modal; and generating the second vector on the basis of the correlation of two different types of vectors obtained from the corrected vector based on the information of the second modal”. It is unclear whether the first vector, the second vector, the corrected vector based on the information of the first model and the corrected vector based on the second modal are referring to vectors in the first processing as recited in claim 1 or the repeated processing as recited in claim 4 line 7-10 because claim 4 line 2 recites the computer repeats one or more times. For examination purposes, Examiner interprets the first vector, the second vector, the corrected vector are vectors performed within the repeated processing as recited in claim 4. One possible amendment is “generating a first new vector on the basis of the correlation of two different types of vectors obtained from the new corrected vector based on the information of the first modal; and generating a second new vector on the basis of the correlation of two different types of vectors obtained from the new corrected vector based on the information of the second modal”.
Claim 4 line 20-24 recites “the generating a third vector includes generating the third vector in which the first vector and the second vector are aggregated on the basis of the correlation of two different types of vectors obtained from the combined vector that includes the predetermined vector, the generated first vector, and the generated second vector.” Again, it is unclear whether the third vector, the first vector, the second vector, the combined vector, the predetermined vector, the generated first vector and the generated second vector are referring to vectors in the first processing as recited in claim 1 or the repeated processing as recited in claim 4 because claim 4 line 2 requires one or more times of processing to be repeated. For examination purposes, Examiner interprets the third vector, the first vector, the second vector, the combined vector, the predetermined vector, the generated first vector and the generated second vector are vectors performed within the repeated processing as recited in claim 4. One possible amendment is “the generating a third vector includes generating a third new vector in which the first new vector and the second new vector are aggregated on the basis of the correlation of two different types of vectors obtained from a new combined vector that includes a predetermined vector, the new generated first vector, and the new generated second vector.”
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites a computer implemented output method
Under Prong One of Step 2A of the USPTO current eligibility guidance (MPEP 2106), the claim recites limitations cover mathematical calculations, relationship, and/or formula, such as correcting a vector based on information of a first modal on the basis of a correlation between the vector based on the information of the first modal and a vector based on information of a second modal different from the first modal (see at least [0103] describes step of correction based on correlation expressed by a degree of similarity, which is expressed by an inner product or a sum of squares of differences. Also see figure 6 illustrates the Target-Attention layer to perform correction that includes a series of mathematical computations); correcting the vector based on the information of the second modal on the basis of the correlation between the vector based on the information of the first modal and the vector based on the information of the second modal (see at least [0109] describes step of correction based on correlation expressed by a degree of similarity, which is expressed by an inner product or a sum of squares of differences. Also see figure 6 illustrates the Target-Attention layer to perform correction that includes a series of mathematical computations); generating a first vector on the basis of a correlation of two different types of vectors obtained from the corrected vector based on the information of the first modal (see at least [0106] describes step of generating vector based on correlation expressed by a degree of similarity, which is expressed by an inner product or a sum of squares of differences. Also see figure 6 illustrates the Self-Attention layer to perform generating that includes a series of mathematical computations); generating a second vector on the basis of the correlation of the two different types of vectors obtained from the corrected vector based on the information of the second modal (see at least [0112] describes step of generating vector based on correlation expressed by a degree of similarity, which is expressed by an inner product or a sum of squares of differences. Also see figure 6 illustrates the Self-Attention layer to perform generating that includes a series of mathematical computations)); generating a third vector in which the first vector and the second vector are aggregated on the basis of the correlation of the two different types of vectors obtained from a combined vector that includes a predetermined vector, the generated first vector, and the generated second vector (see at least [0116] describes step of generating vector based on combining the generated vectors and a predetermined vector. Also see figure 5 step 505 describe the combining step). Therefore, the claim includes limitations that fall within the “Mathematical Concepts” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Under Prong Two of Step 2A, this judicial exception is not integrated into a practical application. The claim additionally recites a computer-implemented output method comprising outputting the generated third vector. However, the additional elements are recited at a high level of generality, i.e., as a generic computer performing a computer function of processing data and outputting data, wherein the step of outputting data is considered as insignificant extra solution activity as the MPEP 2106.05(g) states “all uses of the recited judicial exception require such data gathering or data output”. Such additional elements fail to provide a meaningful limitation on the judicial exception, and amount to no more than mere instructions to apply the exception using generic computer. Thus, the claim is directed to an abstract idea.
Under Step 2B, as discussed with respect to Prong Two of Step 2A, the additional elements in the claim amount no more than mere instructions to apply the exception using a generic component. The same conclusion is reached in step 2B, i.e., mere instruction to apply an exception on a generic element cannot integrate a judicial exception into a practical application at step 2A or provide an inventive concept that is furnished by an element or combination of elements that is recited in the claim in addition to (beyond) the judicial exception. The step of outputting data considered to be insignificant extra-solution activity in step 2A, and are determined to be well-understood, routine, conventional activity in the field. Court decisions cited in MPEP 2106.05(d)(II) section (i), indicate that mere receiving or transmitting data over a network, is well-understood, routing, conventional function when it is claimed in a merely generic manner. Thus, the additional element fails to ensure the claim as a whole amount to significantly more than the judicial exception itself. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Claim 2 further recites steps of correction and generation vectors based on inner products using target-attention layer and self-attention layer to generate the third vector. Such limitations cover mathematical calculations, relationship, and/or formula (see at least figure 6 illustrates self-attention, target-attention, and multi-head attention layers include series of mathematical operations). The claim does not recite additional element that would integrate the judicial exception into a practical application under step 2A prong two or ensure the claim as a whole amount to significantly more than the judicial exception itself under step 2B. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Claims 3 and 6 further recites wherein the computer executes processing comprising determining a situation that regards the first modal and the second modal on the basis of the generated third vector and outputting the situation, wherein the situation is a positive situation or a negative situation. Such step of determining a situation cover mathematical calculations, relationship, and/or formula (see at least [0121-0123] describes solving a problem on a trial basis using the generated third vector and compares an answer with the correct answer data includes a problem of determining a situation. Thus, determining a situation is to solve a problem using the updated target-attention and self-attention in the network). Furthermore, as explained above regarding the step of outputting data, the step of outputting the situation is merely considered as insignificant extra solution activity under step 2A as mere data outputting and determined to be well-understood, routine and conventional under step 2B (see MPEP 2106.05(d)(II) section (i), indicate that mere receiving or transmitting data over a network). Thus, The claim does not recite additional element that would integrate the judicial exception into a practical application under step 2A prong two or ensure the claim as a whole amount to significantly more than the judicial exception itself under step 2B. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Claim 4 further recites wherein the computer repeats, one or more times, processing comprising: setting the generated first vector as a new vector based on the information of the first modal; setting the generated second vector as a new vector based on the information of the second modal; correcting the set vector based on the information of the first modal on the basis of the correlation between the set vector based on the information of the first modal and the set vector based on the information of the second modal; correcting the set vector based on the information of the second modal on the basis of the correlation between the set vector based on the information of the first modal and the set vector based on the information of the second modal; generating the first vector on the basis of the correlation of two different types of vectors obtained from the corrected vector based on the information of the first modal; and generating the second vector on the basis of the correlation of two different types of vectors obtained from the corrected vector based on the information of the second modal, and the generating a third vector includes generating the third vector in which the first vector and the second vector are aggregated on the basis of the correlation of two different types of vectors obtained from the combined vector that includes the predetermined vector, the generated first vector, and the generated second vector. Such limitations cover mathematical calculations, relationship, and/or formula (updating the vector data for iteratively processing or in multiple stages to provide better accuracy of the solution). The claim does not recite additional element that would integrate the judicial exception into a practical application under step 2A prong two or ensure the claim as a whole amount to significantly more than the judicial exception itself under step 2B. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Claim 5 further recites wherein a set of the first modal and the second modal is one of a set of a modal related to an image and a modal related to a document, a set of a modal related to an image and a modal related to a voice, or a set of a modal related to a document in a first language and a modal related to a document in a second language. Such limitations cover mathematical calculations, relationship, and/or formula (merely describing the set of data to be operated on). Alternatively, such limitation is mere generally linking the use of the judicial exception into a particular technological environment or field of use. Therefore, such limitation does not integrate the judicial exception into a practical application under step 2A prong two or ensure the claim as a whole amount to significantly more than the judicial exception itself under step 2B. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Claim 7 further recites wherein the computer executes processing comprising updating the first target-attention layer, the second target-attention layer, the first self-attention layer, the second self-attention layer, and the third self-attention layer on the basis of the generated third vector. Such limitations cover mathematical calculations, relationship, and/or formula (such as updating data to perform in an iteratively manner to provide better accuracy solution). The claim does not recite additional element that would integrate the judicial exception into a practical application under step 2A prong two or ensure the claim as a whole amount to significantly more than the judicial exception itself under step 2B. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Claim 8 recites a product claim having similar limitation as the method claim 1. Thus, it is rejected for the same reasons. claim 8 further recites a non-transitory computer-readable storage medium storing a program for causing a computer to execute processing. Such additional elements are recited at a high level of generality, i.e., as generic computer components performing computer functions of storing and processing data. Such additional elements fail to provide a meaningful limitation on the judicial exception, and amount to no more than mere instructions to apply the exception using generic computer components. Therefore, The additional elements do not integrate the judicial exception into a practical application under step 2A prong two or ensure the claim as a whole amount to significantly more than the judicial exception itself under step 2B. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Claim 9 recites an apparatus claim that would practice the method claim 1. Thus, it is rejected for the same reasons. claim 9 further recites an output apparatus comprising a memory; and a processor coupled to the memory, the processor being configured to perform processing. Such additional elements are recited at a high level of generality, i.e., as generic computer components performing computer functions of storing and processing data. Such additional elements fail to provide a meaningful limitation on the judicial exception, and amount to no more than mere instructions to apply the exception using generic computer components. Therefore, The additional elements do not integrate the judicial exception into a practical application under step 2A prong two or ensure the claim as a whole amount to significantly more than the judicial exception itself under step 2B. Accordingly, the claim is not patent-eligible under 35 U.S.C. 101.
Allowable Subject Matter
Claims 1-9 would be allowable if rewritten or amended to overcome claim objections and the rejections under 35 U.S.C. 112(b) and 101, set forth in this Office action.
The following is a statement of reasons for the indication of allowable subject matter:
Applicant claims a computer-implemented output method comprising: correcting a vector based on information of a first modal on the basis of a correlation between the vector based on the information of the first modal and a vector based on information of a second modal different from the first modal; correcting the vector based on the information of the second modal on the basis of the correlation between the vector based on the information of the first modal and the vector based on the information of the second modal; generating a first vector on the basis of a correlation of two different types of vectors obtained from the corrected vector based on the information of the first modal; generating a second vector on the basis of the correlation of the two different types of vectors obtained from the corrected vector based on the information of the second modal; generating a third vector in which the first vector and the second vector are aggregated on the basis of the correlation of the two different types of vectors obtained from a combined vector that includes a predetermined vector, the generated first vector, and the generated second vector; and outputting the generated third vector.
The primary reasons for indication of allowable subject matter is the limitations in combination of all limitations, such as the steps of generating a first vector on the basis of a correlation of two different types of vectors obtained from the corrected vector based on the information of the first modal; generating a second vector on the basis of the correlation of the two different types of vectors obtained from the corrected vector based on the information of the second modal; generating a third vector in which the first vector and the second vector are aggregated on the basis of the correlation of the two different types of vectors obtained from a combined vector that includes a predetermined vector, the generated first vector, and the generated second vector; and outputting the generated third vector.
Peng – NPL Multi-modality Latent Interaction Network for Visual Question Answering – discloses a multi-modality Latent Interaction module MLI for Visual Question Answering (VQA) as illustrated in figure 2 having a multiple stages of MLI to generate visual features and question features to provide an answer for a question “what sports are these people playing”, wherein the MLI includes a summarization stage that receives first modal [i.e., visual feature] and second modal [i.e., question features], and provide smaller vectors to be performed in propagation and aggregation. Furthermore, aggregation stage includes generating a first vector and second UR and UE based on key and query vectors of the corresponding vectors, wherein the query QR and query QE are obtained from the original vector R and E, rather than the corrected vectors
R
-
and
E
-
. Furthermore, figure 2 illustrates the last step is to combine visual feature and question feature to generate a final vector according to equation 18, wherein such vector generation is not performed on the basis of the two different types of vectors obtained from a combined vector that includes a predetermined vector, the generated first vector, and the generated second vector. Thus, Peng does not teach the limitation as required in the independent claims.
Lu – Hierarchical Question-Image Co-Attention for Visual Question Answering – discloses a novel co attention model for VQA that jointly reasons about image and question attention as illustrated in figure 1, wherein the question and image corresponding to the multi-modal. Section 3 illustrates the mathematical operations to perform the co-attention mechanism. However, Lu does not teach the steps of generating first and second vector based on the correlation of the two different type of vectors obtained from the corrected vectors based on first and second modal and step of generating a third vector.
Vaswani – NPL Attention Is All You Need (IDS filed 04/13/2025) – discloses a simple network architecture, the Transformation, based solely on attention mechanism. Section 3 illustrates a model architecture of the transformer using stacked self-attention, and section 3.2 also describes a multi-head attention. However, Vaswani does not teach the steps of correcting vectors based on the first and second modals, generating first, second, and third vector as required in the claim.
Nguyen – NPL Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering (IDS filed 04/13/2025) - discloses an approach that utilizes co-attention mechanism that enables bi-directional interactions between the two modalities contribute to boost accuracy of prediction of answer. Given representations of an image and a question, it first generates an attention map on image regions for each question word and an attention map on question words for each image region. It then performs computation of attended features, concatenation of multimodal representations, and their transformation by a single layer network with ReLU and a residual connection. Figure 3 illustrates computation of dense co-attention maps and attended representation of image and question. However, Nguyen does not teach generating first and second vector based on the correlation of the two different type of vectors obtained from the corrected vectors based on first and second modal and step of generating a third vector.
Piergiovanni – US 20240119713 – discloses a machine learned visuo-linguistic model that receives image input and text input, wherein the received data is decoded into a plurality of vision embedding tokens and text embedding tokens, which are used as queries and keys vectors [0037-0040] to generate a plurality of text to vision and vision to text intermediate tokens, which are concatenated to generate a first set of output compound token. The machine-learned visuo-linguistic model can further include one or more self-attention layers configured to perform self-attention on the first set of output compound tokens to generate a second set of output compound tokens. The machine-learned visuo-linguistic model can further include a decoder configured to process the second set of output compound tokens generate a prediction 40 for a visuo-linguistic task. However, Piergiovanni does not teach generating first and second vector based on the correlation of the two different type of vectors obtained from the corrected vectors based on first and second modal and step of generating a third vector.
Therefore, none of the closest found prior art teaches the limitations as required in the independent claims. Accordingly, Claims 1-9 would be allowable if rewritten or amended to overcome claim objections and the rejection under 35 U.S.C. 101, set forth in this Office action.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUY DUONG whose telephone number is (571)272-2764. The examiner can normally be reached Mon-Friday 7:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jyoti Mehta can be reached at (571) 270-3995. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUY DUONG/Examiner, Art Unit 2183 (571)272-2764
/JYOTI MEHTA/Supervisory Patent Examiner, Art Unit 2183