The present application, filed on or after 16 March 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This office action is in response to Applicant’s submission filed on 6 February 2024. THIS ACTION IS NON-FINAL.
Status of Claims
Claims 1-20 are pending.
Claim 1-20 are rejected under 35 U.S.C. 101 for being directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
There is no art rejection for claims 1-20.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Judicial Exception
Claims 1-20 of the claimed invention are directed to a judicial exception, an abstract idea, without significantly more.
Regarding claims 1-10,
(Independent Claims) With regards to claim 1 / 8 / 15,
Step 1: The claim recites a process / machine / article of manufacturing, which falls into one of the statutory categories.
Step 2A – Prong 1: the claim, in part, recites: “… generating … first modality embeddings for data points of the first modality in the training dataset; for each first modality embedding, determining a similarity metric to other first modality embeddings; generating … second modality embeddings for data points of the second modality in the training dataset” (mental process and/or math concept), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting generic computer elements, nothing in the claim element precludes the step from practically being performed in the mind. For example, but for the language about generic computer elements, “generating”, “determining”, in the limitation citied above encompasses observing / analyzing / associating data using certain functions, which is based on observation, evaluation, judgement, and/or opinion, that could be performed by human using paper / pen / calculator. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas; Accordingly, the claim recites an abstract idea.
Step 2A – Prong 2: This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of: (a) “…using a first encoder associated with a first modality of the multiple modalities”, “…using a second encoder associated with a second modality of the multiple modalities …”, “training the second encoder…”, “An electronic device comprising: at least one processing device configured to …“, “A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to …“, which is mere instructions to implement an abstract idea using generic computing device, or merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(f)); (b) “accessing a training dataset comprising multiple samples, each sample comprising a data point for each of multiple modalities”, which is extra-solution activity of pre-solution data gathering (see MPEP.2106.05(g)). Accordingly. the additional elements individually or in combination do not integrate the judicial exception into a practical application. The claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the claim recites the additional elements of (a) “…using a first encoder associated with a first modality of the multiple modalities”, “…using a second encoder associated with a second modality of the multiple modalities …”, “training the second encoder…”, “An electronic device comprising: at least one processing device configured to …“, “A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to …“, which is mere instructions to implement an abstract idea using generic computing device, or merely uses a computer as a tool to perform an abstract idea (see MPEP 2106.05(f)); (b) “accessing a training dataset comprising multiple samples, each sample comprising a data point for each of multiple modalities”, which is extra-solution activity of pre-solution data gathering (see MPEP.2106.05(g)). The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory"). Accordingly, the additional elements individually or in combination do not amount to significantly more than the judicial exception. The claim is not patent eligible.
(Dependent claims)
Claims 2-7 / 8-14 / 6-20 are dependent on claim 1 / 7 / 15 and include all the limitations of claim 1 / 8 / 15. Therefore, claims 2-7 / 8-14 / 6-20 recite the same abstract ideas.
With regards to claim 2 / 9 / 16, the claim recites limitation of “… where the contrastive loss function is expressed as:
PNG
media_image1.png
321
777
media_image1.png
Greyscale
”, as drafted, is a process that, under its broadest reasonable interpretation, covers mathematical concepts but for the recitation of generic computer components. That is, the contrastive loss function in math formula, describe mathematical relationships and algorithms. Mathematical relationship and algorithms have been found by the courts to be abstract ideas, e.g., see MPEP 2106.04(a)(2) A. Mathematical Relationships, iv. organizing information and manipulating information through mathematical correlations, Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014). The patentee in Digitech claimed methods of generating first and second data by taking existing information, manipulating the data using mathematical functions, and organizing this information into a new form. The court explained that such claims were directed to an abstract idea because they described a process of organizing information through mathematical correlations, like Flook's method of calculating using a mathematical formula. 758 F.3d at 1350, 111 USPQ2d at 1721. If a claim limitation, under its broadest reasonable interpretation, covers mathematical relationships, then it falls within the “Mathematical Concepts” grouping of abstract ideas. Except citing generic computer elements to implement the abstract idea, there is no additional element showing integration into a practical application or adding something significantly more to the abstract idea. The claim is not patent eligible.
With regards to claim 3 / 10 / 17, the claim recites limitation of “… where the similarity metric wij is expressed as:
PNG
media_image2.png
150
782
media_image2.png
Greyscale
”, as drafted, is a process that, under its broadest reasonable interpretation, covers mathematical concepts but for the recitation of generic computer components. That is, the similarity metric in math formula, describe mathematical relationships and algorithms. Mathematical relationship and algorithms have been found by the courts to be abstract ideas, e.g., see MPEP 2106.04(a)(2) A. Mathematical Relationships, iv. organizing information and manipulating information through mathematical correlations, Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014). The patentee in Digitech claimed methods of generating first and second data by taking existing information, manipulating the data using mathematical functions, and organizing this information into a new form. The court explained that such claims were directed to an abstract idea because they described a process of organizing information through mathematical correlations, like Flook's method of calculating using a mathematical formula. 758 F.3d at 1350, 111 USPQ2d at 1721. If a claim limitation, under its broadest reasonable interpretation, covers mathematical relationships, then it falls within the “Mathematical Concepts” grouping of abstract ideas. Except citing generic computer elements to implement the abstract idea, there is no additional element showing integration into a practical application or adding something significantly more to the abstract idea. The claim is not patent eligible.
With regards to claim 4 / 11 / 18, the claim recites limitation of “… generating, [using a third encoder associated with a third modality of the multiple modalities, ]third modality embeddings for data points of the third modality in the training dataset … based on the contrastive loss function, wherein the contrastive loss function is weighed using additional similarity metrics determined from the second modality” (mental process and/or math concept), which is further process of data analysis / association, which is based on observation, evaluation, judgement, and/or opinion, that could be performed by human using paper / pen / calculator. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Except citing generic computer elements to implement the abstract idea, there is no additional element showing integration into a practical application or adding something significantly more to the abstract idea. The claim is not patent eligible.
The claim recites additional element of “using a third encoder associated with a third modality of the multiple modalities”, “training the third encoder…”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the additional element of “using a third encoder associated with a third modality of the multiple modalities”, “training the third encoder…”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2B the additional elements individually or in combination do not amount to significantly more than the judicial exception.
With regards to claim 5 / 12 / 19, the claim recites additional element of “wherein the first encoder is pre-trained…”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the additional element of “wherein the first encoder is pre-trained …”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2B the additional elements individually or in combination do not amount to significantly more than the judicial exception.
With regards to claim 6 / 13 / 20, the claim recites additional element of “wherein the first encoder and the second encoder are encoders of a multimodal machine learning model…”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the additional element of “wherein the first encoder and the second encoder are encoders of a multimodal machine learning model…”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2B the additional elements individually or in combination do not amount to significantly more than the judicial exception.
With regards to claim 7 / 14, the claim recites additional element of “wherein each of the multiple modalities comprises one of: video, images, audio, text, and sensor data…”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the additional element of “wherein each of the multiple modalities comprises one of: video, images, audio, text, and sensor data…”, which is merely using computer as a tool to perform an abstract idea (see MPEP 2106.05(f)). Accordingly, at Step 2B the additional elements individually or in combination do not amount to significantly more than the judicial exception.
Allowable Subject Matter
Claims 1-20 include allowable subject matter since when reading the claims in light of the specification, as per, MPEP §2111.01 or Toro Co. v. White Consolidated Industries Inc., 199F.3d 1295, 1301, 53 USPQ2d 1065, 1069, 1069 (Fed.Cir. 1999), none of the references of record alone or in combination disclose or suggest the combination of limitations specified in claims 1-20.
In interpreting the claims, in light of the specification filed on 6 February 2024, the Examiner finds the claimed invention to be patentably distinct from the prior arts of record.
Regarding the amended independent claims, the primary reason for the allowance is the inclusion of the specific process / structure of multi-modal association / embedding using multiple encoders based on a contrastive loss function.
None of the cited prior art references, singly or in combination, fully teaches all limitations of independent claims 1, 8 and 15.
Regarding the dependent claims, which include all the limitations of the independent claims, are also allowed.
The followings are references close to the invention claimed:
Russell et al., US-PGPUB NO.20240386048A1 [hereafter Russell] teaches using data encoder wit similarity metric. However Russell does not teach the specific process / structure of multi-modal association / embedding using multiple encoders based on a contrastive loss function.
Harikumar et al., US-PGPUB NO.20240346629A1 [hereafter Harikumar] teaches multi-modal embedding via training multi-modal encoders. However Harikumar does not teach the specific process / structure of multi-modal association / embedding using multiple encoders based on a contrastive loss function.
Wu et al., US-PGPUB NO.20240362269A1 [hereafter Wu] teaches cross modal data access with query embedding. However Wu does not teach the specific process / structure of multi-modal association / embedding using multiple encoders based on a contrastive loss function.
He et al., US-PGPUB NO.20220284246A1 [hereafter He] teaches cross modal retrieval using ML models. However He does not teach the specific process / structure of multi-modal association / embedding using multiple encoders based on a contrastive loss function.
Bronstein et al., “Data fusion through cross-modality metric learning using similarity-sensitive hashing”, 2010 IEEE computer society conference on computer vision and pattern recognition, 2010 [hereafter Bronstein] teaches cross modality data association with similarity metric. However Bronstein does not teach the specific process / structure of multi-modal association / embedding using multiple encoders based on a contrastive loss function.
Zolfaghari et al., “CrossCLR: Cross-modal contrastive learning for multi-modal video representations”, ICCV 2021 [hereafter Zolfaghari] teaches learning with contrastive metric for video representation. However Zolfaghari does not teach the specific process / structure of multi-modal association / embedding using multiple encoders based on a contrastive loss function.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TSU-CHANG LEE whose telephone number is 571-272-3567. The fax number is 571-273-3567.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas, can be reached 571-272-2589.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TSU-CHANG LEE/
Primary Examiner, Art Unit 2128
Prosecution Insights