DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is in response to the application filed on 10/27/2023. Claims 1-20 are pending in the application and have been examined.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/26/2024 and 09/12/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Step 1 Analysis: Claim 1 is directed to a process, which falls within one of the four statutory categories.
Step 2A Prong 1 Analysis: Claim 1 recites, in part, the limitation of:
selecting a first target sample set from the first sample set and the second sample set according to a predetermined rule
determining a first set of samples based on the first target sample set according to a predefined batch size;
These limitations as drafted, are processes that, under broadest reasonable interpretation, covers performance of the limitation in the mind or with the aid of pen and paper which falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2 Analysis: This judicial exception is not integrated into a practical application. The claim recites “training the contrastive learning model using the first set of samples” as an additional element. This element in the claim is recited at a high level of generality such that they amount to no more than mere instructions to implement an abstract idea or other exception on a computer. Accordingly, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea. See MPEP 2106.05(f).
The claim further recites:
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set;
The limitation is a mere data gathering step and thus are insignificant extra-solution activities. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As mentioned above with respect to integration of the abstract idea into a practical application, the additional elements of contrastive learning model to perform the steps of the claimed process amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Furthermore, the limitation of
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set;
is well-understood, routine, and conventional, as evidenced by MPEP 2106.05(d)(II)(I), “receiving or transmitting data over a network”. These limitations, therefore, remain insignificant extra-solution activity even upon reconsideration, and do not amount to significantly more. Even when considered in combination, these additional elements amount to mere instructions to apply the exception of using generic computer components and insignificant extra-solution activity, which cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 2,
The rejection of Claim 1 is further incorporated, and further, the claim recites: wherein the predetermined rule comprises any of: a random selection rule, a polling selection rule, and a selection rule based on sample amount.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 3,
The rejection of Claim 1 is further incorporated, and further, the claim recites: wherein selecting the target sample set according to the selection rule based on a sample amount comprises: determining a first weight for the first sample set and a second weight for the second sample set based on a first sample amount of the first sample set and a second sample amount of the second sample set, respectively; and selecting the target sample set based on the first weight and the second weight.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 4,
The rejection of Claim 3 is further incorporated, and further, the claim recites: wherein selecting the target sample set based on the first weight and the second weight comprises: selecting the target sample set from the first sample set and the second sample set based on a distribution function associated with the first weight and the second weight.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 5,
The rejection of Claim 1 is further incorporated, and further, the claim recites: wherein a sample in the first sample set and the second sample set comprises: data of a first modality, data of a second modality, and a label representing an association relationship between the data of the first modality and the data of the second modality.
This limitation amounts to generally linking the judicial exception to a field of use or technological environment. Please see MPEP 2106.05(h). The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 6,
The rejection of Claim 5 is further incorporated, and further, the claim recites: wherein determining the first set of samples from the first target sample set comprises: selecting a positive sample from the first target sample set, a label in the positive sample indicating that there is an association relationship between the data of the first modality and the data of the second modality in the positive sample; and generating a negative sample based on the data of the first modality in the positive sample and the data of the second modality in the first target sample set; and generating the first set of samples based on the positive sample and the negative sample.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 7,
The rejection of Claim 6 is further incorporated, and further, the claim recites: selecting a further data of the second modality from a data space of the second modality in the first target sample set; and generating the negative sample based on the data of the first modality in the positive sample and further data of the second modality, a label in the negative sample indicating that there is no association relationship between the data of the first modality and the further data of the second modality in the negative sample.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 8,
The rejection of Claim 6 is further incorporated, and further, the claim recites: wherein training the contrastive learning model comprises: with respect to the positive sample in the first set of samples, determining, using the contrastive learning model, a first feature for the data of the first modality and a second feature for the data of the second modality in the positive sample, respectively; determining a loss function for the contrastive learning model based on a difference between the first feature and the second feature; and updating the contrastive learning model in a direction for reducing the loss function.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 9,
The rejection of claim 8 is further incorporated, and further recites: wherein determining the first feature and the second feature comprises: “determining, the first feature and the second feature”
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements: “using a first encoder and a second encoder in the contrastive learning model respectively, the first encoder describing an association relationship between data of the first modality and a feature for the data of the first modality, and the second encoder describing an association relationship between data of the second modality and a feature for the data of the second modality.” These elements in the claim are recited at a high level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component. Please see MPEP 2106.05(f). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is not patent eligible.
Regarding Claim 10,
The rejection of Claim 6 is further incorporated, and further, the claim recites: wherein training of the contrastive learning model comprises: with respect to the negative sample in the first set of samples, determining, using the contrastive learning model, a first feature for the data of the first modality and a second feature for the data of the second modality in the negative sample, respectively; determining a loss function of the contrastive learning model based on a difference between the first feature and the second feature; and updating the contrastive learning model in a direction for increasing the loss function.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 11,
The rejection of Claim 5 is further incorporated, and further, the claim recites: wherein the first modality comprises any of a plurality of modalities: image, text, video, audio, and the second modality comprises a further one of the plurality of modalities.
This limitation amounts to generally linking the judicial exception to a field of use or technological environment. Please see MPEP 2106.05(h). The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding Claim 12,
The rejection of Claim 5 is further incorporated, and further, the claim recites: selecting a second target sample set from the first sample set and the second sample set; determining a second set of samples based on the second target sample set according to a predefined batch size.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements: “and training the contrastive learning model using the second set of samples”. This element in the claim is recited at a high level of generality such that they amount to no more than mere instructions to implement an abstract idea or other exception on a computer. Accordingly, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea. Please see MPEP 2106.05(f). Thus, the claim is not patent eligible.
Regarding Claim 13,
The rejection of Claim 12 is further incorporated, and further, the claim recites: wherein determining the second set of samples comprises: selecting a positive sample from unused positive samples in the second set of samples; generating a negative sample based on the positive sample and data of the second modality in the second target sample set; and determining the second set of samples based on the positive sample and negative sample.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 14,
The rejection of Claim 12 is further incorporated, and further, the claim recites: wherein selecting the first target sample set is independent of selecting the second target sample set, and the first target sample set is different from the second target sample set.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the judicial exceptions. The claim is not patent eligible.
Regarding Claim 15,
The rejection of Claim 1 is further incorporated, and further, the claim recites: determining an association relationship between data in a sample.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 1, thus it recites a judicial exception.
This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements: “that is to be processed using the trained contrastive learning model”. This element in the claim is recited at a high level of generality such that they amount to no more than mere instructions to implement an abstract idea or other exception on a computer. Accordingly, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea. Please see MPEP 2106.05(f). Thus, the claim is not patent eligible.
Regarding Claim 16,
Step 1 Analysis: Claim 16 is directed to a process, which falls within one of the four statutory categories.
Step 2A Prong 1 Analysis: Claim 16 recites, in part, the limitation of:
selecting a first target sample set from the first sample set and the second sample set according to a predetermined rule
determining a first set of samples based on the first target sample set according to a predefined batch size;
These limitations as drafted, are processes that, under broadest reasonable interpretation, covers performance of the limitation in the mind or with the aid of pen and paper which falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2 Analysis: This judicial exception is not integrated into a practical application. The claim recites “at least one processing unit”, “and at least one memory coupled to the at least one processing unit”, “and storing instructions to be executed by the at least one processing unit”, “training the contrastive learning model using the first set of samples” as additional elements. These elements in the claim is recited at a high level of generality such that they amount to no more than mere instructions to implement an abstract idea or other exception on a computer. Accordingly, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea. See MPEP 2106.05(f).
The claim further recites:
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set;
The limitation is a mere data gathering step and thus are insignificant extra-solution activities. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As mentioned above with respect to integration of the abstract idea into a practical application, the additional elements of utilizing a processing unit and memory to perform the steps of the claimed process amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Furthermore, the limitation of
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set;
is well-understood, routine, and conventional, as evidenced by MPEP 2106.05(d)(II)(I), “receiving or transmitting data over a network”. These limitations, therefore, remain insignificant extra-solution activity even upon reconsideration, and do not amount to significantly more. Even when considered in combination, these additional elements amount to mere instructions to apply the exception of using generic computer components and insignificant extra-solution activity, which cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 17,
The rejection of Claim 16 is further incorporated, and further, the claim recites: wherein the predetermined rule comprises any of: a random selection rule, a polling selection rule, and a selection rule based on sample amount.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 16, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 18,
The rejection of Claim 16 is further incorporated, and further, the claim recites: wherein selecting the target sample set according to the selection rule based on a sample amount comprises: determining a first weight for the first sample set and a second weight for the second sample set based on a first sample amount of the first sample set and a second sample amount of the second sample set, respectively; and selecting the target sample set based on the first weight and the second weight.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 16, thus it recites a judicial exception.
The claim does not include any additional elements that amount to integration of the judicial exceptions into a practical application, nor to significantly more than the
judicial exceptions. The claim is not patent eligible.
Regarding Claim 19,
The rejection of Claim 16 is further incorporated, and further, the claim recites: determining an association relationship between data in a sample.
The claim recites additional mental steps in addition to the judicial exception identified in the rejection of claim 16, thus it recites a judicial exception.
This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements: “that is to be processed using the trained contrastive learning model”. This element in the claim is recited at a high level of generality such that they amount to no more than mere instructions to implement an abstract idea or other exception on a computer. Accordingly, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea. Please see MPEP 2106.05(f). Thus, the claim is not patent eligible.
Regarding Claim 20,
Step 1 Analysis: Claim 20 is directed to a process, which falls within one of the four statutory categories.
Step 2A Prong 1 Analysis: Claim 20 recites, in part, the limitation of:
selecting a first target sample set from the first sample set and the second sample set according to a predetermined rule
determining a first set of samples based on the first target sample set according to a predefined batch size;
These limitations as drafted, are processes that, under broadest reasonable interpretation, covers performance of the limitation in the mind or with the aid of pen and paper which falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2 Analysis: This judicial exception is not integrated into a practical application. The claim recites “a non-transient computer-readable storage medium storing a computer program”, and “training the contrastive learning model using the first set of samples” as an additional element. This element in the claim is recited at a high level of generality such that they amount to no more than mere instructions to implement an abstract idea or other exception on a computer. Accordingly, these additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limits on practicing the abstract idea. See MPEP 2106.05(f).
The claim further recites:
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set;
The limitation is a mere data gathering step and thus are insignificant extra-solution activities. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As mentioned above with respect to integration of the abstract idea into a practical application, the additional elements of utilizing the computer-readable storage medium which stores the computer program to perform the steps of the claimed process amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Furthermore, the limitation of
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set;
is well-understood, routine, and conventional, as evidenced by MPEP 2106.05(d)(II)(I), “receiving or transmitting data over a network”. These limitations, therefore, remain insignificant extra-solution activity even upon reconsideration, and do not amount to significantly more. Even when considered in combination, these additional elements amount to mere instructions to apply the exception of using generic computer components and insignificant extra-solution activity, which cannot provide an inventive concept. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1,5,11 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. “Unified Contrastive Learning in Image-Text-Label Space”(herein after Yang (1)), in view of Freitag et al. “Complete Multilingual Neural Machine Translation”
Regarding Claim 1,
Yang (1) discloses,
A method for training a contrastive learning model, (Yang (1), pg.1, abstract, “In this space, we propose a new learning paradigm, called Unified Contrastive Learning (UniCL) with a single learning objective to seamlessly prompt the synergy of two data types”)
comprising: obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set; (Yang(1), pg. 6, section 4.2.1, “We use ImageNet-1K as the base dataset, and gradually add different sets of image-text pairs, including GCC-3M, GCC 15M and YFCC-14M” showing that the training process uses multiple sets of sample data)
and training the contrastive learning model using the first set of samples. (Yang(1), pg.1, abstract “In this space, we propose a new learning paradigm, called Unified Contrastive Learning (UniCL) with a single learning objective to seamlessly prompt the synergy of two data types.” and pg. 6, section 4.2.1 “When combining with image-text pairs, we use a balanced data sampler to ensure that the model is trained with the same number of image-label and image-text pairs per epoch. All models are trained with 500 epochs.”)
However, Yang (1) fails to disclose target sample selection from the different set of samples according to a selection rule and determining a first set of samples from the target set according to a batch size.
Freitag, in the same field of endeavor (training a model using input mapping techniques), teaches,
selecting a first target sample set from the first sample set and the second sample set according to a predetermined rule; (Freitag, pg.553, section 3, “To alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels: i) we choose a target language (based on a temperature-based schedule), ii) uniformly sample a source language. Formally, for a given target language l, let D(l) be the size of the available training examples with target language l, the sample probability with a temperature T is defined as…” stating that the predetermined rule here is the probability sampling.)”
determining a first set of samples based on the first target sample set according to a predefined batch size; (Freitag, pg. 553, section 3, “During training, the scheduler samples a batch of training examples based on the target language only, as opposed to source-target language pair specific sampling” and “All multilingual models are trained for 500,000 updates using an average batch size of around 33,000 sentences” showing the batch size)
Yang (1) and Freitag are both considered to be analogous to the claimed invention because they are in the same field of using input mapping techniques to make a model efficient. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1) and Freitag to select a target sample from the different set of samples according to a selection rule and determining a first set of samples from the target set according to a batch size. The motivation to do so is to “alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels” stating that sampling strategy is used to prevent over-representation/bias towards a specific sample set. (Freitag, pg.553, section 3)
Regarding Claim 5,
the Yang (1)/Freitag combination of claim 1 teaches,
The method according to claim 1 (thus the rejection of claim 1 is incorporated),
Yang (1) further teaches,
wherein a sample in the first sample set and the second sample set comprises: data of a first modality, data of a second modality, and a label representing an association relationship between the data of the first modality and the data of the second modality. (Yang(1), pg. 3, section 3.1, “ We define a triplet-wise data format s =
PNG
media_image1.png
23
154
media_image1.png
Greyscale
,where x ∈ X is the image, and t ∈ T is its corresponding language description (ranging from simple tokens such as category names to free-form text sequences), and y ∈ Y is a label indicating the index of the grouped or unique language description in the dataset”).
Regarding Claim 11,
the Yang (1)/Freitag combination of claim 5 teaches,
The method according to claim 5 (thus the rejection of claim 5 is incorporated),
Yang (1) further teaches,
wherein the first modality comprises any of a plurality of modalities: image, text, video, audio, and the second modality comprises a further one of the plurality of modalities. (Yang(1), pg. 3, section 3.1, “ We define a triplet-wise data format s =
PNG
media_image1.png
23
154
media_image1.png
Greyscale
,where x ∈ X is the image, and t ∈ T is its corresponding language description (ranging from simple tokens such as category names to free-form text sequences” showing that the first modality is the image and the second modality is the text description. )
Regarding Claim 15,
the Yang (1)/Freitag combination of claim 1 teaches,
The method according to claim 1, further comprising: (thus the rejection of claim 1 is incorporated),
Yang (1) further teaches,
determining an association relationship between data in a sample that is to be processed using the trained contrastive learning model. (Yang (1), pg. 3, section 3.2, “For i-th image xi and j-th language description tj in a batch B.…. Their similarity is calculated as
PNG
media_image2.png
45
164
media_image2.png
Greyscale
” showing that the model determines the association by computing similarity between their learned features.)
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Yang (1), in view of
Freitag, and further in view of Wu et al, "Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval"
Regarding Claim 6,
The combination of Yang (1)/Freitag discloses,
The method according to claim 5, wherein determining the first set of samples from the first target sample set comprises: (and thus the rejection of claim 5 is incorporated).
Yang (1)/Freitag combination fails to teach the positive and negative sample and a label in the positive sample showing relationship between the modalities and generating first set of samples based on the positive and negative samples.
Wu, in the same field of endeavor as Yang (1) and Freitag (improving a learning paradigm using input mapping techniques), teaches,
selecting a positive sample from the first target sample set, a label in the positive sample indicating that there is an association relationship between the data of the first modality and the data of the second modality in the positive sample; (Wu, pg. 5, section 3, “The two types of distance are also used to train a multi-task to classify positive and negative text-video sample pairs to establish a semantic alignment relationship between different modalities” suggesting the presence of positive sample and pg.7, section 3.3, “Here, semantic consistency means that the video and caption match with each other, in other words, it is the text-video positive sample pair during the training process.” And pg.8, section 3.3.2, “we record the text-video positive sample pair label as 1, otherwise negative as 0, and ˆy is the probability that the model predicts the sample is a positive example.”)
and generating a negative sample based on the data of the first modality in the positive sample and the data of the second modality in the first target sample set; (Wu, pg. 7, section 3.3.1, “In Formulas (13) and (14), (v,s) represents the text-video positive sample pair, and s− and v−, respectively, represent the negative text sample matching v and the negative video sample of matching s, so as to form (v,s−) and (v−,s), two types of negative sample pair.” showing multiple negative sample pairs)
and generating the first set of samples based on the positive sample and the negative sample. (Wu, pg.7, section 3.3, “Here, semantic consistency means that the video and caption match with each other, in other words, it is the text-video positive sample pair during the training process. On the contrary, the semantic inconsistency indicates the text-video negative sample pair”)
Yang (1), Freitag and Wu are all considered to be analogous to the claimed invention because they are in the same field of using input mapping techniques to make a learning paradigm efficient. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Wu to generate a positive and negative sample and a label in the positive sample showing relationship between the modalities and generate a first set of samples based on the positive and negative samples. The motivation to do so is to “establish a semantic alignment relationship between different modalities” (Wu, pg.5, section 3)
Regarding Claim 7,
The combination of Yang (1)/Freitag discloses,
The method according to claim 6, generating the negative sample comprises: (and thus the rejection of claim 6 is incorporated).
Yang (1)/Freitag combination fails to teach the further data of the second modality and generating the negative sample based on the further data and first modality in the positive sample.
However, Wu teaches,
The method according to claim 6, wherein generating the negative sample comprises:
selecting a further data of the second modality from a data space of the second modality in the first target sample set; (Wu, pg. 7, section 3.3.1, “In Formulas (13) and (14), (v,s) represents the text-video positive sample pair, and s− and v−, respectively, represent the negative text sample matching v and the negative video sample of matching s, so as to form (v,s−) and (v−,s), two types of negative sample pair.” showing multiple negative sample pairs)
and generating the negative sample based on the data of the first modality in the positive sample and further data of the second modality, (Wu, pg.7, section 3.3.1, ““In Formulas (13) and (14), (v,s) represents the text-video positive sample pair, and s− and v−, respectively, represent the negative text sample matching v and the negative video sample of matching s, so as to form (v,s−) and (v−,s), two types of negative sample pair.” showing that the negative pair is based on the first and further data of the second modality in the positive sample (v,s) )
a label in the negative sample indicating that there is no association relationship between the data of the first modality and the further data of the second modality in the negative sample. (Wu, pg.7, section 3.3, “Here, semantic consistency means that the video and caption match with each other, in other words, it is the text-video positive sample pair during the training process”. On the contrary, the semantic inconsistency indicates the text-video negative sample pair.”)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Wu to generate further data of the second modality and generate the negative sample based on the further data and first modality in the positive sample. The motivation to do so is to “establish a semantic alignment relationship between different modalities” (Wu, pg.5, section 3)
Claims 8, 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Yang (1), in view of Freitag, further in view of Wu, and further in view of Babaev et al “Event Sequence Metric Learning”.
Regarding Claim 8,
The combination of Yang (1)/Freitag/Wu discloses,
The method according to claim 6, wherein training the contrastive learning model comprises: (and thus the rejection of claim 6 is incorporated).
Yang (1) further teaches,
with respect to the positive sample in the first set of samples, determining, using the contrastive learning model, a first feature for the data of the first modality and a second feature for the data of the second modality in the positive sample, respectively; (Yang (1), pg.3, section 3.2, “For each image x, an image encoder model fθ parameterized by θ first represents x as a visual feature vector
PNG
media_image3.png
28
235
media_image3.png
Greyscale
.For each language description t ∈ T, we encode it with a text encoder fφ(t) parameterized by φ to get its feature vector
PNG
media_image4.png
19
233
media_image4.png
Greyscale
“ )
and updating the contrastive learning model in the direction for reducing the loss function. (Yang (1), pg.5, section 3.4, algorithm 1,
[AltContent: textbox (The bidirectional loss here implies that the loss function is reducing in one direction and increasing in other direction.)][AltContent: textbox (Loss function )][AltContent: rect][AltContent: rect]
PNG
media_image5.png
741
614
media_image5.png
Greyscale
)
Yang (1)/Freitag/Wu combination fails to teach the loss function based on the difference between the features.
However, Babaev, in the same field of endeavor (using input mapping techniques to make a learning paradigm efficient), teaches,
determining a loss function for the contrastive learning model based on a difference between the first feature and the second feature; (Babaev, pg.4, section 3.4, “Several metric learning losses have been considered-contrastive loss [8], binomial deviance loss [37], triplet loss [9], histogram loss [27] and margin loss [34].”and “
PNG
media_image6.png
31
41
media_image6.png
Greyscale
is a distance function between a i-th labeled sample pair of embeddings X1 and X2, Y is a binary label assigned to a pair: Y = 0 means a similar pair, Y = 1 means dissimilar pair, m > 0 is a margin.” showing the distance/difference function between the embeddings/feature X1 and X2)
Yang (1), Freitag, Wu and Babaev are all considered to be analogous to the claimed invention because they are in the same field of using input mapping techniques to make a learning paradigm efficient. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, Wu and Babaev to teach the loss function based on the difference between the features. The motivation to do so is to “penalize the model only if the negative pair is not distant enough and the distance between embeddings is less than a margin” (Babaev, pg.4, section 3.4)
Regarding Claim 9,
The combination of Yang (1)/Freitag/Wu discloses,
The method according to claim 8, wherein determining the first feature and the second feature comprises: (and thus the rejection of claim 8 is incorporated).
Yang (1) further teaches,
determining, using a first encoder and a second encoder in the contrastive learning model, the first feature and the second feature, respectively, (Yang(1), pg. 3, section 3.2, “For each image x, an image encoder model fθ parameterized by θ first represents x as a visual feature vector
PNG
media_image3.png
28
235
media_image3.png
Greyscale
.For each language description t ∈ T, we encode it with a text encoder fφ(t) parameterized by φ to get its feature vector
PNG
media_image4.png
19
233
media_image4.png
Greyscale
“ )
the first encoder describing an association relationship between data of the first modality and a feature for the data of the first modality, and the second encoder describing an association relationship between data of the second modality and a feature for the data of the second modality.
(Yang(1), pg.3,section 3.2, “For each image x, an image encoder model fθ parameterized by θ first represents x as a visual feature vector
PNG
media_image3.png
28
235
media_image3.png
Greyscale
.For each language description t ∈ T, we encode it with a text encoder fφ(t) parameterized by φ to get its feature vector
PNG
media_image4.png
19
233
media_image4.png
Greyscale
” where the visual encoder maps image x to its visual feature representation and the text encoder maps language description t into its textual feature.)
Regarding Claim 10,
The combination of Yang (1)/Freitag/Wu discloses,
The method according to claim 6, wherein training the contrastive learning model comprises: (and thus the rejection of claim 6 is incorporated).
Yang (1) further teaches,
with respect to the negative sample in the first set of samples, determining, using the contrastive learning model, a first feature for the data of the first modality and a second feature for the data of the second modality in the negative sample, respectively (Yang(1), pg.3, section 3.2, “For each image x, an image encoder model fθ parameterized by θ first represents x as a visual feature vector
PNG
media_image3.png
28
235
media_image3.png
Greyscale
.For each language description t ∈ T, we encode it with a text encoder fφ(t) parameterized by φ to get its feature vector
PNG
media_image4.png
19
233
media_image4.png
Greyscale
“ and figure 2,
[AltContent: rect][AltContent: rect][AltContent: textbox (positive)][AltContent: textbox (positive)][AltContent: rect][AltContent: textbox (Negative )][AltContent: rect][AltContent: textbox (Negative )]
PNG
media_image7.png
820
872
media_image7.png
Greyscale
where each image and language description on which the encoders work consists of positive and negative samples.)
and updating the contrastive learning model in a direction for increasing the loss function (Yang (1), pg.5, section 3.4, algorithm 1,
[AltContent: textbox (The bidirectional loss here implies that the loss function is reducing in one direction and increasing in other direction.)][AltContent: textbox (Loss function )][AltContent: rect][AltContent: rect]
PNG
media_image5.png
741
614
media_image5.png
Greyscale
)
Yang (1)/Freitag/Wu combination fails to teach the loss function based on the difference between the features.
However, Babaev teaches,
determining a loss function of the contrastive learning model based on a difference between the first feature and the second feature; (Babaev, pg.4, section 3.4, “Several metric learning losses have been considered-contrastive loss [8], binomial deviance loss [37], triplet loss [9], histogram loss [27] and margin loss [34].”and “
PNG
media_image6.png
31
41
media_image6.png
Greyscale
is a distance function between a i-th labeled sample pair of embeddings X1 and X2, Y is a binary label assigned to a pair: Y = 0 means a similar pair, Y = 1 means dissimilar pair, m > 0 is a margin.” showing the distance/difference function between the embeddings/feature X1 and X2)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, Wu and Babaev to teach the loss function based on the difference between the features. The motivation to do so is to “penalize the model only if the negative pair is not distant enough and the distance between embeddings is less than a margin” (Babaev, pg.4, section 3.4)
Claims 2, 3, 4, 12,14,16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yang (1), in view of Freitag, and further in view of Yang (U.S. PG Pub 2023/0129870) (hereinafter Yang(2)).
Regarding Claim 2,
the Yang (1)/Freitag combination of claim 1 teaches,
The method according to claim 1 (thus the rejection of claim 1 is incorporated),
The combination of Yang (1)/Freitag fails to teach the predetermined rule for selecting target sample set.
However, Yang (2), in the same field of endeavor (training models using input mapping techniques), teaches,
wherein the predetermined rule comprises any of: a random selection rule, a polling selection rule, and a selection rule based on a sample amount. (Yang (2), paragraph 47, “Specifically, a first sample set is selected from a global sample set including a plurality of sample sets as positive samples”. The citation here corresponds to a selection rule based on a sample amount as it discloses selecting first sample set from a global sample set showing that the selection is based on the number of samples available in the global sample set.)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Yang (2) to select a target sample from the different set of samples according to a selection rule. The motivation to do so is to prevent over-representation/bias towards a specific sample set.
Regarding Claim 3,
the Yang (1)/Freitag combination of claim 1 teaches,
The method according to claim 1(thus the rejection of claim 1 is incorporated),
Yang (1) fails to teach determining the sample weights and selecting sample set based on the weights.
However, Freitag teaches,
wherein selecting the target sample set according to the selection rule based on a sample amount comprises (Freitag, pg.553, section 3, “Formally, for a given target language l, let D(l) be the size of the available training examples with target language l, the sample probability with a temperature T is defined as…” stating that the predetermined rule here is the probability sampling.)
and selecting the target sample set based on the first weight and the second weight. (Freitag, pg. 553, section 3, “we propose a hierarchical sampling strategy with two levels: i) we choose a target language (based on a temperature-based schedule),” where the target data group is chosen using the weights defined from the dataset.)
The combination of Yang (1)/ Freitag fails to disclose determining the weights for the sample sets.
Yang (2), in the same field of endeavor, teaches,
determining a first weight for the first sample set and a second weight for the second sample set based on a first sample amount of the first sample set and a second sample amount of the second sample set, respectively; (Yang(2), paragraph 62, “ According to the definition of formula (2), item vol(A) represents the size of A, and item cut(A, Ā) represents a sum of weights between the data samples of the positive samples and the negative samples.” suggesting the presence of weights for each of the data samples.)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Yang (2) to determine the sample weights and select sample set based on the weights. The motivation to do so is to “alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels” stating that sample weights is used to prevent over-representation of a specific sample set. (Freitag, pg.553, section 3)
Regarding Claim 4,
the Yang (1)/Freitag combination of claim 3 teaches,
The method according to claim 3, wherein selecting the target sample set based on the first weight and the second weight comprises: (thus the rejection of claim 3 is incorporated),
Yang (1) fails to teach the distribution function for selecting the target sample set.
However, Freitag teaches,
selecting the target sample set from the first sample set and the second sample set based on a distribution function (Freitag, pg.553, section 3, “Formally, for a given target language l, let D(l) be the size of the available training examples with target language l, the sample probability with a temperature T is defined as
PNG
media_image8.png
92
219
media_image8.png
Greyscale
” where the Pl values from the sampling distribution used for selection.)
The combination of Yang (1)/Freitag fails to teach the weights associated with it.
Yang (2), in the same field of endeavor, teaches,
associated with the first weight and the second weight. (Yang (2), paragraph 62, “According to the definition of formula (2), item vol(A) represents the size of A, and item cut(A, Ā) represents a sum of weights between the data samples of the positive samples and the negative samples” where weights suggests the presence of multiple weights in the data sample.)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, and Yang (2) to use the distribution function for selecting the target sample set. The motivation to do so is to “alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels” stating that distribution function is ultimately used to prevent over-representation/bias of a specific sample set. (Freitag, pg.553, section 3)
Regarding Claim 12,
The combination of Yang (1)/Freitag discloses,
The method according to claim 5 further comprising (and thus the rejection of claim 5 is incorporated).
The combination of Yang (1)/Freitag fails to disclose the second sample target set, determine second set of samples based on the set and train the learning model based on it.
Yang (2), in the same field of endeavor, teaches,
selecting a second target sample set from the first sample set and the second sample set; (Yang (2), paragraph 47, “Specifically, a first sample set is selected from a global sample set including a plurality of sample sets as positive samples, a second sample set is selected therefrom as negative samples, and the positive samples and the negative samples are used as a piece of training data to generate a corresponding local similarity matrix. A third sample set is selected as positive samples, a fourth sample set is selected as negative samples to form the next piece of training data, another local similarity matrix is obtained, and so on”)
determining a second set of samples based on the second target sample set according to a predefined batch size; (Yang (2), paragraph 68, “According to embodiments of the present disclosure, in the training process, the training data may be divided into mini batches, each mini batch including a plurality of pieces of training data. A loss of a mini batch is obtained by accumulating losses generated by each piece of training data. That is, the first loss is generated from the first sample set and the second sample set, the second loss is generated from the third sample set and the fourth sample set, and so on. These losses may be accumulated to update the model.”)
and training the contrastive learning model using the second set of samples. (Yang (2), paragraph 68, “According to embodiments of the present disclosure, in the training process, the training data may be divided into mini batches, each mini batch including a plurality of pieces of training data.”
Yang (1), Freitag, and Yang (2) are all considered to be analogous to the claimed invention because they are in the same field of training models using input mapping techniques. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, and Yang (2) to disclose the second sample target set, determine second set of samples based on the set and train the learning model based on it. The motivation to do so is to train the model more efficiently using various sets of samples to limit bias.
Regarding Claim 14,
The combination of Yang (1)/Freitag/Yang (2) discloses,
The method according to claim 12 further comprising (and thus the rejection of claim 12 is incorporated).
The combination of Yang (1)/Freitag fails to teach the difference between the first target sample set and second target sample set.
However, Yang (2) further teaches,
wherein selecting the first target sample set is independent of selecting the second target sample set, and the first target sample set is different from the second target sample set. (Yang (2), paragraph 005, “wherein the first sample set is of a first category, and the second sample set is of a second category different from the first category”)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, and Yang (2) to teach the difference between the first target sample set and second target sample set. The motivation to do it is to have different features to prevent bias. “It should be noted that the purpose of contrastive learning training is that trained model 120 may generate similar features from samples belonging to the same category and generate features with large differences from samples belonging to different categories (Yang (2), paragraph 102)
Regarding Claim 16,
Yang (1) teaches,
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set; (Yang(1), pg. 6, section 4.2.1, “We use ImageNet-1K as the base dataset, and gradually add different sets of image-text pairs, including GCC-3M, GCC 15M and YFCC-14M” showing that the training process uses multiple sets of sample data)
and training the contrastive learning model using the first set of samples. (Yang(1), pg.1, abstract “In this space, we propose a new learning paradigm, called Unified Contrastive Learning (UniCL) with a single learning objective to seamlessly prompt the synergy of two data types.” and pg. 6, section 4.2.1 “When combining with image-text pairs, we use a balanced data sampler to ensure that the model is trained with the same number of image-label and image-text pairs per epoch. All models are trained with 500 epochs.”)
Yang (1) fails to teach the processing unit, memory, storing instructions which is executed by the processing unit causing the device to perform a method.
However, Yang (2) discloses,
An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions to be executed by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform a method, the method comprising: (Yang(2), paragraph 007, “According to a third aspect of the present disclosure, an electronic device is provided, including: at least one processing unit; and at least one memory that is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, wherein the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method”)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, and Yang (2) to have a processing unit, memory, storing instructions which is executed by the processing unit causing the device to perform a method. The motivation to do so is “to perform the training method” (Yang (2), paragraph 007)
The combination of Yang (1)/Yang (2) also fails to disclose target sample selection from the different set of samples according to a selection rule and determining a first set of samples from the target set according to a batch size.
However, Freitag, teaches,
selecting a first target sample set from the first sample set and the second sample set according to a predetermined rule; (Freitag, pg.553, section 3, “To alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels: i) we choose a target language (based on a temperature-based schedule), ii) uniformly sample a source language. Formally, for a given target language l, let D(l) be the size of the available training examples with target language l, the sample probability with a temperature T is defined as…” stating that the predetermined rule here is the probability sampling.)”
determining a first set of samples based on the first target sample set according to a predefined batch size; (Freitag, pg. 553, section 3, “During training, the scheduler samples a batch of training examples based on the target language only, as opposed to source-target language pair specific sampling” and “All multilingual models are trained for 500,000 updates using an average batch size of around 33,000 sentences” showing the batch size)
Yang (1), Freitag and Yang (2) are all considered to be analogous to the claimed invention because they are in the same field of using input mapping techniques to make a model efficient. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Yang(2) to select a target sample from the different set of samples according to a selection rule and determining a first set of samples from the target set according to a batch size. The motivation to do so is to “alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels” stating that sampling strategy is used to prevent over-representation/bias towards a specific sample set. (Freitag, pg.553, section 3)
Regarding Claim 17,
the Yang(1)/Yang (2)/Freitag combination of claim 16 teaches,
The electronic device of claim 16 (thus the rejection of claim 16 is incorporated),
The combination of Yang (1)/Freitag fails to teach the predetermined rule for selecting target sample set.
However, Yang (2) teaches,
wherein the predetermined rule comprises any of: a random selection rule, a polling selection rule, and a selection rule based on a sample amount. (Yang (2), paragraph 47, “Specifically, a first sample set is selected from a global sample set including a plurality of sample sets as positive samples”. The citation here corresponds to a selection rule based on a sample amount as it discloses selecting first sample set from a global sample set showing that the selection is based on the number of samples available in the global sample set.)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Yang (2) to select a target sample from the different set of samples according to a selection rule. The motivation to do so is to prevent over-representation/bias towards a specific sample set.
Regarding Claim 18,
the Yang (1)/Yang (2)/Freitag combination of claim 16 teaches,
The electronic device of claim 16 (thus the rejection of claim 16 is incorporated),
Yang (1)/Yang (2) combination fails to teach determining the sample weights and selecting sample set based on the weights.
However, Freitag teaches,
wherein selecting the target sample set according to the selection rule based on a sample amount comprises (Freitag, pg.553, section 3, “Formally, for a given target language l, let D(l) be the size of the available training examples with target language l, the sample probability with a temperature T is defined as…” stating that the predetermined rule here is the probability sampling.)
and selecting the target sample set based on the first weight and the second weight. (Freitag, pg. 553, section 3, “we propose a hierarchical sampling strategy with two levels: i) we choose a target language (based on a temperature-based schedule),” where the target data group is chosen using the weights defined from the dataset.)
The combination of Yang (1)/ Freitag fails to disclose determining the weights for the sample sets.
Yang (2), in the same field of endeavor (training models using input mapping techniques), teaches,
determining a first weight for the first sample set and a second weight for the second sample set based on a first sample amount of the first sample set and a second sample amount of the second sample set, respectively; (Yang(2), paragraph 62, “ According to the definition of formula (2), item vol(A) represents the size of A, and item cut(A, Ā) represents a sum of weights between the data samples of the positive samples and the negative samples.” suggesting the presence of weights for each of the data samples.)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1),Freitag and Yang (2) to determine the sample weights and select sample set based on the weights. The motivation to do so is to “alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels” stating that sample weights is used to prevent over-representation of a specific sample set. (Freitag, pg.553, section 3)
Regarding Claim 19,
the Yang (1)/Freitag/Yang (2) combination of claim 16 teaches,
The electronic device of claim 16, wherein the method further comprises: (thus the rejection of claim 16 is incorporated),
Yang (1) further teaches,
determining an association relationship between data in a sample that is to be processed using the trained contrastive learning model. (Yang (1), pg. 3, section 3.2, “For i-th image xi and j-th language description tj in a batch B.…. Their similarity is calculated as
PNG
media_image2.png
45
164
media_image2.png
Greyscale
” showing that the model determines the association by computing similarity between their learned features.)
Regarding Claim 20,
Yang (1) teaches,
obtaining a plurality of sample sets for training the contrastive learning model, the plurality of sample sets comprising a first sample set and a second sample set; (Yang(1), pg. 6, section 4.2.1, “We use ImageNet-1K as the base dataset, and gradually add different sets of image-text pairs, including GCC-3M, GCC 15M and YFCC-14M” showing that the training process uses multiple sets of sample data)
and training the contrastive learning model using the first set of samples. (Yang(1), pg.1, abstract “In this space, we propose a new learning paradigm, called Unified Contrastive Learning (UniCL) with a single learning objective to seamlessly prompt the synergy of two data types.” and pg. 6, section 4.2.1 “When combining with image-text pairs, we use a balanced data sampler to ensure that the model is trained with the same number of image-label and image-text pairs per epoch. All models are trained with 500 epochs.”)
Yang (1) fails to teach a computer-readable storage medium which stores a computer program that is executed by a processor to perform a method.
However, Yang (2) discloses,
A non-transient computer-readable storage medium storing a computer program thereon, the computer program, when executed by a processor, performing a method, the method comprising: (Yang(2), paragraph 008, “ a computer-readable storage medium is provided” and paragraph 007, “including: at least one processing unit; and at least one memory that is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, wherein the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method”)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, and Yang (2) to have a processing unit, memory, storing instructions which is executed by the processing unit causing the device to perform a method. The motivation to do so is “to perform the training method” (Yang (2), paragraph 007)
The combination of Yang(1)/Yang (2) also fails to disclose target sample selection from the different set of samples according to a selection rule and determining a first set of samples from the target set according to a batch size.
Freitag teaches,
selecting a first target sample set from the first sample set and the second sample set according to a predetermined rule; (Freitag, pg.553, section 3, “To alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels: i) we choose a target language (based on a temperature-based schedule), ii) uniformly sample a source language. Formally, for a given target language l, let D(l) be the size of the available training examples with target language l, the sample probability with a temperature T is defined as…” stating that the predetermined rule here is the probability sampling.)”
determining a first set of samples based on the first target sample set according to a predefined batch size; (Freitag, pg. 553, section 3, “During training, the scheduler samples a batch of training examples based on the target language only, as opposed to source-target language pair specific sampling” and “All multilingual models are trained for 500,000 updates using an average batch size of around 33,000 sentences” showing the batch size)
Yang (1), Freitag and Yang (2) are all considered to be analogous to the claimed invention because they are in the same field of using input mapping techniques to make a model efficient. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Yang(2) to select a target sample from the different set of samples according to a selection rule and determining a first set of samples from the target set according to a batch size. The motivation to do so is to “alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels” stating that sampling strategy is used to prevent over-representation/bias towards a specific sample set. (Freitag, pg.553, section 3)
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Yang (1), in view of Freitag, further in view of Yang (2), further in view of Wu, and further in view of Shi, “Incremental Sampling Without replacement for Sequence Models”
Regarding Claim 13,
The combination of Yang (1)/Freitag/Yang (2) discloses,
The method according to claim 12 wherein determining the second set of samples comprises (and thus the rejection of claim 12 is incorporated).
Yang (1)/Freitag fails to teach the selection of positive sample from the second set of samples.
However, Yang (2) further discloses,
selecting a positive sample from the second set of samples (Yang (2), paragraph 47, “A third sample set is selected as positive samples, a fourth sample set is selected as negative samples to form the next piece of training data” the third sample set being the second set of samples here)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag and Yang (2) to teach the selection of positive sample from the second set of samples. The motivation to do so is to train the model using various samples to prevent bias.
The combination of Yang (1)/Freitag/Yang (2) fails to teach the unused positive samples where a positive sample is selected from.
However, Shi, in the same field of endeavor (sampling data efficiently), teaches,
from unused positive samples in the second set of samples; (Shi, pg.1, abstract, “Unlike prior work, our approach is incremental, i.e., samples can be drawn one at a time, allowing for increased flexibility.” showing that the samples not drawn are unused)
Yang (1), Freitag, Yang (2), and Shi are all considered to be analogous to the claimed invention because they are in the same field of sampling data efficiently. Therefore, it would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1) Freitag, Yang (2), and Shi to select a sample from unused positive sample. The motivation to do so is to allow flexibility (Shi, pg.1, abstract)
The combination of Yang (1)/Freitag/Yang (2) also fails to teach generating negative sample based on positive sample and data of the second modality and determining the second set of samples based on positive and negative sample.
However, Wu teaches,
generating a negative sample based on the positive sample and data of the second modality in the second target sample set; (Wu, pg.7, section 3.3.1, “In Formulas (13) and (14), (v,s) represents the text-video positive sample pair, and s− and v−, respectively, represent the negative text sample matching v and the negative video sample of matching s, so as to form (v,s−) and (v−,s), two types of negative sample pair.” showing that the negative pair is based on the second modality in the positive sample (v,s))
and determining the second set of samples based on the positive sample and negative sample. (Wu, pg.5, section 3, “The two types of distance are also used to train a multi-task to classify positive and negative text-video sample pairs”)
It would’ve been obvious to someone of ordinary skill in art before the filling date of the claimed invention to have combined the teachings of Yang (1), Freitag, Yang(2), Shi, and Wu to generate a negative sample based on positive sample and data of the second modality and determine the second set of sample based on positive and negative sample. The motivation to do so is to “establish a semantic alignment relationship between different modalities” (Wu, pg.5, section 3)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PRAYUKTA GHIMIRE whose telephone number is (571)270-5484. The examiner can normally be reached M-F, 9 am to 5 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PRAYUKTA NMN GHIMIRE/Examiner, Art Unit 2122
/MICHAEL H HOANG/PRIMARY EXAMINER, Art Unit 2122
Prosecution Insights