Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the application and claims filed 03/26/2024. Claims 1-20 are pending and have been examined. Claims 1-20 are rejected.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/26/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract
idea without significantly more.
Claim 1
Step 1: The claim recites a method; therefore, it is directed to the statutory category of process.
Step 2A Prong 1: The claim recites the following abstract idea:
“selecting a set of low-level features based on their correlation with expert knowledge of a domain;” This limitation is a mental process because a person can mentally or with a pen and paper select a set of features based on their relationship or connection with the expert knowledge of a domain. The specification defines “correlation” as “a relationship or connection between the features” (see paragraph [0017]); accordingly, a person can observe candidate features, judge which of them relate to a body of expert knowledge, and pick (select) the set in their mind.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
“A computer-implemented method (...) the method comprising:” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the claim recites a generic, off-the-shelf computer as a tool to perform the recited abstract idea.
“for enabling a machine learning model to run predictions on domains where training data is limited” The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
“training a student machine learning model to have its intermediate feature representations mimic said set of low-level features.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). -- EN: the claim denotes generic training and a generic, off-the-shelf student machine learning model with no additional details or limitations beyond a generic, off-the-shelf machine learning model. The “student machine learning model” is interpreted as applying a generic computer to perform the abstract idea.
Step 2B:
“A computer-implemented method (...) the method comprising:” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the claim recites a generic, off-the-shelf computer as a tool to perform the recited abstract idea. Mere instructions to apply an exception using a generic computer component.
“for enabling a machine learning model to run predictions on domains where training data is limited” The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
“training a student machine learning model to have its intermediate feature representations mimic said set of low-level features.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: generic training using a generic, off-the-shelf machine learning model amounts to no more than mere instructions to apply the exception using a generic computer component.
The additional elements considered individually or in combination do not amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: See the rejection of Claim 1 above, which claim 2 depends on. Claim 2 further recites:
“generating predictions on said domain (...)” This limitation is a mental process because a person can mentally or with a pen and paper generate predictions, i.e., make decisions or judgments, about a domain.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
“(...) using said trained student machine learning model.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the “trained student machine learning model” is interpreted as applying a generic, off-the-shelf machine learning model as a tool to perform the recited abstract idea.
Step 2B:
“(...) using said trained student machine learning model.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: applying a generic, off-the-shelf machine learning model amounts to no more than mere instructions to apply the exception using a generic computer component.
The additional elements considered individually or in combination do not amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: See the rejection of Claim 1 above, which claim 3 depends on. Claim 3 further recites:
“computing a distance between said intermediate feature representations and said set of low-level features to determine a loss.” This limitation falls within the mathematical concepts grouping because computing a distance between two sets of values and determining a loss involves performing a mathematical calculation. See Paragraph 103, “…such a distance corresponds to the cosine distance. Cosine distance = 1 – cosine similarity, where cosine similarity is a metric that determines how two vectors…”
Step 2A Prong 2: The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B: The claim does not recite any additional elements that amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: See the rejection of Claim 3 above, which claim 4 depends on. Claim 4 further recites:
“wherein said intermediate feature representations and said set of low-level features are multi-dimensional vectors.” This limitation falls within the mathematical concepts grouping because it specifies that the data operated upon in the distance computation are multi-dimensional vectors, which are mathematical constructs.
Step 2A Prong 2: The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B: The claim does not recite any additional elements that amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: See the rejection of Claim 3 above, which claim 5 depends on. Claim 5 further recites:
“wherein said distance is a cosine distance.” This limitation falls within the mathematical concepts grouping because a cosine distance is a specific mathematical formula (cosine distance = 1 − cosine similarity, see paragraph 103)
Step 2A Prong 2: The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B: The claim does not recite any additional elements that amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 6
Step 1: A process, as above.
Step 2A Prong 1: See the rejection of Claim 3 above, which claim 6 depends on. Claim 6 further recites:
“wherein said loss is selected from the group consisting of: a classification loss, a mean squared error, a Kullback-Leibler divergence loss, a regression, and a cross entropy loss.” This limitation falls within the mathematical concepts grouping because all of these functions are mathematical equations used to measure the distance between values i.e. loss/discrepancy.
Step 2A Prong 2: The claim does not recite any additional elements that integrate the judicial exception into a practical application.
Step 2B: The claim does not recite any additional elements that amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 7
Step 1: A process, as above.
Step 2A Prong 1: See the rejection of Claim 1 above, which claim 7 depends on.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
“wherein said student machine learning model is trained in a supervised manner.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: this limitation merely amounts to applying a generic, off-the-shelf machine learning model that is trained using generic supervised learning.
Step 2B:
“wherein said student machine learning model is trained in a supervised manner.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: applying generic supervised learning to a generic, off-the-shelf machine learning model amounts to no more than mere instructions to apply the exception using a generic computer component.
The additional elements considered individually or in combination do not amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 8
Step 1: The claim recites a computer program product; therefore, it is directed to the statutory category of manufacture.
Step 2A Prong 1: The claim recites the following abstract idea:
“selecting a set of low-level features based on their correlation with expert knowledge of a domain;” This limitation is a mental process because a person can mentally or with a pen and paper select a set of features based on their relationship or connection with the expert knowledge of a domain. The specification defines “correlation” as “a relationship or connection between the features” (see paragraph [0017]); accordingly, a person can observe candidate features, judge which of them relate to a body of expert knowledge, and pick (select) the set in their mind.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
“A computer program product (...) the computer program product comprising one or more computer readable storage mediums having program code embodied therewith, the program code comprising programming instructions for:” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the claim recites a generic, off-the-shelf computer and storage medium as tools to perform the recited abstract idea.
“for enabling a machine learning model to run predictions on domains where training data is limited” The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
“training a student machine learning model to have its intermediate feature representations mimic said set of low-level features.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the claim denotes generic training and a generic, off-the-shelf student machine learning model with no additional details or limitations beyond a generic, off-the-shelf machine learning model. The “student machine learning model” is interpreted as applying a generic computer to perform the abstract idea.
Step 2B:
“A computer program product (...) the computer program product comprising one or more computer readable storage mediums having program code embodied therewith, the program code comprising programming instructions for:” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the claim recites a generic, off-the-shelf computer and storage medium as tools to perform the recited abstract idea. Mere instructions to apply an exception using a generic computer component.
“for enabling a machine learning model to run predictions on domains where training data is limited” The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
“training a student machine learning model to have its intermediate feature representations mimic said set of low-level features.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: generic training using a generic, off-the-shelf machine learning model amounts to no more than mere instructions to apply the exception using a generic computer component.
The additional elements considered individually or in combination do not amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 9
Claim 9 is a computer program product claim that recites the same limitations as claim 2, therefore claim 9 is rejected using the same rationale as claim 2.
Claim 10
Claim 10 is a computer program product claim that recites the same limitations as claim 3, therefore claim 10 is rejected using the same rationale as claim 3.
Claim 11
Claim 11 is a computer program product claim that recites the same limitations as claim 4, therefore claim 11 is rejected using the same rationale as claim 4.
Claim 12
Claim 12 is a computer program product claim that recites the same limitations as claim 5, therefore claim 12 is rejected using the same rationale as claim 5.
Claim 13
Claim 13 is a computer program product claim that recites the same limitations as claim 6, therefore claim 13 is rejected using the same rationale as claim 6.
Claim 14
Claim 14 is a computer program product claim that recites the same limitations as claim 7, therefore claim 14 is rejected using the same rationale as claim 7.
Claim 15
Step 1: The claim recites a system; therefore, it is directed to the statutory category of machine.
Step 2A Prong 1: The claim recites the following abstract idea:
“selecting a set of low-level features based on their correlation with expert knowledge of a domain;” This limitation is a mental process because a person can mentally or with a pen and paper select a set of features based on their relationship or connection with the expert knowledge of a domain. The specification defines “correlation” as “a relationship or connection between the features” (see paragraph [0017]); accordingly, a person can observe candidate features, judge which of them relate to a body of expert knowledge, and pick (select) the set in their mind.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because the additional elements are as follows:
“A system, comprising: a memory for storing a computer program (...); and a processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program comprising:” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the claim recites a generic, off-the-shelf memory and processor as tools to perform the recited abstract idea.
“for enabling a machine learning model to run predictions on domains where training data is limited” The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
“training a student machine learning model to have its intermediate feature representations mimic said set of low-level features.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). Examiner's Note (EN): the claim denotes generic training and a generic, off-the-shelf student machine learning model with no additional details or limitations beyond a generic, off-the-shelf machine learning model. The “student machine learning model” is interpreted as applying a generic computer to perform the abstract idea.
Step 2B:
“A system, comprising: a memory for storing a computer program (...); and a processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program comprising:” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: the claim recites a generic, off-the-shelf memory and processor as tools to perform the recited abstract idea. Mere instructions to apply an exception using a generic computer component.
“for enabling a machine learning model to run predictions on domains where training data is limited” The limitation amounts to merely indicating a field of use or technological environment in which to apply a judicial exception. This does not amount to significantly more than the exception itself (MPEP 2106.05(h)).
“training a student machine learning model to have its intermediate feature representations mimic said set of low-level features.” Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). EN: generic training using a generic, off-the-shelf machine learning model amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component.
The additional elements considered individually or in combination do not amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claim 16
Claim 16 is a system claim that recites the same limitations as claim 2, therefore claim 16 is rejected using the same rationale as claim 2.
Claim 17
Claim 17 is a system claim that recites the same limitations as claim 3, therefore claim 17 is rejected using the same rationale as claim 3.
Claim 18
Claim 18 is a system claim that recites the same limitations as claim 4, therefore claim 18 is rejected using the same rationale as claim 4.
Claim 19
Claim 19 is a system claim that recites the same limitations as claim 5, therefore claim 19 is rejected using the same rationale as claim 5.
Claim 20
Claim 20 is a system claim that recites the same limitations as claim 6, therefore claim 20 is rejected using the same rationale as claim 6.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Examiner’s Note: Some rejections will include an Examiner’s Note (labeled ‘EN’) to provide additional context or rationale explaining the basis for the rejection.
Claims 1-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Koh et al. (“Concept Bottleneck Models”, hereinafter “Koh”) in view of Sanh et al. (“DistilBERT, a distilled version of BERT”, hereinafter “Sanh”).
Claim 1
Regarding claim 1, Koh teaches:
A computer-implemented method for enabling a machine learning model to run predictions on domains where training data is limited (Koh, Page 5: “Concept bottleneck models are particularly effective on OAI: the sequential bottleneck model with ≈ 25% of the full dataset performs similarly to the standard model.” -- EN: getting the standard task accuracy using only ~25% of the training data denotes enabling a machine learning model to run predictions on a domain where training data is limited; see also Koh, Page 9: “if the set of concepts are good enough, then fewer training examples might be required to achieve a desired accuracy level (as in OAI).”)
selecting a set of low-level features based on their correlation with expert knowledge of a domain (Koh, Page 4: “As concepts, we use k = 10 ordinal variables describing joint space narrowing, bone spurs, calcification, etc.; these clinical concepts are also assessed by radiologists and used directly in the assessment of KLG (Kellgren & Lawrence, 1957).” -- EN: under the broadest reasonable interpretation in light of Merler [0015]-[0016] (“the more specific individual components of a systematic operation, focusing on the details of rudimentary micro functions”), Koh’s human-specified clinical concepts (e.g., bone spurs, joint space narrowing) are specific individual components that read on the claimed low-level features, and choosing those concepts because radiologists assess them and use them directly in the expert KLG diagnosis denotes selecting low-level features based on their correlation with expert knowledge of the domain.)
training a [...] machine learning model to have its intermediate feature representations mimic said set of low-level features (Koh, Page 2: “we simply resize one of the layers to match the number of concepts provided, and add an intermediate loss that encourages the neurons in that layer to align component-wise to the provided concepts.” -- EN: resizing an intermediate layer to the number of concepts and adding a loss that forces those neurons to align component-wise with the provided concepts denotes training a model so that its intermediate feature representations mimic the selected set of low-level features; also see Koh, Page 3: “the bottleneck ĉ = g(x), which we train to align component-wise to the concepts c.”)
Koh does not explicitly disclose:
[training a] student [machine learning model]
However, Sanh teaches:
[training a] student [machine learning model] (Sanh, Page 2: “a compression technique in which a compact model - the student - is trained to reproduce the behaviour of a larger model - the teacher - or an ensemble of models.” -- EN: the compact “student” model trained to reproduce a teacher’s behavior denotes the claimed student machine learning model.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept-bottleneck model of Koh -- whose intermediate layer is trained to align component-wise with the human-specified expert concepts (low-level features) with the student machine learning model trained via knowledge distillation as taught by Sanh, in order to obtain a smaller, faster, and cheaper model. (See Sanh, Abstract: “it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”).
Claim 2
Regarding claim 2, Koh in view of Sanh teaches all the limitations of claim 1, and Koh further teaches:
generating predictions on said domain using said trained student machine learning model (Koh, Page 1: “At test time, they take in an input x, predict concepts ĉ, and then use those concepts to predict the target ŷ (Figure 1).” -- EN: using the trained bottleneck model at test time to predict the target y for inputs drawn from the domain (for example, predicting the KLG osteoarthritis grade on the data-limited OAI domain) denotes generating predictions on said domain using the machine learning model; see also Koh, Page 3: “their prediction ŷ = f(g(x)) relies on the input x entirely through the bottleneck ĉ = g(x)”.)
Claim 3
Regarding claim 3, Koh in view of Sanh teaches all the limitations of claim 1, and Koh further teaches:
computing a distance between said intermediate feature representations and said set of low-level features to determine a loss (Koh, Page 3: "Let L_Cj : R × R → R+ be a loss function that measures the discrepancy between the predicted and true j-th concept"; and "the bottleneck ĉ = g(x), which we train to align component-wise to the concepts c." -- EN: Koh's concept loss L_Cj measures the discrepancy between the predicted concept g_j(x) read from the resized bottleneck layer (the claimed intermediate feature representation) and the true human-specified concept c_j (the claimed selected low-level feature); computing this discrepancy over the k concepts to produce the concept loss denotes computing a distance between the intermediate feature representations and the set of low-level features to determine a loss.
Claim 4
Regarding claim 4, Koh in view of Sanh teaches all the limitations of claim 3, and Koh further teaches:
wherein said intermediate feature representations and said set of low-level features are multi-dimensional vectors (Koh, Page 3: "where c ∈ R^k is a vector of k concepts"; and "g : R^d → R^k maps an input x into the concept space ... [the] bottleneck ĉ = g(x)." -- EN: the predicted concepts ĉ = g(x) read from the bottleneck layer (the claimed intermediate feature representations) and the true concepts c (the claimed selected low-level features) are each elements of R^k -- i.e., k-dimensional vectors)
Claim 5
Regarding claim 5, Koh in view of Sanh teaches all the limitations of claim 3, and Sanh further teaches:
wherein said distance is a cosine distance (Sanh, Abstract: “we introduce a triple loss combining language modeling, distillation and cosine-distance losses.” -- EN: Sanh’s cosine-distance loss denotes the claimed distance being a cosine distance; see also Sanh, Page 2: “a cosine embedding loss (Lcos) which will tend to align the directions of the student and teacher hidden states vectors.”)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept-bottleneck model of Koh -- whose intermediate layer is trained to align component-wise with the use of a cosine distance for the distance-based alignment loss of the Sanh because Sanh teaches that the cosine-distance loss aligns the directional relationship between the student and teacher representations and contributes substantially to the distilled model’s performance. (See Sanh, Page 4: “the two distillation losses account for a large portion of the performance”).
Claim 6
Regarding claim 6, Koh in view of Sanh teaches all the limitations of claim 3, and Koh further teaches:
wherein said loss is selected from the group consisting of: a classification loss, a mean squared error, a Kullback-Leibler divergence loss, a regression, and a cross entropy loss (Koh, Page 4: “We model x-ray grading as a regression problem (minimizing mean squared error) on both the KLG target y and concepts c, following Pierson et al. (2019);” Additionally, Shah also teaches, Sanh, Page 2: “minimizing the cross-entropy between the model’s predicted distribution and the one-hot empirical distribution of training labels.” – EN: this denotes the “mean squared error” and “cross entropy loss” from the claimed Markush group, teaching at least one member of a Markush group which satisfies the limitation)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to implement the loss of the Koh-Sanh student model as a cross-entropy (distillation) loss, as taught by Sanh, because Sanh teaches that this objective results in a rich training signal by leveraging the full teacher distribution (Sanh, Page 2: “This objective results in a rich training signal by leveraging the full teacher distribution.”), for the same reasons set forth with respect to claims 1 and 3.
Claim 7
Regarding claim 7, Koh in view of Sanh teaches all the limitations of claim 1, and Koh further teaches:
wherein said student machine learning model is trained in a supervised manner (Koh, Page 2: “rather than the standard supervised setting we consider here.” -- EN: Koh trains its concept-bottleneck model in the standard supervised setting (i.e., on data points annotated with concept and target labels), which denotes the student machine learning model being trained in a supervised manner.)
Claim 8
Regarding claim 8, Koh teaches:
A computer program product for enabling a machine learning model to run predictions on domains where training data is limited (Koh, Page 5: “Concept bottleneck models are particularly effective on OAI: the sequential bottleneck model with ≈ 25% of the full dataset performs similarly to the standard model.” --EN: attaining standard task accuracy using only ~25% of the training data denotes enabling a machine learning model to run predictions on a domain where training data is limited (the recited “computer program product” / computer-readable-storage-medium structure being separately addressed under Sanh, below); see also Koh, Page 9: “if the set of concepts are good enough, then fewer training examples might be required to achieve a desired accuracy level (as in OAI).”)
selecting a set of low-level features based on their correlation with expert knowledge of a domain (Koh, Page 4: “As concepts, we use k = 10 ordinal variables describing joint space narrowing, bone spurs, calcification, etc.; these clinical concepts are also assessed by radiologists and used directly in the assessment of KLG (Kellgren & Lawrence, 1957).” --EN: under the broadest reasonable interpretation in light of Merler [0015]-[0016] (“the more specific individual components of a systematic operation, focusing on the details of rudimentary micro functions”), Koh’s human-specified clinical concepts (e.g., bone spurs, joint space narrowing) are specific individual components that read on the claimed low-level features, and choosing those concepts because radiologists assess them and use them directly in the expert KLG diagnosis denotes selecting low-level features based on their correlation with expert knowledge of the domain.)
training a [...] machine learning model to have its intermediate feature representations mimic said set of low-level features (Koh, Page 2: “we simply resize one of the layers to match the number of concepts provided, and add an intermediate loss that encourages the neurons in that layer to align component-wise to the provided concepts.” --EN: resizing an intermediate layer to the number of concepts and adding a loss that forces those neurons to align component-wise with the provided concepts denotes training a model so that its intermediate feature representations mimic the selected set of low-level features (Merler [0017]); see also Koh, Page 3: “the bottleneck ĉ = g(x), which we train to align component-wise to the concepts c.”)
Koh does not distinctly disclose:
“the computer program product comprising one or more computer readable storage mediums having program code embodied therewith, the program code comprising programming instructions for”
And
[training a] student [machine learning model]
However, Sanh teaches:
“the computer program product comprising one or more computer readable storage mediums having program code embodied therewith, the program code comprising programming instructions for” (Sanh, Page 2: “We have made the trained weights available along with the training code in the Transformers library from HuggingFace [Wolf et al., 2019].” --EN: providing the executable training code (program code) as a stored, retrievable software library denotes program code embodied on one or more computer readable storage media that comprises programming instructions for performing the recited operations; see also Sanh, Page 4: “the whole model weighs 207 MB (which could be further reduced with quantization).”)
[training a] student [machine learning model] (Sanh, Page 2: “a compression technique in which a compact model - the student - is trained to reproduce the behaviour of a larger model - the teacher - or an ensemble of models.” --EN: the compact “student” model trained to reproduce a teacher’s behavior denotes the claimed student machine learning model.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept-bottleneck model of Koh -- whose intermediate layer is trained to align component-wise with the student machine learning model trained via knowledge distillation and the computer code as taught by Sanh, in order to obtain a smaller, faster, and cheaper model. (motivation from Sanh, Abstract: “it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”). Further, implementing the resulting method as program code on one or more computer readable storage mediums, as taught by Sanh (which makes its training code and trained weights available as a stored software library), is a predictable and conventional implementation that enables storage and distribution of the recited method, yielding the expected result of a model that retains predictive accuracy in domains where training data is limited.
Claims 9-14
Claims 9-14 are the computer-program-product dependents of claim 8 that corresponds to method claims 2-7. Each recites, as program code / programming instructions, the same substantive limitation as its method counterpart and is rejected under the same rationale and over the same reference set forth above.
Claim 15
Regarding claim 15, Koh teaches:
a [...] computer program for enabling a machine learning model to run predictions on domains where training data is limited (Koh, Page 5: “Concept bottleneck models are particularly effective on OAI: the sequential bottleneck model with ≈ 25% of the full dataset performs similarly to the standard model.” --EN: attaining standard task accuracy using only ~25% of the training data denotes a computer program for enabling a machine learning model to run predictions on a domain where training data is limited (the recited memory-and-processor structure being separately addressed under Sanh, below); see also Koh, Page 9: “if the set of concepts are good enough, then fewer training examples might be required to achieve a desired accuracy level (as in OAI).”)
selecting a set of low-level features based on their correlation with expert knowledge of a domain (Koh, Page 4: “As concepts, we use k = 10 ordinal variables describing joint space narrowing, bone spurs, calcification, etc.; these clinical concepts are also assessed by radiologists and used directly in the assessment of KLG (Kellgren & Lawrence, 1957).” --EN: under the broadest reasonable interpretation in light of Merler [0015]-[0016] (“the more specific individual components of a systematic operation, focusing on the details of rudimentary micro functions”), Koh’s human-specified clinical concepts (e.g., bone spurs, joint space narrowing) are specific individual components that read on the claimed low-level features, and choosing those concepts because radiologists assess them and use them directly in the expert KLG diagnosis denotes selecting low-level features based on their correlation with expert knowledge of the domain.)
training a [...] machine learning model to have its intermediate feature representations mimic said set of low-level features (Koh, Page 2: “we simply resize one of the layers to match the number of concepts provided, and add an intermediate loss that encourages the neurons in that layer to align component-wise to the provided concepts.” --EN: resizing an intermediate layer to the number of concepts and adding a loss that forces those neurons to align component-wise with the provided concepts denotes training a model so that its intermediate feature representations mimic the selected set of low-level features (Merler [0017]); see also Koh, Page 3: “the bottleneck ĉ = g(x), which we train to align component-wise to the concepts c.”)
Koh does not distinctly disclose:
a memory for storing a computer program [...] and a processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program (i.e., the recited memory-and-processor hardware that stores and executes the program);
and
[training a] student [machine learning model]
However, Sanh teaches:
a memory for storing a computer program [...] and a processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program (Sanh, Page 3: “DistilBERT was trained on 8 16GB V100 GPUs for approximately 90 hours.” --EN: the 16GB V100 GPUs are processors having connected (16GB) memory on which the program is stored and executed, which denotes a memory for storing the computer program and a processor connected to the memory configured to execute the program instructions; see also Sanh, Page 4, describing on-device inference executed on an iPhone 7 Plus processor.)
[training a] student [machine learning model] (Sanh, Page 2: “a compression technique in which a compact model - the student - is trained to reproduce the behaviour of a larger model - the teacher - or an ensemble of models.” --EN: the compact “student” model trained to reproduce a teacher’s behavior denotes the claimed student machine learning model.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to combine the concept-bottleneck model of Koh -- whose intermediate layer is trained to align component-wise with the student machine learning model trained via knowledge distillation and hardware as taught by Sanh, in order to obtain a smaller, faster, and cheaper model. (See Sanh, Abstract: “it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”). Further, executing the resulting method on a processor connected to a memory, as taught by Sanh (which trained and ran its model on 16GB V100 GPUs and on a mobile processor), is a predictable and conventional implementation that supplies the computational resources necessary to store and execute the program, yielding the expected result of a model that retains predictive accuracy in domains where training data is limited.
Claims 16-20
Claims 16-20 are the system dependents corresponding to method claims 2-6. Each recites, as program instructions executed by the processor, the same substantive limitation as its method counterpart and is rejected under the same rationale and over the same reference set forth above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAYMUR RAHMAN ALI whose telephone number is (571)272-0007. The examiner can normally be reached Mon-Fri. 9:30-6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571)270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NAYMUR RAHMAN ALI/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights