DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 17 is objected to because of the following informalities: to be consistent with claim 8 and to clarify what the “adding” includes, the term “together” should be added at the end of claim 17 as in, “The method of claim 10, wherein using the additive function comprises adding class-wise contribution functions for the plurality of patches together.” Appropriate correction is required.
Claim Interpretation
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“a patch generator, executed by the at least one processor, configured to generate a bag comprising a plurality of patches from an input image, each patch comprising a distinct portion of the input image” in claims 1-9.
“a featurizer, executed by the at least one processor, comprising a neural network model configured to generate a plurality of patch embeddings using at least a portion of the bag” in claims 1-9.
“an attention module, executed by the at least one processor, configured to: determine an attention score for at least some of the plurality of patch embeddings; and generate a plurality of attention weighted patch embeddings by scaling the plurality of patch embeddings using the attention scores” in claim 1-9.
“an additive predictor, executed by the at least one processor, configured to: aggregate the plurality of attention weighted patch embeddings to generate a plurality of patch-wise class contributions, wherein each patch-wise class contribution represents a contribution of a corresponding class; and compute a plurality of predictions from the patch-wise class contributions using an additive function” in claims 1-9.
“generating, using a patch generator, a bag comprising a plurality of patches from an input image, each patch comprising a distinct portion of the input image” in claims 10-18.
“generating, using a featurizer comprising a neural network model, a plurality of patch embeddings using at least a portion of the bag” in claims 10-18.
“determining, using an attention module, an attention score for at least some of the plurality of patch embeddings, generating, using the attention module, a plurality of attention weighted patch embeddings by scaling the plurality of patch embeddings using the attention scores” in claims 10-18.
“aggregating, using an additive predictor, the plurality of attention weighted patch embeddings to generate a plurality of patch-wise class contributions, wherein each patch-wise class contribution represents a contribution of a corresponding class; and computing, using the additive predictor, a plurality of predictions from the patch-wise class contributions using an additive function” in claims 10-18.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f), applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 4 and 13 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claims 4 and 13 recite “determining the sign of the at least one of the plurality of patch-wise class contributions”. A “sign” can be positive, negative, zero, or some other indication, but the claims do not provide sufficient context for a clear antecedent basis of “the sign”. Therefore, claims 4 and 13 are indefinite because the sign” lacks antecedent basis. For purposes of applying prior art, “the sign” is interpreted as “a sign”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5, 8-12, 14, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Attention-based Deep Multiple Instance Learning to Ilse et al. (hereinafter “Ilse” - copy already of record) in view of Data-efficient and weakly supervised computational pathology on whole-slide images to Lu et al. (hereinafter “Lu”).
Regarding claim 1, Ilse teaches a system for additive multiple instance learning (MIL), comprising (Ilse, section 1, “In this paper, we propose a new method that aims at incorporating interpretability to the MIL approach and increasing its flexibility.”):
at least one processor operatively connected to a memory (Ilse discloses neural-network based multiple instance learning, which implicitly teaches a generic computer including a processor and a memory because the disclosed testing results could not otherwise have been obtained without a computer. See Ilse at section 1);
a patch generator (patch generation software), executed by the at least one processor (see above), configured to generate a bag (Ilse, section 2.1, “the bag of instances X”) comprising a plurality of patches from an input image (Ilse, section 2.1, “x1,...,xk”), each patch comprising a distinct portion of the input image (Ilse, section 4.3, “In Figure 5 we show a histopathology image divided into patches containing (mostly) single cells.”);
a featurizer (feature embedding software), executed by the at least one processor (see above), comprising a neural network model (Ilse, section 2.4, “We propose to use a weighted average of instances (low-dimensional embeddings) where weights are determined by a neural network.” ) configured to generate a plurality of patch embeddings using at least a portion of the bag (Ilse, section 2.4, “Let H = {h1,...,hk} be a bag of K embeddings”);
an attention module (attention-based MIL pooling software), executed by the at least one processor (see above), configured to:
determine an attention score (attention weights ak. See Ilse at section 2.4, equations (7)-(9)) for at least some of the plurality of patch embeddings (ak is calculated, in part, using hk. See Ilse at section 2.4); and
generate a plurality of attention weighted patch embeddings by scaling the plurality of patch embeddings using the attention scores (Each embedding hk is multiplied/scaled by its corresponding attention coefficient ak. See Ilse at section 2.4, equation (7)), but does not teach that which is explicitly taught by Lu.
Lu teaches an additive predictor, executed by the at least one processor (Lu, pg. 17, “Intel Xeon CPUs”. The models were trained using “NVIDIA 2080 Ti GPUs”. See Id.), configured to:
aggregate the plurality of attention weighted patch embeddings (Lu, pg. 14, equation (2), “hk”) to generate a plurality of patch-wise class contributions, wherein each patch-wise class contribution represents a contribution of a corresponding class (Lu, pg. 3, “during both training and inference, the model examines and ranks all patches in the tissue regions of a WSI, assigning an attention score to each patch, which informs its contribution or importance to the collective slide-level representation for a specific class”); and
compute a plurality of predictions from the patch-wise class contributions (Lu, pg. 4, “A CLAM model has N parallel attention branches that together calculate N unique slide-level representations, where each representation is determined from a different set of highly attended regions in the image viewed by the network as strong positive evidence for the one of N classes in a multi-class diagnostic task (Fig. 1b,c). Each class-specific slide representation is then examined by a classification layer to obtain the final probability score predictions for the whole slide.”) using an additive function (As shown in Fig. 1(b), Lu uses a softmax function to combine the N class branches prediction scores. The denominator of the softmax function is a linear summation (addition) operation over the exponentiated class scores of the N branches. Thus, the softmax function includes a plurality of predictions.).
Ilse discloses attention-based pooling for multiple-instance learning. Thus, Ilse shows that it was known in the art before the effective filing date of the claimed invention to use a learnable attention mechanism to automatically determine which instances (patches) in a bag are the most important to the bag-level prediction, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, improving accuracy of whole-slide MIL models to produce accurate classifications and interpretable heatmaps. Lu discloses attention-based MIL that computes class-specific attention branches that are combined to produce the bag-level prediction. Thus, Lu shows that it was known in the art before the effective filing date of the claimed invention to calculate attention for each class and additively combine contributions from each class, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, improving accuracy of whole-slide MIL models to produce accurate classifications and interpretable heatmaps.
A person of ordinary skill in the art would have been motivated to modify Ilse’s attention mechanism by computing one attention branch per class and then additively combining them as disclosed by Lu, to thereby generate an attention distribution for each class over the same patches. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of accurately localizing and interpreting pathology features for each class.
Regarding claim 2, Ilse in view of Lu teaches the system of claim 1, wherein the neural network model is trained with weakly annotated data (Ilse, section 4.3, “BREAST CANCER consists of 58 weakly labeled 896 × 768 H&E images. An image is labeled malignant if it contains breast cancer cells, otherwise it is benign. We divide every image into 32 × 32 patches. This results in 672 patches per bag. A patch is discarded if it contains 75% or more of white pixels.”).
Regarding claim 3, Ilse in view of Lu teaches the system of claim 1, wherein the additive predictor is further configured to distinguish between excitatory and inhibitory patch contributions using at least one of the plurality of patch-wise class contributions (Lu, pg. 21, “high attention displayed in red (positive evidence, high contribution to the prediction of the model relative to other patches) and low attention displayed in blue (low contribution to prediction of the model relative to other patches).” Red in the heatmaps indicates an excitatory patch and blue indicates an inhibitory patch.).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 5, Ilse in view of Lu teaches the system of claim 1, wherein computing the plurality of predictions comprises computing a first prediction for a first class and a second prediction for a second class (Each attention branch computes a class-specific prediction score and the collection of scores is fed to a softmax function. See Lu at Fig. 1(c). Each branch score is an indication of the amount of evidence for a specific class. Thus, for classes 1 and 2, first and second scores are predictions for the first and second classes respectively.).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 8, Ilse in view of Lu teaches the system of claim 1, wherein using the additive function comprises adding class-wise contribution functions for the plurality of patches together (As shown in Fig. 1(b), Lu uses a softmax function to combine the N class branches prediction scores. The denominator of the softmax function is a linear summation (addition) operation over the exponentiated class scores of the N branches. Thus, the softmax function includes a plurality of predictions. Each prediction is made by an independent classifier. See Lu at pg. 13. Each classifier is a contribution function because it outputs an indication of how strongly the model thinks the slide belongs to a given class.).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 9, Ilse in view of Lu teaches the system of claim 1, wherein the plurality of patch-wise class contributions are linear (As shown in Fig. 1(b), Lu uses a softmax function to combine the N class branches prediction scores. The denominator of the softmax function is a linear summation (addition) operation over the exponentiated class scores of the N branches.).
The rationale for obviousness is the same as provided for claim 1.
Claims 10-12, 14, 17 and 18 substantially correspond to claims 1-3, 5, 8 and 9 by reciting methods corresponding to the functions of the claimed systems. The rationale for obviousness of each corresponding claim is the same.
Allowable Subject Matter
Claims 4 and 13 would be allowable if rewritten to overcome the rejection under 35 U.S.C. 112(b) set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Claims 6, 7, 15 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathology (published 28 November 2022, hereinafter “Javed”) is pertinent because its authorship is identical to the inventorship of the instant application and it discloses a more comprehensive version of the instant specification. Javed does not qualify as prior art per 35 U.S.C. 102(b)(1)(A) because it was published within the grace period and does not recite any additional authors or leave out any of the inventors. However, Javed is still pertinent because it provides a more detailed explanation of the concepts disclosed herein and provides relevant citations that highlight influential prior art.
TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification to Shao et al. is pertinent because it is one of the iterations of attention-based MIL methods that followed Ilse’s attention-based MIL model but preceded applicant’s additive MIL model (Additive MIL).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN P POTTS whose telephone number is (571)272-6351. The examiner can normally be reached M-F, 9am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN P POTTS/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights