DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 7, 16, and 20 are cancelled.
Claims 1-6, 8-15, 17-19, and 21-23 are pending.
Claims 1-6, 8-15, 17-19, and 21-23 are rejected.
Priority
Applicant’s claim for the benefit of prior-filed applications, U.S. Provisional App. No. 63/179,091 filed 23 April 2021 and PCT/US2022/025699 filed 21 April 2022, under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) are acknowledged.
Therefore, the effective filing date of the claimed invention is 23 April 2021.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 19, Oct. 2023 and 25 Oct. 2023 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the list of cited references was considered in full by the examiner.
Drawings
The drawings were received on 19 Oct. 2023. These drawings are accepted.
Specification
Any references made to Applicant’s specification are made with respect to the published version of the specification.
Claim Interpretation
Claims 10 and 19 recite “providing the tumor image tiles to a neural network, wherein the neural network was trained to identify…, wherein training the neural network involved applying random image compression…”. The limitation regarding how the neural network was previously trained is interpreted to be a product by process limitation that defines the process in which the neural network was previously trained. However, the claim does not require a step of training the neural network. See MPEP 2113 I. "[E]ven though product-by-process claims are limited by and defined by the process, determination of patentability is based on the product itself. The patentability of a product does not depend on its method of production. If the product in the product-by-process claim is the same as or obvious from a product of the prior art, the claim is unpatentable even though the prior product was made by a different process." In re Thorpe, 777 F.2d 695, 698, 227 USPQ 964, 966 (Fed. Cir. 1985).
Claims 12, 14-15, and 17 further limit the product by process limitation of claim 10 above, and therefore are part of the product by process limitation.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 4, 10-15, and 17-19 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claims 4 and 13 are indefinite for recitation of “a Tensorflow and Keras implementation of an Xception neural network model”. Claim 4 contains the trademark/trade names “Tensorflow” and “Keras”. Where a trademark or trade name is used in a claim as a limitation to identify or describe a particular material or product, the claim does not comply with the requirements of 35 U.S.C. 112(b). See Ex parte Simpson, 218 USPQ 1020 (Bd. App. 1982). The claim scope is uncertain since the trademark or trade name cannot be used properly to identify any particular material or product. A trademark or trade name is used to identify a source of goods, and not the goods themselves. Thus, a trademark or trade name does not identify or describe the goods associated with the trademark or trade name. In the present case, the trademark/trade name is used to identify/describe the source of an implementation of a neural network and, accordingly, the identification/description is indefinite.
Claims 10 and 19, and claims dependent therefrom, are indefinite for recitation of “providing the tumor image tiles to a neural network, wherein the neural network was trained…wherein training the neural network involves applying random image compression or random Gaussian blur to the tumor image tiles”. Claims 10 and 19 recite an initial step of obtaining tumor image tiles, and then provides these tumor image tiles to an already trained neural network for classification of the tumor image tiles. However, claims 10 and 19 later recites training the neural network involved applying random image compressing or adding noise to the tumor image tiles, which had yet to be obtained. Accordingly, it is not clear if (1) Applicant intends for the step of providing the tumor image tiles to a neural network to actually be part of the training process, such that claims 10 and 19 require a step of training the neural network, or if (2) Applicant intends for “the tumor image tiles” referred to in the last two lines of the claim to be a different set of training tumor image tiles. Clarification is requested via claim amendment. For purpose of examination, claims 10 and 19 are interpreted to mean “wherein training the neural network involves applying random image compression…to training tumor image tiles”.
Dependent claims 12, 13-14, and 17 recite “wherein the neural network was trained based on/by…the tumor image tiles”, which is indefinite for the same reasons discussed above for claim 10. Consistent with claim 10 above, “the tumor image tiles” used in the process in which the neural network was trained are interpreted to refer to “the training tumor image tiles”.
Claim 23 is indefinite for recitation of “virtual tile sub-images of tumor areas with reduced focal depth”. The term “reduced” in claim 23 is a relative term which renders the claim indefinite. The term “reduced” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. As a result, it is not clear which sub-images of tumor areas would be considered to have a “reduced focal depth”. The claim is interpreted to mean the tile images comprise sub-images of tumor areas with lower resolution compared to other areas of the tumor images.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-6, 8-12, 14-15, 17-19, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Shaul in view of Raedt (2019).
Cited reference:
Shaul et al. US 2022/0237788 A1; effectively filed 20 Nov. 2020 based on priority to PCT/EP2020/0082917; and
Raedt et al. US 2024/0037747 A1; effectively filed 18 Sept. 2020 based on priority to PCT/EP2020/076090.
Regarding claim 1, Shaul discloses a computer-implemented method for tissue image classification (Abstract; [0275]), comprising the following steps:
Shaul discloses receiving a plurality of digital images depicting tissue samples patients, wherein the tissue includes a tumor ([0174], e.g. images contain tumor; [0122], e.g. method used to classify cancer patient tissue as having a morphological state), and splitting each received image into a set of image tiles (i.e. generating tumor image tiles from images of a tumor) ([0008]; claim 1). Shaul further discloses each of the image tiles is assigned a class label relating to a morphological state of a cancer patient tissue ([0223]; [0122]).
Shaul discloses training a multiple-instance-learning (MIL) program in the form of a neural network (claim 1; [0262]; [0278]; [0289], e.g. MIL program is a neural network; [0345]-[0346]), wherein training results in the neural network learning features descriptive of the tissue area depicted in the image tile (i.e. histology features) for classification ([0262] and [0278], e.g. feature extraction module is part of the MIL-program; [0223], e.g. aim of learning is to identify features of instances which are predictive of class membership). Shaul further discloses training the neural network comprises adding random Gaussian blur to the tumor image tiles ([0336]-[0338], e.g. creation of additional training tiles by adding gaussian blur to image tiles).
Regarding claims 10 and 19, Shaul discloses a computer-implemented method for tissue image classification and computer-readable storage medium for implementing the method, (Abstract; [0275]), wherein the method comprises the following steps:
Shaul discloses receiving a digital image of a tissue sample of a patient to be classified (claim 1), wherein the tissue includes a tumor ([0174], e.g. images contain tumor; [0122], e.g. method used to classify cancer patient tissue as having a morphological state), and splitting each received image into a set of image tiles (i.e. generating tumor image tiles from images of a tumor) ([0008]; claim 1).
Shaul discloses providing the tumor image tiles of the patient to the trained MIL program (i.e. the neural network) (Fig. 1; [0174];[0122]) to generate classifications for the morphological state of cancer tissue of the tumor image tiles (Fig. 2; [0122]; claim 1, e.g. trained model is applied to image tiles of a patient). Shaul discloses the neural network was trained to learn features descriptive of the tissue area depicted in the image tile (i.e. histology features) for classification ([0262] and [0278], e.g. feature extraction module is part of the MIL-program; [0223], e.g. aim of learning is to identify features of instances which are predictive of class membership), and by adding random Gaussian blur to the tumor image tiles ([0336]-[0338], e.g. creation of additional training tiles by adding gaussian blur to image tiles).
Shaul further discloses storing and displaying the classification results overlayed on the respective tile ([0281]-[0282]; FIG. 2, e.g. tile gallery with predictive value of classification overlayed on tiles).
Further regarding claims 1, 10, and 19, Shaul does not disclose the tumors are of human papillomavirus positive (HPV+) head and neck squamous cell carcinoma (HNSCC) tumors. Shaul further does not disclose, the labels of the tumor image tiles are indicators of tumor recurrence or that the neural network classifies tumor recurrence.
However, Shaul discloses the MIL-program is a binary MIL-program that uses a “positive class” and a “negative class” ([0031]), and discloses the classes may relate to a cancer patient showing a particular morphological state, a cancer patient showing a response to therapy, a cancer patient showing metastases, etc. ([0122]), demonstrating the image classification is applicable to a broad range of positive/negative class labels.
Furthermore, these limitations were known in the art, before the effective filing date of the claimed invention, as shown by Raedt.
Regarding claims 1 and 10¸ Raedt discloses a method for histological image analysis (Abstract), which comprises generating image tiles for histological images of histopathological samples containing cancer (i.e. tumor) samples and processing the image tiles in a neural network for classification of a “ground-truth” (claim 1; [0021]; [0150]-[0151; [0365]-[0366]). Raedt discloses the classification may be for the recurrence or the absence of recurrence of the cancer ([0698]; [0709]; [0713], e.g. ground truth label is recurrence or no recurrence). Raedt further discloses the histological specimens used in the invention comprise materials of subjects having a metastatic squamous neck cancer with occult primary cancer (head and neck cancer) (i.e. head and neck squamous cell carcinoma tumor) ([0407]), and that human papillomaviruses (HPVs) infections in people cause oropharyngeal cancers ([0638]; [0640]), demonstrating the cancer may be an HPV+ HNSCC tumor.
It would have been prima facie obvious, to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the method of Shaul, to have obtained tumor images from HPV+ HNSCC tumors and trained the neural network to predict tumor recurrence using training data with ground-truth labels indicating tumor recurrence, as shown by Raedt above. One of ordinary skill in the art would have been motivated to combine the methods of Shaul and Raedt in order to define a subgroup of patients at higher risk of cancer recurrence to lower the risk-benefit ratio of adjuvant chemotherapy, as shown by Raedt ([0004]). There would have been a reasonable expectation of success given Shaul discloses the neural network may be used to classify various aspects of cancer tissues and both Shaul and Raedt utilize a neural network to classify image tiles from tumor images.
Regarding the dependent claims:
Regarding claims 2 and 11, Shaul discloses the tumor image tiles are of hematoxylin and eosin stained tumors ([0231]; [0234]).
Regarding claims 3 and 12, Shaul discloses normalizing the mean and std of the color space channels (i.e. normalizing pixel data) in the tumor image tiles used for training ([0288])
Regarding claims 5-6 and 14-15¸ Shaul discloses the data augmentation includes random horizontal or vertical flips (i.e. rotation by 180 degrees) of the image tiles ([0376]).
Regarding claims 8 and 17, Shaul discloses sampling image tiles for training such that the same number of tiles/tissue pattern examples from different types of clusters is drawn, thereby making the training data set more balanced ([0351]).
Regarding claims 9 and 18, Shaul discloses the neural network is a deep convolution neural network ([0160]-[0161]; [0289]-[0290], e.g. Resnet 50 used as the MIL-program).
Regarding claim 21, the limitation regarding the training the neural network comprising applying random image compression is alternative limitation that is not required by the claim. As applied above to claim 1, Shaul discloses the training comprises adding random Gaussian blur to the tumor image tiles.
Regarding claim 22¸ Shaul discloses randomly selecting a subset of tiles of the tissue images, and generating image artifacts including Guassian blur to the randomly selected subset ([0337]).
Therefore the invention is prima facie obvious.
Claims 4, 13, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Shaul in view of Raedt, as applied to claims 1 and 10 above, and further in view of Shaban (2020).
Cited reference: Shaban, Spatial Context in Computational Pathology, 2020, University of Warwick, pg. 1-30.
Regarding claims 4, 13, and 23, Shaul in view of Raedt disclose the methods of claims 1 and 10 as applied above.
Further regarding claims 4 and 13, Shaul further discloses the neural network pre-trained on imageNet (i.e. weights initialized using pretraining) ([0292], e.g. pre-training; [0294], e.g. neural network has weights).
Shaul in view of Raedt, as applied to claims 1 and 10 above, do not disclose the following:
Further regarding claims 4 and 13, Shaul in view of Raedt, as applied to claims 1 and 10 above, do not disclose the neural network is trained by applying an implementation of an Xception neural network.
However, Shaban discloses a method for processing histology images using a neural network by splitting the images into patches (i.e. tiles) (Abstract; pg. 48, para. 2), and discloses there are many state-of-the art supervised image classifiers, such as LR-CNN such as ResNet50, MobileNet, Inception-v3, or Xception (pg. 29, para. 3; pg. 38, para. 2) . Shaban further discloses that Xception networks use separable convolutions which result in a significant reduction in computational complexity, and further shows consistent performance across three folds of the dataset (pg. 37, para. 2; Table 3.3.).
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the convolution neural network of Shaul in view of Raedt, as applied to claims 1 and 10 above, to have used an Xception based convolution neural network as shown by Shaban above (pg. 29, para. 3; pg. 38, para. 2). One of ordinary skill in the art would have been motivated to combine the methods of Shaul in view of Raedt with Shaban in order to significantly reduce the computational complexity of the neural network while achieving consistent classification performance, as shown by Shaban (pg. 37, para. 2; Table 3.3.). This modification would have had a reasonable expectation of success because both Shaul and Shaban use patch-based image classifiers for classification of histology images, Shaban discloses various supervised image classifiers can be used for the same problem (pg. 29, para. 3), and thus the neural network of Shaban is applicable to the method of Shaul.
Regarding claim 23, Shaul in view of Raedt, as applied to claim 1 above, does not disclose the tumor image tiles comprise virtual tile sub-images of tumor areas with reduced focal depth.
However, Shaban further discloses that histology images are large images which do not fit in the memory of a graphical processing unit, and that histology images have multiple resolutions where each resolution is the downsampled version of the highest resolution with a certain downsampling factor (i.e. images have areas with reduced focal depth (pg. 15, para. 2 to pg. 16, para. 2). Shaban discloses many studies leverage the multi-resolution nature of histology images and use patches at both 20x and 10x for classification, wherein lower resolution 10x patches contain 4 times larger context information compared to higher resolution patches (i.e. patches with lower focal depth) (pg. 16, para. 2). Shaban discloses the use of multi-resolution patches results in better classification performance compared to single resolution based convolution neural network classifiers (pg. 16, para. 2). Shaban further discloses the classification of normal versus malignant epithelium in head and neck squamous cell carcinoma (HNSCC) requires a broader spatial context, and the use of lower resolution/magnification patches is the most straightforward approach to increase the spatial context within input patches (pg. 78, para. 1).
It would have been prima facie obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the method of Shaul in view of Raedt, as applied to claim 1 above, to have used tumor image tiles comprising sub-areas of tumor areas with reduced focal depth as shown by Shaban above. One of ordinary skill in the art would have been motivated to combine the methods of Shaul in view of Raedt with Shaban in order to incorporate image patches with larger contextual information, thus improving CNN classification performance compared to using a single resolution, as shown by Shaban (pg. 16, para. 2; pg. 78, para. 1). This modification would have had a reasonable expectation of success given Shaban discloses broader contextual information is important for classifying HNSCC (pg. 78, para. 1), and therefore the multi-resolution patches of Shaban are applicable to the neural network classification method of HNSCC of Shaul in view of Raedt, as applied to claim 1 above.
Therefore, the invention is prima facie obvious.
Conclusion
No claims are allowed.
Claims 1-6, 8-15, 17-19, and 21-23 are patent eligible for the following reasons. Claims 1-6, 8-15, 17-19, and 21-23 do not recite an abstract idea, but recite a natural correlation between histology features of tumor images and a likelihood of tumor recurrence. However, claim 1 recites additional elements of generating tumor image tiles from the tumor images, applying random image compression or adding random Gaussian blur to the image tiles, and training the neural network on the tumor image tiles. Claims 10 and 19 similarly recite the additional elements of obtaining image tiles and applying a trained neural network to the image tiles for classification, wherein the neural network was trained as described for claim 1. Applicant’s specification at para. [0132] discloses the use of random compression and gaussian blur in the training process improves the generalizability of the model to slides that are slightly out of focus. Therefore, the additional elements of training the neural network (claim 1) and applying the trained neural network for image classification (claims 10 and 19) are directed to an improved method of tumor image classification, rather than to the recited law of nature.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAITLYN L MINCHELLA whose telephone number is (571)272-6485. The examiner can normally be reached 7:00 - 4:00 M-Th.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Olivia Wise can be reached at (571) 272-2249. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KAITLYN L MINCHELLA/Primary Examiner, Art Unit 1685
Prosecution Insights