DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-12 and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 2022/0351347 A1) in view of Tong et al. (US 2021/0241122 A1).
With respect to Claim 1, Yang’347 shows one or more computer-readable non-transitory storage media (figure 1 170) comprising instructions executable by one or more processors (160) of a digital pathology image processing system (10 paragraph [0040]) for:
receiving a plurality of digital pathology images (paragraph [0040] interpreting image data from the plurality of raw images acquired during image acquisition process and step 7 generating a representative image from a pathology image (the reconstructed image from steps 2-4)) of histologic samples (paragraphs [0033] and [0104]-[0106] archived clinical histologic/cytologic slides from available databases may be used to take raw images);
assessing physical characteristics of a first histologic sample (paragraphs [0033] and [0104]-[0106]) associated with a first digital pathology image of the plurality of digital pathology images (paragraph [0040] step 8 enumerate/assess abnormalities (tumor cells paragraph [0083]) in the sample from raw and pathology images, paragraph [0190] describes identifying spatial/physical relationships between normal and abnormal tissue); [ ];
performing a [ ] segmentation of the first digital pathology image (paragraph [0040] step 10) based on one or more regions of the first digital pathology image corresponding to one or more predetermined histologic features (paragraph [0108] a demarcated or segmented image may appear to have clearly defined or delineated boundaries (regions) drawn over the image to identify the portions of interest such as viable tumor cells (predetermined histologic features));
generating an assessment regarding a specified condition in the first histologic sample based on the one or more regions of the first digital pathology image corresponding to tumor bed and the one or more regions of the first digital pathology image corresponding to the one or more predetermined histologic features (paragraph [0040] steps 11-12 regard using the segmented image for training a neural network, figure 15 and paragraph [0138] training of the neural network (DCNN 1504) to output the segmented image 1506 with assessment 1508 for, paragraph [0142] image analysis including diagnostic indicator 1516); and
generating a user interface comprising a display of the assessment (paragraph [0142] the diagnostic indicator 1516 may be provided to a user on an interface).
Yang’347 does not specifically show performing a first segmentation of the first digital pathology image based on one or more regions of the first digital pathology image corresponding to tumor bed.
Tong’122 shows performing a first segmentation of the first digital pathology image based on one or more regions of the first digital pathology image corresponding to tumor bed (Figure 1 and paragraph [0027] annotated tissue sample images 111 used for training a prediction model 110. Paragraph [0033] automatically segmenting and classifying areas including tumor bed detection. Paragraph [0034] the segmentation and classification is performed by leverage masks of image bands wherein each image band is a set of image bands representing a corresponding cell type which requires identifying the area via different image bands. Paragraph [0050] multiple masks for multiple image bands each for segmentation including tumor bed detection.); and performing a second segmentation of the first digital pathology image based on one or more regions of the first digital pathology image corresponding to one or more predetermined histologic features (Paragraph [0033] automatically segmenting and classifying areas including predetermined histologic features such as necrosis, stomal tissue, or any other cell type. Paragraph [0034] the segmentation and classification is performed by leverage masks of image bands wherein each image band is a set of image bands representing a corresponding cell type which requires identifying the area via different image bands. Paragraph [0050] multiple masks for multiple image bands each for segmentation including predetermined histologic features such as necrosis, stomal tissue, or any other cell type.).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claim invention to modify Yang’347 to include performing a first segmentation of the first digital pathology image based on one or more regions of the first digital pathology image corresponding to tumor bed method taught by Tong’122. The suggestion/motivation for doing so would have been to improve the system’s ability to be able to have a singular step of plurality of segmentations and classifying areas (paragraphs [0034] and [0050]).
With respect to Claim 2, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein performing the first segmentation of the first digital pathology image based on one or more regions of the first digital pathology image corresponding to tumor bed is performed using a machine-learning model trained to segment tumor bed from non-tumor bed (in Tong’122: paragraph [0020] segmentation task of areas of interest and the classification task in a single step using a generative adversarial network (or GAN) using masks covering each region of interest. The invention similarly employs a GAN to perform both segmentation and classification by using a separate image band for each class).
With respect to Claim 3, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein performing the second segmentation of the first digital pathology image based on one or more regions of the first digital pathology image corresponding to the one or more predetermined histologic features is performed using a machine-learning model trained to segment the one or more predetermined histologic features from tumor bed (in Yang’347: paragraph [0040] steps 11-12 regard using the segmented image for training a neural network, figure 15 and paragraph [0138] training of the neural network (DCNN 1504) to output the segmented image 1506 with assessment 1508 for, paragraph [0142] image analysis including diagnostic indicator 1516 and in Tong’122: paragraph [0020] segmentation task of areas of interest and the classification task in a single step using a generative adversarial network (or GAN) using masks covering each region of interest. The invention similarly employs a GAN to perform both segmentation and classification by using a separate image band for each class).
With respect to Claim 4, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 3, further comprising instructions executable by one or more processors of the digital pathology image processing system for: receiving, by the digital pathology image processing system, feedback from a user operator regarding the assessment; and training the machine-learning model trained to segment the one or more predetermined histologic features from tumor bed based on the feedback (in Yang’347: paragraph [0108] a human annotator can annotate training images in a training dataset to identify and/or enumerate one or more portions of interest. For example, an annotator may identify and/or enumerate viable tumor cells or collections of viable tumor cells. The annotation may serve as the gold standard, or ground truth, for training a model).
With respect to Claim 5, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein the one or more predetermined histologic features comprise necrotic tumor cells, regions of necrosis, viable tumor cells (in Yang’347: tumor cells paragraph [0083]), regions of viable tumor (in Yang’347: paragraph [0040] step 8 enumerate/assess abnormalities (tumor cells paragraph [0083]) in the sample from raw and pathology images, paragraph [0190] describes identifying spatial/physical relationships between normal and abnormal tissue), tumor stroma cells, or regions of tumor stroma.
With respect to Claim 6, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein generating the assessment regarding the specified condition comprises determining whether the specified condition is present (in Yang’347: figure 15 1512 paragraph [0140]).
With respect to Claim 7, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, further comprising instructions executable by one or more processors of the digital pathology image processing system for: computing a first area value of the first histologic sample corresponding to tumor bed based on the one or more regions of the first digital pathology image corresponding to tumor bed and the physical characteristics of the first histologic sample; and computing a second area value of the first histologic sample corresponding to each of the one or more predetermined histologic features based on the one or more regions of the first digital pathology image corresponding to the one or more predetermined histologic features and the physical characteristics of the first histologic sample; wherein the assessment regarding the specified condition in the first histologic sample is generated based on the first area value and the second area value (in Yang’347: figure 15 paragraph [0138] segmented image 1506 comprises demarcated area 1508 wherein boundary lines are determined based on a probability of pixel associated with the boundary line exceeding a threshold compared to other pixels, this describes pixels of the first area/demarcated area 1508 exceed the threshold compared with pixels outside of the demarcated area 1508 falling below the threshold to identify the abnormality).
With respect to Claim 8, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 7, wherein determining whether the specified condition is detected in the first histologic sample based on the first area value and the second area value comprises computing a percentage of the second area relative to the first area corresponding to each of the one or more predetermined histologic features (in Yang’347: paragraphs [0088] and [0141] describes generating a percentage coverage metric of the viable tumor).
With respect to Claim 9, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 8, wherein determining whether the specified condition is detected in the first histologic sample based on the first area value and the second area value comprises determining whether the percentage of the second area value relative to the first area value satisfies one or more predetermined thresholds, wherein the one or more predetermined thresholds are based on the specified condition (in Yang’347: paragraph [0138] segmented image 1506 comprises demarcated area 1508 wherein boundary lines are determined based on a probability of pixel associated with the boundary line exceeding a threshold compared to other pixels, paragraph [0141] describes the diagnostic indicator is achieved based on the percentage area coverage describing a relationship between the two).
With respect to Claim 10, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein: performing the first segmentation of the first digital pathology image based on the one or more regions of the first digital pathology image corresponding to tumor bed comprises producing a first instance of the first digital pathology image including annotations corresponding to the regions of the first digital pathology image corresponding to tumor bed (in Tong’122: Figure 1 and paragraph [0027] annotated tissue sample images 111 used for training a prediction model 110. Paragraph [0033] automatically segmenting and classifying areas including tumor bed detection. Paragraph [0034] the segmentation and classification is performed by leverage masks of image bands wherein each image band is a set of image bands representing a corresponding cell type which requires identifying the area via different image bands. Paragraph [0050] multiple masks for multiple image bands each for segmentation including tumor bed detection.); and performing the second segmentation of the first digital pathology image based on the one or more regions of the first digital pathology image corresponding to the one or more predetermined histologic features comprises producing a second instance of the first digital pathology image including annotations corresponding to the regions of the first digital pathology image corresponding to the one or more predetermined histologic features (in Yang’347: paragraphs [0033] and [0104]-[0106] archived clinical histologic/cytologic slides from available databases may be used to take raw images, paragraph [0040] steps 11-12 regard using the segmented image for training a neural network, figure 15 and paragraph [0138] training of the neural network (DCNN 1504) to output the segmented image 1506 with assessment 1508).
With respect to Claim 11, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein the first histologic sample is further associated with a set of one or more second digital pathology images (in Yang’347: paragraphs [0033] and [0104]-[0106] archived clinical histologic/cytologic slides from available databases may be used to take raw (digital) images; the interpretation here for the second digital pathology image regards the trained model analysis wherein the first digital pathology images regards training the model); wherein the one or more computer-readable non-transitory storage media further comprising instructions executable by one or more processors of the digital pathology image processing system for, for each of the second digital pathology images: assessing physical characteristics of the first histologic sample associated with the second digital pathology image (in Yang’347: figure 15 image analysis staring with 1512 paragraph [0140] assess abnormalities (tumor cells paragraph [0083]) in the sample from raw and pathology images in application outside of training the model as disclosed in paragraph [0040], paragraph [0190] describes identifying spatial/physical relationships between normal and abnormal tissue); performing a first segmentation of the second digital pathology image based on one or more regions of the second digital pathology image corresponding to tumor bed (in Tong’122: Figure 1 and paragraph [0027] annotated tissue sample images 111 used for training a prediction model 110. Paragraph [0033] automatically segmenting and classifying areas including tumor bed detection. Paragraph [0034] the segmentation and classification is performed by leverage masks of image bands wherein each image band is a set of image bands representing a corresponding cell type which requires identifying the area via different image bands. Paragraph [0050] multiple masks for multiple image bands each for segmentation including tumor bed detection.); and performing a second segmentation of the second digital pathology image based on one or more regions of the second digital pathology image corresponding to the one or more predetermined histologic features (in Yang’347: paragraph [0040] interpreting image data from the plurality of raw images acquired during image acquisition process and step 7 generating a representative image from a pathology image (the reconstructed image from steps 2-4), paragraphs [0033] and [0104]-[0106] archived clinical histologic/cytologic slides from available databases may be used to take raw images); wherein generating the assessment regarding the specified condition in the first histologic sample is further based on the one or more regions of the set of second digital pathology images corresponding to tumor bed and the one or more regions of the set of second digital pathology images corresponding to the one or more predetermined histologic features (in Yang’347: paragraph [0040] steps 11-12 regard using the segmented image for training a neural network, figure 15 and paragraph [0138] training of the neural network (DCNN 1504) to output the segmented image 1506 with assessment 1508 for, paragraph [0142] image analysis including diagnostic indicator 1516).
With respect to Claim 12, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, further comprising instructions executable by one or more processors of the digital pathology image processing system for: receiving a human-generated assessment of the first histologic sample (in Yang’347: Figure 15 1510 paragraphs [0101], [0108], and [0139] wherein manual and/or automated annotations for ground truth image); comparing (in Yan’347: 1518) the assessment generated by the first digital pathology image processing system (In Yang’347: 1506) to the human-generated assessment; and generating a user interface comprising a display of the comparison (In Yang’347: paragraph [0142] the diagnostic indicator 1516 may be provided to a user on an interface).
With respect to Claim 14, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, further comprising instructions executable by one or more processors of the digital pathology image processing system for: generating a level of confidence in the assessment (In Yang’347: figure 15 1518 paragraph [0145]).
With respect to Claim 15, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein the user interface comprising the display of the assessment further comprises a display of annotations for the first digital pathology image associated with the segmentations based on the one or more regions of the first digital pathology image corresponding to tumor bed and the one or more regions of the first digital pathology image corresponding to the one or more predetermined histologic features (In Yang’347: figure 1 displays 180 described in paragraph [0042] to display the data of the CRADL system 10 to a pathologist, paragraph [0040] step 6 to display the images on the displays).
With respect to Claims 16 and 19, rejection analogous to those presented for claim 1, are applicable.
With respect to Claims 17 and 20, rejection analogous to those presented for claim 2, are applicable.
With respect to Claim 18, rejection analogous to those presented for claim 3, are applicable.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (US 2022/0351347 A1) in view of Tong et al. (US 2021/0241122 A1) further in view of Peng et al. (US 2021/0019342 A1).
With respect to Claim 13, the combination of Yang’347 and Tong’122 shows the one or more computer-readable non-transitory storage media of claim 1, wherein the assessment regarding the specified condition is further generated based on [metadata] and additional data associated with the first histologic sample (in Yang’347: paragraph [0144] describes successful identification of tumor cells is based on additional data of the sample including accurate delineation with boundaries and percentage area coverage metrics).
Yang’347 and Tong’122 does not specifically disclose the use of metadata.
Peng’342 discloses the use of metadata (paragraph [0041]).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claim invention to modify Yang’347 and Tong’122 to include the use of metadata method taught by Peng’342. The suggestion/motivation for doing so would have been to improve the system’s ability to be able to enabling the model to learn the name of the feature (paragraph [0041]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IRIANA CRUZ whose telephone number is (571)270-3246. The examiner can normally be reached 10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi M. Sarpong can be reached at (571) 270-3438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IRIANA CRUZ/Primary Examiner, Art Unit 2681
Prosecution Insights