DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 07/28/2026 has been entered.
Response to Arguments
Applicant’s arguments see remarks, filed 06/29/2026, with respect to claims 1, 3-11, 13-20 have been considered but are not persuasive.
Applicant argues on page 8-9, “According to amended claim 1, the claimed invention “exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types.” The Examiner alleges that Yip, FIG. 2 and para 0069 discloses “exclude, from a result of the detection, at least one cell, which is not included in a first type of an area, from a first type of cell among the at least one cell included in the at least one tissue.” Applicant respectfully traverses. As highlighted above, Yip discloses that, in counting the number of cancer cells, cells that are located in tumor areas but which belong to cells that are characterized as lymphocytes are excluded. This disclosure of Yip is the opposite to excluding at least one cell, which is not included in an area of a first type (e.g., cancer area) corresponding to a first type of cell (e.g. cancer cell), from the first type of the cell. In other words, the claimed invention determines whether to “exclude” a cell from a first type of cell based on not only the characteristics of the cell itself but also on a spatial/contextual condition – specifically, whether the cell is included in a “an area of the first type among the plurality of types” or not. Yip fails to disclose such feature and only discloses excluding a cell based on the characteristics of the cell. Yip does not disclose exclusion of a cell due to whether the cell is not included in an area of the first type among the plurality of types into which at least one area of the pathological slide image is classified into.”
The office respectfully disagrees. The office finds the term “exclude, from at least one cell of a first type among the detected at least one cell, at least one cell” rendering the claim indefinite. It is not clear what the applicant refers to as being excluded, whether it is a cell or cell type(s), and excluded from which group. For the purposes of examination, the examiner is interpreting the limitation as “exclude at least one cell type which is not included in an area of the first cell type detected among the plurality of types”. Based on the breadth of the claim language in claim 1 and analogous claim 11, and the examiner’s interpretation, YIP et al. (US 20200211189 A1), hereinafter referenced as YIP explicitly teaches and exclude, from at least one cell of a first type among the detected at least one cell (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas but which belong to cells that are characterized as lymphocytes. See also Paragraph [0072].),
at least one cell which is not included in an area of the first type among the plurality of types (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas [wherein tumor areas are an area of the first type] but which belong to cells that are characterized as lymphocytes. Paragraph [0072] – YIP further discloses the digital tissue segmenter 201 may calculate whether the grid areas that are labeled tumor are spatially consolidated or dispersed among non-tumor grid areas. If the digital tissue segmenter 201 determines that the tumor areas are spatially consolidated, the digital tissue segmenter 201 may produce a digital overlay of a recommended cutting boundary that separates the slide regions classified as tumor and the slide regions classified as non-tumor or within the areas classified as non-tumor, proximal to the areas classified as tumor.),
The Applicant further states on page 10, “Applicant respectfully submits that allowance of claim 1 and all claims dependent therefrom is warranted for these reasons.”
The office respectfully disagrees. Based on the breadth of the claim language, claim 1 is rejected 35 U.S.C. 103 as being unpatentable over CHUKKA et al. (US 20200342597 A1), hereinafter referenced as CHUKKA in view of FUCHS (US 20210133966 A1), hereinafter referenced as FUCHS, further in view of YIP (US 20200211189 A1), hereinafter referenced as YIP.
Regarding claim 1, CHUKKA teaches a computing apparatus (Figs. 1 & 2, #200 called a digital pathology system, paragraph [0050]- CHUKKA discloses a digital pathology system 200 for imaging and analyzing specimens is illustrated in Figs. 1 and 2. The digital pathology system 200 may comprise an imaging apparatus 12… and a computer 14) comprising:
at least one memory (Fig. 2, #201 called a memory; Paragraph [0051] – CHUKKA discloses the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201); and
at least one processor (Fig. 2, #209 called processors; Paragraph [0051] – CHUKKA discloses the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201. In paragraph [0052], CHUKKA further discloses the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201, the at least one memory 201 storing non-transitory computer-readable instructions for execution by the one or more processors to cause the one or more processors to execute instructions (or stored data) in a multilayer neural network 220 and in at least one of a ground truth training module 210 or a testing module 230), wherein the at least one processor is configured to:
acquire a pathological slide image (Fig. 3A, step 300 called receive sample image data, Paragraph [0066] – CHUKKA discloses the training of a multilayer neural network comprises the steps of (i) receiving sample image data (step 300) (e.g. using an image acquisition module 202))
showing at least one tissue (Fig. 3A, Paragraph [0067] – CHUKKA further discloses given a sample image, different tissue types and/or cell types may be identified in the sample image);
generate, as feature information, a first image (Figs. 2 & 7, Paragraph [0092] – CHUKKA discloses the image analysis module 207 is run a first time to extract features and classify cells and/or nuclei in a first image; and then run a second time to extract features and classify cells and/or nucleic in a series of addition images. Paragraph [0126] – CHUKKA further discloses the region label image is shown in the bottom right and the cell level label image in the bottom left. The region label image is shown at pixel level and the cell label image shown the cell classification label at the cell level.)
in which at least one area of the pathological slide image is classified into one of a plurality of types (Fig. 3C, Paragraph [0111] – CHUKKA discloses unlabeled image [wherein unlabeled image is the pathological slide image] is supplied to a trained multilayer neural network 220 (step 324) [wherein the multilayer neural network is a first machine learning model]. The skilled artisan will appreciate that the multilayer neural network 220 must have been trained to classify the types of cells present (or suspected to be present) in the unlabeled image. For example, if the network was trained to recognize and classify lymphocytes and tumor cells in a breast cancer sample [wherein lymphocytes and tumor cells are examples of a plurality of cell types], then the unlabeled image must also be of a breast cancer sample. Paragraph [0112] – CHUKKA further discloses a generated foreground mask may be overlaid on top of an unlabeled image, and each of the different predictive pixel labels within any region (i.e. identified cell) may be quantified and then the predictive label having the greatest quantity will be selected as the cell label. For example, if a particular identified cell comprises 100 total pixels, and 55 of those total pixels are predicted to be tumor cell pixels, 40 of those total pixels are predicted to be stromal cells, and 5 of those total pixels are predicted to be lymphocytes, then the cell would be assigned the label of a tumor cell since the majority of the pixels were predicted to be tumor cell pixels. See also Paragraph [0118].),
by analyzing the pathological slide image at a first magnification using a first machine learning model (Fig. 3A, Paragraph [0119] – CHUKKA discloses to train the network, 20 whole slide images @ 20× magnification were used for region and cell annotations (˜20,000 regions, 500,000 cells, 2×108 pixels@ 0.5 μm) [wherein the network is a first machine learning model].);
Although CHUKKA further teaches detect at least one cell included in the at least one tissue (Fig. 3A, Paragraph [0073] – CHUKKA discloses following image acquisition and/or unmixing, input images or unmixed image channel images are provided to a cell detection module 204 to detect cells and subsequently to a cell classification module 205 to classify cells and/or nuclei (step 300). Paragraph [0067] – CHUKKA further discloses given a sample image, different tissue types and/or cell types may be identified in the sample image.)
by analyzing the feature information (Fig. 3A, Paragraph [0073] – CHUKKA discloses the procedures and algorithms described herein may be adapted to identify and classify various types of cells or cell nuclei based on features within the input images. See also Fig. 2A, Paragraph [0092].)
CHUKKA fails to explicitly teach and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
However, FUCHS explicitly teaches and a second image (Fig. 20(a), #2036 called patches, Paragraph [0159] – FUCHS discloses each network 2018 may have one of the patches 2036 at a corresponding magnification factor from one of the tiles 2022 of the biomedical image 2020 as input.),
which is generated based on the pathological slide image (Fig. 20(a), #2020 called biomedical image, Paragraph [0159] – FUCHS discloses each network 2018 may have one of the patches 2036 at a corresponding magnification factor from one of the tiles 2022 of the biomedical image 2020 as input.)
at a second magnification (Fig. 20(a), Paragraph [0159] – FUCHS discloses each network 2018 itself may correspond to or be associated with one of the magnification factors. For example, the first network 2018A may be associated with the first magnification factor (e.g., 20×), the second network 2018B may be associated with the second magnification factor (e.g., 10×), and the third network 2018C may be associated with the third magnification factor (e.g., 20×), and so forth.),
using a second machine learning model (Fig. 20(d), Paragraph [0171] – FUCHS discloses the segmentation model may include a set of networks (e.g. networks 2018) corresponding to the set of magnification factors used to create the patches. Each network may include a set of encoders (e.g., a convolution block 2032) and a set of decoders (e.g., a deconvolution block 2040).);
wherein a field of view of the second magnification is narrower than a field of view of the first magnification (Fig. 20(a), Paragraph [0159] – FUCHS discloses the first network 2018A may be associated with the first magnification factor (e.g., 20×), the second network 2018B may be associated with the second magnification factor (e.g., 10×), and the third network 2018C may be associated with the third magnification factor (e.g., 20×), and so forth [wherein the first magnification factor (e.g., 20×) is narrower than a field of view of the second magnification factor (e.g., 10x)].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; generate, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; detect at least one cell included in the at least one tissue by analyzing the feature information with the teachings of FUCHS having and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
Wherein having CHUKKA’s computing apparatus wherein having and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
CHUKKA in view of FUCHS fail to explicitly teach and exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types,
However, YIP explicitly teaches and exclude, from at least one cell of a first type among the detected at least one cell (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas but which belong to cells that are characterized as lymphocytes. See also Paragraph [0072].),
at least one cell which is not included in an area of the first type among the plurality of types (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas [wherein tumor areas are an area of the first type] but which belong to cells that are characterized as lymphocytes. See also Paragraph [0072].),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA in view of FUCHS of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; generate, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; detect at least one cell included in the at least one tissue by analyzing the feature information and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification with the teachings of YIP having and exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types,
Wherein having CHUKKA’s computing apparatus wherein having and exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types,
The motivation behind the modification would have been to obtain a digital pathology system that considers the whole context of the pathological slide image, since both CHUKKA and YIP relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while YIP has techniques for generating an overlay map on a digital medical image of a slide that may be useful for predicting patient survival in that overlays of such status information can provide an illustrated estimate of how well a patient will respond to certain immunotherapies. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and YIP (US 20200211189 A1), Paragraph [0079].
The applicant further argues on page 10, “Allowance of independent claim 11 is warranted for reasons similar to those provided with respect to claim 1. In view of the above, reconsideration and allowance of this application are now believed to be in order, and such actions are hereby solicited.”
In response, the office does not find this argument persuasive based on the same reasons set forth above for analogous claim 1 and the rejection below and therefore encourages the applicant to amend the claims to overcome the prior arts of record.
Claim Objections
Claim 11 is objected to because of the following informality:
In claim 11, line 11, “at least one cell which is not included in area of the first type” should read “at least one cell which is not included in an area of the first type” in order to be grammatically correct.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 and 11, and associated dependent claims 3-10 and 13-20, are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 1 recites the limitation “exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types,” in lines 11-14. Similarly, claim 11 recites the limitation “excluding, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in area of the first type among the plurality of types,” in lines 10-12. The office finds the term “exclude, from at least one cell of a first type among the detected at least one cell, at least one cell” rendering the claim indefinite. It is not clear what the applicant refers to as being excluded, whether it is a cell or cell type(s), and excluded from which group. For the purposes of examination, the examiner is interpreting the limitation as “exclude at least one cell type which is not included in an area of the first cell type detected among the plurality of types”. The office respectfully requests the Applicant to amend claims 1 and 11 in order to clarify the claimed invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-7, 11, 13-17, and 20 are rejected 35 U.S.C. 103 as being unpatentable over CHUKKA et al. (US 20200342597 A1), hereinafter referenced as CHUKKA in view of FUCHS (US 20210133966 A1), hereinafter referenced as FUCHS, further in view of YIP (US 20200211189 A1), hereinafter referenced as YIP.
Regarding claim 1, CHUKKA teaches a computing apparatus (Figs. 1 & 2, #200 called a digital pathology system, paragraph [0050]- CHUKKA discloses a digital pathology system 200 for imaging and analyzing specimens is illustrated in Figs. 1 and 2. The digital pathology system 200 may comprise an imaging apparatus 12… and a computer 14) comprising:
at least one memory (Fig. 2, #201 called a memory; Paragraph [0051] – CHUKKA discloses the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201); and
at least one processor (Fig. 2, #209 called processors; Paragraph [0051] – CHUKKA discloses the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201. In paragraph [0052], CHUKKA further discloses the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201, the at least one memory 201 storing non-transitory computer-readable instructions for execution by the one or more processors to cause the one or more processors to execute instructions (or stored data) in a multilayer neural network 220 and in at least one of a ground truth training module 210 or a testing module 230), wherein the at least one processor is configured to:
acquire a pathological slide image (Fig. 3A, step 300 called receive sample image data, Paragraph [0066] – CHUKKA discloses the training of a multilayer neural network comprises the steps of (i) receiving sample image data (step 300) (e.g. using an image acquisition module 202))
showing at least one tissue (Fig. 3A, Paragraph [0067] – CHUKKA further discloses given a sample image, different tissue types and/or cell types may be identified in the sample image);
generate, as feature information, a first image (Figs. 2 & 7, Paragraph [0092] – CHUKKA discloses the image analysis module 207 is run a first time to extract features and classify cells and/or nuclei in a first image; and then run a second time to extract features and classify cells and/or nucleic in a series of addition images. Paragraph [0126] – CHUKKA further discloses the region label image is shown in the bottom right and the cell level label image in the bottom left. The region label image is shown at pixel level and the cell label image shown the cell classification label at the cell level.)
in which at least one area of the pathological slide image is classified into one of a plurality of types (Fig. 3C, Paragraph [0111] – CHUKKA discloses unlabeled image [wherein unlabeled image is the pathological slide image] is supplied to a trained multilayer neural network 220 (step 324) [wherein the multilayer neural network is a first machine learning model]. The skilled artisan will appreciate that the multilayer neural network 220 must have been trained to classify the types of cells present (or suspected to be present) in the unlabeled image. For example, if the network was trained to recognize and classify lymphocytes and tumor cells in a breast cancer sample [wherein lymphocytes and tumor cells are examples of a plurality of cell types], then the unlabeled image must also be of a breast cancer sample. Paragraph [0112] – CHUKKA further discloses a generated foreground mask may be overlaid on top of an unlabeled image, and each of the different predictive pixel labels within any region (i.e. identified cell) may be quantified and then the predictive label having the greatest quantity will be selected as the cell label. For example, if a particular identified cell comprises 100 total pixels, and 55 of those total pixels are predicted to be tumor cell pixels, 40 of those total pixels are predicted to be stromal cells, and 5 of those total pixels are predicted to be lymphocytes, then the cell would be assigned the label of a tumor cell since the majority of the pixels were predicted to be tumor cell pixels. See also Paragraph [0118].),
by analyzing the pathological slide image at a first magnification using a first machine learning model (Fig. 3A, Paragraph [0119] – CHUKKA discloses to train the network, 20 whole slide images @ 20× magnification were used for region and cell annotations (˜20,000 regions, 500,000 cells, 2×108 pixels@ 0.5 μm) [wherein the network is a first machine learning model].);
Although CHUKKA further teaches detect at least one cell included in the at least one tissue (Fig. 3A, Paragraph [0073] – CHUKKA discloses following image acquisition and/or unmixing, input images or unmixed image channel images are provided to a cell detection module 204 to detect cells and subsequently to a cell classification module 205 to classify cells and/or nuclei (step 300). Paragraph [0067] – CHUKKA further discloses given a sample image, different tissue types and/or cell types may be identified in the sample image.)
by analyzing the feature information (Fig. 3A, Paragraph [0073] – CHUKKA discloses the procedures and algorithms described herein may be adapted to identify and classify various types of cells or cell nuclei based on features within the input images. See also Fig. 2A, Paragraph [0092].)
CHUKKA fails to explicitly teach and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
However, FUCHS explicitly teaches and a second image (Fig. 20(a), #2036 called patches, Paragraph [0159] – FUCHS discloses each network 2018 may have one of the patches 2036 at a corresponding magnification factor from one of the tiles 2022 of the biomedical image 2020 as input.),
which is generated based on the pathological slide image (Fig. 20(a), #2020 called biomedical image, Paragraph [0159] – FUCHS discloses each network 2018 may have one of the patches 2036 at a corresponding magnification factor from one of the tiles 2022 of the biomedical image 2020 as input.)
at a second magnification (Fig. 20(a), Paragraph [0159] – FUCHS discloses each network 2018 itself may correspond to or be associated with one of the magnification factors. For example, the first network 2018A may be associated with the first magnification factor (e.g., 20×), the second network 2018B may be associated with the second magnification factor (e.g., 10×), and the third network 2018C may be associated with the third magnification factor (e.g., 20×), and so forth.),
using a second machine learning model (Fig. 20(d), Paragraph [0171] – FUCHS discloses the segmentation model may include a set of networks (e.g. networks 2018) corresponding to the set of magnification factors used to create the patches. Each network may include a set of encoders (e.g., a convolution block 2032) and a set of decoders (e.g., a deconvolution block 2040).);
wherein a field of view of the second magnification is narrower than a field of view of the first magnification (Fig. 20(a), Paragraph [0159] – FUCHS discloses the first network 2018A may be associated with the first magnification factor (e.g., 20×), the second network 2018B may be associated with the second magnification factor (e.g., 10×), and the third network 2018C may be associated with the third magnification factor (e.g., 20×), and so forth [wherein the first magnification factor (e.g., 20×) is narrower than a field of view of the second magnification factor (e.g., 10x)].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; generate, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; detect at least one cell included in the at least one tissue by analyzing the feature information with the teachings of FUCHS having and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
Wherein having CHUKKA’s computing apparatus wherein having and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
CHUKKA in view of FUCHS fail to explicitly teach and exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types,
However, YIP explicitly teaches and exclude, from at least one cell of a first type among the detected at least one cell (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas but which belong to cells that are characterized as lymphocytes. See also Paragraph [0072].),
at least one cell which is not included in an area of the first type among the plurality of types (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas [wherein tumor areas are an area of the first type] but which belong to cells that are characterized as lymphocytes. See also Paragraph [0072].),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA in view of FUCHS of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; generate, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; detect at least one cell included in the at least one tissue by analyzing the feature information and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification with the teachings of YIP having and exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types,
Wherein having CHUKKA’s computing apparatus wherein having and exclude, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in an area of the first type among the plurality of types,
The motivation behind the modification would have been to obtain a digital pathology system that considers the whole context of the pathological slide image, since both CHUKKA and YIP relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while YIP has techniques for generating an overlay map on a digital medical image of a slide that may be useful for predicting patient survival in that overlays of such status information can provide an illustrated estimate of how well a patient will respond to certain immunotherapies. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and YIP (US 20200211189 A1), Paragraph [0079].
Regarding claim 3, CHUKKA and FUCHS in view of YIP teach the computing apparatus of claim 1,
CHUKKA further teaches wherein the one of the plurality of types includes at least one of a cancer area (Fig. 6, Paragraph [0007] – CHUKKA discloses the present disclosure is a method of training a multilayer neural network to detect and classify different cell types (e.g. tumor cells, stromal cells, lymphocytes, etc.) and tissue regions (e.g. tumor regions [wherein tumor regions are a cancer area], lymphocyte-rich regions, stromal regions, necrotic regions [wherein lymphocyte-rich/stromal/necrotic regions are a plurality of types], etc.) within a sample image (e.g. a sample image of a biological sample tissue having one or more stains)),
a cancer stroma area (Fig. 6, Paragraph [0007] – CHUKKA discloses the present disclosure is a method of training a multilayer neural network to detect and classify different cell types (e.g. tumor cells, stromal cells, lymphocytes, etc.) and tissue regions (e.g. tumor regions, lymphocyte-rich regions, stromal regions, necrotic regions, etc.) within a sample image (e.g. a sample image of a biological sample tissue having one or more stains)),
a necrosis area (Fig. 6, Paragraph [0007] – CHUKKA discloses the present disclosure is a method of training a multilayer neural network to detect and classify different cell types (e.g. tumor cells, stromal cells, lymphocytes, etc.) and tissue regions (e.g. tumor regions, lymphocyte-rich regions, stromal regions, necrotic regions, etc.) within a sample image (e.g. a sample image of a biological sample tissue having one or more stains)),
and a background area (Fig. 4B, Paragraph [0065] – CHUKKA discloses a tissue region mask may be created by identifying the tissue regions and automatically or semi-automatically excluding the background regions (e.g. regions of a whole slide image corresponding to glass with no sample, such as where there exists only white light from the imaging source).).
Regarding claim 4, CHUKKA and FUCHS in view of YIP teach the computing apparatus of claim 1,
Although CHUKKA further teaches wherein the at least one processor (Fig. 2, #209 called processors; Paragraph [0051])
is further configured to detect the at least one cell (Fig. 3C, Paragraph [0112]- CHUKKA discloses using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325) [wherein identified denotes detecting])
CHUKKA fails to explicitly teach on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image.
However, YIP explicitly teaches on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image (Fig. 3A, 3B, Paragraph [0062] - YIP discloses FIG. 3A illustrates a tissue class overlay map created by the overlay map generator of the digital tissue segmenter 201. FIG. 3B illustrates a cell outer edge overlay map created by the overlay map generator of the digital tissue segmenter 201. The overlay map generator may display the digital overlays as transparent or opaque layers that cover the slide image [wherein the layers that cover the slide image are a third image, wherein the slide image is a first image, and the digital overlay is a fourth image], aligned such that the slide location shown in the overlay and the slide image are in the same location on the display).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; generate, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, with the teachings of YIP having on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image.
Wherein having CHUKKA’s computing apparatus wherein having on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image.
The motivation behind the modification would have been to obtain a digital pathology system that considers the whole context of the pathological slide image, since both CHUKKA and YIP relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while YIP has techniques for generating an overlay map on a digital medical image of a slide that may be useful for predicting patient survival in that overlays of such status information can provide an illustrated estimate of how well a patient will respond to certain immunotherapies. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and YIP (US 20200211189 A1), Paragraph [0079].
Regarding claim 5, CHUKKA and FUCHS in view of YIP teach the computing apparatus of claim 1,
CHUKKA further teaches wherein the at least one processor (Fig. 2, #209 called processors; Paragraph [0051])
is further configured to detect the at least one cell (Fig. 3C, Paragraph [0112] - CHUKKA discloses using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325))
by using the first image in at least one of intermediate operations (Fig. 3C, Paragraph [0112]- CHUKKA discloses using the pixel level predictive classifications [wherein the first image is pixel level predictive classifications], a label is then assigned to each identified cell (step 325))
of the second machine learning model (Fig. 3C, step 324 called apply a trained multi-layer neural network [wherein multi-layer neural network is the second machine learning model] to provide predictive labels for each pixel within the unlabeled image)
that uses the pathological slide image as an input (Fig. 3C, step 320 called receive an unlabeled image [wherein unlabeled image is the pathological slide image]).
Regarding claim 6, CHUKKA and FUCHS in view of YIP teach the computing apparatus of claim 1,
CHUKKA further teaches wherein the at least one processor (Fig. 2, #209 called processors; Paragraph [0051]) is further configured to detect the at least one cell (Fig. 3C, Paragraph [0112])
CHUKKA fails to explicitly teach by mutually using information generated from at least one of intermediate operations of the first machine learning model and information generated from at least one of intermediate operations of the second machine learning model in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model.
However, FUCHS explicitly teaches by mutually using information generated from at least one of intermediate operations of the first machine learning model (Fig. 20(e), Paragraph [0145] – FUCHS discloses the terminal convolution block 2046 may have a set of feature maps 2038 as input. The set of transform layers 2048A-N of the terminal convolution block 2046 may be applied to the input, such as the set of feature maps 2038′, in any sequence (such as the one depicted), outputted by one of the networks 2018 [wherein one of the networks 2018 is the first machine learning model]. The set of feature maps 2038′ may be the resultant output of one of the networks 2018 from processing one of the patches 2036 and other input feature maps 2038 inputted to the network 2018.)
and information generated from at least one of intermediate operations of the second machine learning model (Fig. 20(e), Paragraph [0145] – FUCHS discloses the terminal convolution block 2046 may have a set of feature maps 2038 as input. The set of transform layers 2048A-N of the terminal convolution block 2046 may be applied to the input, such as the set of feature maps 2038′, in any sequence (such as the one depicted), outputted by one of the networks 2018 [wherein one of the networks 2018 is the second machine learning model]. The set of feature maps 2038′ may be the resultant output of one of the networks 2018 from processing one of the patches 2036 and other input feature maps 2038 inputted to the network 2018.)
in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model (Fig. 20(f), Paragraph [0151] – FUCHS discloses the network 2018 may have one of the patches 2036 of a tile 2022 in the biomedical image 2020 (depicted generally to the left) and set of feature maps 2038′ [wherein feature maps 2038’ is at least one of the intermediate operations] outputted from other networks 2018 (depicted generally below) as an input.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; detect at least one cell included in the at least one tissue with the teachings of FUCHS having by mutually using information generated from at least one of intermediate operations of the first machine learning model and information generated from at least one of intermediate operations of the second machine learning model in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model.
Wherein having CHUKKA’s computing apparatus wherein having by mutually using information generated from at least one of intermediate operations of the first machine learning model and information generated from at least one of intermediate operations of the second machine learning model in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
Regarding claim 7, CHUKKA and FUCHS in view of YIP teach the computing apparatus of claim 1,
Although CHUKKA further teaches and the generated data includes at least one of a first patch magnified at the first magnification, at least one annotated tissue based on the first patch (Fig. 1, Paragraph [0119] - CHUKKA discloses to train the network, 20 whole slide images @ 20× magnification were used for region and cell annotations (˜20,000 regions, 500,000 cells, 2×108 pixels @ 0.5 μm)),
CHUKKA fails to explicitly teach wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image, a second patch magnified at the second magnification, at least one annotated cell based on the second patch, and information regarding a positional relationship between the first patch and the second patch.
However, FUCHS explicitly teaches wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image (Fig. 2, Paragraph [0065] – FUCHS discloses to train the multi-class segmentation DMMN, patches are extracted from whole slide images and the corresponding annotations (thus similar/consistent to applicants’ disclosure in the specification, Paragraph [0060] "However, the first machine learning model and the second machine learning model may be the same machine learning model or may be separate machine learning models.")),
a second patch magnified at the second magnification (Fig. 2, Paragraph [0071] – FUCHS discloses a set of multi-magnification patches may be extracted to train the DMMN. The first patch is extracted from the center of the input patch with size of 256×256 pixels in 20×.),
at least one annotated cell based on the second patch (Fig. 2, Paragraph [0099] – FUCHS discloses to train the multi-class segmentation DMMN, patches are extracted from whole slide images (WSIs) and the corresponding annotations.),
and information regarding a positional relationship between the first patch and the second patch (Fig. 2, Paragraph [0089]- FUCHS discloses although the annotation was partially done, the model was able to learn not only spatial characteristics within a class but also spatial relationship between classes. The DMMN architecture see all 20×, 10×, and 5× magnifications to have a wider field-of-view to make more accurate predictions.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; detect at least one cell included in the at least one tissue with the teachings of FUCHS having wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image, a second patch magnified at the second magnification, at least one annotated cell based on the second patch, and information regarding a positional relationship between the first patch and the second patch.
Wherein having CHUKKA’s computing apparatus wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image, a second patch magnified at the second magnification, at least one annotated cell based on the second patch, and information regarding a positional relationship between the first patch and the second patch.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
Regarding claim 11, CHUKKA teaches a method of analyzing a pathological slide image (Figs. 1 & 2, Paragraph [0049]- CHUKKA discloses at least some embodiments of the present disclosure relate to computer systems and methods for analyzing digital images captured from biological samples, including tissue samples), the method comprising:
acquiring a pathological slide image (Fig. 3A, step 300 called receive sample image data; Paragraph [0066] - CHUKKA discloses the training of a multilayer neural network comprises the steps of (i) receiving sample image data (step 300) (e.g. using an image acquisition module 202))
showing at least one tissue (Fig. 3A, Paragraph [0067] - CHUKKA further discloses given a sample image, different tissue types and/or cell types may be identified in the sample image);
generating, as feature information, a first image (Figs. 2 & 7, Paragraph [0092] – CHUKKA discloses the image analysis module 207 is run a first time to extract features and classify cells and/or nuclei in a first image; and then run a second time to extract features and classify cells and/or nucleic in a series of addition images. Paragraph [0126] – CHUKKA further discloses the region label image [wherein the region label image is a first image] is shown in the bottom right and the cell level label image in the bottom left. The region label image is shown at pixel level and the cell label image shown the cell classification label at the cell level.)
in which at least one area of the pathological slide image is classified into one of a plurality of types (Fig. 3C, Paragraph [0111] – CHUKKA discloses unlabeled image [wherein unlabeled image is the pathological slide image] is supplied to a trained multilayer neural network 220 (step 324) [wherein the multilayer neural network is a first machine learning model]. The skilled artisan will appreciate that the multilayer neural network 220 must have been trained to classify the types of cells present (or suspected to be present) in the unlabeled image. For example, if the network was trained to recognize and classify lymphocytes and tumor cells in a breast cancer sample [wherein lymphocytes and tumor cells are examples of a plurality of cell types], then the unlabeled image must also be of a breast cancer sample. Paragraph [0112] – CHUKKA further discloses a generated foreground mask may be overlaid on top of an unlabeled image, and each of the different predictive pixel labels within any region (i.e. identified cell) may be quantified and then the predictive label having the greatest quantity will be selected as the cell label. For example, if a particular identified cell comprises 100 total pixels, and 55 of those total pixels are predicted to be tumor cell pixels, 40 of those total pixels are predicted to be stromal cells, and 5 of those total pixels are predicted to be lymphocytes, then the cell would be assigned the label of a tumor cell since the majority of the pixels were predicted to be tumor cell pixels. See also Paragraph [0118].),
by analyzing the pathological slide image at a first magnification using a first machine learning model (Fig. 3A, Paragraph [0119] – CHUKKA discloses to train the network, 20 whole slide images @ 20× magnification were used for region and cell annotations (˜20,000 regions, 500,000 cells, 2×108 pixels@ 0.5 μm) [wherein the network is a first machine learning model].);
Although CHUKKA further teaches and detecting at least one cell included in the at least one tissue (Fig. 3A, Paragraph [0073] – CHUKKA discloses following image acquisition and/or unmixing, input images or unmixed image channel images are provided to a cell detection module 204 to detect cells and subsequently to a cell classification module 205 to classify cells and/or nuclei (step 300). Paragraph [0067] – CHUKKA further discloses given a sample image, different tissue types and/or cell types may be identified in the sample image.)
by analyzing the feature information (Fig. 3A, Paragraph [0073] – CHUKKA discloses the procedures and algorithms described herein may be adapted to identify and classify various types of cells or cell nuclei based on features within the input images. See also Fig. 2A, Paragraph [0092].)
CHUKKA fails to explicitly teach and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
However, FUCHS explicitly teaches and a second image (Fig. 20(a), #2036 called patches, Paragraph [0159] – FUCHS discloses each network 2018 may have one of the patches 2036 at a corresponding magnification factor from one of the tiles 2022 of the biomedical image 2020 as input.),
which is generated based on the pathological slide image (Fig. 20(a), #2020 called biomedical image, Paragraph [0159] – FUCHS discloses each network 2018 may have one of the patches 2036 at a corresponding magnification factor from one of the tiles 2022 of the biomedical image 2020 as input.)
at a second magnification, (Fig. 20(a), Paragraph [0159] – FUCHS discloses each network 2018 itself may correspond to or be associated with one of the magnification factors. For example, the first network 2018A may be associated with the first magnification factor (e.g., 20×), the second network 2018B may be associated with the second magnification factor (e.g., 10×), and the third network 2018C may be associated with the third magnification factor (e.g., 20×), and so forth.)
using a second machine learning model (Fig. 20(d), Paragraph [0171] – FUCHS discloses the segmentation model may include a set of networks (e.g. networks 2018) corresponding to the set of magnification factors used to create the patches. Each network may include a set of encoders (e.g., a convolution block 2032) and a set of decoders (e.g., a deconvolution block 2040).);
wherein a field of view of the second magnification is narrower than a field of view of the first magnification (Fig. 20(a), Paragraph [0159] – FUCHS discloses the first network 2018A may be associated with the first magnification factor (e.g., 20×), the second network 2018B may be associated with the second magnification factor (e.g., 10×), and the third network 2018C may be associated with the third magnification factor (e.g., 20×), and so forth [wherein the first magnification factor (e.g., 20×) is narrower than a field of view of the second magnification factor (e.g., 10x)].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; and detecting at least one cell included in the at least one tissue by analyzing the feature information with the teachings of FUCHS having and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein having and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
CHUKKA in view of FUCHS fail to explicitly teach and excluding, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in area of the first type among the plurality of types,
However, YIP explicitly teaches and excluding, from at least one cell of a first type among the detected at least one cell (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas but which belong to cells that are characterized as lymphocytes. See also Paragraph [0072].),
at least one cell which is not included in area of the first type among the plurality of types (Fig. 2, Paragraph [0069] – YIP discloses the digital tissue segmenter 201 may exclude cell nuclei or outer edges of cells [wherein cell nuclei or outer edges of cells are at least one cell] that are located in tumor areas [wherein tumor areas are an area of the first type] but which belong to cells that are characterized as lymphocytes. See also Paragraph [0072].),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA in view of FUCHS of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; and detecting at least one cell included in the at least one tissue by analyzing the feature information and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; wherein a field of view of the second magnification is narrower than a field of view of the first magnification with the teachings of YIP having and excluding, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in area of the first type among the plurality of types,
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein having and excluding, from at least one cell of a first type among the detected at least one cell, at least one cell which is not included in area of the first type among the plurality of types,
The motivation behind the modification would have been to obtain a digital pathology system that considers the whole context of the pathological slide image, since both CHUKKA and YIP relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while YIP has techniques for generating an overlay map on a digital medical image of a slide that may be useful for predicting patient survival in that overlays of such status information can provide an illustrated estimate of how well a patient will respond to certain immunotherapies. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and YIP (US 20200211189 A1), Paragraph [0079].
Regarding claim 13, CHUKKA and FUCHS in view of YIP teach the method of claim 11,
CHUKKA further teaches wherein the one of the plurality of types includes at least one of a cancer area (Fig. 6, Paragraph [0007] – CHUKKA discloses the present disclosure is a method of training a multilayer neural network to detect and classify different cell types (e.g. tumor cells, stromal cells, lymphocytes, etc.) and tissue regions (e.g. tumor regions [wherein tumor regions are a cancer area], lymphocyte-rich regions, stromal regions, necrotic regions, etc [wherein lymphocyte-rich/stromal/necrotic regions are a plurality of types].) within a sample image (e.g. a sample image of a biological sample tissue having one or more stains)),
a cancer stroma area (Fig. 6, Paragraph [0007] – CHUKKA discloses the present disclosure is a method of training a multilayer neural network to detect and classify different cell types (e.g. tumor cells, stromal cells, lymphocytes, etc.) and tissue regions (e.g. tumor regions, lymphocyte-rich regions, stromal regions, necrotic regions, etc.) within a sample image (e.g. a sample image of a biological sample tissue having one or more stains)),
a necrosis area (Fig. 6, Paragraph [0007] – CHUKKA discloses the present disclosure is a method of training a multilayer neural network to detect and classify different cell types (e.g. tumor cells, stromal cells, lymphocytes, etc.) and tissue regions (e.g. tumor regions, lymphocyte-rich regions, stromal regions, necrotic regions, etc.) within a sample image (e.g. a sample image of a biological sample tissue having one or more stains)),
and a background area (Fig. 4B, Paragraph [0065] – CHUKKA discloses a tissue region mask may be created by identifying the tissue regions and automatically or semi-automatically excluding the background regions (e.g. regions of a whole slide image corresponding to glass with no sample, such as where there exists only white light from the imaging source).).
Regarding claim 14, CHUKKA and FUCHS in view of YIP teach the method of claim 11,
Although CHUKKA further teaches wherein the detecting includes detecting the at least one cell (Fig. 3C, Paragraph [0112]- CHUKKA discloses using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325) [wherein identified denotes detecting])
CHUKKA fails to explicitly teach on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image.
However, YIP explicitly teaches on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image (Fig. 3A, 3B, Paragraph [0062] - YIP discloses FIG. 3A illustrates a tissue class overlay map created by the overlay map generator of the digital tissue segmenter 201. FIG. 3B illustrates a cell outer edge overlay map created by the overlay map generator of the digital tissue segmenter 201. The overlay map generator may display the digital overlays as transparent or opaque layers that cover the slide image [wherein the layers that cover the slide image are a third image, wherein the slide image is a first image, and the digital overlay is a fourth image], aligned such that the slide location shown in the overlay and the slide image are in the same location on the display).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; and detecting at least one cell included in the at least one tissue by analyzing the feature information and a second image, with the teachings of YIP having on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image.
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein having on the basis of a third image in which the first image is merged with a fourth image acquired from a portion corresponding to the first image within the pathological slide image.
The motivation behind the modification would have been to obtain a digital pathology system that considers the whole context of the pathological slide image, since both CHUKKA and YIP relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while YIP has techniques for generating an overlay map on a digital medical image of a slide that may be useful for predicting patient survival in that overlays of such status information can provide an illustrated estimate of how well a patient will respond to certain immunotherapies. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and YIP (US 20200211189 A1), Paragraph [0079].
Regarding claim 15, CHUKKA and FUCHS in view of YIP teach the method of claim 11,
CHUKKA further teaches wherein the detecting includes detecting the at least one cell (Fig. 3C, Paragraph [0112]- CHUKKA discloses using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325) [wherein identified denotes detecting])
by using the first image in at least one of intermediate operations (Fig. 3C, Paragraph [0112]- CHUKKA discloses using the pixel level predictive classifications [wherein the first image is pixel level predictive classifications], a label is then assigned to each identified cell (step 325))
of the second machine learning model (Fig. 3C, step 324 called apply a trained multi-layer neural network [wherein multi-layer neural network is the second machine learning model] to provide predictive labels for each pixel within the unlabeled image)
that uses the pathological slide image as an input (Fig. 3C, step 320 called receive an unlabeled image [wherein unlabeled image is the pathological slide image]).
Regarding claim 16, CHUKKA and FUCHS in view of YIP teach the method of claim 11,
CHUKKA further teaches wherein the detecting includes detecting the at least one cell (Fig. 3C, Paragraph [0112]- CHUKKA discloses using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325) [wherein identified denotes detecting])
CHUKKA fails to explicitly teach by mutually using information generated from at least one of intermediate operations of the first machine learning model and information generated from at least one of intermediate operations of the second machine learning model in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model.
However, FUCHS explicitly teaches by mutually using information generated from at least one of intermediate operations of the first machine learning model (Fig. 20(e), Paragraph [0145] – FUCHS discloses the terminal convolution block 2046 may have a set of feature maps 2038 as input. The set of transform layers 2048A-N of the terminal convolution block 2046 may be applied to the input, such as the set of feature maps 2038′, in any sequence (such as the one depicted), outputted by one of the networks 2018 [wherein one of the networks 2018 is the first machine learning model]. The set of feature maps 2038′ may be the resultant output of one of the networks 2018 from processing one of the patches 2036 and other input feature maps 2038 inputted to the network 2018.)
and information generated from at least one of intermediate operations of the second machine learning model (Fig. 20(e), Paragraph [0145] – FUCHS discloses the terminal convolution block 2046 may have a set of feature maps 2038 as input. The set of transform layers 2048A-N of the terminal convolution block 2046 may be applied to the input, such as the set of feature maps 2038′, in any sequence (such as the one depicted), outputted by one of the networks 2018 [wherein one of the networks 2018 is the second machine learning model]. The set of feature maps 2038′ may be the resultant output of one of the networks 2018 from processing one of the patches 2036 and other input feature maps 2038 inputted to the network 2018.)
in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model (Fig. 20(f), Paragraph [0151] – FUCHS discloses the network 2018 may have one of the patches 2036 of a tile 2022 in the biomedical image 2020 (depicted generally to the left) and set of feature maps 2038′ [wherein feature maps 2038’ is at least one of the intermediate operations] outputted from other networks 2018 (depicted generally below) as an input.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; and detecting at least one cell included in the at least one tissue by analyzing the feature information and a second image, with the teachings of FUCHS having by mutually using information generated from at least one of intermediate operations of the first machine learning model and information generated from at least one of intermediate operations of the second machine learning model in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model.
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein having by mutually using information generated from at least one of intermediate operations of the first machine learning model and information generated from at least one of intermediate operations of the second machine learning model in at least one of the intermediate operations of the first machine learning model and at least one of the intermediate operations of the second machine learning model.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
Regarding claim 17, CHUKKA and FUCHS in view of YIP teach the method of claim 11,
Although CHUKKA further teaches and the generated data includes at least one of a first patch magnified at the first magnification, at least one annotated tissue based on the first patch (Fig. 1, Paragraph [0119] - CHUKKA discloses to train the network, 20 whole slide images @ 20× magnification were used for region and cell annotations (˜20,000 regions, 500,000 cells, 2×108 pixels @ 0.5 μm)),
CHUKKA fails to explicitly teach wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image, a second patch magnified at the second magnification, at least one annotated cell based on the second patch, and information regarding a positional relationship between the first patch and the second patch.
However, FUCHS explicitly teaches wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image (Fig. 2, Paragraph [0065] – FUCHS discloses to train the multi-class segmentation DMMN, patches are extracted from whole slide images and the corresponding annotations (thus similar/consistent to applicants’ disclosure in the specification, Paragraph [0060] "However, the first machine learning model and the second machine learning model may be the same machine learning model or may be separate machine learning models.")),
a second patch magnified at the second magnification (Fig. 2, Paragraph [0071] – FUCHS discloses a set of multi-magnification patches may be extracted to train the DMMN. The first patch is extracted from the center of the input patch with size of 256×256 pixels in 20×.),
at least one annotated cell based on the second patch (Fig. 2, Paragraph [0099] – FUCHS discloses to train the multi-class segmentation DMMN, patches are extracted from whole slide images (WSIs) and the corresponding annotations.),
and information regarding a positional relationship between the first patch and the second patch (Fig. 2, Paragraph [0089]- FUCHS discloses although the annotation was partially done, the model was able to learn not only spatial characteristics within a class but also spatial relationship between classes. The DMMN architecture see all 20×, 10×, and 5× magnifications to have a wider field-of-view to make more accurate predictions.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; and detecting at least one cell included in the at least one tissue by analyzing the feature information and a second image, with the teachings of FUCHS having wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image, a second patch magnified at the second magnification, at least one annotated cell based on the second patch, and information regarding a positional relationship between the first patch and the second patch.
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein at least one of the first machine learning model and the second machine learning model is trained by data generated on the basis of at least one patch included in the pathological slide image, a second patch magnified at the second magnification, at least one annotated cell based on the second patch, and information regarding a positional relationship between the first patch and the second patch.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
Regarding claim 20, CHUKKA teaches a non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the method of claim 11 (Fig. 2, #201 called a memory, Paragraph [0052]- CHUKKA discloses the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201, the at least one memory 201 storing non-transitory computer-readable instructions for execution by the one or more processors to cause the one or more processors to execute instructions (or stored data) in a multilayer neural network 220 and in at least one of a ground truth training module 210 or a testing module 230.).
Claims 8 and 18 are rejected 35 U.S.C. 103 as being unpatentable over CHUKKA et al. (US 20200342597 A1), hereinafter referenced as CHUKKA in view of FUCHS (US 20210133966 A1), hereinafter referenced as FUCHS, further in view of YIP (US 20200211189 A1), hereinafter referenced as YIP, and further in view of ZHANG (US 20240331868 A1), hereinafter referenced as ZHANG.
Regarding claim 8, CHUKKA and FUCHS in view of YIP teach the computing apparatus of claim 1,
Although CHUKKA further teaches wherein the at least one processor (Fig. 2, #209 called processors; Paragraph [0051]) is further configured to:
and detect the at least one cell (Fig. 3C, Paragraph [0112] – CHUKKA discloses using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325))
by using a result of extracting the cells and the feature information (Fig. 2A, Paragraph [0092] – CHUKKA discloses the image analysis module 207 is run a first time to extract features and classify cells and/or nuclei in a first image).
CHUKKA fails to explicitly teach generate the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification; extract cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification;
However, ZHANG explicitly teaches generate the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification (Fig. 1, Paragraph [0050-0051] – ZHANG discloses the step S200 comprises the following steps. Step S210, performing horizontal and vertical reversal of the pathological image data of each sample using a fixed-size square sliding window (e.g., 224*224 pixels) at a predetermined step size (e.g., 10% or 15% of the side length of the sliding window), to obtain a plurality of small sliding window regions corresponding to each pathological image data, the labels of the sliding window regions being consistent with the labels of the corresponding complete pathological image data. See also paragraph [0040].);
extract cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification (Fig. 1, Paragraph [0066] – ZHANG discloses the trained optimal classifier model (e.g. the RegNet model) is used, with the fully connected layer removed, as a feature extractor to extract the feature vectors of s sliding window images for each sample (one sliding window image corresponds to one feature vector).);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; detect at least one cell included in the at least one tissue with the teachings of ZHANG having generate the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification; extract cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification;
Wherein having CHUKKA’s computing apparatus wherein generate the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification; extract cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification;
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and ZHANG relate to deep learning based pathological image detection and prediction processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while ZHANG has a deep learning-based cancer prognosis survival prediction method and device, and a storage medium wherein prognosis risk evaluation is performed on the new sample to enhance the efficiency of diagnosis and treatment in the clinical field and improve the accuracy of risk evaluation results. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and ZHANG (US 20240331868 A1), Paragraph [0096].
Regarding claim 18, CHUKKA and FUCHS in view of YIP teach the method of claim 11,
Although CHUKKA further teaches and detecting the at least one cell (Fig. 3C, Paragraph [0112] – CHUKKA discloses using the pixel level predictive classifications, a label is then assigned to each identified cell (step 325))
by using a result of extracting the cells and the feature information (Fig. 2A, Paragraph [0092] - CHUKKA discloses the image analysis module 207 is run a first time to extract features and classify cells and/or nuclei in a first image).
CHUKKA fails to explicitly teach wherein the generating includes generating the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification, and the detecting includes: extracting cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification;
However, ZHANG explicitly teaches wherein the generating includes generating the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification (Fig. 1, Paragraph [0050-0051] – ZHANG discloses the step S200 comprises the following steps. Step S210, performing horizontal and vertical reversal of the pathological image data of each sample using a fixed-size square sliding window (e.g., 224*224 pixels) at a predetermined step size (e.g., 10% or 15% of the side length of the sliding window), to obtain a plurality of small sliding window regions corresponding to each pathological image data, the labels of the sliding window regions being consistent with the labels of the corresponding complete pathological image data. See also paragraph [0040].),
and the detecting includes: extracting cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification (Fig. 1, Paragraph [0066] – ZHANG discloses the trained optimal classifier model (e.g. the RegNet model) is used, with the fully connected layer removed, as a feature extractor to extract the feature vectors of s sliding window images for each sample (one sliding window image corresponds to one feature vector).);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, with the teachings of ZHANG having wherein the generating includes generating the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification, and the detecting includes: extracting cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification;
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein the generating includes generating the feature information by analyzing, in a sliding window method, the pathological slide image magnified at a third magnification, and the detecting includes: extracting cells from the pathological slide image by analyzing, in the sliding window method, the pathological slide image magnified at a fourth magnification;
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and ZHANG relate to deep learning based pathological image detection and prediction processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while ZHANG has a deep learning-based cancer prognosis survival prediction method and device, and a storage medium wherein prognosis risk evaluation is performed on the new sample to enhance the efficiency of diagnosis and treatment in the clinical field and improve the accuracy of risk evaluation results. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and ZHANG (US 20240331868 A1), Paragraph [0096].
Claims 9 and 19 are rejected 35 U.S.C. 103 as being unpatentable over CHUKKA et al. (US 20200342597 A1), hereinafter referenced as CHUKKA in view of FUCHS (US 20210133966 A1), hereinafter referenced as FUCHS, further in view of YIP (US 20200211189 A1), hereinafter referenced as YIP, and further in view of STUMPE (US 20210224541 A1), hereinafter referenced as STUMPE.
Regarding claim 9, CHUKKA and FUCHS in view of YIP teach the computing apparatus of claim 1,
CHUKKA and YIP fail to explicitly teach but both magnifications have same resolution.
However, FUCHS explicitly teaches but both magnifications have same resolution (Fig. 20(a), Paragraph [0161] – FUCHS discloses the tile 2022′ may be of the same resolution and magnification factor as the patch 2036A fed into the first network 2018A. By applying the networks 2018 of the segmentation model 2014 to patches 2036 from more tiles 2022 of the biomedical image 2020, additional tiles 2022′ for the segmented image 2020′ may be generated. See also Paragraph [0127].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; detect at least one cell included in the at least one tissue with the teachings of YIP having but both magnifications have same resolution.
Wherein having CHUKKA’s computing apparatus wherein having but both magnifications have same resolution.
The motivation behind the modification would have been to obtain a digital pathology system that considers the whole context of the pathological slide image, since both CHUKKA and YIP relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while YIP has techniques for generating an overlay map on a digital medical image of a slide that may be useful for predicting patient survival in that overlays of such status information can provide an illustrated estimate of how well a patient will respond to certain immunotherapies. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and YIP (US 20200211189 A1), Paragraph [0079].
CHUKKA and FUCHS in view of YIP fail to explicitly teach wherein a magnification of an image used for training of the second machine learning model and a magnification of an image used for an inference by the second machine learning model are different from each other, wherein the magnification of an image used for training has a narrow field of view, and the magnification of an image used for an inference has a wide field of view,
However, STUMPE explicitly teaches wherein a magnification of an image used for training of the second machine learning model (Fig. 8, Paragraph [0107] – STUMPE discloses each of the pattern recognizers takes the form of a deep convolutional neural network trained on a set of digital slide images at a particular magnification. For example, pattern recognizer 406A is trained on 40× magnification slide images. Pattern recognizer 406B is trained on 20× magnification slide images. Pattern recognizer 406C is trained on 10× magnification slide images. Pattern recognizer 406D is trained on 5× magnification slide images.)
and a magnification of an image used for an inference by the second machine learning model are different from each other (Fig. 8, Paragraph [0066] – STUMPE discloses in some implementations it may be possible to perform inference on a digital image that is the entire field of view of the microscope. In other situations, it may be desirable to perform inference on only a portion of the image, such as several 299×299 rectangular patches of pixels located about the center of the field of view, or on some larger portion of the field of view. See also Paragraph [0102-0103].),
wherein the magnification of an image used for training has a narrow field of view (Fig. 8, Paragraph [0107] – STUMPE discloses each of the pattern recognizers takes the form of a deep convolutional neural network trained on a set of digital slide images at a particular magnification. For example, pattern recognizer 406A is trained on 40× magnification slide images [wherein 40x is a narrow field of view].),
and the magnification of an image used for an inference has a wide field of view (Fig. 6, Paragraph [0102-0103] – STUMPE discloses if the operator changes the objective lens 108 (e.g., to zoom in or out) a new image is captured. The new images are sent to the compute unit 126 [wherein the operator can adjust the objective lens to have a wide field of view]. At step 312 the image of the field of view is provided as input to the relevant machine learning pattern recognizer 200 in the compute unit 126 (FIG. 5) to perform inference.),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; detect at least one cell included in the at least one tissue with the teachings of STUMPE having wherein a magnification of an image used for training of the second machine learning model and a magnification of an image used for an inference by the second machine learning model are different from each other, wherein the magnification of an image used for training has a narrow field of view, and the magnification of an image used for an inference has a wide field of view,
Wherein having CHUKKA’s computing apparatus wherein a magnification of an image used for training of the second machine learning model and a magnification of an image used for an inference by the second machine learning model are different from each other, wherein the magnification of an image used for training has a narrow field of view, and the magnification of an image used for an inference has a wide field of view,
The motivation behind the modification would have been to obtain an enhanced digital pathology system with an improved microscope system, since both CHUKKA and STUMPE relate to pathological imaging systems and processes to detect and classify biological samples, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while STUMPE has an improved microscope system and method for assisting a pathologist in classifying biological samples where continuous capture of images by the camera 124, rapid performance of interference on the images by the pattern recognizer, and generation and projection of enhancements as overlays onto the field of view, enables the system 100 of FIG. 1 to continue to provide enhancements to the field of view and assist the pathologist in characterizing or classifying the specimen in substantial real time. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and STUMPE (US 20210224541 A1), Paragraph [0047].
Regarding claim 19, CHUKKA and FUCHS in view of YIP teach the method of claim 11,
CHUKKA and YIP fail to explicitly teach but both magnifications have same resolution.
However, FUCHS explicitly teaches but both magnifications have same resolution (Fig. 20(a), Paragraph [0161] – FUCHS discloses the tile 2022′ may be of the same resolution and magnification factor as the patch 2036A fed into the first network 2018A. By applying the networks 2018 of the segmentation model 2014 to patches 2036 from more tiles 2022 of the biomedical image 2020, additional tiles 2022′ for the segmented image 2020′ may be generated. See also Paragraph [0127].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, by analyzing the pathological slide image at a first magnification using a first machine learning model; and detecting at least one cell included in the at least one tissue by analyzing the feature information and a second image, with the teachings of FUCHS having but both magnifications have same resolution.
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein having but both magnifications have same resolution.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
CHUKKA and FUCHS in view of YIP fail to explicitly teach wherein a magnification of an image used for training of the second machine learning model and a magnification of an image used for an inference by the second machine learning model are different from each other, wherein the magnification of an image used for training has a narrow field of view, and the magnification of an image used for an inference has a wide field of view,
However, STUMPE explicitly teaches wherein a magnification of an image used for training of the second machine learning model (Fig. 8, Paragraph [0107] – STUMPE discloses each of the pattern recognizers takes the form of a deep convolutional neural network trained on a set of digital slide images at a particular magnification. For example, pattern recognizer 406A is trained on 40× magnification slide images. Pattern recognizer 406B is trained on 20× magnification slide images. Pattern recognizer 406C is trained on 10× magnification slide images. Pattern recognizer 406D is trained on 5× magnification slide images.)
and a magnification of an image used for an inference by the second machine learning model are different from each other (Fig. 8, Paragraph [0066] – STUMPE discloses in some implementations it may be possible to perform inference on a digital image that is the entire field of view of the microscope. In other situations, it may be desirable to perform inference on only a portion of the image, such as several 299×299 rectangular patches of pixels located about the center of the field of view, or on some larger portion of the field of view. See also Paragraph [0102-0103].),
wherein the magnification of an image used for training has a narrow field of view (Fig. 8, Paragraph [0107] – STUMPE discloses each of the pattern recognizers takes the form of a deep convolutional neural network trained on a set of digital slide images at a particular magnification. For example, pattern recognizer 406A is trained on 40× magnification slide images [wherein 40x is a narrow field of view].),
and the magnification of an image used for an inference has a wide field of view (Fig. 6, Paragraph [0102-0103] – STUMPE discloses if the operator changes the objective lens 108 (e.g., to zoom in or out) a new image is captured. The new images are sent to the compute unit 126 [wherein the operator can adjust the objective lens to have a wide field of view]. At step 312 the image of the field of view is provided as input to the relevant machine learning pattern recognizer 200 in the compute unit 126 (FIG. 5) to perform inference.),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a method of analyzing a pathological slide image, the method comprising: acquiring a pathological slide image showing at least one tissue; generating, as feature information, a first image in which at least one area of the pathological slide image is classified into one of a plurality of types, and detecting at least one cell included in the at least one tissue by analyzing the feature information and a second image, which is generated based on the pathological slide image at a second magnification, using a second machine learning model; with the teachings of STUMPE having wherein a magnification of an image used for training of the second machine learning model and a magnification of an image used for an inference by the second machine learning model are different from each other, wherein the magnification of an image used for training has a narrow field of view, and the magnification of an image used for an inference has a wide field of view,
Wherein having CHUKKA’s method of analyzing a pathological slide image wherein a magnification of an image used for training of the second machine learning model and a magnification of an image used for an inference by the second machine learning model are different from each other, wherein the magnification of an image used for training has a narrow field of view, and the magnification of an image used for an inference has a wide field of view,
The motivation behind the modification would have been to obtain an enhanced digital pathology system with an improved microscope system, since both CHUKKA and STUMPE relate to pathological imaging systems and processes to detect and classify biological samples, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while STUMPE has an improved microscope system and method for assisting a pathologist in classifying biological samples where continuous capture of images by the camera 124, rapid performance of interference on the images by the pattern recognizer, and generation and projection of enhancements as overlays onto the field of view, enables the system 100 of FIG. 1 to continue to provide enhancements to the field of view and assist the pathologist in characterizing or classifying the specimen in substantial real time. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and STUMPE (US 20210224541 A1), Paragraph [0047].
Claim 10 is rejected 35 U.S.C. 103 as being unpatentable over CHUKKA et al. (US 20200342597 A1), hereinafter referenced as CHUKKA in view of FUCHS (US 20210133966 A1), hereinafter referenced as FUCHS, further in view of YIP (US 20200211189 A1), hereinafter referenced as YIP, and further in view of NICULA (US 20210327534 A1), hereinafter referenced as NICULA.
Regarding claim 10, CHUKKA teaches the computing apparatus of claim 1,
CHUKKA further teaches wherein the feature information includes at least one of a cancer area (Fig. 6, Paragraph [0100] – CHUKKA discloses FIG. 6 illustrates different regions and different cells which may be classified. In this example, biologically feasible combinations of cell types in tissue regions include (i) tumor cells in tumor regions [wherein tumor regions is a cancer area]; (ii) stromal cells in tumor regions; (iii) lymphocytes in tumor regions; (iv) tumor cells in stromal regions; (v) stromal cells in stromal regions; (vi) lymphocytes in stromal regions; (vii) tumor cells in lymphocyte-rich regions; and (viii) lymphocytes in lymphocyte-rich regions.),
a tumor cell density (Fig. 3C, Paragraph [0112]- CHUKKA discloses each of the different predictive pixel labels within any region (i.e. identified cell) may be quantified [wherein quantified denotes cell density]),
and information regarding a biomarker score (Paragraph [0093] - CHUKKA discloses after features are derived, the feature may be used alone or in conjunction with training data (e.g. during training, example cells are presented together with a ground truth identification provided by an expert observer according to procedures known to those of ordinary skill in the art) to classify nuclei or cells. In some embodiments, the system can include a classifier that was trained based at least in part on a set of training or reference slides for each biomarker.),
and the at least one processor (Fig. 2, #209 called processors; Paragraph [0051]) is further configured to further detect at least one of biomarker information (Paragraph [0093] - CHUKKA discloses the system can include a classifier that was trained based at least in part on a set of training or reference slides for each biomarker.),
Although CHUKKA further teaches immune phenotype information (Fig. 3A, Paragraph [0073]- CHUKKA discloses image acquisition and/or unmixing, input images or unmixed image channel images are provided to a cell detection module 204 to detect cells and subsequently to a cell classification module 205 to classify cells and/or nuclei (step 300) … the cell types identified may be dependent on the sample image type and staining, e.g. in the context of immune cells, it could be that different types of immune cells including CD3 and CD8 [wherein types of immune cells including CD3 and CD8 are immune phenotype information] are detected and classified),
CHUKKA fails to explicitly teach lesion information.
However, FUCHS explicitly teaches lesion information (Fig. 20(b), Paragraph [0125]- FUCHS discloses each region of interest 2026 may correspond to tumor, lesion, or other injury in the imaged tissue sample used to generate the sample biomedical image 2020B).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; detect at least one cell included in the at least one tissue with the teachings of FUCHS having lesion information.
Wherein having CHUKKA’s computing apparatus wherein having lesion information.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and FUCHS relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while FUCHS has a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue where DMMNs may be developed to combine feature maps in various magnification for more accurate segmentation predictions, and partial annotations may be used to save annotation time for pathologists and still achieve high performance. Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and FUCHS (US 20210133966 A1), Paragraph [0064].
CHUKKA and FUCHS in view of YIP fail to explicitly teach genomic mutation information, and genomic signature information which are expressed on the at least one tissue.
However, NICULA explicitly teaches genomic mutation information, and genomic signature information which are expressed on the at least one tissue (Fig. 8B, Paragraph [0244]- NICULA discloses step 852 of the method 850 can include obtaining, via one or more processors, a training dataset from one or more training subjects… The training dataset can include any biological or genomic information of the training subjects including, but not limited to, information relating to the primary nucleic acid sequence of all or a portion of the genome (e.g., the presence or absence of a nucleotide polymorphism, indel, sequence rearrangement, mutational frequency, etc.), the copy number of one or more particular nucleotide sequences within the genome (e.g., copy number, allele frequency fractions, single chromosome or entire genome ploidy, etc.), the epigenetic status of all or a portion of the genome (e.g., covalent nucleic acid modifications such as methylation, histone modifications, nucleosome positioning, etc.), and the expression profile of the organism's genome (e.g., gene expression levels, isotype expression levels, gene expression ratios, etc.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of CHUKKA and FUCHS in view of YIP of having a computing apparatus comprising: at least one memory; and at least one processor, wherein the at least one processor is configured to: acquire a pathological slide image showing at least one tissue; detect at least one cell included in the at least one tissue with the teachings of NICULA having genomic mutation information, and genomic signature information which are expressed on the at least one tissue.
Wherein having CHUKKA’s computing apparatus wherein having genomic mutation information, and genomic signature information which are expressed on the at least one tissue.
The motivation behind the modification would have been to obtain an automated digital pathology system that can make more accurate predictions, since both CHUKKA and NICULA relate to cell detection and tissue classification processes, wherein CHUKKA has an automated system and methods for training a multilayer neural network to classify cells and regions where the combination of the results of various classification problems can then be done automatically and would not depend on any post-processing, while NICULA has patch convolutional neural networks that classify subjects for a disease condition, such as cancer, using genotypic information from such subjects, hence providing a method to assess and optimize patch selection (e.g., to minimize the set of patches thus improving computational efficiency and/or reducing cost). Please see CHUKKA (US 20200342597 A1), Paragraph [0022], and NICULA (US 20210327534 A1), Paragraph [0289].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
Bredno et al. (US 20170372117 A1) - Disclosed, among other things, is a computer device and computer-implemented method of classifying cells within an image of a tissue sample comprising providing the image of the tissue sample as input; computing nuclear feature metrics from features of nuclei within the image; computing contextual information metrics based on nuclei of interest with the image; classifying the cells within the image using a combination of the nuclear feature metrics and contextual information metrics.…. Fig. 1, Abstract.
Tandon et al. (US 20180211380 A1) - A system for imaging biological samples and analyzing images of the biological samples is provided. The system can automatically analyze images of biological samples to classify cells of interest using machine learning techniques. Some implementations can diagnose diseases associated with specific cell types. Devices, methods, and computer program product for imaging and analyzing biological samples are also provided.…. Fig. 1, Abstract.
Arar et al. (US 20190080450 A1) - The invention relates to the automated determination of the staining quality of an IHC stained biological sample. A plurality of features is extracted from a digital IHC stained tissue image. The features are input into a first classifier configured to identify the extended tissue type of the depicted tissue as a function of the extracted features. An extended tissue type is a tissue type with a defined expression level of the tumor marker. In addition, the extracted features are input into a second classifier configured to identify a contrast level of the depicted tissue as a function of at least some second ones of the extracted features. The contrast level indicates the intensity contrast of pixels of the stained tissue. Then, a staining quality score of the image is computed as a function of the identified extended tissue type and the identified contrast level..…. Fig. 1, Abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEZAWIT N SHIMELES whose telephone number is (571)272-7663. The examiner can normally be reached M-F 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEZAWIT NOLAWI SHIMELES/ Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights