Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 9 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. See MPEP 2106 and 2106.03 for guidance. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter of a process, machine, manufacture, or composition of matter. “A computer program” as recited is not patent eligible subject matter because it is “software/data per se”. A recommended remedy for claiming a computer program is to have it embodied within a “non-transitory” computer readable medium. See also USPTO Published 2019 Patent Eligibility Guidance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 11-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bussola et al (NPL: Quantification of the Immune Content in Neuroblastoma: Deep Learning and Topological Data Analysis in Digital Pathology) in view of Alfonso et al (NPL: In-silico insights on the prognostic potential of immune cell infiltration patterns in the breast lobular epithelium).
Regarding claim 11, Bussola et al teaches a computer-implemented method, comprising the steps of: a') receiving (402) at least one digital image comprising one or more objects of interest captured at a first point in time (Fig 22, see input image, EUNet analysis, density map prediction using EfficientNet-b3, and density map creation, utilized using a computer accessing a cloud platform, see Materials and Methods, Pages 17-26);
b') classifying (404) (Figures 1, 5, and 22, and 4.3. EUNet Training and Evaluation, lymphocyte counting task was censored as a classification task, by manually defining classes of lymphocyte density. The density classes used can be represented by the set C = f0, 1, 2, 3, 4, 5, 6g, as shown in Table 2. i.e. the EUNet is utilized for classifying the one or more objects of interest (lymphocytes));
c') segmenting (406) (4.4. Lymphocytes Spatial Identification and Page 24-25, Otsu thresholding algorithm [132] is used to find an optimal value to discretize the density maps in two levels: lymphocytes and background. The Otsu algorithm is the de facto standard for discriminating foreground and background pixels within an image. In detail, the optimal threshold is identified by minimizing intra-class intensity variance (equivalent to maximizing inter-class variance). Since the Otsu algorithm is the one-dimensional discrete analog of Fisher’s discriminant analysis, this procedure coincides with globally optimizing k-means clustering on the intensity histogram. Pixels with values under the threshold are assigned to the background, while pixels with values over the threshold are assigned to the lymphocyte class. i.e. segmenting the classified one or more objects of interest (lymphocytes), using machine-learning);
d') obtaining using a tiling algorithm (408) tiles of the received digital image comprising the classified and segmented one or more objects of interest (Page 25 and 4.4. Lymphocytes Spatial Identification, in order to obtain the coordinates of the center, for each connected component in the mask, the coordinates of the center of mass are computed and used as a proxy for the coordinates of the predicted lymphocytes. The goodness of the detection is evaluated by the three metrics Precision, Recall and F1-score, using as input the two sets of points T and P. Also see Page 21, this approach allows the model to encode the confidence of the annotation during the training phase, and also to leverage the surrounding context for the prediction. To define a density map, let T be an RGB tile of shape (N N 3), and A its set of annotations. Also see 4. Materials and Methods, Pages 17-26). i.e. obtaining tiles of the classified and segmented one or more objects of interest);
e') mapping on a grid using a mapping algorithm (410) the obtained tiles of the received digital image comprising the classified and segmented one or more objects of interest (Page 18, extraction scheme was designed by overlaying a grid on the tissue area detected on each slide, where each cell of the grid represented a tile. A random number of tiles ranging from 20 to 175 were extracted with random uniform probability, in order to have a representative sample of tissue per slide. Also see Figure 20. Visualization of the random extraction pattern for the tile extraction in a grid-like fashion. A portion of an CD3+ stained WSI used for the NeSTBG dataset is portrayed (at magnification 1.25). The size of each tile is representative of the real portion of tissue captured with a 512 512 tile at 20 magnification. Also see 4. Materials and Methods, Pages 17-26. i.e. mapping the obtained tiles of interest on a grid comprising the classified and segmented one or more objects of interest).
Bussola et al teaches estimating the spatial distribution of the classified and segmented one or more objects of interest in the mapped tiles of the received digital image (4. Materials and Methods, Pages 17-26), but does not teach simulating at one or more points in time using a parameter-based model (the classified and segmented one or more objects of interest).
In a similar field of endeavor, Alfonso et al teaches, f) simulating at one or more points in time using a parameter-based model (412) the spatial distribution of the classified and segmented one or more objects of interest in the mapped tiles of the received digital image (Figure 8 and Epithelial damage induces clustering patterns of immune cells, spatial distribution of immune cells is determined by the radial distribution function g(r) that describes how density varies as a function of the distance from a reference immune cell r75. The function g(r) is calculated at the follicular (day 5) and luteal (day 25) phase of the menstrual cycle, as well as in the middle (day 14). We then consider the mean g(r) values over 10 simulations and fit them to the power law g(r) ≈ b · r−m. The parameter m represents the decreasing slope of g(r) and b the probability to find a neighbor immune cell, i.e. in direct contact. Figure 8 shows estimates of the spatial distribution of immune infiltrates in the lobular epithelium for different damage rates of epithelial cells kdge, hormone levels θ and effector killing rates kkill. Also see, Results, due to the stochastic nature of the model, we performed for each parameter constellation 10 simulations each consisting of 12 menstrual cycles (about 1 year) and monitored the average statistics. The relative number of damaged epithelial and immune cells are provided with respect to the total number of epithelial cells in the simulated cross-sections of TDLUs. i.e. simulating at one or more points in time (12 cycles, also see the radial distribution according to different time points in Figure 8) the spatial distribution of the cells, see Figure 4 and 9).
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date to incorporate the teachings of Bussola et al (NPL: Quantification of the Immune Content in Neuroblastoma: Deep Learning and Topological Data Analysis in Digital Pathology) in view of Alfonso et al (NPL: In-silico insights on the prognostic potential of immune cell infiltration patterns in the breast lobular epithelium) so that the method comprises simulating at one or more points in time using a parameter-based model (412) the spatial distribution of the classified and segmented one or more objects of interest in the mapped tiles of the received digital image. Doing so would allow for the interpretation of snapshot-like single time points of breast tissue samples in the context of a mathematical model by considering crucial aspects of the underlying dynamic processes, and would therefore allow for a dynamic understanding of these early events [immune responses] is a crucial step towards investigating the prognostic role of inflammation in the breast lobular tissue (Alfonso et al, Page 4 and Introduction).
Regarding claim 12, Bussola et al teaches the method of claim 11, wherein the received at least one digital image is a digitized image of a biopsy (Page 2, the approach is demonstrated on the neuroblastoma specimens with T-Lymphocytes -Bambino Gesù (NeSTBG), an original dataset of samples from NB patients, provided by Ospedale Pediatrico Bambino Gesù (OPBG) in Rome, also see 4. Materials and Methods, Pages 17-26).
Regarding claim 13, Bussola et al teaches the method of claim 12, wherein the one or more objects of interest comprise tumor cells and/or immune cells (Page 2, the approach is demonstrated on the neuroblastoma specimens with T-Lymphocytes -Bambino Gesù (NeSTBG), an original dataset of samples from NB patients, provided by Ospedale Pediatrico Bambino Gesù (OPBG) in Rome, also see 4. Materials and Methods, Pages 17-26).
Regarding claim 14, Bussola et al teaches the method of claim 11, further comprising: classifying (404) using a first machine-learning algorithm the one or more objects of interest in the received images (Figures 1, 5, and 22, and 4.3. EUNet Training and Evaluation, lymphocyte counting task was censored as a classification task, by manually defining classes of lymphocyte density. The density classes used can be represented by the set C = f0, 1, 2, 3, 4, 5, 6g, as shown in Table 2. i.e. the EUNet is utilized for classifying the one or more objects of interest (lymphocytes)); and segmenting (406) using a second machine-learning algorithm the classified one or more objects of interest in the received images ( (4.4. Lymphocytes Spatial Identification and Page 24-25, Otsu thresholding algorithm [132] is used to find an optimal value to discretize the density maps in two levels: lymphocytes and background. The Otsu algorithm is the de facto standard for discriminating foreground and background pixels within an image. In detail, the optimal threshold is identified by minimizing intra-class intensity variance (equivalent to maximizing inter-class variance). Since the Otsu algorithm is the one-dimensional discrete analog of Fisher’s discriminant analysis, this procedure coincides with globally optimizing k-means clustering on the intensity histogram. Pixels with values under the threshold are assigned to the background, while pixels with values over the threshold are assigned to the lymphocyte class. i.e. segmenting the classified one or more objects of interest (lymphocytes), using machine-learning). Otsu thresholding algorithm can be considered separate machine learning algorithm from the classification algorithm).
Regarding claim 15, Bussola et al does not teach the method of claim 11, wherein the one or more points in time differ from the first point in time at which the received digital image is captured.
In a similar field of endeavor, Alfonso et al teaches, the method of claim 11, wherein the one or more points in time differ from the first point in time at which the received digital image is captured (Figure 8 and Epithelial damage induces clustering patterns of immune cells, spatial distribution of immune cells is determined by the radial distribution function g(r) that describes how density varies as a function of the distance from a reference immune cell r75. The function g(r) is calculated at the follicular (day 5) and luteal (day 25) phase of the menstrual cycle, as well as in the middle (day 14). We then consider the mean g(r) values over 10 simulations and fit them to the power law g(r) ≈ b · r−m. The parameter m represents the decreasing slope of g(r) and b the probability to find a neighbor immune cell, i.e. in direct contact. Figure 8 shows estimates of the spatial distribution of immune infiltrates in the lobular epithelium for different damage rates of epithelial cells kdge, hormone levels θ and effector killing rates kkill. Also see, Results, due to the stochastic nature of the model, we performed for each parameter constellation 10 simulations each consisting of 12 menstrual cycles (about 1 year) and monitored the average statistics. The relative number of damaged epithelial and immune cells are provided with respect to the total number of epithelial cells in the simulated cross-sections of TDLUs. i.e. simulating at one or more points in time (12 cycles, also see the radial distribution according to different time points in Figure 8) the spatial distribution of the cells, see Figure 4 and 9).
Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date to incorporate the teachings of Bussola et al (NPL: Quantification of the Immune Content in Neuroblastoma: Deep Learning and Topological Data Analysis in Digital Pathology) in view of Alfonso et al (NPL: In-silico insights on the prognostic potential of immune cell infiltration patterns in the breast lobular epithelium) so that wherein the one or more points in time differ from the first point in time at which the received digital image is captured. Doing so would allow for the interpretation of snapshot-like single time points of breast tissue samples in the context of a mathematical model by considering crucial aspects of the underlying dynamic processes, and would therefore allow for a dynamic understanding of these early events [immune responses] is a crucial step towards investigating the prognostic role of inflammation in the breast lobular tissue (Alfonso et al, Page 4 and Introduction).
Allowable Subject Matter
Claims 1-7 allowed.
The following is an examiner’s statement of reasons for allowance:
Regarding claim 1, the prior art fails to anticipate, disclose, or render obvious the specific limitations presented in claim 1, alone or in combination. The applicant has claimed in the independent claim 1: a computer-implemented method for optimizing the parameter values of a parameter-based model, comprising the steps of:
a) receiving (302) at least two digital images, wherein the at least two digital images comprise the same one or more objects of interest captured at different points in time;
b) classifying (304)
c) segmenting (306)
d) obtaining using a tiling algorithm (308) tiles of the received digital images comprising the classified and segmented one or more objects of interest;
e) mapping on a grid using a mapping algorithm (310) the obtained tiles of the received digital images comprising the classified and segmented one or more objects of interest;
f) extracting the radial distribution function (312) of the classified and segmented one or more objects of interest in the mapped tiles of the received one or more digital images other than the received digital image captured at the first point in time;
g) simulating at several points in time using the parameter-based model (314) with at least one set of parameter values (3 14A) the classified and segmented one or more objects of interest in the mapped tiles of the received digital image captured at the first point in time, wherein the several points in time comprise the points in time at which the received one or more digital images are captured for which the radial distribution function is extracted;
h) extracting the radial distribution functions (316) of the classified and segmented one or more objects of interest in the mapped tiles simulated at the several points in time with the at least one set of parameter values; and
i) calculating a spatial agreement measure (318) based on the overlap at corresponding points in time of the extracted radial distribution function of the classified and segmented one or more objects of interest in the mapped tiles simulated with at least one set of parameter values and of the extracted radial distribution function of the classified and segmented one or more objects of interest in the mapped tiles of the received digital images.
The closest prior art(s) of reference is Bussola et al (NPL: Quantification of the in view of Alfonso et al, which, however, does not teach the specific claimed limitations regarding: extracting the radial distribution functions (316) of the classified and segmented one or more objects of interest in the mapped tiles simulated at the several points in time with the at least one set of parameter values; and i) calculating a spatial agreement measure (318) based on the overlap at corresponding points in time of the extracted radial distribution function of the classified and segmented one or more objects of interest in the mapped tiles simulated with at least one set of parameter values and of the extracted radial distribution function of the classified and segmented one or more objects of interest in the mapped tiles of the received digital images.
. No prior art alone or in combination teaches the specific limitations of claim 1. Therefore, claims 1-7 are allowed.
Regarding claim 9, the examiner would like to note that claim 9 contains corresponding allowable subject matter to claim 1. Claim 1 is objected to as potentially allowable if the 35 U.S.C 101 rejection is resolved.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 20170076448 A1
US 11508120 B2
US 10991097 B2
Klauschen, F., Müller, K. R., Binder, A., Bockmayr, M., Hägele, M., Seegerer, P., ... & International Immuno-Oncology Biomarker Working Group. (2018, October). Scoring of tumor-infiltrating lymphocytes: From visual estimation to machine learning. In Seminars in cancer biology (Vol. 52, pp. 151-157). Academic Press.
Schwen, L. O., Andersson, E., Korski, K., Weiss, N., Haase, S., & Gaire, F. (2018). Data-Driven Discovery of Immune Contexture Biomarkers. Front Oncol. 2018; 8: 627–.
Kather, J. N. et al. In silico modeling of immunotherapy and stroma-targeting therapies in human colorectal cancer. Cancer Res. 77, 6442–6452 (2017).
Wang, D., Khosla, A., Gargeya, R., Irshad, H., & Beck, A. H. (2016). Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718.
Cui, L., Li, H., Hui, W., Chen, S., Yang, L., Kang, Y., Bo, Q., & Feng, J. (2020). A deep learning-based framework for lung cancer survival analysis with biomarker interpretation. BMC bioinformatics, 21(1), 112. https://doi.org/10.1186/s12859-020-3431-z
Yu, K. H., Zhang, C., Berry, G. J., Altman, R. B., Ré, C., Rubin, D. L., & Snyder, M. (2016). Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nature communications, 7, 12474. https://doi.org/10.1038/ncomms12474
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACK PETER KRAYNAK whose telephone number is (703)756-1713. The examiner can normally be reached Monday - Friday 7:30 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACK PETER KRAYNAK/ Examiner, Art Unit 2668
/UTPAL D SHAH/Primary Examiner, Art Unit 2668