DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 31-33, 37-40, 44-46 and 50 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US PG PUB. 2021/0350538A1 (hereinafter “Taube”).
Regarding claim 31, Taube discloses a method (Taube, Figs. 1A-1B, Figs. 4-5), comprising:
using at least one computer hardware processor to perform (Taube, ¶0070-0071):
obtaining at least one multiplexed immunofluorescence (MxIF) image of a same tissue sample (Taube, ¶0017-0020; “a specimen (e.g., a slice of a tumor tissue and/or the like) may be prepared in a microscope slide, and positioned under the microscope for image capturing (e.g., for capturing of high power fields or field images) by the microscope. In some implementations, the microscope may be configured to capture multiple overlapping, deep field images, covering a portion, or an entirety, of the specimen, in a redundant manner.”);
obtaining information indicative of locations of cells in the at least one MxIF image, wherein the information indicative of locations of cells includes information indicating cell boundaries of the at least some of the cells (Taube, Fig. 1B, ¶0031-0032; “the characterization platform may identify the pixels that correspond to each cell and/or subcellular component, identify a center of each cell, determine an outline of each cell and/or subcellular component,”);
identifying multiple groups of cells in the at least one MxIF image at least in part by: identifying pixel intensity values for at least some of the cells using the at least one MxIF image and the information indicative of locations of cells; and grouping the at least some of the cells into the multiple groups (Tuabe, ¶0015; “performing image segmentation and obtaining flux measurements (e.g., of cell markers, where appropriate color transformations may be associated with different tissue types, where available color information (e.g., all available color information) may be used to aid tissue/cell segmentation and classification, and where machine learning techniques may be utilized to cluster the color space into multiple regions that each corresponds to biologically meaningful morphological components for different tissue types)”); at least in part by:
for each particular cell of the at least some of the cells, using the information indicating cell boundaries of at least some of the cells to identify pixels within the particular cell's boundary (Taube, ¶0031, 0097-0099; “the characterization platform may perform image segmentation to identify pixels, in the primary areas of the processed field images, that correspond to cells (e.g., some or all of the cells)”), and
calculating at least one feature value using the identified pixel intensity values for the pixels identified within the particular cell's boundary (Tuabe, ¶0032-0034, 0097-0099, Fig. 5; “the information may include data regarding classification types of the one or more cells.”); and
determining similarities among the calculated feature values (Taube, ¶0015, 0032-0033; “machine learning techniques may be utilized to cluster the color space into multiple regions that each corresponds to biologically meaningful morphological components for different tissue types”); and
determining at least one characteristic of the tissue sample using the multiple groups (Taube, Fig. 4, ¶0083; “causing, based on the information, an action to be performed relating to identifying features related to normal tissue, diagnosis or prognosis of disease, or factors used to select therapy (block 460)”).
Regarding claim 32, claim 31 is incorporated, and Taube further discloses wherein determining the at least one characteristic comprises determining information about cell types in the tissue sample, determining cell masks, and/or determining spatial distribution of the cell types (Taube, ¶0032-0033; “the characterization platform may determine cell phenotypic data for an immune cell subset, determine spatial position information of a given cell relative to a tumor and other immune cell subsets, and compute spatial relations between immune cells and tumor cells (e.g., conditioned on expression levels of several different markers, and on positions of such markers relative to tissue boundaries). This permits, for example, correlation of tissue architectural features with clinical parameters, such as age, survival, therapies received, and/or the like, as well as with other profiling features”).
Regarding claim 33, claim 31 is incorporated, and Taube further discloses wherein obtaining the information indicative of locations of cells in the at least one MxIF image comprises using a neural network (Taube, ¶0031; “the characterization platform may, as part of image segmentation, utilize one or more image analysis and/or deep learning techniques (e.g., deep convolutional neural networks) to analyze the multispectral data in the processed field images”)
Regarding claim 37, claim 31 is incorporated, and Taube further discloses wherein the at least one MxIF image comprises a plurality of channels that are associated with respective markers in a plurality of markers (Taube, ¶0014-0015, 0037; “the characterization platform may be configured to execute an automated protocol or pipeline for analyzing multiplex immunofluorescent and/or immunohistochemistry data generated on tissue sections, including levels of expression of multiple markers.”) .
Claim 38 recites a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform steps which correspond to the steps recited in method claim 31, the rejection of which is applicable here, and Taube further discloses at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions (Taube, ¶0070-0071).
Claim 39 recites a system having features which correspond to the elements recited in method claim 32, the rejection of which is applicable here.
Claim 40 recites a system having features which correspond to the elements recited in method claim 33, the rejection of which is applicable here.
Claim 44 recites a system having features which correspond to the elements recited in method claim 37, the rejection of which is applicable here.
Claim 45 recites at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform steps corresponding to the steps recited in method claim 31, the rejection of which is applicable here, and Taube further discloses at least one non-transitory computer-readable storage medium storing processor-executable instructions (Taube, ¶0070-0071).
Claim 46 recites at least one non-transitory computer-readable storage medium having features which correspond to the elements recited in method claim 32, the rejection of which is applicable here.
Claim 50 recites at least one non-transitory computer-readable storage medium having features which correspond to the elements recited in method claim 37, the rejection of which is applicable here.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 34, 41 and 47 are rejected under 35 U.S.C. 103 as being unpatentable over Taube, as applied to claims 31, 38 and 45 above, in view of “U-Net: Convolutional Networks for Biomedical Image Segmentation” (hereinafter “Ronneberger”; applicant-submitted prior art).
Regarding claim 34, claim 33 is incorporated, and Taube does not expressly teach the limitations as further claimed, but, in an analogous field of endeavor, Ronneberger does as follows.
Ronneberger teaches wherein the neural network is implemented using a U-Net architecture or a region-based convolutional neural network architecture, and wherein the neural network comprises at least one million parameters (Ronneberger, Introduction, Fig. 1, Conclusion; U-net architecture of a convolutional neural network, which are commonly known to comprise millions of parameters, is applied to biomedical image segmentation.).
Ronneberger is considered analogous art because it pertains to biomedical image segmentation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the image segmentation taught by Taube to be performed using a U-net architecture comprising millions of parameters, as taught by Ronneberger, in order to improve segmentation accuracy (Ronneberger, Conclusion).
Claim 41 recites a system having features which correspond to the elements recited in method claim 34, the rejection of which is applicable here.
Claim 47 recites at least one non-transitory computer-readable storage medium having features which correspond to the elements recited in method claim 34, the rejection of which is applicable here.
Claims 35-36 and 42-43 are rejected under 35 U.S.C. 103 as being unpatentable over Taube, as applied to claims 31 and 38 above, in view of “CGC-Net: Cell graph convolutional network for grading of colorectal cancer histology images” (hereinafter “Zhou”; applicant-submitted prior art).
Regarding claim 35, claim 33 is incorporated, and Taube does not expressly teach the limitations as further claimed, but, in an analogous field of endeavor, Zhou does as follows.
Zhou teaches wherein grouping the at least some of the cells comprises clustering using a graph neural network different from the neural network (Zhou, Abstract, Sections 3.2-3.3, Figure 3, Section 4.3; "One Adaptive GraphSage is applied to generate the embedding matrix M(i). Meanwhile, the nodes are passed through another GraphSage followed by a linear function to generate the assignment matrix S⁽ⁱ⁾ the probability of each node being assigned to each cluster" after which "the clusters are considered as new nodes for the following layer" followed by hierarchical combination of features for cell classification).
Zhou is considered analogous art because it pertains to tissue image analysis for tissue microenvironment classification. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the cell and tissue classification method taught by Taube to include determining the distribution of cell clusters in the tissue image using a graph neural network, as taught by Zhou, in order to more efficiently identify cell-to-cell interactions between various present cell types (Zhou, Section 2).
Regarding claim 36, claim 35 is incorporated, and Zhou in the combination further teaches wherein the clustering comprises performing hierarchical clustering, density-based clustering, k-means clustering, self-organizing map clustering, or minimum spanning tree clustering (Zhou, Section 3.3; "Combining Hierarchical features for graph-level classification: We utilise a max operation for the node embeddings at each stage to get a fixed-size representation. Then the concatenation of multi-level representations is fed into the linear layer to get the prediction for 3-class classification.").
Claim 42 recites a system having features which correspond to the elements recited in method claim 35, the rejection of which is applicable here.
Claim 43 recites a system having features which correspond to the elements recited in method claim 36, the rejection of which is applicable here.
Claims 48 and 49 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Taube in view of Ronneberger, as applied to claim 47 above, and further in view of “CGC-Net: Cell graph convolutional network for grading of colorectal cancer histology images” (hereinafter “Zhou”; applicant-submitted prior art).
Regarding claim 48, claim 47 is incorporated, and the combination of Taube in view of Ronneberger does not expressly teach the limitations as further claimed, but, in an analogous field of endeavor, Zhou does as follows.
Zhou teaches wherein grouping the at least some of the cells comprises clustering using a graph neural network different from the neural network (Zhou, Abstract, Sections 3.2-3.3, Figure 3, Section 4.3; "One Adaptive GraphSage is applied to generate the embedding matrix M(i). Meanwhile, the nodes are passed through another GraphSage followed by a linear function to generate the assignment matrix S⁽ⁱ⁾ the probability of each node being assigned to each cluster" after which "the clusters are considered as new nodes for the following layer" followed by hierarchical combination of features for cell classification).
Zhou is considered analogous art because it pertains to tissue image analysis for tissue microenvironment classification. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the cell and tissue classification algorithm taught by the combination of Taube in view of Ronneberger to include determining the distribution of cell clusters in the tissue image using a graph neural network, as taught by Zhou, in order to more efficiently identify cell-to-cell interactions between various present cell types (Zhou, Section 2).
Regarding claim 49, claim 48 is incorporated, and Zhou in the combination further teaches wherein the clustering comprises performing hierarchical clustering, density-based clustering, k-means clustering, self-organizing map clustering, or minimum spanning tree clustering (Zhou, Section 3.3; "Combining Hierarchical features for graph-level classification: We utilise a max operation for the node embeddings at each stage to get a fixed-size representation. Then the concatenation of multi-level representations is fed into the linear layer to get the prediction for 3-class classification.").
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 31, 38 and 45 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 30 of U.S. Patent No. 11,915,422. Although the claims at issue are not identical, they are not patentably distinct from each other because, aside from minor differences in claim language, the broadly recited features of each of claims 31, 38 and 45 of the instant application are anticipated by claim 30 of the patent, which is narrower in scope. The table below shows the corresponding features between the claim of the patent and claim 31 of the application, by way of example.
Claim 30 of the Patent
Claim 31 of the Instant Application
A method, comprising:
using at least one computer hardware processor to perform:
obtaining at least one multiplexed immunofluorescence (MxIF) image of a same tissue sample, wherein the at least one MxIF image comprises a plurality of channels that are associated with respective markers in a plurality of markers;
obtaining, using a neural network, information indicative of locations of cells in the at least one MxIF image, wherein the neural network is implemented using a U-Net architecture or a region-based convolutional neural network architecture, and wherein the neural network comprises at least one million parameters;
identifying multiple groups of cells in the at least one MxIF image at least in part by:
identifying pixel intensity values for at least some of the cells using the at least one MxIF image and the information indicative of locations of cells; and
grouping the at least some of the cells into the multiple groups at least in part by clustering using a graph neural network different from the neural network, wherein the information indicative of locations of cells includes information indicating cell boundaries of the at least some of the cells, wherein the clustering comprises grouping the at least some of the cells by:
for each particular cell of the at least some of the cells, using the information indicating cell boundaries of at least some of the cells to identify pixels within the particular cell's boundary, and
calculating at least one feature value using the identified pixel intensity values for the pixels identified within the particular cell's boundary; and
determining similarities among the calculated feature values; and
determining at least one characteristic of the tissue sample using the multiple groups, wherein determining the at least one characteristic comprises determining information about cell types in the tissue sample, determining cell masks, and/or determining spatial distribution of the cell types.
A method, comprising:
using at least one computer hardware processor to perform:
obtaining at least one multiplexed immunofluorescence (MxIF) image of a same tissue sample;
obtaining information indicative of locations of cells in the at least one MxIF image, wherein the information indicative of locations of cells includes information indicating cell boundaries of the at least some of the cells;
identifying multiple groups of cells in the at least one MxIF image at least in part by:
identifying pixel intensity values for at least some of the cells using the at least one MxIF image and the information indicative of locations of cells; and
grouping the at least some of the cells into the multiple groups at least in part by:
for each particular cell of the at least some of the cells, using the information indicating cell boundaries of at least some of the cells to identify pixels within the particular cell's boundary, and
calculating at least one feature value using the identified pixel intensity values for the pixels identified within the particular cell's boundary; and
determining similarities among the calculated feature values; and
determining at least one characteristic of the tissue sample using the multiple groups.
Dependent claims 36, 43 and 49 are additionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 31 of U.S. Patent No. 11,915,422. Although the claims at issue are not identical, they are not patentably distinct from each other because, aside from minor differences in claim language, the recited corresponding features of each of claims 36, 43 and 49 of the instant application are anticipated by the features of claim 31 of the patent.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The additionally cited reference US 2009/0238457 pertains to segmentation of clustered cells in microscopy images.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMAH A BEG whose telephone number is (571)270-7912. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HENOK SHIFERAW can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMAH A BEG/Primary Examiner, Art Unit 2676