DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are currently pending and under exam herein.
Priority
The instant Application is a Continuation of US 17/549,040, filed 13 December 2021, now US Patent 11,727,674, which claims priority to US Provisional application 63/199,185, filed 11 December 2020. Priority is acknowledged for each of claims 1-20 to the EFD of 11 December 2020.
Information Disclosure Statement
The Information Disclosure Statements filed 15 September 2023; 24 September 2024; and 5 February 2026 are in compliance with the provisions of 37 CFR 1.97 and have therefore been considered. Signed copies of the IDS are included with this Office Action.
It is noted that certain references lack appropriate page numbers and/or dates as is required under 37 CFR 1.97. The Examiner has annotated the references herein. Applicant is kindly reminded to provide proper citations in compliance with 37 CFR 1.97 in all future submissions to the office.
Drawings
The Drawings submitted 15 June 2023 are accepted.
It is noted that the Petition to Accept Color Drawings in this application has been granted, as indicated in the separate mailing dated 9 August 2023.
Specification
Note: All reference to the “Specification” in this Office Action is with respect to the
PG Publication: 2023/0343074A1.
Claim Objections
Claims 19 and 20 are objected to because of the following informalities:
Claims 19 and 20 recite, “for each multiplex IHC image, detect mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color; determine the location of each IHC stain color and determining the location of the associated stained target molecules; detect individual cell locations and determining which individual cells are lymphocyte”, wherein the claim language is grammatically inconsistent and should be amended to recite, “for each multiplex IHC image, detect mixture colors comprised of more than one IHC stain and identifying identify the IHC stains that comprise each mixture color; determine the location of each IHC stain color and determining determine the location of the associated stained target molecules; detect individual cell locations and determining determine which individual cells are lymphocyte”.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The instant rejection reflects the framework as outlined in the MPEP at 2106.04:
Framework with which to Evaluate Subject Matter Eligibility:
(1) Are the claims directed to a process, machine, manufacture or composition of matter;
(2A) Prong One: Do the claims recite a judicially recognized exception, i.e. a law of nature, a natural phenomenon, or an abstract idea;
Prong Two: If the claims recite a judicial exception under Prong One, then is the judicial exception integrated into a practical application (Prong Two); and
(2B) If the claims do not integrate the judicial exception, do the claims provide an inventive concept.
Framework Analysis as Pertains to the Instant Claims:
Step 1 Analysis: Are claims directed to process, machine, manufacture/composition of matter
With respect to step (1): Claims 1-19 are directed to a method and a system: YES. Claim 20 is directed to a computer-readable medium having stored thereon a set of computer-executable instructions: NO. Claim 20 is non-statutory as the “computer-readable medium” reads on carrier waves which are non-statutory for including a transitory propagating signals which is not proper patentable subject matter as it does not fit within any of the four statutory categories of invention (In re Nuijten, Federal. Circuit, 2006). It is noted that the recitation of a “non-transitory computer-readable medium” would overcome the rejection with respect to this issue relating to claim 20. Claim 20 will be further treated below in the interest of compact prosecution and with the assumption that this issue will be resolved by claim amendment.
Step 2A, Prong 1 Analysis: Do claims recite abstract idea
With respect to step (2A)(1), the claims recite abstract ideas. The MPEP at 2106.04(a)(2) further explains that abstract ideas are defined as:
mathematical concepts, (mathematical formulas or equations, mathematical relationships and mathematical calculations);
certain methods of organizing human activity (fundamental economic practices or principles, managing personal behavior or relationships or interactions between people); and/or
mental processes (procedures for observing, evaluating, analyzing/ judging and organizing information).
With respect to the instant claims, under the (2A)(1) evaluation, the claims are found herein to recite abstract ideas that fall into the grouping of mental processes (in particular procedures for observing, analyzing and organizing information) and in conjunction with mathematical concepts (in particular mathematical relationships and formulas).
Note: The claim elements are italicized herein to highlight the judicial exceptions in the claim steps and underlined to represent the additional claim elements.
Independent Claim 1:
A computer-implemented method comprising:
a. obtaining, at one or more processors, at least one H&E slide image associated with a biological specimen;
b. obtaining, at the one or more processors, one or more multiplex immunohistochemistry (IHC) images associated with the biological specimen, wherein each multiplex IHC image includes at least two IHC stains, where each IHC stain has a unique color and a unique target molecule;
c. for each multiplex IHC image, detecting, at the one or more processors, mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color;
d. determining, at the one or more processors, the location of each IHC stain color and determining the location of the associated stained target molecules;
e. detecting, at the one or more processors, individual cell locations and determining which individual cells are lymphocytes;
f. for each H&E image and IHC image associated with the biological specimen, aligning and/or registering, at the one or more processors, images such that for each physical location in the biological specimen, all pixels associated with that physical location are aligned;
g. for each target molecule, marking, at the one or more processors, the location on the H&E image that corresponds to the locations of the target molecules stained on the IHC layers;
h. for each cell having a location that corresponds to the location of one or more IHC stains, calculating, at the one or more processors, the percentage of stained pixels overlapping the cell that is associated with each IHC stain to determine an IHC stain profile for each cell; and
i. storing marked and unmarked versions of the H&E image in a training data set.
Independent Claims 19 and 20 are further directed to the system with a processor and memory with computer-executable instructions (claim 19) and a computer-readable medium having stored thereon a set of computer-executable instructions (claim 20) that perform the methods as above.
Hence, the claims explicitly recite numerous elements that, individually and in combination, constitute abstract ideas.
The abstract ideas recited in the claims are evaluated under the Broadest Reasonable Interpretation (BRI) and determined herein to each cover performance either in the mind (calculations by hand or pen and paper or computer as a tool) and performance by mathematical operation (normalization; curve fitting; etc.). When evaluated under the BRI of the claim the following is assessed:
With respect to the step of “detecting, at the one or more processors, mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color”, the step of “detecting” herein is interpreted as a mental observation of making a detecting of color mixtures in a slide, for example, assessment under a microscope after receiving data of a slide.
With respect to the step directed to, “determining, at the one or more processors, the location of each IHC stain color and determining the location of the associated stained target molecules”, the claim step is interpreted under the BRI as one which is mental observation of making a determination of a location by, for example, looking at a microscopic slide and assessing location of each stain therein.
With respect to the step of, “detecting, at the one or more processors, individual cell locations and determining which individual cells are lymphocytes”, the claim step is interpreted under the BRI as one which is mental observation of making a determination of a location by, for example, looking at a microscopic slide and assessing cell type in that location.
With respect to the step of, “aligning and/or registering, at the one or more processors, images such that for each physical location in the biological specimen, all pixels associated with that physical location are aligned”, the claim step is interpreted under the BRI as one which is mental operation of making note of pixel alignment in a slide using pen and paper or using the computer as a tool (register is interpreted only as “marking” or making note of herein).
With respect to the step of, “marking, at the one or more processors, the location on the H&E image that corresponds to the locations of the target molecules stained on the IHC layers”, the claim step is interpreted under the BRI as one which is mental operation of making note of a location in a slide using pen and paper or using the computer as a tool (“marking” is interpreted only as making note of herein).
With respect to the step of, “calculating, at the one or more processors, the percentage of stained pixels overlapping the cell that is associated with each IHC stain to determine an IHC stain profile for each cell”, said operation is interpreted under the BRI as one which is a mathematical process of calculating percentages and is therefore abstract.
These recitations are similar to the concepts of collecting information, analyzing it and providing certain results from the collection and analysis (Electric Power Group, LLC, v. Alstom (830 F.3d 1350, 119 USPQ2d 1739 (Fed. Cir. 2016)), organizing and manipulating information through mathematical correlations (Digitech Image Techs., LLC v Electronics for Imaging, Inc. (758 F.3d 1344, 111 U.S.P.Q.2d 1717 (Fed. Cir. 2014)) and comparing information regarding a sample or test to a control or target data in (Univ. of Utah Research Found. v. Ambry Genetics Corp. (774 F.3d 755, 113 U.S.P.Q.2d 1241 (Fed. Cir. 2014) and Association for Molecular Pathology v. USPTO (689 F.3d 1303, 103 U.S.P.Q.2d 1681 (Fed. Cir. 2012)) that the courts have identified as concepts that can be practically performed in the human mind with pen and paper, and can include mathematical concepts.
Further, see MPEP § 2106.04(a)(2), subsection III. The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation (see, e.g., Benson, 409 U.S. at 67, 65, 175 USPQ at 674-75, 674: noting that the claimed "conversion of [binary-coded decimal] numerals to pure binary numerals can be done mentally," i.e., "as a person would do it by head and hand."); Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1139, 120 USPQ2d 1473, 1474 (Fed. Cir. 2016): holding that claims to a mental process of "translating a functional description of a logic circuit into a hardware component description of the logic circuit" are directed to an abstract idea, because the claims "read on an individual performing the claimed steps mentally or with pencil and paper"). Nor do the courts distinguish between claims that recite mental processes performed by humans and claims that recite mental processes performed on a computer. As the Federal Circuit has explained, "[c]ourts have examined claims that required the use of a computer and still found that the underlying, patent-ineligible invention could be performed via pen and paper or in a person’s mind" (see Versata Dev. Group v. SAP Am., Inc., 793 F.3d 1306, 1335, 115 USPQ2d 1681, 1702 (Fed. Cir. 2015); Mortgage Grader, Inc. v. First Choice Loan Servs. Inc., 811 F.3d 1314, 1324, 117 USPQ2d 1693, 1699 (Fed. Cir. 2016): holding that computer-implemented method for "anonymous loan shopping" was an abstract idea because it could be "performed by humans without a computer").
Steps of the dependent claims have been further analyzed and also found to limit the above judicial exceptions: claim 5: determining the location by setting an intensity threshold and comparing intensity of the stain color; claim 6: generating an overlay; claim 7: detecting using neural network (the neural network is not defined other than by the use of said network and therefore the claim is akin to using a computer as a tool); claim 9: assigning a tissue class to the image; claim 10: associating an immunotherapy response score with a stored image; claim 11: limits the response score type; claims 13: repeating the steps as above; claim 14: optimize…the histology-based machine learning model to receive a subsequent unmarked H&E image and generate a report, which steps directed to “optimization” may be performed mentally by an operator to set specific guidelines wherein the machine learning model is a tool by which to perform optimization and generate a report. It is noted that if the machine learning model included specific operation as how to optimize said images, the claim may be found eligible. Claim 15: further limit on the type of status or the basis of the response class. It is noted that if the alternative embodiment of “generate a report” in claim 14 were directed to an actual “identification” of a response class that includes the number and location of a predicted molecule and comparison to a threshold for each molecule and classes are actually identified, then the claim may be found eligible herein. Claims 15-18 further limit claim 14 above.
Step 2A, Prong 2 Analysis: Integration to a Practical Application
Because the claims do recite judicial exceptions, direction under (2A)(2) provides that the claims must be examined further to determine whether they integrate the abstract ideas into a practical application (MPEP 2106.04(d). A claim can be said to integrate a judicial exception into a practical application when it applies, relies on, or uses the judicial exception in a manner that imposes a meaningful limit on the judicial exception. This is performed by analyzing the additional elements of the claim to determine if the abstract idea is integrated into a practical application (MPEP 2106.04(d).I.; MPEP 2106.05(a-h)). If the claim contains no additional elements beyond the abstract idea, the claim is said to fail to integrate the abstract idea into a practical application (MPEP 2106.04(d).III).
With respect to the instant recitations, the claims recite the additional elements as underlined above.
With respect to the steps directed to “obtaining…at least one H&E slide image” and “obtaining…one or more multiplex immunohistochemistry (IHC) images” said operations are directed to data gathering operations. With respect to the steps that include a computer-implement method, system and processor and memory with instructions executable by a processor said elements serve as the vehicle for data gathering in the claim. As such, the additional elements in the instant claims directed to data gathering perform functions of collecting the data needed to carry out the abstract idea. Data gathering does not impose any meaningful limitation on the abstract idea, or on how the abstract idea is performed. Data gathering steps are not sufficient to integrate an abstract idea into a practical application. (MPEP 2106.05(g). Further to “storing…versions… in a training data set”, there is no particular data structure that is stored and therefore the “storing” operation is an “extra-solution” step of data storage only.
Further, the system, processor, memory and instructions are part of a general purpose computer system as is the recitation of a neural network and machine learning, as there are no details herein wherein of how the specific computer structures are used to implement the judicial exceptions beyond generic computing operations, i.e., the computer elements of the claims do not provide improvements to the functioning of the computer itself (see: DDR Holdings, LLC v. Hotels.com LP); they do not provide improvements to any other technology or technical field (see: Diamond v. Diehr); nor do they utilize a particular machine (see: Eibel Process Co. v. Minn. & Ont. Paper Co.). Hence, these are mere instructions to apply the judicial exception using a computer, and therefore the claim does not provide integration into a practical application of any judicial exception.
Step 2B Analysis: Do Claims Provide an Inventive Concept
The claims are lastly evaluated using the (2B) analysis, wherein it is determined that because the claims recite abstract ideas, and do not integrate that abstract ideas into a practical application, the claims also lack a specific inventive concept. Applicant is reminded that the judicial exception alone cannot provide the inventive concept or the practical application and that the identification of whether the additional elements amount to such an inventive concept requires considering the additional elements individually and in combination to determine if they provide significantly more than the judicial exception. (MPEP 2106.05.A i-vi).
With respect to the instant claims, the additional elements of data gathering described above do not rise to the level of significantly more than the judicial exception. As directed in the Berkheimer memorandum of 19 April 2018 and set forth in the MPEP, determinations of whether or not additional elements (or a combination of additional elements) may provide significantly more and/or an inventive concept rests in whether or not the additional elements (or combination of elements) represents well-understood, routine, conventional activity. Said assessment is made by a factual determination stemming from a conclusion that an element (or combination of elements) is widely prevalent or in common use in the relevant industry, which is determined by either a citation to an express statement in the specification or to a statement made by an applicant during prosecution that demonstrates a well-understood, routine or conventional nature of the additional element(s); a citation to one or more of the court decisions as discussed in MPEP 2106(d)(II) as noting the well-understood, routine, conventional nature of the additional element(s); a citation to a publication that demonstrates the well-understood, routine, conventional nature of the additional element(s); and/or a statement that the examiner is taking official notice with respect to the well-understood, routine, conventional nature of the additional element(s).
With respect to the instant claims, the claim elements directed to a computer system with sequencing information are data gathering elements as in 2A, prong 2 and that under the assessment herein under 2B encompass steps that are routine, well-understood and conventional steps. For example, the courts have recognized the following laboratory techniques as well-understood, routine, conventional activity in the life science arts when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity (see MPEP 2106.05(d)II.): determining the level of a biomarker in blood by any means (Mayo, 566 U.S. at 79, 101 USPQ2d at 1968; Cleveland Clinic Foundation v. True Health Diagnostics, LLC, 859 F.3d 1352, 1362, 123 USPQ2d 1081, 1088 (Fed. Cir. 2017)); detecting DNA or enzymes in a sample (Sequenom, 788 F.3d at 1377-78, 115 USPQ2d at 1157); Cleveland Clinic Foundation 859 F.3d at 1362, 123 USPQ2d at 1088 (Fed. Cir. 2017)) and as such the steps directed to obtaining slide images is equivalent to those routine laboratory steps. Further, the prior art to Koelzer et al. (Virchows Archiv (2019) 474:511–522) disclose that, “The current convergence of new imaging technologies with tissue-based multiplexed immunohistochemistry (IHC) (reviewed in (“Multiplexing”) and molecular phenotyping, our ability to digitize and process large collections of histology slides, and the promise of supporting human interpretation through automated analysis and artificial intelligence will have a dramatic impact on the field” (p.512) and the staining techniques of IHC and H&E (Figure 1). Further the prior art to Hofman et al. (Cancers 2019, 11, 283:22 pages) discuss programs such as HALO that include image classifications that include H&E stain and IHC (p. 10).
With respect to the claims to the system and processor, memory and instruction, the computer-related elements or the general purpose computer do not rise to the level of significantly more than the judicial exception. Further, the Specification also discloses that computer processors and systems, as example, are generic computing systems [0196]; [0198]-[0199]; [0296]. The additional elements are set forth at such a high level of generality that they can be met by a general purpose computer. Therefore, the computer components constitute no more than a general link to a technological environment, which is insufficient to constitute an inventive concept that would render the claims significantly more than an abstract idea (see MPEP 2106.05(b)I-III).
The dependent claims have been analyzed with respect to step 2B and none of these claims provide a specific inventive concept, as they all fail to rise to the level of significantly more than the identified judicial exception.
For these reasons, the claims, when the limitations are considered individually and as a whole, are rejected under 35 USC § 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
1. Claims 1-6, 8-9 and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US2017/0270346 to Ascierto et al.
With respect to claim 1 the prior art to Ascierto et al. disclose:
A computer-implemented method (Ascierto et al. at [0231]:
obtaining, at one or more processors, at least one H&E slide image associated with a biological specimen, (Ascierto et al. disclose H&E images at [0205]);
obtaining, at the one or more processors, one or more multiplex immunohistochemistry (IHC) images associated with the biological specimen, wherein each multiplex IHC image includes at least two IHC stains, where each IHC stain has a unique color and a unique target molecule (Ascierto et al. disclose, “each of the multiple marker images corresponding to a different immune cell marker” [0009]; “said biomarker may be selectively stained and the color signal emitted by said stain may be captured in a respective color channel of a multiplex image” [0031];
for each multiplex IHC image, detecting, at the one or more processors, mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color (Ascierto et al. disclose “the method comprises generating the marker images by applying a color unmixing procedure on a single multiplex slide comprising a tumor tissue section, each color channel corresponding to one of the immune cell markers…” [0089];
determining, at the one or more processors, the location of each IHC stain color and determining the location of the associated stained target molecules (Ascierto et al. disclose, “In addition, or alternatively, the overlay image may display and indicate the location of multiple different immune cell types in the context of various tumors” [0092]; and “Based on the size and location of each individual tumor cell cluster, a set of regions of interest are defined. The slide image (whole slide or portion thereof) is divided into multiple areas, i.e., according to the identified region, for example, the inter-tumor area, peri-tumor area and intra-tumor area” [0115])
detecting, at the one or more processors, individual cell locations and determining which individual cells are lymphocytes (Ascierto et al. disclose determination of cells, including lymphocytes at [0033]);
for each H&E image and IHC image associated with the biological specimen, aligning and/or registering, at the one or more processors, images such that for each physical location in the biological specimen, all pixels associated with that physical location are aligned, (Ascierto et al. disclose image registration at locations at [0182]; [0223]);
for each target molecule, marking, at the one or more processors, the location on the H&E image that corresponds to the locations of the target molecules stained on the IHC layers (Ascierto et al. disclose, “in accordance with the present invention, involves reading images of individual markers or stains from an unmixed multiplex slide, or from multiple slides of serial sections, and computing a tumor region mask from the tumor marker image or hematoxylin and eosin (H&E) stained slide. Based on the size and location of each individual tumor cell cluster, a set of regions of interest are defined. The slide image (whole slide or portion thereof) is divided into multiple areas, i.e., according to the identified region, for example, the inter-tumor area, peri-tumor area and intra-tumor area. FIG. 4 shows an example of a melanoma slide being partitioned into multiple regions. An inter-marker image registration algorithm is used to map the regions to each of the marker images respectively corresponding to immune-histochemistry (IHC) slides from serial sections of IHC slides with different markers” [0115]; and “In embodiments of the present invention, fields of view generated in different images, for example, images of serial tissue sections stained with same or different stains, are registered in a single image. For example, in embodiments of the present invention, FOVs of H&E images are registered in a same coordinate system or image with FOVs identified in an IHC image. In other embodiments of the present invention, FOVs identified in individual color channel images (e.g., individual marker channel images), derived from an image of a biological specimen (e.g., a tissue sample) stained with a multiplex assay, are registered in a single one of the images, merged, and/or registered in a same coordinate system” [0123]);
for each cell having a location that corresponds to the location of one or more IHC stains, calculating, at the one or more processors, the percentage of stained pixels overlapping the cell that is associated with each IHC stain to determine an IHC stain profile for each cell (Ascierto et al. discloses, “The identification of the tumor-related regions is performed according to embodiments of the invention in a two-step approach: at first, the inner-tumor region identification module 113 identifies pixel blobs of high intensity values in the tumor image, e.g. by applying a threshold algorithm or by evaluating annotations and location information already comprised in the tumor image” [0184]… “FOVs are identified as sub-areas within the respective tumor regions or extended tumor regions in dependence on the intensity of groups of pixels in a respective marker image. For example, the regions may be assigned a color (via creation of a heat map) and ranked according to the appearance and/or staining intensity of the groups of pixels (i.e., candidate FOVs) in the marker image of the biological sample” [0091]);
storing marked and unmarked versions of the H&E image in a training data set, (Ascierto et al. disclose data techniques as disclosed for training a machine learning algorithm and as such would inherently include storing those data for a training data set [0098].
With respect to claims 19 and 20, said claims include the system and computer-readable medium to perform the above process and therefore the prior art to Ascierto et al. teach the embodiments as claims in 19 and 20 herein. Ascierto et al. further include: computer-implemented workflow [0115]; computer network [0175]; memory [0176]; processors [0203]; and computer-readable media [0203].
With respect to claim 2, Ascierto et al. disclose H&E images stained only with H&E [0205].
With respect to claim 3, Ascierto et al. disclose, “ H&E image or from a digital image of the same or an adjacent tissue section stained with a tumor-cell specific stain” [0078]; and further discloses, “ A tumor mask is computed from, for example, the unmixed tumor marker channel of a multiplex image, a single stain slide with tumor staining, and/or an H&E slide by a tumor segmentation algorithm in accordance with embodiments of the present invention” [0118].
With respect to claim 4, Ascierto et al. disclose, “For example, the tumor mask may be a mask derived from the H&E image or from a digital image of the same” [0078].
With respect to claim 5, Ascierto et al. disclose, “ processing the marker image for identifying pixel areas whose pixel intensity values are local intensity maxima within the marker image” [0014] and further, “applying a cell detection algorithm on pixel intensity information of the marker image and automatically counting all detected cells within said field of view” [0022]; and “The intensity of the emitted color signal of said stain will correlate with the amount of the biomarker expressed by said immune cells and thus will correlate with the number and density of immune cells of said particular immune cell type in any region of the slide the marker image was derived from” [0031].
With respect to claim 6, Ascierto et al. disclose “a “mask” as used herein is a derivative of a digital image wherein each pixel in the mask is represented as a binary value, e.g. “1” or “0” (or “true” or “false”). By overlaying a digital image with said mask, all pixels of the digital image mapped to a mask pixel of a particular one of the binary values are hidden, removed or otherwise ignored or filtered out in further processing steps applied on the digital image. For example, a mask can be generated from an original digital image by assigning all pixels of the original image with an intensity value above a threshold to true and otherwise false, thereby creating a mask that will filter out all pixels overlaid by a “false” masked pixel” [0114].
With respect to claim 8, Ascierto et al. disclose image deconvolution of IHC slides [0170].
With respect to claim 9, Ascierto et al. disclose, “in embodiments of the present invention, a system automatically generates a region around locations (e.g., tumor regions) in an image corresponding to the presence or identification of melanoma in an image of a stained biological specimen or sample, for example in a Hematoxylin and Eosin (H&E) image [0118] wherein “melanoma” is a tissue class.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
1. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over US2017/0270346 to Ascierto et al., as applied to claim 1 above and in view of US6,463,438 to Veltri et al.
The prior art to Ascierto et al. discloses the methods of claim 1, as discussed above. However, Ascierto et al. does not specifically disclose detecting the cell locations by a neural network.
The prior art to Veltri et al. discloses a neural network system to detect cell abnormalities (abstract). Specifically, Veltri et al. disclose methods for which abnormal cells may be located in an image using neural network procedures that include, for example, feature generation techniques are applied to extract cellular features from an image (col. 8) and feature extraction from cell images wherein feature vectors are processes by said network (col. 10-11).
As such, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have utilized a neural network process to detect cell locations by the techniques as described by Veltri et al. with the methods as disclosed in Ascierto et al. for image processing, as both references are in the same filed of endeavor for detecting images on stained slides (see Veltri et al. at abstract; col. 5, lns. 13-23). The prior art to Ascierto et al. further disclose that, “the automated identification of immune cell types, their respective count and their cell densities in predefined tumor regions within the tumor or at the periphery of the tumor may be beneficial as the reproducibility of immune score computation is further increased” [0030] and that “a human operator of the system may easily add additional rules or modify the criteria and/or thresholds evaluated by existing rules as to support the automated identification of further immune cell types and/or to adapt the rules to more accurately identify immune cell types and/or tumor-related regions relevant for the prognosis of tumors of a particular cancer type” [0040]. Thus motivation exists for modifications to further optimize the teachings of Ascierto et al. Finally Ascierto et al. include that machine learning techniques may be utilized [0098]. As such, one of skill in the art would have had a reasonable expectation in the combination herein.
2. Claims 7 and 10-18 are rejected under 35 U.S.C. 103 as being unpatentable over US2017/0270346 to Ascierto et al., as applied to claim 1 above and in view of US20220405919 to Yip et al.
The prior art to Ascierto et al. discloses the methods of claim 1, as discussed above. However, Ascierto et al. does not specifically disclose limitations as in claims 7 and 10-18 as claimed.
The prior art to Yip et al. discloses the following:
With respect to claim 7, Yip et al. disclose a deep learning system by which a convolutional neural network is employed for assessment of H&E and IHC slides [0010]; [0042]; [0065].
With respect to claim 10, Yip et al. disclose classification frameworks for labeled and unlabeled images [0016]; [0041] and treatment response prediction using the deep learning framework [0085]; [0090]; [0096]. Slide classifiers can include data such as RNA data (clinical data) for prediction of biomarker status [0017].
With respect to claim 11, Yip et al. disclose biomarkers that include TILs [0098]; PD-L1 [0098]; [0259].
With respect to claim 12, Yip et al. disclose patient response to a specific cancer treatment, MSI, and TMB at [0259].
With respect to claim 13, Yip et al. disclose a plurality of training data for the deep learning system that includes various tissue types of training data [0189].
With respect to claim 14, Yip et al. disclose H&E images as discussed above and data associated therewith. Further, Yip et al. disclose optimization to generate predicted biomarker status at [0088]; [0093].
With respect to claim 15, Yip et al. disclose PD-L1 identification at [0098]; [0259].
With respect to claim 16, Yip et al. disclose unlabeled H&E training slides [0041]. Yip et al. disclose labeled biomarker images from training sets at [0102]; [0198]. Yip et al. disclose IHC stained images [0198]. Further, Yip et al. disclose, “For example, new (unlabeled or labeled) histopathology images may be received at the block 610 from the physical clinical records system or primary care system and applied to a trained deep learning framework which applies its trained cell segmentation, tissue classification model, and biomarker classification models, and the block 612 determines a biomarker prediction score. That prediction score can be determined for an entire histopathology image or for different regions across the image. For example, for each image, the block 612 may generate an absolute count of how many biomarkers are on the image, a percentage of the number of cells in tumor regions that are associated with each of the biomarkers, and/or a designation of any biomarker classifications or other information. In some examples, the deep learning framework may identify predicted biomarkers for all identified tissue classes within the image” [0202]; Further, Yip et al. disclose slide pairs that include H&E and IHC stains with corresponding areas [0337].
With respect to claim 17, Yip et al. disclose immunotherapy response predictions at [0085]; [0130].
With respect to claim 18, Yip et al. disclose deep learning neural nets at [0010]; [0042]; [0065] [whole reference].
As such, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Ascierto et al. and Yip et al. to arrive at the instant invention because both references are in the same field of endeavor for stained (H&E and IHC) slide image analysis. The prior art to Ascierto et al. discloses that, in fact, machine learning systems may be utilized and are contemplated for training data sets [0098]. As such, the implementation of the sets as described by Yip et al. would have been met with a reasonable expectation of success. Further, Yip et al. disclose that said systems are applied to H&E and IHC slide images (as described above).
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
1. Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim1-30 of U.S. Patent No. 11,727,674. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of patent ‘674 are directed to:
A method for using a machine learning model to analyze at least one hematoxylin and eosin (H&E) slide image, the method comprising:
a. receiving, at one or more processors, the H&E slide image and providing the H&E slide image to a machine learning model;
b. predicting, at the one or more processors, locations of molecules in the H&E slide image using the machine learning model where the machine learning model is trained using a training data set comprising a plurality of unmarked H&E images and a plurality of marked H&E images, each marked H&E image being associated with one unmarked H&E image and each marked H&E image including a location of one or more molecules determined by analyzing a multiplex IHC image having at least two IHC stains, wherein each IHC stain has a unique color and a unique target molecule and wherein analyzing the multiplex IHC image includes determining an IHC stain that contributes to any two or more overlapping or adjacent IHC stains and comparing each IHC stain in the multiplex IHC image to a threshold;
c. analyzing the number of predicted molecules and locations of the predicted molecules predicted by the machine learning model; and
d. assigning an immunotherapy response class to the H&E slide image, based on the number of predicted molecules and/or locations of the predicted molecules.
The instant claims include steps in claim 1 and dependent claims that include:
A computer-implemented method comprising:
a. obtaining, at one or more processors, at least one H&E slide image associated with a biological specimen;
b. obtaining, at the one or more processors, one or more multiplex immunohistochemistry (IHC) images associated with the biological specimen, wherein each multiplex IHC image includes at least two IHC stains, where each IHC stain has a unique color and a unique target molecule;
c. for each multiplex IHC image, detecting, at the one or more processors, mixture colors comprised of more than one IHC stain and identifying the IHC stains that comprise each mixture color;
d. determining, at the one or more processors, the location of each IHC stain color and determining the location of the associated stained target molecules;
e. detecting, at the one or more processors, individual cell locations and determining which individual cells are lymphocytes;
f. for each H&E image and IHC image associated with the biological specimen, aligning and/or registering, at the one or more processors, images such that for each physical location in the biological specimen, all pixels associated with that physical location are aligned;
g. for each target molecule, marking, at the one or more processors, the location on the H&E image that corresponds to the locations of the target molecules stained on the IHC layers;
h. for each cell having a location that corresponds to the location of one or more IHC stains, calculating, at the one or more processors, the percentage of stained pixels overlapping the cell that is associated with each IHC stain to determine an IHC stain profile for each cell; and
i. storing marked and unmarked versions of the H&E image in a training data set.
Each of the dependent claims are also included in the variants above and as such the claim include variations one of the other and are subject to non-statutory double patenting.
Conclusion
No claims are allowed.
E-mail Communications Authorization
Per updated USPTO Internet usage policies, Applicant and/or applicant’s representative is encouraged to authorize the USPTO examiner to discuss any subject matter concerning the above application via Internet e-mail communications. See MPEP 502.03. To approve such communications, Applicant must provide written authorization for e-mail communication by submitting following form via EFS-Web or Central Fax (571-273-8300): PTO/SB/439. Applicant is encouraged to do so as early in prosecution as possible, so as to facilitate communication during examination.
Written authorizations submitted to the Examiner via e-mail are NOT proper. Written authorizations must be submitted via EFS-Web or Central Fax (571-273-8300). A paper copy of e-mail correspondence will be placed in the patent application when appropriate. E-mails from the USPTO are for the sole use of the intended recipient, and may contain information subject to the confidentiality requirement set forth in 35 USC § 122. See also MPEP 502.03.
Inquiries
Papers related to this application may be submitted to Technical Center 1600 by facsimile transmission. Papers should be faxed to Technical Center 1600 via the PTO Fax Center. The faxing of such papers must conform to the notices published in the Official Gazette, 1096 OG 30 (November 15, 1988), 1156 OG 61 (November 16, 1993), and 1157 OG 94 (December 28, 1993) (See 37 CFR § 1.6(d)). The Central Fax Center Number is (571) 273-8300.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Lori A. Clow, whose telephone number is (571) 272-0715. The examiner can normally be reached on Monday-Thursday from 11:00AM to 9:00PM ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Karlheinz Skowronek can be reached on (571) 272-9047.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to (571) 272-0547.
Patent applicants with problems or questions regarding electronic images that can be viewed in the Patent Application Information Retrieval system (PAIR) can now contact the USPTO’s Patent Electronic Business Center (Patent EBC) for assistance. Representatives are available to answer your questions daily from 6 am to midnight (EST). The toll free number is (866) 217-9197. When calling please have your application serial or patent number, the type of document you are having an image problem with, the number of pages and the specific nature of the problem. The Patent Electronic Business Center will notify applicants of the resolution of the problem within 5-7 business days. Applicants can also check PAIR to confirm that the problem has been corrected. The USPTO’s Patent Electronic Business Center is a complete service center supporting all patent business on the Internet. The USPTO’s PAIR system provides Internet-based access to patent application status and history information. It also enables applicants to view the scanned images of their own application file folder(s) as well as general patent information available to the public.
/Lori A. Clow/Primary Examiner, Art Unit 1687
Prosecution Insights