DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed April 16th 2026 has been entered. Claims 1-20 are pending in the application. Claims 16-20 are newly added. Applicant’s amendments to the Claims 1, 12, and 15 have overcome the rejections previously set forth in the Non-Final Office Action mailed October 16th 2025. Newly found references Stumpe (US 20200394825 A1), Freytag (WO 2021198247 A1), Seppo (US 20150065371 A1), and Chukka (US 20160019695 A1) were used for the amended claims.
Response to Arguments
The Examiner thanks the Applicant for the thorough review of the previous Office Action and for the clarifications presented at the interview. Because some of the same art was cited in the new grounds of rejection, the relevant arguments are addressed below.
The Applicant argues that: “Bhargava fails to contemplate that these images can be combined one or more times to produce one or more new multiplexed virtual images” (Pg. 8). The Examiner respectfully disagrees, because Bhargava teaches: “Using a combination of multiple staining results, it is possible to subsequently deduce the cell types and/or molecular transformations present” [0143] under the heading “Multiplexing Computed Stain Images”.
The Applicant also argues that: “Batenchuk still fails to cure Bhargava's aforementioned deficiencies. Like Bhargava, Batenchuk is completely silent as to virtual staining patterns or virtually stained image annotations. As such, Batenchuk's operations on physically stained images do not provide support for the claimed annotating, parsing, and overlaying steps that operate on detected virtual staining patterns in a virtually stained image” (Pg. 10).
The Examiner respectfully disagrees that “Batenchuk is completely silent as to virtual staining patterns or virtually stained image annotations”, because Batenchuk teaches: “The image may be of an immuno-histochemistry (IHC) slide, spatial transcriptomics, virtual stain, and/or a multiplexed immunofluorescence (mIF) slide” [0029] (emphasis added) and that “Image processing system 120 can include a feature detector 144 that identifies and tags each patch as depicting one or more biological features. The feature detector may use detected edges, colors (e.g., RGB value of a pixel, block of pixels or area defined by one or more edges), intensities (e.g., of a pixel, block of pixels or area defined by one or more edges), and/or shapes (e.g., defined by detected edges, one or more pixels, etc.) to perform […] pattern analysis” [0047] (emphasis added). Put simply, Batenchuk teaches a feature detector that may operate on a virtual stain that performs pattern analysis to identify biological features. The Examiner submits that it would be reasonable to interpret the biological features as patterns detected from the virtual stain, and that said biological features may be analogous to virtual staining patterns.
Batenchuk further teaches that: “In some instances, image annotation device 116 can transform detected annotated biological features as input annotation data (e.g., that indicate which pixel(s) of the image correspond to particular annotation characteristics)” [0032]. Therefore, the Examiner understands Batenchuk also to teach generating annotations of the biological features.
The Applicant also argues that: “As explained above with respect to claim 15, a skilled artisan reading Bhargava would have no reason to depart from Bhargava's one-stain-per-output model, which generates computed stain images that replicate physical chemical stains. Batenchuk fails to provide any such reason” (Pg. 10). Assuming, for the sake of argument, that Bhargava is limited to a “one-stain-per-output model”, the Examiner submits that one of ordinary skill in the art would be motivated to combine the teachings of Batenchuk with Bhargava, because only producing one stain every output would not be as efficient as producing multiple at once. Batenchuk teaches that: “Performing a TMB diagnostic can take up to three weeks which—on average, for patients with Stage-IV Non-Small Cell Lung Cancer (NSCLC)—represents 25-40% of patients' remaining life span when not treated. Further, the DNA sequencing frequently fails. There is insufficient material for the test 40% of the time, and the test fails quality control 40% of the time. Determining an optimal treatment plan in this instance is tricky, in that 55% of patients are TMB low or negative, and an assumption of either TMB test result would frequently be wrong and lead to sub-optimal treatment. Thus, the time from biopsy to a diagnostic test result may take up a significant portion of the patient's remaining life. Thus, it would be advantageous to identify a fast and reliable technique for estimating a patient's PD-L1, TMB, and IFNγ status” [0005-0006].
Arguments not explicitly addressed in this section were considered moot because the new grounds of rejection does not rely on any previously cited reference for a teaching specifically challenged in that respective argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 3, 4, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Bhargava (US 20140270457 A1) in view of Batenchuk (US 20200123618 A1) and Stumpe (US 20200394825 A1).
Regarding claim 1:
Bhargava teaches:
A method for generating inference-based virtually stained image annotations, comprising:
providing one or more neural networks (Bhargava: feed-forward neural network [00119]) executed by image processing software running on one or more processors of a computing device (Bhargava: The computer-executable instructions can be part of, for example, a dedicated software application […] Such software can be executed, for example, on a single local computer [0049]), wherein the one or more neural networks are trained with a plurality of images of chemical stains (Bhargava: Thus, a plurality of samples stained with a target stain (such as H&E, Bismark Brown, Nile Blue or antibody specific for a target protein […] can be used to train a network [0045]) of one or more first endogenous signals (see Note 1A);
obtaining data corresponding to a biological sample (Bhargava: in process block 110, a spectroscopic image (e.g., IR absorbance data) of an unstained sample (such as one containing a tumor or portion thereof) are acquired, [0052]);
Note 1A: Bhargava specifically defines some of the terms used in their disclosure: “Stain: A dye or other label used in microscopy, for example to enhance contrast in the microscopic image or to detect particular structures in biological tissues, such as cell populations” [0028] (emphasis added) and “Detect: To determine if an agent (such as a signal, protein, cellular structure, organism, or cell) is present or absent”.
Therefore, when Bhargava teaches a stain, one of ordinary skill in the art would understand that the stain may be of a signal. Additionally, because Bhargava teaches the signal as an alternative to endogenous structures such as a protein, cell, or cellular structure, the Examiner submits that it would be obvious to a PHOSITA that the signal may be endogenous.
Bhargava fails to teach:
obtaining producing an image of the biological sample including one or more second endogenous signals identified by annotating techniques;
detecting one or more virtual staining patterns of the one or more second endogenous signals in the image of the biological sample using the one or more neural networks;
obtaining and producing an image of the biological sample including one or more second endogenous signals identified by annotating techniques;
overlaying the virtual staining patterns detected in the image of the biological sample using spatial matching techniques to create the inference-based virtually stained image annotations.
Batenchuk teaches:
obtaining and producing an image of the biological sample (Batenchuk: An image of a biological sample can be accessed, Abstract) including one or more second endogenous signals (Batenchuk: The image can be segmented into a set of patches. Edge detection performed on each patch can be used to identify one or more biological features represented by the patch, Abstract) identified by annotating techniques (Batenchuk: An identification of the one or more biological features depicted by a patch may be assigned (e.g., through a label, a data structure, annotation, metadata, image parameter and/or the like) to the patch [0048]);
detecting one or more virtual staining patterns of the one or more second endogenous signals (see Note 12A) in the image of the biological sample using the one or more neural networks (Batenchuk: Feature detector may use segmentation, convolutional neural network, object recognition, pattern analysis, machine-learning, a pathologist, and/or the like to detect the one or more biological features [0048]);
obtaining and producing an image of the biological sample (Batenchuk: image processing system 120 processes an image to generate one or more feature maps. A feature map can be data structure that includes an indication of the one or more biological features assigned to each patch of the image [0049]) including one or more second endogenous signals (Batenchuk: image processing system to detect biological features to infer PD-L1 status [0022]; see Note 1B) identified by annotating techniques (Batenchuk: An identification of the one or more biological features depicted by a patch may be assigned (e.g., through a label, a data structure, annotation, metadata, image parameter and/or the like) to the patch. [0048]);
Note 1B: Batenchuk teaches: “PD-L1 protein expression can represent or indicate a level of expression of corresponding PD-1 receptor. PD-L1, upon binding to the PD-1 receptor, may cause a transmission of an inhibitory signal that reduces antigen-specific T-cells and apoptosis in regulatory T-cells.” [0053]. That is, inferring PD-L1 status as cited above from [0022] may include detecting an endogenous signal (such as the inhibitory signal in [0053]), and therefore, the biological features taught by Batenchuk are analogous to the endogenous signals of the present application.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Batenchuk with Bhargava. Detecting one or more virtual staining patterns of the one or more second endogenous signals and obtaining and producing an image of the biological sample including one or more second endogenous signals identified by annotating techniques, as in Batenchuk, would benefit the Bhargava teachings by quickly and accurately measuring biological features in the biological sample: “Performing a TMB diagnostic can take up to three weeks which—on average, for patients with Stage-IV Non-Small Cell Lung Cancer (NSCLC)—represents 25-40% of patients' remaining life span when not treated. Further, the DNA sequencing frequently fails. There is insufficient material for the test 40% of the time, and the test fails quality control 40% of the time. Determining an optimal treatment plan in this instance is tricky, in that 55% of patients are TMB low or negative, and an assumption of either TMB test result would frequently be wrong and lead to sub-optimal treatment. Thus, the time from biopsy to a diagnostic test result may take up a significant portion of the patient's remaining life. Thus, it would be advantageous to identify a fast and reliable technique for estimating a patient's PD-L1, TMB, and IFNγ status. (Batenchuk, [0005-0006]).
Bhargava in view of Batenchuk still fails to explicitly teach:
overlaying the virtual staining patterns detected in the image of the biological sample using spatial matching techniques to create the inference-based virtually stained image annotations.
Stumpe teaches:
overlaying the virtual staining patterns detected in the image (see Note 1C) of the biological sample (Stumpe: The virtual stained image could be displayed also as an overlay on the H&E stained image of the specimen, […] essentially recoloring the H&E image into the respective stain image. [0066]) using spatial matching techniques (Stumpe: One possible approach is to perform a coarse global matching via rotation and transformation on a thumbnail level, and then match pairs of image patches. [0049]) to create the inference-based virtually stained image annotations (see Note 1D).
Note 1C: The Examiner submits that it would be obvious to overlay the virtual stain based on the virtual staining patterns detected in the image with the biological sample, because Stumpe teaches: “The underlying assumption is that there is a causation or correlation between the morphological features in the tissue, and the local protein expression patterns.” [0057] and “Since the training data for the virtual staining model will be pairs of (almost) identical image patches of H&E vs, special stain (or unstained vs. special stain) the images need to be aligned as perfectly as possible. With whole slide image sizes of 100,000×100,000 pixels and potentially local tissue deformations, the registration is not a trivial task of an affine transform but more likely requires some local warping as well.” [0049].
Note 1D: Batenchuk teaches that: “image annotation device 116 can transform detected annotated biological features as input annotation data (e.g., that indicate which pixel(s) of the image correspond to particular annotation characteristics). For example, image annotation device 116 associates a patient's smoking history with particular pixels in the image that depict possible smoking damage or staining” [0032]. Put simply, Batenchuk teaches that an image may be transformed to include annotation data about the biological features (previously analogized to virtual staining patterns of endogenous signals in Note 1B above)
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Stumpe with Bhargava in view of Batenchuk. It would be obvious to one of ordinary skill in the art to overlay virtual stain data in order to annotate the image of the biological sample as in Stumpe, because overlaying the virtual staining pattern clearly communicates to a user what portion of the image each pattern relates to: “Explainability—the importance of showing pathologists the visualization of the stain that leads to the diagnosis can hardly be overestimated. It helps to establish trust in the methodology of this disclosure as well as any particular predictions that accompany the virtual stain image. This is especially strong if we can overlay the virtual IHC with the actual H&E images or an unstained image of the tissue sample.” (Stumpe, [0079]).
Regarding claim 2:
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 1 (as shown above),
Bhargava fails to explicitly teach:
wherein the annotations comprise features used by the one or more neural networks to perform semantic segmentation.
Batenchuk teaches:
wherein the annotations comprise features used by the one or more neural networks to perform semantic segmentation (Batenchuk: In some instances, segmentation can be performed, such that individual pixels and/or detected figures can be associated with a particular structure such as, but not limited to, the biological sample [0034]; see Note 2A).
Note 2A: Batenchuk teaches: “Feature detector may use segmentation, convolutional neural network, object recognition, pattern analysis, machine-learning, a pathologist, and/or the like to detect the one or more biological features,” [0048]. Batenchuk uses “and/or”, indicating that both a convolutional neural network and segmentation may be used to detect biological features. Therefore, it would be obvious to one of ordinary skill in the art to perform semantic segmentation with the one or more neural networks using features.
Regarding claim 3:
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 1 (as shown above), wherein the obtaining and producing of the image of the biological sample comprises incorporates sequencing or imaging mass spectroscopy (Bhargava: The disclosed methods can include obtaining a spectroscopic image (e.g., infrared (IR) imaging data) of the sample, Abstract).
Regarding claim 4:
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 1 (as shown above), wherein the obtaining and producing of the image of the biological sample (Batenchuk: Image collection system 104 can be configured such that each portion of a sample (e.g., slide) is manually loaded onto a stage prior to imaging and/or such that a set of portions of one or more samples (e.g., a set of slides) are automatically and sequentially loaded onto a stage [0029]) comprises incorporates an immunohistochemistry or immunofluorescence technique (Batenchuk: The image may be of an immuno-histochemistry (IHC) slide, [0029]).
Regarding claim 11:
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 1 (as shown above), further comprising multiplexing the overlayed virtual staining patterns (Bhargava: Multiplexing Computed Stain Images, [0142]) with one or both of existing virtual stains (Bhargava: Using a combination of multiple staining results, it is possible to subsequently deduce the cell types and/or molecular transformations present, [0143]) or conventional assay readouts.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Bhargava (US 20140270457 A1) in view of Batenchuk (US 20200123618 A1), Stumpe (US 20200394825 A1) and Gaiser (NPL: Automated analysis of protein expression and gene amplification within the same cells of paraffin-embedded tumour tissue).
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 4 (as shown above),
Bhargava in view of Batenchuk and Stumpe fails to teach:
wherein the immunohistochemistry or immunofluorescence technique comprises directly comparing virtual staining patterns for two or more antibody clones on a tissue section.
Gaiser teaches:
wherein the immunohistochemistry or immunofluorescence technique comprises directly comparing virtual (see Note 5B) staining patterns for two or more antibody clones on a tissue section (see Note 5A).
Note 5A: Gaiser teaches: “For IHC CD133 detection an anti-CD133 rabbit monoclonal antibody (1:20; clone C24B9, Cel} Signaling, Danvers, MA, USA) was used,” (Pg. 2, Section 2.2: Algorithm for combined IHC and FISH, par. 1). Gaiser further teaches: “Slides were washed in 1x PB8 and thereafter incubated for I h at room temperature (RT) with the secondary Goal Anti-Rabbit IgG-FITC antibody (1:200; clone 4030-02, SouthernBiotech, Birmingham, AL, USA).” (Pg. 2, Section 2.2: Algorithm for combined IHC and FISH, par. 1). Gaiser further teaches: “Performing fluorescence immunophenotyping and FISH on the same tumour tissue slide enables the comparison of protein expression and gene copy numbers detected within the same tumour cells.” (Pg. 3, Section 4: Discussion, par. 1). That is, using two antibody clones, Gaiser was enabled to compare protein expression and gene copy numbers within the same cells. To the best of the examiner’s knowledge, this method is analogous to comparing staining patterns for two or more antibody clones on a tissue section.
Note 5B: Gaiser teaches that an “we developed a protocol for subsequent protein and gene copy number detection and analysis on the same slide using an automated image acquisition and analysis software including image relocation and a function to mark and count in areas of interest (Fig. 1).” (Pg. 2, par. 1). Gaiser further showcases a “Screen capture of image analysis” in Figure 3. It is reasonable to conclude that while Gaiser does not virtually stain images, Gaiser does compare “virtual” staining patterns (i.e., the patterns have been digitized before comparison).
Furthermore, while Gaiser does not explicitly teach virtually staining the tissue sample, Bhargava teaches that a computer may generate a stained image without having to stain the tissue manually: “FIGS. 1C-1E show how the disclosed methods can be used to obtain an image (FIG. 1E) that looks similar to the H&E image (FIG. 1B), without staining the tissue.” [0042]. Therefore, it is reasonable to conclude that it would be obvious to one of ordinary skill in the art to simulate the method of Gaiser using a virtual staining system on a computer.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Gaiser with Bhargava in view of Batenchuk and Stumpe. Comparing staining patterns for two or more antibody clones on a tissue section, as in Gaiser, would benefit the Bhargava in view of Batenchuk and Stumpe teachings by enabling the user to quickly identify patterns they are looking for without staining the physical tissue sample.
Claims 12, 13, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Bhargava (US 20140270457 A1) in view of Batenchuk (US 20200123618 A1).
Regarding claim 12:
Bhargava teaches:
A method of generating virtually stained image annotations, comprising:
obtaining an image of a biological sample (Bhargava: obtaining a spectroscopic image of the sample, [0007]);
generating, based on the image, a virtually stained image of the biological sample using a machine learning algorithm (Bhargava: information is applied to a network, such as a neural network, […] to produce an algorithm and parameters for the network needed to generate an output computed stain image for the test sample [0074]) executed via a computer program running on a processor (Bhargava: Any of the methods described herein can be implemented by computer-executable instructions [...] Such instructions can cause a computer to perform the method. The technologies described herein can be implemented in a variety of programming languages [0109]);
overlaying the staining images detected in the image of the biological sample (Bhargava: multiple stained images can be generated from the sample for different regions and at different scales. Images can be overlaid, merged or multiply highlighted, [0016]).
Bhargava fails to teach:
detecting, via the machine learning algorithm, virtual staining patterns of endogenous signals in the biological sample;
annotating detected virtual staining patterns in the virtually stained image of the biological sample with annotations for one or more biomarkers;
parsing the annotations of the virtually stained image of the biological sample; and
overlaying the virtually stained image of the biological sample with the parsed annotations.
Batenchuk teaches:
detecting, via the machine learning algorithm, virtual staining patterns of endogenous signals (see Note 12A) in the biological sample (Batenchuk: Feature detector may use segmentation, convolutional neural network, object recognition, pattern analysis, machine-learning, a pathologist, and/or the like to detect the one or more biological features [0048]);
annotating detected virtual staining patterns in the virtually stained image of the biological sample with annotations for one or more biomarkers (Batenchuk: Feature detector 144 may use the original image and/or just the processed image to detect one or more biological features that are shown by the image [0048]);
parsing the annotations of the virtually stained image of the biological sample (Batenchuk: Each patch can be analyzed by the feature detector to determine the one or more biological features shown in the patch [0048]); and
transforming the virtually stained image of the biological sample with the parsed annotations (Batenchuk: image annotation device 116 can transform detected annotated biological features as input annotation data (e.g., that indicate which pixel(s) of the image correspond to particular annotation characteristics). For example, image annotation device 116 associates a patient's smoking history with particular pixels in the image that depict possible smoking damage or staining [0032]; see also Note 12B).
Note 12A: Batenchuk teaches: “The image may be of an immuno-histochemistry (IHC) slide, spatial transcriptomics, virtual stain, and/or a multiplexed immunofluorescence (mIF) slide” [0029] (emphasis added) and that “Image processing system 120 can include a feature detector 144 that identifies and tags each patch as depicting one or more biological features. The feature detector may use detected edges, colors (e.g., RGB value of a pixel, block of pixels or area defined by one or more edges), intensities (e.g., of a pixel, block of pixels or area defined by one or more edges), and/or shapes (e.g., defined by detected edges, one or more pixels, etc.) to perform […] pattern analysis” [0047] (emphasis added). Put simply, Batenchuk teaches a feature detector that may operate on a virtual stain that performs pattern analysis to identify biological features. The Examiner submits that it would be reasonable to interpret the biological features as patterns detected from the virtual stain, and that said biological features may be analogous to virtual staining patterns.
The specification of the present application only mentions endogenous signals once, but does not define the term: “the disclosure utilizes patterns from endogenous signals to digitally generate the tissue staining patterns” [0017]. Therefore, the Examiner interprets endogenous signal under its plain and ordinary meaning, namely, any signal that originates from an organism.
Batenchuk further teaches: “FIG. 1 depicts an exemplary system for using an image processing system to extract image parameters that define biological features used to infer PD-L1 status” [0014], and that “PD-L1 protein expression can represent or indicate a level of expression of corresponding PD-1 receptor. PD-L1, upon binding to the PD-1 receptor, may cause a transmission of an inhibitory signal that reduces antigen-specific T-cells and apoptosis in regulatory T-cells.” [0053]. In other words, the PD-L1 expression is an example of a signal originating from the body, which the biological features or patterns are used to detect. Therefore, the Examiner submits that it is reasonable to conclude that the biological features taught by Batenchuk are analogous to “virtual staining patterns of endogenous signals” as claimed.
Note 12B: The Examiner submits that one of ordinary skill in the art would find it obvious to overlay the virtually stained image of the biological sample with the parsed annotations, because:
Bhargava teaches that stained images may be overlaid on a biological sample in [0016] cited above, and
Batenchuk showcases in Fig. 5 that a biological sample may have annotations overlaid onto the image of the sample (e.g., the annotated “Leucocyte” figure is overlaid onto the full image of the biological sample).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Batenchuk with Bhargava. Annotating the virtually stained image of the biological sample with annotations for one or more biomarkers; parsing the annotations of the virtually stained image of the biological sample; and overlaying the virtually stained image of the biological sample with the parsed annotations, as in Batenchuk, would benefit the Bhargava teachings by quickly and accurately measuring biological features in the biological sample: “Performing a TMB diagnostic can take up to three weeks which—on average, for patients with Stage-IV Non-Small Cell Lung Cancer (NSCLC)—represents 25-40% of patients' remaining life span when not treated. Further, the DNA sequencing frequently fails. There is insufficient material for the test 40% of the time, and the test fails quality control 40% of the time. Determining an optimal treatment plan in this instance is tricky, in that 55% of patients are TMB low or negative, and an assumption of either TMB test result would frequently be wrong and lead to sub-optimal treatment. Thus, the time from biopsy to a diagnostic test result may take up a significant portion of the patient's remaining life. Thus, it would be advantageous to identify a fast and reliable technique for estimating a patient's PD-L1, TMB, and IFNγ status. (Batenchuk, [0005-0006]).
Regarding claim 13:
Bhargava in view of Batenchuk teaches:
The method of claim 12 (as shown above),
Bhargava fails to explicitly teach:
further comprising semantically segmenting the virtually-stained image of the biological sample.
Batenchuk teaches:
further comprising semantically segmenting the virtually-stained image of the biological sample (Batenchuk: In some instances, segmentation can be performed, such that individual pixels and/or detected figures can be associated with a particular structure such as, but not limited to, the biological sample [0034]; see Note 2A above).
Regarding claim 16:
Bhargava in view of Batenchuk teaches:
The method of claim 12 (as shown above),
wherein annotating the detected virtual staining patterns comprises
separating the virtually stained image into a plurality of component images, each depicting a respective biomarker of the one or more biomarkers (Batenchuk: segmentation can be performed, such that individual pixels and/or detected figures can be associated with a particular structure such as, but not limited to, the biological sample [0034]), and
recoloring at least one of the plurality of component images such that each respective biomarker is visually differentiated (Batenchuk: The feature map may be rendered (e.g., as depicted in FIGS. 4 and 5) in which the identifier of each patch may be represented as a color [0049]; see Note 16A).
Note 16A: The right half of Figure 4 of Batenchuk depicts a feature map with various cells where each patch of the image has an assigned color and annotation (as shown by the legend in the lower right).
Claim 7, 8, 9, 10, and 17 is rejected under 35 U.S.C. 103 as being unpatentable over Bhargava (US 20140270457 A1) in view of Batenchuk (US 20200123618 A1), Stumpe (US 20200394825 A1), and Kincaid (US 20100128988 A1).
Regarding claim 7:
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 1 (as shown above),
Bhargava in view of Batenchuk and Stumpe fails to explicitly teach:
wherein an opacity level of the virtual staining patterns may be adjusted to enable focusing on specific antibody clones.
Kincaid teaches:
wherein an opacity level of the virtual staining patterns may be adjusted (Kincaid: In at least one embodiment, the adjusting comprises varying the transparency/opacity of the virtual staining [0022]) to enable focusing on specific antibody clones (see Note 7A).
Note 7A: Allowing the user to adjust the opacity level of each virtual staining pattern would inherently enable the user to better focus on the elements of the virtual staining patterns.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Kincaid with Bhargava in view of Batenchuk and Stumpe. Having an opacity level of the virtual staining patterns be adjusted to enable focusing on specific antibody clones, as in Kincaid, would benefit the Bhargava in view of Batenchuk teachings by enabling the user to quickly identify patterns they are looking for and save time that would be otherwise spent searching for them.
Regarding claim 8:
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 1 (as shown above),
Bhargava in view of Batenchuk and Stumpe fails to teach:
wherein the virtual staining patterns render pattern differences that allow for quantitative analysis.
Kincaid teaches:
wherein the virtual staining patterns render pattern differences that allow for quantitative analysis (Kincaid: Different combinations of selecting and deselecting can be performed until the user positively identifies the individual attributes that are being represented by a combination color. [0076]; Kincaid: each of the attributes can be quantitated and these quantitated, extracted values (e.g., numbers) 52 can then be displayed in a table 50 that can be displayed on the display 10 of user interface 100 [0068], see Note 8A).
Note 8A: Kincaid teaches: “in FIG. 4, all three attributes 54 have been selected 56 for display on all cells 14 (all cells having been selected by checking attribute box feature 56 for that column) from which the attributes were measured and/or calculated. In image 12 display 10 shows cells 14 characterized by only one of the attributes as shown by a color of red, green or blue. However, when more than one attribute is present in a cell, the colors become mixed,” [0076]. That is, the biological features in the virtual stain may be color coded, enabling a viewer to visualize the differences between various patterns and features.
Kincaid further teaches in [0068] that attributes may be “quantitated” and displayed to a user interface. Kincaid further teaches that: “Attributes can include, but are not limited to, size (e.g., cell size or size of a particular type of sub-cellular component), shape (e.g., cell shape or shape of a sub-cellular component), number of nuclei, etc.” While Kincaid lists biological features of the cell, it would be obvious to one of ordinary skill in the art to expand the attributes to include, for example, the biological features taught by Batenchuk, because Batenchuk teaches: “For each portion of the one or more portions of the microscopic image, a biological feature is identified. The biological feature is of a set of possible biological features that is represented by the portion,” [0007]. Therefore, by “quantitating” the biological features and displaying them to a user, Kincaid enables quantitative analysis.
Quantitating and color coding the biological features, as in Kincaid, is analogous to rendering pattern differences that allow for quantitative analysis, as the user will be able to quickly see what cells have which attributes and how many have a certain attribute in a given area.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Kincaid with Bhargava in view of Batenchuk and Stumpe. Having the virtual staining patterns render pattern differences that allow for quantitative analysis, as in Kincaid, would benefit the Bhargava in view of Batenchuk and Stumpe teachings by enabling the user to quickly identify patterns they are looking for and save time that would be otherwise spent searching for them.
Regarding claim 9:
Bhargava in view of Batenchuk and Stumpe teaches:
The method of claim 1 (as shown above),
Bhargava in view of Batenchuk and Stumpe fails to teach:
further comprising individually manipulating a first virtual staining pattern of the overlayed virtual staining patterns.
Kincaid teaches:
further comprising individually manipulating (Kincaid: In at least one embodiment, the user interface includes a user selectable feature for each of the attributes displayed in a table, wherein selection of a feature for an attribute causes the user interface to virtually stain the image with virtual staining for the selected attribute, [0030], Kincaid: selecting, by a user, one of the cells in the image; and selecting, by a user, a feature for finding all other cells in the image having similar attributes, wherein upon the selecting a feature [0019]; see Note 9A) a first virtual staining pattern of the overlayed virtual staining patterns (Kincaid: staining representing classes of cells or sub-cellular components may replace or overlay any existing virtual stain being viewed, [0086]).
Note 9A: Kincaid teaches that individual features of a cell can be selected by the user. It is reasonable to conclude that the selecting of biological features taught by Kincaid is analogous to “individual manipulation” of the stains of the present application. Therefore, when combined with the teachings of Bhargava in view of Batenchuk, it would be obvious to allow the user to individually manipulate a virtual staining pattern of the overlaid virtual staining pattern.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Kincaid with Bhargava in view of Batenchuk and Stumpe. Individually manipulating a virtual staining pattern of the overlayed virtual staining patterns, as in Kincaid, would benefit the Bhargava in view of Batenchuk and Stumpe teachings by enabling the user to fine tune the annotations and patterns to best fit how they would analyze the patterns of the biological sample (Kincaid: The user is provided flexibility to combine attributes and thus display any combination of attributes desired on the image 12 [0076]).
Regarding claim 10:
Bhargava in view of Batenchuk, Stumpe and Kincaid teaches:
The method of claim 9 (as shown above), wherein the manipulating includes adjusting an intensity of each virtual staining pattern (Kincaid: For each attribute the value can be encoded by varying the intensity of the attribute's color in proportion to the value. This color gradient provides a visual representation of the attribute values so that variations in attribute values can be visually perceived across the population of cells/sub-cellular components [0074]) in real time (Kincaid: the image is an image produced by an instrument in real time [0015]).
Regarding claim 17:
Bhargava in view of Batenchuk teaches:
The method of claim 12 (as shown above),
Bhargava in view of Batenchuk fails to teach:
wherein parsing the annotations comprises applying a thresholding operator or a classifier to the annotations of the virtually stained image.
Kincaid teaches:
wherein parsing the annotations comprises applying a thresholding operator or a classifier to the annotations of the virtually stained image (Kincaid: Such annotation then appears on a per-cell basis in the table 50. Such metadata can then be used for classification purposes, [0071]; see Note 17A).
Note 17A: Kincaid teaches cells that have been virtually stained: “FIG. 3 shows the results of such virtual staining, where the cells having significant values for attribute 1 have been virtually stained in various intensities of the color red” [0075]. Kincaid further teaches that the cells may be annotated: “a pathologist may want to add manually produced annotations about cells that, for whatever reason, the pathologist finds to be particularly interesting.” [0071]. Kincaid further teaches that classification may be applied based on the annotations, as in [0071] cited above.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Kincaid with Bhargava in view of Batenchuk. Applying a classifier to the annotations of the virtually stained image, as in Kincaid, would benefit the Bhargava in view of Batenchuk teachings by automatically providing quantitative analysis of data based on annotated input: “It would be desirable to provide a solution in which a user can analyze quantitative information data in the context of the image or images from which the quantitative information data was derived.” (Kincaid, [0006]).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Bhargava (US 20140270457 A1) in view of Batenchuk (US 20200123618 A1) and Freytag (WO 2021198247 A1).
Regarding claim 14:
Bhargava in view of Batenchuk teaches:
The method of claim 12 (as shown above), further comprising
Bhargava in view of Batenchuk fails to explicitly teach:
training the machine learning algorithm with a plurality of virtual staining patterns of endogenous signals.
Freytag teaches:
training the machine learning algorithm with a plurality of virtual staining patterns of endogenous signals (Freytag: training output images comprising one or more virtual stains corresponding to the one or more chemical stains, Abstract; see Note 14A).
Note 14A: In Note 1A, it was discussed that due to the definitions of terms in Bhargava, one of ordinary skill in the art would understand the staining patterns taught by Bhargava to be “of endogenous signals”. When the teachings of Freytag are combined with Bhargava, it would be obvious to one of ordinary skill in the art to train the machine learning algorithm based on virtual staining patterns of endogenous signals.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Chukka with Bhargava in view of Batenchuk. Training the machine learning algorithm with a plurality of virtual staining patterns of endogenous signals, as in Chukka, would benefit the Bhargava in view of Batenchuk teachings by enabling machine-generated stains to be used instead of physical stains: “Preferably, several chemical stains are used to fully assess the pathology case. Typically, only one chemical stain can be applied to a tissue sample. Thus, if several chemical stains are required for diagnosis, several tissue samples each corresponding to a different section or slice of the wax block have to be prepared. Moreover, different chemical stains may require different staining protocols. Thus, the known chemical staining techniques are labour- and cost-intensive” (Freytag, Pg. 2, Background, par. 6).
Claims 15, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bhargava (US 20140270457 A1) in view of Batenchuk (US 20200123618 A1) and Seppo (US 20150065371 A1).
Regarding claim 15:
Bhargava teaches:
A method of generating an inference-based virtually stained image, comprising:
providing a neural network (Bhargava: feed-forward neural network [0049]) executed by image processing software running on one or more processors of a computing device (Bhargava: The computer-executable instructions can be part of, for example, a dedicated software application […] Such software can be executed, for example, on a single local computer [0049]), wherein the neural network is trained with a plurality of images of chemical stains of one or more first endogenous signals (Bhargava: a plurality of samples stained with a target stain (such as H&E, Bismark Brown, Nile Blue or antibody specific for a target protein, that is directly or indirectly labeled, e.g., with a fluorophore or enzyme), which are the same sample type of interest (e.g., breast tissue, lung tissue, water sample) can be used to train a network [0045]; see Note 1A);
obtaining and producing an image of a biological sample (Bhargava: obtaining a spectroscopic image of the sample, [0007]);
outputting two virtual images (Bhargava: In some examples, the method includes imaging a single sample multiple times, for example to obtain or generate a plurality of images using the method [0060]) corresponding to one or more second endogenous signals (see Note 1A),
wherein the two virtual images are configured to be combined one or more times to produce one or more new multiplexed virtual images (Bhargava: Multiplexing Computed Stain Images Using a combination of multiple staining results, it is possible to subsequently deduce the cell types and/or molecular transformations present [0143]).
Bhargava fails to teach:
detecting, using the neural network, one or more virtual staining patterns in the image of the biological sample;
parsing, using the neural network, one or more second endogenous signals of the one or more detected virtual staining patterns; and
outputting two virtual images corresponding to each of the one or more second endogenous signals
Batenchuk teaches:
detecting, using the neural network, one or more virtual staining patterns in the image of the biological sample (Batenchuk: Feature detector may use segmentation, convolutional neural network, object recognition, pattern analysis, machine-learning, a pathologist, and/or the like to detect the one or more biological features [0048]);
parsing, using the neural network (Batenchuk: The feature detector may use detected edges, colors […] to perform segmentation, pattern analysis, object recognition, and/or the like, as input into, convolutional neural network, a machine-learning model, and/or the like to identify the one or more biological feature [0047]), one or more second endogenous signals of the one or more detected virtual staining patterns (Batenchuk: image processing system to detect biological features to infer PD-L1 status [0022]; see Note 1B); and
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Batenchuk with Bhargava. Detecting, using the neural network, one or more virtual staining patterns in the image of the biological sample, as in Batenchuk, would benefit the Bhargava teachings by quickly and accurately measuring biological features in the biological sample: “Performing a TMB diagnostic can take up to three weeks which—on average, for patients with Stage-IV Non-Small Cell Lung Cancer (NSCLC)—represents 25-40% of patients' remaining life span when not treated. Further, the DNA sequencing frequently fails. There is insufficient material for the test 40% of the time, and the test fails quality control 40% of the time. Determining an optimal treatment plan in this instance is tricky, in that 55% of patients are TMB low or negative, and an assumption of either TMB test result would frequently be wrong and lead to sub-optimal treatment. Thus, the time from biopsy to a diagnostic test result may take up a significant portion of the patient's remaining life. Thus, it would be advantageous to identify a fast and reliable technique for estimating a patient's PD-L1, TMB, and IFNγ status. (Batenchuk, [0005-0006]).
Bhargava in view of Batenchuk fails to explicitly teach:
parsing, using the neural network, one or more second endogenous signals of the one or more detected virtual staining patterns; and
outputting two virtual images corresponding to each of the one or more second endogenous signals
Seppo teaches:
Parsing one or more second endogenous signals of the one or more detected virtual staining patterns (Seppo: detecting, by fluorescence, signals from the first binder and the fluorescent marker [0009])
outputting two virtual images corresponding to each of the one or more second endogenous signals (Seppo: detecting, by fluorescence, signals from the first binder and the fluorescent marker; (d) generating the first image of at least part of the sample from the detected fluorescent signals […] detecting, by fluorescence, signals from the probes for each of the target nucleic acid sequences and the fluorescent marker; and (d) generating the second image of at least part of the sample from the detected fluorescent signal” [0010] (emphasis added))
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Seppo with Bhargava in view of Batenchuk. Outputting two virtual images corresponding to each of the one or more second endogenous signals, as in Seppo, would benefit the Bhargava in view of Batenchuk teachings by improving the detection of multiple signals in a single sample: “There is still a need for improved methods for the detection of multiple targets in the same biological sample and the creation of a composite image to compare expression on a cell by cell basis, allowing the inclusion or exclusion of cells based on protein expression, apoptosis etc.” [0008].
Regarding claim 19:
Bhargava in view of Batenchuk, and Seppo teaches:
The method of claim 15 (as shown above),
wherein the two virtual images depict staining patterns of different cellular sub-populations partitioned between the two virtual images (Bhargava: a plurality of computed stain images for at least two different targets can be generated from a single sample, such as at least 3, at least 4, at least 5, at least 6, […] at least 100 different proteins, different pathogens, different cells, or combinations thereof. [0060]).
Regarding claim 20:
Bhargava in view of Batenchuk, and Seppo teaches:
The method of claim 15 (as shown above),
wherein the one or more new multiplexed virtual images are produced by
assigning a respective color in RGB color space to each of the two virtual images (Seppo: For example a FISH signal for a gene of interest would be colored red and another region of the same chromosome would be colored green making it easy to distinguish relative amounts of the two types of signals in a given cell or area of tissue. [0141]; see Note 20A) and
recombining the two virtual images into a single multiplexed image (Seppo: Preferably, the first and second images are overlaid and a composite image is further created. A composite image allows direct comparison of results obtained from the first image with that from the second image on a cell by cell basis. [0142]).
Note 20A: Seppo teaches in [0141] that signals may be assigned red or green color. One of ordinary skill in the art would recognize that Seppo teaches mapping signals to the RGB (red-green-blue) color space.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Bhargava (US 20140270457 A1) in view of Batenchuk (US 20200123618 A1) and Chukka (US 20160019695 A1).
Regarding claim 18:
Bhargava in view of Batenchuk teaches:
The method of claim 12 (as shown above), wherein overlaying the virtually stained image of the with the parsed annotations comprises
Bhargava in view of Batenchuk fails to teach:
using a spatial matching technique between the parsed annotations and the virtually stained image.
Chukka teaches:
using a spatial matching technique (Chukka: In further embodiments, matching tissue structure also involves mapping at least a portion of the originally selected images (for example, the first image and the second image) to a common grid based on the global transformation parameters [0007]) between the parsed annotations and the virtually stained image (Chukka: transferring annotations from one aligned image to another aligned image on the basis of matching tissue structure, Abstract).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Chukka with Bhargava in view of Batenchuk. Using a spatial matching technique between the parsed annotations and the virtually stained image, as in Chukka, would benefit the Bhargava in view of Batenchuk teachings by automatically adjusting annotations to fit updated data without requiring the user to manually recreate annotation data by hand.
Allowable Subject Matter
Claim 6 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 6 recites: “creating a unique virtual multiclonal antibody stain from multiple monoclonal clones, and titering to a desired expression level.”
Bhargava, Batenchuk, and Kincaid fail to teach monoclonal antibodies or titering to a desired expression level.
Gaiser teaches a “anti-CD133 rabbit monoclonal antibody” that is used to stain a tissue sample. However, Gaiser does not teach creating a unique, virtual multiclonal antibody stain from the monoclonal antibody. Furthermore, Gaiser does not teach titering to a desired expression level.
Seppo, Chukka, Stumpe, and Freytag also do not teach the aforementioned limitations.
Therefore, none of the other prior art searched or on the record teaches, suggests, or renders obvious the limitations of claim 6.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Brozek (WO 2022090205 A1) teaches: “performing a color deconvolution of the image according to a given color so as to obtain a deconvoluted image in which each pixel is associated with an intensity indicating the contribution of the given color for a same pixel location in the image, wherein the given color is selected in accordance with at least one dye used during said immunostaining” (Pg. 3, par. 6).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT ALEXANDER PROVIDENCE whose telephone number is (571)270-5765. The examiner can normally be reached Monday-Thursday 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached on (571)270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VINCENT ALEXANDER PROVIDENCE/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights