DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20, as originally filed, are currently pending and have been considered below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lahiani, Amal, et al. "Generalizing multistain immunohistochemistry tissue segmentation using one-shot color deconvolution deep neural networks." arXiv preprint arXiv:1805.06958 (2018), hereinafter, “Lahiani”, and further in view of Levy, Joshua J., et al. "Preliminary evaluation of the utility of deep generative histopathology image translation at a mid-sized NCI cancer center." BioRxiv (2020): 2020-01, hereinafter, “Levy”.
As per claim 1, Lahiani discloses a computer-implemented method for training a model, comprising:
performing color deconvolution on a set of training images, stained according to a first staining process, to generate a plurality of channels that correspond to dyes used in the first staining process and dyes used in a second staining process (Lahiani, page 1, 1 Introduction, Slides are typically stained in at least 3 different Immunohistochemistry (IHC) staining methods with different stain types and colors in addition to the traditional H&E; Lahiani, page 2, 2 Dataset, We then split this dataset into training (51 slides) and testing (26 slides) sets; Lahiani, page 4, 3.2 Network architecture: CD-UNET, As the training dataset is composed of different stainings, the number of principle color shades in the training images was 6: pink, blue, purple, brown, yellow and red. We chose then the first layer to have 6 (1 x 1) filters, each filter corresponding to a color ... In order to allow the color deconvolution segment to learn this non-linearity of the physical model, each of the (1 x 1) convolution layers is followed by a nonlinear function. The proposed color deconvolution network architecture (CD-UNET) is composed of 2 main parts (Figure 3). The first part is a color deconvolution segment composed of 2 layers of (1x1) convolution with ReLU and batch normalization. The second part is a UNET fully convolutional network);
training a machine learning model, using the selected channels of the set of training images, to function with images stained according to the first staining process and images stained according to the second staining process (Lahiani, page 1, 1 Introduction, Slides are typically stained in at least 3 different Immunohistochemistry (IHC) staining methods with different stain types and colors in addition to the traditional H&E; Lahiani, page 2, 2 Dataset, We then split this dataset into training (51 slides) and testing (26 slides) sets; Lahiani, As the training dataset is composed of different stainings, the number of principle color shades in the training images was 6: pink, blue, purple, brown, yellow and red. We chose then the first layer to have 6 (1 x 1) filters, each filter corresponding to a color ... In order to allow the color deconvolution segment to learn this non-linearity of the physical model, each of the (1 x 1) convolution layers is followed by a nonlinear function. The proposed color deconvolution network architecture (CD-UNET) is composed of 2 main parts (Figure 3). The first part is a color deconvolution segment composed of 2 layers of (1x1) convolution with ReLU and batch normalization. The second part is a UNET fully convolutional network; Lahiani, page 8, 5 Discussion and conclusions, multiple stains were simultaneously used in order to train a unified segmentation model that deals with multi stain histopathology images).
Lahiani does not explicitly disclose the following limitation as further recited however Levy discloses
selecting a channel from the plurality of channels that corresponds to a dye used in the second staining process (Levy, page 3, 2 Methods, 2.1 Data Preparation, For the SOX10 IHC, an unpaired dataset containing a total of 15,000 H&E and 15,000 IHC, 256 pixel x 256 pixel subimages of skin and lymph node histology were acquired to train a CycleGAN model ... 15,000 paired images for training a Pix2Pix model. The Pix2Pix IHC model data was split into 60% training, 20% validation and 20% testing. The CycleGAN model data was split into 60% training and 20% validation sets and shared the same test set with the Pix2Pix model; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks)
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Levy with Lahiani because they are in the same field of invention. One skilled in the art would have been motivated to select a channel as taught by Levy in the system of Lahiani dependent on the image, the number of colors in the dataset and diagnostic requirements (Levy, Abstract).
As per claim 2, Lahiani and Levy discloses the method of claim 1, wherein the first staining process is hematoxylin and eosin (H&E) staining and the second staining process is immunohistochemistry (IHC) staining (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stain; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images).
As per claim 3, Lahiani and Levy disclose the method of claim 2, wherein the plurality of channels include a hematoxylin (H) channel, an eosin (E) channel, and a 3,3′-diaminobenzidine (D) channel (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stains; Lahiani, page 7, Figure 7: Example of the output of the color deconvolution segment on an H&E image: (a) Original image (b) First output of the segment: corresponds to the hematoxylin channel (blue cells) (c) Second output of the segment: corresponds the eosin channel (pink connective tissue) (d) Third output: corresponds to the background in this case; Lahiani, page 7, 4.1.2 Color deconvolution segment output, In order to demonstrate the effect of the color deconvolution segment, we visualize its outputs using different stains (H&E and IHC). Figure 7 shows an example of an input image and its corresponding outputs from the color deconvolution segment of the network; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks).
As per claim 4, Lahiani and Levy disclose the method of claim 3, wherein selecting the channel from the plurality of channels includes selecting the H channel (Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks).
As per claim 5, Lahiani and Levy disclose the method of claim 1, wherein the machine learning model is a neural network model that includes a convolutional neural network that detects tumor cells (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stains ... We then split this dataset into training (51 slides) and testing (26 slides) sets. The various regions on the slides were annotated with one of the following categories: "Tissue" - i.e. normal tissue, "Tumor", "Necrosis"; Lahiani, page 4, 3.2 Network architecture: CD-UNET, The proposed color deconvolution network architecture (CD-UNET) is composed of 2 main parts (Figure 3). The first part is a color deconvolution segment composed of 2 layers of (1x1) convolution with ReLU and batch normalization. The second part is a UNET fully convolutional network resulting in a pixel wise segmentation of the input image).
As per claim 6, Lahiani and Levy disclose the method of claim 1, wherein the set of training images is stored as three-channel red-green-blue (RGB) images before color deconvolution (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stains ... All these IHC stainings use a blue (Hematoxylin) counterstain ... These 77 slides compose our dataset for this project. We then split this dataset into training (51 slides) and testing (26 slides) sets ... Each high resolution whole slide image was split into overlapping 512 x 512 x 3 RGB tiles).
As per claim 7., Lahiani disclose a computer-implemented method for processing an image, comprising:
performing color deconvolution on an input image, stained according to a second staining process, to generate a plurality of channels that correspond to dyes used in a first staining process and dyes using in the second staining process (Lahiani, page 1, 1 Introduction, Slides are typically stained in at least 3 different Immunohistochemistry (IHC) staining methods with different stain types and colors in addition to the traditional H&E; Lahiani, page 2, 2 Dataset, We then split this dataset into training (51 slides) and testing (26 slides) sets; Lahiani, page 4, 3.2 Network architecture: CD-UNET, As the training dataset is composed of different stainings, the number of principle color shades in the training images was 6: pink, blue, purple, brown, yellow and red. We chose then the first layer to have 6 (1 x 1) filters, each filter corresponding to a color ... In order to allow the color deconvolution segment to learn this non-linearity of the physical model, each of the (1 x 1) convolution layers is followed by a nonlinear function. The proposed color deconvolution network architecture (CD-UNET) is composed of 2 main parts (Figure 3). The first part is a color deconvolution segment composed of 2 layers of (1x1) convolution with ReLU and batch normalization. The second part is a UNET fully convolutional network).
Lahiani does not explicitly disclose the following limitations as further recited however Levy discloses
combining channels of the plurality of channels that correlate with a channel used to train a machine learning model to produce a single combined channel (Levy, page 3, 2 Methods, 2.1 Data Preparation, For the SOX10 IHC, an unpaired dataset containing a total of 15,000 H&E and 15,000 IHC, 256 pixel x 256 pixel subimages of skin and lymph node histology were acquired to train a CycleGAN model ... 15,000 paired images for training a Pix2Pix model. The Pix2Pix IHC model data was split into 60% training, 20% validation and 20% testing. The CycleGAN model data was split into 60% training and 20% validation sets and shared the same test set with the Pix2Pix model; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks);
processing the combined channel using the machine learning model to aid in medical decision making (Levy, page 5, 3 Preliminary Results, SOX10 is a nuclear transcription factor that is used in IHC for the identification of cells with melanocytic / neural crest origin (melanocytes, melanoma, etc). Immunohistochemical stains generate a distinctive brown color (DAB) on counterstained (hematoxylin) tissue sections allowing for relatively simple color deconvolution. Previous deep learning approaches have been able to utilize mappings between H&E and IHC tissue to learn antibody driven features in the H&E that may be correspondent to the separation of tumorous/non-tumorous tissue ... The ability of a deep learning model to predict the expression of DAB immunohistochemistry for a nuclear transcription factor (SOX10) from an HE stained image ... We also investigated each algorithm’s ability to identify melanocytic tissue); and
automatically adjusting a patient's treatment based on an output of the machine learning model (Levy, Abstract, Evaluation of a tissue biopsy is often required for the diagnosis and prognostic staging of a disease ... their incorporation into a clinical workflow; Levy, page 7, 4 Discussion, render a diagnosis).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Levy with Lahiani because they are in the same field of invention. One skilled in the art would have been motivated to select a channel as taught by Levy in the system of Lahiani dependent on the image, the number of colors in the dataset and diagnostic requirements (Levy, Abstract).
As per claim 8, Lahiani and Levy disclose the method of claim 7, wherein the first staining process is hematoxylin and eosin (H&E) staining and the second staining process is immunohistochemistry (IHC) staining (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stain; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images).
As per claim 9, Lahiani and Levy disclose the method of claim 8, wherein the plurality of channels include a hematoxylin (H) channel, an eosin (E) channel, and a 3,3′-diaminobenzidine (D) channel (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stains; Lahiani, page 7, Figure 7: Example of the output of the color deconvolution segment on an H&E image: (a) Original image (b) First output of the segment: corresponds to the hematoxylin channel (blue cells) (c) Second output of the segment: corresponds the eosin channel (pink connective tissue) (d) Third output: corresponds to the background in this case; Lahiani, page 7, 4.1.2 Color deconvolution segment output, In order to demonstrate the effect of the color deconvolution segment, we visualize its outputs using different stains (H&E and IHC). Figure 7 shows an example of an input image and its corresponding outputs from the color deconvolution segment of the network; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks).
As per claim 10, Lahiani and Levy disclose the method of claim 9, wherein combining channels of the plurality of channels includes combining the H and D channels (Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks).
As per claim 11, Lahiani and Levy disclose the method of claim 10, wherein combining channels includes setting pixel values of the combined channel according to maximum values of corresponding pixels in the H and D channels (Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks; Lahiani, page 7, 4.2 Feature visualization and pixel attribution, A noise image is inserted to the network, a specific pixel and category in the network output is set as the target, and several iterations of gradient ascent are run in order to modify the input image pixels to receive a high value in the target pixel. Using this we can create examples of input images, that cause a high activation at the target pixel for each of the categories (Figure 8)).
As per claim 12, Lahiani and Levy disclose the method of claim 10, wherein combining channels includes setting pixel values of the combined channel according to a linear combination of corresponding pixels in the H and D channels (Levy, page 3, 2 Methods, 2.1 Data Preparation, For the SOX10 IHC, an unpaired dataset containing a total of 15,000 H&E and 15,000 IHC, 256 pixel x 256 pixel subimages of skin and lymph node histology were acquired to train a CycleGAN model ... 15,000 paired images for training a Pix2Pix model. The Pix2Pix IHC model data was split into 60% training, 20% validation and 20% testing. The CycleGAN model data was split into 60% training and 20% validation sets and shared the same test set with the Pix2Pix model).
As per claim 13, Lahiani and Levy disclose the method of claim 7, wherein automatically adjusting the patient's treatment includes automatically administering an anti-cancer medication responsive to a determination that the input image indicates a tumor (Lahiani, Abstract, A key challenge in cancer immunotherapy biomarker research is quantification of pattern changes in microscopic whole slide images of tumor biopsies. Different cell types tend to migrate into various tissue compartments and form variable distribution patterns. Drug development requires correlative analysis of various biomarkers in and between the tissue compartments ... segment and annotate digitized slide images with multiple stainings into compartments of tumor, healthy tissue, and necrosis. We address the task in the context of drug development where multiple stains, tissue and tumor types exist; Lahiani, page 1, 1 Introduction, automatic tumor detection ... drug development, and specifically in the development of immunotherapy drugs ... common to have 5-6 cancer types in a clinical trial).
As per claim 14, Lahiani discloses a system for processing an image, comprising:
a hardware processor; and a memory that stores a computer program which (Lahiani, page 2, Dataset, The memory bound on the compute hardware limits the size of the input images), when executed by the hardware processor causes the hardware processor to:
perform color deconvolution on an input image, stained according to a second staining process, to generate a plurality of channels that correspond to dyes used in a first staining process and dyes using in the second staining process (Lahiani, page 1, 1 Introduction, Slides are typically stained in at least 3 different Immunohistochemistry (IHC) staining methods with different stain types and colors in addition to the traditional H&E; Lahiani, page 2, 2 Dataset, We then split this dataset into training (51 slides) and testing (26 slides) sets; Lahiani, page 4, 3.2 Network architecture: CD-UNET, As the training dataset is composed of different stainings, the number of principle color shades in the training images was 6: pink, blue, purple, brown, yellow and red. We chose then the first layer to have 6 (1 x 1) filters, each filter corresponding to a color ... In order to allow the color deconvolution segment to learn this non-linearity of the physical model, each of the (1 x 1) convolution layers is followed by a nonlinear function. The proposed color deconvolution network architecture (CD-UNET) is composed of 2 main parts (Figure 3). The first part is a color deconvolution segment composed of 2 layers of (1x1) convolution with ReLU and batch normalization. The second part is a UNET fully convolutional network).
Lahhiani does not explicitly disclose the following limitations as further recited however Levy discloses
combine channels of the plurality of channels that correlate with a channel used to train a machine learning model to produce a single combined channel (Levy, page 3, 2 Methods, 2.1 Data Preparation, For the SOX10 IHC, an unpaired dataset containing a total of 15,000 H&E and 15,000 IHC, 256 pixel x 256 pixel subimages of skin and lymph node histology were acquired to train a CycleGAN model ... 15,000 paired images for training a Pix2Pix model. The Pix2Pix IHC model data was split into 60% training, 20% validation and 20% testing. The CycleGAN model data was split into 60% training and 20% validation sets and shared the same test set with the Pix2Pix model; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks);
process the combined channel using the machine learning model to aid in medical decision making (Levy, page 5, 3 Preliminary Results, SOX10 is a nuclear transcription factor that is used in IHC for the identification of cells with melanocytic / neural crest origin (melanocytes, melanoma, etc). Immunohistochemical stains generate a distinctive brown color (DAB) on counterstained (hematoxylin) tissue sections allowing for relatively simple color deconvolution. Previous deep learning approaches have been able to utilize mappings between H&E and IHC tissue to learn antibody driven features in the H&E that may be correspondent to the separation of tumorous/non-tumorous tissue ... The ability of a deep learning model to predict the expression of DAB immunohistochemistry for a nuclear transcription factor (SOX10) from an HE stained image ... We also investigated each algorithm’s ability to identify melanocytic tissue); and
automatically adjust a patient's treatment based on an output of the machine learning model (Levy, Abstract, Evaluation of a tissue biopsy is often required for the diagnosis and prognostic staging of a disease ... their incorporation into a clinical workflow; Levy, page 7, 4 Discussion, render a diagnosis).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Levy with Lahiani because they are in the same field of invention. One skilled in the art would have been motivated to select a channel as taught by Levy in the system of Lahiani dependent on the image, the number of colors in the dataset and diagnostic requirements (Levy, Abstract).
As per claim 15, Lahiani and Levy disclose the system of claim 14, wherein the first staining process is hematoxylin and eosin (H&E) staining and the second staining process is immunohistochemistry (IHC) staining (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stain; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images).
As per claim 16, Lahiani and Levy disclose the system of claim 15, wherein the plurality of channels include a hematoxylin (H) channel, an eosin (E) channel, and a 3,3′-diaminobenzidine (D) channel (Lahiani, page 2, 2 Dataset, We selected 77 whole slide images of Colorectal Carcinoma metastases in liver tissue from biopsy slides stained with H&E (blue, pink) and 8 additional immunohistochemistry (IHC) assays stains; Lahiani, page 7, Figure 7: Example of the output of the color deconvolution segment on an H&E image: (a) Original image (b) First output of the segment: corresponds to the hematoxylin channel (blue cells) (c) Second output of the segment: corresponds the eosin channel (pink connective tissue) (d) Third output: corresponds to the background in this case; Lahiani, page 7, 4.1.2 Color deconvolution segment output, In order to demonstrate the effect of the color deconvolution segment, we visualize its outputs using different stains (H&E and IHC). Figure 7 shows an example of an input image and its corresponding outputs from the color deconvolution segment of the network; Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks).
As per claim 17, Lahiani and Levy disclose the system of claim 16, wherein the computer program further causes the hardware processor to combine the H and D channels (Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks).
As per claim 18, Lahiani and Levy disclose the system of claim 17, wherein the computer program further causes the hardware processor to set pixel values of the combined channel according to maximum values of corresponding pixels in the H and D channels (Levy, pages 3-4, 2.2 Analytic Approach, For the SOX10 dataset, both CycleGAN and Pix2Pix models were fit to the synthetic IHC dataset and each model was trained for approximately 150 epochs. These models were then utilized to convert H&E images into SOX10 IHC stained images. Color deconvolution algorithms were able to decompose both the real and generated IHC stained images into SOX10-positive (via 3,3 ′ -Diaminobenzidine (DAB) color deconvolution) and SOX10-negative (via hematoxylin color deconvolution) binary masks; Lahiani, page 7, 4.2 Feature visualization and pixel attribution, A noise image is inserted to the network, a specific pixel and category in the network output is set as the target, and several iterations of gradient ascent are run in order to modify the input image pixels to receive a high value in the target pixel. Using this we can create examples of input images, that cause a high activation at the target pixel for each of the categories (Figure 8)).
As per claim 19, Lahiani and Levy disclose the system of claim 17, wherein the computer program further causes the hardware processor to set pixel values of the combined channel according to a linear combination of corresponding pixels in the H and D channels (Levy, page 3, 2 Methods, 2.1 Data Preparation, For the SOX10 IHC, an unpaired dataset containing a total of 15,000 H&E and 15,000 IHC, 256 pixel x 256 pixel subimages of skin and lymph node histology were acquired to train a CycleGAN model ... 15,000 paired images for training a Pix2Pix model. The Pix2Pix IHC model data was split into 60% training, 20% validation and 20% testing. The CycleGAN model data was split into 60% training and 20% validation sets and shared the same test set with the Pix2Pix model).
As per claim 20, Lahiani and Levy disclose the system of claim 14, wherein the computer program further causes the hardware processor to automatically administer an anti-cancer medication responsive to a determination that the input image indicates a tumor (Lahiani, Abstract, A key challenge in cancer immunotherapy biomarker research is quantification of pattern changes in microscopic whole slide images of tumor biopsies. Different cell types tend to migrate into various tissue compartments and form variable distribution patterns. Drug development requires correlative analysis of various biomarkers in and between the tissue compartments ... segment and annotate digitized slide images with multiple stainings into compartments of tumor, healthy tissue, and necrosis. We address the task in the context of drug development where multiple stains, tissue and tumor types exist; Lahiani, page 1, 1 Introduction, automatic tumor detection ... drug development, and specifically in the development of immunotherapy drugs ... common to have 5-6 cancer types in a clinical trial).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRACY MANGIALASCHI whose telephone number is (571)270-5189. The examiner can normally be reached M-F, 9:30AM TO 6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRACY MANGIALASCHI/Primary Examiner, Art Unit 2668
Prosecution Insights