Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 2 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Xianyong et al. ‘PICaSSO Histologic Remission Index (PHRI) in ulcerative colitis: development of a novel simplified histological score for monitoring mucosal healing and predicting clinical outcomes and its applicability in an artificial intelligence system’ (hereinafter Xianyong).
Regarding claim 1, Xianyong discloses computer-implemented method, comprising: determining, within an image of a biological sample from an intestine of a patient, a plurality of image patches, wherein each image patch of the plurality of image patches depicts a portion of the biological sample (Xianyong in [Page – 890, Paragraph – 2 (right)] discloses about rectum and sigmoid (intestine of a patient), “In the same areas of rectum and sigmoid assessed and video recorded on endoscopy, at least two targeted mucosal biopsies were taken resulting in a total of 614 biopsies for histopathological analysis”. Xianyong in [Page – 890, Paragraph – 3 (right)] discloses, “The H&E- stained glass slides of colorectal biopsies were scanned at 40× (0.25 μm per pixel) using Aperio Digital Pathology Scanning system (Leica Biosystem, Illinois, USA)”. Xianyong in [Page – 891, Paragraph – 3 (right)] discloses plurality of patches, “a first model identified patches (areas of the WSI) containing neutrophils, while a second model, using a multiple instance learning approach, combined the features of each patch in the slide into a final dichotomous result (presence or absence of active disease) following the PHRI”); determining a plurality of bowel disease indication groups based at least on the plurality of image patches, wherein each bowel disease indication group of the plurality of bowel disease indication groups corresponding to a subset of the plurality of image patches (Xianyong in [Page – 891, Paragraph – 3 (right)] discloses about bowel disease, “a first model identified patches (areas of the WSI) containing neutrophils, while a second model, using a multiple instance learning approach, combined the features of each patch in the slide into a final dichotomous result (presence or absence of active disease) following the PHRI”. “The PHRI scores of rectum and sigmoid were considered individually”. Bowel diseases associated with the Patches of sigmoid biopsy and Bowel diseases associated with the Patches of rectal biopsy equates to disease indication of group 1 and 2. Additionally, patches belonging to corresponding biopsy implies to subset of the plurality of image patches);
generating a group-level histological score for each bowel disease indication group of the plurality of bowel disease indication groups based at least on the subset of the plurality of image patches contained in each respective group (Xianyong in [Page – 891, Paragraph – 3 (left)] discloses about PHRI score (histological score) for rectum and sigmoid (group level of sigmoid and rectum), “The PHRI scores of rectum and sigmoid were considered individually”. Furthermore, Xianyong in [Page – 889, Paragraph – 2(left)] discloses, “we trained and tested a novel deep learning strategy based on a CNN architecture to detect neutrophils, calculate PHRI and identify active from quiescent UC using a subset of 138 biopsies”); and generating an aggregated histological score for the biological sample based on the generated group-level histological score for each bowel disease indication group (Xianyong in [Page – 891, Paragraph – 3 (left)] discloses, “The PHRI scores of rectum and sigmoid were considered individually as well as combined in the total score (PHRI_ total, ie, the sum of PHRI scores of both rectum and sigmoid) or the maximum score (PHRI_max, ie, the higher score between rectum and sigmoid)”), wherein the aggregated histological score is indicative of a disease burden in the intestine of the patient (Xianyong in [Page – 895, Paragraph – 1 (right)] discloses, “PHRI is the easiest to apply in daily practice as a universal histological indicator and quantitative measurement (grading tool) of disease activity in UC”).
Summary of Citations (Xianyong)
[Page – 889, Paragraph – 2(left)]; “we trained and tested a novel deep learning strategy based on a CNN architecture to detect neutrophils, calculate PHRI and identify active from quiescent UC using a subset of 138 biopsies”.
[Page – 890, Paragraph – 2 (right)]; “In the same areas of rectum and sigmoid assessed and video recorded on endoscopy, at least two targeted mucosal biopsies were taken resulting in a total of 614 biopsies for histopathological analysis”.
[Page – 890, Paragraph – 3 (right)]; “The H&E- stained glass slides of colorectal biopsies were scanned at 40× (0.25 μm per pixel) using Aperio Digital Pathology Scanning system (Leica Biosystem, Illinois, USA)”.
[Page – 891, Paragraph – 3 (right)]; “a first model identified patches (areas of the WSI) containing neutrophils, while a second model, using a multiple instance learning approach, combined the features of each patch in the slide into a final dichotomous result (presence or absence of active disease) following the PHRI”.
[Page – 891, Paragraph – 3 (left)]; “The PHRI scores of rectum and sigmoid were considered individually as well as combined in the total score (PHRI_ total, ie, the sum of PHRI scores of both rectum and sigmoid) or the maximum score (PHRI_max, ie, the higher score between rectum and sigmoid)”.
[Page – 895, Paragraph – 1 (right)]; “PHRI is the easiest to apply in daily practice as a universal histological indicator and quantitative measurement (grading tool) of disease activity in UC”.
Regarding claim 2, Xianyong discloses the method of claim 1, wherein the group-level histological score and the aggregated histological score are each one of a Nancy Histological Index (NHI) score, a Robarts Histopathology Index (RHI) score, a Geboes Scale score, and a Global Histology Activity Score (GHAS) (Xianyong in [Page – 890, Paragraph – 3(right)] discloses about group level histological score, “For each biopsy from each segment, the worst features were scored applying five different histological scoring schemes—Geboes Score (GS),30 Robarts Histological Index (RHI),31 Nancy Histological Index (NHI),32 extent, chronicity, activity and plus (ECAP) score33 34 and Villa nacci Simplified Score (VSS) 35 The average values of each score and subscore for both rectum and sigmoid were also separately analysed”. Additionally, Xianyong in [Page – 895, Paragraph – 2(right)] discloses about aggregated histological score, “PHRI makes it easier to perform histological scoring on multiple biopsies from different segments of colon in patients with extensive colitis, to achieve an entire assessment and generate a global (total, maximum, or average) score per colon”).
Summary of Citations (Xianyong)
[Page – 890, Paragraph – 3(right)]; “For each biopsy from each segment, the worst features were scored applying five different histological scoring schemes—Geboes Score (GS),30 Robarts Histological Index (RHI),31 Nancy Histological Index (NHI),32 extent, chronicity, activity and plus (ECAP) score33 34 and Villa nacci Simplified Score (VSS).35 The average values of each score and subscore for both rectum and sigmoid were also separately analysed”.
[Page – 895, Paragraph – 2(right)]; “PHRI makes it easier to perform histological scoring on multiple biopsies from different segments of colon in patients with extensive colitis, to achieve an entire assessment and generate a global (total, maximum, or average) score per colon”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 3 is rejected under 35 U.S.C 103 as being unpatentable over Xianyong in view of Maclean Patent Application Publication No. CN-114599665-A (hereinafter Maclean).
Regarding claim 3, Xianyong discloses the method of claim 1, the group-level histological score and the aggregated histological score is at least one of a first score indicating no disease burden (Xianyong in [Page – 891, Paragraph – 3 (left)] discloses about group-level histological score and the aggregated histological score, “The PHRI scores of rectum and sigmoid were considered individually as well as combined in the total score (PHRI_ total, ie, the sum of PHRI scores of both rectum and sigmoid) or the maximum score (PHRI_max, ie, the higher score between rectum and sigmoid)”. Xianyong in [Page – 893, Paragraph – 2(right)] discloses about first score, “in patients in ER (MES 0), of which only 10.9% had PHRI >0 (presence of neutrophilic infiltration) and 89.1% had PHRI of 0 (no neutrophilic infiltration)”).
Xianyong doesn’t disclose about the following limitation as further recited in the claim.
Maclean discloses a second score indicating low disease burden in the intestine of the patient, a third score indicating a moderate disease burden in the intestine of the patient, and a fourth score indicating a high disease burden in the intestine of the patient (Maclean in [0566] discloses, “inflammatory parameters (mucosal/submucosal inflammation, erosion, gland loss, hyperplasia, edema, transmural inflammation, serosal inflammation) were scored on a scale of 1-4 (1 ═ minimal, 2 ═ mild, 3 ═ moderate, 4 ═ significant) to assess histopathology of the proximal, middle and distal colon”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Maclean into the system of Xianyong because converting the information into a consistent measure of how severe the intestinal disease is would make the result more useful for disease assessment and treatment.
Summary of Citations (Maclean)
Paragraph [0566]; “inflammatory parameters (mucosal/submucosal inflammation, erosion, gland loss, hyperplasia, edema, transmural inflammation, serosal inflammation) were scored on a scale of 1-4 (1 ═ minimal, 2 ═ mild, 3 ═ moderate, 4 ═ significant) to assess histopathology of the proximal, middle and distal colon”.
Summary of Citations (Xianyong)
[Page – 891, Paragraph – 3 (left)]; “The PHRI scores of rectum and sigmoid were considered individually as well as combined in the total score (PHRI_ total, ie, the sum of PHRI scores of both rectum and sigmoid) or the maximum score (PHRI_max, ie, the higher score between rectum and sigmoid)”.
[Page – 893, Paragraph – 2(right)]; “in patients in ER (MES 0), of which only 10.9% had PHRI >0 (presence of neutrophilic infiltration) and 89.1% had PHRI of 0 (no neutrophilic infiltration)”.
Claims 4 ,5, 9, 11, 14 and 19 are rejected under 35 U.S.C 103 as being unpatentable over Xianyong in view of Chen Patent Application Publication No. US-20170076448-A1 (hereinafter Chen).
Regarding claim 4, Xianyong discloses the method of claim 1.
Xianyong doesn’t disclose about the following limitation as further recited in the claim.
Chen discloses the subset of the plurality of image patches is formed by at least clustering one or more similar image patches of the plurality of image patches based at least on one or more pixel-wise features (Chen in [0066] discloses about image patches, “For data augmentation, ten images are sub-cropped (corners and centers with or without horizontal flips) of size 57×57 from the original image patch of size 64×64”. Additionally, in Chen in [0068] discloses about clustering, “At 156 , a clustering process, utilizing a Voronoi diagram of clusters model, is applied to locate clusters of high confidence examples of neutrophils and non-neutrophil cells. In one example, density-based clustering is applied to the trustable examples of neutrophils, to capture the potential clustering behavior of neutrophils”. Lastly, Chen in [0028] discloses about pixel wise features, “Di (p) evaluates the probability that a pixel belongs to a given boundary according to the photometric properties of the pixel, such as its hue, saturation, and value, or brightness, referred to as the HSV values of the pixel”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Chen into the system of Xianyong because it would reduce the complexity of analyzing large number of image patches.
Summary of Citations (Chen)
Paragraph [0028]; “In one implementation, three photometric priors are used and included as energy terms in the energy minimization. A first energy term, Di (p), penalizes a pixel p, if it is not likely to be on a given boundary, Bi . Essentially, Di (p) evaluates the probability that a pixel belongs to a given boundary according to the photometric properties of the pixel, such as its hue, saturation, and value, or brightness, referred to as the HSV values of the pixel”.
Paragraph [0066]; “For data augmentation, ten images are sub-cropped (corners and centers with or without horizontal flips) of size 57×57 from the original image patch of size 64×64”.
Paragraph [0068]; “At 156 , a clustering process, utilizing a Voronoi diagram of clusters model, is applied to locate clusters of high confidence examples of neutrophils and non-neutrophil cells. In one example, density-based clustering is applied to the trustable examples of neutrophils, to capture the potential clustering behavior of neutrophils”.
Regarding claim 5, Xianyong discloses the method of claim 1, wherein a presence of a first (Xianyong in [Page – 889, Paragraph – 3(right)] discloses, “PHRI is a new score based simply on the presence or absence of neutrophils (yes/no) and it provides excellent diagnostic accuracy, the strongest correlation to endoscopic activity among several histological scores”. A patch containing neutrophils corresponds to the feature being present. And presence of neutrophil finding receives a PHRI score of 1 absence of the neutrophil feature receives a score of 0 (disclosed in table 1).
Xianyong doesn’t disclose about the following limitation as further recited in the claim.
Chen discloses pixel-wise feature of the one or more pixel-wise features (Chen in [0044] discloses about pixel wise feature, “each of the plurality of superpixels is classified as one of epithelium, lumen, and extracellular material. This classification can be performed with any appropriate classifier with classification features drawn from properties of the pixels comprising the superpixel, such as texture, color content, and similar features”).
Summary of Citations (Xianyong)
[Page – 889, Paragraph – 3(right)]; “PHRI is a new score based simply on the presence or absence of neutrophils (yes/no) and it provides excellent diagnostic accuracy, the strongest correlation to endoscopic activity among several histological scores”.
Summary of Citations (Chen)
Paragraph [0044]; “At 102 , a superpixel segmentation of a tissue image, comprising a plurality of superpixels, is generated. At 104 , each of the plurality of superpixels is classified as one of epithelium, lumen, and extracellular material. This classification can be performed with any appropriate classifier with classification features drawn from properties of the pixels comprising the superpixel, such as texture, color content, and similar features”.
Regarding claim 9, Chen in the combination discloses the method of claim 4, wherein the one or more pixel-wise features includes at least one of a shape, a color, a size, a presence of a dye (Chen in [0044] discloses, “classification can be performed with any appropriate classifier with classification features drawn from properties of the pixels comprising the superpixel, such as texture, color content, and similar features”) and an intensity associated with a pixel of the image of the biological sample (Chen in [0028] discloses about pixel brightness (intensity)).
Summary of Citations (Chen)
Paragraph [0028]; “Essentially, Di (p) evaluates the probability that a pixel belongs to a given boundary according to the photometric properties of the pixel, such as its hue, saturation, and value, or brightness”.
Paragraph [0044]; “At 102 , a superpixel segmentation of a tissue image, comprising a plurality of superpixels, is generated. At 104 , each of the plurality of superpixels is classified as one of epithelium, lumen, and extracellular material. This classification can be performed with any appropriate classifier with classification features drawn from properties of the pixels comprising the superpixel, such as texture, color content, and similar features”.
Regarding claim 11, Xianyong discloses the method of claim 4, wherein the group-level histological score is generated based at least on one or more of a (Xianyong in [Page – 891, Paragraph – 3 (right)] discloses about bowel disease, “a first model identified patches (areas of the WSI) containing neutrophils, while a second model, using a multiple instance learning approach, combined the features of each patch in the slide into a final dichotomous result (presence or absence of active disease) following the PHRI”. Xianyong in [Page – 891, Paragraph – 3 (left)] further discloses, “The PHRI scores of rectum and sigmoid were considered individually”. Bowel diseases associated with the Patches of sigmoid biopsy and Bowel diseases associated with the Patches of rectal biopsy equates to disease indication of group 1 and 2. Additionally, patches belonging to corresponding biopsy implies to subset of the plurality of image patches).
Xianyong doesn’t disclose about the following limitation as further recited in the claim and strike through above.
Chen discloses quantity of the one or more pixel-wise features (Chen in [0057] discloses, “the locations and number of neutrophils with respect to different tissue layers”), a distribution of the one or more pixel-wise features within the subset of the plurality of image patches (Chen in [0068] discloses about clustering and evaluates the spatial arrangement (distribution) of neutrophils, “density-based clustering is applied to the trustable examples of neutrophils, to capture the potential clustering behavior of neutrophils”).
Summary of Citations (Chen)
Paragraph [0057]; “Identifying neutrophils, a major type of immune cell, is critical to the diagnosis of inflammatory diseases, as the locations and number of neutrophils with respect to different tissue layers can be used to determine whether there is clinically significant acute inflammation”.
Paragraph [0068]; “density-based clustering is applied to the trustable examples of neutrophils, to capture the potential clustering behavior of neutrophils”.
Summary of Citations (Xianyong)
[Page – 891, Paragraph – 3 (right)]; “a first model identified patches (areas of the WSI) containing neutrophils, while a second model, using a multiple instance learning approach, combined the features of each patch in the slide into a final dichotomous result (presence or absence of active disease) following the PHRI”.
[Page – 891, Paragraph – 3 (left)]; “The PHRI scores of rectum and sigmoid were considered individually as well as combined in the total score (PHRI_ total, ie, the sum of PHRI scores of both rectum and sigmoid) or the maximum score (PHRI_max, ie, the higher score between rectum and sigmoid)”.
Regarding claim 14, Xianyong discloses the method of claim 1.
Chen further discloses generating a first visual representation of a reduced dimension representation of the plurality of image patches (Chen in [0051] discloses about plurality of image patches, “two small image patches containing the same candidate object”. Furthermore, Chen in [0066] discloses about reduced dimension representation, “ten images are sub-cropped (corners and centers with or without horizontal flips) of size 57×57 from the original image patch of size 64×64”).
Summary of Citations (Chen)
Paragraph [0051]; “two small image patches containing the same candidate object, with or without the background masked out, are provided to two convolutional neural networks (CNN) with the same architecture”.
Paragraph [0066]; “ten images are sub-cropped (corners and centers with or without horizontal flips) of size 57×57 from the original image patch of size 64×64”.
Regarding claim 19, Xianyong discloses the method of claim 1, wherein the group-level histological score and the aggregated histological score are each generated by applying at least one machine learning model trained to generate the group-level histological score (Xianyong in [Page – 895, Paragraph – 2(right)] discloses about aggregated histological score, “PHRI makes it easier to perform histological scoring on multiple biopsies from different segments of colon in patients with extensive colitis, to achieve an entire assessment and generate a global (total, maximum, or average) score per colon”. Additionally, Xianyong in [Page – 889, Paragraph – 2(left)] discloses about ML model).
Chen further discloses the aggregated histological score by at least determining a representational encoding of the subset of the plurality of image patches (Chen in [0065] discloses, “deep convolutional neural networks (CNNs) are applied to learn a hierarchical representation of the low, mid, and high level features of the cells”).
Summary of Citations (Chen)
Paragraph [0065]; “deep convolutional neural networks (CNNs) are applied to learn a hierarchical representation of the low, mid, and high level features of the cells”.
Summary of Citations (Xianyong)
[Page – 889, Paragraph – 2(left)]; “we trained and tested a novel deep learning strategy based on a CNN architecture to detect neutrophils, calculate PHRI and identify active from quiescent UC using a subset of 138 biopsies”.
[Page – 895, Paragraph – 2(right)]; “PHRI makes it easier to perform histological scoring on multiple biopsies from different segments of colon in patients with extensive colitis, to achieve an entire assessment and generate a global (total, maximum, or average) score per colon”.
Claim 8 is rejected under 35 U.S.C 103 as being unpatentable over Xianyong in view of Chen and further in view of Niels ‘Utilizing Deep Learning to Analyze Whole Slide Images of Colonic Biopsies for Associations Between Eosinophil Density and Clinicopathologic Features in Active Ulcerative Colitis, Inflammatory Bowel Diseases’ (hereinafter Niels).
Regarding claim 8, Xianyong in the combination discloses method of claim 4.
Xianyong and Chen in the combination doesn’t disclose about the following limitation as further recited in the claim.
Niels discloses the one or more pixel-wise features is representative of a presence in the biological sample of at least one of an erosion of tissue, a neutrophil, a lymphoid structure, a crypt abscess, and debris within an epithelium of the tissue (Niels in [Page – 541, Paragraph – 2(right)] discloses, “The sigmoid colon biopsies were scored according to the Geboes score, which has subscores for crypt architecture, lamina propria chronic inflammation, lamina propria eosinophils, lamina propria neutrophils, neutrophils in epithelium, crypt destruction, and surface epithelial injury”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Niels into the system of Xianyong in view of Chen because assessing diseases based on indicators of inflammation and tissue damage would make the assessment more accurate.
Summary of Citations (Niels)
[Page – 541, Paragraph – 2(right)]; “The sigmoid colon biopsies were scored according to the Geboes score, which has subscores for crypt architecture, lamina propria chronic inflammation, lamina propria eosinophils, lamina propria neutrophils, neutrophils in epithelium, crypt destruction, and surface epithelial injury”.
Claims 23 , 24, 25, 26, 31, 32, 33, 40, 41 and 42 are rejected under 35 U.S.C 103 as being unpatentable over Chen in view of Kondo ‘Highly Multiplexed Image Analysis of Intestinal Tissue Sections in Patients With Inflammatory Bowel Disease’ (hereinafter Kondo).
Regarding claim 23, Chen discloses computer-implemented method (Chen in [0017] discloses about computer implemented method), comprising: receiving an image of a biological sample from an intestine of a patient (Chen in [0016] discloses image of intestine, “the imager 12 can be a digital camera and/or a microscope configured to capture histology tissue images. In one example, the tissue can be taken from the gastrointestinal tract of a human being”), each portion of the plurality of portions corresponding to one cell of the plurality of cells (Chen in [0058] discloses, “a segmentation of the tissue image is generated such that the lobes of each cell, particularly the neutrophil, are grouped into one segment, which contains no lobes from other cells” wherein one segment equates to one cell).
Chen doesn’t disclose about the following limitation as further recited in the claim
Kondo discloses the image depicting a plurality of cells of the biological sample; segmenting the received image into a plurality of portions (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3)”), identifying, based at least on the segmented image, a first spatial coordinate associated with each cell of the plurality of cells within the image (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “Cell segmentation yielded 521,783 captured cells, each with a unique protein expres sion profile and x- and y-coordinates indicating cell location within an image”); identifying a first cell type associated with the first spatial coordinate ((Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method”); and generating, based at least on the first spatial coordinate and the first cell type, a visual representation including the image of the biological sample (Kondo in [Page – 1941, Paragraph – 4(right)] discloses, “Cell type annotation was validated by pseudoflow plots for cell-type–specific expression patterns and by overlaying the x- and y-coordinates of cells onto IMC pseudocolor images (Supplementary Figures 4 and 5)”).
It would have been obvious to one of ordinary skill in art before the effective filling date of the claimed invention to integrate the technique of Chen into the system of Kondo because using cell location and cell types would help the system to generate better visual representation of pathology results.
Summary of Citations (Kondo)
[Page – 1941, Paragraph – 4(right)]; “Cell type annotation was validated by pseudoflow plots for cell-type–specific expression patterns and by overlaying the x- and y-coordinates of cells onto IMC pseudocolor images (Supplementary Figures 4 and 5)”.
[Page – 1943, Paragraph – 2(left)]; “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3) ... We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method (Supplementary Table 4 and Figure 1A–C)”.
Summary of Citations (Chen)
Paragraph [0016]; “the imager 12 can be a digital camera and/or a microscope configured to capture histology tissue images. In one example, the tissue can be taken from the gastrointestinal tract of a human being”.
Paragraph [0058]; “a segmentation of the tissue image is generated such that the lobes of each cell, particularly the neutrophil, are grouped into one segment, which contains no lobes from other cells”.
Paragraph [0017]; “The captured image is provided to an analysis component 20 via an imager interface 22 ... on an application specific integrated circuit, software or firmware stored on a non-transitory computer readable medium and executed by an associated processor, or a mix of both”.
Regarding claim 24, Kondo in the combination discloses the method of claim 23, wherein the identifying is further based at least on a plurality of annotations identifying a plurality of cell types depicted in a plurality of images of biological samples (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3) ... We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method (Supplementary Table 4 and Figure 1A–C)”).
Summary of Citations (Kondo)
[Page – 1943, Paragraph – 2(left)]; “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3) ... We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method (Supplementary Table 4 and Figure 1A–C)”.
Regarding claim 25, Kondo in the combination discloses the method of claim 23, wherein the first cell type is at least one of a neutrophil, a plasma cell, a lymphocyte, an intraepithelial lymphocyte, an eosinophil, a Mast cell, a macrophage, a goblet cell, an enterocyte, an endothelial cell, a fibroblast, a smooth muscle cell, and an endothelial cell (Kondo in [Page – 1943, Paragraph – 2(left)] discloses about identifying plasma cell).
Summary of Citations (Kondo)
[Page – 1943, Paragraph – 2(left)]; “we identified T helper cells, cytotoxic T cells, Tregs, B cells, plasma cells, resident tissue macrophages, and monocytes”.
Regarding claim 26, Kondo in the combination discloses the method of claim 23, wherein the image further depicts a plurality of tissue regions of the biological sample (Kondo in [Page – 1941, Paragraph – 2(right)] discloses, “A decision forest learning method was used to detect regions of tissue versus background in each image using Vis software (Visiopharm)”), and wherein the method further comprises identifying, based at least on the segmented image, a second spatial coordinate associated with each tissue region of the plurality of tissue regions and a first tissue region type associated with the second spatial coordinate (Kondo in [Page – 1949, Paragraph – 3(right)] discloses, “one could use IMC to perform tissue-compartmentalized assessments of immune cell types (ie, epithelium vs stroma, tumor vs nontumor) for the identification of spatially resolved biomarkers in human disease”).
Summary of Citations (Kondo)
[Page – 1941, Paragraph – 2(right)]; “A decision forest learning method was used to detect re gions of tissue versus background in each image using Vis software (Visiopharm)”.
[Page – 1949, Paragraph – 3(right)]; “one could use IMC to perform tissue-compartmentalized assessments of immune cell types (ie, epithelium vs stroma, tumor vs nontumor) for the identification of spatially resolved biomarkers in human disease”.
Regarding claim 31, Kondo in the combination discloses the method of claim 23, wherein the identifying includes (Kondo in [Page – 1941, Paragraph – 4(right)] discloses, “boolean rules were used to assign each cell to a cell type (Supplementary Table 4)”).
Chen doesn’t disclose about the following limitation as further recited in the claim.
Chen discloses generating a metric (Chen in [0016] discloses, “samples classified with a probability greater than 0.9 are retained as high confidence examples”).
Summary of Citations (Kondo)
[Page – 1941, Paragraph – 4(right)]; “boolean rules were used to assign each cell to a cell type (Supplementary Table 4)”.
Summary of Citations (Chen)
Paragraph [0016]; “samples classified with a probability greater than 0.9 are retained as high confidence examples”.
Regarding claim 32, Kondo in the combination discloses the method of claim 23, further comprising: generating spatial tabular data including the first spatial coordinate associated with each cell of the plurality of cells within the image and the first cell type associated with the first spatial coordinate (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3) ... We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method (Supplementary Table 4 and Figure 1A–C)”).
Summary of Citations (Kondo)
[Page – 1943, Paragraph – 2(left)]; “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3) ... We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method (Supplementary Table 4 and Figure 1A–C)”.
Regarding claim 33, Kondo in the combination discloses the method of claim 23, further comprising: generating a histological score for the biological sample (Kondo in [Page – 1943, Paragraph – 1(right)] discloses, “inflammation grades to the 48 images by assessing the histopathologic features of adjacent H&E-stained sections (Table 1) ... The following inflammation grades were assigned to each image: (1) control (all CI and CC images), (2) uninflamed, (3) mildly chronic, (4) chronic, (5) active, (6) chronic and active, and (7) severely chronic and active”) based at least on the first spatial coordinate and the first cell type associated with the first spatial coordinate (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3)), wherein the histological score is indicative of a disease burden in the intestine of the patient (Kondo in [Page – 1943, Paragraph – 1(right)] discloses, “The following inflammation grades were assigned to each image: (1) control (all CI and CC images), (2) uninflamed, (3) mildly chronic, (4) chronic, (5) active, (6) chronic and active, and (7) severely chronic and active”).
Summary of Citations (Kondo)
[Page – 1943, Paragraph – 1(right)]; “inflammation grades to the 48 images by assessing the histopathologic features of adjacent H&E-stained sections (Table 1) ... The following inflammation grades were assigned to each image: (1) control (all CI and CC images), (2) uninflamed, (3) mildly chronic, (4) chronic, (5) active, (6) chronic and active, and (7) severely chronic and active”.
Regarding claim 40, Kondo in the combination discloses the method of claim 23, wherein the first spatial coordinate is two-dimensional (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3)”).
Summary of Citations (Kondo)
[Page – 1943, Paragraph – 2(left)]; “Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3)”.
Regarding claim 41, Kondo in the combination discloses he method of claim 23, further comprising: generating, based at least on the first spatial coordinate and the first cell type (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3) ... We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method (Supplementary Table 4 and Figure 1A–C)”), an overlay indicating the first cell type at the first spatial coordinate, wherein the overlay includes at least one of a mask, a color, and a pattern (Kondo in [Page – 1941, Paragraph – 4(right)] discloses, “Cell type annotation was validated by pseudoflow plots for cell-type–specific expression patterns and by overlaying the x- and y-coordinates of cells onto IMC pseudocolor images”).
Summary of Citations (Kondo)
[Page – 1943, Paragraph – 2(left)]; “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm) (Supplementary Figure 2). Cell segmentation yielded 521,783 captured cells, each with a unique protein expression profile and x- and y-coordinates indicating cell location within an image (Supplementary Table 3) ... We defined marker expression parameters to assign 13 cell types of epithelial, stromal, or immune origin using a boolean method (Supplementary Table 4 and Figure 1A–C)”.
[Page – 1941, Paragraph – 4(right)]; “Cell type annotation was validated by pseudoflow plots for cell-type–specific expression patterns and by overlaying the x- and y-coordinates of cells onto IMC pseudocolor images”
Regarding claim 42, Kondo in the combination discloses the method of claim 23, wherein the image is segmented by applying a machine learning model trained (Kondo in [Page – 1941, Paragraph – 2(right)] discloses, “A decision forest learning method was used to detect regions of tissue versus background in each image using Vis software (Visiopharm)”) to perform per-cell segmentation and per-tissue region segmentation (Kondo in [Page – 1943, Paragraph – 2(left)] discloses, “We acquired 57 multiplex images, which were processed for cell segmentation using Vis software (Visiopharm). Furthermore, Kondo in [Page – 1941, Paragraph – 4(right)] discloses about tissue region segmentation, “For immune cell composition analyses, lymphoid tissues and submucosal tissues were masked and removed using EBImage to prevent skewing”).
Kondo doesn’t disclose about the following limitation as further recited in the claim.
Chen discloses by at least assigning to each pixel in the image, a cell segmentation label indicating whether the pixel is associated with a cell type of a cell depicted in the image (Chen in [0039] discloses, “allowing for efficient segmentation of the nucleus and cytoplasm of each cell. At 80 , the cell is classified according to the determined set of boundaries. In one example, cells are classified into plasma cells and lymphocytes based on features extracted from the segmented subcellular structures”) and a tissue region label indicating whether the pixel is associated with a tissue region type of a tissue region depicted in the image (Chen in [0044] discloses, “a superpixel segmentation of a tissue image, comprising a plurality of superpixels, is generated. At 104 , each of the plurality of superpixels is classified as one of epithelium, lumen, and extracellular material. This classification can be performed with any appropriate classifier with classification features drawn from properties of the pixels comprising the superpixel, such as texture, color content, and similar features”).
Summary of Citations (Chen)
Paragraph [0039]; “allowing for efficient segmentation of the nucleus and cytoplasm of each cell. At 80 , the cell is classified according to the determined set of boundaries. In one example, cells are classified into plasma cells and lymphocytes based on features extracted from the segmented subcellular structures”.
Paragraph [0044]; “a superpixel segmentation of a tissue image, comprising a plurality of superpixels, is generated. At 104 , each of the plurality of superpixels is classified as one of epithelium, lumen, and extracellular material. This classification can be performed with any appropriate classifier with classification features drawn from properties of the pixels comprising the superpixel, such as texture, color content, and similar features”.
Summary of Citations (Kondo)
[Page – 1941, Paragraph – 2(right)]; “A decision forest learning method was used to detect regions of tissue versus background in each image using Vis software (Visiopharm)”.
[Page – 1941, Paragraph – 4(right)]; “For immune cell composition analyses, lymphoid tissues and submucosal tissues were masked and removed using EBImage to prevent skewing”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to ZAID MUHAMMAD SALEH whose telephone number is (703)756-1684.
The examiner can normally be reached M-F 8 am - 5 pm ET. Examiner interviews are available
via telephone, in-person, and video conferencing using a USPTO supplied web-based
collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO
Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.If attempts to
reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be
reached on (571)272-7332. The fax phone number for the organization where this application or
proceeding is assigned is 571-273-8300. Information regarding the status of published or
unpublished applications may be obtained from Patent Center. Unpublished application
information in Patent Center is available to registered users. To file and manage patent
submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center
and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For
additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative, call 800-786
9199 (IN USA OR CANADA) or 571-272-1000.
/ZAID MUHAMMAD SALEH/
Examiner, Art Unit 2668
08/10/2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights