DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-13, 15-19, and 21-27 are pending for examination in the application filed 10/03/2025. Claims 1, 22, and 25 are currently amended and claims 14 and 20 were previously canceled.
Priority
Acknowledgement is made of Applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been received for parent application DE102021125512.0, filing date 10/01/2021.
Response to Arguments
The objection regarding misnumbered claim 27 has been withdrawn in view of the correction. The 35 U.S.C. 112(b) rejections of claims 1, 22, and 25 have been withdrawn in view of the amendments.
Applicant’s arguments with respect to independent claims 1, 22, and 25 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument, as facilitated by the newly added amendments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-8, 13, 17, 21-27 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US20140286562A1) in view of Guetter (US20200167584A1) and Kannan (US20210325308A1).
Regarding claim 1, Chen teaches a computer-implemented method for processing a microscopy image to predict one or more properties of structures at improved accuracy, ([0005] a method performed by one or more processors. [0052] In the example of FIG. 1, system 110 identifies nuclei in representations 126, 128 through numerous segmentations, including, e.g., segmentations 122, 124. The fact that two segmentations 122, 124 are associated with representations 126, 128 strengthens an accuracy of the prediction generated by system 110 that representations 126, 128 are representations of nuclei), the method comprising:
obtaining the microscopy image imaging a plurality of types of one structure, with the plurality of types of the structure having a different appearance in the microscopy image in relation to a structure property ([0005] includes receiving an image to be segmented into one or more representations of one or more biological structures. [0008] the received image includes a microscopy image. [0004] In this example, cell nuclei are segmented from images, e.g., before a feature of the cell nuclei can be quantitatively analyzed and studied for the effects of drugs, genes, and diseases (e.g., cancer). Generally, a feature includes a characteristic of a structure. There are numerous types of features, including, e.g., size, shape, chromatin distribution, location in a cell, and so forth),
for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by adjusting an image property of the microscopy image ([0005] computing a normalized cross correlation of the received image against one or more of the biological structures in the set of biological structures. [0032] In this example, system 110 selects the normalized cross correlation of the set of sub-windows as the optimization criteria to be used in the rigid alignment procedure. The rigid alignment procedure consists of searching for the spatial transformations (translations and rotations) that "best match" the images being registered. The "best match" is computed through a maximization of the cross correlation between the images' intensity values),
in which the structure has an appearance in relation to the structure property that corresponds to a given reference value ([0044] For an image from which a specified type of structure is to be segmented, system 110 computes the normalized cross correlation between each filter and the image. System 110 selects the pixels for which the normalized cross correlation falls above an input threshold. The selected pixels are seed data for use segmenting an image from data set 104. Generally, seed data includes an item of data used in deriving values for other items of data. The seed data represents an estimate of the spatial organization of structures in the image to be segmented).
Chen does not teach determining one or more properties of the structure of each type by applying an image processing algorithm to the plurality of normalized representations of the microscopy image.
Guetter, in the same field of endeavor of cell image analysis, teaches determining one or more properties of the structure of each type by applying an image processing algorithm to the plurality of normalized representations of the microscopy image ([0019] According to one embodiment, the normalization module may be adapted to normalize the counter stained cell nuclei image into a normalized counter stain image. Thereafter, an image operator module may combine the normalized complement of the stain image and the normalized counter stain image in order to generate a combined normalized image. A segmentation module may then automatically extract the individual cells in the combined normalized image. [0006] the assay can be assessed by an observer, typically through a microscope, or image data can be acquired from the assay for further processing. [0007] For image data processed by automated methods, depicted on a display, or for an assay viewed by an observer, a relation may be determined between the local appearance of the stained tissue and the applied stains and biomarkers to determine a model of the biomarker distribution in the stained tissue).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to determine one or more properties of the structure of each type by applying an image processing algorithm to the normalized representation because "certain cell surface receptors may demonstrate both membrane staining (where the cell surface receptors may function as ligand receptors), cytoplasmic staining (where the cell surface receptors may exert some effector functions), and nuclear staining. In each of these cases, the detection of the nuclei may depend upon the staining colors and whether these staining colors are mutually exclusive" [Guetter 0008].
Chen does not explicitly teach for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by determining an associated entity of the microscopy image for each type by duplicating an entirety of the microscopy image, and adjusting an image property of the entity of the microscopy image, in which the structure of has an appearance in relation to the structure property that corresponds to a given reference value, wherein, in each case, for each entity, the image property is adjusted differently.
Kannan, in the same field of endeavor of microscopy image analysis, teaches for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by determining an associated entity of the microscopy image for each type by duplicating an entirety of the microscopy image, and adjusting an image property of the entity of the microscopy image ([Abstract] The method includes receiving a brightfield image generated by a brightfield microscopy imaging modality of at least a portion of cells included in a specimen. [0128] At 916, the process 900 can provide the brightfield image to a trained model. In some embodiments, the model can include the generator 408 in FIG. 4 trained using the process 800 in FIG. 8, the trained model 508, and/or the neural network 600 trained using the process 800 in FIG. 8. In some embodiments, the trained model can include three neural networks, and each neural network can receive a copy of the brightfield image and output a different channel (e.g., red, green, or blue) of an artificial fluorescent image. [0076] In some embodiments, the fluorescent imaging can include producing three channels of data for each cell. The three channels of data can include a blue/all nuclei channel, a green/apoptotic channel, and a red/pink/dead channel), in which the structure of has an appearance in relation to the structure property that corresponds to a given reference value, wherein, in each case, for each entity, the image property is adjusted differently ([0076] The three channels of data can include a blue/all nuclei channel, a green/apoptotic channel, and a red/pink/dead channel. [0089] For the three pixel intensities corresponding to the three fluorophores used to generate the fluorescent image, the input range can be re-scaled using the mode of the pixel intensity distribution as the lower bound value and 1/10th the maximum pixel intensity as the upper bound. The contrast enhancement process can choose the upper bound in order to avoid oversaturated pixels and focus on cell signal. The contrast enhancement process can normalize each pixel intensity based on the lower bound and the upper bound, which function as a min/max range, using a min-max norm, and then each pixel can be multiplied by the output range [0,255]. For the brightfield image 404, the contrast enhancement process can determine an input range by uniformly stretching the 2nd and 98th percentile of pixel intensities to the output range [0,255]. [0090] For images with low signal, background noise may be included in the output range. To minimize any remaining back-ground noise, the contrast enhancement process can clip the minimum pixel value by two integer values for the red and green channels, and by three integer values for the blue channel, where the intensity range is wider on average. The maximum pixel values can be increased accordingly to preserve intensity range per image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Kannan to determine associated entities of the microscopy image by duplicating the microscopy image and adjusting image properties of the entities of the microscopy images differently because "a trained model is used to predict the percentages of live and dead cells in a sample (i.e., values which would typically be determined using fluorescent stains, such as Caspase-3/7 and TO-PRO-3 stain), using only a brightfield image as input. These visualization methods may then be used in a high-throughput screen for drugs that kill cancer cells within tumor organoids generated from patient tumor samples" [0049].
Regarding claim 2, Chen, Guetter, and Kannan teach the method of claim 1. Guetter teaches wherein each type of the structure is associated with one or more image portions of the microscopy image, in which portions the respective type of the structure occurs or is dominant, wherein the image processing algorithm is applied taking account of the corresponding one or more image portions ([0020] A segmentation module may then automatically detect all the cells from the binary complement stain image. A nuclei segmentation module segments all nuclei (e.g., blue ellipses in blue image channel) from the counter stain image. An object operator module then combines the cell detection results from both images into the final cell detection output).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to apply the image processing algorithm taking into account image portions so that "As a result, the complement image 234 clearly reflects the previously masked light-colored nuclei" [Guetter 0116].
Regarding claim 3, Chen, Guetter, and Kannan teach the method of claim 2. Guetter teaches wherein the image processing algorithm is applied to the normalized representations and those parts of an output of the image processing algorithm which are associated with different image portions to the respectively assigned one or more image portions are discarded in each case ([0116] The images generated by color deconvolution module 220 are then individually transformed into binary images by means of simple or sophisticated segmentation into foreground and background. For example, in this exemplary illustration, the foreground is composed of all areas in the image that are covered by a brown DAB stain. A stain image complement module 232 is adapted to automatically generate a binary complement 234 of stain image 230. In the complement of a binary image, zeros become ones and ones become zeros, leading to the reversal of color. [0118] an image operator module 270 (FIG. 2) is adapted to combine the normalized complement 262 of the stain image 230 and the normalized binary image of the counter stain image 264, to generate a combined normalized image 272. A segmentation module 275 (FIG. 2) of cell detection engine 114 then detects the nuclei within the combined, normalized image 272, and outputs a combined detected cells image 280 (FIGS. 2, 3, 4, 9).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to apply the image processing algorithm to the normalized representations associated with image portions so that "As a result, the complement image 234 clearly reflects the previously masked light-colored nuclei" [Guetter 0116].
Regarding claim 5, Chen, Guetter, and Kannan teach the method of claim 2. Guetter teaches for each type: masking image regions other than the respective one or more image regions in the corresponding normalized representation prior to the application of the image processing algorithm to the respective normalized representation ([0116] The images generated by color deconvolution module 220 are then individually transformed into binary images by means of simple or sophisticated segmentation into foreground and background. For example, in this exemplary illustration, the foreground is composed of all areas in the image that are covered by a brown DAB stain. A stain image complement module 232 is adapted to automatically generate a binary complement 234 of stain image 230. In the complement of a binary image, zeros become ones and ones become zeros, leading to the reversal of color. [0118] an image operator module 270 (FIG. 2) is adapted to combine the normalized complement 262 of the stain image 230 and the normalized binary image of the counter stain image 264, to generate a combined normalized image 272. A segmentation module 275 (FIG. 2) of cell detection engine 114 then detects the nuclei within the combined, normalized image 272, and outputs a combined detected cells image 280 (FIGS. 2, 3, 4, 9).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to mask image regions so that "As a result, the complement image 234 clearly reflects the previously masked light-colored nuclei" [Guetter 0116].
Regarding claim 6, Chen, Guetter, and Kannan teach the method of claim 2. Guetter teaches for each type: determining the respective one or more image portions on the basis of an object recognition of the plurality of types of the structure in the microscopy image ([0099] After acquiring image 210, image analysis system 100 may pass the image 210 to an object identifier 110, which functions to identify and mark relevant objects and other features within image 210 that will later be used for cell detection. Object identifier 110 may extract from (or generate for) each image 210 a plurality of image features characterizing the various objects in the image as a well as pixels representing expression of the biomarker(s)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to use object recognition because "the features extracted by object identifier 110 may include features or feature vectors sufficient to categorize the objects in the sample as biomarker-positive objects of interest (as denoted by the red arrow arrow 810 in FIG. 8) or biomarker-negative objects of interest (as denoted by the blue arrow arrow 820 in FIG. 8) and/or by level or intensity of biomarker staining of the object…Thus, the objects may then be categorized at least on the basis of biomarker expression (for example, biomarker-positive or biomarker-negative cells) and, if relevant, a sub-type of the object (e.g. tumor cell, immune cell, etc.)" [Guetter 0100].
Regarding claim 7, Chen, Guetter, and Kannan teach the method of claim 6. Chen further teaches wherein the object recognition uses the appearances of the plurality of types of the structure as prior knowledge ([0024] For example, using the template, the system may automatically (or semi-automatically) identify representations of nuclei in various images. [0025] In this example, the system is configured to semiautomatically generate the template using training images, in which a user has manually delineated structures that are present in the training images. The template represents a structure that is based on features of the delineated structures. For example, the template may include an average structure of a cell).
Regarding claim 8, Chen, Guetter, and Kannan teach the method of claim 6. Chen further teaches wherein the object recognition predicts the appearances of the plurality of types of the structure ([0024] For example, using the template, the system may automatically (or semi-automatically) identify representations of nuclei in various images. [0059] Using the template, system 110 generates (508) a statistical model of spatial transformations representing possible variations in the shape of the sample nuclei that were marked in training image 109).
Regarding claim 13, Chen, Guetter, and Kannan teach the method of claim 2. Guetter teaches for each type: segmenting contrast values of the microscopy image, and determining the one or more image portions on the basis of the segmentation ([0070] An object isolation process in which an object of interest (e.g., a cell, tumor, or bone, etc.) is separated from a background of varying, for example, the optical complexity. Conventional image segmentation techniques can be broadly divided into three categories: [0071] 1. Edge- or contour-based approaches, in which sustained intensity changes, or edges, are marked (usually with a discrete derivative operator). [0072] 2. Direct region segmentation approaches, in which the image is partitioned according to some homogeneity criterion. [0073] 3. Gray-level thresholding, where bright (or dark) objects are separated from the background by the use of an appropriate threshold value. [0020] A segmentation module may then automatically detect all the cells from the binary complement stain image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to segment contrast values to determine image portions because "certain cell surface receptors may demonstrate both membrane staining (where the cell surface receptors may function as ligand receptors), cytoplasmic staining (where the cell surface receptors may exert some effector functions), and nuclear staining. In each of these cases, the detection of the nuclei may depend upon the staining colors and whether these staining colors are mutually exclusive" [Guetter 0008].
Regarding claim 17, Chen, Guetter, and Kannan teach the method of claim 1. Chen further teaches wherein the structure property comprises a size of the structures such that the different types of the structure appear in the microscopy image with different sizes ([0004] In this example, cell nuclei are segmented from images, e.g., before a feature of the cell nuclei can be quantitatively analyzed and studied for the effects of drugs, genes, and diseases (e.g., cancer). Generally, a feature includes a characteristic of a structure. There are numerous types of features, including, e.g., size, shape, chromatin distribution, location in a cell, and so forth),
and wherein the size of the microscopy image is adjusted in order to obtain the respective normalized representation such that the size of the respective type corresponds to a given size reference value ([0031] To promote uniformity in the size of the rectangular sub-windows, system 110 adds additional data (e.g., a binary string of zeros) to an image of a structure included in the rectangular sub-windows, e.g., to render each sub-window of the same size (in terms of number of pixels in each dimension) as the largest rectangular sub-window in the set).
Regarding claim 21, Chen, Guetter, and Kannan teach the method of claim 1. Chen further teaches wherein the appearance of the structure in respect of the structure property in the normalized representation is closer to an appearance of the structure in respect of the structure property in training images of the image processing algorithm ([0005] wherein the training image is associated with a level of modality that corresponds to a level of modality associated with the image to be segmented).
Regarding claim 22, Chen teaches a device for processing a microscopy image, the device comprising a processor configured to ([0011] In still another aspect of the disclosure, an electronic system includes one or more processors; and one or more machine-readable media configured to store instructions that are executable by the one or more processors to perform operations):
obtain the microscopy image imaging a plurality of types of one structure, with the plurality of types of the structure having a different appearance in the microscopy image in relation to a structure property ([0005] includes receiving an image to be segmented into one or more representations of one or more biological structures. [0008] the received image includes a microscopy image. [0004] In this example, cell nuclei are segmented from images, e.g., before a feature of the cell nuclei can be quantitatively analyzed and studied for the effects of drugs, genes, and diseases (e.g., cancer). Generally, a feature includes a characteristic of a structure. There are numerous types of features, including, e.g., size, shape, chromatin distribution, location in a cell, and so forth),
for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by adjusting an image property of the microscopy image ([0005] computing a normalized cross correlation of the received image against one or more of the biological structures in the set of biological structures. [0032] In this example, system 110 selects the normalized cross correlation of the set of sub-windows as the optimization criteria to be used in the rigid alignment procedure. The rigid alignment procedure consists of searching for the spatial transformations (translations and rotations) that "best match" the images being registered. The "best match" is computed through a maximization of the cross correlation between the images' intensity values),
in which the structure has an appearance in relation to the structure property that corresponds to a given reference value ([0044] For an image from which a specified type of structure is to be segmented, system 110 computes the normalized cross correlation between each filter and the image. System 110 selects the pixels for which the normalized cross correlation falls above an input threshold. The selected pixels are seed data for use segmenting an image from data set 104. Generally, seed data includes an item of data used in deriving values for other items of data. The seed data represents an estimate of the spatial organization of structures in the image to be segmented).
Chen does not teach apply an image processing algorithm to the plurality of normalized representations of the microscopy image.
Guetter, in the same field of endeavor of cell image analysis, teaches apply an image processing algorithm to the plurality of normalized representations of the microscopy image ([0019] According to one embodiment, the normalization module may be adapted to normalize the counter stained cell nuclei image into a normalized counter stain image. Thereafter, an image operator module may combine the normalized complement of the stain image and the normalized counter stain image in order to generate a combined normalized image. A segmentation module may then automatically extract the individual cells in the combined normalized image. [0006] the assay can be assessed by an observer, typically through a microscope, or image data can be acquired from the assay for further processing. [0007] For image data processed by automated methods, depicted on a display, or for an assay viewed by an observer, a relation may be determined between the local appearance of the stained tissue and the applied stains and biomarkers to determine a model of the biomarker distribution in the stained tissue).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Chen with the teachings of Guetter to apply an image processing algorithm to the normalized representation because "certain cell surface receptors may demonstrate both membrane staining (where the cell surface receptors may function as ligand receptors), cytoplasmic staining (where the cell surface receptors may exert some effector functions), and nuclear staining. In each of these cases, the detection of the nuclei may depend upon the staining colors and whether these staining colors are mutually exclusive" [Guetter 0008].
Chen does not explicitly teach for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by determining an associated entity of the microscopy image for each type by duplicating an entirety of the microscopy image, and adjusting an image property of the entity of the microscopy image, in which the structure of has an appearance in relation to the structure property that corresponds to a given reference value, wherein, in each case, for each entity, the image property is adjusted differently.
Kannan, in the same field of endeavor of microscopy image analysis, teaches for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by determining an associated entity of the microscopy image for each type by duplicating an entirety of the microscopy image, and adjusting an image property of the entity of the microscopy image ([Abstract] The method includes receiving a brightfield image generated by a brightfield microscopy imaging modality of at least a portion of cells included in a specimen. [0128] At 916, the process 900 can provide the brightfield image to a trained model. In some embodiments, the model can include the generator 408 in FIG. 4 trained using the process 800 in FIG. 8, the trained model 508, and/or the neural network 600 trained using the process 800 in FIG. 8. In some embodiments, the trained model can include three neural networks, and each neural network can receive a copy of the brightfield image and output a different channel (e.g., red, green, or blue) of an artificial fluorescent image. [0076] In some embodiments, the fluorescent imaging can include producing three channels of data for each cell. The three channels of data can include a blue/all nuclei channel, a green/apoptotic channel, and a red/pink/dead channel), in which the structure of has an appearance in relation to the structure property that corresponds to a given reference value, wherein, in each case, for each entity, the image property is adjusted differently ([0076] The three channels of data can include a blue/all nuclei channel, a green/apoptotic channel, and a red/pink/dead channel. [0089] For the three pixel intensities corresponding to the three fluorophores used to generate the fluorescent image, the input range can be re-scaled using the mode of the pixel intensity distribution as the lower bound value and 1/10th the maximum pixel intensity as the upper bound. The contrast enhancement process can choose the upper bound in order to avoid oversaturated pixels and focus on cell signal. The contrast enhancement process can normalize each pixel intensity based on the lower bound and the upper bound, which function as a min/max range, using a min-max norm, and then each pixel can be multiplied by the output range [0,255]. For the brightfield image 404, the contrast enhancement process can determine an input range by uniformly stretching the 2nd and 98th percentile of pixel intensities to the output range [0,255]. [0090] For images with low signal, background noise may be included in the output range. To minimize any remaining back-ground noise, the contrast enhancement process can clip the minimum pixel value by two integer values for the red and green channels, and by three integer values for the blue channel, where the intensity range is wider on average. The maximum pixel values can be increased accordingly to preserve intensity range per image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Chen with the teachings of Kannan to determine associated entities of the microscopy image by duplicating the microscopy image and adjusting image properties of the entities of the microscopy images differently because "a trained model is used to predict the percentages of live and dead cells in a sample (i.e., values which would typically be determined using fluorescent stains, such as Caspase-3/7 and TO-PRO-3 stain), using only a brightfield image as input. These visualization methods may then be used in a high-throughput screen for drugs that kill cancer cells within tumor organoids generated from patient tumor samples" [0049].
Regarding claim 23, Chen, Guetter, and Kannan teach the device of claim 22. Guetter teaches for each type: mask image regions other than the respective one or more image regions in the corresponding normalized representation prior to the application of the image processing algorithm to the respective normalized representation ([0116] The images generated by color deconvolution module 220 are then individually transformed into binary images by means of simple or sophisticated segmentation into foreground and background. For example, in this exemplary illustration, the foreground is composed of all areas in the image that are covered by a brown DAB stain. A stain image complement module 232 is adapted to automatically generate a binary complement 234 of stain image 230. In the complement of a binary image, zeros become ones and ones become zeros, leading to the reversal of color. [0118] an image operator module 270 (FIG. 2) is adapted to combine the normalized complement 262 of the stain image 230 and the normalized binary image of the counter stain image 264, to generate a combined normalized image 272. A segmentation module 275 (FIG. 2) of cell detection engine 114 then detects the nuclei within the combined, normalized image 272, and outputs a combined detected cells image 280 (FIGS. 2, 3, 4, 9).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to mask image regions so that "As a result, the complement image 234 clearly reflects the previously masked light-colored nuclei" [Guetter 0116].
Regarding claim 24, Chen, Guetter, and Kannan teach the method of claim 1. Chen further teaches wherein a first image content of the microscopy image and a second image content of the normalized representation of the microscopy image are the same ([0032] In this example, system 110 selects the normalized cross correlation of the set of sub-windows as the optimization criteria to be used in the rigid alignment procedure. The rigid alignment procedure consists of searching for the spatial transformations (translations and rotations) that "best match" the images being registered. The "best match" is computed through a maximization of the cross correlation between the images' intensity values. [0006] In some implementations, segmenting includes: using the normalized cross correction of the received image to the seed data to spatially register one or more items of seed data to the one or more biological structures in the received image).
Regarding claim 25, Chen teaches a computer-implemented method for processing a microscopy image to apply image improvement to a microscopy image, ([0005] a method performed by one or more processors. [0052] [0052] In the example of FIG. 1, system 110 identifies nuclei in representations 126, 128 through numerous segmentations, including, e.g., segmentations 122, 124. The fact that two segmentations 122, 124 are associated with representations 126, 128 strengthens an accuracy of the prediction generated by system 110 that representations 126, 128 are representations of nuclei), the method comprising:
obtaining the microscopy image imaging a plurality of types of a structure, with the plurality of types of the structure having a different appearance in the microscopy image in relation to a structure property ([0005] includes receiving an image to be segmented into one or more representations of one or more biological structures. [0008] the received image includes a microscopy image. [0004] In this example, cell nuclei are segmented from images, e.g., before a feature of the cell nuclei can be quantitatively analyzed and studied for the effects of drugs, genes, and diseases (e.g., cancer). Generally, a feature includes a characteristic of a structure. There are numerous types of features, including, e.g., size, shape, chromatin distribution, location in a cell, and so forth),
for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by adjusting an image property of the microscopy image ([0005] computing a normalized cross correlation of the received image against one or more of the biological structures in the set of biological structures. [0032] In this example, system 110 selects the normalized cross correlation of the set of sub-windows as the optimization criteria to be used in the rigid alignment procedure. The rigid alignment procedure consists of searching for the spatial transformations (translations and rotations) that "best match" the images being registered. The "best match" is computed through a maximization of the cross correlation between the images' intensity values),
in which the structure has an appearance in relation to the structure property that corresponds to a given reference value ([0044] For an image from which a specified type of structure is to be segmented, system 110 computes the normalized cross correlation between each filter and the image. System 110 selects the pixels for which the normalized cross correlation falls above an input threshold. The selected pixels are seed data for use segmenting an image from data set 104. Generally, seed data includes an item of data used in deriving values for other items of data. The seed data represents an estimate of the spatial organization of structures in the image to be segmented).
Chen does not teach improving a structure-related appearance of the microscopy image by applying an image processing algorithm to the plurality of normalized representations of the microscopy image.
Guetter, in the same field of endeavor of cell image analysis, teaches improving a structure-related appearance of the microscopy image by applying an image processing algorithm to the plurality of normalized representations of the microscopy image ([0019] According to one embodiment, the normalization module may be adapted to normalize the counter stained cell nuclei image into a normalized counter stain image. Thereafter, an image operator module may combine the normalized complement of the stain image and the normalized counter stain image in order to generate a combined normalized image. A segmentation module may then automatically extract the individual cells in the combined normalized image. [0006] the assay can be assessed by an observer, typically through a microscope, or image data can be acquired from the assay for further processing. [0007] For image data processed by automated methods, depicted on a display, or for an assay viewed by an observer, a relation may be determined between the local appearance of the stained tissue and the applied stains and biomarkers to determine a model of the biomarker distribution in the stained tissue).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to determine one or more properties of the structure of each type by applying an image processing algorithm to the normalized representation because "certain cell surface receptors may demonstrate both membrane staining (where the cell surface receptors may function as ligand receptors), cytoplasmic staining (where the cell surface receptors may exert some effector functions), and nuclear staining. In each of these cases, the detection of the nuclei may depend upon the staining colors and whether these staining colors are mutually exclusive" [Guetter 0008].
Chen does not explicitly teach for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by determining an associated entity of the microscopy image for each type by duplicating an entirety of the microscopy image, and adjusting an image property of the entity of the microscopy image, in which the structure of has an appearance in relation to the structure property that corresponds to a given reference value, wherein, in each case, for each entity, the image property is adjusted differently.
Kannan, in the same field of endeavor of microscopy image analysis, teaches for each of the plurality of types: in each case obtaining a corresponding normalized representation of the microscopy image by determining an associated entity of the microscopy image for each type by duplicating an entirety of the microscopy image, and adjusting an image property of the entity of the microscopy image ([Abstract] The method includes receiving a brightfield image generated by a brightfield microscopy imaging modality of at least a portion of cells included in a specimen. [0128] At 916, the process 900 can provide the brightfield image to a trained model. In some embodiments, the model can include the generator 408 in FIG. 4 trained using the process 800 in FIG. 8, the trained model 508, and/or the neural network 600 trained using the process 800 in FIG. 8. In some embodiments, the trained model can include three neural networks, and each neural network can receive a copy of the brightfield image and output a different channel (e.g., red, green, or blue) of an artificial fluorescent image. [0076] In some embodiments, the fluorescent imaging can include producing three channels of data for each cell. The three channels of data can include a blue/all nuclei channel, a green/apoptotic channel, and a red/pink/dead channel), in which the structure of has an appearance in relation to the structure property that corresponds to a given reference value, wherein, in each case, for each entity, the image property is adjusted differently ([0089] In some embodiments, the desired intensity range of an input image to be stretched can be decided on a per image basis as follows: For the three pixel intensities corresponding to the three fluorophores used to generate the fluorescent image, the input range can be re-scaled using the mode of the pixel intensity distribution as the lower bound value and 1/10th the maximum pixel intensity as the upper bound. The contrast enhancement process can choose the upper bound in order to avoid oversaturated pixels and focus on cell signal. The contrast enhancement process can normalize each pixel intensity based on the lower bound and the upper bound, which function as a min/max range, using a min-max norm, and then each pixel can be multiplied by the output range [0,255]. For the brightfield image 404, the contrast enhancement process can determine an input range by uniformly stretching the 2nd and 98th percentile of pixel intensities to the output range [0,255]. [0090] For images with low signal, background noise may be included in the output range. To minimize any remaining back-ground noise, the contrast enhancement process can clip the minimum pixel value by two integer values for the red and green channels, and by three integer values for the blue channel, where the intensity range is wider on average. The maximum pixel values can be increased accordingly to preserve intensity range per image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Kannan to determine associated entities of the microscopy image by duplicating the microscopy image and adjusting image properties of the entities of the microscopy images differently because "a trained model is used to predict the percentages of live and dead cells in a sample (i.e., values which would typically be determined using fluorescent stains, such as Caspase-3/7 and TO-PRO-3 stain), using only a brightfield image as input. These visualization methods may then be used in a high-throughput screen for drugs that kill cancer cells within tumor organoids generated from patient tumor samples" [0049].
Regarding claim 26, Chen, Guetter, and Kannan teach the method of claim 25. Guetter teaches wherein the image processing algorithm comprises at least one of an image-to- image transformation, a reconstruction of one or more image parts, or an image modality transformation ([0116] The images generated by color deconvolution module 220 are then individually transformed into binary images by means of simple or sophisticated segmentation into foreground and background).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to perform an image-to-image transformation so that "As a result, the complement image 234 clearly reflects the previously masked light-colored nuclei" [Guetter 0116].
Regarding claim 27, Chen, Guetter, and Kannan teach the method of claim 1. Guetter teaches wherein the image processing algorithm is not applied to the non-normalized microscopy image ([0019] According to one embodiment, the normalization module may be adapted to normalize the counter stained cell nuclei image into a normalized counter stain image. Thereafter, an image operator module may combine the normalized complement of the stain image and the normalized counter stain image in order to generate a combined normalized image. A segmentation module may then automatically extract the individual cells in the combined normalized image).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Guetter to not apply an image processing algorithm to the non-normalized representation because "certain cell surface receptors may demonstrate both membrane staining (where the cell surface receptors may function as ligand receptors), cytoplasmic staining (where the cell surface receptors may exert some effector functions), and nuclear staining. In each of these cases, the detection of the nuclei may depend upon the staining colors and whether these staining colors are mutually exclusive" [Guetter 0008].
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Guetter, Kannan and Cohen (US20130044940A1).
Regarding claim 4, Chen, Guetter, and Kannan teach the method of claim 2. Cohen, in the same field of endeavor of microscopy image analysis, teaches wherein the image processing algorithm is applied to the normalized representations ([0025] The preprocessing module 120 may preprocess the microscopy image 102 and prepare it for subsequent processing and analysis of the microscopy image sections. For example, the preprocessing module 120 may perform various image corrections before the microscopy image 102 is divided into image sections), and wherein the image processing algorithm is in each case applied selectively to the respective one or more image portions associated with the corresponding type ([Abstract] Multiple image sections are respectively assigned to multiple processing units, and the processing units respectively process the image sections in parallel).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Cohen to apply the image processing algorithm selectively to the respective one or more image portions associated with the corresponding type to result in improved identification and measurement of objects in microscopy image via multiple processing units in parallel, which also saves processing time (Cohen [0020]).
Claims 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Guetter, Kannan and Stember (US20230377195A1).
Regarding claim 9, Chen, Guetter, and Kannan teach the method of claim 2. Stember, in the same field of endeavor of biomedical image analysis, teaches for each type: determining the respective one or more image portions using a clustering algorithm ([0018] the computing system may establish a clustering model comprising a second set of transform layers to generate the plurality of clusters having a set number of clusters using a training dataset).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Stember to determine image portions using a clustering algorithm because "Each of the plurality of clusters may correspond to a region in the biomedical image having a respective characteristic" [Stember 0011].
Regarding claim 10, Chen, Guetter, Kannan, and Stember teach the method of claim 9. Stember teaches wherein clusters are determined on the basis of a distance of latent feature representations of a machine-learned encoding branch which encodes the microscopy image ([0288] Each region 2930 may correspond to a common latent feature shared among the feature vectors 2920 within the feature space 2915 in the region 2930, and at least one of the regions 2930 may correspond to the SOI 2910 in the image 2905. Each region 2930 may correspond to a portion of the feature space 2915. In some embodiments, each region 2930 may correspond to the portion of the feature space 2915 based on a distance about the associated centroid 825 in the feature space 2915. The distance may be, for example, proximity in terms of Euclidean distance or L-norm distance, among others, to the centroid 825 defining the respective region 2930. [0256] Referring now to FIG. 25A, depicted is a block diagram of an architecture 2500 of the localization model 2335 of the system 2300 for localizing images. Under the architecture 2500, the localization model 2335 may include at least one encoder block 2505. [0264] the imaging device 2310 may support other imaging modalities besides the ones listed above, such as optical photography or microscopy, among others).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Stember to determine clusters based on distance of latent feature representations for "identifying at least one cluster corresponding to at least one structure of interest (SOI) in the biomedical image" [Stember 0015].
Regarding claim 11, Chen, Guetter, Kannan, and Stember teach the method of claim 9. Stember teaches wherein clusters are determined on the basis of a segmentation of contrast values of the microscopy image or on the basis of an adjustment parameter determined pixel-by-pixel using an image-to-image transformation ([0120] This approach separates the image into N.sub.sp superpixels using nearest-neighbors with the features of red-green-blue color channel pixel intensity each in the range from 0 to 255 and x, y position. This tends to cluster together regions of the image that are of similar color and spatially close, with an example superpixel tiling displayed on the right image in FIG. 5. [0013] In some embodiments, the computing system may apply the biomedical image to the segmentation model for a number of iterations. The number of iterations may be identified based on a set number of clusters. In some embodiments, the computing system may select, from the plurality of clusters, the at least one cluster as the at least one segment based on the state and the action performed by the agent in accordance with the output).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Stember to determine clusters based on the segmentation of contrast values because "The goal is for each pixel in a given superpixel to be the one type of feature" [Stember 0032].
Regarding claim 12, Chen, Guetter, Kannan, and Stember teach the method of claim 9. Stember teaches wherein a quantity of the plurality of types is given as a boundary condition of the clustering algorithm ([0013] In some embodiments, the computing system may apply the biomedical image to the segmentation model for a number of iterations. The number of iterations may be identified based on a set number of clusters. In some embodiments, the computing system may select, from the plurality of clusters, the at least one cluster as the at least one segment based on the state and the action performed by the agent in accordance with the output. [0125] Training the clustering CNN is somewhat unique in that it ends when the number of clusters/classes, N.sub.cl reaches down to 25. [0154] For simplicity, the number of clusters produced were restricted by the clustering CNN to be N.sub.cl 2 clusters or classes, the lesion mask and its complement. In general, the clustering CNN would tile the image into N.sub.cl clusters. Through trial and error, it was found that for these images, around 25 clusters gave the best segmentation of important structures).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Stember to use a quantity of the plurality of types for the clustering algorithm because "Each cluster 2935 may correspond to a region or an area in the image 2905 having a common visual characteristic…The number of clusters 2935 in the image 2905 may correspond to the number of regions 2930 (and by extension the number of centroids 2925) in the feature space 2915 as defined by the cluster generator 2835" [Stember 0292].
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Guetter, Kannan and Caicedo (Caicedo, Juan C., et al. "Data-analysis strategies for image-based cell profiling." Nature methods 14.9 (2017): 849-863).
Regarding claim 15, Chen, Guetter, and Kannan teach the method of claim 1. Caicedo, in the same field of endeavor of cell image analysis, teaches wherein the outputs provided by the image processing algorithm for each of the plurality of normalized representations encode the one or more properties of the respective type of the structure for different image positions in the microscopy image ([pg.851, Feature Extraction] The phenotypic characteristics of each cell are measured in a step called feature extraction, which provides the raw data for profiling. [pg. 851, Microenvironment and context features] Microenvironment and context features. These features include counts and spatial relationships among cells in the field of view (on the basis of the number of and distance to cells in a neighborhood) as well as its position relative to a cell colony. [pg. 853 Feature transformation and normalization] Morphological profiles include features that display varying shapes of statistical distributions. It is therefore essential to transform feature values with simple mathematical operations, such that the values are approximately normally distributed and mean centered and have comparable s.d.),
the method further comprising: image position-resolved combining of the outputs of the image processing algorithm ([pg. 855, Classification] Single-cell data points from all treatment conditions are then assigned to one of the subpopulations identified in the previous step. This assignment can be done by using a feature-evaluation rule, such as proximity, similarity, or feature weighting. This step is necessary because subpopulation identification is typically performed only on a subset of cells.; [pg. 855, Aggregation] For each treatment condition, vectors are calculated and yield the number (or fraction) of cells within each subpopulation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Caicedo to encode the type of structure for image positions because defining subpopulations can assist in inferring biological meaning, by identifying over- and underrepresented subpopulations of cells under a given treatment condition and by improving understanding of the dynamics of how cells transition between different phenotypes (Caicedo [pg. 855, Aggregation]).
Regarding claim 16, Chen, Guetter, and Kannan teach the method of claim 1. Caicedo teaches wherein the outputs provided by the image processing algorithm for each of the plurality of normalized representations encode the one or more hidden properties of the structure as a sample-global parameter ([pg. 855, Step 5: single-cell data aggregation] Profiles are data representations that describe the morphological state of an individual cell or a population of cells. Population-level (also called image-level or well-level) representations are obtained by aggregating the measurements of single cells into a single vector to summarize the typical features of the population, so that populations can be compared (Fig. 5)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Caicedo to encode properties of the structure as a global parameter because defining subpopulations can assist in inferring biological meaning, by identifying over- and underrepresented subpopulations of cells under a given treatment condition and by improving understanding of the dynamics of how cells transition between different phenotypes (Caicedo [pg. 855, Aggregation]).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Guetter, Kannan and Bayley (WO2017149296A1).
Regarding claim 18, Chen, Guetter, and Kannan teach the method of claim 1. Chen further teaches wherein the structures are cells ([0004] In this example, cell nuclei are segmented from images, e.g., before a feature of the cell nuclei can be quantitatively analyzed and studied for the effects of drugs, genes, and diseases (e.g., cancer)), wherein the structure property is a size of the cells ([0004] Generally, a feature includes a characteristic of a structure. There are numerous types of features, including, e.g., size, shape, chromatin distribution, location in a cell, and so forth.), wherein the image property is a scaling of the microscopy image ([0031] To promote uniformity in the size of the rectangular sub-windows, system 110 adds additional data (e.g., a binary string of zeros) to an image of a structure included in the rectangular sub-windows, e.g., to render each sub-window of the same size (in terms of number of pixels in each dimension) as the largest rectangular sub-window in the set.).
Chen does not teach wherein a quantity of the cells is determined as a sample-global parameter on the basis of an output of the image processing algorithm.
Bayley, in the same field of endeavor of cell analysis, teaches wherein a quantity of the cells is determined as a sample-global parameter on the basis of an output of the image processing algorithm ([pg. 25] The automated cell counter gives the live cell density, dead cell density and total cell density. The average of the two live cell density counts was used as the value for the initial cell density).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Bayley to use a quantity of cells for the sample-global parameter to enhance the method via enabling counting of different cell types that can be used for further calculations or processing (Bayley [pg. 25]).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Guetter, Kannan, Bayley, and Chung (KR20180105439A).
Regarding claim 19, Chen, Guetter, Kannan, and Bayley teach the method of claim 18. Chung, in the same field of endeavor of cell image analysis, teaches wherein the image processing algorithm is applied to each of the plurality of normalized representations and the respective output comprises a density map, the density map encoding a probability for the presence or absence of cells of the respective type ([pg. 5 para. 2] As shown in FIGS. 4 and 5, a plurality of object tracking operations first converts an image frame stored in gray scale of an original microscope image into a probability density map. That is, And converting the pixel values into a probability density map in an image frame. Here, transforming an image frame into a probability density map means transforming each object (cell sample) existing in an image frame into a two-dimensional probability density function expressed as a normal distribution).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Chen with the teachings of Chung to have a probability density map because it enables the identification (via position and size) and subsequent analysis of cells (Chung [Abstract]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights