DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 9-12, 14-16 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitation "wherein obtaining a translational alignment vector" in lines 1-2. There is insufficient antecedent basis for this limitation in the claim. Specifically, parent claim 1 discloses a step of “determining a translational alignment vector” however not “obtaining” as referenced in claim 9.
Claims 10-12 are rejected by the virtue of their dependency upon rejected claim 9 rejected above.
Claim 14 recites “applying the image mask of the pair of images to obtain a pair of updated images” however it is unclear to what the mask is being applied. Further, parent claim 1 recites “generating an image mask for the pair of images” and then claim 14 recites “the image mask of the pair of images”. It’s further unclear if the mask is “for” or “of” the pair of images. Perhaps the “of” should be changed to “to”?
Claims 15 and 16 are rejected by the virtue of it’s dependency upon claim 14 rejected above.
Claim 20 recites “applying the image mask of the pair of images to obtain a pair of updated images” however it is unclear to what the mask is being applied. Further, parent claim 19 recites “generating an image mask for the pair of images” and then claim 20 recites “the image mask of the pair of images”. It’s further unclear if the mask is “for” or “of” the pair of images. Perhaps the “of” should be changed to “to”?
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over US 2025/0246010 to Jesudason et al. (“Jesudason”) in view of USPN 6,067,373 to Ishida et al. (“Ishida”).
Regarding claim 1, Jesudason discloses a computer-implemented method, comprising:
accessing a pair of images of a tissue sample, comprising a first image of a first modality and a second image of a second modality (Fig. 2B, element 210; paragraphs 43, 55 and 56, wherein first and second images tissue cells is accessed, the images being of different visualization modalities, such as mxIF and H&E);
generating a plurality of image patch pairs from the pair of images, wherein each image patch pair comprises a first image patch sampled from a first location in the first image and a second image patch sampled from a second location in the second image, wherein the first location in the first image corresponds to the second location in the second image (Fig. 2B, element 212; paragraph 56, wherein corresponding image regions/patches are generated from each of images);
determining a
determining an image distance metric between the first image patch and the second image patch of each image patch pair, the image distance metric indicating a matching level between the first image patch and the second image patch of each image patch pair (Fig. 2B, element 214; paragraphs 46-48, 56, wherein regions/patches in each of the images are matched and used to align the images); and
As noted by the double strike-throughs above, Jesudason does not disclose expressly determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric and generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs.
Ishida discloses a process for registering/aligning a pair of images that includes determining a plurality of translational alignment vectors corresponding to a plurality of image patch/region pairs by: determining an image distance metric between the first image patch and the second image patch of each image patch pair, the image distance metric indicating a matching level between the first image patch and the second image patch of each image patch pair; and determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric (Figs. 4A-4D; column 6, line 1 – column 7, line 26, wherein regions of interest (ROI) (i.e. image patches) are matched across the images and shift vectors are determined by optimizing (i.e. minimizing) shift values (i.e. distance metric) associated with the ROI using cross-correlation); and
generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs (Fig. 4A, element 480, column 1, line 55 – column 2, line 63 and column 6, line 1 – column 7, line 26, wherein a subtraction image (i.e. image mask) for the pair of images is ultimately generated based on the shift vectors (i.e. translation alignment vectors) corresponding to each ROI).
Jesudason & Ishida are combinable because they are from the same art of image processing, specifically image alignment.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to incorporate the technique of determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric and generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs, as taught by Ishida, into the process of aligning image pairs disclosed by Jesudason.
The suggestion/motivation for doing so would have been to overcome severe misregistration errors due to inclination and/or rotation between a pair of images, as well as providing an image subtraction technique to aid in detecting of changes between images (Ishida, column 2, lines 21-27).
Therefore, it would have been obvious to combine Ishida with Jesudason to obtain the invention as specified in claim 1.
Regarding claim 2, the combination of Jesudason and Ishida discloses the computer-implemented method of claim 1, further comprising:
prior to accessing a pair of images corresponding to a tissue sample, comprising a first image of a first modality and a second image of a second modality, obtaining the first image of the first modality and the second image of the second modality (Jesudason, Fig. 1A; paragraphs 35-39 and 43, wherein images of two different modalities are first captured); and
aligning the first image of the first modality and the second image of the second modality (Jesudason, Fig. 2B, element 214; paragraphs 46-48, 56, wherein the images are aligned/registered).
Regarding claim 3, the combination of Jesudason and Ishida disclose the computer-implemented method of claim 1, wherein the first modality is a bright-field imaging (Jesudason, paragraph 43, wherein a bright-field microscopy imaging modality maybe used instead of the mxIF or H&E modalities).
Regarding claim 4, the combination of Jesudason and Ishida disclose the computer-implemented method of claim 1, wherein the first modality is fluorescence imaging (Jesudason, paragraphs 35 and 43, wherein a fluorescence imaging modality (FISH) maybe used instead of the mxIF or H&E modalities).
Regarding claim 17, Jesudason discloses a non-transitory computer-readable storage device comprising computer-executable instructions that, when executed by a computer system, cause the computer system to perform operations (page 13, item “48”) comprising:
accessing a pair of images of a tissue sample, comprising a first image of a first modality and a second image of a second modality (Fig. 2B, element 210; paragraphs 43, 55 and 56, wherein first and second images tissue cells is accessed, the images being of different visualization modalities, such as mxIF and H&E);
generating a plurality of image patch pairs from the pair of images, wherein each image patch pair comprises a first image patch sampled from a first location in the first image and a second image patch sampled from a second location in the second image, wherein the first location in the first image corresponds to the second location in the second image (Fig. 2B, element 212; paragraph 56, wherein corresponding image regions/patches are generated from each of images);
determining a
determining an image distance metric between the first image patch and the second image patch of each image patch pair, the image distance metric indicating a matching level between the first image patch and the second image patch of each image patch pair (Fig. 2B, element 214; paragraphs 46-48, 56, wherein regions/patches in each of the images are matched and used to align the images); and
As noted by the double strike-throughs above, Jesudason does not disclose expressly determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric and generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs.
Ishida discloses a process for registering/aligning a pair of images that includes determining a plurality of translational alignment vectors corresponding to a plurality of image patch/region pairs by: determining an image distance metric between the first image patch and the second image patch of each image patch pair, the image distance metric indicating a matching level between the first image patch and the second image patch of each image patch pair; and determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric (Figs. 4A-4D; column 6, line 1 – column 7, line 26, wherein regions of interest (ROI) (i.e. image patches) are matched across the images and shift vectors are determined by optimizing (i.e. minimizing) shift values (i.e. distance metric) associated with the ROI using cross-correlation); and
generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs (Fig. 4A, element 480, column 1, line 55 – column 2, line 63 and column 6, line 1 – column 7, line 26, wherein a subtraction image (i.e. image mask) for the pair of images is ultimately generated based on the shift vectors (i.e. translation alignment vectors) corresponding to each ROI).
Jesudason & Ishida are combinable because they are from the same art of image processing, specifically image alignment.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to incorporate the technique of determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric and generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs, as taught by Ishida, into the process of aligning image pairs disclosed by Jesudason.
The suggestion/motivation for doing so would have been to overcome severe misregistration errors due to inclination and/or rotation between a pair of images, as well as providing an image subtraction technique to aid in detecting of changes between images (Ishida, column 2, lines 21-27).
Therefore, it would have been obvious to combine Ishida with Jesudason to obtain the invention as specified in claim 17.
Regarding claim 19, Jesudason discloses a computer system, comprising: a memory configured to store computer-executable instructions; and a processor configured to access the memory and execute the computer- executable instructions to perform operations (page 13, item “47”) comprising:
accessing a pair of images of a tissue sample, comprising a first image of a first modality and a second image of a second modality (Fig. 2B, element 210; paragraphs 43, 55 and 56, wherein first and second images tissue cells is accessed, the images being of different visualization modalities, such as mxIF and H&E);
generating a plurality of image patch pairs from the pair of images, wherein each image patch pair comprises a first image patch sampled from a first location in the first image and a second image patch sampled from a second location in the second image, wherein the first location in the first image corresponds to the second location in the second image (Fig. 2B, element 212; paragraph 56, wherein corresponding image regions/patches are generated from each of images);
determining a
determining an image distance metric between the first image patch and the second image patch of each image patch pair, the image distance metric indicating a matching level between the first image patch and the second image patch of each image patch pair (Fig. 2B, element 214; paragraphs 46-48, 56, wherein regions/patches in each of the images are matched and used to align the images); and
As noted by the double strike-throughs above, Jesudason does not disclose expressly determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric and generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs.
Ishida discloses a process for registering/aligning a pair of images that includes determining a plurality of translational alignment vectors corresponding to a plurality of image patch/region pairs by: determining an image distance metric between the first image patch and the second image patch of each image patch pair, the image distance metric indicating a matching level between the first image patch and the second image patch of each image patch pair; and determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric (Figs. 4A-4D; column 6, line 1 – column 7, line 26, wherein regions of interest (ROI) (i.e. image patches) are matched across the images and shift vectors are determined by optimizing (i.e. minimizing) shift values (i.e. distance metric) associated with the ROI using cross-correlation); and
generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs (Fig. 4A, element 480, column 1, line 55 – column 2, line 63 and column 6, line 1 – column 7, line 26, wherein a subtraction image (i.e. image mask) for the pair of images is ultimately generated based on the shift vectors (i.e. translation alignment vectors) corresponding to each ROI).
Jesudason & Ishida are combinable because they are from the same art of image processing, specifically image alignment.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to incorporate the technique of determining a translational alignment vector for the first image patch in reference to the second image patch by minimizing the image distance metric and generating an image mask for the pair of images based on the plurality of translational alignment vectors corresponding to the plurality of image patch pairs, as taught by Ishida, into the process of aligning image pairs disclosed by Jesudason.
The suggestion/motivation for doing so would have been to overcome severe misregistration errors due to inclination and/or rotation between a pair of images, as well as providing an image subtraction technique to aid in detecting of changes between images (Ishida, column 2, lines 21-27).
Therefore, it would have been obvious to combine Ishida with Jesudason to obtain the invention as specified in claim 19.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over US 2025/0246010 to Jesudason et al. (“Jesudason”) in view of USPN 6,067,373 to Ishida et al. (“Ishida”) in further view of US 2017/0136948 to Sypitkowski et al. (“Sypitkowski”).
Regarding claim 6, the combination of Jesudason and Ishida discloses the computer implemented method of claim 1.
However, the combination does not disclose expressly converting the plurality of image patch pairs to a plurality of grayscale patch pairs, wherein a pixel intensity in each grayscale patch pair is between 0.0 and 1.0.
Sypitkowski discloses a process of aligning images that includes converting the plurality of image patches to a plurality of grayscale patches, wherein a pixel intensity in each grayscale patch pair is between 0.0 and 1.0 (Fig. 4 and paragraph 40, wherein conversion to grayscale is performed and the intensity values on a scale of 0-1 is common practice in the art).
Jesudason, Ishida & Sypitkowski are combinable because they are from the same art of image processing, specifically alignment.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to incorporate the technique of converting image regions/patches to grayscale, as taught by Sypitkowski, into the process for aligning images disclosed by the combination of Jesudason and Ishida.
The suggestion/motivation for doing so would have been to reduce computation time (Sypitkowski, paragraph 40).
Therefore, it would have been obvious to combine Sypitkowski with Jesudason and Ishida to obtain the invention as specified in claim 6.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over US 2025/0246010 to Jesudason et al. (“Jesudason”) in view of USPN 6,067,373 to Ishida et al. (“Ishida”) in further view of the article “Beyond Mutual Information: a simple and robust alternative” to Haber et al. (“Haber”).
Regarding claim 7, the combination of Jesudason and Ishida discloses the computer implemented method of claim 1.
However, the combination does not disclose expressly wherein the image distance metric is determined using a normalized gradient field technique.
Haber discloses a process for determining the distance between images (i.e. image distance metric) of different modalities that uses a normalized gradient field technique (section 3).
Jesudason, Ishida and Haber are combinable because they are from the same art of image processing.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to incorporate the technique of determining an image distance metric between images using a normalized gradient field technique, as taught by Haber, into the process for determining matches/similarities between images as disclosed by the combination of Jesudason and Ishida.
The suggestion/motivation for doing so would have been to provide a normalized gradient based approach that is deterministic, much simpler, easier to interpret, fast and straightforward to implement, faster to compute, and also much more suitable to optimization (Haber, Abstract).
Therefore, it would have been obvious to combine Haber with Jesudason and Ishida to obtain the invention as specified in claim 7.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over US 2025/0246010 to Jesudason et al. (“Jesudason”) in view of USPN 6,067,373 to Ishida et al. (“Ishida”) in further view of the article “Normalized Total Gradient: A New Measure for Multispectral Image Registration” to Chen et al. (“Chen”).
Regarding claim 8, the combination of Jesudason and Ishida discloses the computer implemented method of claim 1.
However, the combination does not disclose expressly wherein the image distance metric is determined using a normalized total gradient technique.
Chen discloses a process for determining the distance/similarity between images (i.e. image distance metric) of different modalities that uses a normalized total gradient technique (Abstract and Introduction, paragraphs 4-8).
Jesudason, Ishida and Chen are combinable because they are from the same art of image processing.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to incorporate the technique of determining an image distance metric between images using a normalized total gradient technique, as taught by Chen, into the process for determining matches/similarities between images as disclosed by the combination of Jesudason and Ishida.
The suggestion/motivation for doing so would have been provide a unimodal/multimodal image registration technique that is better both quantitively and qualitatively (Chen, Abstract).
Therefore, it would have been obvious to combine Chen with Jesudason and Ishida to obtain the invention as specified in claim 8.
Claims 13 and 18 is rejected under 35 U.S.C. 103 as being unpatentable over US 2025/0246010 to Jesudason et al. (“Jesudason”) in view of USPN 6,067,373 to Ishida et al. (“Ishida”) in further view of US 2023/0316458 to Ren et al. (“Ren”).
Regarding claim 8, the combination of Jesudason and Ishida discloses the computer implemented method of claim 1.
However, the combination does not disclose expressly wherein the image mask is a binary mask comprising a plurality of binary values corresponding to a plurality of locations of the plurality of image patch pairs.
Ren discloses a process for aligning images in which a binary mask is generated corresponding to locations of a plurality of image patches/regions in overlapping images (Fig. 7A and paragraph 148).
Jesudason, Ishida and Ren are combinable because they are from the same art of image processing.
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to incorporate the technique of generating a binary mask corresponding to a plurality of locations of the plurality of image patches, as taught by Ren, into the process for determining matches/similarities between images as disclosed by the combination of Jesudason and Ishida.
The suggestion/motivation for doing so would have been to provide prioritized objects in the images (Ren, paragraph 148).
Therefore, it would have been obvious to combine Ren with Jesudason and Ishida to obtain the invention as specified in claim 13.
Claim 18 recites the same limitations to those of claim 13 and is therefore rejected for the same reasoning indicated above with regard to 13.
Allowable Subject Matter
Claim 5 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON W CARTER whose telephone number is (571)272-7445. The examiner can normally be reached 8am - 5pm (Mon - Fri).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON W CARTER/Primary Examiner, Art Unit 2661
Prosecution Insights