Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-16 are rejected under 35 U.S.C. 103 as being unpatentable over HORSTMEYER et al (US 20230070475) in view of MICHAILOVICH (Image stitching by means of adaptive normalization).
As per claim 1, Horstmeyer teaches the claimed “apparatus” comprising: “an acquiring unit which acquires a plurality of images captured by changing a position of a focal plane relative to a subject in an image-capturing direction” (Horstmeyer, [0132] - In FIG. 6B, the cameras are controlled to be focused on different focus depth planes 653. The changes of focus depth planes can be performed by a moving mechanism moving the camera array 675. Alternatively, the changes of focus depth planes can be performed by another moving mechanism moving the sample (not shown). Similar processes can be repeated to form multiple image representations 651, each at a different focus depth plane 653. The multiple image representations can be stacked to form a focus stack 654 of image representation); “a selecting unit which selects, out of the plurality of images, two or more images which have higher contrast than a reference value” (Horstmeyer, [0145] - For a total of N focus depth planes, there are N composite image representations, if the images captured by the cameras are stitched together; [0300] - From this multiple image representations, software can also be utilized to produce an all-in-focus image. In some embodiments, an all-in-focus image is formed by using a sharpness metric to identify image locations that are in-focus within each stitched composite image representation, repeating this process for all stitched composite image representations within the multiple stitched image representations, and then merging the identified image locations that are in-focus into the final all-in-focus image); and “a composing unit which composes the two or more images to generate a composite image” (Horstmeyer, [0010] - In some embodiments, the formation of the each image representation comprises forming an image having higher contrast or resolution than an image of the multiple images. The formation of the each image representation comprises determining lateral and depth information of objects in the sample at the focus plane. The formation of each image representation comprises determining lateral locations of the objects in the sample and thickness of the objects at the lateral locations at the focus plane. The formation of the each image representation comprises determining complex values representing the objects in the sample at the focus plane, with magnitude components of the complex values showing lateral locations of the objects and phase components of the complex values showing thicknesses of the objects; [0300] - From this multiple image representations, software can also be utilized to produce an all-in-focus image. In some embodiments, an all-in-focus image is formed by using a sharpness metric to identify image locations that are in-focus within each stitched composite image representation, repeating this process for all stitched composite image representations within the multiple stitched image representations, and then merging the identified image locations that are in-focus into the final all-in-focus image) (see also Michailovich, Introduction - the process of combining together the independently acquired fragments of a digital image to form a complete representation of the entire scene is known as image stitching or mosaicking). Thus, it would have been obvious, in view of Michailovich, to configure Horstmeyer’s apparatus as claimed by combining images of different focused regions to yield a represented image of all focused objects. The motivation is to enhance the visualization of images captured in different focus planes.
Claim 2 adds into claim 1 “a sub-image generating unit which generates, from each of the plurality of images, a sub-image of at least one position in the image which is a common position among the plurality of images, wherein the sub-image includes a plurality of sub-images, the selecting unit selects, out of the plurality of sub-images for each position in the image, two or more sub-images which have higher contrast than the reference value, to select, out of the plurality of images, the two or more images which have higher contrast than the reference value, and the composing unit composes the two or more sub-images for each position in the image to generate a composite sub-image, to compose the two or more images to generate the composite image” (Horstmeyer, [0300] - From this multiple image representations, software can also be utilized to produce an all-in-focus image. In some embodiments, an all-in-focus image is formed by using a sharpness metric to identify image locations that are in-focus within each stitched composite image representation, repeating this process for all stitched composite image representations within the multiple stitched image representations, and then merging the identified image locations that are in-focus into the final all-in-focus image; Figure 2A – different objects (e.g., 250) having different depths are captured by different focus planes and combined to a representing image having all objects 250 being focused).
Claim 3 adds into claim 2 “a size detecting unit which detects a size of the subject in the image, wherein the sub-image generating unit generates each sub-image in the size detected by the size detecting unit” (Horstmeyer, [0186] - FIGS. 14A-14D illustrate configurations for volumetric representations according to some embodiments. FIG. 14A shows a volumetric representation 1452, which describes a 3 dimensional volume of a sample including any organisms, e.g., the locations, lateral sizes and thickness of the organisms. FIG. 14B shows a volumetric representation having multiple 3D image layers 1455, with each 3D layer showing lateral locations, sizes and thicknesses of the organisms. FIG. 14C shows a volumetric representation having multiple 2D image layers 1428, e.g., images 1428, with each image showing lateral locations and sizes of the organisms in higher contrast or resolution due to the multiple illumination patterns, as compared to images captured under a single illumination pattern).
Claim 4 adds into claim 2 “a position detecting unit which detects a position of the subject in the image, wherein the sub-image generating unit generates each sub-image at the position detected by the position detecting unit” (Horstmeyer, [0010] - In some embodiments, the formation of the each image representation comprises forming an image having higher contrast or resolution than an image of the multiple images. The formation of the each image representation comprises determining lateral and depth information of objects in the sample at the focus plane. The formation of each image representation comprises determining lateral locations of the objects in the sample and thickness of the objects at the lateral locations at the focus plane. The formation of the each image representation comprises determining complex values representing the objects in the sample at the focus plane, with magnitude components of the complex values showing lateral locations of the objects and phase components of the complex values showing thicknesses of the objects; [0186] - FIG. 14C shows a volumetric representation having multiple 2D image layers 1428, e.g., images 1428, with each image showing lateral locations and sizes of the organisms in higher contrast or resolution due to the multiple illumination patterns, as compared to images captured under a single illumination pattern).
Claim 5 adds into claim 1 “wherein the sub-image generating unit generates, from each of the plurality of images, a sub-image for each of the plurality of positions in the image, and the selecting unit, for each of the plurality of positions in the image, selects two sub-images which have higher contrast than the reference value and which are captured by using, as focal planes, two positions which are spaced apart in the image-capturing direction by a predetermined common interval” (Horstmeyer, [0010] - In some embodiments, the formation of the each image representation comprises forming an image having higher contrast or resolution than an image of the multiple images. The formation of the each image representation comprises determining lateral and depth information of objects in the sample at the focus plane. The formation of each image representation comprises determining lateral locations of the objects in the sample and thickness of the objects at the lateral locations at the focus plane. The formation of the each image representation comprises determining complex values representing the objects in the sample at the focus plane, with magnitude components of the complex values showing lateral locations of the objects and phase components of the complex values showing thicknesses of the objects; [0186] - FIG. 14C shows a volumetric representation having multiple 2D image layers 1428, e.g., images 1428, with each image showing lateral locations and sizes of the organisms in higher contrast or resolution due to the multiple illumination patterns, as compared to images captured under a single illumination pattern; [0300] - From this multiple image representations, software can also be utilized to produce an all-in-focus image. In some embodiments, an all-in-focus image is formed by using a sharpness metric to identify image locations that are in-focus within each stitched composite image representation, repeating this process for all stitched composite image representations within the multiple stitched image representations, and then merging the identified image locations that are in-focus into the final all-in-focus image; Figure 2A – different objects (e.g., 250) having different depths are captured by different focus planes and combined to a representing image having all objects 250 being focused).
Claim 6 adds into claim 2 “wherein the sub-image generating unit generates, from each of the plurality of images, a sub-image for each of the plurality of positions in the image, and the composing unit merges separate sub-images for separate positions in the image and which include the composite sub-image” (Horstmeyer, [0010] - In some embodiments, the formation of the each image representation comprises forming an image having higher contrast or resolution than an image of the multiple images. The formation of the each image representation comprises determining lateral and depth information of objects in the sample at the focus plane. The formation of each image representation comprises determining lateral locations of the objects in the sample and thickness of the objects at the lateral locations at the focus plane. The formation of the each image representation comprises determining complex values representing the objects in the sample at the focus plane, with magnitude components of the complex values showing lateral locations of the objects and phase components of the complex values showing thicknesses of the objects; [0186] - FIG. 14C shows a volumetric representation having multiple 2D image layers 1428, e.g., images 1428, with each image showing lateral locations and sizes of the organisms in higher contrast or resolution due to the multiple illumination patterns, as compared to images captured under a single illumination pattern; [0300] - From this multiple image representations, software can also be utilized to produce an all-in-focus image. In some embodiments, an all-in-focus image is formed by using a sharpness metric to identify image locations that are in-focus within each stitched composite image representation, repeating this process for all stitched composite image representations within the multiple stitched image representations, and then merging the identified image locations that are in-focus into the final all-in-focus image; Figure 2A – different objects (e.g., 250) having different depths are captured by different focus planes and combined to a representing image having all objects 250 being focused), and also adjusts brightness between the separate sub-images” which would have been obvious when combining images of different contrast and brightness (e.g., Michailovich, Figure 2 - Stitching by feathering in the original (left) and normalized (right) domains). Thus, it would have been obvious, in view of Michailovich, to configure Horstmeyer’s apparatus as claimed by performing the brightness adjustment during a process of combining images of different focused regions to yield a represented image of all focused objects. The motivation is to enhance the visualization of images captured in different focus planes by reducing brightness variability artifacts, which cause the average image brightness to vary from sub-image to sub-image.
Claim 7 adds into claim 6 “wherein in response to any of a plurality of sub-images generated by the sub-image generating unit for a same position in the image having equal or lower contrast than the reference value, the selecting unit selects a single sub-image out of the plurality of sub-images for the same position, and the composing unit merges separate sub-images including the single sub-image as separate sub-images and sub-images for the same position” which would have been obvious in view of Horstmeyer’s objects in different depths (i.e., each focused sub-image associated with its corresponding focus plane) being captured and combined (Horstmeyer, [0300] - From this multiple image representations, software can also be utilized to produce an all-in-focus image. In some embodiments, an all-in-focus image is formed by using a sharpness metric to identify image locations that are in-focus within each stitched composite image representation, repeating this process for all stitched composite image representations within the multiple stitched image representations, and then merging the identified image locations that are in-focus into the final all-in-focus image; Figure 2A – different objects (e.g., 250) having different depths are captured by different focus planes and combined to a representing image having all objects 250 being focused).
Claim 8 adds into claim 2 “wherein the acquiring unit acquires the plurality of images captured with the plurality of subjects arranged at different positions from each other in the image-capturing direction” (Horstmeyer, [0010] - In some embodiments, the formation of the each image representation comprises forming an image having higher contrast or resolution than an image of the multiple images. The formation of the each image representation comprises determining lateral and depth information of objects in the sample at the focus plane. The formation of each image representation comprises determining lateral locations of the objects in the sample and thickness of the objects at the lateral locations at the focus plane. The formation of the each image representation comprises determining complex values representing the objects in the sample at the focus plane, with magnitude components of the complex values showing lateral locations of the objects and phase components of the complex values showing thicknesses of the objects; Figure 2A – different objects (e.g., 250) having different depths are captured by different focus planes and combined to a representing image having all objects 250 being focused).
Claim 9 adds into claim 1 “an image-capturing unit which captures the plurality of images, wherein the acquiring unit acquires the plurality of images from the image-capturing unit” (Horstmeyer, [0010] - In some embodiments, the formation of the each image representation comprises forming an image having higher contrast or resolution than an image of the multiple images. The formation of each image representation comprises determining lateral and depth information of objects in the sample at the focus plane. The formation of each image representation comprises determining lateral locations of the objects in the sample and thickness of the objects at the lateral locations at the focus plane. The formation of each image representation comprises determining complex values representing the objects in the sample at the focus plane, with magnitude components of the complex values showing lateral locations of the objects and phase components of the complex values showing thicknesses of the objects).
Claim 10 adds into claim 9 “an image-capturing control unit which changes a position of a focal plane relative to the subject in the image-capturing direction, and causes the image-capturing unit to capture the plurality of images” (Horstmeyer, [0132] - In FIG. 6B, the cameras are controlled to be focused on different focus depth planes 653. The changes of focus depth planes can be performed by a moving mechanism moving the camera array 675. Alternatively, the changes of focus depth planes can be performed by another moving mechanism moving the sample (not shown). Similar processes can be repeated to form multiple image representations 651, each at a different focus depth plane 653. The multiple image representations can be stacked to form a focus stack 654 of image representation).
Claim 11 adds into claim 10 “wherein the image-capturing control unit, in response to a not-in-focus image being captured, causes the image-capturing unit to capture the plurality of images once again with narrower intervals between adjacent focal planes in the image-capturing direction” (Horstmeyer, [0132] - In FIG. 6B, the cameras are controlled to be focused on different focus depth planes 653. The changes of focus depth planes can be performed by a moving mechanism moving the camera array 675. Alternatively, the changes of focus depth planes can be performed by another moving mechanism moving the sample (not shown). Similar processes can be repeated to form multiple image representations 651, each at a different focus depth plane 653. The multiple image representations can be stacked to form a focus stack 654 of image representation).
Claim 12 adds into claim 1 “wherein the subject is a cell or a microorganism” (Horstmeyer, [0067] - The material of interest can include cellular matter, stem cells, human-derived cells, organism-derived cells, plant-derived cells, small model organisms, other vertebrate or invertebrate organisms, plant material, plant cells, biochemical suspensions, organoid tissue, organoids, cellular structures, in vitro cells, in vitro tissue, tissue, cytopathology material, bacteria, bacterial colonies, protein crystals, other proteins, viruses, DNA, RNA, or such proteins, viruses, DNA, RNA attached to microbeads or microspheres).
Claim 13 adds into claim 12 “wherein the subject is adhered to a flexible and translucent sheet-shaped member” which would have been obvious in Horstmeyer’s teaching of the microscope viewing image organoid specimens (Horstmeyer, [0062] - These are small clusters of cells, typically grown from human cells extracted within clinical applications, which is of particular interest to be observed at high resolution. Since the cells are distributed in 3 dimensions within different areas of a well plate that is multiple mm across and multiple mm deep, the present microscope utilizing a combination of 3D high resolution thin specimen imaging together with the focal stack imaging for extending the high resolution in the axial dimension can be an effective microscopic imaging technology available to capture the desired information).
Claim 14 adds into claim 12 “wherein the subject is adhered to an interior of a culture bag” which would have been obvious in Horstmeyer’s teaching of the microscope viewing image organoid specimens (Horstmeyer, [0062] - These are small clusters of cells, typically grown from human cells extracted within clinical applications, which is of particular interest to be observed at high resolution. Since the cells are distributed in 3 dimensions within different areas of a well plate that is multiple mm across and multiple mm deep, the present microscope utilizing a combination of 3D high resolution thin specimen imaging together with the focal stack imaging for extending the high resolution in the axial dimension can be an effective microscopic imaging technology available to capture the desired information).
Claims 15 and 16 claim a method and a non-statutory computer readable medium based on the apparatus of claims 1-14; therefore, they are rejected under a similar rationale.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/Primary Examiner, Art Unit 2616
Prosecution Insights