DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/22/2025 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 16, 31 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 16, 18-33 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2020/0074669 by Okabe et al. in view of US 2011/0267267 by Hasegawa et al.
Regarding claim 16, Okabe et al. discloses a system for a microscope system, the system comprising one or more processors and one or more storage devices (fig. 1), wherein the system is configured to:
obtain imaging sensor data from at least one optical imaging sensor of a microscope of the microscope system (paragraph 0045 teaches “The microscope 700 includes a first microscope 710 and a second microscope 720 that capture images of a specimen based on mutually different microscopies.”);
generate a plurality of different preview images based on the imaging sensor data, the plurality of different preview images being continuously updated (in addition to discussion above, fig. 18, paragraph 0118 teaches “In addition, in the aforementioned example, as the microscope image group 450, the first microscope image 401 and the second microscope image 402 have been acquired from an image acquired by observing one specimen in chronological order.”, paragraph 0149 teaches “In a case where the plurality of the second magnified images 412 are sequentially displayed, the image generator 160 identifies a start position of scanning, a scanning direction, and an end position of scanning, based on the input from the user or the default setting, and sequentially displays the second magnified image 412 of a well corresponding to the scanning position”);
obtain a control input signal from an input device of the microscope system (in addition to discussion above, paragraph 0136 teaches “For example, a menu enumerating types of image processing and values of parameters as options may be displayed, and the contents instructed by the menu may be selected. Further, for example, in a case of designating the display region of the magnified image, an actual size of the specimen may be designated by a numerical value, or may be designated by the number of pixels on the display 170, the magnification relative to the size of the original region, or the like. Alternatively, the display condition may be selected from a plurality of first magnified images 411 having different sizes which are displayed.”);
generate a display signal for a display device of the microscope system, the display signal comprising showing two or more preview images of the plurality of different preview images in smaller size windows and a further image in a larger size window (in addition to discussion above, paragraph 0114 teaches “As a result, the first magnified image 411 and the second magnified image 412, which are partial magnified images of the first microscope image 401 and the second microscope image 402, which are selected from the microscope image group 450 generated by the time-lapse imaging at different times, can be displayed on the display 170 and performed comparative observation.”, paragraph 0054 teaches “Thus, in the first microscope image 401 and the second microscope image 402, the sizes, slopes, and positions of the specimen images 501 and 502 in each image are different. Thus, even if the first microscope image 401 and the second microscope image 402 are displayed side by side on the display 170, it is difficult for the user to perform comparative observation in both images.”, paragraph 0052); and
control based on the input signal such that the further image is populated based on a selection of one of the two or more different preview images via the user interface (in addition to discussion above, paragraph 0127 teaches “The image processor 102 is different from the image processor 101 in that the image processor 102 includes a processing accepting unit 151 that accepts a display condition input from the input device 130 in addition to the accepting unit 150 that accepts the position designated by the input device 130. The display condition is a condition in which the first magnified image 411 once generated is image processed to change the display on the display 170. The display condition includes a range magnified by the first magnified image 411, an image magnification, an image brightness, an image contrast, an image rotation, and an image reversal.”, paragraph 0142 teaches “In this example, when performing comparative observation of the first magnified image 411 and the second magnified image 412, the user can instruct the process accepting unit 151 for the scanning mode. In the scanning mode, a starting position of scanning, a scanning direction and an end position of scanning are set by default, but they may be designated by the user via the input device 130.”, paragraph 0108-0109 teaches “In step S102 of FIG. 13, the image associating unit 140 accepts designation of the first microscope image 401 and the second microscope image 402 acquired from different time ranges in the microscope image group 450 via the input device 130. That is, in this example, as the first microscope image 401 and the second microscope image 402, microscope images captured at mutually different times are identified.”, paragraph 0046-0047 teaches “The first microscope 710 captures images of the specimen by Structured Illumination Microscopy (SIM). The SIM illuminates the specimen with a structured illumination having a periodic illumination pattern and generates a super-resolution microscope image representing a microstructure of the specimen smaller than a wavelength of the illumination light, based on interference fringes generated in the image of the specimen…..On the other hand, the second microscope 720 captures images of the specimen by STochastic Optical Reconstruction Microscopy (STORM). STORM can reconstruct a fluorescent image (super-resolution microscope image) with a resolution higher than the illumination light wavelength by overlaying position information of fluorescent dyes detected with high precision from a plurality of fluorescent images.”).
Okabe et al. fails to disclose the display signal comprising a user interface showing two or more preview images of the plurality of different preview images in smaller size windows and a further image in a larger size window; control the user interface based on the input signal such that the further image is populated based on a selection of one of the two or more different preview images via the user interface; wherein the plurality of preview images differ with regards to at least one of an imaging mode being used for generating a respective preview image, a frequency spectrum being used for generating the respective preview image, an illumination being used for generating the respective preview image, or post-processing being applied to generate the respective preview image.
Hasegawa et al. discloses the display signal comprising a user interface showing two or more preview images of the plurality of different preview images in smaller size windows and a further image in a larger size window; control the user interface based on the input signal such that the further image is populated based on a selection of one of the two or more different preview images via the user interface; wherein the plurality of preview images differ with regards to at least one of an imaging mode being used for generating a respective preview image, a frequency spectrum being used for generating the respective preview image, an illumination being used for generating the respective preview image, or post-processing being applied to generate the respective preview image (paragraph 0052 teaches “An image pyramid structure 50 in this embodiment is an image group (whole image group) generated for the same image obtained from a single observation target 15 (see FIG. 4) by an optical microscope in different resolutions.”, paragraph 0055 teaches “First, a digital image of the original image obtained by an optical microscope (not shown) at a predetermined observation magnification is prepared. This original image corresponds to the image having the largest size, which is the lowermost image of the image pyramid structure 50 shown in FIG. 3. In other words, the original image is an image having the highest resolution. Therefore, as the lowermost image of the image pyramid structure 50, an image that is observed at a relatively high magnification and then obtained by the optical microscope is used.”, paragraph 0058 teaches “The whole image group forming the image pyramid structure 50 may be generated by a known compression method, or generated by a known compression method used when a thumbnail image is generated, for example.”, paragraph 0059 teaches “Specifically, the PC 100 displays an image of any part selected by the user from an image having any resolution selected by the user. By such processing, the user can obtain a feeling of observing an observation target 15 while changing an observation magnification. In other words, the PC 100 functions as a virtual microscope. The virtual observation magnification used here corresponds to the resolution in actuality.”, paragraph 0066 teaches “Displayed in the area A is a pathological image 203 corresponding to a thumbnail image 201 selected by a user via the mouse 10 from the plurality of thumbnail images 201 arranged in the area B.”).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the ability to include the plurality of preview images differ, as taught by Hasegawa et al. into the system of Okabe et al., because such incorporation would allow more options to a user to have variety of preview, thus increase user flexibility of the system.
Regarding claim 18, the system wherein the plurality of preview images comprise at least one preview image that is based on reflectance imaging and at least one preview image that is based on fluorescence imaging (in addition to discussion above, Okabe et al., paragraph 0047 teaches “On the other hand, the second microscope 720 captures images of the specimen by STochastic Optical Reconstruction Microscopy (STORM). STORM can reconstruct a fluorescent image (super-resolution microscope image) with a resolution higher than the illumination light wavelength by overlaying position information of fluorescent dyes detected with high precision from a plurality of fluorescent images..”, paragraph 0168).
Regarding claim 19, the system wherein the plurality of preview images comprise a preview image with a first lower image contrast and a further preview image with a second higher image contrast (in addition to discussion above, Okabe et al., paragraph 0095 teaches “Here, the setting conditions include image-forming magnification of the microscope, stage coordinates, and the like. The image associating unit 140 may associate the microscope images with each other by using both the image-forming magnification and the stage coordinates, or may associate by using either the image-forming magnification or the stage coordinates.”, paragraph 0098 teaches “the method of associating the microscope images using the aforementioned setting conditions of the microscope at the time of imaging the microscope images, the image generator 160 identifies the second magnified region in step S106 in accordance with the identification of the first magnified region in step S105, based on the size, the orientation and the position of each image identified in the association between the first microscope image 401 and the second microscope image 402”, paragraph 0127 teaches “The display condition is a condition in which the first magnified image 411 once generated is image processed to change the display on the display 170. The display condition includes a range magnified by the first magnified image 411, an image magnification, an image brightness, an image contrast, an image rotation, and an image reversal.”, paragraph 0168).
Regarding claim 20, the system wherein the plurality of preview images comprise one or more of at least one preview image that is based on high-dynamic-range imaging, at least one preview image in which reflections have been reduced compared to an unprocessed version of the imaging sensor data, and at least one preview image that is based on multi-spectral imaging (as discussed above, Okabe et al., paragraph 0045-0046).
Regarding claim 21, the system wherein the plurality of preview images comprise at least one preview image being generated based on a user-specified setting, wherein the system is configured to generate the user interface with means for defining the user- specified setting (in addition to discussion above, Okabe et al., paragraph 0067-0068 teaches “In this case, the image generator 160 displays a cursor 530 on the first microscope image 401 as illustrated in FIG. 9 by input from the user by the input device 130, for example, via a movement of the mouse. Furthermore, the accepting unit 150 accepts the position of a point designated by operation from the user of depressing a mouse button as position information (for example, coordinate of a point).”, paragraph 0168).
Regarding claim 22, the system wherein the system is configured to obtain additional imaging sensor data from an external device being coupled with the microscope system, the system being configured to generate the plurality of preview images further based on the additional imaging sensor data (in addition to discussion above, Okabe et al.paragraph 0045 teaches “The microscope 700 includes a first microscope 710 and a second microscope 720 that capture images of a specimen based on mutually different microscopies. The first microscope 710 and the second microscope 720 share at least part of optical systems in both microscopes (that is, at least part of optical axes in both microscopes are coaxial). Note that, the optical systems in both the first microscope 710 and the second microscope 720 may be independent optical systems.”).
Regarding claim 23, the system wherein the system is configured to select the two or more of the plurality of different preview images being initially shown in the user interface based on information on preview images being preferred by a user of the microscope system (in addition to discussion above, Okabe et al., paragraph 0108-0109 teaches “In step S102 of FIG. 13, the image associating unit 140 accepts designation of the first microscope image 401 and the second microscope image 402 acquired from different time ranges in the microscope image group 450 via the input device 130. That is, in this example, as the first microscope image 401 and the second microscope image 402, microscope images captured at mutually different times are identified.”, paragraph 0114 teaches “As a result, the first magnified image 411 and the second magnified image 412, which are partial magnified images of the first microscope image 401 and the second microscope image 402, which are selected from the microscope image group 450 generated by the time-lapse imaging at different times, can be displayed on the display 170 and performed comparative observation.”).
Regarding claim 24, the system wherein the system is configured to include a subset of the plurality of preview images in the user interface, the system being configured to forego generating the preview images not shown in the user interface (in addition to discussion above, Okabe et al., fig. 18, paragraph 0118 teaches “In addition, in the aforementioned example, as the microscope image group 450, the first microscope image 401 and the second microscope image 402 have been acquired from an image acquired by observing one specimen in chronological order.”, paragraph 0149 teaches “In a case where the plurality of the second magnified images 412 are sequentially displayed, the image generator 160 identifies a start position of scanning, a scanning direction, and an end position of scanning, based on the input from the user or the default setting, and sequentially displays the second magnified image 412 of a well corresponding to the scanning position”, paragraph 0115, 0143-0144, 0149).
Regarding claim 25, the system wherein the plurality of preview images have a lower resolution and/or a lower frame rate than the optical imaging data (in addition to discussion above, Hasegawa et al., paragraph 0057 teaches “As shown in FIG. 4, the scanner apparatus or a general-purpose computer (not shown) generates a plurality of images having resolutions reduced stepwise, from the image having the largest size obtained as described above. Then, the scanner apparatus or the general-purpose computer stores those images in units of "tiles" of a predetermined size, for example. The size of one tile is, for example, 256.times.256 (pixels). The image group thus generated forms the image pyramid structure 50, and the image pyramid structure 50 is stored in the storage 108 of the PC 100. In reality, the PC 100 may only have to store those images having different resolutions and resolution information items in association with each other. It should be noted that the PC 100 shown in FIG. 1 may generate and store the image pyramid structure 50.”, paragraph 0059 teaches “Specifically, the PC 100 displays an image of any part selected by the user from an image having any resolution selected by the user. By such processing, the user can obtain a feeling of observing an observation target 15 while changing an observation magnification. In other words, the PC 100 functions as a virtual microscope. The virtual observation magnification used here corresponds to the resolution in actuality.”).
Regarding claim 26, the system wherein the plurality of preview images have a lower resolution and/or a lower frame rate than the further image (in addition to discussion above, Hasegawa et al., paragraph 0057 teaches “As shown in FIG. 4, the scanner apparatus or a general-purpose computer (not shown) generates a plurality of images having resolutions reduced stepwise, from the image having the largest size obtained as described above. Then, the scanner apparatus or the general-purpose computer stores those images in units of "tiles" of a predetermined size, for example. The size of one tile is, for example, 256.times.256 (pixels). The image group thus generated forms the image pyramid structure 50, and the image pyramid structure 50 is stored in the storage 108 of the PC 100. In reality, the PC 100 may only have to store those images having different resolutions and resolution information items in association with each other. It should be noted that the PC 100 shown in FIG. 1 may generate and store the image pyramid structure 50.”, paragraph 0059 teaches “Specifically, the PC 100 displays an image of any part selected by the user from an image having any resolution selected by the user. By such processing, the user can obtain a feeling of observing an observation target 15 while changing an observation magnification. In other words, the PC 100 functions as a virtual microscope. The virtual observation magnification used here corresponds to the resolution in actuality.”).
The motivation for combining references has been discussed in independent claim above.
Regarding claim 27, the system wherein the control input signal is obtained from a touch interface of a touch screen of the microscope system, the display signal being generated for a display of the touch screen (in addition to discussion above, Okabe et al., paragraph 0068 teaches “Note that instead of pressing the mouse button, the accepting unit 150 may accept designation of a point by pressing a return key, touching the display 170 in a case where the display 170 is a touch panel, or the like.”).
The motivation for combining references has been discussed in independent claim above.
Regarding claim 28, the system wherein the control input signal is obtained from one or more input devices arranged at one or more handles of the microscope system (in addition to discussion above, Okabe et al., paragraph 0042 teaches “The input device 130 is operated in a case where an instruction from a user is input to the image processor 101. As the input device 130, an existing general-purpose input device such as a pointing device such as a mouse, a keyboard, or a touch panel may be used. Although the input device 130 is for input to the image processor 101, it may also be provided for operating a first microscope 110 and a second microscope 120.”).
Regarding claim 29, the system wherein the system is configured to provide the display signal to one of a display of a touch screen of the microscope system, one or more ocular displays of the microscope system, and a head-mounted display of the microscope system (in addition to discussion above, Okabe et al., paragraph 0068 teaches “Note that instead of pressing the mouse button, the accepting unit 150 may accept designation of a point by pressing a return key, touching the display 170 in a case where the display 170 is a touch panel, or the like.”).
Regarding claim 30, a surgical microscope system comprising a microscope, a display device, a control input device and the system according to claim 16 (Okabe et al., fig. 1).
Claim 31 is rejected for the same reason as discussed in the corresponding claim 16 above.
Claim 32 is rejected for the same reason as discussed in the corresponding claim 22 above.
Claim 33 is rejected for the same reason as discussed in the corresponding claim 30 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIGAR CHOWDHURY whose telephone number is (571)272-8890. The examiner can normally be reached Monday-Friday 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thai Tran can be reached on 571-272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIGAR CHOWDHURY/Primary Examiner, Art Unit 2484