DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 2/20/2026 has been entered. Claims 1-2 and 4-20 remain pending the application.
Response to Arguments
Applicant's arguments filed on 2/20/2026 have been fully considered but they are moot.
The Applicant’s interview summary suggests that the Examiner recommended this amendment but the Examiner does not recall doing so and the Examiner’s interview summary does not reflect this statement. Nevertheless, the following rejection is presented.
Applicant argues on pages 7-9 that the previous art does not disclose the newly added limitations to the claims related to the application parameter including one or more medical/surgical operation related parameters. This argument is moot in view of the new grounds of rejection necessitated by amendment which relies on portions of Themelis to disclose these limitations in the claims as outlined in the rejection below. Specifically, the type of dye injected would fall under the broadest reasonable interpretation of medical/surgical operation related parameters because an injection of a type of dye is a medical/surgical operation related parameter. Accordingly, this argument is moot.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 5, 8-9, 11, 15-16, and 18-20 are rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Themelis (US20230137862).
Regarding claims 1 and 16, Themelis discloses a surgical microscope and a method for providing an image representation by a surgical microscope (Themelis, Para 2; “This disclosure relates to near-infrared (NIR) imaging used in endoscopic surgical systems”), comprising:
obtaining or capturing at least one application parameter, the at least one application parameter including one or more medical/surgical operation related parameters (Themelis, Para 51; “Accordingly, the system may be configured to determine a feature detected in the second component to fail the visibility criterion if a contrast or intensity of fluorescence imaging indicator highlighting the feature in the digital view relative to adjacent portions of the digital view is below a contrast threshold or intensity threshold, respectively. For example, the fluorescence imaging indicator may correspond to the pseudocolor representation of the second component, at the location of the feature of interest. The contrast may be determined by comparing the color values (e.g., the Red-Green-Blue (RGB) color values) of the pixels of the pseudocolor representation with the color values (e.g., RGB color values) of pixels of the color representation that are adjacent to the pixels of the pseudocolor representation. If the difference in color values is smaller than a threshold (in all color channels), the contrast may be deemed to fail the contrast threshold. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view).”) (Themelis, Para 49; “In addition to, or as an alternative to, the size and intensity of the fluorescence emissions, the shape of the fluorescence emissions may be of interest, too. For example, fluorescent dyes may be used to clearly identify aneurysms in vascular surgery. However, in this case, the aneurysm might not emit fluorescence emissions, in contrast to neighboring vessels. Therefore, in this case, the feature of interest, i.e., the aneurysm, may be detected due to a lack of fluorescence emissions compared to neighboring vessels or portions of the same vessel. For example, the system may be configured to use object detection to identify the shape of a blood vessel or network of blood vessels, and to detect a feature of interest if a portion of the blood vessel or network of blood vessels emits less (or none) fluorescence emissions compared to adjacent portions of the blood vessel or network of blood vessels. For example, the object detection may be applied on the second component, on the first component, or on a combination of the first and second component. In a more general manner, object detection may be used to distinguish features from features of interest. For example, the system may be configured to perform object detection based on the first and/or second component of the optical imaging data to determine whether a feature is a feature of interest. […] Based on whether the potential feature of interest also emits fluorescence emissions (i.e., based on the second component), it may be deemed to be a feature of interest.”) (Themelis, Para 39-40; “If a fluorescent dye is used in the surgical site, the emitted light excites the dye, and light is emitted in a second wavelength band (also called the fluorescence emission wavelength band). This light can be recorded by a camera sensor, such as the optical imaging sensor, and displayed on a display of the surgical microscope. Compared to reflectance imaging, in reflectance imaging, light that is emitted by the fluorescent dye has a low illumination intensity, and thus cannot be distinguished without digital processing if the surgical site is illuminated for reflectance imaging. […] To enable concurrent reflectance imaging and fluorescent imaging, the fluorescence emission wavelength band (or fluorescence emission wavelength bands, if multiple fluorescent dyes having different fluorescence emission wavelength bands are used) may be sensed separately by the at least one optical imaging sensor, and light in the fluorescence emission wavelength band(s) (and optionally in the fluorescence excitation wavelength band(s)) may be omitted or filtered out from the illumination being used for reflectance imaging.”),
capturing a color image representation of a capture region by a camera, the color image representation being an RGB image that includes signals from red, green, and blue color channels of the camera (Themelis, Para 40; “In some examples, different optical imaging sensors are used for fluorescence imaging and reflectance imaging. For example, the microscope may comprise a first optical imaging sensor being configured to perform reflectance imaging and a second optical imaging sensor being configured to perform fluorescence imaging.”) (Themelis, Para 42; “In general, the reflectance imaging may be used to generate a color image of the surgical site, while the fluorescence imaging may be used to generate an image of the surgical site that represents the intensity of the fluorescence emissions, e.g., as monochrome image. The digital view may be generated based on these images.”) (Themelis, Para 51; “The contrast may be determined by comparing the color values (e.g., the Red-Green-Blue (RGB) color values) of the pixels of the pseudocolor representation with the color values (e.g., RGB color values) of pixels of the color representation that are adjacent to the pixels of the pseudocolor representation. If the difference in color values is smaller than a threshold (in all color channels), the contrast may be deemed to fail the contrast threshold. Similarly, if the intensity of the fluorescence emissions, and thus also of the pseudocolor representation of the fluorescence emissions, is below the intensity threshold, the intensity of the pseudocolor representation (i.e., the fluorescence imaging indicator) may be deemed to fail the intensity threshold.”),
capturing a fluorescence image representation of the capture region by a fluorescence camera (Themelis, Para 40; “In some examples, different optical imaging sensors are used for fluorescence imaging and reflectance imaging. For example, the microscope may comprise a first optical imaging sensor being configured to perform reflectance imaging and a second optical imaging sensor being configured to perform fluorescence imaging.”) (Themelis, Para 42; “In general, the reflectance imaging may be used to generate a color image of the surgical site, while the fluorescence imaging may be used to generate an image of the surgical site that represents the intensity of the fluorescence emissions, e.g., as monochrome image. The digital view may be generated based on these images.”),
processing the fluorescence image representation by a processing device to optimize it for an overlay with the color image representation, wherein a type of processing steps and a processing sequence for the processing of the fluorescence image representation is defined based on the at least one application parameter (Themelis, Para 51-52; “Another potential visibility criteria are the contrast and the intensity of the representation of the feature of interest. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view). […] For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. For the third approach of making the surgeon aware that the feature of interest can be perceived by switching the digital view, an indicator, such as a textual indicator or symbol, for prompting a surgeon to switch the digital view may be used. In summary, the visual indicator may be one of a blinking indicator, an indicator having an increased contrast relative to adjacent portions of the digital view, an indicator drawing an outline around the feature of interest, and an indicator for prompting a surgeon to switch the digital view to show fluorescence emissions in isolation. It may be superimposed on the digital view (e.g., when using an outline or a blinking indicator), integrated as part of the digital view (e.g., when increasing the contrast of a portion of the digital view), or be shown next to the digital view (as prompt for switching the digital view).”) (Themelis, Para 47; describing this),
overlaying the color image representation with the processed fluorescence image representation, and providing an image signal that encodes the overlaid image representation (Themelis, Para 52; “Accordingly, the system may be configured to include the visual indicator in the display signal as one of an overlay (e.g., by superimposing the visual indicator on the digital view), an indicator included in a composite reflectance and fluorescence image (i.e., the third representation, e.g., by altering the composition of the composite reflectance and fluorescence image)”),
wherein the processing the fluorescence image representation comprises generating a detail image of the fluorescence image representation, wherein the generation of the detail image takes place taking into account the at least one application parameter, and wherein the overlay with the color image representation with the processed fluorescence image representation is carried out starting from the generated detail image (Themelis, Para 51-52; “Another potential visibility criteria are the contrast and the intensity of the representation of the feature of interest. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view). […] For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. For the third approach of making the surgeon aware that the feature of interest can be perceived by switching the digital view, an indicator, such as a textual indicator or symbol, for prompting a surgeon to switch the digital view may be used. In summary, the visual indicator may be one of a blinking indicator, an indicator having an increased contrast relative to adjacent portions of the digital view, an indicator drawing an outline around the feature of interest, and an indicator for prompting a surgeon to switch the digital view to show fluorescence emissions in isolation. It may be superimposed on the digital view (e.g., when using an outline or a blinking indicator), integrated as part of the digital view (e.g., when increasing the contrast of a portion of the digital view), or be shown next to the digital view (as prompt for switching the digital view).”).
Regarding claim 2, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis further discloses wherein the at least one application parameter is at least partially automatically recognized and/or defined and transferred to the processing device (Themelis, Para 8; “The system comprises one or more processors and one or more storage devices. The system is configured to obtain imaging sensor data from at least one optical imaging sensor of the microscope. The imaging sensor data comprises a first component that is based on reflectance imaging and a second component that is based on fluorescence imaging. The system is configured to generate a digital view of a surgical site based on at least the first component of the imaging sensor data. The system is configured to detect at least one feature of interest in the second component of the imaging sensor data. The system is configured to generate a visual indicator of the at least one feature of interest. The system is configured to provide a display signal to a display device of the surgical microscope system, the display signal comprising the digital view and the visual indicator. By detecting the feature of interest in the fluorescence imaging based data, the occurrence of a feature of interest that may or may not be apparent to the surgeon can be detected. By including a visual indicator, the surgeon may be made aware of the feature of interest, in particular when it is otherwise imperceptible or hard to perceive in the digital view.”) (Themelis, Para 73; “A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.”).
Regarding claim 5, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis further discloses wherein the generation of the detail image comprises recognizing and/or extracting structures in the fluorescence image representation, wherein the recognition and/or extraction of the structures takes place taking into account the at least one application parameter (Themelis, Para 49; “In addition to, or as an alternative to, the size and intensity of the fluorescence emissions, the shape of the fluorescence emissions may be of interest, too. For example, fluorescent dyes may be used to clearly identify aneurysms in vascular surgery. However, in this case, the aneurysm might not emit fluorescence emissions, in contrast to neighboring vessels. Therefore, in this case, the feature of interest, i.e., the aneurysm, may be detected due to a lack of fluorescence emissions compared to neighboring vessels or portions of the same vessel. For example, the system may be configured to use object detection to identify the shape of a blood vessel or network of blood vessels, and to detect a feature of interest if a portion of the blood vessel or network of blood vessels emits less (or none) fluorescence emissions compared to adjacent portions of the blood vessel or network of blood vessels. For example, the object detection may be applied on the second component, on the first component, or on a combination of the first and second component. In a more general manner, object detection may be used to distinguish features from features of interest. For example, the system may be configured to perform object detection based on the first and/or second component of the optical imaging data to determine whether a feature is a feature of interest. For example, the object detection may be based on image segmentation and image classification. For example, a machine-learning model may be trained to perform image segmentation and classification on the first component, e.g., to segment the first component into smaller image portions (image segmentation), with each image portion showing a feature, and with the image classification being used to classify the features shown in the smaller image portions, e.g., into different types of objects. Depending on the classification, a feature may be deemed to be a potential feature of interest. Based on whether the potential feature of interest also emits fluorescence emissions (i.e., based on the second component), it may be deemed to be a feature of interest.”).
Regarding claim 8, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis further discloses wherein the color image representation is processed by the processing device to optimize the overlay, wherein a type of the processing the color image representation is defined based on the at least one application parameter (Themelis, Para 47; “For example, the system may be configured to use object detection to identify the shape of a blood vessel or network of blood vessels, and to detect a feature of interest if a portion of the blood vessel or network of blood vessels emits less (or none) fluorescence emissions compared to adjacent portions of the blood vessel or network of blood vessels.”) (Themelis, Para 51-52; “Another potential visibility criteria are the contrast and the intensity of the representation of the feature of interest. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view). […] For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. For the third approach of making the surgeon aware that the feature of interest can be perceived by switching the digital view, an indicator, such as a textual indicator or symbol, for prompting a surgeon to switch the digital view may be used. In summary, the visual indicator may be one of a blinking indicator, an indicator having an increased contrast relative to adjacent portions of the digital view, an indicator drawing an outline around the feature of interest, and an indicator for prompting a surgeon to switch the digital view to show fluorescence emissions in isolation. It may be superimposed on the digital view (e.g., when using an outline or a blinking indicator), integrated as part of the digital view (e.g., when increasing the contrast of a portion of the digital view), or be shown next to the digital view (as prompt for switching the digital view).”).
Themelis is interpreted as disclosing these limitations in the claims because its description is similar that of the specification of the instant application such as in [0033-0034].
Regarding claim 9, Themelis is interpreted as disclosing all of the limitations of claim 8 as discussed above
Themelis further discloses wherein the processing the color image representation comprises generating a color detail image of the color image representation, wherein the generation of the color detail image takes place taking into account the at least one application parameter, and wherein the overlay with the fluorescence image representation is carried out based on the generated color detail image (Themelis, Para 47; “For example, the system may be configured to use object detection to identify the shape of a blood vessel or network of blood vessels, and to detect a feature of interest if a portion of the blood vessel or network of blood vessels emits less (or none) fluorescence emissions compared to adjacent portions of the blood vessel or network of blood vessels.”) (Themelis, Para 51-52; “Another potential visibility criteria are the contrast and the intensity of the representation of the feature of interest. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view). […] For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. For the third approach of making the surgeon aware that the feature of interest can be perceived by switching the digital view, an indicator, such as a textual indicator or symbol, for prompting a surgeon to switch the digital view may be used. In summary, the visual indicator may be one of a blinking indicator, an indicator having an increased contrast relative to adjacent portions of the digital view, an indicator drawing an outline around the feature of interest, and an indicator for prompting a surgeon to switch the digital view to show fluorescence emissions in isolation. It may be superimposed on the digital view (e.g., when using an outline or a blinking indicator), integrated as part of the digital view (e.g., when increasing the contrast of a portion of the digital view), or be shown next to the digital view (as prompt for switching the digital view).”).
Themelis is interpreted as disclosing these limitations in the claims because its description is similar that of the specification of the instant application such as in [0033-0034].
Regarding claim 11, Themelis is interpreted as disclosing all of the limitations of claim 8 as discussed above
Themelis further discloses wherein the processing the fluorescence image presentation taking into account the at least one application parameter comprises: a smoothing of the fluorescence image representation by a filter (Themelis, Para 40; “In some examples, wavelength bands are recorded separately by one or more optical imaging sensors (e.g., for the purposes of multispectral images, e.g., using filters that are placed in front of the optical imaging sensors), and one or more of the wavelength bands may be used for fluorescence imaging, e.g. by blocking light from the emission spectrum of the illumination system of the surgical microscope system that intersects with the fluorescence emission wavelength band.”), a contrast enhancement and/or a brightness compensation of the smoothed fluorescence image representation and a merging of the generated detail image with the fluorescence image representation thus processed, wherein the overlay with the color image representation is carried out based on the merged image representation (Themelis, Para 52; “For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. For the third approach of making the surgeon aware that the feature of interest can be perceived by switching the digital view, an indicator, such as a textual indicator or symbol, for prompting a surgeon to switch the digital view may be used. In summary, the visual indicator may be one of a blinking indicator, an indicator having an increased contrast relative to adjacent portions of the digital view, an indicator drawing an outline around the feature of interest, and an indicator for prompting a surgeon to switch the digital view to show fluorescence emissions in isolation. It may be superimposed on the digital view (e.g., when using an outline or a blinking indicator), integrated as part of the digital view (e.g., when increasing the contrast of a portion of the digital view), or be shown next to the digital view (as prompt for switching the digital view). Accordingly, the system may be configured to include the visual indicator in the display signal as one of an overlay (e.g., by superimposing the visual indicator on the digital view), an indicator included in a composite reflectance and fluorescence image (i.e., the third representation, e.g., by altering the composition of the composite reflectance and fluorescence image), and a warning image (to be shown superimposed on the digital view or next to the digital view).”).
Regarding claim 15, Themelis is interpreted as disclosing all of the limitations of claim 8 as discussed above
Themelis further discloses wherein the processing the fluorescence image representation comprises converting intensity values of the fluorescence image representation or of the processed fluorescence image representation or of the detail image into color values according to a color map specified by the at least one application parameter (Themelis, Para 51-52; “Another potential visibility criteria are the contrast and the intensity of the representation of the feature of interest. As outlined above, in the combined view (i.e., the third representation), a pseudocolor representation of the fluorescence emissions is overlaid over the color image of the reflectance image. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view). The purpose of the visual indicator is to make the surgeon aware of the feature of interest, e.g., by highlighting the feature of interest, by delineating the feature of interest from adjacent portions of the digital view, or by making the surgeon aware that the feature of interest can be perceived by switching the digital view, e.g., to the second or third representation. For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. […] Accordingly, the system may be configured to include the visual indicator in the display signal as one of an overlay (e.g., by superimposing the visual indicator on the digital view), an indicator included in a composite reflectance and fluorescence image (i.e., the third representation, e.g., by altering the composition of the composite reflectance and fluorescence image), and a warning image (to be shown superimposed on the digital view or next to the digital view).”).
Regarding claim 18, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis further discloses wherein the at least one application parameter includes information about a planned and/or performed operation or treatment (Themelis, Para 39-40; “If a fluorescent dye is used in the surgical site, the emitted light excites the dye, and light is emitted in a second wavelength band (also called the fluorescence emission wavelength band). This light can be recorded by a camera sensor, such as the optical imaging sensor, and displayed on a display of the surgical microscope. Compared to reflectance imaging, in reflectance imaging, light that is emitted by the fluorescent dye has a low illumination intensity, and thus cannot be distinguished without digital processing if the surgical site is illuminated for reflectance imaging. To enable concurrent reflectance imaging and fluorescent imaging, the fluorescence emission wavelength band (or fluorescence emission wavelength bands, if multiple fluorescent dyes having different fluorescence emission wavelength bands are used) may be sensed separately by the at least one optical imaging sensor, and light in the fluorescence emission wavelength band(s) (and optionally in the fluorescence excitation wavelength band(s)) may be omitted or filtered out from the illumination being used for reflectance imaging. In some examples, different optical imaging sensors are used for fluorescence imaging and reflectance imaging. For example, the microscope may comprise a first optical imaging sensor being configured to perform reflectance imaging and a second optical imaging sensor being configured to perform fluorescence imaging. The system may be configured to obtain the first component from the first optical imaging sensor being configured to perform reflectance imaging and to obtain the second component from the second optical imaging sensor being configured to perform fluorescence imaging. In some examples, wavelength bands are recorded separately by one or more optical imaging sensors (e.g., for the purposes of multispectral images, e.g., using filters that are placed in front of the optical imaging sensors), and one or more of the wavelength bands may be used for fluorescence imaging, e.g. by blocking light from the emission spectrum of the illumination system of the surgical microscope system that intersects with the fluorescence emission wavelength band.”).
Regarding claim 19, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis further discloses wherein the at least one application parameter includes parameters that describe and/or determine the overlay between the color image representation and the processed fluorescence image representation (Themelis, Para 51-52; “Another potential visibility criteria are the contrast and the intensity of the representation of the feature of interest. As outlined above, in the combined view (i.e., the third representation), a pseudocolor representation of the fluorescence emissions is overlaid over the color image of the reflectance image. While the color of the pseudocolor representation is chosen to provide high contrast in usual surgical settings […] For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. For the third approach of making the surgeon aware that the feature of interest can be perceived by switching the digital view, an indicator, such as a textual indicator or symbol, for prompting a surgeon to switch the digital view may be used. In summary, the visual indicator may be one of a blinking indicator, an indicator having an increased contrast relative to adjacent portions of the digital view, an indicator drawing an outline around the feature of interest, and an indicator for prompting a surgeon to switch the digital view to show fluorescence emissions in isolation. It may be superimposed on the digital view (e.g., when using an outline or a blinking indicator), integrated as part of the digital view (e.g., when increasing the contrast of a portion of the digital view), or be shown next to the digital view (as prompt for switching the digital view). Accordingly, the system may be configured to include the visual indicator in the display signal as one of an overlay (e.g., by superimposing the visual indicator on the digital view), an indicator included in a composite reflectance and fluorescence image (i.e., the third representation, e.g., by altering the composition of the composite reflectance and fluorescence image), and a warning image (to be shown superimposed on the digital view or next to the digital view).”).
Regarding claims 20, Themelis discloses a method for providing an image representation by a surgical microscope (Themelis, Para 2; “This disclosure relates to near-infrared (NIR) imaging used in endoscopic surgical systems”), comprising:
obtaining or capturing at least one application parameter (Themelis, Para 51; “Accordingly, the system may be configured to determine a feature detected in the second component to fail the visibility criterion if a contrast or intensity of fluorescence imaging indicator highlighting the feature in the digital view relative to adjacent portions of the digital view is below a contrast threshold or intensity threshold, respectively. For example, the fluorescence imaging indicator may correspond to the pseudocolor representation of the second component, at the location of the feature of interest. The contrast may be determined by comparing the color values (e.g., the Red-Green-Blue (RGB) color values) of the pixels of the pseudocolor representation with the color values (e.g., RGB color values) of pixels of the color representation that are adjacent to the pixels of the pseudocolor representation. If the difference in color values is smaller than a threshold (in all color channels), the contrast may be deemed to fail the contrast threshold. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view).”),
capturing a color image representation of a capture region by a camera (Themelis, Para 40; “In some examples, different optical imaging sensors are used for fluorescence imaging and reflectance imaging. For example, the microscope may comprise a first optical imaging sensor being configured to perform reflectance imaging and a second optical imaging sensor being configured to perform fluorescence imaging.”) (Themelis, Para 42; “In general, the reflectance imaging may be used to generate a color image of the surgical site, while the fluorescence imaging may be used to generate an image of the surgical site that represents the intensity of the fluorescence emissions, e.g., as monochrome image. The digital view may be generated based on these images.”) (Themelis, Para 51; “The contrast may be determined by comparing the color values (e.g., the Red-Green-Blue (RGB) color values) of the pixels of the pseudocolor representation with the color values (e.g., RGB color values) of pixels of the color representation that are adjacent to the pixels of the pseudocolor representation. If the difference in color values is smaller than a threshold (in all color channels), the contrast may be deemed to fail the contrast threshold. Similarly, if the intensity of the fluorescence emissions, and thus also of the pseudocolor representation of the fluorescence emissions, is below the intensity threshold, the intensity of the pseudocolor representation (i.e., the fluorescence imaging indicator) may be deemed to fail the intensity threshold.”),
capturing a fluorescence image representation of the capture region by a fluorescence camera (Themelis, Para 40; “In some examples, different optical imaging sensors are used for fluorescence imaging and reflectance imaging. For example, the microscope may comprise a first optical imaging sensor being configured to perform reflectance imaging and a second optical imaging sensor being configured to perform fluorescence imaging.”) (Themelis, Para 42; “In general, the reflectance imaging may be used to generate a color image of the surgical site, while the fluorescence imaging may be used to generate an image of the surgical site that represents the intensity of the fluorescence emissions, e.g., as monochrome image. The digital view may be generated based on these images.”),
processing the fluorescence image representation by a processing device to optimize it for an overlay with the color image representation, wherein a type of processing steps and a processing sequence for the processing of the fluorescence image representation is defined based on the at least one application parameter that includes a density of fluorescent dye present in the capture region (Themelis, Para 51-52; “Another potential visibility criteria are the contrast and the intensity of the representation of the feature of interest. […] if the intensity of the fluorescence emissions, and thus the intensity of the pseudocolor representation, is low, and/or the color of adjacent portions of the color representation of the first component is similar, the visibility of the feature of interest may be impacted. Accordingly, the system may be configured to determine a feature detected in the second component to fail the visibility criterion if a contrast or intensity of fluorescence imaging indicator highlighting the feature in the digital view relative to adjacent portions of the digital view is below a contrast threshold or intensity threshold, respectively. […] In summary, the at least one visibility criterion may be based on at least one of a size of the one or more features of interest, a contrast of the one or more features of interest relative to the digital view (e.g., the contrast of the pseudocolor representation of the fluorescence emissions of the feature of interest relative to the adjacent color representation of the surgical site), a visual intensity of the one or more features of interest in the digital view, and the presence of an indicator highlighting the one or more features of interest in the digital view (i.e., is the pseudocolor representation shown or not, i.e., is the third or first representation shown in the digital view). […] For the first approach of highlighting the feature of interest, a blinking visual indicator, an indicator having an increased contrast, or an indicator drawing an outline around the feature of interest may be used. For the second approach of delineating the feature of interest, the indicator having an increased contrast, or the indicator drawing an outline around the feature of interest may be used. For the third approach of making the surgeon aware that the feature of interest can be perceived by switching the digital view, an indicator, such as a textual indicator or symbol, for prompting a surgeon to switch the digital view may be used. In summary, the visual indicator may be one of a blinking indicator, an indicator having an increased contrast relative to adjacent portions of the digital view, an indicator drawing an outline around the feature of interest, and an indicator for prompting a surgeon to switch the digital view to show fluorescence emissions in isolation. It may be superimposed on the digital view (e.g., when using an outline or a blinking indicator), integrated as part of the digital view (e.g., when increasing the contrast of a portion of the digital view), or be shown next to the digital view (as prompt for switching the digital view).”),
overlaying the color image representation with the processed fluorescence image representation, and providing an image signal that encodes the overlaid image representation (Themelis, Para 52; “Accordingly, the system may be configured to include the visual indicator in the display signal as one of an overlay (e.g., by superimposing the visual indicator on the digital view), an indicator included in a composite reflectance and fluorescence image (i.e., the third representation, e.g., by altering the composition of the composite reflectance and fluorescence image)”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Themelis and Selya et al. (US20220338721, hereafter Selya).
Regarding claim 4, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis does not clearly and explicitly disclose wherein the generation of the detail image comprises recognizing and removing a background in the fluorescence image representation.
In an analogous fluorescence imaging field of endeavor Selya discloses recognizing and removing a background in a fluorescence image representation (Selya, Para 27; “FIG. 11 illustrates an example of one or more image frames where background fluorescence removal was performed according to one or more algorithms based on a threshold background fluorescence value”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein the generation of the detail image comprises recognizing and removing a background in the fluorescence image representation in order to remove inconvenient random increment and therefore improve the detection of weak fluorescent signals and reduce noise as taught by Selya (Selya, Para 6-8).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Themelis and Dicarlo et al. (US20220015616, hereafter Dicarlo)
Regarding claim 6, Themelis discloses all of the limitations of claim 5 as discussed above.
Themelis does not clearly and explicitly disclose wherein the overlaying comprises identifying a picture element set which reproduces the recognized and/or extracted structures, and, corresponding to the identified picture element set, changing at least one color channel in the color image representation.
In an analogous overlying fluorescence images with color images, Dicarlo discloses wherein the overlaying comprises identifying a picture element set which reproduces the recognized and/or extracted structures, and, corresponding to the identified picture element set, changing at least one color channel in the color image representation (Dicarlo, Para 48; “In some implementations, the combiner engine 535 can be configured to execute additional image processing to improve the quality of the visual representation presented to the surgeon. In some cases, simply adding or overlaying shades of green on the approximate full-color image can cause color transitions that make portions of the overall image appear unnatural. For example, a red liver may appear to go from red to orange to yellow to green as the level of green is increased to represent increasing ICG concentrations. In some cases, this may make it challenging to detect low levels of ICG in the liver as the green color may not be visible until the level of ICG reaches a threshold amount. In some implementations, this issue may be addressed using a non-linear color mapping that is triggered upon detection of ICG is detected. In using such a mapping, instead of just adding green to the existing color at the portion where ICG is detected, the existing color is desaturated to black-and-white as green is added. In some implementations, this may prevent confusing color transitions by allowing the corresponding portions to transition from the original color directly to green. For example, a portion of an image representing locations of ICG in the liver can be made to transition from red to green without interim color transitions.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein the overlaying comprises identifying a picture element set which reproduces the recognized and/or extracted structures, and, corresponding to the identified picture element set, changing at least one color channel in the color image representation in order to improve the quality of the visual for the operator as taught by DiCarlo (Dicarlo, Para 48).
Claims 7 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Themelis and Doser et al. (US20190041333, hereafter Doser).
Regarding claim 7, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis does not clearly and explicitly disclose wherein the processing the fluorescence image representation comprises post-processing the generated detail image, wherein the overlaying with the color image representation is carried out based on the post-processed detail image.
In an analogous fluorescence imaging field of endeavor Doser discloses post-processing an image, wherein the post-processed image is overlaid and displayed (Doser, Para 59; “By way of example, an overall image can be visualized to a user, in particular in real time, said image being produced by a juxtaposition or superposition of the two image types explained above. Here, the individual images or the overall image also can be subjected to image preparation and/or post-processing in order to improve the representation”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein the processing the fluorescence image representation comprises post-processing the generated detail image, wherein the overlaying with the color image representation is carried out based on the post-processed detail image in order to improve the visual representation as taught by Doser (Doser, Para 59).
Regarding claim 12, Themelis is interpreted as disclosing all of the limitations of claim 11 as discussed above
Themelis does not clearly and explicitly disclose wherein the merged image representation is post-processed before the overlay taking into account the at least one application parameter.
In an analogous fluorescence imaging field of endeavor Doser discloses post-processing an image, wherein the post-processed image is overlaid and displayed (Doser, Para 59; “By way of example, an overall image can be visualized to a user, in particular in real time, said image being produced by a juxtaposition or superposition of the two image types explained above. Here, the individual images or the overall image also can be subjected to image preparation and/or post-processing in order to improve the representation”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein the merged image representation is post-processed before the overlay taking into account the at least one application parameter in order to improve the visual representation as taught by Doser (Doser, Para 59).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Themelis and Hay (US20160217588).
Regarding claim 10, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis does not clearly and explicitly disclose wherein values in the generated detail image are amplified by a specified gain factor prior to the overlaying the color image representation with the processed fluorescence image representation.
In an analogous fluorescence imaging field of endeavor Hay discloses wherein values in a generated image are amplified by a specified gain factor prior to an overlay of an image (Hay, Para 49; “FIG. 14 illustrates the result when an amplification factor is applied to the difference image of FIG. 12, which is then overlaid onto the original image.”) (Hay, Para 133; “Another method that can be used to further enhance the inventive technique is to amplify motion by an amplification factor. This factor determines the strength of the overlay of the difference image sequence on top of a static image from the original motion sequence”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein values in the generated detail image are amplified by a specified gain factor prior to the overlaying the color image representation with the processed fluorescence image representation as taught by Hay in order to more easily visual the fluorescent data and therefore improve usability.
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Themelis and Baumgart (US20130216119).
Regarding claim 13, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis does not clearly and explicitly disclose wherein the processed fluorescence image representation and the color image representation are overlaid by alpha- blending and/or channel replacement in a suitable color space.
In an analogous fluorescence imaging field of endeavor Baumgart discloses wherein images are overlaid by alpha- blending and/or channel replacement in a suitable color space (Baumgart, Para 18; “The vessel luminance intensity is used in step 312 as an alpha-blending factor in blending a filtered difference image derived in step 309 and a neutral background 315. In addition to being used directly as the mask for blending the difference frame S and the background B, the threshold image derived in step 309 is used to derive a different mask for this purpose. Operations performed by processor 15 on the threshold image used to generate this mask include, blurring the mask to soften the edges of the vessels, enlarging or reducing the areas shown to be vessels and connecting adjacent vessel segments”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein the processed fluorescence image representation and the color image representation are overlaid by alpha- blending and/or channel replacement in a suitable color space in order to soften the edges of the vessels, enlarging or reducing the areas shown to be vessels and connecting adjacent vessel segments allowing for a more natural look and therefore improving visualization by a user as taught by Baumgart (Baumgart, Para 18).
Regarding claim 14, Themelis as modified by Baumgart above is interpreted as disclosing all of the limitations of claim 13 as discussed above
Themelis does not clearly and explicitly disclose wherein the processed fluorescence image representation and the color image representation are overlaid by alpha-bending and a used alpha value and/or another blending parameter is a function of intensity values of the fluorescence image representation and/or of the processed fluorescence image representation and/or of the detail image.
In an analogous fluorescence imaging field of endeavor Baumgart discloses processing using alpha-blending and a used alpha value and/or another blending parameter is a function of intensity values of an image representation (Baumgart, Para 18; “The vessel luminance intensity is used in step 312 as an alpha-blending factor in blending a filtered difference image derived in step 309 and a neutral background 315. In addition to being used directly as the mask for blending the difference frame S and the background B, the threshold image derived in step 309 is used to derive a different mask for this purpose. Operations performed by processor 15 on the threshold image used to generate this mask include, blurring the mask to soften the edges of the vessels, enlarging or reducing the areas shown to be vessels and connecting adjacent vessel segments”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein the processed fluorescence image representation and the color image representation are overlaid by alpha-bending and a used alpha value and/or another blending parameter is a function of intensity values of the fluorescence image representation and/or of the processed fluorescence image representation and/or of the detail image in order to soften the edges of the vessels, enlarging or reducing the areas shown to be vessels and connecting adjacent vessel segments allowing for a more natural look and therefore improving visualization by a user as taught by Baumgart (Baumgart, Para 18).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Themelis and Shelton et la. (US20220331047, hereafter Shelton).
Regarding claim 17, Themelis is interpreted as disclosing all of the limitations of claim 1 as discussed above
Themelis does not clearly and explicitly disclose wherein the at least one application parameter includes information on visual impairments of a user of the surgical microscope and/or preferences of the user.
In an analogous intraoperative display for surgical systems field of endeavor Shelton discloses considering information on visual impairments of a user of the surgical microscope and/or preferences of the user for generation of a displayed overlay (Shelton, Para 471; “overlaid information based on predefined parameters associated with the user, such as the user's experience level, the user's age, the user's eye sight, the user's preferences, as examples”) (Shelton, Para 242; “comprise a change in one or more display elements of an overlay including a change in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof, based on changes in display priority values”) (Shelton, Para 270; “An AR system may be viewable to one or more viewers, and may include differences among views available for the one or more viewers […] Aspects such as a color of an object, lighting, or other changes may be made among the views”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Themelis wherein the at least one application parameter includes information on visual impairments of a user of the surgical microscope and/or preferences of the user in order to customize the display for a specific user as taught by Shelton (Shelton, Para 599) which increases ease of use.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to John Li whose telephone number is (313)446-4916. The examiner can normally be reached Monday to Thursday; 5:30 AM to 3:30 PM Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN D LI/Primary Examiner, Art Unit 3798