DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on March 22nd, 2023, January 4th, 2024, July 15th, 2024, and July 29th, 2026 have been considered and the listed references were noted.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “46” has been used to designate both stereo vision display 45 and left earpiece 46.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: 45 in Figure 1. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
In Paragraph [0050], "…wavelengths (420 The…" should read "…wavelengths (420). The…"
Appropriate correction is required.
Claim Objections
Claim 58 is objected to because of the following informalities:
"...wherein obtaining the representation of the first image…comprises:…" is repeated twice. The examiner believes this was a typographical error, and recommends the second instance of "...wherein obtaining the representation of the first image…comprises:…" be changed to "…wherein obtaining the representation of the second image of the surgical scene comprises:"
Appropriate correction is required.
Status of Claims
Claims 1-43 are canceled. Claims 44-63 are pending.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
The claims of the instant application are rejected on the ground of nonstatutory double patenting as being unpatentable over claims of U.S. Patent No 11,648,059. Although the conflicting claims are not identical, they are not patentably distinct from each other because the claims of the instant application for the most part, with the exception of the limitations regarding the first and second wavelength ranges in the independent claims, are generic to all that is recited in claims of the above-listed parent U.S. patent. Therefore, the present rejection of claims is based on obviousness-type non-statutory double patenting. For Example, the following chart compares Claim 44 of the instant application with Claim 1 of the Patent No. 11,648,059.
Instant Application
Patent Application No. 11658059
A method of providing visual feedback of a surgical process using a visual representation of a surgical scene rendered on one or more display devices, the method comprising:
obtaining a representation of a first image of the surgical scene using electromagnetic radiation of a first wavelength range
obtaining a representation of a second image of the surgical scene using electromagnetic radiation of a second wavelength range
presenting the visual representation of the surgical scene on the one or more display devices, wherein the visual representation is rendered using the representation of the first image and the representation of the second image.
A method of providing visual feedback of a surgical process using a visual representation of a surgical scene rendered on one or more displays associated with a surgical device, the method comprising:
obtaining a representation of a first image of the surgical scene using electromagnetic radiation of a first wavelength range that lies outside the visible range of wavelengths and lies within the near-infrared or infrared range of wavelengths, wherein an amount of electromagnetic radiation of the first wavelength range received by one or more sensors from a first tissue type is lower than an amount of electromagnetic radiation of the first wavelength range received for a second tissue type;
obtaining a representation of a second image of the surgical scene using electromagnetic radiation of a second wavelength range that lies outside the visible range of wavelengths and lies within the near-infrared or infrared range of wavelengths, wherein an amount of electromagnetic radiation of the second wavelength range received by the one or more sensors from the second tissue type is substantially different than an amount of electromagnetic radiation of the second wavelength range received for the first tissue type; and
presenting the visual representation of the surgical scene on the one or more displays, wherein the visual representation is rendered using the representation of the first image and the representation of the second image.
As noted from the chart above, Claim 1 of U.S. Patent No. 11,648,059 does not recite the limitation “first wavelength range between 1200nm and 1350nm” or “second wavelength range between 1200nm and 1350nm”. However, in an analogous field of endeavor, Yaroslavsky et al. (US 2009/0076396 - IDS) discloses “In accordance with another aspect of the invention, the filtering means is configured to pass only light within a wavelength range in which cancerous tissue and healthy tissue have absorption coefficients that are approximately equal, and in which cancerous (or pre-cancerous) tissue has a scattering coefficient different from that of healthy tissue, causing cancerous tissue to appear darker or brighter than healthy tissue. The filtering means can be configured and adapted to filter at least one of the illumination light and the reflected light to pass only light within a range of about 1050 and about 1400 nm, including any of the subranges within the above mentioned range” (Emphasis added for clarification. Light is being illuminated on tissue at filtered wavelengths between 1050-1400nm. In this instance, if cancerous and healthy tissue are being evaluated and imaged separately and they can be evaluated within any of the subranges as described in Yaroslavsky, it is being interpreted using BRI that the first and second wavelength ranges can both be found as subranges within the range depicted in this paragraph) (Yaroslavsky, Paragraph [0014]). Consequently, before the effective filing date of the instant application, it would have been obvious to one of ordinary skill in the art to combine the recited limitations of Claim 1 of the parent application 11,648,059 with the teachings of Yaroslavsky to determine the first and second wavelength ranges between 1200-1350nm. One of ordinary skill in the art would be motivated to combine the elements recited in Claim 1 of the parent patent with the teachings of Yaroslavsky to ensure illumination of a surgical scene at the appropriate wavelengths to ensure the robust characterization of target tissues. Therefore, it would have been obvious to combine the limitations of Claim 1 of the parent patent with Yaroslavsky to obtain the invention of Claim 1 of the instant application. Due to the described analysis, all independent claims of the instant application are rejected in the same manner as Claim 1. In addition, all dependent claims in the instant application are also rejected in the same manner as Claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 44-63 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite a method directed to providing visual feedback of a surgical process using a visual representation of a surgical scene rendered on one or more display devices. With respect to the analysis of independent method claim 44:
STEP 1:
With regard to Step 1: the instant claim is directed to a method; and therefore, the claim is directed to one of the statutory categories of invention.
STEP 2A, Prong One:
With regard to 2A, Prong One, the limitations “obtaining a representation of a first image of the surgical scene using electromagnetic radiation of a first wavelength range between 1200 and 1350 nm”, “obtaining a representation of a second image of the surgical scene using electromagnetic radiation of a second wavelength range between 1200 and 1350 nm”, and “the visual representation is rendered using the representation of the first image and the representation of the second image” as drafted, recite an abstract idea, such as a process that, under its broadest reasonable interpretation, covers the data gathering of important imaging areas (such as the surgical scene) and organizing the acquired images onto a display to allow a surgeon, radiology technician, or physician carefully assess tissues during a surgical procedure through observation, evaluation, judgement, and opinion. This is the concept that falls under the grouping of abstract idea mental processes for monitoring and determination (evaluation, judgement, and/or opinion of obtaining first and second images of different representations of a surgical scene and providing a visual representation of the surgical scene based on the two images).
STEP 2A, Prong Two:
The 2019 PEG defines the phrase "evaluate whether the claim recites additional elements that integrate the exception into a practical application of the exception". Therefore, additional elements, or a combination of additional elements in the claim, are required to apply, rely on, or use the judicial exception. In the instant case, in this instance, the additional limitations are "presenting the visual representation of the surgical scene on the one or more display devices" which are essentially considered insignificant extra-solution activities of acquiring input. Therefore, the additional limitation does not apply, rely on, or use the judicial exception as an indication of integration of the judicial exception into a practical application. In addition, the method in Claim 44 recites additional elements of a visual representation of a surgical scene being rendered on one or more display devices, which do not integrate the above-described abstract idea into a practical application. Accordingly, claim 44 recites an abstract idea.
Step 2B:
Because the claim fails under Step 2A, the claim is further evaluated under Step 2B. The claim herein does not include additional elements that are sufficient to amount to significantly more than the judicial exception, because as discussed above with respect to the integration of the abstract idea into practical application, the additional elements in the claim are merely insignificant extra-solution activities, which does not amount to significantly more than an abstract idea. Therefore, claim 44 is not patent eligible. Independent Claims 57 and 63 is analyzed in the same manner, and found not patent eligible under this section of the rules.
In addition, with regard to dependent claims 45-55 and 58-62 viewed individually, these additional elements, under their broadest reasonable interpretation, cover features expanding upon the limitations as an abstract idea (mental processes), and do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 44, 53, 56, 57, and 60 are rejected under 35 U.S.C. 103 as being unpatentable over Demos (US 2008/0051664) in view of Yaroslavsky (US 2009/0076396).
Regarding Claim 44, Demos discloses “A method of providing visual feedback of a surgical process using a visual representation of a surgical scene rendered on one or more display devices, the method comprising:” (Demos, Paragraph [0026] and Figure 1 discloses: “a diagram that illustrates exemplary embodiments of a system constructed in accordance with the present invention is shown in FIG. 1. The system, designated generally by the reference numeral 100, provides a method and apparatus for interrogating and characterizing human tissue components in a clinical environment from a specimen. System 100, as shown in FIG. 1, designed as a portable, compact apparatus, includes the following basic components: a CPU with software for sample image processing (not shown); a lens system 10 for image collection; an image acquisition system 12; a substantially monochromatic 20 light source; an optical band-pass filters 62; an optical polarization filters 58; a sample holder (not shown); a white light source 80; and the remaining components (discussed below).”
PNG
media_image1.png
587
841
media_image1.png
Greyscale
Here, we can see how this system can serve as a method in a clinical environment for interrogating and characterizing human tissue; Paragraph [0024] discloses “A useful feature of the present invention is that fresh surgical resections collected from patients may be measured in-vitro (e.g., in an artificial environment) and in-vivo (e.g., during medical biopsy or intervention procedures) immediately upon collection.”; This paragraph shows how this can be done in a surgical scene, as the tissue has been resected via a surgical process in order to be analyzed further; Paragraph [0036] discloses “... The output of image acquisition system 12 is coupled to a computer (not shown), e.g., a laptop computer, and image processed by commercially available image processing software, such as, Roper Scientific Winspec/32 and/or Winview/32 software, and displayed on, for example, a computer screen for human eye diagnosis or for image software analysis.”; This shows the image of the tissue from the surgical scene having a visual representation on the display of a computer); “obtaining a representation of a first image of the surgical scene using electromagnetic radiation of a first wavelength”“obtaining a representation of a second image of the surgical scene using electromagnetic radiation of a second wavelength” (Demos Paragraph [0037], discloses “The NIR polarized elastic light scattering method of the present invention to delineate differences in absorption and scattering in human tissue components allows an end-user to acquire clinical diagnostic deep-subsurface (e.g., at least 1 cm) images. An illumination wavelength, preferably greater than 500 nm from an electromagnetic radiation source is utilized to provide mean photon penetration depth larger than 1 mm.”; It is important to note that near-infrared (NIR) light is a form of electromagnetic radiation; Paragraph [0046] and Figure 3 discloses “FIG. 3 shows a set of images obtained from two liver specimens. FIG. 3a, 3b, and 3c, show a histologic section of a specimen 206, taken from a benign growth as determined by clinical assessment. FIG. 3d, 3e, and 3f, show a histologic section of a specimen 208 of a well-circumscribed 0.8.times.0.5-cm nodule. The histologic features of the nodule, as determined by clinical assessment, are those of a hepatoblastoma (i.e., a tumor of the liver). FIG. 3a and 3d are NIR autofluorescence images recorded under 632-nm illumination, respectively. FIG. 3b and 3e are NIR cross-polarized light scattering images recorded under 700-nm illumination respectively.”
PNG
media_image2.png
572
802
media_image2.png
Greyscale
Figure 3, from Demos
From here, we see that the images of the liver tissue are being taken at two or more separate wavelengths of light, thus representing both limitations of obtaining representations of a first and second image of a surgical scene at a first and second wavelength); (Demos, Paragraph [0044] and Figure 2 discloses “FIG. 2 illustrates a capability of the present invention with a set of images of an approximately 4-cm.times.3-cm human breast tissue specimen 204 with multifocal high grade ductal carcinoma 210 shown in FIG. 2d, surrounded by fibrous supporting tissue 215 with an adjacent area of fatty 220 (i.e., adipose) infiltration as shown in FIG. 2f. FIGS. 2a and 2b show autofluorescence images in the 700-nm and 1000-nm spectral region under a 532-nm and a 633-nm substantially monochromatic illumination, respectively. FIG. 2c shows a light scattering image of specimen 204 under 700-nm illumination. FIG. 2d shows a novel feature of the present invention wherein a ratio of the autofluorescence image of FIG. 2b, divided by the light scattering image of FIG. 2c, improves visibility and contrast of a pair of higher emission 1-mm diameter ductal carcinoma lesions 210 as determined by histopathological (i.e., microscopic tissue disease) assessment (i.e., hematoxylin-eosin stain). In addition, the ratio image provides better delineation of the tumor margins.”
PNG
media_image3.png
595
783
media_image3.png
Greyscale
Figure 2, from Demos
As shown, images of breast tissue are being recorded at two different wavelengths, similar to what was discussed in Figure 3. In terms of the visual representation being rendered using both the first and second image, Demos clearly describes this through the image created by taking the ratio of the autofluorescence image (Fig. 2B) and the light scattering image (Fig. 2C), which creates an entirely new image seen in Figure 2D).
Demos does not explicitly disclose “a first wavelength range between 1200-1350nm” OR “a second wavelength range of 1200-1350nm”. However, in an analogous field of endeavor, Yaroslavsky discloses “In accordance with another aspect of the invention, the filtering means is configured to pass only light within a wavelength range in which cancerous tissue and healthy tissue have absorption coefficients that are approximately equal, and in which cancerous (or pre-cancerous) tissue has a scattering coefficient different from that of healthy tissue, causing cancerous tissue to appear darker or brighter than healthy tissue. The filtering means can be configured and adapted to filter at least one of the illumination light and the reflected light to pass only light within a range of about 1050 and about 1400 nm, including any of the subranges within the above mentioned range.” (Yaroslavsky, Paragraph [0014]). Here, in this paragraph, light is being illuminated on tissue at filtered wavelengths between 1050-1400nm. In this instance, if cancerous and healthy tissue are being evaluated and imaged separately and they can be evaluated within any of the subranges as described in Yaroslavsky, it is being interpreted using BRI that the first and second wavelength ranges can both be found as subranges within the range depicted in this paragraph. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method of obtaining a visual representation of a surgical scene by obtaining visual representations of the first and second images of the scene seen in Demos with the wavelength ranges seen in Yaroslavsky to result in an improved visual feedback method. By combining the method seen in Demos with the wavelength ranges seen in Yaroslavsky, one of ordinary skill in the art allows for multiple angles of the surgical site to be taken to enable a more clear visual representation of the surgical scene to enable surgeons working on patients using telerobotic surgery machines to perform their work accurately while mitigating errors. Thus, it would have been obvious to combine the Demos and Yaroslavsky references to achieve the same method seen in Claim 44.
Regarding Claim 53, the combination of Demos and Yaroslavsky discloses “The method of claim 44, wherein the first wavelength range is between 1300 and 1350 nm, and the second wavelength range is between 1200 and 1250 nm” (Yaroslavsky, Paragraph [0066], discloses “In accordance with another aspect of the invention, the filtering means is configured to pass only light within a wavelength range in which cancerous tissue and healthy tissue have absorption coefficients that are approximately equal, and in which cancerous (or pre-cancerous) tissue has a scattering coefficient different from that of healthy tissue, causing cancerous tissue to appear darker or brighter than healthy tissue. The filtering means can be configured and adapted to filter at least one of the illumination light and the reflected light to pass only light within a range of about 1050 and about 1400 nm, including any of the subranges within the above mentioned range.”; Recall from Claim 44 the explanation behind how the first and second wavelength ranges can be found as subranges within the total range that the light source has. Here, we see that through the first wavelength range being clearly stated in the paragraph. That also means that the second wavelength range can be a subrange when analyzing a separate tissue sample). The proposed combination as well as the motivation for combining the Demos and Yaroslavsky references presented in the rejection of Claim 44, apply to Claim 53 and are incorporated herein by reference. Thus, the method recited in Claim 53 is met by Demos and Yaroslavsky.
Regarding Claim 56, the combination of Demos and Yaroslavsky discloses “The method of claim 44, wherein: obtaining the representation of the first image of the surgical scene using electromagnetic radiation of the first wavelength range between 1200 and 1350 nm comprises:” (Please see the above described analysis for Claim 44) “receiving an amount of electromagnetic radiation of the first wavelength range from a first tissue type that is substantially different than an amount of electromagnetic radiation of the first wavelength range received from a second tissue type” (Demos, Paragraph [0044] and Figure 2, discloses “FIG. 2 illustrates a capability of the present invention with a set of images of an approximately 4-cm.times.3-cm human breast tissue specimen 204 with multifocal high grade ductal carcinoma 210 shown in FIG. 2d, surrounded by fibrous supporting tissue 215 with an adjacent area of fatty 220 (i.e., adipose) infiltration as shown in FIG. 2f. FIGS. 2a and 2b show autofluorescence images in the 700-nm and 1000-nm spectral region under a 532-nm and a 633-nm substantially monochromatic illumination, respectively. FIG. 2c shows a light scattering image of specimen 204 under 700-nm illumination. FIG. 2d shows a novel feature of the present invention wherein a ratio of the autofluorescence image of FIG. 2b, divided by the light scattering image of FIG. 2c, improves visibility and contrast of a pair of higher emission 1-mm diameter ductal carcinoma lesions 210 as determined by histopathological (i.e., microscopic tissue disease) assessment (i.e., hematoxylin-eosin stain). In addition, the ratio image provides better delineation of the tumor margins. This effect is also demonstrated in the examples shown in succeeding FIGS. 3 through 5. Furthermore, FIG. 2e shows an inter-image ratio of a cross-polarized light scattering 1000 nm band-pass image (not shown) divided by the cross-polarized light scattering 700 nm band-pass image of FIG. 2c. FIG. 2f shows an inter-image ratio of cross-polarized light scattering 700 nm band-pass image of FIG. 2c divided by the autofluorescence image after 532 nm illumination of FIG. 2a.”
PNG
media_image3.png
595
783
media_image3.png
Greyscale
From the aforementioned paragraph, we can see that the breast tissue specimen receives four separate wavelengths of light, with each wavelength being substantially different from one another. Due to the wavelengths being different, the images are able to highlight different portions of the tissue (thus creating a first tissue type (healthy tissue) and second tissue type (cancerous tissue) that are seen at specific wavelengths); and “obtaining the representation of the second image of the surgical scene using electromagnetic radiation of the second wavelength range between 1200 and 1350 nm comprises” (For the sake of brevity, the analysis for this limitation has been omitted. Please see the above described analysis for Claim 44); “receiving an amount of electromagnetic radiation of the second wavelength range from the second tissue type that is substantially different from an amount of electromagnetic radiation of the second wavelength range received from the first tissue type.” (Demos, Paragraph [0044] and Figure 2, see both citations above; The same citation and explanation can be used in this scenario, as the image of the second tissue type (cancerous tissue) can be evaluated at a separate wavelength, which vary from the image of first tissue type (healthy tissue) that is evaluated at another substantially different wavelength).
Claim 57 recites a system with elements corresponding to the steps recited in Claim 44. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Demos and Yaroslavsky references, presented in rejection of Claim 44, apply to this claim. Finally, the combination of Demos and Yaroslavsky references discloses one or more display devices as well as one or more processing devices, (for example, see Demos, Paragraphs [0024], [0026], and [0036]).
Claim 60 recites a system with elements corresponding to the steps recited in Claim 53. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Demos and Yaroslavsky references, presented in rejection of Claim 53, apply to this claim.
Claims 45-46, 48-52, 55, 58-59, and 62-63 are rejected under 35 U.S.C. 103 as being unpatentable over Demos in view of Yaroslavsky, and further in view of DiCarlo et al. (WO-2016100214-A1).
Regarding Claim 45, the combination of Demos and Yaroslavsky discloses “The method of claim 44, further comprising:” (Please see the above-described analysis regarding Claim 44) DiCarlo, Paragraph [0021] and Figure 1)
PNG
media_image4.png
837
748
media_image4.png
Greyscale
As clearly shown in [0021] and Figure 1, the surgeon offers user input for the visual representation of the surgical scene, controlling the surgical device to perform the surgery based on the clearer visual representation of the ureters that are being operated on. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the reception of user input for controlling a surgical device seen in DiCarlo to result in an improved visual feedback method. By combining the method with the technique of receiving input from a user (surgeon), one of ordinary skill in the art enables the surgeon to appropriately respond to the surgical scene adequately based off of the image that is depicted when the method has been performed. Thus, it would have been obvious for one of ordinary skill in the art to combine the Demos, Yaroslavsky, and DiCarlo references to achieve the same method seen in Claim 45.
Regarding Claim 46, the combination of Demos and Yaroslavsky discloses “The method of claim 44, wherein obtaining the representation of the first image or obtaining the representation of the second image comprises:” (Please see the above-described analysis regarding Claim 44); (DiCarlo, Paragraph [0011]). It is important to note that although the wavelengths shown here are not exactly the same as the ones mentioned in Claim 44, this paragraph mainly demonstrates the concept of the surgical scene being illuminated at different wavelength ranges. The first and second wavelength ranges have already been interpreted from Yaroslavsky, Paragraph [0014] (as per the analysis of Claim 44). The electromagnetic radiation part of the limitation will be further disclosed from DiCarlo in the next paragraph. DiCarlo also discloses “In normal operation, i.e., where a color image of a surgical site is presented on stereoscopic display unit 351, image capture sensor 321R captures a plurality of scenes, one for each color channel in surgical system 300. In another aspect for a ureter analysis mode, appropriate band pass filters are used in image capture sensor 321R so that a plurality of scenes is captured, one for each of the low, medium, and high wavebands described above. Each of the plurality of band pass filters passes one of the plurality of wavebands that are reflected differently be ureter tissue and non-ureter tissue. In this aspect of the ureter analysis mode, illuminator 310 includes at least a white light source and an infrared illumination source.” (DiCarlo, Paragraph [0044]). Here, the sensor is clearly used to sense different portions of the infrared light spectrum (which was already defined previously as a form of electromagnetic radiation) based on the wavelengths of light passed through the filters. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the DiCarlo technique of illuminating the surgical scene while using one or more sensors to generate the representation of a first or second image to result in an improved visual feedback method. By using the method seen in the combination of Demos and Yaroslavsky with the DiCarlo technique, one of ordinary skill in the art allows for multiple varieties of tissues to be analyzed based on specific features or characteristics that make them visible to the scene to allow surgeons to focus only on the selected areas to avoid damaging other surrounding tissues or organs. Thus, it would have been obvious to combine the Demos, Yaroslavsky, and DiCarlo references to achieve the same method described in Claim 46.
Regarding Claim 48, the combination of Demos, Yaroslavsky discloses “The method of claim 44, wherein obtaining the representation of the first image or obtaining the representation of the second image comprises: illuminating the surgical scene using broadband electromagnetic radiation” (Demos, Paragraph [0030], discloses “Monochromatic light source 20 is used for photoexcitation to provide NIR autofluorescence images, including a polarizer 58 and a narrow band-pass filter 62 positioned to ensure a predetermined narrow band of electromagnetic radiation with a predetermined polarization orientation to uniformly illuminate sample 46. White light source 80, to provide elastic light scattered images, additionally has one or more polarization filters 90 positioned to orient one or more illumination output beams 92 to a predetermined polarization orientation prior to uniformly illuminating a tissue specimen.”; Paragraph [0036] discloses “Autofluorescence emission is generated from light sources 20, or 30, and then collected from tissue sample 46 in a back-scattering geometry, as generally shown by optical rays 94, by lens system 10 having one or more interchangeable camera lenses, preferably a 50-mm focal length camera lens. An analyzing polarizer 74 is positioned before lens system 10 on a translation stage such that parallel linear polarization, orthogonal cross-polarization, orthogonal elliptical polarization, same elliptical polarization, opposite circular polarization, or non-polarization analysis of the autofluorescence emission may be employed. A bandpass filter 72 is additionally positioned before lens system 10 to ensure a proper spectral band selection between about 650 nm and about 1500 nm for imaging.”; Here, we can see that the monochromatic light source and white light source are being used with polarization filters that increase the normally narrow band wavelengths in the electromagnetic radiation from the NIR autofluorescence images to have broadband electromagnetic radiation through wavelengths being accommodated from 650nm to 1500nm); “generating the representation of the first image” (Demos, Paragraph [0037], discloses “The NIR polarized elastic light scattering method of the present invention to delineate differences in absorption and scattering in human tissue components allows an end-user to acquire clinical diagnostic deep-subsurface (e.g., at least 1 cm) images. An illumination wavelength, preferably greater than 500 nm from an electromagnetic radiation source is utilized to provide mean photon penetration depth larger than 1 mm.”; Paragraph [0046] and Figure 3 discloses “FIG. 3 shows a set of images obtained from two liver specimens. FIG. 3a, 3b, and 3c, show a histologic section of a specimen 206, taken from a benign growth as determined by clinical assessment. FIG. 3d, 3e, and 3f, show a histologic section of a specimen 208 of a well-circumscribed 0.8.times.0.5-cm nodule. The histologic features of the nodule, as determined by clinical assessment, are those of a hepatoblastoma (i.e., a tumor of the liver). FIG. 3a and 3d are NIR autofluorescence images recorded under 632-nm illumination, respectively.”
PNG
media_image2.png
572
802
media_image2.png
Greyscale
); Yaroslavsky, Paragraph [0066], discloses “High-contrast images of cancer tissue and adjacent healthy tissue can be produced with filtering means (or without filtering in the case of monochromatic lasers, LEDs, etc., for example) that pass light in the range from about 1050 and about 1400 nm, or any subrange thereof. Those skilled in the art will readily appreciate that within this range, narrow ranges can also be used to produce images in which cancer tissue is contrasted against healthy adjacent tissue. By way of example and not limitation, a range of 1300 nm to 1350 nm can provide high contrast images, however, those skilled in the art will readily appreciate that any subset of ranges in the range of about 1040 nm to about 1400 nm can be used without departing from the spirit and scope of the invention.”; As we can remember from the analysis of Claim 44, the first wavelength can be derived from subranges within the range seen in this paragraph); and “generating the representation of the second image” (Demos, Paragraph [0037], discloses “The NIR polarized elastic light scattering method of the present invention to delineate differences in absorption and scattering in human tissue components allows an end-user to acquire clinical diagnostic deep-subsurface (e.g., at least 1 cm) images. An illumination wavelength, preferably greater than 500 nm from an electromagnetic radiation source is utilized to provide mean photon penetration depth larger than 1 mm.”; Paragraph [0046] and Figure 3 (see above) discloses “FIG. 3 shows a set of images obtained from two liver specimens. FIG. 3a, 3b, and 3c, show a histologic section of a specimen 206, taken from a benign growth as determined by clinical assessment. FIG. 3d, 3e, and 3f, show a histologic section of a specimen 208 of a well-circumscribed 0.8.times.0.5-cm nodule. The histologic features of the nodule, as determined by clinical assessment, are those of a hepatoblastoma (i.e., a tumor of the liver). FIG. 3a and 3d are NIR autofluorescence images recorded under 632-nm illumination, respectively.”);Yaroslavsky, Paragraph [0066], discloses “High-contrast images of cancer tissue and adjacent healthy tissue can be produced with filtering means (or without filtering in the case of monochromatic lasers, LEDs, etc., for example) that pass light in the range from about 1050 and about 1400 nm, or any subrange thereof. Those skilled in the art will readily appreciate that within this range, narrow ranges can also be used to produce images in which cancer tissue is contrasted against healthy adjacent tissue. By way of example and not limitation, a range of 1300 nm to 1350 nm can provide high contrast images, however, those skilled in the art will readily appreciate that any subset of ranges in the range of about 1040 nm to about 1400 nm can be used without departing from the spirit and scope of the invention.”; As we can remember from the analysis of Claim 44, the second wavelength can be derived from subranges within the range seen in this paragraph).
The combination of Demos and Yaroslavsky does not explicitly disclose “using data captured by a first sensor of the one or more sensors configured to sense a first portion of the electromagnetic radiation reflected or transmitted from the surgical scene, wherein the first portion of the electromagnetic radiation passes through a first filter configured to selectively pass electromagnetic radiation” or “using data captured by the first sensor or a second sensor of the one or more sensors, the first sensor or the second sensor configured to sense a second portion of the electromagnetic radiation reflected or transmitted from the surgical scene, wherein the second portion of the electromagnetic radiation passes through a second filter configured to selectively pass electromagnetic radiation”. However, in an analogous field of endeavor, DiCarlo discloses “In normal operation, i.e., where a color image of a surgical site is presented on stereoscopic display unit 351, image capture sensor 321R captures a plurality of scenes, one for each color channel in surgical system 300. In another aspect for a ureter analysis mode, appropriate band pass filters are used in image capture sensor 321R so that a plurality of scenes is captured, one for each of the low, medium, and high wavebands described above. Each of the plurality of band pass filters passes one of the plurality of wavebands that are reflected differently be ureter tissue and non-ureter tissue. In this aspect of the ureter analysis mode, illuminator 310 includes at least a white light source and an infrared illumination source.” (DiCarlo, Paragraph [0044]). From this, we can see the sensor being able to capture the first and second portion of electromagnetic radiation, as it has filters for both the low, medium, and high bands which covers the entire spectrum of the broadband electromagnetic radiation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the DiCarlo technique of using the data captured by the sensor to sense specific portions of electromagnetic radiation through a filter to achieve the same visual feedback method seen in Claim 48.
Regarding Claim 49, the combination of Demos, Yaroslavsky discloses “The method of claim 44, wherein obtaining the representation of the first image or obtaining the representation of the second image comprises: illuminating the surgical scene using broadband electromagnetic radiation that includes multiple wavelengths” (Demos, Paragraph [0030], discloses “Monochromatic light source 20 is used for photoexcitation to provide NIR autofluorescence images, including a polarizer 58 and a narrow band-pass filter 62 positioned to ensure a predetermined narrow band of electromagnetic radiation with a predetermined polarization orientation to uniformly illuminate sample 46. White light source 80, to provide elastic light scattered images, additionally has one or more polarization filters 90 positioned to orient one or more illumination output beams 92 to a predetermined polarization orientation prior to uniformly illuminating a tissue specimen.”; Paragraph [0036] discloses “Autofluorescence emission is generated from light sources 20, or 30, and then collected from tissue sample 46 in a back-scattering geometry, as generally shown by optical rays 94, by lens system 10 having one or more interchangeable camera lenses, preferably a 50-mm focal length camera lens. An analyzing polarizer 74 is positioned before lens system 10 on a translation stage such that parallel linear polarization, orthogonal cross-polarization, orthogonal elliptical polarization, same elliptical polarization, opposite circular polarization, or non-polarization analysis of the autofluorescence emission may be employed. A bandpass filter 72 is additionally positioned before lens system 10 to ensure a proper spectral band selection between about 650 nm and about 1500 nm for imaging.”; Here, we can see that the monochromatic light source and white light source are being used with polarization filters that increase the normally narrow band wavelengths in the electromagnetic radiation from the NIR autofluorescence images to have broadband electromagnetic radiation through wavelengths being accommodated from 650nm to 1500nm (which, as can be seen, contain multiple wavelengths)); “generating the representation of the first image” (Demos, Paragraph [0037], discloses “The NIR polarized elastic light scattering method of the present invention to delineate differences in absorption and scattering in human tissue components allows an end-user to acquire clinical diagnostic deep-subsurface (e.g., at least 1 cm) images. An illumination wavelength, preferably greater than 500 nm from an electromagnetic radiation source is utilized to provide mean photon penetration depth larger than 1 mm.”; Paragraph [0046] and Figure 3 discloses “FIG. 3 shows a set of images obtained from two liver specimens. FIG. 3a, 3b, and 3c, show a histologic section of a specimen 206, taken from a benign growth as determined by clinical assessment. FIG. 3d, 3e, and 3f, show a histologic section of a specimen 208 of a well-circumscribed 0.8.times.0.5-cm nodule. The histologic features of the nodule, as determined by clinical assessment, are those of a hepatoblastoma (i.e., a tumor of the liver). FIG. 3a and 3d are NIR autofluorescence images recorded under 632-nm illumination, respectively.”
PNG
media_image2.png
572
802
media_image2.png
Greyscale
); Yaroslavsky, Paragraph [0066], discloses “High-contrast images of cancer tissue and adjacent healthy tissue can be produced with filtering means (or without filtering in the case of monochromatic lasers, LEDs, etc., for example) that pass light in the range from about 1050 and about 1400 nm, or any subrange thereof. Those skilled in the art will readily appreciate that within this range, narrow ranges can also be used to produce images in which cancer tissue is contrasted against healthy adjacent tissue. By way of example and not limitation, a range of 1300 nm to 1350 nm can provide high contrast images, however, those skilled in the art will readily appreciate that any subset of ranges in the range of about 1040 nm to about 1400 nm can be used without departing from the spirit and scope of the invention.”; As we can remember from the analysis of Claim 44, the first wavelength can be derived from subranges within the range seen in this paragraph); (Demos, Paragraph [0037], discloses “The NIR polarized elastic light scattering method of the present invention to delineate differences in absorption and scattering in human tissue components allows an end-user to acquire clinical diagnostic deep-subsurface (e.g., at least 1 cm) images. An illumination wavelength, preferably greater than 500 nm from an electromagnetic radiation source is utilized to provide mean photon penetration depth larger than 1 mm.”; Paragraph [0046] and Figure 3 (see above) discloses “FIG. 3 shows a set of images obtained from two liver specimens. FIG. 3a, 3b, and 3c, show a histologic section of a specimen 206, taken from a benign growth as determined by clinical assessment. FIG. 3d, 3e, and 3f, show a histologic section of a specimen 208 of a well-circumscribed 0.8.times.0.5-cm nodule. The histologic features of the nodule, as determined by clinical assessment, are those of a hepatoblastoma (i.e., a tumor of the liver). FIG. 3a and 3d are NIR autofluorescence images recorded under 632-nm illumination, respectively.”);Yaroslavsky, Paragraph [0066], discloses “High-contrast images of cancer tissue and adjacent healthy tissue can be produced with filtering means (or without filtering in the case of monochromatic lasers, LEDs, etc., for example) that pass light in the range from about 1050 and about 1400 nm, or any subrange thereof. Those skilled in the art will readily appreciate that within this range, narrow ranges can also be used to produce images in which cancer tissue is contrasted against healthy adjacent tissue. By way of example and not limitation, a range of 1300 nm to 1350 nm can provide high contrast images, however, those skilled in the art will readily appreciate that any subset of ranges in the range of about 1040 nm to about 1400 nm can be used without departing from the spirit and scope of the invention.”; As we can remember from the analysis of Claim 44, the second wavelength can be derived from subranges within the range seen in this paragraph)
The combination of Demos and Yaroslavsky does not explicitly disclose “using data captured by a first sensor configured to selectively sense a first portion of the electromagnetic radiation” … “as reflected or transmitted from the surgical scene” or “using data captured by the first sensor or a second sensor configured to selectively sense a second portion of the electromagnetic radiation”… “as reflected or transmitted from the surgical scene”. However, in an analogous field of endeavor, DiCarlo discloses “In normal operation, i.e., where a color image of a surgical site is presented on stereoscopic display unit 351, image capture sensor 321R captures a plurality of scenes, one for each color channel in surgical system 300. In another aspect for a ureter analysis mode, appropriate band pass filters are used in image capture sensor 321R so that a plurality of scenes is captured, one for each of the low, medium, and high wavebands described above. Each of the plurality of band pass filters passes one of the plurality of wavebands that are reflected differently be ureter tissue and non-ureter tissue. In this aspect of the ureter analysis mode, illuminator 310 includes at least a white light source and an infrared illumination source.” (DiCarlo, Paragraph [0044]). From this, we can see the sensor being able to capture the first and second portion of electromagnetic radiation, as it has filters for both the low, medium, and high bands which covers the entire spectrum of the broadband electromagnetic radiation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the DiCarlo technique of using the data captured by the sensor to sense specific portions of electromagnetic radiation through a filter to achieve the same visual feedback method seen in Claim 49.
Regarding Claim 50, the combination of Demos and Yaroslavsky discloses “The method of claim 44, further comprising: obtaining a representation of a third image of the surgical scene using a third wavelength range that lies outside the visible range of wavelengths” (Demos, Paragraph [0009], discloses “Another aspect of the present invention provides a diagnostic apparatus that includes: at least two substantially narrow-band electromagnetic radiation wavelength sources for simultaneous illumination of one or more tissue components, an arranged interior examination device adapted to transmit the wavelength sources for illumination of the tissue components and further adapted to relay a near-infrared scattered radiation of the illuminated tissue components; and a detector adapted to simultaneously capture one or more images produced by the near-infrared scattered radiation from the tissue components so that characterization of the one or more tissue components is capable of being performed.”; It is important to note that near-infrared light falls outside the visible range of wavelengths (400-700nm), as near-infrared has wavelengths spanning from 700nm-2,500nm); (Demos, Paragraph [0030], discloses “Monochromatic light source 20 is used for photoexcitation to provide NIR autofluorescence images, including a polarizer 58 and a narrow band-pass filter 62 positioned to ensure a predetermined narrow band of electromagnetic radiation with a predetermined polarization orientation to uniformly illuminate sample 46. White light source 80, to provide elastic light scattered images, additionally has one or more polarization filters 90 positioned to orient one or more illumination output beams 92 to a predetermined polarization orientation prior to uniformly illuminating a tissue specimen.”; Paragraph [0031] discloses: “Sample 46 is illuminated with a set of one or more images, preferably seven, recorded for each sample.”; Here, a visual representation is clearly seen for the image acquired by illuminating the NIR light source, thus resulting in a clearer representation of the tissue observed in that wavelength of light). The combination of Demos and Yaroslavsky does not explicitly disclose “presenting the visual representation of the surgical scene on the one or more displays”. However, in an analogous field of endeavor, DiCarlo discloses “For a stereoscopic endoscope, the surgical site scene includes two scenes, a left scene and a right scene. Two sets of data frames, a left set and right set, are captured by image capture system 120. The two sets are processed by a ureter detection module 136 of ureter analysis module 135 to create a left ureter enhanced image and a right ureter enhanced image that are sent to a stereoscopic display unit in surgeon's control console 114. The left ureter enhanced image and the right ureter enhanced image are included in ureter enhanced image 140” (DiCarlo, Paragraph [0025]). DiCarlo further discloses “The left ureter enhanced image and the right ureter enhanced image are presented on the stereoscopic display unit in surgeon's control console 114, sometimes referred to as surgeon's console 114 or simply console 114, to create a three-dimensional scene of surgical site 103 with the ureters highlighted.”. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the DiCarlo technique of presenting the visual representation of the surgical scene on one or more displays to result in an improved visual feedback method to accommodate a tertiary wavelength range for a tertiary image. By combining the method seen in the combination of Demos and Yaroslavsky with the DiCarlo technique, one of ordinary skill in the art allows for tissues (such as cancerous tumors or tissues) to be visible when telerobotic surgery is performed on a patient to allow for a surgeon to carefully remove those tissues that may cause future problem for them. Thus, it would have been obvious for one of ordinary skill in the art to combine the Demos, Yaroslavsky, and DiCarlo references to achieve the same method described in Claim 50.
Regarding Claim 51, the combination of Demos, Yaroslavsky, and DiCarlo discloses “The method of claim 50, wherein the third wavelength range is selected such that an absorption of electromagnetic radiation in the third wavelength range for lipid is substantially equal to an absorption of electromagnetic radiation in the third wavelength range for collagen.” (Demos, Paragraph [0049] and Figure 5a-5c, discloses “FIG. 5c (i.e., the ratio of NIR cross-polarized image under 700-nm illumination FIG. 5b over NIR autofluorescence image under 632-nm illumination FIG. 5a) shows a high visibility, high contrast bladder cancerous tissue 510 from surrounding normal bladder tissue 520 similar to the case for uterine tissue shown in FIG. 4. FIG. 5a shows necrotic tissue 505 (cells that have died as a result of cancerous growth) as a high intensity region. Moreover, FIG. 5a (NIR autofluorescence image under 632-nm illumination) and FIG. 5b (NIR cross-polarized image under 700-nm illumination) also show cancerous tissue 510 as a dark feature that is about 0.40 less in digitized intensity with respect to the surrounding lighter featured normal bladder tissue 520. However, better contrast and better visibility is still the image shown in FIG. 5c as compared in FIG. 5a and FIG. 5b separately.”; Here, we can see that the tissues that are visualized as high-intensity regions have been clearly identified in terms of normal healthy tissue and cancerous tissue. As collagen and lipids act as intrinsic optical absorbers and structural chromophores inside the bladder wall, one of ordinary skill in the art could visualize the collagen and lipid within each of the healthy, cancerous, and necrotic tissue through the method of targeting them with NIR light as shown in this paragraph).
PNG
media_image5.png
182
466
media_image5.png
Greyscale
Regarding Claim 52, the combination of Demos, Yaroslavsky, and DiCarlo discloses “The method of claim 50, wherein the third wavelength range is selected such that an absorption of electromagnetic radiation in the third wavelength range for lipid is substantially different from an absorption of electromagnetic radiation in the third wavelength range for collagen.” (Demos, Paragraph [0037], discloses “The NIR polarized elastic light scattering method of the present invention to delineate differences in absorption and scattering in human tissue components allows an end-user to acquire clinical diagnostic deep-subsurface (e.g., at least 1 cm) images. An illumination wavelength, preferably greater than 500 nm from an electromagnetic radiation source is utilized to provide mean photon penetration depth larger than 1 mm. Linear cross-polarization and spectral analysis of the scattered photons substantially removes the photon information from the orthogonal illumination polarization resulting from the surface and allows substantially all of the scattered photons from the subsurface tissue to be imaged.”)
Regarding Claim 55, the combination of Demos and Yaroslavsky discloses “The method of claim 44, wherein: the surgical scene comprises a first tissue type and a second tissue type” (Yaroslavsky, Paragraph [0014], discloses “In accordance with another aspect of the invention, the filtering means is configured to pass only light within a wavelength range in which cancerous tissue and healthy tissue have absorption coefficients that are approximately equal, and in which cancerous (or pre-cancerous) tissue has a scattering coefficient different from that of healthy tissue, causing cancerous tissue to appear darker or brighter than healthy tissue.”); (DiCarlo, Paragraph [0050]). As described before in the analysis for Claim 51, collagen and lipids act as intrinsic optical absorbers and structural chromophores inside the bladder wall, so by DiCarlo’s invention having multiple variations of colors for specific areas within the image, one of ordinary skill in the art can interpret this as the collagen and lipids surrounding the bladder wall and bladder tissue as well as the ureters to have visible differences with one another. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the technique of having a visual representation that enhances the visible differences between a ureter and the surrounding lipid layers to achieve the same visual feedback method described in Claim 55.
Regarding Claim 58, the combination of Demos and Yaroslavsky discloses “The surgical system of claim 57, further comprising:” (Please see the above-described analysis regarding Claim 57) wavebands described above. Each of the plurality of band pass filters passes one of the plurality of wavebands that are reflected differently be ureter tissue and non-ureter tissue. In this aspect of the ureter analysis mode, illuminator 310 includes at least a white light source and an infrared illumination source.” (DiCarlo, Paragraph [0044]). Here we can see, as shown before previous analyses of claims, the sensors receiving the electromagnetic radiation (infrared illumination source) through receiving images filtered at different wavelengths of infrared light). DiCarlo further discloses “For a stereoscopic endoscope, the surgical site scene includes two scenes, a left scene and a right scene. Two sets of data frames, a left set and right set, are captured by image capture system 120. The two sets are processed by a ureter detection module 136 of ureter analysis module 135 to create a left ureter enhanced image and a right ureter enhanced image that are sent to a stereoscopic display unit in surgeon's control console 114. The left ureter enhanced image and the right ureter enhanced image are included in ureter enhanced image 140.” (DiCarlo, Paragraph [0025]). This paragraph shows the image capture system that is further described in Paragraph [0044], obtaining two images (using the image capture sensor) of the left and right ureter enhanced image to carefully allow the surgeon to inspect before performing any procedures. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system seen in the combination of Demos and Yaroslavsky with the DiCarlo technique of using one or more sensors to capture first and second image representations of the surgical scene to achieve the same visual feedback system described in Claim 58.
Claim 59 recites a system with elements corresponding to the steps recited in Claim 45. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Demos Yaroslavsky, and DiCarlo references, presented in rejection of Claim 45, apply to this claim.
Claim 62 recites a system with elements corresponding to the steps recited in Claim 55. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Demos Yaroslavsky, and DiCarlo references, presented in rejection of Claim 55, apply to this claim.
Regarding Claim 63, Demos discloses (Demos Paragraph [0037], discloses “The NIR polarized elastic light scattering method of the present invention to delineate differences in absorption and scattering in human tissue components allows an end-user to acquire clinical diagnostic deep-subsurface (e.g., at least 1 cm) images. An illumination wavelength, preferably greater than 500 nm from an electromagnetic radiation source is utilized to provide mean photon penetration depth larger than 1 mm.”; It is important to note that near-infrared (NIR) light is a form of electromagnetic radiation; Paragraph [0046] and Figure 3 discloses “FIG. 3 shows a set of images obtained from two liver specimens. FIG. 3a, 3b, and 3c, show a histologic section of a specimen 206, taken from a benign growth as determined by clinical assessment. FIG. 3d, 3e, and 3f, show a histologic section of a specimen 208 of a well-circumscribed 0.8.times.0.5-cm nodule. The histologic features of the nodule, as determined by clinical assessment, are those of a hepatoblastoma (i.e., a tumor of the liver). FIG. 3a and 3d are NIR autofluorescence images recorded under 632-nm illumination, respectively. FIG. 3b and 3e are NIR cross-polarized light scattering images recorded under 700-nm illumination respectively.”
PNG
media_image2.png
572
802
media_image2.png
Greyscale
Figure 3, from Demos
From here, we see that the images of the liver tissue are being taken at two or more separate wavelengths of light, thus representing both limitations of obtaining representations of a first and second image of a surgical scene at a first and second wavelength); (Demos, Paragraph [0044] and Figure 2 discloses “FIG. 2 illustrates a capability of the present invention with a set of images of an approximately 4-cm.times.3-cm human breast tissue specimen 204 with multifocal high grade ductal carcinoma 210 shown in FIG. 2d, surrounded by fibrous supporting tissue 215 with an adjacent area of fatty 220 (i.e., adipose) infiltration as shown in FIG. 2f. FIGS. 2a and 2b show autofluorescence images in the 700-nm and 1000-nm spectral region under a 532-nm and a 633-nm substantially monochromatic illumination, respectively. FIG. 2c shows a light scattering image of specimen 204 under 700-nm illumination. FIG. 2d shows a novel feature of the present invention wherein a ratio of the autofluorescence image of FIG. 2b, divided by the light scattering image of FIG. 2c, improves visibility and contrast of a pair of higher emission 1-mm diameter ductal carcinoma lesions 210 as determined by histopathological (i.e., microscopic tissue disease) assessment (i.e., hematoxylin-eosin stain). In addition, the ratio image provides better delineation of the tumor margins.”
PNG
media_image3.png
595
783
media_image3.png
Greyscale
Figure 2, from Demos
As shown, images of breast tissue are being recorded at two different wavelengths, similar to what was discussed in Figure 3. In terms of the visual representation being rendered using both the first and second image, Demos clearly describes this through the image created by taking the ratio of the autofluorescence image (Fig. 2B) and the light scattering image (Fig. 2C), which creates an entirely new image seen in Figure 2D).
Demos does not explicitly disclose “a first wavelength range between 1200-1350nm” OR “a second wavelength range of 1200-1350nm”. However, in an analogous field of endeavor, Yaroslavsky discloses “In accordance with another aspect of the invention, the filtering means is configured to pass only light within a wavelength range in which cancerous tissue and healthy tissue have absorption coefficients that are approximately equal, and in which cancerous (or pre-cancerous) tissue has a scattering coefficient different from that of healthy tissue, causing cancerous tissue to appear darker or brighter than healthy tissue. The filtering means can be configured and adapted to filter at least one of the illumination light and the reflected light to pass only light within a range of about 1050 and about 1400 nm, including any of the subranges within the above mentioned range.” (Yaroslavsky, Paragraph [0014]). Here, in this paragraph, light is being illuminated on tissue at filtered wavelengths between 1050-1400nm. In this instance, if cancerous and healthy tissue are being evaluated and imaged separately and they can be evaluated within any of the subranges as described in Yaroslavsky, it is being interpreted using BRI that the first and second wavelength ranges can both be found as subranges within the range depicted in this paragraph. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method of obtaining a visual representation of a surgical scene by obtaining visual representations of the first and second images of the scene seen in Demos with the wavelength ranges seen in Yaroslavsky to achieve the same limitations described in Claim 63.
The combination of Demos and Yaroslavsky does not explicitly disclose “One or more machine-readable non-transitory storage devices encoded with machine-readable instructions configured to cause one or more processing devices to perform operations comprising:”. However, in an analogous field of endeavor, DiCarlo discloses “Herein, a computer program product comprises a computer readable medium configured to store computer readable code needed for any part of or all of the processes described herein, or in which computer readable code for any part of or all of those processes is stored. Some examples of computer program products are CD-ROM discs, DVD discs, flash memory, ROM cards, floppy discs, magnetic tapes, computer hard drives, servers on a network and signals transmitted over a network representing computer readable program code. A non-transitory tangible computer program product comprises a tangible computer readable medium configured to store computer readable instructions for any part of or all of the processes or in which computer readable instructions for any part of or all of the processes is stored. Non-transitory tangible computer program products are CD-ROM discs, DVD discs, flash memory, ROM cards, floppy discs, magnetic tapes, computer hard drives and other physical storage mediums.” (DiCarlo, Paragraph [0071]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the non-transitory computer-readable storage device seen in DiCarlo to achieve the same non-transitory computer readable storage device described in Claim 63.
Claims 47 are rejected under 35 U.S.C. 103 as being unpatentable over Demos in view of Yaroslavsky, and further in view of DiCarlo and Homyk et al (US 2016/0198961).
Regarding Claim 47, the combination of Demos, Yaroslavsky, and DiCarlo discloses “The method of claim 46, wherein illuminating the surgical scene using electromagnetic radiation in the first wavelength range or the second wavelength range comprises: illuminating the surgical scene using electromagnetic radiation in the first wavelength range during a first time period” (DiCarlo, Paragraph [0011], discloses “In one aspect, a surgical site is illuminated with a plurality of different light spectrums. For example, the surgical site is illuminated sequentially with at least two of the plurality of different light spectrums. In one aspect, the plurality of light spectrums includes a light spectrum of wavelengths in a range from 450 nm to 580 nm, a light spectrum of wavelengths in a range from 640 nm to 750 nm, and a light spectrum of wavelengths in a range from 900 to 1080 nm.”); second emitted lights (e.g., a single phototransistor, photodiode, or other light-sensitive element could be used, during the first and second periods of time, to detect one or more properties of the first and second emitted lights). Additionally or alternatively, different sets of one or more light sensitive elements of the light sensor could be operated to receive the first and second emitted lights. This could include the first and second emitted lights differing in wavelength or some other spectrographic property, and first and second sets, respectively, of one or more light-sensitive elements of the light sensor being configured to detect light having the respective different wavelengths or other spectrographic properties (e.g., by the first and second sets of light-sensitive elements having respective different filters configured to pass respective different bands of wavelengths of light, being disposed at respective different locations within a spectrometer, e.g., relative to a diffraction grating, prism, or other optically dispersive element).” (Homyk, Paragraph [0050]). Paragraph [0050] is a key paragraph, as it directly described the environment (which will be further described in the subsequent paragraphs to establish the environment as a surgical scene) that is being illuminated with another wavelength range of light that is non-overlapping with the previous time period. Homyk further discloses that “In some embodiments, the above described system may be implemented as a stationary measurement device that may be brought into contact or proximity with a target environment. For example the system could be configured to emit beams of coherent light toward a biological tissue undergoing a surgical intervention, and to determine flow properties of the biological tissue based on received light emitted from the biological tissue responsive to the multiple beams of illumination.” (Homyk, Paragraph [0039]). Now, from this paragraph, we see that this environment is clearly a surgical scene due to the biological tissue undergoing a surgical intervention. Homyk finally discloses that “The wavelength(s) of the beam(s) of coherent illumination could be within a near-infrared (NIR) transparency window of biological tissue (e.g., between approximately 650 and approximately 950 nanometers and/or between approximately 1000 and approximately 1350 nanometers). In some examples, the wavelengths of two (or more) beams of illumination could differ such that the beams of illumination preferentially interact with respective different sets of elements and/or regions of the environment (e.g., overlapping sets of elements and/or regions)” (Homyk, Paragraph [0079]). As we have seen in the above analyses for numerous claims, NIR is a form of electromagnetic radiation, and this paragraph confirms that the environment was being illuminated using that form of radiation. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos, Yaroslavsky, and DiCarlo with the illumination of the surgical scene using electromagnetic radiation on non-overlapping time periods seen in Homyk to achieve the same visual feedback method described in Claim 47.
Claims 54 and 61 are rejected under 35 U.S.C. 103 as being unpatentable over Demos in view of Yaroslavsky, and further in view of Hannaford et al. (US 10092355 B1 w/ EFD of November 21st, 2014).
Regarding Claim 54, the combination of Demos and Yaroslavsky discloses “The method of claim 44, wherein: the surgical scene comprises a first tissue type and a second tissue type” (Yaroslavsky, Paragraph [0014], discloses “In accordance with another aspect of the invention, the filtering means is configured to pass only light within a wavelength range in which cancerous tissue and healthy tissue have absorption coefficients that are approximately equal, and in which cancerous (or pre-cancerous) tissue has a scattering coefficient different from that of healthy tissue, causing cancerous tissue to appear darker or brighter than healthy tissue.”); Yaroslavsky, Paragraph [0066], discloses “High-contrast images of cancer tissue and adjacent healthy tissue can be produced with filtering means (or without filtering in the case of monochromatic lasers, LEDs, etc., for example) that pass light in the range from about 1050 and about 1400 nm, or any subrange thereof. Those skilled in the art will readily appreciate that within this range, narrow ranges can also be used to produce images in which cancer tissue is contrasted against healthy adjacent tissue. By way of example and not limitation, a range of 1300 nm to 1350 nm can provide high contrast images, however, those skilled in the art will readily appreciate that any subset of ranges in the range of about 1040 nm to about 1400 nm can be used without departing from the spirit and scope of the invention.”; As we can remember from the analysis of Claim 44, the first and second wavelengths can be derived from subranges within the range seen in this paragraph). The combination of Demos and Yaroslavsky does not explicitly disclose “an absorption characteristic of the first tissue type in the first wavelength range is different from an absorption characteristic of the second tissue type”. However, in an analogous field of endeavor, Hannaford discloses the following in Col 10, lines 5-41:
PNG
media_image6.png
801
505
media_image6.png
Greyscale
Here, the absorption spectrum of the tissues are interpreted as absorption characteristics for cancerous and non-cancerous tissue. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method seen in the combination of Demos and Yaroslavsky with the technique of identifying the differing absorption characteristics between the first and second tissue types for two different wavelength ranges to achieve the same visual feedback method described in Claim 54
Claim 61 recites a system with elements corresponding to the steps recited in Claim 54. Therefore, the recited elements of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Demos Yaroslavsky, and Hannaford references, presented in rejection of Claim 54, apply to this claim.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Butte et al. (US 2016/0364858) teaches systems and methods for imaging a sample using components such as an image sensor, a laser for emitting excitation light for infrared or near-infrared fluorophores, etc.
McDowall et al. (WO 2012003127 A1) teaches an image capture system that acquires an image for each of the visible color components illuminating the surgical site and a fluorescence image, which is excited by the fluorescence excitation component from the illuminator.
Wood et al. (WO 2014139020 A1) teaches systems, methods and devices are provided for illuminating tissue with monochromatic or broadband light and imaging light that has been reflected back from the tissue.
Sarvazyan (US 2015/0141847) teaches systems and methods for hyperspectral analysis of cardiac tissue.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SORIE I KOROMA JR whose telephone number is (571)272-9259. The examiner can normally be reached Monday - Friday 8AM-6:00PM; Alternate Fridays Off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SORIE I KOROMA JR/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights