DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/09/2025 was filed in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description:
FIG. 10: Although paragraph [0119] (see specification section below) includes the labels 18d, these labels do not appear in this figure. Rather this figure includes the labels 18b. The examiner recommends clarifying which is the correct label and updating the drawing or specification accordingly.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “142” has been used to designate both proximity sensors and what appears to be the second end of the tubular sleeve 132d.
FIG. 14: This figure includes two labels 142. The examiner believes the 142 in line with the labels 10 and 132d is intended to be 136d since paragraph [0141]: “[…] one or more scanning systems 10 oriented from a second end 136d of the tubular sleeve 132d or colonoscopy tool. As shown in FIG. 14”, includes this label, but the figure does not.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
[0007]: As written it reads “Many medical facilities use CT scans to measure the herniated area to allow selection of patch size prior to surgery”. However, this is the first instance of the term “CT”, therefore, the term should be spelled out to provide clarity.
[0020]: As written the sentence does not end with a period (“.”).
[0076]: As written it reads “meeting operating objectives of the optical three-dimensional medical scanning and mapping system 10a requires the right balance between the system parameters”. However, this sentence does not end with a period (“.”).
[0119]: As written it reads “FIG. 10 illustrates an exemplary embodiment of a system 10d having two or more cameras 18d and two or more optical sources 16d. The system 10d includes two infrared cameras 18b, two pattern generators 28d, two optical sources 16d and a visible camera. The use of multiple cameras 18d placed at different angles […] shadowed from the view of one camera 18d […] Multiple cameras 18d may also provide overlapping FOVs [….] The cameras 18d can be located at different distances from the optical source 16d”. However, the examiner believes the instances of 18d are typos since FIG. 10 only includes labels 18b.
[0162]: As written it reads “The measurements may also provide data useful for documenting the procedure for later review and reference”. However, this sentence does not end with a period (“.”).
Appropriate correction is required.
Claim Objections
Claims 1 and 12 are objected to because of the following informalities:
Regarding claim 1, as written it reads “A system for providing three-dimensional imaging of a surgical site during surgery, comprising: a camera configured to obtain image data representative of a two-dimensional visible light image of a surgical site”. However, the examiner believes the second instance of “a surgical site” (underlined above) should be “the surgical site” to avoid potential antecedent basis issues.
Regarding claim 12, as written it reads “wherein image reconstruction system is configured to merge the point clouds providing a merged point cloud, and apply color to the merged point cloud using color information included in the two-dimensional visible light image”. However, to be grammatically correct, the examiner believes “the” should be added between “wherein” and “image” (underlined above).
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1-12, and 19-20 rejected under 35 U.S.C. 112(a) as failing to comply with the written description requirement.
The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, at the time the application was filed, had possession of the claimed invention.
Regarding claims 1, 11 and 19, as written the claims read “a depth sensor separate from the camera and configured to obtain depth data representative of a depth map of the surgical site” (Claim 1); “a depth sensor separate from the camera and configured to generate a depth map of the surgical site” (Claim 11); and “obtaining, by a depth sensor separate from the camera, depth data representative of a depth map of the surgical site” (Claim 19). However, there is a lack of written description for the term “depth sensor”.
Regarding claims 2-10, 12 and 20, due to their dependence on claims 1, 11 and 19, respectively, these claims inherit the rejection under 35 U.S.C. 112(a).
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-12, and 19-20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding claims 1, 11 and 19, as written the claims read “a depth sensor separate from the camera and configured to obtain depth data representative of a depth map of the surgical site” (Claim 1); “a depth sensor separate from the camera and configured to generate a depth map of the surgical site” (Claim 11); and “obtaining, by a depth sensor separate from the camera, depth data representative of a depth map of the surgical site” (Claim 19). However, due to the lack of written description for the term “depth sensor” it is unclear which component within the systems shown in the figures is intended to correspond to the “depth sensor”. The examiner would recommend clarifying whether this depth sensor corresponds to the proximity sensor 142, the time-of-flight camera 18c, or another camera/sensor.
Additionally, the examiner notes that the phrase “separate from the camera” can be interpreted broadly to include 1) a depth sensor/camera located within the same channel of an endoscope (i.e. same face) as a separate optical camera; 2) a depth sensor located in a separate endoscope/catheter from an optical camera; 3) a depth sensor located outside of an endoscope/patient, and/or the like. The examiner would therefore recommend clarifying the scope of the phrase “a depth sensor separate from the camera”.
Regarding claims 2-10, 12 and 20, due to their dependence on claims 1, 11 and 19, respectively, these claims inherit the rejection under 35 U.S.C. 112(b).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, 6, 8-11, 13-16 and 19 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Krieger et al. WO 2015/081213 A1 “Krieger”.
Regarding claim 1, Krieger teaches “A system for providing three-dimensional imaging of a surgical site during surgery, comprising:” (“The system and similarly the method describe an optical camera that captures at least one optical image of an area of interest, a depth sensor that captures at least one depth map of the area of interest, and circuitry configured to correlate depth information of the at least one depth map to the at least one optical image to generate a depth image, correct the at least one optical image by applying a model to address alteration in the respective at least one optical image, the model using information from the depth image, and output the corrected at least one optical image for display in 2D and/or as a 3D surface” [Page 19, Lines 5-12]; “The system further including the optical camera and the depth sensor being positioned at the distal end of a laparoscopic tool” [Page 19, Lines 13-14]; “In the case of the laparoscopic embodiment, discussed later, the constant updating allows for the computer workstation to keep a relatively local, up-to-date 3D model of the environment which is in the FOV. This can be useful not only for the image correction applications, but also for intraoperative navigation of other tools” [Page 10, Line 31-Page 11, Line 2]. Therefore, since the circuitry of the system outputs the corrected at least one optical image for display as a 3D surface and in the laparoscopic embodiment, an up-to-date 3D model of the environment which is in the FOV (i.e. field of view) is provided (i.e. useful for intraoperative navigation), the system represents a system for providing three-dimensional imaging of a surgical site during surgery.);
“a camera configured to obtain image data representative of a two-dimensional visible light image of a surgical site” (See [Page 10, Line 31-Page 11, Line 2] above; “In one embodiment, shown in FIG. 2, an exemplary imaging system is implemented in a laparoscope 12 that contains a 3D camera 13, a light source 14, and an optical camera 15 directed at an interesting object. The optical camera 15 may be a multispectral camera for image acquisition that captures several images at discrete and narrow bands of electromagnetic spectrum and the 3D camera may be a light- field camera for depth information. The images from these cameras are fed to a computer workstation and monitor (such as the computing device shown in Figure 14), which the surgeon may operate via convenient human interaction methods. The use of a light-field camera is new to the field of medical imaging, as many systems rely on optical or electromagnetic trackers to obtain depth information” [Page 13, Lines 1-10]. As shown in FIG. 2, the 3D camera 13, the light source 14 and the optical camera 15 are all distinct components included within the laparoscope 12. Since the optical camera 15 may be a multispectral camera for image acquisition in narrow bands of electromagnetic spectrum (i.e. the visible light spectrum being part of the electromagnetic spectrum), the system includes a camera (i.e. optical camera 15) configured to obtain image data representative of a two-dimensional visible light image of a surgical site.);
“a depth sensor separate from the camera and configured to obtain depth data representative of a depth map of the surgical site” (See 3D camera/light-field camera in [Page 13, Lines 1-10] above and “a depth sensor that captures at least one depth map of the area of interest” [Abstract]; “An optical image 3 is acquired along with a depth map 2 over at least a portion of a field-of-view (FOV). […] Information about the depth of corresponding optical image 3 pixels can be determined based on their position in the depth map 2, which can be obtained from a 3D-capable depth sensor. Example images developed from optical image 3 and depth map 2 are shown in FIGS. 9 and 13. […] FIG. 13 presents exemplary depth images 2 obtained from a light-field camera that would provide information about the 3D structure of the object of interest” [Page 6, Detailed Description, Line 24-Page 7, Line 11]. As shown in FIG. 2, the optical camera 15 (i.e. camera) is separate from the 3D camera 13 (i.e. depth sensor). Therefore, the system includes a depth sensor separate from the camera and configured to obtain depth data representative of a depth map (i.e. depth map 2) of the surgical site.).; and
“an image reconstruction system communicatively coupled to the camera and the depth sensor, the image reconstruction system configured to generate, based on the image data and the depth data, a three-dimensional image of the surgical site” (See [Page 19, Lines 5-12] and [Page 10, Line 31-Page 11, Line 2] above. Therefore, since the circuitry is “configured to correlate depth information of the at least one depth map to the at least one optical image to generate a depth image, correct the at least one optical image by applying a model to address alteration in the respective at least one optical image, the model using information from the depth image, and output the corrected at least one optical image for display in 2D and/or as a 3D surface, the circuitry represents an image reconstruction system communicatively coupled to the camera and the depth sensor, the image reconstruction system configured to generate, based on the image data (i.e. optical image) and the depth data (i.e. depth map), a three-dimensional image of the surgical site (i.e. 3D surface).).
Regarding claim 2, Krieger discloses all features of the claimed invention as discussed with respect to claim 1 above, and Krieger further teaches “further comprising: a first optical source emitting visible light” (See [Page 13, Lines 1-10] as discussed in claim 1 above. In this case, the light source 14 represents a first optical source emitting visible light (i.e. within the narrow bands of the electromagnetic spectrum).); and
“wherein the camera is configured to obtain the image data using visible light by detecting the visible light after the visible light reflects off tissue within the surgical site” (See [Page 13, Lines 1-10] as discussed in claim 1 above. Since the optical camera 15 may be a multispectral camera for image acquisition in narrow bands of electromagnetic spectrum (i.e. the visible light spectrum being part of the electromagnetic spectrum), the camera (i.e. optical camera 15) is configured to obtain the image data using visible light by detecting the visible light after the visible light reflects off tissue within the surgical site.).
Regarding claim 3, Krieger discloses all features of the claimed invention as discussed with respect to claim 2 above, and Krieger further teaches “wherein the depth sensor is configured to obtain depth data by: detecting the visible light after the visible light reflects off tissue within the surgical site; and generating, based on the detected visible light, the depth data” (See [Page 13, Lines 1-10] as discussed in claim 1 above and “If a particular band of frequencies would enhance contrast of a selected set of features in an imaged object, then the depth of those features could be more easily and robustly determined by the light-field camera with a particular multispectral filter. This could help provide more selective, highly accurate depth maps for certain objects of interest. For example, detailed maps of vasculature surrounding a cancerous region can be captured simply by imaging the tumor” [Page 12, Lines 29-34]. In this case, since the 3D camera 13 is a light-field camera used for obtaining depth information, (i.e. useful in generating highly accurate depth maps for objects of interest such as vasculature surrounding a tumor, see [Page 12, Lines 29-34]), and light is emitted by the light source 14, the depth sensor is configured to obtain depth data by: detecting the visible light after the visible light reflects off tissue within the surgical site (i.e. a tumor, for example); and generating, based on the detected visible light, the depth data (i.e. a depth map).).
Regarding claim 6, Krieger discloses all features of the claimed invention as discussed with respect to claim 1 above, and Krieger further teaches “wherein generation of the three-dimensional image of the surgical site includes: generating images of the surgical site at a first perspective, providing first perspective images, and generating images of the surgical site at a second perspective, providing second perspective images” (“The surgeon may use the user interface to change views, to change distortion 4 (in FIG. 1) or deformation model 5(in FIG. 1) parameters, to rotate, scale, or translate 3D views, or to make other parameter changes that influences the display shown on monitors” [Page 16, Lines 1-4]. In this case, in order for the surgeon to be able to change views and/or rotate, scale or translate 3D views, the system has to first generate multiple 3D views for selection by the surgeon. Therefore, the generation of the three-dimensional image of the surgical site includes: generating images of the surgical site at a first perspective, providing first perspective images, generating images of the surgical site at a second perspective and providing second perspective images.).
Regarding claim 8, Krieger discloses all features of the claimed invention as discussed with respect to claim 1 above, and Krieger further teaches “wherein the camera configured to obtain image data representative of the two-dimensional visible light image of the surgical site is a single visible light camera” (See [Page 13, Lines 1-10] and [Page 19, Lines 5-12] as discussed in claim 1 above. As shown in FIG. 2, there is only one optical camera 15. In this case, in order for the circuitry to output the corrected at least one optical image (i.e. obtained by the optical camera 15/the visible light camera) for display in 2D, the visible light camera (i.e. optical camera 15) must be configured to obtain image data representative of a two-dimensional visible light image of the surgical site. Thus, the camera (i.e. optical camera 15) configured to obtain image data representative of a two-dimensional visible light image of the surgical site is a single visible light camera.).
Regarding claim 9, Krieger discloses all features of the claimed invention as discussed with respect to claim 1 above, and Krieger further teaches “wherein the system further comprises: an interactive interface communicatively coupled to the image reconstruction system and configured to facilitate virtual interaction by a user of a medical procedure with respect to a patient, the interactive interface comprising: a display configured to display the three-dimensional image of the surgical site” (“A surgeon may interact with the displayed images through a user interface that includes an input device (computer mouse, keyboard, microphone, buttons, joystick, touch screen, gesture control, etc.) to select various features and options. The surgeon can optionally switch between two-dimensional and three-dimensional views or can visualize them side by side on displays. The surgeon may use the user interface to change views, to change distortion 4 (in FIG. 1) or deformation model 5(in FIG. 1) parameters, to rotate, scale, or translate 3D views, or to make other parameter changes that influences the display shown on monitors” [Page 15, Line 31-Page 16, Line 4].
Therefore, the system further comprises an interactive interface (i.e. user interface with input device) communicatively coupled to the image reconstruction system (i.e. circuitry) and configured to facilitate virtual interaction by a user of a medical procedure (i.e. surgeon) with respect to a patient, the interactive interface comprising: a display configured to display the three-dimensional image of the surgical site (i.e. to display/switch between two-dimensional and three-dimensional views).).
Regarding claim 10, Krieger discloses all features of the claimed invention as discussed with respect to claim 1 above, and Krieger further teaches “wherein the depth sensor is a time-of-flight camera configured to capture optical intensity arriving at each pixel of the sensor after reflection of light from tissue at the surgical site” (“If a particular band of frequencies would enhance contrast of a selected set of features in an imaged object, then the depth of those features could be more easily and robustly determined by the light-field camera with a particular multispectral filter. This could help provide more selective, highly accurate depth maps for certain objects of interest. For example, detailed maps of vasculature surrounding a cancerous region can be captured simply by imaging the tumor” [Page 12, Lines 29-34]. In this case, in order to obtain highly accurate depth maps for objects of interest, such as tumors, the depth sensor (i.e. 3D camera 13) must be configured to capture optical intensity (i.e. light) arriving at each pixel of the sensor after reflection of light from tissue at the surgical site (i.e. cancerous region surrounding the tumor). Therefore, the depth sensor (i.e. 3D camera 13) is a time-of-flight camera configured to capture optical intensity arriving at each pixel of the sensor after reflection of light from tissue at the surgical site (i.e. tumor, for example).).
Regarding claim 11, Krieger teaches “A system for providing three-dimensional imaging of a surgical site during surgery, comprising:” (See [Page 19, Lines 5-12]; [Page 19, Lines 13-14]; and [Page 10, Line 31-Page 11, Line 2] as discussed in claim 1 above. Therefore, since the circuitry of the system outputs the corrected at least one optical image for display as a 3D surface and in the laparoscopic embodiment, an up-to-date 3D model of the environment which is in the FOV (i.e. field of view) is provided (i.e. useful for intraoperative navigation), the system represents a system for providing three-dimensional imaging of a surgical site during surgery.);
“an optical hardware system that includes: a camera configured to capture a two-dimensional visible light image of the surgical site” (See [Page 10, Line 31-Page 11, Line 2] and [Page 13, Lines 1-10] as discussed in claim 1 above. As shown in FIG. 2, the optical camera 15 (i.e. camera), the light source 14, and the 3D camera 13 (i.e. depth camera/sensor) are all included within the laparoscope 12. Therefore, the system includes an optical hardware system that includes a camera configured to capture a two-dimensional visible light image of the surgical site); and
“a depth sensor separate from the camera and configured to generate a depth map of the surgical site” (See 3D camera/light-field camera in [Page 13, Lines 1-10], [Abstract]; and [Page 6, Detailed Description, Line 24-Page 7, Line 11] as discussed in claim 1 above. Therefore, optical hardware of the system includes a depth sensor (i.e. 3D camera 13) separate from the camera (i.e. optical camera 15) and configured to obtain depth data representative of a depth map (i.e. depth map 2) of the surgical site.); and
“an image reconstruction system communicatively coupled to the optical hardware system to generate, based on the two-dimensional visible light image of the surgical site and the depth map, two perspective images of the surgical site that when presented concurrently form a three-dimensional view of the surgical site” (See [Page 19, Lines 5-12] above, “In another embodiment, shown in FIG. 4, the imaging system is comprised of an endoscope, with an optical camera 15 and a stereo vision camera 16. The stereo vision camera has at least two cameras are arranged relative to each other such that images of an object can be captured from different viewpoints. These images are then processed to extract 3D information pertaining to an object. The increased physical separation between different cameras will lead to an improved 3D depth perception than that of the close mounted dual cameras in existing systems.” [Page 15, Line 31-Page 16, Line 4]; “A surgeon may interact with the displayed images through a user interface that includes an input device (computer mouse, keyboard, microphone, buttons, joystick, touch screen, gesture control, etc.) to select various features and options. The surgeon can optionally switch between two-dimensional and three-dimensional views or can visualize them side by side on displays. The surgeon may use the user interface to change views, to change distortion 4 (in FIG. 1) or deformation model 5(in FIG. 1) parameters, to rotate, scale, or translate 3D views, or to make other parameter changes that influences the display shown on monitors” [Page 15, Line 31-Page 16, Line 4]. Therefore, since the circuitry is configured to output corrected at least one optical image for display in 2D and/or as a 3D surface (See [Page 19, Lines 5-12]), a stereo vision camera 16 with at least two cameras can be utilized to capture images from different viewpoints and these images are processed to extract 3D information leading to improved 3D depth perception and the surgeon can switch between views to show information in a side-by-side manner/change what is displayed (see [Page 15, Line 31-Page 16, Line 4]), the system includes an image reconstruction system (i.e. circuitry) communicatively coupled to the optical hardware system to generate, based on the two-dimensional visible light image (i.e. optical image) of the surgical site and the depth map (i.e. depth map 2), two perspective images of the surgical site that when presented concurrently form a three-dimensional view of the surgical site.).
Regarding claim 13, Krieger teaches “A system for providing three-dimensional imaging of a surgical site during surgery, comprising:” (See [Page 19, Lines 5-12]; [Page 19, Lines 13-14]; and [Page 10, Line 31-Page 11, Line 2] as discussed in claim 1 above. Therefore, since the circuitry of the system outputs the corrected at least one optical image for display as a 3D surface and in the laparoscopic embodiment, an up-to-date 3D model of the environment which is in the FOV (i.e. field of view) is provided (i.e. useful for intraoperative navigation), the system represents a system for providing three-dimensional imaging of a surgical site during surgery.);
“an optical hardware system that includes: a visible light camera configured to obtain visible light images of the surgical site; and a time-of-flight camera separate from the visible light camera and configured to obtain depth data representative of a depth map of the surgical site” (See [Page 10, Line 31-Page 11, Line 2] and [Page 13, Lines 1-10] as discussed in claim 1 above and [Page 13, Lines 1-10], [Abstract]; and [Page 6, Detailed Description, Line 24-Page 7, Line 11] as discussed in claim 1 above. Therefore, optical hardware of the system includes a visible light camera configured to obtain visible light images of the surgical site (i.e. optical camera 15) and a time-of-flight camera (i.e. 3D camera 13) separate from the camera (i.e. optical camera 15) and configured to obtain depth data representative of a depth map (i.e. depth map 2) of the surgical site.); and
“an image reconstruction system communicatively coupled to the optical hardware system and configured to:” “generate, based on the visible light images and the depth data, a three-dimensional depth map”; “generate, based on the visible light images and three-dimensional depth map, a three-dimensional model of anatomy within the surgical site”; and “instruct an interface to display the three-dimensional model of the surgical site” (See ([Page 19, Lines 5-12] as discussed in claim 1 above. In this case, since the circuitry is configured to correlate depth information of the at least one depth map to the at least one optical image to generate a depth image (i.e. a three-dimensional depth map), correct the at least one optical image by applying a model to address alteration in the respective at least one optical image, the model using information from the depth image (i.e. a three-dimensional model of the anatomy within the surgical site), and output the corrected at least one optical image for display in 2D and/or as a 3D surface (i.e. display three-dimensional model of the surgical site), the system includes an image reconstruction system communicatively coupled to the optical hardware system and configured to:” “generate, based on the visible light images and the depth data, a three-dimensional depth map”; “generate, based on the visible light images and three-dimensional depth map, a three-dimensional model of anatomy within the surgical site”; and “instruct an interface to display the three-dimensional model of the surgical site”.).
Regarding claim 14, Krieger discloses all features of the claimed invention as discussed with respect to claim 13 above, and Krieger further teaches “wherein the visible light camera is configured to obtain image data representative of a two-dimensional visible light image of the surgical site” (See [Page 13, Lines 1-10] and [Page 19, Lines 5-12] as discussed in claim 1 above. In this case, in order for the circuitry to output the corrected at least one optical image (i.e. obtained by the optical camera 15/the visible light camera) for display in 2D, the visible light camera (i.e. optical camera 15) must be configured to obtain image data representative of a two-dimensional visible light image of the surgical site. Thus, the visible light camera (i.e. optical camera 15) is configured to obtain image data representative of a two-dimensional visible light image of the surgical site.).
Regarding claim 15, Krieger discloses all features of the claimed invention as discussed with respect to claim 13 above, and Krieger further teaches “further comprising a first optical source configured to emit visible light” (See [Page 13, Lines 1-10] as discussed in claim 1 above. In this case, the light source 14 represents a first optical source configured to emit visible light (i.e. within the narrow bands of the electromagnetic spectrum).).
Regarding claim 16, Krieger discloses all features of the claimed invention as discussed with respect to claim 15 above, and Krieger further teaches “wherein the time-of-flight camera is configured to obtain depth data by: detecting the visible light after the visible light reflects off tissue within the surgical site; and, generating, based on the detected visible light, the depth data” (See [Page 13, Lines 1-10] as discussed in claim 1 above and [Page 12, Lines 29-34] as discussed in claim 3 above. In this case, since the 3D camera 13 is a light-field camera used for obtaining depth information, (i.e. useful in generating highly accurate depth maps for objects of interest such as vasculature surrounding a tumor, see [Page 12, Lines 29-34]), and light is emitted by the light source 14, the depth sensor is configured to obtain depth data by: detecting the visible light after the visible light reflects off tissue within the surgical site (i.e. a tumor, for example); and generating, based on the detected visible light, the depth data (i.e. a depth map).).
Regarding claim 19, Krieger teaches “A method, comprising:” (“A method for corrected imaging that includes capturing at least one optical image of an area of interest using an optical camera, capturing at least one depth map of the area of interest using a depth sensor, correlating depth information of the at least one depth map to the at least one optical image to generate a depth image, correcting the at least one optical image by applying a model to address alteration in the respective at least one optical image, the model using information from at least one of the depth image, optical image, and prior information, and outputting the corrected at least one optical image for display in 2D and/or as a 3D surface” [Page 20, Lines 8-15]. Therefore. Krieger describes a method.);
“obtaining, by a camera, image data representative of a two-dimensional visible light image of a surgical site” (See [Page 13, Lines 1-10] as discussed in claim 1 above, and [Page 20, Lines 8-15] above. Therefore, since the method includes capturing at least one optical image of an area of interest using an optical camera, the method involves obtaining, by a camera, image data representative of a two-dimensional visible light image of a surgical site.);
“obtaining, by a depth sensor separate from the camera, depth data representative of a depth map of the surgical site” (See [Page 20, Lines 8-15] above. Since the method includes capturing at least one depth map of the area of interest using a depth sensor, the method involves obtaining, by a depth sensor (i.e. 3D camera 13) separate from the camera (i.e. optical camera 15), depth data representative of a depth map of the surgical site.); and
“generating, by an image reconstruction system, a three-dimensional image of the surgical site using the image data and the depth data” (See [Page 20, Lines 8-15] above. Since the method includes outputting the corrected at least one optical image for display as a 3D surface, the method involves generating, by an image reconstruction system (i.e. circuitry), a three-dimensional image of the surgical site using the image data and the depth data.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 4, 17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krieger et al. WO 2015/081213 A1 as applied to claims 2, 15, and 19 above, and further in view of Seibel et al. US 2002/0139920 A1 “Seibel”.
Regarding claims 4 and 17, Krieger discloses all features of the claimed invention as discussed with respect to claim 2 above. However, Krieger does not teach “further comprising: a second optical source emitting infrared light; and wherein the depth sensor is configured to obtain the depth data by: detecting the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data” (Claim 4); “further comprising: a second optical source emitting infrared light; and wherein the time-of-flight camera is configured to obtain the depth data by: detecting the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data” (Claim 17).
Seibel is within the same field of endeavor as the claimed invention because it involves a minimally invasive medical image acquisition system which utilizes multiple forms of light (see [Abstract] and [0126]).
Seibel teaches “further comprising: a second optical source emitting infrared light” (Claim 4); “further comprising: a second optical source emitting infrared light” (Claim 17) (“Referring to FIG. 1, a miniature image acquisition system 10 includes an illuminating subsystem 12, a collector or detector subsystem 14 and in some embodiments a host system 16. The illuminating subsystem 12 emits light onto an object. […] In various applications, the miniature image acquisition system 10 embodies an endoscope, boroscope, bar code reader or another device for acquiring images” [0048]; “In one implementation the illuminating subsystem 12 (FIGS. 1 and 2) includes a light source 24 (FIG. 2) which is formed by a visible light source and an infrared light source. The visible light and infrared light are emitted from a common illuminating fiber 26” [0126]; “The infrared light source preferably is a modulated laser infrared source which outputs infrared light in the GHz frequency range. Fast photon infrared detectors detect the returning infrared light. Preferably, the infrared photon detectors generate detection signals at the same frequency as the infrared light source” [0127]. Therefore, the system further comprises a second optical source emitting infrared light.);
“wherein the depth sensor is configured to obtain the depth data by: detecting the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data” (Claim 4); “wherein the time-of-flight camera is configured to obtain the depth data by: detecting the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data” (Claim 17) (See [0127] above and “Depth enhancement is determined by correlating images detected by the respective detectors, or alternatively by a range finding method based on phase difference, time of flight, frequency or interferometry” [Abstract]; “Referring to FIG. 24, a method 230 for acquiring an image including depth information is presented. At step 232, a beam of light 30/32 is output from a resonant fiber waveguide 36 (see FIG. 2). The waveguide 36 scans the output beam along a scan path. The scan lens 39 focuses the beam onto an object's surface (i.e., target). At a given time the beam is focused to impinge on a spot of the target, the spot being an illuminated spot, the illuminated spot varying with time as the beam is scanned along the target surface” [0134]; “At step 234, returning light from the target surface is detected by at least one detector 50. At step 236, a plurality of pixels for an image are acquired from the returning light detected by the detector either at the distal tip or after a light-collecting optical fiber. At step 238, range information also is acquired from the returning light using any of the range finding methods described above (e.g., phase shift, time of flight, frequency, interferometry), subsequent to, or preferably in parallel to step 236” [0135]; “At step 240, the range information is used to map surface, edges and vertices of the scene imaged (e.g., topography of target). At step 242, the three dimensional image data derived in step 240 is processed to achieve pixel data for display on a display screen or other output device, such as a stereographic head mounted display” [0136]; “In still another embodiment, the output beam includes a beam of light for imaging the object being scanned. Such output beam is either monochromatic or a color beam of visible light. In addition, the output beam also includes a coincident beam for range finding. The coincident beam is either visible light or non-visible light” [0137]; “wherein the output beam of light comprises visible light and infrared light, wherein a first detector of the plurality of detectors detects returning visible light and a second detector of the plurality of detectors detects returning infrared light” [Claim 13]; “With regard to step 218, the depth map then is derived from the orientation map. A depth map represents the shape of the object in a different manner. Specifically, rather than specifying orientation information, the depth map specifies relative height above a reference plane” [0118]; “With regard to step 220, the 3-dimensional mesh is derived from the depth map. The vertices of a planar mesh are displaced by an amount related to the gray scale pixel values of the depth map. […] Further information derived from other sensors (infrared, ultraviolet, polarization) or from measurements (contour maps) can be included” [0121].
In this case, infrared light is a form of non-visible light (see [0099]: “In other embodiments, one or more of the at least three sensors captures non-visible light (e.g., ultraviolet, infrared) coincidentally output by the waveguide (e.g., an ultraviolet or infrared beam is passed through a common waveguide with the visible light)”) which is output by the light source 24/resonant fiber waveguide 36 and subsequently detected by either the detector 50/second detector. Since infrared light is detected by a second detector and information from infrared sensors can be included in the 3-dimensional mesh which is derived from the depth map, the depth/time-of-flight sensor is configured to obtain depth data: by detecting the infrared light after the infrared light reflect off tissue within the surgical site; and generate, based on the detected infrared light, the depth data.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Krieger such that it further comprises a second optical source emitting infrared light and the depth/time-of-flight sensor is configured to obtain the depth data by: detecting the infrared light after the infrared light reflects off tissue within the surgical site; generating, based on the detected infrared light, the depth data as disclosed Seibel in order to effectively distinguish the location at which an anatomical target is located. Infrared light is one of a finite number of light types which can be emitted to an anatomical target and subsequently detected for use in identifying depth information with a reasonable expectation of success. Thus, modifying the system of Krieger such that it further comprises a second optical source emitting infrared light and the depth/time-of-flight sensor is configured to obtain the depth data by: detecting the infrared light after the infrared light reflects off tissue within the surgical site; generating, based on the detected infrared light, the depth data as disclosed Seibel would yield the predictable result of effectively distinguishing the location at which an anatomical target is located.
Regarding claim 20, Krieger discloses all features of the claimed invention as discussed with respect to claim 19 above, and Krieger further teaches “wherein the depth sensor includes a time-of-flight camera” (See [Page 12, Lines 29-34] as discussed in claim 10. In this case, in order to obtain highly accurate depth maps for objects of interest, such as tumors, the depth sensor (i.e. 3D camera 13) must be configured to capture optical intensity (i.e. light) arriving at each pixel of the sensor after reflection of light from tissue at the surgical site (i.e. cancerous region surrounding the tumor). Therefore, the depth sensor (i.e. 3D camera 13) is a time-of-flight camera.)., and
However, Krieger does not teach that “the method further comprises the steps of: emitting infrared light at the surgical site; detecting, by the time-of-flight camera, the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data”.
Seibel teaches that “the method further comprises the steps of: emitting infrared light at the surgical site; detecting, by the time-of-flight camera, the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data” (See [0127]; [Abstract]; [0134]; [0135]; [0136]; [0137]; [Claim 13]; [0118]; and [0121] as discussed in claims 4 and 17 above.
In this case, infrared light is a form of non-visible light (see [0099]: “In other embodiments, one or more of the at least three sensors captures non-visible light (e.g., ultraviolet, infrared) coincidentally output by the waveguide (e.g., an ultraviolet or infrared beam is passed through a common waveguide with the visible light)”) which is output by the light source 24/resonant fiber waveguide 36 and subsequently detected by either the detector 50/second detector. Since infrared light is detected by a second detector and information from infrared sensors can be included in the 3-dimensional mesh which is derived from the depth map, the depth/time-of-flight sensor is configured to obtain depth data: by detecting the infrared light after the infrared light reflect off tissue within the surgical site; and generate, based on the detected infrared light, the depth data.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Krieger such that it further comprises emitting infrared light at the surgical site; detecting, by the time-of-flight camera, the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data, as disclosed Seibel in order to effectively distinguish the location at which an anatomical target is located. Infrared light is one of a finite number of light types which can be emitted to an anatomical target and subsequently detected for use in identifying depth information with a reasonable expectation of success. Thus, method of Krieger such that it further comprises emitting infrared light at the surgical site; detecting, by the time-of-flight camera, the infrared light after the infrared light reflects off tissue within the surgical site; and generating, based on the detected infrared light, the depth data, as disclosed Seibel, would yield the predictable result of effectively distinguishing the location at which an anatomical target is located.
Claim(s) 5 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krieger et al. WO 2015/081213 A1 as applied to claims 2 and 15 above, and further in view of Bayer et al. US 2015/0223676 A1 “Bayer”.
Regarding claims 5 and 18, Krieger discloses all features of the claimed invention as discussed with respect to claims 2 and 15 above, and Krieger further teaches “wherein the system further comprises a tubular sleeve, and […] the depth sensor is fixed at one end of the tubular sleeve proximate to the surgical site” (Claim 5); “wherein the system further comprises a tubular sleeve […] the time-of-flight camera […] fixed at one end of the tubular sleeve proximate to the surgical site” (Claim 18) (See [Page 19, Lines 13-14] and [Page 13, Lines 1-10] as discussed in claim 1 above. The laparoscope 12, in this case, represents a tubular sleeve. In this case, the 3D camera 13 represents the time-of-flight camera which is located at a specific position within the laparoscope 12. Thus, the system further comprises a tubular sleeve (i.e. laparoscope 12) and the time-of-flight camera/depth sensor is fixed at one end of the tubular sleeve proximate to the surgical site (i.e. the distal end of the laparoscopic tool).).
However, Krieger does not teach that “the first optical source” is “fixed at one end of the tubular sleeve proximate to the surgical site” (Claims 5 and 18).
Bayer is within a related field of endeavor to the claimed invention because it involves an endoscope with one or more light sources included therein (see FIG. 2).
Bayer teaches that “the first optical source” is “fixed at one end of the tubular sleeve proximate to the surgical site” (Claims 5 and 18) (“As illustrated in FIG. 2, the insertion tube 12 may additionally include one or more light sources 24, such as light emitting diodes (LEDs) or fiber optical delivery of light from an external light source, and an imaging device 26. […] Each light source 24, individually, can be turned on or off. The intensity of each can be adjusted to achieve optimum imaging” [0047]. As shown in FIG. 2, the light sources 24 are located at the distal end of the insertion tube 12. Therefore, the first optical source (i.e. one of the light sources 24) is fixed at one end of the tubular sleeve proximate to the surgical site.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Krieger such that the first optical source is fixed at one end of the tubular sleeve proximate to the surgical site as disclosed in Bayer in order to easily facilitate the passage of light to a surgical site, wherein the intensity of the light is adjusted to achieve optimum imaging (see Bayer: [0047]). Fixing a light/optical source at an end of the tubular sleeve of an endoscope is one of a finite number of techniques which can be used to deliver light/optical signals to a surgical site with a reasonable expectation of success. Thus, modifying the system of Krieger such that the first optical source is fixed at one end of the tubular sleeve proximate to the surgical site as disclosed in Bayer would yield the predictable result of easily facilitating the passage of light to a surgical site, wherein the intensity of the light is adjusted to achieve optimum imaging (see Bayer: [0047]).
Claim(s) 7 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krieger et al. WO 2015/081213 A1 as applied to claims 1 and 11 above, and further in view of Reiter et al. US 2014/0336461 A1 “Reiter”.
Regarding claim 7, Krieger discloses all features of the claimed invention as discussed with respect to claim 1 above. However, Krieger does not teach “wherein generation of the three-dimensional image includes constructing multiple point clouds; and adding image data to the point clouds”.
Reiter is within a related field of endeavor as the claimed invention because it involves a surgical structured light system that provides real-time dynamic 3D visual information of the surgical environment, allowing for improved navigation and safety within the surgical field (see [Abstract]).
Reiter teaches “wherein generation of the three-dimensional image includes constructing multiple point clouds; and adding image data to the point clouds” (“One embodiment of the present disclosure comprises software including registration and modeling software to take the 3D point clouds, build meshes from these clouds, and texture map the 2D imagery onto the models. This software is operated by processors housed within the user interface which includes a display for the surgeon's viewing of the surgical site” [0064]; “Because colorized 3D point cloud information is provided, the surgeon is able to move, rotate and zoom (virtually) on the reconstructed 3D model to view the anatomy from various viewpoints, a capability which is not possible with current intra-operative imaging techniques” [0036].
Therefore, since the registration and modeling software takes the 3D point clouds (i.e. more than one), builds meshes from these clouds, and texture maps 2D imagery onto these models (i.e. reconstructed 3D model, which is provided to a surgeon for use in viewing anatomy from various viewpoints, see [0036]), the generation of the three-dimensional image includes constructing multiple point clouds; and adding image data (i.e. 2D imagery, for example) to the point clouds.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Krieger such that the generation of the three-dimensional image includes constructing multiple point clouds and adding image data to the point clouds as disclosed in Reiter in order to provide a surgeon with 3D point cloud information and thereby allow the surgeon to move, rotate and zoom (i.e. virtually) on the reconstructed 3D model to view the anatomy from various viewpoints when performing an assessment thereof (see Reiter: [0036]). Constructing 3D point clouds/meshes and mapping 2D imagery onto these models is one of a finite number of techniques which can be used to provide a surgeon with multiple views of a patient’s anatomy with a reasonable expectation of success. Thus, modifying the system of Krieger such that the generation of the three-dimensional image includes constructing multiple point clouds and adding image data to the point clouds as disclosed in Reiter would yield the predictable result of providing a surgeon with 3D point cloud information and thereby allow the surgeon to move, rotate and zoom (i.e. virtually) on the reconstructed 3D model to view the anatomy from various viewpoints when performing an assessment thereof (see Reiter: [0036]).
Regarding claim 12, Krieger discloses all features of the claimed invention as discussed with respect to claim 11 above, and Krieger further teaches “wherein the depth sensor includes multiple cameras” (“In another embodiment, shown in FIG. 4, the imaging system is comprised of an endoscope, with an optical camera 15 and a stereo vision camera 16. The stereo vision camera has at least two cameras are arranged relative to each other such that images of an object can be captured from different viewpoints. These images are then processed to extract 3D information pertaining to an object. The increased physical separation between different cameras will lead to an improved 3D depth perception than that of the close mounted dual cameras in existing systems.” [Page 15, Line 31 -Page 16, Line 4]. Therefore, since the stereo vision camera 16 includes at least two cameras to capture images of an object and these images are processed to extract 3D information for improved 3D dept perception, the stereo vision camera 16 represents a depth sensor which includes multiple cameras.);
However, Krieger does not teach that the multiple cameras are “configured to provide point clouds, wherein image reconstruction system is configured to merge the point clouds providing a merged point cloud, and apply color to the merged point cloud using color information included in the two-dimensional visible light image”.
Reiter teaches that the multiple cameras are “configured to provide point clouds, wherein image reconstruction system is configured to merge the point clouds providing a merged point cloud, and apply color to the merged point cloud using color information included in the two-dimensional visible light image” (See [0064] and [0036] as discussed in claim 7 above, and “FIG. 7 depicts a representative screen capture of an exemplary interface including an additional Side View of the colorized 3D point cloud information of the cylinder portrayed on FIG. 6. Using the SSL thumbstick (FIG. 5), the surgeon is able to move, rotate and zoom (virtually) on our reconstructed 3D model without the need to reposition the laparoscopic camera or take her/his hands of the laparoscopic tools” [0037]; “Recover the 3D point cloud using the depth image at every pixel using Eqn. 3” [0092]; “2) Colorizing the Point Cloud: Finally, because it is desirable to display a graphically-pleasing colored 3D point cloud (FIG. 7), the stereo extrinsics is used between the pattern and imaging cameras to assign an RGB color to every 3D point recovered from the depth image” [0093].
Therefore, the system includes multiple cameras (i.e. pattern and imaging cameras, see [0093]) which are configured to provide point clouds (i.e. see [0064]) and the image reconstruction system is configured to merge the point clouds providing a merged point cloud (i.e. colored 3D point cloud shown in FIG. 7), and apply color to the merged point cloud using color information included in the two-dimensional visible light image (i.e. assign an RGB color to every 3D point recovered from the depth image).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Krieger such that the multiple cameras are configured to provide point clouds, and wherein the image reconstruction system is configured to merge the point clouds providing a merged point cloud, and apply color to the merged point cloud using color information included in the two-dimensional visible light image as disclosed in Reiter in order to provide a surgeon with 3D point cloud information and thereby allow the surgeon to move, rotate and zoom (i.e. virtually) on the reconstructed 3D model to view the anatomy from various viewpoints when performing an assessment thereof (see Reiter: [0036]). Constructing 3D point clouds/meshes, mapping 2D imagery onto these models and merging/coloring these models/point clouds is one of a finite number of techniques which can be used to provide a surgeon with multiple views of a patient’s anatomy with a reasonable expectation of success. Thus, modifying the system of Krieger such that the multiple cameras are configured to provide point clouds, and wherein the image reconstruction system is configured to merge the point clouds providing a merged point cloud, and apply color to the merged point cloud using color information included in the two-dimensional visible light image as disclosed in Reiter would yield the predictable result of providing a surgeon with 3D point cloud information and thereby allow the surgeon to move, rotate and zoom (i.e. virtually) on the reconstructed 3D model to view the anatomy from various viewpoints when performing an assessment thereof (see Reiter: [0036]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Liu US 2018/0270474 A1 “Liu” is pertinent to the applicant’s disclosure because it discloses “an optical imaging system which utilizes a 3D light scanner to capture topography information, color reflectance information and fluorescence information of a target object being imaged, such as a surgical patient” [Abstract].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAITLYN E SEBASTIAN whose telephone number is (571)272-6190. The examiner can normally be reached Mon.- Fri. 7:30-4:30 (Alternate Fridays Off).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne M Kozak can be reached at (571) 270-0552. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KAITLYN E SEBASTIAN/Examiner, Art Unit 3797
Prosecution Insights