Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/20/26 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 7-9, 11-13, 18-20 and 22, are rejected under 35 U.S.C. 103 as being unpatentable over “Single-Shot Structured Light Sensor for 3D Dense and Dynamic Reconstruction” by Feifei Gu, et al. (hereinafter Gu) in view of U.S. Patent Application Publication 2014/0336461 A1 (hereinafter Reiter) in view of U.S. Patent Application Publication 2007/0204684 A1 (hereinafter Muhlhoff) in view of U.S. Patent Application Publication 2012/0320186 A1 (hereinafter Urban) in view of “Online Reconstruction of Retinal 3D map from Stereo in Close-to-Realtime for Eye Surgery” by S. Hecker, et al. (hereinafter Hecker) in view of “An Accurate Speckle 3D Reconstruction System Based on Binocular Endoscope” by Junguang Guo, et al. (hereinafter Guo).
Regarding claim 1, the limitations “A … visualization apparatus, comprising: two or more imaging devices configured to acquire … illuminated speckle images of an organ … from multiple different directions; and a processor which is configured to: … estimate spatial coordinates of speckles from the acquired … illuminated speckle images; generate from the estimated spatial coordinates of speckles a three-dimensional (3D) point cloud of locations; generate from the 3D point cloud of locations, a 3D map of an anatomical surface of the organ; and display the 3D map of the anatomical surface in real time” are taught by Gu (Gu, e.g. abstract, sections 1-4, discloses a dynamic 3D reconstruction system using a structured light sensor. Gu, e.g. figures 1, 3, 9, sections 1-3, teaches that the sensor comprises a speckle projection component for projecting speckles onto an object surface, and two cameras capturing images of the object surface from stereoscopically offset viewpoints, i.e. as claimed the two imaging device acquire speckle images from multiple different directions, where the subject can be a human organ, e.g. in figure 17 a human head is the reconstructed object. Further, Gu, e.g. section 1, paragraph 1, sections 2.2- 2.4, teaches that stereo matching is performed using the speckles in the captured images to estimate the lateral and depth coordinates of matched speckles, which can be used to triangulate a 3D point cloud, e.g. figures 15-17, middle column show reconstructed point clouds, i.e. as claimed a 3D point cloud of locations is generated from the estimated coordinates of the matched speckles in the speckle images. Finally, Gu, e.g. sections 3, 3.4, 3.5, figures 15-17, teaches that 3D models can be generated from the reconstructed and textured point cloud and displayed at a frame rate of 30 frames per second, i.e. the 3D model corresponds to the claimed 3D map of the anatomical surface of the organ, which can be displayed in real time.)
The limitations “A medical visualization apparatus, comprising: two or more imaging devices configured to acquire … illuminated speckle images of an organ of a patient from multiple different directions; and a processor, which is configured to: … display the 3D map of the anatomical surface in real time according to a requested gazing direction relative to the organ” are implicitly taught by Gu (While, as discussed above, Gu’s system allows for generating a 3D map/model from a 3D point cloud generated from stereoscopic speckle images, as claimed, and Gu’s system can be used to model human organs, e.g. the mouth, nose, ears, and eyes of the human in figure 17, Gu does not explicitly discuss using the structured light sensor in a medical context, i.e. the claimed medical visualization of a patient. It is noted that this difference is merely an intended use, i.e. Gu’s system would not require any modification to be used by a medical professional for generating 3D models of a patient. Further, while Gu indicates that 3D models are reconstructed for each captured frame, e.g. figures 15-17, sections 3.4, 3.5, Gu does not discuss viewpoint control, per se, although it is noted that one of ordinary skill in the art would have found it implicit, if not inherent, that Gu’s reconstructed 3D models could be displayed from any requested viewing direction. In the interest of compact prosecution, Reiter is cited for teaching that a structured light reconstruction system can be used as part of a medical procedure, and providing user controls allowing control of the virtual viewpoint and direction.) However, these limitations are taught by Reiter (Reiter, e.g. abstract, paragraphs 40-106, discloses a surgical structured light system which is used to capture 3D point clouds of organs for interactive display during a surgical procedure. Note: the cited Reiter patent application publication substantially comprises the Reiter non-patent literature publication previously applied in the 7/15/25 Office Action, e.g. Reiter, figures 1-9, 84-102, correspond to the NPL publication’s figures and section II B and section III. Reiter, e.g. figures 4, 8, 9, shows that the structured light reconstruction can be applied to different types of organs. Further, Reiter, e.g. figures 5-8, paragraphs 94-98, teaches that it is beneficial to provide controls for virtual transformations of the reconstructed model, i.e. allowing the user to request viewpoints having different location, rotation, or scale relative to the 3D model, corresponding to the claimed display of the 3D map of the anatomical surface according to a requested gazing direction relative to the organ, where analogous to Gu, as noted above, Reiter teaches that the reconstruction of the organ can be provided in real-time, e.g. paragraphs 3, 18-27, 40-43, 47.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s structured light 3D reconstruction system to include Reiter’s user viewpoint controls in order to allow a user to control the relative viewpoint and orientation used for displaying the reconstructed 3D models. Further, as noted above, Gu’s system would not require any modification to be used by a medical professional for generating 3D models of a patient, such that Gu’s modified system providing user viewpoint controls could be used for medical visualization of a patient’s organ according to a requested relative viewing direction, as claimed.
The limitations “A patient visualization apparatus, comprising: two or more imaging devices configured to acquire flash LED illuminated speckle images of an organ of a patient from multiple different directions; and a processor which is configured to: … estimated spatial coordinates of speckles from the acquired flash LED illuminated speckle images” are implicitly taught by Gu in view of Reiter (Gu, e.g. figure 1, teaches using an infrared laser with the speckle pattern projector for illuminating the subject, and does not discuss flashing the laser, per se. Further, while not relied on in the above modification of Gu to use Reiter’s user viewpoint controls, Reiter, e.g. paragraphs 62, 69, teaches that an infrared LED can be used as an alternative to an infrared laser illumination source, thereby motivating one of ordinary skill in the art to modify Gu’s system to substitute an LED for the laser. Further, Reiter, e.g. paragraphs 45, 49, 59, teaches that visible light illumination can be flashed at a high speed so as not to be perceived by a human viewer but still detected by the camera, but Reiter suggests this as an alternative to the infrared illumination. While one of ordinary skill in the art would have found it implicit that the infrared LED illumination source could also be flashed rather than continuous, i.e. as is common with illumination in both conventional photography and structured light systems, in the interest of compact prosecution, Muhlhoff is cited for explicitly teaching flashing an LED light source in a structured light scanning system.) However, this limitation is taught by Reiter in view of Muhlhoff (As noted above, Reiter, e.g. paragraphs 62, 69, teaches that an infrared LED can be used as an alternative to an infrared laser illumination source, thereby motivating one of ordinary skill in the art to modify Gu’s system to substitute an LED for the laser. Further, Reiter, e.g. paragraphs 45, 49, 59, teaches that visible light illumination can be flashed at a high speed so as not to be perceived by a human viewer but still detected by the camera, but Reiter suggests this as an alternative to the infrared illumination. Muhlhoff, e.g. abstract, paragraphs 53-94, describes two embodiments of a high speed tire deformation scanning system, including the second embodiment based on structured light, e.g. paragraphs 83-94. Muhlhoff, e.g. paragraphs 60, 66, 74-77, 84, 85, 91-94, teaches that the illumination/projection means may use a flashed LED turned on and off to generate speckled projection images captured by the camera, where Muhlhoff, e.g. paragraphs 77, 93 emphasizes the importance of keeping the illumination period short in order to ensure that the images are sharp and suitable for processing, i.e. as noted above, one of ordinary skill in the art would have found it implicit that the infrared LED illumination source could also be flashed rather than continuous.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s structured light 3D reconstruction system, including Reiter’s user viewpoint controls, to substitute an LED for the laser illumination source as taught by Reiter, controlled to flash for very short periods as taught by Muhlhoff, because Reiter teaches the substitution of the LED for the laser as discussed above, and Reiter suggests using high frequency bursts of illumination, where one of ordinary skill in the art would have found it implicit that the infrared LED illumination source could be flashed rather than continuous, i.e. as taught by Muhlhoff a short illumination time period leads to sharper images which are more suitable for processing. In Gu’s modified system, the infrared LED light source would be flashed in synchronization with the cameras, i.e. at up to 60 times per second as in Gu, section 3, paragraph 1.
The limitation “a processor, which is configured to: control movement of the two or more imaging devices in at least one direction to align the two or more imaging devices with respect to the organ for acquiring the flash LED illuminated speckle images” is not explicitly taught by Gu in view of Reiter and Muhlhoff (As discussed above, Gu’s modified system providing user viewpoint controls could be used for medical visualization of a patient’s organ according to a requested relative viewing direction, and in Gu’s modified system, the infrared LED light source would be flashed in synchronization with the cameras. While the cameras and flash LED in Gu’s modified system are easily movable, i.e. as shown in figure 9, the entire assembly is only a few inches in size, Gu does not address positioning the camera/light source assembly with respect to the subject, or more specifically the processor being used to control movement of the camera/light source assembly to align the cameras with a subject. In the interest of compact prosecution, Urban is cited for teaching a surgical microscope control system for controlling movement of a surgical microscope to a predefined orientation/position relative to a patient organ, in view of Hecker and Guo collectively teaching that one of ordinary skill in the art would recognize that Gu’s structured light 3D reconstruction system can be incorporated into a into a digital microscope embodiment.) However, this limitation is taught by Urban in view of Hecker and Guo (Urban, e.g. abstract, paragraphs 1-32, describes a system for controlling a surgical microscope, which is controlled by a processor executing a program as part of a surgical navigation system, e.g. paragraphs 8, 18, 25, 31 and uses a tracking system, motors, brakes, e.g. paragraphs 4, 9, 10, 17, 22-24, 29, 30, to control the movement and articulation of the arm holding the microscope at the distal end in order to return the optics/microscope to a previously stored position/orientation relative to a patient/patient’s organ, e.g. paragraphs 4-8, 11-16, 26-31. Further, Hecker, e.g. sections Introduction, Materials and Methods, Results, and Discussions and Conclusions, describes a surgical system for performing retinal surgery using stereo digital microscopes, including performing stereo matching and point cloud construction in real-time, where the reconstructed surface is displayed during the surgical procedure. That is, Hecker’s system, analogous to Gu, performs stereo point cloud construction, and analogous to Reiter, relies on a microscope embodiment for generating models of organs. Furthermore, while one of ordinary skill in the art would recognize that a digital microscope embodiment could require using a smaller structured light sensor designed for a shorter working distance from the sensor, i.e. Gu, section 3, paragraph 1, indicates a minimum efficient working distance of .03m or 30mm, Guo, e.g. abstract, sections I-III describes an analogous speckle pattern based stereo structured light reconstruction sensor which is embodied in an endoscope, e.g. figures 1, 6, having a smaller camera baseline, e.g. section I, paragraph 3, requiring a modified calibration and epipolar rectification technique, e.g. section II, but provides a smaller minimum efficient working distance of .005m or 5mm, e.g., section III, paragraph 1. That is, one of ordinary skill in the art would recognize that Guo’s endoscopic sensor embodiment and modified calibration/rectification technique would allow Gu’s structured light 3D reconstruction system to be incorporated into a digital microscope embodiment analogous to Hecker’s stereo reconstruction digital microscope, thereby providing a real-time structured light 3D reconstruction system for a surgical procedure, analogous to Hecker’s system.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s structured light 3D reconstruction system, including Reiter’s user viewpoint controls, substituting a flashed LED for the laser illumination source as taught by Reiter in view of Muhlhoff, to be used as the microscope in Urban’s processor controlled surgical microscope, by incorporating Gu’s modified structured light 3D reconstruction system into a digital microscope embodiment for performing stereo reconstruction during a surgical procedure as taught by Hecker, using Guo's description of the endoscopic sensor embodiment and modified calibration/rectification technique to perform stereo reconstruction with a small minimum efficient working distance as low as 5mm. In the modified system, the digital microscope embodiment of Gu’s modified structured 3D light reconstruction system would be mounted at the distal end of Urban’s articulated arm, e.g. paragraph 22, and the same software interface providing Reiter’s viewpoint controls would additionally support Urban’s navigation controls, e.g. Urban, paragraph 25, including allowing the user to return to a stored previous position/orientation with respect to the patient/patient’s organ(s), e.g. Urban, paragraphs 26-27, such that the processor in the modified system would both control movement of the microscope cameras and flash LEDs as taught by Urban, and perform the processing and display steps taught by Gu in view of Reiter.
Regarding claims 2 and 7, the limitations “wherein the processor is configured to generate a video stream of the organ and overlay the video stream onto the 3D map” and “wherein the imaging devices are configured to acquire a stream of the speckle images at a video rate, and wherein the processor is configured to generate and display the 3D map at the video rate” are taught by Gu (Gu, e.g. section 3, paragraph 1, indicates that the stereo cameras capture images at up to 60 frames per second, i.e. the video stream of speckle images is captured at a video rate. Further, Gu, e.g. section 3.5, figures 16, 17, teaches that the whole pipeline of 3D reconstruction can be performed at over 30 frames per second, where each reconstructed 3D model is overlaid with the corresponding color content of the captured images as shown in the two right side columns of figures 16 and 17. That is, as recited in claim 2, the video stream of the organ is superimposed onto the 3D map, each pair of captured images being used to color the 3D model reconstructed from the pair of captured images, and as recited in claim 7, a stream of the speckle images is captured at the video rate, and the 3D model is reconstructed and displayed at the video rate, where the video rate is the under 30ms per frame, or over 30 frames per second.)
Regarding claim 8, the limitation “wherein the processor is further configured to, in response to a request to change the gazing direction, display the 3D map of the anatomical surface in real time as viewed from a new direction” is taught by Gu in view of Reiter (As discussed in the claim 1 rejection above, Gu’s structured light 3D reconstruction system would be modified to include Reiter’s user viewpoint controls in order to allow a user to control the relative viewpoint and orientation used for displaying the reconstructed 3D models, i.e. a requested change in the relative viewpoint or orientation would cause the system to render the subsequent image(s) from the changed relative viewpoint or orientation, as claimed.)
Regarding claim 9, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claims 1, 2, 7, and 8 above, i.e. as discussed in the claim 8 rejection, Gu’s modified system is responsive to requested changes in viewpoint/orientation, and as discussed in the claims 2 and 7 rejection, Gu’s system performs capture, reconstruction, and display at a video rate of over 30 fps, such that Gu’s modified system would be responsive to viewpoint change requests at the video rate, i.e. once the input is received by the system, the change can be incorporated for the subsequently rendered frame, where as discussed in the claim 1 rejection, the reconstruction and display can be performed in real time.
Regarding claim 11, the limitation “wherein the organ is an eye” is taught by Gu (Gu, e.g. figure 17, shows an example where the upper body of a human is reconstructed, including their head. Although Gu protects the subject’s privacy by blanking part of the color images/rendered models, the recovered point clouds, e.g. second and third columns, do not have holes, i.e. the eyes are reconstructed as part of the recovered point clouds. Alternately, if the recovered point cloud did not include the eyes, there would be no need to blank the rendered images of the point clouds with texture or 3D models in the right most columns. Furthermore, in the claim 1 modification in view of Urban, Hecker, and Guo, the digital microscope embodiment of Gu’s modified structured 3D light reconstruction system would be mounted at the distal end of Urban’s articulated arm, and would be used for performing eye surgery, i.e. the organ would be the patient’s eye.)
Regarding claim 12, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 1 above.
Regarding claim 13, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 2 above.
Regarding claim 18, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 7 above.
Regarding claim 19, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 8 above.
Regarding claim 20, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 9 above.
Regarding claim 22, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 11 above.
Claims 3, 5, 6, 14, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over “Single-Shot Structured Light Sensor for 3D Dense and Dynamic Reconstruction” by Feifei Gu, et al. (hereinafter Gu) in view of U.S. Patent Application Publication 2014/0336461 A1 (hereinafter Reiter) in view of U.S. Patent Application Publication 2007/0204684 A1 (hereinafter Muhlhoff) in view of U.S. Patent Application Publication 2012/0320186 A1 (hereinafter Urban) in view of “Online Reconstruction of Retinal 3D map from Stereo in Close-to-Realtime for Eye Surgery” by S. Hecker, et al. (hereinafter Hecker) in view of “An Accurate Speckle 3D Reconstruction System Based on Binocular Endoscope” by Junguang Guo, et al. (hereinafter Guo) as applied to claims 1 and 12 above, and further in view of “Real-time 3D semi-local surface patch extraction using GPGPU Application to 3D object recognition” by Sergio Orts-Escolano, et al. (hereinafter Orts-Escolano).
Regarding claim 3, the limitations “wherein the processor is configured to generate the 3D map of the anatomical surface by performing at least the steps of: estimating lateral coordinates of the speckles; performing depth analysis to estimate depth coordinates of the speckles; aggregating estimated spatial locations of the speckles into a 3D point cloud set” are taught by Gu (Gu, e.g. sections 2-2.4, describes the details of calibrating the cameras in order to generate rectified images allowing for feature matching, i.e. matching corresponding speckles captured in the images, to be conducted along the rectified epipolar lines as shown in figure 4, corresponding to estimating lateral coordinates of speckles in the speckle images, i.e. each matched feature is a speckle having a location in the left view image and a location in the right view image. Gu, e.g. section 2.4, indicates that feature matching allows for determining an optimal disparity map D, which corresponds to a depth estimate of the matched features, i.e. increased magnitude of disparity correlates to increased depth, and Gu, e.g. figures 15-17, second column, indicates that an estimated depth map is determined. Finally, Gu, e.g. section 2.4, paragraph 7, indicates that after stereo matching, triangulation is used to recover the 3D point cloud, as in figures 15-17, middle column.)
The limitations “dividing the 3D point cloud data set into unit areas, and estimate respective orientations of the unit areas; and using the orientations of the unit areas, reconstructing from the 3D point cloud a mesh approximation of a variably curved surface in 3D space” are not explicitly taught by Gu (Gu, e.g. figures 15-17, right column, indicates that the 3D point clouds with textures are used to generate a 3D model. Gu does not explicitly indicate the nature of the 3D model, i.e. it is implicitly distinct from the point cloud with texture in the fourth column, but Gu does not further describe how the 3D model is generated, i.e. using estimated orientations of the unit areas of the point cloud as claimed, or whether the 3D model is a mesh, per se, as claimed.) However, these limitations are taught by Orts-Escolano (Orts-Escolano, e.g. abstract, sections 1-5, describes a system for performing real-time 3D mesh reconstruction from 3D sensor based range maps using a GPU, including from structured light sensors, e.g. section 2.2, paragraph 1. Orts-Escolano, e.g. section 2, describes the processing steps including generating a 3D point cloud from the input data, e.g. section 2.1, generating normals for each point in the 3D point cloud, e.g. section 2.1.2, which provides the necessary information for generating a triangle mesh using the points of the point cloud, e.g. section 2.2. Orts-Escolano, e.g. section 2.2, paragraph 4, indicates that the GPU implementation achieves a 3D mesh reconstruction rate of approximately 30 frames per second, i.e. analogous to Gu’s 3D reconstruction system, generates 3D models at 30 frames per second. It is noted that the claimed unit areas are interpreted as corresponding to points of the point cloud, i.e. each point has a determined normal as in section 2.1.2, which is used in defining triangles from the points as in section 2.2, equation 5, where an edge validity for two points depends, in part, on the angle between the normal vectors of the points.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s structured light 3D reconstruction system, including Reiter’s user viewpoint controls, substituting a flashed LED for the laser illumination source as taught by Reiter in view of Muhlhoff, incorporated into a digital microscope embodiment in view of Hecker and Guo, being used as the microscope in Urban’s processor controlled surgical microscope, to use Orts-Escolano’s real-time approximate mesh reconstruction technique for generating 3D mesh models from Gu’s reconstructed point clouds with texture because, as noted above, Gu indicates that the 3D point clouds with textures are used to generate a 3D model, but does not describe how the 3D model is generated, and Orts-Escolano discloses an analogous technique for generating 3D mesh models from 3D point clouds determined from structured light sensors at a rate of 30 frames per second, including the details of generating the 3D mesh model from the point cloud, i.e. by estimating per-point normals used to determine which pairs of points have valid edges for defining the triangle mesh elements, such that one of ordinary skill in the art would be motivated to look to Orts-Escolano for details of implementing Gu’s 3D model generation step. In Gu’s modified system, the output 3D models, e.g. right column of figures 15-17, would be determined from the 3D point clouds with texture using Orts-Escolano’s technique, i.e. estimating point cloud normals as in section 2.1.2, and triangle mesh elements having edges determined to be valid in part based on the estimated normals, corresponding to the claimed using estimating unit area orientations to reconstruct a mesh approximation of the 3D point cloud.
Regarding claim 5, the limitation “wherein the mesh approximation is a triangular mesh” is taught by Gu in view of Orts-Escolano (As discussed in the claim 3 rejection above, in Gu’s modified system, the output 3D models, e.g. right column of figures 15-17, would be determined from the 3D point clouds with texture using Orts-Escolano’s technique, i.e. estimating point cloud normals as in section 2.1.2, and triangle mesh elements having edges determined to be valid in part based on the estimated normals, i.e. the resulting mesh approximation is a triangular mesh.)
Regarding claim 6, the limitation “wherein edges of the mesh approximation are locations of a subset of the data points of the 3D point cloud” is taught by Gu in view of Orts-Escolano (As discussed in the claim 3 rejection above, in Gu’s modified system, the output 3D models, e.g. right column of figures 15-17, would be determined from the 3D point clouds with texture using Orts-Escolano’s technique, i.e. estimating point cloud normals as in section 2.1.2, and triangle mesh elements having edges determined to be valid in part based on the estimated normals, i.e. the included edges are defined using the locations/points of the point cloud. It is noted that the claim uses the term “subset”, and the broadest reasonable interpretation of the “subset” would include using all of the points of the point cloud, i.e. by definition, a set is a subset of itself. However, it is additionally noted that although Gu does not address limiting the 3D model to less than the entire set of points in the point cloud, Orts-Escolano, e.g. section 2.1.1, suggests a noise removal operation could be performed on the 3D point cloud, i.e. removing a subset of points determined to be noise, leaving a remaining subset of points for use in defining the mesh, such that even if the claim were amended to require using less than the entire set of data points of the 3D point cloud, the narrower scope would still be obvious in view of Orts-Escolano’s noise removal operation.)
Regarding claim 14, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 3 above.
Regarding claim 16, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 5 above.
Regarding claim 17, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 6 above.
Claims 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over “Single-Shot Structured Light Sensor for 3D Dense and Dynamic Reconstruction” by Feifei Gu, et al. (hereinafter Gu) in view of U.S. Patent Application Publication 2014/0336461 A1 (hereinafter Reiter) in view of U.S. Patent Application Publication 2007/0204684 A1 (hereinafter Muhlhoff) in view of U.S. Patent Application Publication 2012/0320186 A1 (hereinafter Urban) in view of “Online Reconstruction of Retinal 3D map from Stereo in Close-to-Realtime for Eye Surgery” by S. Hecker, et al. (hereinafter Hecker) in view of “An Accurate Speckle 3D Reconstruction System Based on Binocular Endoscope” by Junguang Guo, et al. (hereinafter Guo) in view of “Real-time 3D semi-local surface patch extraction using GPGPU Application to 3D object recognition” by Sergio Orts-Escolano, et al. (hereinafter Orts-Escolano) as applied to claims 3 and 14 above, and further in view of U.S. Patent Application Publication 2015/0077528 A1 (hereinafter Awdeh).
Regarding claim 4, the limitation “wherein the processor is further configured to graphically indicate different anatomical portions of the variable curved surface” is not explicitly taught by Gu (Gu does not address graphically indicating anatomical portions of the reconstructed 3D model, per se. As discussed in the claim 1 rejection above, Gu’s modified structured light 3D reconstruction system incorporated into a digital microscope embodiment in view of Hecker and Guo, being used as the microscope in Urban’s processor controlled surgical microscope, i.e. in the modified system, the organ is an eye and the reconstruction is being performed during eye surgery.) However, this limitation is taught by Awdeh (Awdeh, e.g. abstract, paragraphs 43-149, discloses a system for real time acquisition and display of image data to an operator performing a surgical procedure on an eye using a digital microscope, e.g. paragraphs 43, 44, 46-48. Awdeh, e.g. paragraphs 61, 64, 72, 73, further teaches that anatomical landmarks or regions can be identified in the image data of the patient’s organ, and be color coded or otherwise patterned to distinguish the anatomical regions, such as color coding a patient’s cornea, limbus, iris, pupil, etc., as in the case of eye surgery, corresponding to the claim limitation of graphically indicating different anatomical portions of an organ surface.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s structured light 3D reconstruction system, including Reiter’s user viewpoint controls, substituting a flashed LED for the laser illumination source as taught by Reiter in view of Muhlhoff, incorporated into a digital microscope embodiment in view of Hecker and Guo, being used as the microscope in Urban’s processor controlled surgical microscope, using Orts-Escolano’s real-time approximate mesh reconstruction technique, to perform Awdeh’s anatomical region color-coding technique in order to graphically indicate to the surgeon/operator the different anatomical regions of the reconstructed 3D mesh model.
Regarding claim 15, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 4 above.
Claims 10 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over “Single-Shot Structured Light Sensor for 3D Dense and Dynamic Reconstruction” by Feifei Gu, et al. (hereinafter Gu) in view of U.S. Patent Application Publication 2014/0336461 A1 (hereinafter Reiter) in view of U.S. Patent Application Publication 2007/0204684 A1 (hereinafter Muhlhoff) in view of U.S. Patent Application Publication 2012/0320186 A1 (hereinafter Urban) in view of “Online Reconstruction of Retinal 3D map from Stereo in Close-to-Realtime for Eye Surgery” by S. Hecker, et al. (hereinafter Hecker) in view of “An Accurate Speckle 3D Reconstruction System Based on Binocular Endoscope” by Junguang Guo, et al. (hereinafter Guo) as applied to claims 1 and 12 above, and further in view of “Development and Evaluation of a Miniature Trinocular Camera System for Surgical Measurement Applications” by Niklas Conen, et al. (hereinafter Conen).
Regarding claim 10, the limitation “wherein the imaging devices comprise three digital microscopes” is partially taught by Gu in view of Hecker and Guo (In the claim 1 modification, Gu’s system is incorporated into a digital microscope embodiment for performing stereo reconstruction during a surgical procedure as taught by Hecker, using Guo's description of the endoscopic sensor embodiment and modified calibration/rectification technique to perform stereo reconstruction with a small minimum efficient working distance as low as 5mm. While Gu, Hecker, and Guo all rely on stereo structured light imaging techniques, i.e. two cameras, or microscopes as in Hecker and Guo, the references do not address using a third microscope.) However, this limitation is taught by Conen (Conen, e.g. abstract, sections 1, 3-6, describes a trinocular microscope system for performing real-time 3D reconstruction of patients for minimally invasive surgery applications. Conen, e.g. abstract, section 4.3, teaches that the reconstruction uses a semi-global matching process, i.e. analogous to Gu, sections 1, 2.2-2.4 performing stereo reconstruction using a semi-global matching process, where Conen, e.g. section 3.1, paragraph 2, sections 4.1, 4.2, 5.1, 5.3-5.5, figures 7, 8, teaches that the trinocular reconstruction is similar to stereo reconstruction, except extended to the third image/viewpoint, and can achieve better reconstruction results due to the additional viewpoint.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gu’s structured light 3D reconstruction system, including Reiter’s user viewpoint controls, substituting a flashed LED for the laser illumination source as taught by Reiter in view of Muhlhoff, incorporated into a digital microscope embodiment in view of Hecker and Guo, being used as the microscope in Urban’s processor controlled surgical microscope, to use Conen’s trinocular camera and trinocular reconstruction technique in order to improve the 3D reconstruction results by incorporating information from the third viewpoint. In Gu’s modified system, in addition to using an additional camera in the trinocular arrangement taught by Conen, the stereo reconstruction technique would be extended to match with the third viewpoint image to improve the reconstruction result as taught by Conen, i.e. Gu’s modified system would have, as claimed, three digital microscopes.
Regarding claim 21, the limitations are similar to those treated in the above rejection(s) and are met by the references as discussed in claim 10 above.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-22 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT BADER whose telephone number is (571)270-3335. The examiner can normally be reached 11-7 m-f.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT BADER/Primary Examiner, Art Unit 2611
Prosecution Insights