DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 5-6, 8, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Schneider et al. (US 2013/0338505) in the view Song et al. (NPL: “MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing”).
Regarding claim 1, Schneider teaches an imaging system comprising (fig. 7, para. 0021; robotic system, patient and ultrasound probe):
a laparoscopic ultrasound probe configured to be inserted through an access port and to obtain a plurality of 2D ultrasound images of a tissue (fig. 7, para. 0049; The ultrasound probe 100 will be placed through an incision but outside the cannula lumen. This means that no extra incision is needed in the patient's abdominal wall 753);
a laparoscopic camera configured to capture a video stream of the tissue (fig. 7, para. 0049; The laparoscope 720 is the camera that is used in laparoscopic minimally invasive surgery to allow the surgeon to see inside the patient's abdominal cavity 751. In various embodiments of the invention, the laparoscope 720 can be monocular or stereoscopic.);
an image processing device configured to (para. 0050; The ultrasound machine 701 is capable of processing the electrical signals from the transducer array 105 and collecting the ultrasound data which can be processed in different ways to produce several image types, including B-mode images, Doppler images and continuous wave Doppler audio.):
generate a depth map of a surgical site from the video stream to estimate a 3D location of the laparoscopic ultrasound probe in frame of reference of the laparoscopic camera (paras. 0004, 0051, and claims 5 and 8; a stereo camera system, meaning the surgeon can see his/her tools and the anatomy in three dimensions, thus regaining depth perception. Optical markers on the exterior surface of the probe housing that that can be localized by an endoscopic camera. identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system. The examiner notes that the stereo camera generate depth maps and the laparoscopic probe is tracked using the optical markers attached to the probe and shown in the video.);
localizing the laparoscopic ultrasound probe in the cameras coordinate system (claim 8; identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system.);
receive a volumetric image of tissue formed from a first modality images (para. 0052; Visualization of pre-operative images, such as CT or MRI volumes, can also be provided with this invention.);
generate an ultrasound volume from the plurality of 2D ultrasound images (para. 0051; It is also possible to combine a sequence of ultrasound data into 3D volumes.);
register the ultrasound volume with the volumetric image of the tissue (paras. 0052-0053; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
generate an overlay of the volumetric image of tissue and a 2D ultrasound image of the plurality of 2D ultrasound images (para. 0052; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
a screen configured to display the video stream showing the laparoscopic ultrasound probe and the overlay extending from the laparoscopic ultrasound probe (paras. 0051-0053; t is possible to display the ultrasound data, such a B-mode ultrasound image 104 to the surgeon in its correct spatial location with respect to the ultrasound probe 100. It is also possible to combine a sequence of ultrasound data into 3D volumes. These volumes can be visualized by the surgeon through conventional rendering techniques. In some embodiment of the invention, this visualization could be accomplished through stereographic video systems and displayed through a robotic surgeon's console. When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.).
However, Schneider fails to disclose localizing the laparoscopic camera in a world coordinate system based on the depth map.
Song, in the same field of endeavor, teaches localizing the laparoscopic camera in a world coordinate system based on the depth map (pages 1-2, introduction section, Inspired by the research done by Mahmoud et al. [4] [5] [13] which demonstrate the robustness of camera pose estimation from ORB-SLAM, we figure out ORB-SLAM is suitable to be improved and coupled with dense deformable SLAM. In this paper, we propose MIS-SLAM which builds on our preliminary work [12] with the following major improvements: (1) We proposed a heterogeneous framework to make full use of both GPU (dense deformable SLAM) and CPU (ORB-SLAM) to recover the dense deformed 3D structure of the soft-tissues in MIS scenario. Computational power of CPU is fully exploited to run an improved ORB-SLAM to provide complementary information to GPU modules. (2) Modules from GPU and CPU are deeply integrated to boost performance and enhance the efficiency. Sparse ORB features as well as global pose are uploaded to GPU).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider to include SIM-SLAM pipeline to determine the position and orientation of the laparoscopic camera in world coordinate system of Song because it increase the accuracy of pose estimation of fast moving camera and improves registration as disclosed within Song in introduction section.
Regarding claim 5, Schneider teaches the imaging system according to claim 1, wherein the laparoscopic camera is a stereoscopic camera and the video stream includes a pair of stereo image streams (para. 0049; The laparoscope 720 is the camera that is used in laparoscopic minimally invasive surgery to allow the surgeon to see inside the patient's abdominal cavity 751. In various embodiments of the invention, the laparoscope 720 can be monocular or stereoscopic.).
Regarding claim 6, Schneider teaches the imaging system according to claim 5, wherein the image processing device is further configured to localize the laparoscopic ultrasound probe in the cameras coordinate system (claim 8).
However, fails to explicitly teach localize the laparoscopic camera in the world coordinate system using a Deformable Visual Simultaneous Localization and Mapping (DV-SLAM) pipeline based on the pair of stereo image streams.
Song, in the same field of endeavor, teaches the laparoscopic camera in the world coordinate system using a Deformable Visual Simultaneous Localization and Mapping (DV-SLAM) pipeline based on the pair of stereo image streams (pages 1-2, introduction section, To broaden surgeons’ field of view, 3D laparoscopy, or binoculars is applied to generate 2 images from different viewing point so that a 3D geometry based on parallax is created in surgeons’ mind for better understanding of the environment. Recently, similar stereo vision is adopted by some AR devices for enhancing MIS procedures. Therefore, it will be very helpful if stereo vision related approaches in computer vision community could be integrated, extended and improved to recover deformable shape in real-time while estimating the pose of the camera. In our previous work, we proposed a dynamic reconstruction of deformable soft-tissue with stereo scope [12]. A warping field based on the embedded deformation nodes approach is introduced with 3D dense shapes recovered from sequences pairs of stereo images. With the help of general-purpose computing on graphics processing units (GPGPU), all the processes are achieved in real-time. Mentioned in [12], the first and most important challenge to the pipeline is the fast movement of scope. Fast movement not only makes visual odometry unstable but also causes blurry images which make registrations even worse. This issue has also been reported in [2] and [3]. Inspired by the research done by Mahmoud et al. [4] [5] [13] which demonstrate the robustness of camera pose estimation from ORB-SLAM, we figure out ORB-SLAM is suitable to be improved and coupled with dense deformable SLAM. In this paper, we propose MIS-SLAM which builds on our preliminary work [12] with the following major improvements: (1) We proposed a heterogeneous framework to make full use of both GPU (dense deformable SLAM) and CPU (ORB-SLAM) to recover the dense deformed 3D structure of the soft-tissues in MIS scenario. Computational power of CPU is fully exploited to run an improved ORB-SLAM to provide complementary information to GPU modules. (2) Modules from GPU and CPU are deeply integrated to boost performance and enhance the efficiency. Sparse ORB features as well as global pose are uploaded to GPU).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider to include SIM-SLAM pipeline to determine the position and orientation of the laparoscopic camera in world coordinate system of Song because it recovers deformable shape in real-time while estimating the pose of fast moving camera without any blurring in the images which improves registration as disclosed within Song in introduction section.
Regarding claim 8, Schneider teaches the imaging system according to claim 1, wherein the image processing device is further configured to generate the ultrasound volume by matching a plurality of key points in each 2D ultrasound image of the plurality of 2D ultrasound images (paras. 0051 and 0054; Because the ultrasound probe 100 can be tracked, it is possible to display the ultrasound data, such a B-mode ultrasound image 104 to the surgeon in its correct spatial location with respect to the ultrasound probe 100. It is also possible to combine a sequence of ultrasound data into 3D volumes. The ultrasound images 104 can be stitched together to make a 3D volume along the direction of travel 801 if the motion is in the elevation direction (out-of-plane)); and provide visual guidance encompassing a tumor along with a negative resection margin around the tumor (para. 0005; Ultrasound is currently used during liver resection to localize and stage the lesions as well as determine the extent of the resection required. Similarly, ultrasound is used during partial nephrectomy to determine the resection margins and identify proper blood flow through the kidney after the tumour is removed.).
Regarding claim 19, Schneider teaches a method for intraoperative imaging of tissue, the method comprising (fig. 7, para. 0021; robotic system, patient and ultrasound probe):
obtaining a plurality of 2D ultrasound images from a laparoscopic ultrasonic probe (fig. 7, paras. 0049 and 0051; The ultrasound probe 100 will be placed through an incision but outside the cannula lumen. This means that no extra incision is needed in the patient's abdominal wall 753);
obtaining a video stream from a laparoscopic camera (fig. 7, para. 0049; The laparoscope 720 is the camera that is used in laparoscopic minimally invasive surgery to allow the surgeon to see inside the patient's abdominal cavity 751. In various embodiments of the invention, the laparoscope 720 can be monocular or stereoscopic.);
generating a depth map of a surgical site from the video stream to estimate a 3D location of the laparoscopic ultrasound probe in frame of reference of the laparoscopic camera (paras. 0004, 0051, and claims 5 and 8; a stereo camera system, meaning the surgeon can see his/her tools and the anatomy in three dimensions, thus regaining depth perception. Optical markers on the exterior surface of the probe housing that that can be localized by an endoscopic camera. identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system. The examiner notes that the stereo camera generate depth maps and the laparoscopic probe is tracked using the optical markers attached to the probe and shown in the video.);
localizing the laparoscopic ultrasound probe in the cameras coordinate system (claim 8; identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system.);
receiving a 3D model of tissue formed from a first modality images (para. 0052; Visualization of pre-operative images, such as CT or MRI volumes, can also be provided with this invention.);
generating an ultrasound volume from the plurality of 2D ultrasound images (para. 0051; It is also possible to combine a sequence of ultrasound data into 3D volumes.);
registering the ultrasound volume with the 3D model (paras. 0052-0053; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
generate an overlay of the 3D model and a 2D ultrasound image of the plurality of 2D ultrasound images (para. 0052; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
displaying the video stream, the video stream including the laparoscopic ultrasound probe and the overlay including the 3D model and the 2D ultrasound image extending along an imaging plane of the laparoscopic ultrasonic probe (paras. 0051-0053; it is possible to display the ultrasound data, such a B-mode ultrasound image 104 to the surgeon in its correct spatial location with respect to the ultrasound probe 100. It is also possible to combine a sequence of ultrasound data into 3D volumes. These volumes can be visualized by the surgeon through conventional rendering techniques. In some embodiment of the invention, this visualization could be accomplished through stereographic video systems and displayed through a robotic surgeon's console. When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.).
However, Schneider fails to disclose localizing the laparoscopic camera in a world coordinate system based on the depth map.
Song, in the same field of endeavor, teaches localizing the laparoscopic camera in a world coordinate system based on the depth map (pages 1-2, introduction section, Inspired by the research done by Mahmoud et al. [4] [5] [13] which demonstrate the robustness of camera pose estimation from ORB-SLAM, we figure out ORB-SLAM is suitable to be improved and coupled with dense deformable SLAM. In this paper, we propose MIS-SLAM which builds on our preliminary work [12] with the following major improvements: (1) We proposed a heterogeneous framework to make full use of both GPU (dense deformable SLAM) and CPU (ORB-SLAM) to recover the dense deformed 3D structure of the soft-tissues in MIS scenario. Computational power of CPU is fully exploited to run an improved ORB-SLAM to provide complementary information to GPU modules. (2) Modules from GPU and CPU are deeply integrated to boost performance and enhance the efficiency. Sparse ORB features as well as global pose are uploaded to GPU).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider to include SIM-SLAM pipeline to determine the position and orientation of the laparoscopic camera in world coordinate system of Song because it increase the accuracy of pose estimation of fast moving camera and improves registration as disclosed within Song in introduction section.
Claim(s) 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Schneider et al. (US 2013/0338505) in the view Song et al. (NPL: “MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing”) in further view of Shekhar et al. (US 2020/0197102).
Regarding claim 2, modified Schneider teaches the imaging system according to claim 1, however, fails to explicitly teach wherein the image processing device is further configured to generate a plurality of virtual fiducial markers on the laparoscopic ultrasound probe using a deep learning image processing algorithm.
Shekhar, in the same field of endeavor, teaches generate a plurality of virtual fiducial markers on the laparoscopic ultrasound probe using a deep learning image processing algorithm (paras. 0055-0056; the computer vision-based object tracking workflow of the present disclosure describes a workflow for image line detection 526 and image point detection 527 following image pre-processing of a video image. First, a region of interest (ROI) can be generated for each frame of a first imaging modality using fast visual tracking based on robust discriminative correlation filters such that subsequent processing focuses on the imaging tip, as shown at No. 618 in FIG. 6A. Based on this coarse estimate of the probe's location, the bounding box surrounding the imaging tip can be intended to include at least some portion of the top, middle, and tip of the transducer as seen by the camera. In order to find the straight edges of these features of the transducer, the image can first be converted to a gray scale image based on brightness, followed by Canny edge detection, for instance, as shown in FIG. 6B. A Probabilistic Hough Transform (PHT) can be used to extract a first set of lines from the edge detection result within the ROI, an example of which is shown in FIG. 6C. As shown in FIG. 6D, the first line set can be filtered, for instance, by creating a coarse grain 2D histogram with the axes defined by PHT parameters (r, θ; magnitude and direction) and values of the histogram defined by the sum of the lengths of lines in the bin. All lines not contained within the highest peak present in the 2D histogram, therefore, can be removed to generate a second set of lines that corresponds with the long edges and is substantially parallel to an object of interest, which may be a projection of a transducer, or virtual object, or an actual transducer, or real object, according to the imaging modality. From this smaller, second set of lines, a fine grain 2D histogram, based on the PHT parameters (r, θ; magnitude and direction), can be created. The two highest peaks in this histogram, determined to be the longest lines, may represent the top and middle of the transducer of either the projection of the transducer or the actual transducer, and be selected. The indices of the peaks, defining the ROI of the first imaging modality, can then be used in one of at least one cost function for the optimization of the virtual object location. According to an embodiment, the above-described line feature extraction can be performed via PHT, as described above, or via custom designed markers including AR marker, checkerboard marker, dot marker, and the like, and machine learning-based binary segmentation of the tool including, among others, convolutional neural networks.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the optical markers of Schneider in the view of Song to include virtual AR markers to determine the position and orientation of the laparoscopic ultrasound probe in the video stream of Shekhar because it reduces a line of sight problem caused by optical markers tracking method and reduce registration errors within Shekhar in paras. 0004 and 0040.
Regarding claim 3, modified Schneider teaches the imaging system according to claim 2, however, fails to explicitly teach localize the laparoscopic ultrasound probe in the video stream based on the virtual fiducial markers.
Shekhar, in the same field of endeavor, teaches localize the laparoscopic ultrasound probe in the video stream based on the virtual fiducial markers (paras. 0055-0056; the computer vision-based object tracking workflow of the present disclosure describes a workflow for image line detection 526 and image point detection 527 following image pre-processing of a video image. First, a region of interest (ROI) can be generated for each frame of a first imaging modality using fast visual tracking based on robust discriminative correlation filters such that subsequent processing focuses on the imaging tip, as shown at No. 618 in FIG. 6A. Based on this coarse estimate of the probe's location, the bounding box surrounding the imaging tip can be intended to include at least some portion of the top, middle, and tip of the transducer as seen by the camera. In order to find the straight edges of these features of the transducer, the image can first be converted to a gray scale image based on brightness, followed by Canny edge detection, for instance, as shown in FIG. 6B. A Probabilistic Hough Transform (PHT) can be used to extract a first set of lines from the edge detection result within the ROI, an example of which is shown in FIG. 6C. As shown in FIG. 6D, the first line set can be filtered, for instance, by creating a coarse grain 2D histogram with the axes defined by PHT parameters (r, θ; magnitude and direction) and values of the histogram defined by the sum of the lengths of lines in the bin. All lines not contained within the highest peak present in the 2D histogram, therefore, can be removed to generate a second set of lines that corresponds with the long edges and is substantially parallel to an object of interest, which may be a projection of a transducer, or virtual object, or an actual transducer, or real object, according to the imaging modality. From this smaller, second set of lines, a fine grain 2D histogram, based on the PHT parameters (r, θ; magnitude and direction), can be created. The two highest peaks in this histogram, determined to be the longest lines, may represent the top and middle of the transducer of either the projection of the transducer or the actual transducer, and be selected. The indices of the peaks, defining the ROI of the first imaging modality, can then be used in one of at least one cost function for the optimization of the virtual object location. According to an embodiment, the above-described line feature extraction can be performed via PHT, as described above, or via custom designed markers including AR marker, checkerboard marker, dot marker, and the like, and machine learning-based binary segmentation of the tool including, among others, convolutional neural networks.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the optical markers of Schneider in the view of Song to include virtual AR markers to determine the position and orientation of the laparoscopic ultrasound probe in the video stream of Shekhar because it reduces a line of sight problem caused by optical markers tracking method and reduce registration errors within Shekhar in paras. 0004 and 0040.
Regarding claim 4, modified Schneider teaches the imaging system according to claim 3, however, fails to explicitly teach estimate a pose and orientation of the laparoscopic ultrasound probe from the video stream based on the virtual fiducial markers.
Shekhar, in the same field of endeavor, teaches estimate a pose and orientation of the laparoscopic ultrasound probe from the video stream based on the virtual fiducial markers (paras. 0055-0056; the computer vision-based object tracking workflow of the present disclosure describes a workflow for image line detection 526 and image point detection 527 following image pre-processing of a video image. First, a region of interest (ROI) can be generated for each frame of a first imaging modality using fast visual tracking based on robust discriminative correlation filters such that subsequent processing focuses on the imaging tip, as shown at No. 618 in FIG. 6A. Based on this coarse estimate of the probe's location, the bounding box surrounding the imaging tip can be intended to include at least some portion of the top, middle, and tip of the transducer as seen by the camera. In order to find the straight edges of these features of the transducer, the image can first be converted to a gray scale image based on brightness, followed by Canny edge detection, for instance, as shown in FIG. 6B. A Probabilistic Hough Transform (PHT) can be used to extract a first set of lines from the edge detection result within the ROI, an example of which is shown in FIG. 6C. As shown in FIG. 6D, the first line set can be filtered, for instance, by creating a coarse grain 2D histogram with the axes defined by PHT parameters (r, θ; magnitude and direction) and values of the histogram defined by the sum of the lengths of lines in the bin. All lines not contained within the highest peak present in the 2D histogram, therefore, can be removed to generate a second set of lines that corresponds with the long edges and is substantially parallel to an object of interest, which may be a projection of a transducer, or virtual object, or an actual transducer, or real object, according to the imaging modality. From this smaller, second set of lines, a fine grain 2D histogram, based on the PHT parameters (r, θ; magnitude and direction), can be created. The two highest peaks in this histogram, determined to be the longest lines, may represent the top and middle of the transducer of either the projection of the transducer or the actual transducer, and be selected. The indices of the peaks, defining the ROI of the first imaging modality, can then be used in one of at least one cost function for the optimization of the virtual object location. According to an embodiment, the above-described line feature extraction can be performed via PHT, as described above, or via custom designed markers including AR marker, checkerboard marker, dot marker, and the like, and machine learning-based binary segmentation of the tool including, among others, convolutional neural networks.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the optical markers of Schneider in the view of Song to include virtual AR markers to determine the position and orientation of the laparoscopic ultrasound probe in the video stream of Shekhar because it reduces a line of sight problem caused by optical markers tracking method and reduce registration errors within Shekhar in paras. 0004 and 0040.
Claim(s) 9, 12-13, 17-18, and 20-23 are rejected under 35 U.S.C. 103 as being unpatentable over Schneider et al. (US 2013/0338505) in the view Song et al. (NPL: “MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing”) and Pheiffer et al. (US 2019/0336109).
Regarding claim 9, modified Schneider teaches the imaging system according to claim 1, wherein the image processing device is further configured to generate a 3D model based on the volumetric image of tissue formed from the first modality images (para. 0052; Visualization of pre-operative images, such as CT or MRI volumes, can also be provided with this invention. When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.).
However, fails to explicitly teach deform the 3D model to conform to the ultrasound volume.
Pheiffer, in the same field of endeavor, teaches deform the 3D model to conform to the ultrasound volume (para. 0040; non-rigid registration is performed using a biomechanical tissue model with patient-specific boundary conditions and patient-specific material properties based on the intra-operative 3D B-mode US volume and the intra-operative 3D USE volume. The biomechanical tissue model simulates biomechanical deformation of the tissue of the target organ. According to an embodiment of the present invention, the biomechanical tissue model is used to drive the non-rigid registration by computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition. Patient-specific boundary conditions for the biomechanical tissue model are computed based on the intra-operative 3D USE volume and are used to constrain the simulated deformation of the pre-operative target organ such that positions of surface correspondences match on the deformed pre-operative target organ and the intra-operative target organ. Patient-specific material properties for the biomechanical tissue model, such as a distribution of spatially varying values of a tissue property, are determined for the pre-operative tissue domain based on the intra-operative 3D USE volume.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider in the view of Song to non-ridge registration for computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition of Pheiffer because it helps to accurately align organ structures in the pre-operative medical image data and the intra-operative US due within Pheiffer in para. 0031.
Regarding claim 12, Schneider teaches a surgical robotic system comprising (fig. 7, para. 0021; robotic system, patient and ultrasound probe):
a first robotic arm including a laparoscopic ultrasound probe configured to be inserted through an access port and to obtain intraoperatively a plurality of 2D ultrasound images of a tissue (fig. 7, paras. 0027 and 0049; the grasping interface 102 is 140 meant to be used with a grasping instrument such as the one shown in FIG. 5. The grasping interface 102 allows for the ultrasound probe 100 to be grasped by a robotic or laparoscopic grasping tool 500, where a grasping tool 500 is defined as having a first and second grasping jaws 503, 504 in an opposed facing relationship, with a variable grasping angle 550 between the grasping jaws 503, 504. The ultrasound probe 100 will be placed through an incision but outside the cannula lumen. This means that no extra incision is needed in the patient's abdominal wall 753);
a second robotic arm including a laparoscopic camera configured to capture a video stream of the tissue (fig. 7, paras. 0027 and 0049; the grasping interface 102 is 140 meant to be used with a grasping instrument such as the one shown in FIG. 5. The grasping interface 102 allows for the ultrasound probe 100 to be grasped by a robotic or laparoscopic grasping tool 500, where a grasping tool 500 is defined as having a first and second grasping jaws 503, 504 in an opposed facing relationship, with a variable grasping angle 550 between the grasping jaws 503, 504. The laparoscope 720 is the camera that is used in laparoscopic minimally invasive surgery to allow the surgeon to see inside the patient's abdominal cavity 751. In various embodiments of the invention, the laparoscope 720 can be monocular or stereoscopic.);
an image processing device configured to (para. 0050; The ultrasound machine 701 is capable of processing the electrical signals from the transducer array 105 and collecting the ultrasound data which can be processed in different ways to produce several image types, including B-mode images, Doppler images and continuous wave Doppler audio.):
generate a depth map of a surgical site from the video stream to estimate 3D location of the laparoscopic ultrasound probe in frame of reference of the laparoscopic camera (paras. 0004, 0051, and claims 5 and 8; a stereo camera system, meaning the surgeon can see his/her tools and the anatomy in three dimensions, thus regaining depth perception. Optical markers on the exterior surface of the probe housing that that can be localized by an endoscopic camera. identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system. The examiner notes that the stereo camera generate depth maps and the laparoscopic probe is tracked using the optical markers attached to the probe and shown in the video.);
localize the laparoscopic ultrasound probe in the cameras coordinate system (claim 8; identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system.);
receive a 3D model of tissue formed from a first modality images (para. 0052; Visualization of pre-operative images, such as CT or MRI volumes, can also be provided with this invention.);
generate an ultrasound volume from the plurality of 2D ultrasound images (para. 0051; It is also possible to combine a sequence of ultrasound data into 3D volumes.);
register the ultrasound volume with the 3D model of the tissue (paras. 0052-0053; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
generate an overlay of the 3D model of tissue and a 2D ultrasound image of the plurality of 2D ultrasound images (para. 0052; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
a screen configured to display the video stream showing the laparoscopic ultrasound probe and the overlay including the 3D model and the 2D ultrasound image extending along an imaging plane of the laparoscopic ultrasonic probe(paras. 0051-0053; it is possible to display the ultrasound data, such a B-mode ultrasound image 104 to the surgeon in its correct spatial location with respect to the ultrasound probe 100. It is also possible to combine a sequence of ultrasound data into 3D volumes. These volumes can be visualized by the surgeon through conventional rendering techniques. In some embodiment of the invention, this visualization could be accomplished through stereographic video systems and displayed through a robotic surgeon's console. When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.).
However, Schneider fails to disclose localizing the laparoscopic camera in a world coordinate system based on the depth map and deform at least one of the 3D model or the ultrasound volume based on at least one of pressure applied on the tissue by the laparoscopic ultrasound probe or breathing motion.
Song, in the same field of endeavor, teaches localizing the laparoscopic camera in a world coordinate system based on the depth map (pages 1-2, introduction section, Inspired by the research done by Mahmoud et al. [4] [5] [13] which demonstrate the robustness of camera pose estimation from ORB-SLAM, we figure out ORB-SLAM is suitable to be improved and coupled with dense deformable SLAM. In this paper, we propose MIS-SLAM which builds on our preliminary work [12] with the following major improvements: (1) We proposed a heterogeneous framework to make full use of both GPU (dense deformable SLAM) and CPU (ORB-SLAM) to recover the dense deformed 3D structure of the soft-tissues in MIS scenario. Computational power of CPU is fully exploited to run an improved ORB-SLAM to provide complementary information to GPU modules. (2) Modules from GPU and CPU are deeply integrated to boost performance and enhance the efficiency. Sparse ORB features as well as global pose are uploaded to GPU).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider to include SIM-SLAM pipeline to determine the position and orientation of the laparoscopic camera in world coordinate system of Song because it increase the accuracy of pose estimation of fast moving camera and improves registration as disclosed within Song in introduction section.
However, Schneider in the view of Song fail to explicitly teach deform at least one of the 3D model or the ultrasound volume based on at least one of pressure applied on the tissue by the laparoscopic ultrasound probe or breathing motion.
Pheiffer, in the same field of endeavor, teaches deform at least one of the 3D model or the ultrasound volume based on at least one of pressure applied on the tissue by the laparoscopic ultrasound probe or breathing motion (paras. 0003 and 0040; computer-based registration of pre-operative volumes (such as CT or MR) to intra-operative ultrasound volumes is a challenging task due to organ deformation which occurs as a result of soft tissue motion, for example from the application of the ultrasound probe. Non-rigid registration is performed using a biomechanical tissue model with patient-specific boundary conditions and patient-specific material properties based on the intra-operative 3D B-mode US volume and the intra-operative 3D USE volume. The biomechanical tissue model simulates biomechanical deformation of the tissue of the target organ. According to an embodiment of the present invention, the biomechanical tissue model is used to drive the non-rigid registration by computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition. Patient-specific boundary conditions for the biomechanical tissue model are computed based on the intra-operative 3D USE volume and are used to constrain the simulated deformation of the pre-operative target organ such that positions of surface correspondences match on the deformed pre-operative target organ and the intra-operative target organ. Patient-specific material properties for the biomechanical tissue model, such as a distribution of spatially varying values of a tissue property, are determined for the pre-operative tissue domain based on the intra-operative 3D USE volume.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider in the view of Song to non-ridge registration for computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition of Pheiffer because it helps to accurately align organ structures in the pre-operative medical image data and the intra-operative US due within Pheiffer in para. 0031.
Regarding claim 13, Schneider teaches the surgical robotic system according to claim 12, wherein the image processing device is further configured to localize the laparoscopic ultrasound probe based on kinematic data of the first robotic arm (para. 0054; The image can be tracked through the use of the embedded EM sensor 470, optical sensing of the probe with the stereo laparoscope, or forward robot kinematics, given the known position of the grasper 500).
Regarding claim 17, Schneider teaches the surgical robotic system according to claim 12, wherein the laparoscopic camera is a stereoscopic camera and the video stream includes a pair of stereo image streams (para. 0049; The laparoscope 720 is the camera that is used in laparoscopic minimally invasive surgery to allow the surgeon to see inside the patient's abdominal cavity 751. In various embodiments of the invention, the laparoscope 720 can be monocular or stereoscopic.).
Regarding claim 18, Schneider teaches the surgical robotic system according to claim 17, wherein the image processing device is further configured to localize the laparoscopic ultrasound probe in the cameras coordinate system (claim 8).
However, fails to explicitly teach localize the laparoscopic camera in the world coordinate system using a Deformable Visual Simultaneous Localization and Mapping (DV-SLAM) pipeline based on the pair of stereo image streams.
Song, in the same field of endeavor, teaches the laparoscopic camera in the world coordinate system using a Deformable Visual Simultaneous Localization and Mapping (DV-SLAM) pipeline based on the pair of stereo image streams (pages 1-2, introduction section, To broaden surgeons’ field of view, 3D laparoscopy, or binoculars is applied to generate 2 images from different viewing point so that a 3D geometry based on parallax is created in surgeons’ mind for better understanding of the environment. Recently, similar stereo vision is adopted by some AR devices for enhancing MIS procedures. Therefore, it will be very helpful if stereo vision related approaches in computer vision community could be integrated, extended and improved to recover deformable shape in real-time while estimating the pose of the camera. In our previous work, we proposed a dynamic reconstruction of deformable soft-tissue with stereo scope [12]. A warping field based on the embedded deformation nodes approach is introduced with 3D dense shapes recovered from sequences pairs of stereo images. With the help of general-purpose computing on graphics processing units (GPGPU), all the processes are achieved in real-time. Mentioned in [12], the first and most important challenge to the pipeline is the fast movement of scope. Fast movement not only makes visual odometry unstable but also causes blurry images which make registrations even worse. This issue has also been reported in [2] and [3]. Inspired by the research done by Mahmoud et al. [4] [5] [13] which demonstrate the robustness of camera pose estimation from ORB-SLAM, we figure out ORB-SLAM is suitable to be improved and coupled with dense deformable SLAM. In this paper, we propose MIS-SLAM which builds on our preliminary work [12] with the following major improvements: (1) We proposed a heterogeneous framework to make full use of both GPU (dense deformable SLAM) and CPU (ORB-SLAM) to recover the dense deformed 3D structure of the soft-tissues in MIS scenario. Computational power of CPU is fully exploited to run an improved ORB-SLAM to provide complementary information to GPU modules. (2) Modules from GPU and CPU are deeply integrated to boost performance and enhance the efficiency. Sparse ORB features as well as global pose are uploaded to GPU).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider to include SIM-SLAM pipeline to determine the position and orientation of the laparoscopic camera in world coordinate system of Song because it recovers deformable shape in real-time while estimating the pose of fast moving camera without any blurring in the images which improves registration as disclosed within Song in introduction section.
Regarding claim 20, Schneider teaches the method according to claim 19, however, fails to explicitly teach further comprising: deforming at least one of the 3D model or the ultrasound volume based on at least one of pressure applied on the tissue by the laparoscopic ultrasound probe or breathing motion to generate a deformed model.
Pheiffer, in the same field of endeavor, teaches deforming at least one of the 3D model or the ultrasound volume based on at least one of pressure applied on the tissue by the laparoscopic ultrasound probe or breathing motion to generate a deformed model (paras. 0003 and 0040; computer-based registration of pre-operative volumes (such as CT or MR) to intra-operative ultrasound volumes is a challenging task due to organ deformation which occurs as a result of soft tissue motion, for example from the application of the ultrasound probe. Non-rigid registration is performed using a biomechanical tissue model with patient-specific boundary conditions and patient-specific material properties based on the intra-operative 3D B-mode US volume and the intra-operative 3D USE volume. The biomechanical tissue model simulates biomechanical deformation of the tissue of the target organ. According to an embodiment of the present invention, the biomechanical tissue model is used to drive the non-rigid registration by computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition. Patient-specific boundary conditions for the biomechanical tissue model are computed based on the intra-operative 3D USE volume and are used to constrain the simulated deformation of the pre-operative target organ such that positions of surface correspondences match on the deformed pre-operative target organ and the intra-operative target organ. Patient-specific material properties for the biomechanical tissue model, such as a distribution of spatially varying values of a tissue property, are determined for the pre-operative tissue domain based on the intra-operative 3D USE volume.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider in the view of Song to non-ridge registration for computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition of Pheiffer because it helps to accurately align organ structures in the pre-operative medical image data and the intra-operative US due within Pheiffer in para. 0031.
Regarding claim 21, modified Schneider teaches method according to claim 20, further comprising: displaying a suggested path of resection of tumor along with a selected negative resection margin from tumor based on the deformed model registered with ultrasound volume (para. 0005; Ultrasound is currently used during liver resection to localize and stage the lesions as well as determine the extent of the resection required. Similarly, ultrasound is used during partial nephrectomy to determine the resection margins and identify proper blood flow through the kidney after the tumour is removed.).
Regarding claim 22, Schneider teaches an imaging system comprising (fig. 7, para. 0021; robotic system, patient and ultrasound probe):
an intraoperative imaging device configured to be inserted through an access port and to obtain a plurality of signals from a tissue (fig. 7, para. 0049; The ultrasound probe 100 will be placed through an incision but outside the cannula lumen. This means that no extra incision is needed in the patient's abdominal wall 753);
a laparoscopic camera configured to capture a video stream of the tissue (fig. 7, para. 0049; The laparoscope 720 is the camera that is used in laparoscopic minimally invasive surgery to allow the surgeon to see inside the patient's abdominal cavity 751. In various embodiments of the invention, the laparoscope 720 can be monocular or stereoscopic.);
an image processing device configured to (para. 0050; The ultrasound machine 701 is capable of processing the electrical signals from the transducer array 105 and collecting the ultrasound data which can be processed in different ways to produce several image types, including B-mode images, Doppler images and continuous wave Doppler audio.):
generate a 3D reconstruction of a surgical site from the laparoscopic camera video stream to estimate 3D location of the intraoperative imaging device in a frame of reference of the laparoscopic camera (paras. 0004, 0051, and claims 5 and 8; a stereo camera system, meaning the surgeon can see his/her tools and the anatomy in three dimensions, thus regaining depth perception. Optical markers on the exterior surface of the probe housing that that can be localized by an endoscopic camera. identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system. The examiner notes that the stereo camera generate depth maps and the laparoscopic probe is tracked using the optical markers attached to the probe and shown in the video.);
localizing the laparoscopic ultrasound probe in the cameras coordinate system (claim 8; identifying the visible markers displayed on the ultrasound transducer in camera images and computing their coordinates with respect to a camera coordinate system.);
receive a volumetric image of tissue formed from a pre-operative imaging modality (para. 0052; Visualization of pre-operative images, such as CT or MRI volumes, can also be provided with this invention.);
generate a multi-frame representation from a plurality of signals from the intraoperative imaging device (para. 0051; It is also possible to combine a sequence of ultrasound data into 3D volumes.);
register the multi-frame representation with the volumetric image of the tissue (paras. 0052-0053; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
generate an overlay of the volumetric image of tissue and multi-frame representation (para. 0052; When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.); and
display the video stream showing data based on the plurality of signals from the intraoperative imaging device and the overlay extending from the intraoperative imaging device (paras. 0051-0053; it is possible to display the ultrasound data, such a B-mode ultrasound image 104 to the surgeon in its correct spatial location with respect to the ultrasound probe 100. It is also possible to combine a sequence of ultrasound data into 3D volumes. These volumes can be visualized by the surgeon through conventional rendering techniques. In some embodiment of the invention, this visualization could be accomplished through stereographic video systems and displayed through a robotic surgeon's console. When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI, the pre-operative images can be displayed in their correct spatial location with respect to the ultrasound probe 100 and surgeons view through the laparoscope 720.).
However, Schneider fails to disclose localizing the laparoscopic camera in a world coordinate system based on the depth map and deform the volumetric image of the tissue according to the multi-frame representation.
Song, in the same field of endeavor, teaches localizing the laparoscopic camera in a world coordinate system based on the depth map (pages 1-2, introduction section, Inspired by the research done by Mahmoud et al. [4] [5] [13] which demonstrate the robustness of camera pose estimation from ORB-SLAM, we figure out ORB-SLAM is suitable to be improved and coupled with dense deformable SLAM. In this paper, we propose MIS-SLAM which builds on our preliminary work [12] with the following major improvements: (1) We proposed a heterogeneous framework to make full use of both GPU (dense deformable SLAM) and CPU (ORB-SLAM) to recover the dense deformed 3D structure of the soft-tissues in MIS scenario. Computational power of CPU is fully exploited to run an improved ORB-SLAM to provide complementary information to GPU modules. (2) Modules from GPU and CPU are deeply integrated to boost performance and enhance the efficiency. Sparse ORB features as well as global pose are uploaded to GPU).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider to include SIM-SLAM pipeline to determine the position and orientation of the laparoscopic camera in world coordinate system of Song because it increase the accuracy of pose estimation of fast moving camera and improves registration as disclosed within Song in introduction section.
However, Schneider in the view of Song fail to explicitly teach deform the volumetric image of the tissue according to the multi-frame representation.
Pheiffer, in the same field of endeavor, teaches deform the volumetric image of the tissue according to the multi-frame representation (paras. 0003 and 0040; computer-based registration of pre-operative volumes (such as CT or MR) to intra-operative ultrasound volumes is a challenging task due to organ deformation which occurs as a result of soft tissue motion, for example from the application of the ultrasound probe. Non-rigid registration is performed using a biomechanical tissue model with patient-specific boundary conditions and patient-specific material properties based on the intra-operative 3D B-mode US volume and the intra-operative 3D USE volume. The biomechanical tissue model simulates biomechanical deformation of the tissue of the target organ. According to an embodiment of the present invention, the biomechanical tissue model is used to drive the non-rigid registration by computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition. Patient-specific boundary conditions for the biomechanical tissue model are computed based on the intra-operative 3D USE volume and are used to constrain the simulated deformation of the pre-operative target organ such that positions of surface correspondences match on the deformed pre-operative target organ and the intra-operative target organ. Patient-specific material properties for the biomechanical tissue model, such as a distribution of spatially varying values of a tissue property, are determined for the pre-operative tissue domain based on the intra-operative 3D USE volume.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider in the view of Song to non-ridge registration for computing deformation between the target organ in the pre-operative medical image volume and the intra-operative US acquisition of Pheiffer because it helps to accurately align organ structures in the pre-operative medical image data and the intra-operative US due within Pheiffer in para. 0031.
Regarding claim 23, Schneider teaches the imaging system according to claim 22, wherein the intraoperative imaging device is a laparoscopic ultrasound probe and the pre-operative imaging modality is at least one MRI or CT (para. 0052; Visualization of pre-operative images, such as CT or MRI volumes, can also be provided with this invention. When the intra-operative ultrasound images 104 provided from the ultrasound probe 100 are registered to pre-operative images, such as CT or MRI).
Claim(s) 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Schneider et al. (US 2013/0338505) in the view Song et al. (NPL: “MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing”) and Pheiffer et al. (US 2019/0336109) and in further view of Shekhar et al. (US 2020/0197102).
Regarding claim 14, modified Schneider teaches the surgical robotic system according to claim 12, however, fails to explicitly teach wherein the image processing device is further configured to generate a plurality of virtual fiducial markers on the laparoscopic ultrasound probe using a deep learning image processing algorithm.
Shekhar, in the same field of endeavor, teaches generate a plurality of virtual fiducial markers on the laparoscopic ultrasound probe using a deep learning image processing algorithm (paras. 0055-0056; the computer vision-based object tracking workflow of the present disclosure describes a workflow for image line detection 526 and image point detection 527 following image pre-processing of a video image. First, a region of interest (ROI) can be generated for each frame of a first imaging modality using fast visual tracking based on robust discriminative correlation filters such that subsequent processing focuses on the imaging tip, as shown at No. 618 in FIG. 6A. Based on this coarse estimate of the probe's location, the bounding box surrounding the imaging tip can be intended to include at least some portion of the top, middle, and tip of the transducer as seen by the camera. In order to find the straight edges of these features of the transducer, the image can first be converted to a gray scale image based on brightness, followed by Canny edge detection, for instance, as shown in FIG. 6B. A Probabilistic Hough Transform (PHT) can be used to extract a first set of lines from the edge detection result within the ROI, an example of which is shown in FIG. 6C. As shown in FIG. 6D, the first line set can be filtered, for instance, by creating a coarse grain 2D histogram with the axes defined by PHT parameters (r, θ; magnitude and direction) and values of the histogram defined by the sum of the lengths of lines in the bin. All lines not contained within the highest peak present in the 2D histogram, therefore, can be removed to generate a second set of lines that corresponds with the long edges and is substantially parallel to an object of interest, which may be a projection of a transducer, or virtual object, or an actual transducer, or real object, according to the imaging modality. From this smaller, second set of lines, a fine grain 2D histogram, based on the PHT parameters (r, θ; magnitude and direction), can be created. The two highest peaks in this histogram, determined to be the longest lines, may represent the top and middle of the transducer of either the projection of the transducer or the actual transducer, and be selected. The indices of the peaks, defining the ROI of the first imaging modality, can then be used in one of at least one cost function for the optimization of the virtual object location. According to an embodiment, the above-described line feature extraction can be performed via PHT, as described above, or via custom designed markers including AR marker, checkerboard marker, dot marker, and the like, and machine learning-based binary segmentation of the tool including, among others, convolutional neural networks.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the optical markers of Schneider in the view of Song and Pheiffer to include virtual AR markers to determine the position and orientation of the laparoscopic ultrasound probe in the video stream of Shekhar because it reduces a line of sight problem caused by optical markers tracking method and reduce registration errors within Shekhar in paras. 0004 and 0040.
Regarding claim 15, modified Schneider teaches the surgical robotic system according to claim 14, however, fails to explicitly teach localize the laparoscopic ultrasound probe in the video stream based on the virtual fiducial markers.
Shekhar, in the same field of endeavor, teaches localize the laparoscopic ultrasound probe in the video stream based on the virtual fiducial markers (paras. 0055-0056; the computer vision-based object tracking workflow of the present disclosure describes a workflow for image line detection 526 and image point detection 527 following image pre-processing of a video image. First, a region of interest (ROI) can be generated for each frame of a first imaging modality using fast visual tracking based on robust discriminative correlation filters such that subsequent processing focuses on the imaging tip, as shown at No. 618 in FIG. 6A. Based on this coarse estimate of the probe's location, the bounding box surrounding the imaging tip can be intended to include at least some portion of the top, middle, and tip of the transducer as seen by the camera. In order to find the straight edges of these features of the transducer, the image can first be converted to a gray scale image based on brightness, followed by Canny edge detection, for instance, as shown in FIG. 6B. A Probabilistic Hough Transform (PHT) can be used to extract a first set of lines from the edge detection result within the ROI, an example of which is shown in FIG. 6C. As shown in FIG. 6D, the first line set can be filtered, for instance, by creating a coarse grain 2D histogram with the axes defined by PHT parameters (r, θ; magnitude and direction) and values of the histogram defined by the sum of the lengths of lines in the bin. All lines not contained within the highest peak present in the 2D histogram, therefore, can be removed to generate a second set of lines that corresponds with the long edges and is substantially parallel to an object of interest, which may be a projection of a transducer, or virtual object, or an actual transducer, or real object, according to the imaging modality. From this smaller, second set of lines, a fine grain 2D histogram, based on the PHT parameters (r, θ; magnitude and direction), can be created. The two highest peaks in this histogram, determined to be the longest lines, may represent the top and middle of the transducer of either the projection of the transducer or the actual transducer, and be selected. The indices of the peaks, defining the ROI of the first imaging modality, can then be used in one of at least one cost function for the optimization of the virtual object location. According to an embodiment, the above-described line feature extraction can be performed via PHT, as described above, or via custom designed markers including AR marker, checkerboard marker, dot marker, and the like, and machine learning-based binary segmentation of the tool including, among others, convolutional neural networks.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the optical markers of Schneider in the view of Song and Pheiffer to include virtual AR markers to determine the position and orientation of the laparoscopic ultrasound probe in the video stream of Shekhar because it reduces a line of sight problem caused by optical markers tracking method and reduce registration errors within Shekhar in paras. 0004 and 0040.
Regarding claim 16, modified Schneider teaches the surgical robotic system according to claim 15, however, fails to explicitly teach estimate a pose and orientation of the laparoscopic ultrasound probe from the video stream based on the virtual fiducial markers.
Shekhar, in the same field of endeavor, teaches estimate a pose and orientation of the laparoscopic ultrasound probe from the video stream based on the virtual fiducial markers (paras. 0055-0056; the computer vision-based object tracking workflow of the present disclosure describes a workflow for image line detection 526 and image point detection 527 following image pre-processing of a video image. First, a region of interest (ROI) can be generated for each frame of a first imaging modality using fast visual tracking based on robust discriminative correlation filters such that subsequent processing focuses on the imaging tip, as shown at No. 618 in FIG. 6A. Based on this coarse estimate of the probe's location, the bounding box surrounding the imaging tip can be intended to include at least some portion of the top, middle, and tip of the transducer as seen by the camera. In order to find the straight edges of these features of the transducer, the image can first be converted to a gray scale image based on brightness, followed by Canny edge detection, for instance, as shown in FIG. 6B. A Probabilistic Hough Transform (PHT) can be used to extract a first set of lines from the edge detection result within the ROI, an example of which is shown in FIG. 6C. As shown in FIG. 6D, the first line set can be filtered, for instance, by creating a coarse grain 2D histogram with the axes defined by PHT parameters (r, θ; magnitude and direction) and values of the histogram defined by the sum of the lengths of lines in the bin. All lines not contained within the highest peak present in the 2D histogram, therefore, can be removed to generate a second set of lines that corresponds with the long edges and is substantially parallel to an object of interest, which may be a projection of a transducer, or virtual object, or an actual transducer, or real object, according to the imaging modality. From this smaller, second set of lines, a fine grain 2D histogram, based on the PHT parameters (r, θ; magnitude and direction), can be created. The two highest peaks in this histogram, determined to be the longest lines, may represent the top and middle of the transducer of either the projection of the transducer or the actual transducer, and be selected. The indices of the peaks, defining the ROI of the first imaging modality, can then be used in one of at least one cost function for the optimization of the virtual object location. According to an embodiment, the above-described line feature extraction can be performed via PHT, as described above, or via custom designed markers including AR marker, checkerboard marker, dot marker, and the like, and machine learning-based binary segmentation of the tool including, among others, convolutional neural networks.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the optical markers of Schneider in the view of Song and Pheiffer to include virtual AR markers to determine the position and orientation of the laparoscopic ultrasound probe in the video stream of Shekhar because it reduces a line of sight problem caused by optical markers tracking method and reduce registration errors within Shekhar in paras. 0004 and 0040.
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Schneider et al. (US 2013/0338505) in the view Song et al. (NPL: “MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing”) in further view of Wittmeier et al. (EP 2886059).
Regarding claim 7, modified Schneider teaches imaging system according to claim 1, however, fails to explicitly teach wherein the image processing device is further configured to compensate for deformation of the ultrasound volume in response to breathing motion based on respiratory data from a pulse oximeter apparatus.
Wittmeier, in the same field of endeavor, teaches compensate for deformation of the ultrasound volume in response to breathing motion based on respiratory data from a pulse oximeter apparatus (para. 0024; Preferably, the phase of the cyclical movement and/or deformation is also determined, wherein the phase is determined on the basis of the sonography data and/or with the aid of an additional measuring unit. If, for example, the cyclical movement and/or deformation is defined by the pulse stroke of a patient, the determination of the phase can be carried out with the aid of an ECG measuring device and/or a pulse oximeter. If the cyclical movement and/or deformation is defined, for example, by respiration, the phase can preferably be determined with the aid of a spirometer. In these cases, the step of providing sonographic data comprises the additional step of providing data of one or more measurement units containing information regarding the phase of the cyclical movement and/or deformation. An image-based determination of the phase in the case of respiration is described, for example, in "Manifold Learning for Image-Based Braking Gating with Application to 4D Ultrasound" (Wahinger, C.; Yigitsoy, M.; Navab, N., Medical Image Computing and Computer-Assisted Intervention" MICCAI 2010, pp. 26-3 3, 2010.).).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider in the view of Song to include the deformation of ultrasound image based on respiration data from a pulse oximeter of Wittmeier because it improves image guidance by compensating for motion due to breathing of the subject as disclosed within Wittmeier in paras. 0006-0007.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over Schneider et al. (US 2013/0338505) in the view Song et al. (NPL: “MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing”) and Pheiffer et al. (US 2019/0336109) in further view of Song et al. (NPL: “CT2US: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data”, hereinafter Song II).
Regarding claim 10, modified Schneider teaches imaging system according to claim 9, however, fails to explicitly teach wherein the image processing device is further configured to transfer a slice of the 3D model to a corresponding 2D ultrasound image of the ultrasound volume using a neural network.
Song II, in the same field of endeavor, teaches the image processing device is further configured to transfer a slice of the 3D model to a corresponding 2D ultrasound image of the ultrasound volume using a neural network (pages 3-4, synthesizing US from CT section; The work to synthesize US from CT is first tried in [26], where the author transformed the CT image into the US image domain for cross- modal registration. In addition, Field II [27,28] is also a classic algo rithm for synthesizing US from CT. Even ignoring the quality of syn thesized images, the main problem with these traditional methods is that the running time is too long to be used in the application of deep learning. For example, it takes more than 10 h for the Field II algorithm to synthesize a US image from a CT image. Therefore, we turned to finding ways based on learning to realize the synthesizing from CT to US. Generative adversarial network (GAN) is currently the mainstream image synthesis method, and has been widely used in the field of generating natural images and industrial ultrasound images [29,30]. CycleGAN, which is used for unpaired image-to-image translation using cycle-consistent adversarial networks, has been applied for cross-modal medical image synthesis [31,32]. In this paper, we adopted CycleGAN to synthesize kidney ultrasound images from CT slices. CycleGAN performs as the translator between CT and US modalities in our method. By learning the mapping relationship between the two modalities, it con ducts nonlinear transformations and builds the transition dataset. The workflow is shown in Fig. 2. The central idea behind CycleGAN is that an excellent translator should be two-way reversible. In our task, the synthesized ultrasound image by the GAN generator with the input CT should be the same as the original CT after reconstruction by the GAN reverse generator.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider in the view of Song and Pheiffer to include neural network model to transfer CT image slice to ultrasound image using cross modality transfer learning of Song II because it improves registration and segmentation as disclosed within Song II in page 8.
Claim(s) 11 is rejected under 35 U.S.C. 103 as being unpatentable over Schneider et al. (US 2013/0338505) in the view Song et al. (NPL: “MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing”) in further view of Miller et al. (US 2020/0261169).
Regarding claim 11, modified Schneider teaches the imaging system according to claim 1, however, fails to explicitly teach further comprising a laparoscopic imaging probe selected from the group consisting of a gamma radiation probe, Raman spectroscopy probes, multispectral probe, a time-resolved fluorescence spectroscopy (ms-TRFS) probe, and an auto fluorescence probe.
Miller, in the same field of endeavor, teaches a laparoscopic imaging probe selected from the group consisting of a gamma radiation probe, Raman spectroscopy probes, multispectral probe, a time-resolved fluorescence spectroscopy (ms-TRFS) probe, and an auto fluorescence probe (para. 0027; The imaging system 15 may comprise an endoscopic imaging system using optical imaging technology, or comprise another type of imaging system using other technology (e.g. ultrasonic, fluoroscopic, etc.). An operator input system 16 allows an operator such as a surgeon or other type of clinician S to view images of or representing the procedure site and to control the operation of the medical tool 14 and/or the medical tool 15.).
It would have been obvious to an ordinary skilled in the art before the invention was made to modify the system of Schneider in the view of Song to include fluoroscopic imaging probe of Miller because it would provide complementary real time visualization of the surgical tools and the anatomy that are not visible in the ultrasonic or stereoscopic images.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZAINAB M ALDARRAJI whose telephone number is (571)272-8726. The examiner can normally be reached Monday-Thursday7AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carey Michael can be reached at (571) 270-7235. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZAINAB MOHAMMED ALDARRAJI/ Patent Examiner, Art Unit 3797
Prosecution Insights