DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to towards a "computer program product" that includes "code instructions", which broadly encompasses a computer program per se. Such computer programs, per se, are not, in and of themselves, methods or machines, nor are they physical products of manufacture or compositions of matter. Therefore, such programs do not fall into any of the categories of eligible subject matter defined in 35 U.S.C. § 101 and are not, by themselves, eligible for patent protection. Such programs can be eligible for patent protection if claimed as embodied on or in a computer readable storage device or medium, but only if the claim clearly and unambiguously excludes transitory, propagating signals from the full scope of the claimed subject matter, as such signals are also not eligible under 35 U.S.C. § 101. It is suggested that amending the claim language to define the computer program product as having the code instructions embodied on a "non-transitory computer-readable medium storing a computer program" would satisfy these requirements and would limit the claimed invention to eligible subject matter.
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to a computer readable medium or a memory that does not exclude transitory forms of signal transmission (often referred to as "signals per se"), such as a propagating electrical or electromagnetic signal or carrier wave, and therefore does not fall within at least one of the four categories (a process, machine, manufacture, or composition of matter). It is suggested that amending the claim language to define the computer readable medium as "a non-transitory computer-readable medium" to satisfy the requirements and limit the claimed invention to eligible subject matter. The Examiner notes that once claim 11 is amended, claim 12 becomes redundant and suggests canceling claim 12.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 5-9, and 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Shen et al. (US 2022/0319031 A1; hereinafter “Shen”) in view of Jayarathne et al., “Robust, Intrinsic Tracking of a Laparoscopic Ultrasound Probe for Ultrasound-Augmented Laparoscopy 2018 (hereinafter “Jayarathne”), and further in view of Montaña-Brown et al., “Vessel segmentation for automatic registration of untracked laparoscopic ultrasound to CT of the liver,” (2021), (hereinafter “Montana”).
Regarding claim 1:
Shen discloses: a computer-implemented method for determining a pose of a probe with respect to volumetric scan data (Shen discloses image based localization of a medical instrument having an imaging device, wherein video image data acquired by the imaging device are related to a virtual three dimensional anatomical model generated from preoperative CT image data. See Shen ¶¶0126–0131; see also ¶0140 and FIG. 11. Shen's collection of CT slices and corresponding three dimensional anatomical model reads on the claimed volumetric scan data), comprising:
receiving image data obtained from a first probe (Shen teaches an imaging device such as a video camera that captures video image data of the internal anatomical structure and generates a first depth map. See Shen ¶¶0127, 0137 and FIG. 10. Shen teaches using machine learning, including a convolutional neural network (“CNN”), to derive the first depth map from the video image data. See Shen ¶¶0127, 0137);
determining, using a machine learning algorithm, a pose of at least one of the first probe and the second probe relative to the volumetric scan data, from the image data obtained from a first probe; wherein the first probe is or comprises a video camera (Shen teaches generating CT derived second depth maps representing the virtual anatomical model, comparing the video derived first depth map with the CT derived depth maps, identifying a corresponding candidate depth map, and generating the camera pose based thereon. See Shen ¶¶0128–0130 and FIG. 11. Shen states that the resulting camera pose may be defined relative to the preoperative CT image data. See Shen ¶0130. Shen further teaches employing a CNN to solve a transformation matrix and estimate the camera pose parameters. See Shen ¶0131. Shen additionally teaches neural network processing wherein deep features are extracted from video derived and CT derived point clouds, correspondence is established between the point clouds, and an optimal transformation and corresponding camera pose are determined. See Shen ¶0147);
Shen does not specifically teach: receiving image data obtained from a second probe, the second probe is located at least partially within the field of view of the video camera.
However, in the same field of endeavor, Jayarathne teaches this: (Jayarathne teaches that, during ultrasound guided laparoscopic tasks, the imaging tip of the LUS probe is visible in the camera image, thereby permitting image based estimation of the 6DoF pose of the LUS probe in the camera coordinate system. See Jayarathne, p. 461, §I.”A Related Work”. Jayarathne further teaches that image data from both probes (a laparoscopic camera and a laparoscopic ultrasound (“LUS”) probe) are acquired in the same system. In §III.C, “US-Video Overlay,” Jayarathne states acquiring 17 US/video image pairs, determining the LUS probe pose relative to the camera for each corresponding video frame, and using the determined pose to overlay the ultrasound image into the laparoscopic video. See Jayarathne, p. 466, §III.C).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the image based CT referenced localization system of Shen to include Jayarathne's LUS probe and associated ultrasound image data because Jayarathne teaches that laparoscopic video provides a surface view while ultrasound provides complementary tomographic information concerning hidden structures, and that spatially relating the ultrasound information to the laparoscopic-video frame facilitates intraoperative visualization. See Jayarathne, pp. 460–461.
Shen in view of Jayarathne does not expressly teach applying machine learning to the LUS image data (image data obtained from second probe) as part of determining a probe pose relative to the volumetric CT data.
However, in the same field, of endeavor, Montana teaches this (Montana teaches an automatic LUS to CT registration framework in which 2 D image data obtained from a laparoscopic ultrasound probe are processed using deep learning. Specifically, Montana teaches training a modified 2 D U Net to segment vessels directly from the LUS images. See Montana, p. 1152, “LUS vessel segmentation model.” Montana then integrates the machine learning generated LUS segmentation into an LUS to CT registration framework. Montana generates possible registration solutions by intersecting segmented CT vessel models with LUS planes parameterized by virtual probe poses, stores the resulting feature vector pose pairs, and, given a segmented LUS image, determines the pose that provides the best content match. See Montana, p. 1153, “Integration of segmentation in a registration framework.”).
Therefore, It would have been obvious to one of ordinary skill in the art to further modify the combined system of Shen and Jayarathne further apply Montana's known ML-assisted LUS to CT registration technique to the LUS data of the modified Shen/Jayarathne system to provide automatic CT space localization of the ultrasound acquisition while reducing reliance on manual initialization or external tracking. Montana expressly identifies those objectives. See Montana, pp. 1152–1153.
Regarding claim 3:
wherein determining the pose comprises determining at least one of a position and an orientation of the respective probe (Jayarathne teaches estimating the 6DoF pose of the LUS probe in the camera coordinate system. See Jayarathne, p. 461, §I.A. A 6-DoF pose includes translational position and rotational orientation, and Jayarathne separately evaluates translational and rotational pose errors. See Fig. 5 and accompanying discussion).
Regarding claim 5:
wherein the machine learning algorithm comprises a neural network, and optionally comprises a convolutional neural network (Shen teaches generating the depth map using a CNN (¶0127), employing a CNN to solve the transformation matrix and estimate camera pose parameters (¶0131), and using neural network encoders to extract deep features and determine the camera pose (¶0147)).
Regarding claim 6:
wherein the image data from at least one of the first probe and the second probe is segmented (Montana acquires 2D LUS images and trains a modified 2D UNet to segment vessels from the LUS images. See Montana, p. 1152, “LUS vessel segmentation model.”).
Regarding claim 7:
wherein the volumetric scan data and the image data from the first probe and the second probe is of an organ, and optionally wherein the organ is one of a liver, a kidney and a pancreas (Montana is directed to registration of laparoscopic ultrasound to preoperative CT for laparoscopic liver resection, using liver vessel information. See Montana, Abstract, p. 1151. Montana further teaches acquiring LUS images by sweeping the LUS probe over the surface of the liver. See p. 1152, “Data description.”. Shen also teaches that while the illustrated luminal network 130 is a bronchial network of airways 150 within the patient's lung, this disclosure is not limited to only the illustrated example, and may be used to navigate any type of luminal network, such as bronchial networks, renal networks, cardiovascular networks (e.g., arteries and veins), gastrointestinal tracts, urinary tracts, etc. See Shen ¶0113).
Regarding claim 8:
image data from the second probe is segmented to identify one or more internal structures of the organ, optionally one or more blood vessels of the organ (Montana teaches using a modified 2D UNet to segment liver vessels from LUS images and using the segmented vessel information for LUS to CT registration. See Montana, p. 1152, “LUS vessel segmentation model”; p. 1153, “Integration of segmentation in a registration framework.”).
Regarding claim 9:
wherein the second probe is or comprises an ultrasound probe, and optionally is or comprises a laparoscopic ultrasound probe or an endoscopic ultrasound probe (Jayarathne is directed to tracking a laparoscopic ultrasound probe relative to a laparoscopic camera and, in the experimental implementation, employs a clinical LUS probe having a fiducial pattern attached to its imaging tip. See Jayarathne, Abstract; p. 466, §III.A (“Experimental Setup”)).
Regarding claims 11-13: Shen ¶0052 teaches a system and a non-transitory computer-readable storage medium.
Regarding claims 14: see ¶8 above in claim 1 rejection.
Regarding claims 15: see ¶26 above in claim 9 rejection.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Shen in view of Jayarathne and Montana as applied to claim 1 above, and further in view of further in view of Guo et al., “Deep Adaptive Registration of Multi-modal Prostate Images” (2020) “Guo”).
Regarding claim 2:
Shen in view of Jayarathne and Montana does not expressly teach concatenating the image data from the first probe and the second probe and determining a pose from the concatenated image data.
However, in a related field, Guo teaches: concatenating the image data from the first probe and the second probe and determining a pose from the concatenated image data (Guo teaches multimodal registration using MR image data and TRUS image data. Guo states that the MR and TRUS ROI volumes are normalized and that their concatenation serves as the input to the multistage registration network. See Guo, §2.1, “Registration Workflow,” Fig. 2. Guo further teaches that the concatenated MR and TRUS input is processed by 3D convolutional layers to extract hybrid structural features, which are supplied to dense layers that regress six rigid-transformation parameters: Δtx, Δty, Δtz, Δαx, Δαy, Δαz. Guo states that the trained network estimates these 6DoF to adjust the position of the moving TRUS image. See Guo, §2.1, “Registration Workflow,” Fig. 2).
Therefore, It would have been obvious to one of ordinary skill in the art to configure the multimodal probe pose system of Shen in view of Jayarathne and Montana according to Guo by concatenating the received image data from the two probes before pose determination, because Guo teaches concatenating heterogeneous medical image inputs to jointly extract multimodal features and determine a rigid 6DoF spatial transformation. Applying the same known input fusion technique of Gue to the video and ultrasound image data would have been a predictable use of the Guo’s technique for its established purpose of jointly processing different modalities for spatial registration.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Shen in view of Jayarathne and Montana as applied to claim 1 above, and further in view of further in view of En et al. "Rpnet: An end-to-end network for relative camera pose estimation 2018. “En”).
Regarding claim 4:
Shen in view of Jayarathne and Montana does not expressly teach wherein the machine learning algorithm comprises a first path configured to determine a pose of the first probe and a second path configured to determine a pose of the second probe.
However, in a related field, En teaches: wherein the machine learning algorithm comprises a first path configured to determine a pose of the first probe and a second path configured to determine a pose of the second probe (EN teaches a Siamese Network with two branches regressing one pose per image, thereby outputting one pose per image. See En, §3, “Relative Pose Inference with RPNet”, Fig. 1).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the multimodal probe pose system of Shen in view of Jayarathne and Montana to implement En’s two branch architecture in order to estimate the probes poses separately using a known parallel pose regression architecture.
Claims 10 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Shen in view of Jayarathne and Montana as applied to claim 1 above, and further in view of further in view of Azizian et al (US 2023/0218356 A1, “Azizian”)
Regarding claim 10:
Shen in view of Jayarathne and Montana does not expressly teach comprising displaying image data from at least one of the first probe and the second probe overlaid on the volumetric scan data.
However, in a related field, Azizian teaches: comprising displaying image data from at least one of the first probe and the second probe overlaid on the volumetric scan data (EN teaches that he display 150 includes an endoscopic image 152 and an anatomic model image 154. The endoscopic image 152 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 154. See ¶0038 and FIGS. 3-6. Azizian further teaches that registering the endoscopic image dataset with the anatomic model dataset allows the two dimensional stereoscopic endoscope image data to be projected or mapped onto a three-dimensional volume in space defined by the anatomic model. See ¶0037. Azizian further states that anatomic model datasets include computed tomography (CT) or magnetic resonance imaging (MRI). See ¶0036).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the multimodal probe pose system of Shen in view of Jayarathne and Montana to display image data from at least one of the first probe and the second probe overlaid on the volumetric scan data in order to provide the surgeon with enhanced perspective of the surgical environment beyond the endoscopic view. See Azizian ¶0003.
Regarding claims 16: see ¶43 above in claim 10 rejection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665
Prosecution Insights