DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 7 is objected to because of the following informalities:
Lines 4, 5, and 6/7 recite “the plurality of 3D probe poses” and needs to be changed to “the one or more 3D probe poses” to match the earlier recitation of 3D probe poses.
Claim 8 is objected to because of the following informalities:
Line 2 recites “the one or more of the plurality of 3D probe poses” and needs to be changed to “the one or more 3D probe poses” to match the earlier recitation of 3D probe poses.
Claims 17, 18, 27, and 28 recited similar objections.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-10, 16-20, and 26-30 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 6, the limitation “the metrics of the selected predicted 3D probe poses” lacks antecedent basis. For examination purposes, it shall be considered as “metrics of the selected predicted 3D probe poses”.
Claims 16 and 26 suffer from the same issues.
Regarding claim 7, the limitation “the additional 3D probe poses” lacks antecedent basis. For examination purposes, it shall be considered as “additional 3D probe poses”.
Claims 17 and 27 suffer from the same issues.
Claims that are not discussed above but are cited to be rejected under 35 U.S.C. 112(b) are also rejected because they inherit the indefiniteness of the claims they respectively depend upon.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-6, 11, 13-16, 21 and 23-26 are rejected under 35 U.S.C. 103 as being unpatentable over Matsumoto et al., (US20240407755A1), and in further view of Ramalhinho et al., (US20220249056A1) and Djajadiningrat et al., (US20190117190A1).
Regarding claim 1, Matsumoto teaches a method implemented on at least one processor, a memory, and a communication platform, comprising ([0060]):
projecting, during a surgery, one or more three-dimensional (3D) probe information onto a two-dimensional (2D) image at corresponding 2D probe locations (fig. 18 Rendering is processing to project the preoperative preparation information 23 based on the preoperative 3D image 34 onto a two-dimensional projection plane 63 [0097]-[0098]), wherein the 2D image is acquired by a laparoscopic camera inserted into a patient’s body in a surgery directed to a target organ ([0047]-[0048] endoscope, which can be laparoscope, has an camera build in and can be a charge-coupled device image sensor, which takes 2D images);
receiving 2D ultrasound images acquired by an ultrasound probe at or near one of the 2D probe locations ([0049] the probe acquires B mode images; fig. 17 is an example of images taken [0095]).
However, the combination of references still fails to explicitly disclose with respect to each of at least some of the 2D ultrasound image, detecting an anatomical structure from the 2D image, generating an anatomical structure mask (ASM) for the 2D ultrasound image based on the detected anatomical structure, and predicting, via an ASM-pose mapping model, one or more 3D probe pose estimates of the ultrasound probe based on the ASM.
In the same laparoscope field of endeavor, Ramalhinho teaches with respect to each of at least some of the 2D ultrasound image ([0062] extracting a feature vector from each ultrasound image in an acquired sequence of vessel/non-vessel segmentation; [0068] the candidate simulated images are obtained for each acquired 2D ultrasound image in the sequence), detecting an anatomical structure from the 2D image ([0073] acquired ultrasound is segmented to identify vessels and then extract vessel features for comparison with the simulated images) generating an anatomical structure mask (ASM) for the 2D ultrasound image based on the detected anatomical structure ([0062] the acquired ultrasound images are segmented into vessel/non-vessel regions and extract features from that segmentation for each image; the segmentation constitutes a mask identifying the anatomical vessel regions), and predicting, via an ASM-pose mapping model, one or more 3D probe pose estimates of the ultrasound probe based on the ASM ([0074]-[0076] the simulated vessel images are generated for known 3D probe positions, rotations, and translations, and each vessel feature vector is associated with its corresponding probe pose from the lookup tables).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
However, the combination of references is still silent regarding generating an optimal 3D probe pose estimate based on the 3D probe pose estimates.
In the same ultrasound probe localization field of endeavor, Djajadiningrat teaches generating an optimal 3D probe pose estimate based on the 3D probe pose estimates ([0025] pose information is obtained from the images generated from the data stream; [0026] the best pose is chosen, meaning the optimal pose is also generated).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method modified Matsumoto with the pose processing of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill would understand that combining the generation of the optimal 3D probe pose of Djajadiningrat with the predicting ASM pose probe mapping model of modified Matsumoto would read upon the limitation “generating an optimal 3D probe pose estimate based on the 3D probe pose estimates predicted based on ASMs obtained from the 2D ultrasound images acquired by the ultrasound probe at varying 3D probe poses.”
Regarding claim 3, modified Matsumoto teaches the method of claim 1, but fails to explicitly disclose wherein each of the one or more 3D probe estimates is predicted with a metric indicative of a confidence level of the 3D probe pose estimate.
In the same laparoscope field of endeavor, Ramalhinho teaches wherein each of the one or more 3D probe estimates is predicted with a metric indicative of a confidence level of the 3D probe pose estimate ([0057]-[0059] a similarity value is determined for each simulated image, and the best matching candidate is selected based on that value, and identifying the probe pose from the selected candidate; [0076] selecting the vectors with the lowest distance values, which corresponds to the respective probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 5, modified Matsumoto teaches the method of claim 4, but fails to explicitly disclose determining a trajectory of the ultrasound probe according to the varying 3D probe poses; identifying 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory.
In the same ultrasound field of endeavor, Ramalhinho teaches determining a trajectory of the ultrasound probe according to the varying 3D probe poses ([0068] using a Viterbi algorithm to determine the most likely sequence of simulated images and the corresponding probe pose for each ultrasound image; [0077] each image corresponds to a successive pose along the path swept by the moving probe; [0081] the lower cost path is used found to find the optimal sequence of candidate probe poses); identifying 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory ([0068]obtaining possible candidate images and corresponding probe poses for each acquire ultrasound image in the sequence; [0074]-[0076] feasible candidate probe poses are identified for each input ultrasound image by comparing feature vectors and selecting candidate vectors having corresponding probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the probe processing of Ramalhinho, as this would help determine an optimal lower cost path through the probe poses (see Ramalhinho [0081]).
However the combination of references still fails to explicitly disclose determining the optimal 3D probe pose estimate based on the 3D probe pose estimates
In the same ultrasound field of endeavor, Djajadiningrat teaches determining the optimal 3D probe pose estimate based on the 3D probe pose estimates ([0025] system receives a sequence of ultrasound images having respective probe pose information and generates a target probe pose based on the image selection and respective pose information; [0026] user selects a preferred image providing the probe pose, and determines the require pose from the probe pose information associated with the information in the sequence; [0048]-[0049] selecting the image providing the best anatomical view and retrieving the probe pose belonging to the selected image.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the identification process of the 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory as taught by modified Matsumoto with the optimal 3D probe pose estimation process of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill in the art would understand this combination would result in the reading of the limitation “determining the optimal 3D probe pose estimate based on the 3D probe pose estimates predicted with respect to the varying 3D probe poses along the trajectory.”
Regarding claim 6, modified Matsumoto teaches the method of claim 6, but fails to explicitly disclose selecting one of the 3D probe pose estimates with respect to each of the varying probe poses along the trajectory so that the metrics of the selected predicted 3D probe poses are maximized; and determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses along the trajectory.
In the same ultrasound field of endeavor, Ramalhinho teaches selecting one of the 3D probe pose estimates with respect to each of the varying probe poses along the trajectory so that the metrics of the selected predicted 3D probe poses are maximized ([0026] selecting, for each ultrasound image, candidate images having the lowest distance or highest similarity; [0068] the system obtains candidate images for each acquired ultrasound image and applying a Viterbi algorithm to determine the most likely sequence of simulated images and the corresponding probe pose for each acquire image; [0077]-[0078] successive acquired images are associated with successive probe poses along the path swept by the probe and representing the probability that each acquired image matches each candidate pose; [0081] the optimal sequence of candidate poses is selected as the lowest cost path, with the candidate probabilities optionally weighted by the image similarity distance metric).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the probe processing of Ramalhinho, as this would help determine an optimal lower cost path through the probe poses (see Ramalhinho [0081]).
However, the combination to references still fail to explicitly disclose determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses.
In the same ultrasound field of endeavor, Djajadiningrat teaches determining the optimal 3D probe poses based on the 3D probe pose ([[0025] system receives a sequence of ultrasound images having respective probe pose information and generates a target probe pose based on the image selection and respective pose information; [0026] user selects a preferred image providing the probe pose, and determines the require pose from the probe pose information associated with the information in the sequence; [0048]-[0049] selecting the ultrasound image providing the best anatomical view and retrieving the probe pose belonging to that selected image as the target probe pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the identification process of the 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory as taught by modified Matsumoto with the optimal 3D probe pose estimation process of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill in the art would understand this combination would result in the reading of the limitation “determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses along the trajectory.”
Regarding claim 11, Matsumoto teaches a machine readable and non-transitory medium having information recorded thereon, wherein the information, when read by the machine, causes the machine to perform the following steps ([0060];[0200]):
projecting, during a surgery, one or more three-dimensional (3D) probe information onto a two-dimensional (2D) image at corresponding 2D probe locations (fig. 18 Rendering is processing to project the preoperative preparation information 23 based on the preoperative 3D image 34 onto a two-dimensional projection plane 63 [0097]-[0098]), wherein the 2D image is acquired by a laparoscopic camera inserted into a patient’s body in a surgery directed to a target organ ([0047]-[0048] endoscope, which can be laparoscope, has an camera build in and can be a charge-coupled device image sensor, which takes 2D images);
receiving 2D ultrasound images acquired by an ultrasound probe at or near one of the 2D probe locations ([0049] the probe acquires B mode images; fig. 17 is an example of images taken [0095]).
However, the combination of references still fails to explicitly disclose with respect to each of at least some of the 2D ultrasound image, detecting an anatomical structure from the 2D image, generating an anatomical structure mask (ASM) for the 2D ultrasound image based on the detected anatomical structure, and predicting, via an ASM-pose mapping model, one or more 3D probe pose estimates of the ultrasound probe based on the ASM.
In the same laparoscope field of endeavor, Ramalhinho teaches with respect to each of at least some of the 2D ultrasound image ([0062] extracting a feature vector from each ultrasound image in an acquired sequence of vessel/non-vessel segmentation; [0068] the candidate simulated images are obtained for each acquired 2D ultrasound image in the sequence), detecting an anatomical structure from the 2D image ([0073] acquired ultrasound is segmented to identify vessels and then extract vessel features for comparison with the simulated images) generating an anatomical structure mask (ASM) for the 2D ultrasound image based on the detected anatomical structure ([0062] the acquired ultrasound images are segmented into vessel/non-vessel regions and extract features from that segmentation for each image; the segmentation constitutes a mask identifying the anatomical vessel regions), and predicting, via an ASM-pose mapping model, one or more 3D probe pose estimates of the ultrasound probe based on the ASM ([0074]-[0076] the simulated vessel images are generated for known 3D probe positions, rotations, and translations, and each vessel feature vector is associated with its corresponding probe pose from the lookup tables).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
However, the combination of references is still silent regarding generating an optimal 3D probe pose estimate based on the 3D probe pose estimates.
In the same ultrasound probe localization field of endeavor, Djajadiningrat teaches generating an optimal 3D probe pose estimate based on the 3D probe pose estimates ([0025] pose information is obtained from the images generated from the data stream; [0026] the best pose is chosen, meaning the optimal pose is also generated).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method modified Matsumoto with the pose processing of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill would understand that combining the generation of the optimal 3D probe pose of Djajadiningrat with the predicting ASM pose probe mapping model of modified Matsumoto would read upon the limitation “generating an optimal 3D probe pose estimate based on the 3D probe pose estimates predicted based on ASMs obtained from the 2D ultrasound images acquired by the ultrasound probe at varying 3D probe poses.”
Regarding claim 13, modified Matsumoto teaches the medium of claim 11, but fails to explicitly disclose wherein each of the one or more 3D probe estimates is predicted with a metric indicative of a confidence level of the 3D probe pose estimate.
In the same laparoscope field of endeavor, Ramalhinho teaches wherein each of the one or more 3D probe estimates is predicted with a metric indicative of a confidence level of the 3D probe pose estimate ([0057]-[0059] a similarity value is determined for each simulated image, and the best matching candidate is selected based on that value, and identifying the probe pose from the selected candidate; [0076] selecting the vectors with the lowest distance values, which corresponds to the respective probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 15, modified Matsumoto teaches the medium of claim 14, but fails to explicitly disclose determining a trajectory of the ultrasound probe according to the varying 3D probe poses; identifying 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory.
In the same ultrasound field of endeavor, Ramalhinho teaches determining a trajectory of the ultrasound probe according to the varying 3D probe poses ([0068] using a Viterbi algorithm to determine the most likely sequence of simulated images and the corresponding probe pose for each ultrasound image; [0077] each image corresponds to a successive pose along the path swept by the moving probe; [0081] the lower cost path is used found to find the optimal sequence of candidate probe poses); identifying 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory ([0068]obtaining possible candidate images and corresponding probe poses for each acquire ultrasound image in the sequence; [0074]-[0076] feasible candidate probe poses are identified for each input ultrasound image by comparing feature vectors and selecting candidate vectors having corresponding probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the probe processing of Ramalhinho, as this would help determine an optimal lower cost path through the probe poses (see Ramalhinho [0081]).
However the combination of references still fails to explicitly disclose determining the optimal 3D probe pose estimate based on the 3D probe pose estimates
In the same ultrasound field of endeavor, Djajadiningrat teaches determining the optimal 3D probe pose estimate based on the 3D probe pose estimates ([[0025] system receives a sequence of ultrasound images having respective probe pose information and generates a target probe pose based on the image selection and respective pose information; [0026] user selects a preferred image providing the probe pose, and determines the require pose from the probe pose information associated with the information in the sequence; [0048]-[0049] selecting the image providing the best anatomical view and retrieving the probe pose belonging to the selected image.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the identification process of the 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory as taught by modified Matsumoto with the optimal 3D probe pose estimation process of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill in the art would understand this combination would result in the reading of the limitation “determining the optimal 3D probe pose estimate based on the 3D probe pose estimates predicted with respect to the varying 3D probe poses along the trajectory.”
Regarding claim 16, modified Matsumoto teaches the medium of claim 15, but fails to explicitly disclose selecting one of the 3D probe pose estimates with respect to each of the varying probe poses along the trajectory so that the metrics of the selected predicted 3D probe poses are maximized; and determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses along the trajectory.
In the same ultrasound field of endeavor, Ramalhinho teaches selecting one of the 3D probe pose estimates with respect to each of the varying probe poses along the trajectory so that the metrics of the selected predicted 3D probe poses are maximized ([0026] selecting, for each ultrasound image, candidate images having the lowest distance or highest similarity; [0068] the system obtains candidate images for each acquired ultrasound image and applying a Viterbi algorithm to determine the most likely sequence of simulated images and the corresponding probe pose for each acquire image; [0077]-[0078] successive acquired images are associated with successive probe poses along the path swept by the probe and representing the probability that each acquired image matches each candidate pose; [0081] the optimal sequence of candidate poses is selected as the lowest cost path, with the candidate probabilities optionally weighted by the image similarity distance metric).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the probe processing of Ramalhinho, as this would help determine an optimal lower cost path through the probe poses (see Ramalhinho [0081]).
However, the combination to references still fail to explicitly disclose determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses.
In the same ultrasound field of endeavor, Djajadiningrat teaches determining the optimal 3D probe poses based on the 3D probe pose ([[0025] system receives a sequence of ultrasound images having respective probe pose information and generates a target probe pose based on the image selection and respective pose information; [0026] user selects a preferred image providing the probe pose, and determines the require pose from the probe pose information associated with the information in the sequence; [0048]-[0049] selecting the ultrasound image providing the best anatomical view and retrieving the probe pose belonging to that selected image as the target probe pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the identification process of the 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory as taught by modified Matsumoto with the optimal 3D probe pose estimation process of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill in the art would understand this combination would result in the reading of the limitation “determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses along the trajectory.”
Regarding claim 21, Matsumoto teaches a system comprising ([0060]):
an ultrasound probe pose projector implemented by a process and configured for projecting, during a surgery, one or more three-dimensional (3D) probe information onto a two-dimensional (2D) image at corresponding 2D probe locations (fig. 18 Rendering is processing to project the preoperative preparation information 23 based on the preoperative 3D image 34 onto a two-dimensional projection plane 63 [0097]-[0098]), wherein the 2D image is acquired by a laparoscopic camera inserted into a patient’s body in a surgery directed to a target organ ([0047]-[0048] endoscope, which can be laparoscope, has an camera build in and can be a charge-coupled device image sensor, which takes 2D images); and
an ultrasound-based 3D anatomical structure visualizer implemented by a processor and configured for ([0016] processor is used to generate virtual images and display them):
receiving 2D ultrasound images acquired by an ultrasound probe at or near one of the 2D probe locations ([0049] the probe acquires B mode images; fig. 17 is an example of images taken [0095]).
However, the combination of references still fails to explicitly disclose with respect to each of at least some of the 2D ultrasound image, detecting an anatomical structure from the 2D image, generating an anatomical structure mask (ASM) for the 2D ultrasound image based on the detected anatomical structure, and predicting, via an ASM-pose mapping model, one or more 3D probe pose estimates of the ultrasound probe based on the ASM.
In the same laparoscope field of endeavor, Ramalhinho teaches with respect to each of at least some of the 2D ultrasound image, detecting an anatomical structure from the 2D image ([0073] feature vector comprises size and position of each vessel intersection and is extracted from the ultrasound image and are used to identify the vessels) generating an anatomical structure mask (ASM) for the 2D ultrasound image based on the detected anatomical structure (fig. 5 segmenting an ultrasound image into vessels and non-vessels and a binary image containing vessel sections may be generated [0069]), and predicting, via an ASM-pose mapping model, one or more 3D probe pose estimates of the ultrasound probe based on the ASM ([0074]-[0076] the simulated vessel images are generated for known 3D probe positions, rotations, and translations, and each vessel feature vector is associated with its corresponding probe pose from the lookup tables).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
However, the combination of references are still silent regarding generating an optimal 3D probe pose estimate based on the 3D probe pose estimates.
In the same ultrasound probe localization field of endeavor, Djajadiningrat teaches generating an optimal 3D probe pose estimate based on the 3D probe pose estimates ([0025] pose information is obtained from the images generated from the data stream; [0026] the best pose is chosen, meaning the optimal pose is also generated).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method modified Matsumoto with the pose processing of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill would understand that combining the generation of the optimal 3D probe pose of Djajadiningrat with the predicting ASM pose probe mapping model of modified Matsumoto would read upon the limitation “generating an optimal 3D probe pose estimate based on the 3D probe pose estimates predicted based on ASMs obtained from the 2D ultrasound images acquired by the ultrasound probe at varying 3D probe poses.”
Regarding claim 23, modified Matsumoto teaches the system of claim 21, but fails to explicitly disclose wherein each of the one or more 3D probe estimates is predicted with a metric indicative of a confidence level of the 3D probe pose estimate.
In the same laparoscope field of endeavor, Ramalhinho teaches wherein each of the one or more 3D probe estimates is predicted with a metric indicative of a confidence level of the 3D probe pose estimate ([0057]-[0059] a similarity value is determined for each simulated image, and the best matching candidate is selected based on that value, and identifying the probe pose from the selected candidate; [0076] selecting the vectors with the lowest distance values, which corresponds to the respective probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 25, modified Matsumoto teaches the system of claim 24, but fails to explicitly disclose determining a trajectory of the ultrasound probe according to the varying 3D probe poses; identifying 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory.
In the same ultrasound field of endeavor, Ramalhinho teaches determining a trajectory of the ultrasound probe according to the varying 3D probe poses ([0068] using a Viterbi algorithm to determine the most likely sequence of simulated images and the corresponding probe pose for each ultrasound image; [0077] each image corresponds to a successive pose along the path swept by the moving probe; [0081] the lower cost path is used found to find the optimal sequence of candidate probe poses); identifying 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory ([0068]obtaining possible candidate images and corresponding probe poses for each acquire ultrasound image in the sequence; [0074]-[0076] feasible candidate probe poses are identified for each input ultrasound image by comparing feature vectors and selecting candidate vectors having corresponding probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the probe processing of Ramalhinho, as this would help determine an optimal lower cost path through the probe poses (see Ramalhinho [0081]).
However the combination of references still fails to explicitly disclose determining the optimal 3D probe pose estimate based on the 3D probe pose estimates
In the same ultrasound field of endeavor, Djajadiningrat teaches determining the optimal 3D probe pose estimate based on the 3D probe pose estimates ([[0025] system receives a sequence of ultrasound images having respective probe pose information and generates a target probe pose based on the image selection and respective pose information; [0026] user selects a preferred image providing the probe pose, and determines the require pose from the probe pose information associated with the information in the sequence; [0048]-[0049] selecting the image providing the best anatomical view and retrieving the probe pose belonging to the selected image.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the identification process of the 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory as taught by modified Matsumoto with the optimal 3D probe pose estimation process of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill in the art would understand this combination would result in the reading of the limitation “determining the optimal 3D probe pose estimate based on the 3D probe pose estimates predicted with respect to the varying 3D probe poses along the trajectory.”
Regarding claim 26, modified Matsumoto teaches the system of claim 25, but fails to explicitly disclose selecting one of the 3D probe pose estimates with respect to each of the varying probe poses along the trajectory so that the metrics of the selected predicted 3D probe poses are maximized; and determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses along the trajectory.
In the same ultrasound field of endeavor, Ramalhinho teaches selecting one of the 3D probe pose estimates with respect to each of the varying probe poses along the trajectory so that the metrics of the selected predicted 3D probe poses are maximized ([0026] selecting, for each ultrasound image, candidate images having the lowest distance or highest similarity; [0068] the system obtains candidate images for each acquired ultrasound image and applying a Viterbi algorithm to determine the most likely sequence of simulated images and the corresponding probe pose for each acquire image; [0077]-[0078] successive acquired images are associated with successive probe poses along the path swept by the probe and representing the probability that each acquired image matches each candidate pose; [0081] the optimal sequence of candidate poses is selected as the lowest cost path, with the candidate probabilities optionally weighted by the image similarity distance metric).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the probe processing of Ramalhinho, as this would help determine an optimal lower cost path through the probe poses (see Ramalhinho [0081]).
However, the combination to references still fail to explicitly disclose determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses.
In the same ultrasound field of endeavor, Djajadiningrat teaches determining the optimal 3D probe poses based on the 3D probe pose ([[0025] system receives a sequence of ultrasound images having respective probe pose information and generates a target probe pose based on the image selection and respective pose information; [0026] user selects a preferred image providing the probe pose, and determines the require pose from the probe pose information associated with the information in the sequence; [0048]-[0049] selecting the ultrasound image providing the best anatomical view and retrieving the probe pose belonging to that selected image as the target probe pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the identification process of the 3D probe pose estimates predicted with respect to each of the varying 3D probe poses along the trajectory as taught by modified by modified Matsumoto with the optimal 3D probe pose estimation process of Djajadiningrat, as this would assist the user in correctly positioning the ultrasound probe (see Djajadiningrat [0011]). One of ordinary skill in the art would understand this combination would result in the reading of the limitation “determining the optimal 3D probe pose estimate based on the selected predicted 3D probe poses along the trajectory.”
Claims 2, 12, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Matsumoto in view of Ramalhinho and Djajadiningrat as applied to claim 1 above, and further in view of Di Vece et al., “Ultrasound plane pose regression: assessing generalized pose coordinates in the fetal brain” JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021
Regarding claim 2, modified Matsumoto teaches the method of claim 1, but fails to explicitly disclose wherein the one or more 3D probe poses are specified by a user with respect to a 3D model of the target organ rendered to the user and used of creating pose mapping model.
In the same ultrasound field of endeavor, Di Vece teaches wherein the one or more 3D probe poses are specified by a user with respect to a 3D model of the target organ rendered to the user and used for creating pose mapping model (pg. 4 and figs 2/4 the clinician specifies and saves a desired ultrasound plane pose within a rendered 3D fetal brain volume, and the selected pose and nearby rotated and translated poses are used to create pose labeled training images for the image to pose regression model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the probe processing of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model.
However, the combination of references are silent regarding ASM pose mapping model based on ASMs obtained with respect the one or more 3D probe poses.
In the same ultrasound field endeavor, Ramalhinho teaches ASM pose mapping model based on ASMs obtained with respect the one or more 3D probe poses ([0069] an image containing the anatomical vessel sections are generated for each combination of probe position, rotation, and translation; [0071] the vessel anatomy from the binary image is encoded into a feature vector and associating the vector with the corresponding probe pose; [0073]-[0076] ultrasound image is segmented to identify vessels and compare the anatomical vector with the store features associated with poses to map the segmented image to the 3D probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 12, modified Matsumoto teaches the medium of claim 11, but fails to explicitly disclose wherein the one or more 3D probe poses are specified by a user with respect to a 3D model of the target organ rendered to the user and used of creating pose mapping model.
In the same ultrasound field of endeavor, Di Vece teaches wherein the one or more 3D probe poses are specified by a user with respect to a 3D model of the target organ rendered to the user and used for creating pose mapping model (pg. 4 and figs 2/4 the clinician specifies and saves a desired ultrasound plane pose within a rendered 3D fetal brain volume, and the selected pose and nearby rotated and translated poses are used to create pose labeled training images for the image to pose regression model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the probe processing of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model.
However, the combination of references are silent regarding ASM pose mapping model based on ASMs obtained with respect the one or more 3D probe poses.
In the same ultrasound field endeavor, Ramalhinho teaches ASM pose mapping model based on ASMs obtained with respect the one or more 3D probe poses ([0069] an image containing the anatomical vessel sections are generated for each combination of probe position, rotation, and translation; [0071] the vessel anatomy from the binary image is encoded into a feature vector and associating the vector with the corresponding probe pose; [0073]-[0076] ultrasound image is segmented to identify vessels and compare the anatomical vector with the store features associated with poses to map the segmented image to the 3D probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 22, modified Matsumoto teaches the system of claim 21, but fails to explicitly disclose wherein the one or more 3D probe poses are specified by a user with respect to a 3D model of the target organ rendered to the user and used of creating pose mapping model.
In the same ultrasound field of endeavor, Di Vece teaches wherein the one or more 3D probe poses are specified by a user with respect to a 3D model of the target organ rendered to the user and used for creating pose mapping model (pg. 4 and figs 2/4 the clinician specifies and saves a desired ultrasound plane pose within a rendered 3D fetal brain volume, and the selected pose and nearby rotated and translated poses are used to create pose labeled training images for the image to pose regression model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the probe processing of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model.
However, the combination of references are silent regarding ASM pose mapping model based on ASMs obtained with respect the one or more 3D probe poses.
In the same ultrasound field endeavor, Ramalhinho teaches ASM pose mapping model based on ASMs obtained with respect the one or more 3D probe poses ([0069] an image containing the anatomical vessel sections are generated for each combination of probe position, rotation, and translation; [0071] the vessel anatomy from the binary image is encoded into a feature vector and associating the vector with the corresponding probe pose; [0073]-[0076] ultrasound image is segmented to identify vessels and compare the anatomical vector with the store features associated with poses to map the segmented image to the 3D probe poses.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Claims 4, 14, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Matsumoto in view of Ramalhinho and Djajadiningrat as applied to claim 1 above, and further in view of Sauer et al., (US20020140709A1).
Regarding claim 4, modified Matsumoto teaches the method of claim 3, but fails to teach wherein the varying 3D probe poses are due to motions of the ultrasound probe, wherein the motions include variations in terms of 3D coordinate or 3D orientation of the ultrasound probe.
In the same ultrasound field of endeavor, Ramalhinho teaches wherein the varying 3D probe poses are due to motions of the ultrasound probe, wherein the motions include variations in terms of 3D coordinate or 3D orientation of the ultrasound probe ([0060] for determining a sequence of probe poses corresponding with a sequence of ultrasound images obtained by sweeping the probe over tissue, such as an organ, by registering the sequence of ultrasound images with volumetric scan data; [0077]- [0078] each successive acquired image and pose to movement of the probe along the swept path as modeling the transition between candidate poses for consecutive images using differences in rotation and translation including a 3D difference between probe contact points and an angular difference between imaging plane normals).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
However, the combination of references fails to explicitly disclose around the selected 2D probe location.
In the same instrument tracking field of endeavor, Sauer teaches around the selected 2D probe location ([0056] the user pre selects a target and path, and then displays them as a graphical guide; [0058] using the graphical guide to mark the instrument tip position or entry point;[0113] positioning the instrument tip at an entry point and then swiveling the instrument around the tip; [0117] positioning the tip at the entry point and rotating the instrument around the tip on the entry point).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the laparoscopic ultrasound probe of modified Matsumoto with the selected entry point displayed in the registered camera view that corresponds to the selected 2D probe location as taught by Sauer, as this would enable accurate positioning of medical instrument relative to a target without substantial occlusion (see Sauer [0056]).
Regarding claim 14, modified Matsumoto teaches the medium of claim 13, but fails to teach wherein the varying 3D probe poses are due to motions of the ultrasound probe, wherein the motions include variations in terms of 3D coordinate or 3D orientation of the ultrasound probe.
In the same ultrasound field of endeavor, Ramalhinho teaches wherein the varying 3D probe poses are due to motions of the ultrasound probe, wherein the motions include variations in terms of 3D coordinate or 3D orientation of the ultrasound probe ([0060] for determining a sequence of probe poses corresponding with a sequence of ultrasound images obtained by sweeping the probe over tissue, such as an organ, by registering the sequence of ultrasound images with volumetric scan data; [0077]- [0078] each successive acquired image and pose to movement of the probe along the swept path as modeling the transition between candidate poses for consecutive images using differences in rotation and translation including a 3D difference between probe contact points and an angular difference between imaging plane normals).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
However, the combination of references fails to explicitly disclose around the selected 2D probe location.
In the same instrument tracking field of endeavor, Sauer teaches around the selected 2D probe location ([0056] the user pre selects a target and path, and then displays them as a graphical guide; [0058] using the graphical guide to mark the instrument tip position or entry point;[0113] positioning the instrument tip at an entry point and then swiveling the instrument around the tip; [0117] positioning the tip at the entry point and rotating the instrument around the tip on the entry point).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the laparoscopic ultrasound probe of modified Matsumoto with the selected entry point displayed in the registered camera view that corresponds to the selected 2D probe location as taught by Sauer, as this would enable accurate positioning of medical instrument relative to a target without substantial occlusion (see Sauer [0056]).
Regarding claim 24, modified Matsumoto teaches the system of claim 23, but fails to teach wherein the varying 3D probe poses are due to motions of the ultrasound probe, wherein the motions include variations in terms of 3D coordinate or 3D orientation of the ultrasound probe.
In the same ultrasound field of endeavor, Ramalhinho teaches wherein the varying 3D probe poses are due to motions of the ultrasound probe, wherein the motions include variations in terms of 3D coordinate or 3D orientation of the ultrasound probe ([0060] for determining a sequence of probe poses corresponding with a sequence of ultrasound images obtained by sweeping the probe over tissue, such as an organ, by registering the sequence of ultrasound images with volumetric scan data; [0077]- [0078] each successive acquired image and pose to movement of the probe along the swept path as modeling the transition between candidate poses for consecutive images using differences in rotation and translation including a 3D difference between probe contact points and an angular difference between imaging plane normals).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
However, the combination of references fails to explicitly disclose around the selected 2D probe location.
In the same instrument tracking field of endeavor, Sauer teaches around the selected 2D probe location ([0056] the user pre selects a target and path, and then displays them as a graphical guide; [0058] using the graphical guide to mark the instrument tip position or entry point;[0113] positioning the instrument tip at an entry point and then swiveling the instrument around the tip; [0117] positioning the tip at the entry point and rotating the instrument around the tip on the entry point).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the laparoscopic ultrasound probe of modified Matsumoto with the selected entry point displayed in the registered camera view that corresponds to the selected 2D probe location as taught by Sauer, as this would enable accurate positioning of medical instrument relative to a target without substantial occlusion (see Sauer [0056]).
Claims 7-10, 17-20, and 27-30 are rejected under 35 U.S.C. 103 as being unpatentable over Matsumoto in view of Ramalhinho and Djajadiningrat as applied to claim 1 above, and further in view of Bakhishev et al., (US20230263573A1), Di Vece and Liao et al., (US20170337682A1).
Regarding claim 7, modified Matsumoto teaches the method of claim 1, but fails to teach rendering, to a user prior to the surgery, a 3D model constructed to model the target organ.
In the same image guided probe field of endeavor, Bakhishev teaches rendering, to a user prior to the surgery, a 3D model constructed to model the target organ ([0103]-[0105] the system generates and displays a 3D representation of the patient’s anatomy during the preoperative surgery planning phase).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the 3D rendering of the anatomy as taught by Bakhishev, as this would improve the accuracy and precision of measurements based on placed virtual landmarks (see Bakhishev [0004]).
However the combination of references still fails to explicitly disclose receiving instructions from the user specifying the plurality of 3D probe poses and additional 3D probe poses automatically generating with respect to the plurality of 3D probe poses.
In the same ultrasound field of endeavor, Di Vece teaches receiving instructions from the user specifying the plurality of 3D probe poses (pg. 4/fig. 4 a reconstructed 3D fetal brain ultrasound volume is rendered to a clinician, and the slicing plane is selected to the desired standard plane and saving the corresponding 6D pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the probe processing of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model.
However the combination of references still fails to explicitly disclose obtaining simulated 3D probe poses including the user specified plurality of 3D probe poses and the additional 3D probe poses automatically.
In the same image registration field of endeavor Liao teaches obtaining simulated 3D probe poses including the user specified plurality of 3D probe poses and the additional 3D probe poses automatically ([0064] for each manually labeled ground truth pose, a predetermined number of rigid body perturbation’s within predetermined ranges of transformation parameters are randomly generated).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the clinician specified ultrasound plane poses of modified Matsumoto with the training set augmentation of Liao, as this would allow for efficient and reliably training (see Liao [0064]).
However the combination of references fails to explicitly teach creating the ASM-pose mapping model based on the simulated 3D probe poses ([0069] for each combination of Ps, R, and d, a binary image containing sections is generated; [0071] the feature vector represents the segmented vessel information and concatenated that information with the corresponding probe pose; [0075]-[0076] storing the feature vectors in loop tables, and retrieving candidate feature vectors with their corresponding probe poses)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 8, modified Matsumoto teaches the method of claim 7, but fails to explicitly disclose wherein the additional 3D probe poses are generated randomly within a 3D region defined based on one or more of the plurality of 3D probe poses.
In the same image registration field of endeavor, Liao teaches wherein the additional 3D probe poses are generated randomly within a 3D region defined based on one or more of the plurality of 3D probe poses ([0084] randomly generating rigid body perturbations within 3D translation ranges).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the clinician specified ultrasound plane poses of modified Matsumoto with the training set augmentation of Liao, as this would allow for efficient and reliably training (see Liao [0064]).
Regarding claim 9, modified Matsumoto teaches the method of claim 7, but fails to explicitly disclose with respect to each of the simulated 3D probe poses: determining a perspective based on the simulated 3D probe pose with respect to the rendered 3D model obtaining a simulated two-dimensional (2D) ultrasound image as acquired from the perspective by a simulated ultrasound probe placed at the simulated 3D probe pose, deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound image, and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose; and obtaining the ASM-pose mapping model based on the simulated mapping pairs.
In the same ultrasound field of endeavor, Ramalhinho teaches with respect to each of the simulated 3D probe poses: determining a perspective based on the simulated 3D probe pose with respect to the rendered 3D model ([0069] for each simulated prob position Ps, a virtual reference orientation is determined relative the organ surface normal, rotations and translations are applied and thereby defining the probe imaging plane that intersects the segmented 3D organ model), obtaining a simulated two-dimensional (2D) ultrasound image as acquired from the perspective by a simulated ultrasound probe placed at the simulated 3D probe pose ([0069] for each combination of simulated probe poses, a binary simulated ultrasound image is generated containing the vessel sections intersected by the imaging plane), deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound image ([0069] the target organ model is segmented into blood vessels and non-blood vessels and generated a binary image containing the vessel sections visible at the simulated probe pose), and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose ([0071] the vessel regions are encoded from the binary image as feature triplets and concatenating the mask derived feature information with the corresponding probe pose); and obtaining the ASM-pose mapping model based on the simulated mapping pairs ([0075]-[0076] the feature vectors are organized into lookup tables, and searched for the matching mask derived features, and the corresponding probe poses are retrieved.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the method of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 10, modified Matsumoto teaches the method of claim 9, but fails to explicitly teach generating training data for machine learning based on the simulated mapping pairs.
In the same ultrasound field of endeavor, Di Vece teaches generating training data for machine learning based on the simulated mapping pairs (pg. 4 synthetic ultrasound slices are generated and saving the slices together with their respective poses to provide an automated method of generating a high amount of training data having reliable ground truth pose labels); and training, using the training data, the pose mapping model (pg. 4-5 training a 18 layer residual CNN using ultrasound images and their corresponding ground truth 6D poses so that the network learns to predict a 6D pose from an input ultrasound image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the simulated ASM pose mapping pairs of modified Matsumoto with the machine learning training procedure of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model. One of understand that this combination would result in the reading of the limitation “training, using the training data, the ASM-pose mapping model”.
Regarding claim 17, modified Matsumoto teaches the medium of claim 11, but fails to teach rendering, to a user prior to the surgery, a 3D model constructed to model the target organ.
In the same image guided probe field of endeavor, Bakhishev teaches rendering, to a user prior to the surgery, a 3D model constructed to model the target organ ([0103]-[0105] the system generates and displays a 3D representation of the patient’s anatomy during the preoperative surgery planning phase).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the 3D rendering of the anatomy as taught by Bakhishev, as this would improve the accuracy and precision of measurements based on placed virtual landmarks (see Bakhishev [0004]).
However the combination of references still fails to explicitly disclose receiving instructions from the user specifying the plurality of 3D probe poses and additional 3D probe poses automatically generating with respect to the plurality of 3D probe poses.
In the same ultrasound field of endeavor, Di Vece teaches receiving instructions from the user specifying the plurality of 3D probe poses (pg. 4/fig. 4 a reconstructed 3D fetal brain ultrasound volume is rendered to a clinician, and the slicing plane is selected to the desired standard plane and saving the corresponding 6D pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the probe processing of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model.
However the combination of references still fails to explicitly disclose obtaining simulated 3D probe poses including the user specified plurality of 3D probe poses and the additional 3D probe poses automatically.
In the same image registration field of endeavor Liao teaches obtaining simulated 3D probe poses including the user specified plurality of 3D probe poses and the additional 3D probe poses automatically ([0064] for each manually labeled ground truth pose, a predetermined number of rigid body perturbation’s within predetermined ranges of transformation parameters are randomly generated).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the clinician specified ultrasound plane poses of modified Matsumoto with the training set augmentation of Liao, as this would allow for efficient and reliably training (see Liao [0064]).
However the combination of references fails to explicitly teach creating the ASM-pose mapping model based on the simulated 3D probe poses ([0069] for each combination of Ps, R, and d, a binary image containing sections is generated; [0071] the feature vector represents the segmented vessel information and concatenated that information with the corresponding probe pose; [0075]-[0076] storing the feature vectors in loop tables, and retrieving candidate feature vectors with their corresponding probe poses)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 18, modified Matsumoto teaches the medium of claim 17, but fails to explicitly disclose wherein the additional 3D probe poses are generated randomly within a 3D region defined based on one or more of the plurality of 3D probe poses.
In the same image registration field of endeavor, Liao teaches wherein the additional 3D probe poses are generated randomly within a 3D region defined based on one or more of the plurality of 3D probe poses ([0084] randomly generating rigid body perturbations within 3D translation ranges).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the clinician specified ultrasound plane poses of modified Matsumoto with the training set augmentation of Liao, as this would allow for efficient and reliably training (see Liao [0064]).
Regarding claim 19, modified Matsumoto teaches the medium of claim 17, but fails to explicitly disclose with respect to each of the simulated 3D probe poses: determining a perspective based on the simulated 3D probe pose with respect to the rendered 3D model obtaining a simulated two-dimensional (2D) ultrasound image as acquired from the perspective by a simulated ultrasound probe placed at the simulated 3D probe pose, deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound image, and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose; and obtaining the ASM-pose mapping model based on the simulated mapping pairs.
In the same ultrasound field of endeavor, Ramalhinho teaches with respect to each of the simulated 3D probe poses: determining a perspective based on the simulated 3D probe pose with respect to the rendered 3D model ([0069] for each simulated prob position Ps, a virtual reference orientation is determined relative the organ surface normal, rotations and translations are applied and thereby defining the probe imaging plane that intersects the segmented 3D organ model), obtaining a simulated two-dimensional (2D) ultrasound image as acquired from the perspective by a simulated ultrasound probe placed at the simulated 3D probe pose ([0069] for each combination of simulated probe poses, a binary simulated ultrasound image is generated containing the vessel sections intersected by the imaging plane), deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound image ([0069] the target organ model is segmented into blood vessels and non-blood vessels and generated a binary image containing the vessel sections visible at the simulated probe pose), and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose ([0071] the vessel regions are encoded from the binary image as feature triplets and concatenating the mask derived feature information with the corresponding probe pose); and obtaining the ASM-pose mapping model based on the simulated mapping pairs ([0075]-[0076] the feature vectors are organized into lookup tables, and searched for the matching mask derived features, and the corresponding probe poses are retrieved.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the medium of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 20, modified Matsumoto teaches the medium of claim 19, but fails to explicitly teach generating training data for machine learning based on the simulated mapping pairs.
In the same ultrasound field of endeavor, Di Vece teaches generating training data for machine learning based on the simulated mapping pairs (pg. 4 synthetic ultrasound slices are generated and saving the slices together with their respective poses to provide an automated method of generating a high amount of training data having reliable ground truth pose labels); and training, using the training data, the pose mapping model (pg. 4-5 training a 18 layer residual CNN using ultrasound images and their corresponding ground truth 6D poses so that the network learns to predict a 6D pose from an input ultrasound image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the simulated ASM pose mapping pairs of modified Matsumoto with the machine learning training procedure of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model. One of understand that this combination would result in the reading of the limitation “training, using the training data, the ASM-pose mapping model”.
Regarding claim 27, modified Matsumoto teaches the system of claim 21, but fails to teach rendering, to a user prior to the surgery, a 3D model constructed to model the target organ.
In the same image guided probe field of endeavor, Bakhishev teaches rendering, to a user prior to the surgery, a 3D model constructed to model the target organ ([0103]-[0105] the system generates and displays a 3D representation of the patient’s anatomy during the preoperative surgery planning phase).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the 3D rendering of the anatomy as taught by Bakhishev, as this would improve the accuracy and precision of measurements based on placed virtual landmarks (see Bakhishev [0004]).
However the combination of references still fails to explicitly disclose receiving instructions from the user specifying the plurality of 3D probe poses and additional 3D probe poses automatically generating with respect to the plurality of 3D probe poses.
In the same ultrasound field of endeavor, Di Vece teaches receiving instructions from the user specifying the plurality of 3D probe poses (pg. 4/fig. 4 a reconstructed 3D fetal brain ultrasound volume is rendered to a clinician, and the slicing plane is selected to the desired standard plane and saving the corresponding 6D pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the probe processing of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model.
However the combination of references still fails to explicitly disclose obtaining simulated 3D probe poses including the user specified plurality of 3D probe poses and the additional 3D probe poses automatically.
In the same image registration field of endeavor Liao teaches obtaining simulated 3D probe poses including the user specified plurality of 3D probe poses and the additional 3D probe poses automatically ([0064] for each manually labeled ground truth pose, a predetermined number of rigid body perturbation’s within predetermined ranges of transformation parameters are randomly generated).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the clinician specified ultrasound plane poses of modified Matsumoto with the training set augmentation of Liao, as this would allow for efficient and reliably training (see Liao [0064]).
However the combination of references fails to explicitly teach creating the ASM-pose mapping model based on the simulated 3D probe poses ([0069] for each combination of Ps, R, and d, a binary image containing sections is generated; [0071] the feature vector represents the segmented vessel information and concatenated that information with the corresponding probe pose; [0075]-[0076] storing the feature vectors in loop tables, and retrieving candidate feature vectors with their corresponding probe poses)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 28, modified Matsumoto teaches the system of claim 27, but fails to explicitly disclose wherein the additional 3D probe poses are generated randomly within a 3D region defined based on one or more of the plurality of 3D probe poses.
In the same image registration field of endeavor, Liao teaches wherein the additional 3D probe poses are generated randomly within a 3D region defined based on one or more of the plurality of 3D probe poses ([0084] randomly generating rigid body perturbations within 3D translation ranges).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the clinician specified ultrasound plane poses of modified Matsumoto with the training set augmentation of Liao, as this would allow for efficient and reliably training (see Liao [0064]).
Regarding claim 29, modified Matsumoto teaches the system of claim 27, but fails to explicitly disclose with respect to each of the simulated 3D probe poses: determining a perspective based on the simulated 3D probe pose with respect to the rendered 3D model obtaining a simulated two-dimensional (2D) ultrasound image as acquired from the perspective by a simulated ultrasound probe placed at the simulated 3D probe pose, deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound image, and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose; and obtaining the ASM-pose mapping model based on the simulated mapping pairs.
In the same ultrasound field of endeavor, Ramalhinho teaches with respect to each of the simulated 3D probe poses: determining a perspective based on the simulated 3D probe pose with respect to the rendered 3D model ([0069] for each simulated prob position Ps, a virtual reference orientation is determined relative the organ surface normal, rotations and translations are applied and thereby defining the probe imaging plane that intersects the segmented 3D organ model), obtaining a simulated two-dimensional (2D) ultrasound image as acquired from the perspective by a simulated ultrasound probe placed at the simulated 3D probe pose ([0069] for each combination of simulated probe poses, a binary simulated ultrasound image is generated containing the vessel sections intersected by the imaging plane), deriving an anatomical structure mask (ASM) based on anatomical structures in the simulated 2D ultrasound image ([0069] the target organ model is segmented into blood vessels and non-blood vessels and generated a binary image containing the vessel sections visible at the simulated probe pose), and creating a simulated mapping pair involving the ASM and the simulated 3D probe pose ([0071] the vessel regions are encoded from the binary image as feature triplets and concatenating the mask derived feature information with the corresponding probe pose); and obtaining the ASM-pose mapping model based on the simulated mapping pairs ([0075]-[0076] the feature vectors are organized into lookup tables, and searched for the matching mask derived features, and the corresponding probe poses are retrieved.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the system of modified Matsumoto with the vessel segmentation processing of Ramalhinho, as this would allow a simple method of registering ultrasound scan images (see Ramalhinho [0006]).
Regarding claim 30, modified Matsumoto teaches the system of claim 29, but fails to explicitly teach generating training data for machine learning based on the simulated mapping pairs.
In the same ultrasound field of endeavor, Di Vece teaches generating training data for machine learning based on the simulated mapping pairs (pg. 4 synthetic ultrasound slices are generated and saving the slices together with their respective poses to provide an automated method of generating a high amount of training data having reliable ground truth pose labels); and training, using the training data, the pose mapping model (pg. 4-5 training a 18 layer residual CNN using ultrasound images and their corresponding ground truth 6D poses so that the network learns to predict a 6D pose from an input ultrasound image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to modify the simulated ASM pose mapping pairs of modified Matsumoto with the machine learning training procedure of Di Vece, as using a clinician specified pose and generating nearby poses through rotations and translations provides an automated way of generating a large amount training data with reliable ground truth pose labels (see Di Vece pg. 5-6). This would facilitate effective training of the pose model. One of understand that this combination would result in the reading of the limitation “training, using the training data, the ASM-pose mapping model”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL Y FANG whose telephone number is (571)272-0952. The examiner can normally be reached Mon - Friday 9:30 am - 6:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at 5712722714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL YIMING FANG/ Examiner, Art Unit 3798
/PASCAL M BUI PHO/ Supervisory Patent Examiner, Art Unit 3798
