DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Claim 1 & 11 recites the limitation of “upon determining that a sequence of previous 3D probe poses exists, the sequence of previous 3D probe poses representing prior 3D probe poses of the ultrasound probe” which in an interpretation it may be construed as a conditional limitation where the conditional limitations may not be given a full weight in light of the below decisions as for considering the other case scenario of “upon determining that a sequence of previous 3D probe poses exists” not being advanced… which the claim would not require this limitation to be a positive recitation.
In the recent Ex parte Gopalan decision, the PTAB addressed a claim where all of the features were recited in a conditional manner. A first step of “identifying … an outlier” was performed if “traffic is outside of a prediction interval.” A second step of “identifying” was performed “only when a count of outliers … is greater than or equal to two, and exceeds an anomaly threshold.” These were the only two elements of the independent claim. Thus, if the traffic is never outside Gopalan’s prediction interval, then the steps of the method are never performed.
However, the PTAB distinguished Schulhauser and noted that this construction “would render the entire claim meaningless.” Gopalan at p. 5. The Board went on to state, “Although each of these steps is conditional, they are integrated into one method or path and do not cause the claim to diverge into two methods or paths, as in Schulhauser. Thus, we conclude that the broadest reasonable interpretation of claim 1 requires the performance of both steps…” Id. at p. 6.”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ramalhinho et al (US20220249056A1; hereinafter referred to as Ramalhinho) in view of Yan et al (US20160367216A1; hereinafter referred to as Yan).
Regarding Claim 1, Ramalhinho discloses a method (“A computer implemented method is disclosed for identifying a pose of a probe by registering an ultrasound image from with volumetric scan data.” [Abstract]), comprising:
receiving an ultrasound image acquired by an ultrasound probe deployed at a three- dimensional (3D) probe pose during a medical procedure involving a target organ (“The ultrasound probe 24 may be a laparoscopic ultrasound probe that is configured to obtain ultrasound data for generating an ultrasound image of an organ during a laparoscopic surgical procedure.” [0066]);
detecting each anatomical structure from the ultrasound image (“a corresponding feature vector must be extracted from the ultrasound image. For embodiments where the feature vector encodes the position and area of vessels intersecting the imaging plane, the ultrasound image must be segmented to identify the vessels, to produce a feature vector that can be compared with the feature vectors obtained from each simulated ultrasound image. The ultrasound image may be automatically segmented, for example using a convolutional neural network (e.g. as described in reference 10).” [0073]);
generating an anatomical structure mask (ASM) for each anatomical structure (“The model of the volumetric scan data may be segmented to indicate “blood vessel” and “not blood vessel”.” [0069], “FIG. 5 shows an example simulated ultrasound image 20, comprising black regions that are not vessels, and white regions 22 that are vessels. In this example image there are 8 separate (non-contiguous) vessel regions, in various positions. The vessel information content of the simulated ultrasound image 20 may be captured in a feature vector ƒ. In one embodiment, the centroid (with respect to the probe co-ordinates x and y) and the area of each vessel may be encoded in a feature triplet ƒi. The triplets may be concatenated with the pose of the probe (comprising the position P, rotation R and translation d), to form the feature vector ƒ as illustrated in FIG. 5.” [0071]);
estimating an identifier for each ASM, resulting in an ASM/identifier pair (“FIG. 5 shows an example simulated ultrasound image 20, comprising black regions that are not vessels, and white regions 22 that are vessels. In this example image there are 8 separate (non-contiguous) vessel regions, in various positions. The vessel information content of the simulated ultrasound image 20 may be captured in a feature vector ƒ. In one embodiment, the centroid (with respect to the probe co-ordinates x and y) and the area of each vessel may be encoded in a feature triplet ƒi. The triplets may be concatenated with the pose of the probe (comprising the position P, rotation R and translation d), to form the feature vector ƒ as illustrated in FIG. 5.” [0071],” To perform an efficient search, it is possible (but not essential) to only search for feature vectors that have a similar number of triplets (corresponding with vessel sections) to the input ƒ1. Feature vectors may be grouped in lookup tables FM according to their size M. The search for the best candidates ƒ* may as expressed in equation (2):” [0075]);
upon determining that a sequence of previous 3D probe poses exists, the sequence of previous 3D probe poses representing prior 3D probe poses of the ultrasound probe (“Once a set of k possible matches {J1i, . . . , JKi} are obtained for image Ii a transition probability may be used to determine a set of simulated images from J that match the set of acquired images {I1, . . . , IN} acquired by sweeping the probe over the surface of the organ. Under these conditions, each successive acquired image will correspond with a successive pose along the path swept by the probe as it moves over the surface of the organ. This imposes a kinematic constraint on the set of images selected from J to match the acquired images {I1, . . . , IN} because solutions that require very high acceleration and/or velocity are very unlikely to be correct.” [0077]):
resulting in a predicted 3D probe pose (“This can be formulated as a hidden Markov model, as shown in FIG. 6. Nodes in FIG. 6 represent a probability of each acquired image Ii matching a candidate simulated image Jki, and edges represent a probability associated with an input kinematic prior. The kinematic prior may be selected to constrain the selected images to match a condition of the kinematics of the acquisition. For example, it may be assumed that there is smoothness in the acquisition, and/or it may be assumed that the probe follows a continuous path along a direction normal to the imaging plane without moving backwards.” [0078], “During optimisation, a constraint may be implemented to reject candidate simulated image sets that do not fulfil specific kinematic conditions. For example, a sweep direction may be defined as the difference between the first two probe positions Pk2−Pk1 in the candidate simulated image set. The probability P(Ii|Jki) may be set to 0 (or reduced by a predetermined amount or ratio) if the angle between Pki+i−Pki and the sweep direction is above 90 degrees (or some other predetermined angle or variance).” [0082])
creating a virtual ultrasound image based on the predicted 3D probe pose (“FIG. 4 illustrates the generation of an example simulated US image from CT scan data. A model of the liver 14 is obtained from the CT scan data along with a model of the inferior vena cava 10. The blood vessels 22 of the liver 14, including the hepatic vein 12 and portal vein 16, as are also modelled. The simulated probe position P is shown, with probe body co-ordinates x, y and z. The field of view of the simulated scan 20 is indicated, which intersects with a number of blood vessels 22 a, 22 b etc.” [0070], “This can be formulated as a hidden Markov model, as shown in FIG. 6. Nodes in FIG. 6 represent a probability of each acquired image Ii matching a candidate simulated image Jki, and edges represent a probability associated with an input kinematic prior. The kinematic prior may be selected to constrain the selected images to match a condition of the kinematics of the acquisition. For example, it may be assumed that there is smoothness in the acquisition, and/or it may be assumed that the probe follows a continuous path along a direction normal to the imaging plane without moving backwards.” [0078]);
extracting at least one ASM/identifier pair from the virtual ultrasound image, resulting in at least one extracted ASM/identifier pair; generating an ASM/identifier representation of the virtual ultrasound image based on the at least one extracted ASM/identifier pair (“The set of simulated US images J may be obtained by intersecting a segmented model of the volumetric scan data with 2D planes, bounded by an LUS field of view. The model of the volumetric scan data may be segmented to indicate “blood vessel” and “not blood vessel”. A set of evenly distributed points PS may be generated over the surface of the organ of interest (e.g. liver). At each of these points PS a virtual reference orientation may be created, orthogonal to the organ surface normal and with the imaging plane aligned with the sagittal plane. At each point PS, different combinations of rotations Rx, Ry, and Rz may be applied to generate simulated US images corresponding with rotated projections parameterised by R=[{right arrow over (x)}, {right arrow over (y)}, {right arrow over (z)}]. In addition, at each point PS, a number of translations d may be applied along the organ surface normal, simulating the case in which the probe compresses the tissue of the organ and images deeper structures. For each combination of PS, R and d, a binary image containing vessel sections may be generated.” [0069], “FIG. 5 shows an example simulated ultrasound image 20, comprising black regions that are not vessels, and white regions 22 that are vessels. In this example image there are 8 separate (non-contiguous) vessel regions, in various positions. The vessel information content of the simulated ultrasound image 20 may be captured in a feature vector ƒ. In one embodiment, the centroid (with respect to the probe co-ordinates x and y) and the area of each vessel may be encoded in a feature triplet ƒi. The triplets may be concatenated with the pose of the probe (comprising the position P, rotation R and translation d), to form the feature vector ƒ as illustrated in FIG. 5.” [0071]);
and estimating the 3D probe pose based on the ASM/identifier representation of the virtual ultrasound image via an ASM-pose mapping model, wherein the ASM-pose mapping model that maps the ASM/label representation to a 3D probe pose(“extracting a feature vector from each of the simulated ultrasound images and from the ultrasound image; comparing the feature vector from the ultrasound image with each feature vector of the simulated ultrasound images to determine a distance or similarity for each simulated ultrasound image; and selecting at least one candidate image, the at least one candidate image comprising a subset of the simulated ultrasound images that best matches the ultrasound image, based on the distance or similarity; identifying the pose of the probe from the at least one candidate image” [Claim 1], “To perform an efficient search, it is possible (but not essential) to only search for feature vectors that have a similar number of triplets (corresponding with vessel sections) to the input ƒ1. Feature vectors may be grouped in lookup tables FM according to their size M. The search for the best candidates ƒ* may as expressed in equation (2)” [0075], “Once a set of k possible matches {J1i, . . . , JKi} are obtained for image Ii a transition probability may be used to determine a set of simulated images from J that match the set of acquired images {I1, . . . , IN} acquired by sweeping the probe over the surface of the organ. Under these conditions, each successive acquired image will correspond with a successive pose along the path swept by the probe as it moves over the surface of the organ.” [0077]).
Ramalhinho does not specifically disclose the identifier being a label, and predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses.
However, in a similar field of endeavor, Tregidgo teaches a proposed registration approach using Content-Based Image Retrieval [Abstract]
Tregidgo also teaches the identifier being a label (“In this work, we generalise our CBIR system to include multiple labels in the vessel feature encoding which increases registration performance. In the specific case of liver imaging, this is possible by labelling different vessels as branches of the portal vein or branches of the hepatic vein.” [Pg. 1043 Col. 2])
It would have been obvious to an ordinary skilled person in the art before the effective filing
date of the claimed invention to modify the system of Ramalhinho as outlined above with the identifier being a label as taught by Tregidgo, because it enables the registration problem to be accurately initialised without tracking data [Pg. 1043 Col. 2].
Ramalhinho in view of Tregidgo does not specifically teach predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses.
However, in a similar field of endeavor, Sprung teaches A method of determining a three-dimensional motion of a movable ultrasound probe [Abstract].
Sprung also teaches predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses (“the method comprises determining, from the probe motion indicator (from the relative three-dimensional displacement and rotation between the ultrasound image frames), a probe position and orientation of the ultrasound probe. The probe position and orientation may be obtained by discrete integration of multiple probe motion indicators.” [0103], “the method comprises directly predicting the ultrasound probe motion from the stream of ultrasound images, without the input of any external tracking system, and optionally based on only the image data” [0108])
It would have been obvious to an ordinary skilled person in the art before the effective filing
date of the claimed invention to modify the system of Ramalhinho in view of Tregidgo as outlined above with predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses as taught by Sprung, because it can allow for estimating the in-plane motion quite reliably [0004].
Regarding Claim 2, Ramalhinho discloses the step of predicting comprises: generating a first trajectory of 3D coordinates of the prior 3D probe poses in the sequence (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065]);
generating a second trajectory of 3D orientations of the prior 3D probe poses in the sequence; generating the predicted 3D probe pose based on the first and the second trajectories (“The mean number of plausible paths 2 for each of the nine sweep registrations vs the number of images is shown in FIG. 8. Since the Viterbi algorithm is recursive on the number of columns in the hidden Markov model (shown in FIG. 6), results are displayed as a function of the number of images used so far in the optimisation (from 2 to 20). FIG. 8 therefore shows the number of kinematically possible paths for N images (i.e. with a non-zero probability, based on the constraints defined above). The number of plausible trajectories found by the algorithm converges to 1 if enough images are used (N=17 in this case).” [0085], multiple trajectories are predicted as being the potentially correct trajectory as more images are acquired the projected trajectory narrows to the closest matching trajectory),
wherein the virtual ultrasound image is created based on a 3D model for the target organ in accordance with the predicted 3D probe pose (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065]).
Regarding Claim 3, Ramalhinho discloses the step of extracting the at least one ASM from a virtual ultrasound image comprises: identifying each 2D structure in the virtual ultrasound image corresponding to a 3D anatomical structure in the 3D model; generating a mask for the 2D structure to create a corresponding ASM(“FIG. 1 is a sequence of steps 30 according to an embodiment of the invention, for determining a probe pose corresponding with an ultrasound image by registering the ultrasound image with volumetric scan data. At step 31, the volumetric scan data is processed to determine a plurality of simulated ultrasound images corresponding with different poses of the probe (e.g. at least one of position, orientation, depth/deformation). At step 32, a feature vector is extracted from each of the simulated ultrasound images, and from the ultrasound image. The feature vector may comprise a position and size of each vessel intersection with the respective image. The feature vector may be obtained by segmentation of the images into vessels and not-vessels. At step 33, the feature vector from each simulated ultrasound image is compared with the feature vector from the ultrasound image to determine a distance or similarity value.” [0054-0057]) ;
Ramalhinho does not specifically disclose assigning a label for the 3D anatomical structure retrieved from the 3D model to the ASM to generate the ASM/label pair.
However, in a similar field of endeavor, Tregidgo teaches assigning a label for the 3D anatomical structure retrieved from the 3D model to the ASM to generate the ASM/label pair ((“In this work, we generalise our CBIR system to include multiple labels in the vessel feature encoding which increases registration performance. In the specific case of liver imaging, this is possible by labelling different vessels as branches of the portal vein or branches of the hepatic vein.” [Pg. 1043 Col. 2])
It would have been obvious to an ordinary skilled person in the art before the effective filing
date of the claimed invention to modify the system of Ramalhinho as outlined above with assigning a label for the 3D anatomical structure retrieved from the 3D model to the ASM to generate the ASM/label pair as taught by Tregidgo, because it enables the registration problem to be accurately initialised without tracking data [Pg. 1043 Col. 2].
Regarding Claim 4, Ramalhinho discloses further comprising refining the estimated 3D probe pose by updating the ASM/label representation based on the estimated 3D probe pose to generate an updated ASM/label representation for the ultrasound image, wherein the updating comprises: creating a new virtual ultrasound image based on the estimated 3D probe pose in accordance with a 3D model for the target organ, the new virtual ultrasound image providing ground truth ASM/label pairs (“FIG. 2 illustrates a sequence of steps 40, according to an embodiment of the invention, for determining a sequence of probe poses corresponding with a sequence of ultrasound images obtained by sweeping the probe over tissue, such as an organ, by registering the sequence of ultrasound images with volumetric scan data. At step 41, the volumetric scan data is processed to determine a plurality of simulated ultrasound images corresponding with different poses of the probe (e.g. at least one of position, orientation, depth/deformation). At step 42, a feature vector is extracted from each of the simulated ultrasound images, and from each of the sequence of ultrasound images. The feature vector may comprise a position and size of each vessel intersection with the respective image. The feature vector may be obtained by segmentation of the images into vessels and not-vessels.” [0060-0062]);
extracting one or more virtual ASM/label pairs from the new virtual ultrasound image; for each ASM/label pair generated based on the ultrasound image, Identifying a corresponding virtual ASM/label pair, revising the ASM/label pair from the ultrasound image if it satisfies at least one predetermined criterion with respect to the virtual ASM/label pair; and generating an update ASM/label representation based on the revised ASM/label pair (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065], “In the example described herein, it is implicit that the organ does not deform. In some embodiments, the set of simulated ultrasound images obtained from the volumetric scan may be parameterised to include deformation (e.g. in the y direction). In some embodiments the depth d parameter may represent deformation of the organ in a direction normal to the surface of the organ (rather than a simple translation without deformation). Higher accuracies may be achievable with parameterisation including deformation.” [0096]).
Regarding Claim 5, Ramalhinho discloses the step of revising according to at least one predetermined criterion comprises: if an overlap between the ASM in the ASM/label pair and the ASM in the virtual ASM/label pair is not acceptable, removing the ASM/label pair; if the overlap between the ASM in the ASM/label pair and the ASM in the virtual ASM/label pair is acceptable, replacing the label from the ASM/label pair with the label from the virtual ASM/label pair pair (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065], “The mean number of plausible paths 2 for each of the nine sweep registrations vs the number of images is shown in FIG. 8.” [0085], “In the example described herein, it is implicit that the organ does not deform. In some embodiments, the set of simulated ultrasound images obtained from the volumetric scan may be parameterised to include deformation (e.g. in the y direction). In some embodiments the depth d parameter may represent deformation of the organ in a direction normal to the surface of the organ (rather than a simple translation without deformation). Higher accuracies may be achievable with parameterisation including deformation.” [0096], as shown in Fig.8 the more ultrasound images analyzed the less possibilities for the potential number of paths (3D probe poses)).
PNG
media_image1.png
345
417
media_image1.png
Greyscale
Regarding Claim 6, Ramalhinho discloses further comprising refining the estimated 3D probe pose to generate an updated 3D probe pose based on the updated ASM/label representation, wherein the step of refining the estimated 3D probe pose comprises: obtaining a new 3D probe pose based on the updated ASM/label representation via the ASM-pose mapping model; if the new 3D probe pose and the estimated 3D probe pose satisfy a configured condition, outputting the new 3D probe pose as the updated 3D probe pose; and if the new 3D probe pose and the estimated 3D probe pose do not satisfy the configured condition, outputting the estimated 3D probe pose as the updated 3D probe pose (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065]).
Regarding Claim 7, Ramalhinho discloses that the configured condition defines that a deviation between the new 3D probe pose and the 3D probe pose is at an acceptable range (“At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0065], “imposing a transition probability penalty when a probe path direction deviates from an initial direction by more than a threshold amount.” [0024]).
Regarding Claim 8, Ramalhinho discloses further comprising adding the updated 3D probe pose to the sequence of previous 3D probe poses of the ultrasound probe (“At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0065], as shown in Fig.8 the more ultrasound images analyzed the less possibilities for the potential number of paths (3D probe poses) and the sequence is optimized for the highest probability sequence).
PNG
media_image1.png
345
417
media_image1.png
Greyscale
Regarding Claim 9, Ramalhinho discloses the ASM-pose mapping model is created prior to the medical procedure by: retrieving a 3D model of the target organ; generating a plurality of virtual 3D probe poses in connection with an ultrasound probe; creating a virtual ultrasound image with respect to each of the plurality of virtual 3D probe poses; obtaining an ASM-pose pairing for each of the plurality of virtual 3D probe poses, where the ASM-pose pairing includes a virtual 3D probe pose and an ASM/label representation for the virtual ultrasound image created with respect to the virtual 3D probe pose; and establishing the ASM-pose mapping model for mapping an ASM/label representation to a 3D probe pose based on the ASM-pose pairings (“FIG. 1 is a sequence of steps 30 according to an embodiment of the invention, for determining a probe pose corresponding with an ultrasound image by registering the ultrasound image with volumetric scan data. At step 31, the volumetric scan data is processed to determine a plurality of simulated ultrasound images corresponding with different poses of the probe (e.g. at least one of position, orientation, depth/deformation). At step 32, a feature vector is extracted from each of the simulated ultrasound images, and from the ultrasound image. The feature vector may comprise a position and size of each vessel intersection with the respective image. The feature vector may be obtained by segmentation of the images into vessels and not-vessels. At step 33, the feature vector from each simulated ultrasound image is compared with the feature vector from the ultrasound image to determine a distance or similarity value. At step 34, a candidate image is selected as the best match, based on the distance or similarity. At step 35, the pose of the probe is identified from the candidate image” [0054-0059]).
Regarding Claim 10, Ramalhinho discloses the step of establishing comprises: generating training data based on the ASM-pose pairings; obtaining, via machine learning based on the training data, the ASM-pose mapping model to learn relationships between ASM/label representations and 3D probe poses. (“The feature vector may be extracted using a convolutional neural network. The convolutional neural network may have been trained to distinguish between ultrasound images.” [0029-0030]).
Regarding Claim 11, A machine-readable and non-transitory medium having information recorded thereon, wherein the information, when read by the machine, (“A computer implemented method is disclosed for identifying a pose of a probe by registering an ultrasound image from with volumetric scan data.” [Abstract], “there is provided a non-transient machine readable medium comprising instructions for configuring a processor to perform the method of the first aspect, including any of the optional features thereof.” [0035]) causes the machine to perform the following steps::
receiving an ultrasound image acquired by an ultrasound probe deployed at a three- dimensional (3D) probe pose during a medical procedure involving a target organ (“The ultrasound probe 24 may be a laparoscopic ultrasound probe that is configured to obtain ultrasound data for generating an ultrasound image of an organ during a laparoscopic surgical procedure.” [0066]);
detecting each anatomical structure from the ultrasound image (“a corresponding feature vector must be extracted from the ultrasound image. For embodiments where the feature vector encodes the position and area of vessels intersecting the imaging plane, the ultrasound image must be segmented to identify the vessels, to produce a feature vector that can be compared with the feature vectors obtained from each simulated ultrasound image. The ultrasound image may be automatically segmented, for example using a convolutional neural network (e.g. as described in reference 10).” [0073]);
generating an anatomical structure mask (ASM) for each anatomical structure (“The model of the volumetric scan data may be segmented to indicate “blood vessel” and “not blood vessel”.” [0069], “FIG. 5 shows an example simulated ultrasound image 20, comprising black regions that are not vessels, and white regions 22 that are vessels. In this example image there are 8 separate (non-contiguous) vessel regions, in various positions. The vessel information content of the simulated ultrasound image 20 may be captured in a feature vector ƒ. In one embodiment, the centroid (with respect to the probe co-ordinates x and y) and the area of each vessel may be encoded in a feature triplet ƒi. The triplets may be concatenated with the pose of the probe (comprising the position P, rotation R and translation d), to form the feature vector ƒ as illustrated in FIG. 5.” [0071]);
estimating an identifier for each ASM, resulting in an ASM/identifier pair (“FIG. 5 shows an example simulated ultrasound image 20, comprising black regions that are not vessels, and white regions 22 that are vessels. In this example image there are 8 separate (non-contiguous) vessel regions, in various positions. The vessel information content of the simulated ultrasound image 20 may be captured in a feature vector ƒ. In one embodiment, the centroid (with respect to the probe co-ordinates x and y) and the area of each vessel may be encoded in a feature triplet ƒi. The triplets may be concatenated with the pose of the probe (comprising the position P, rotation R and translation d), to form the feature vector ƒ as illustrated in FIG. 5.” [0071],” To perform an efficient search, it is possible (but not essential) to only search for feature vectors that have a similar number of triplets (corresponding with vessel sections) to the input ƒ1. Feature vectors may be grouped in lookup tables FM according to their size M. The search for the best candidates ƒ* may as expressed in equation (2):” [0075]);
upon determining that a sequence of previous 3D probe poses exists, the sequence of previous 3D probe poses representing prior 3D probe poses of the ultrasound probe (“Once a set of k possible matches {J1i, . . . , JKi} are obtained for image Ii a transition probability may be used to determine a set of simulated images from J that match the set of acquired images {I1, . . . , IN} acquired by sweeping the probe over the surface of the organ. Under these conditions, each successive acquired image will correspond with a successive pose along the path swept by the probe as it moves over the surface of the organ. This imposes a kinematic constraint on the set of images selected from J to match the acquired images {I1, . . . , IN} because solutions that require very high acceleration and/or velocity are very unlikely to be correct.” [0077]):
resulting in a predicted 3D probe pose (“This can be formulated as a hidden Markov model, as shown in FIG. 6. Nodes in FIG. 6 represent a probability of each acquired image Ii matching a candidate simulated image Jki, and edges represent a probability associated with an input kinematic prior. The kinematic prior may be selected to constrain the selected images to match a condition of the kinematics of the acquisition. For example, it may be assumed that there is smoothness in the acquisition, and/or it may be assumed that the probe follows a continuous path along a direction normal to the imaging plane without moving backwards.” [0078], “During optimisation, a constraint may be implemented to reject candidate simulated image sets that do not fulfil specific kinematic conditions. For example, a sweep direction may be defined as the difference between the first two probe positions Pk2−Pk1 in the candidate simulated image set. The probability P(Ii|Jki) may be set to 0 (or reduced by a predetermined amount or ratio) if the angle between Pki+i−Pki and the sweep direction is above 90 degrees (or some other predetermined angle or variance).” [0082])
creating a virtual ultrasound image based on the predicted 3D probe pose (“FIG. 4 illustrates the generation of an example simulated US image from CT scan data. A model of the liver 14 is obtained from the CT scan data along with a model of the inferior vena cava 10. The blood vessels 22 of the liver 14, including the hepatic vein 12 and portal vein 16, as are also modelled. The simulated probe position P is shown, with probe body co-ordinates x, y and z. The field of view of the simulated scan 20 is indicated, which intersects with a number of blood vessels 22 a, 22 b etc.” [0070], “This can be formulated as a hidden Markov model, as shown in FIG. 6. Nodes in FIG. 6 represent a probability of each acquired image Ii matching a candidate simulated image Jki, and edges represent a probability associated with an input kinematic prior. The kinematic prior may be selected to constrain the selected images to match a condition of the kinematics of the acquisition. For example, it may be assumed that there is smoothness in the acquisition, and/or it may be assumed that the probe follows a continuous path along a direction normal to the imaging plane without moving backwards.” [0078]);
extracting at least one ASM/identifier pair from the virtual ultrasound image, resulting in at least one extracted ASM/identifier pair; generating an ASM/identifier representation of the virtual ultrasound image based on the at least one extracted ASM/identifier pair (“The set of simulated US images J may be obtained by intersecting a segmented model of the volumetric scan data with 2D planes, bounded by an LUS field of view. The model of the volumetric scan data may be segmented to indicate “blood vessel” and “not blood vessel”. A set of evenly distributed points PS may be generated over the surface of the organ of interest (e.g. liver). At each of these points PS a virtual reference orientation may be created, orthogonal to the organ surface normal and with the imaging plane aligned with the sagittal plane. At each point PS, different combinations of rotations Rx, Ry, and Rz may be applied to generate simulated US images corresponding with rotated projections parameterised by R=[{right arrow over (x)}, {right arrow over (y)}, {right arrow over (z)}]. In addition, at each point PS, a number of translations d may be applied along the organ surface normal, simulating the case in which the probe compresses the tissue of the organ and images deeper structures. For each combination of PS, R and d, a binary image containing vessel sections may be generated.” [0069], “FIG. 5 shows an example simulated ultrasound image 20, comprising black regions that are not vessels, and white regions 22 that are vessels. In this example image there are 8 separate (non-contiguous) vessel regions, in various positions. The vessel information content of the simulated ultrasound image 20 may be captured in a feature vector ƒ. In one embodiment, the centroid (with respect to the probe co-ordinates x and y) and the area of each vessel may be encoded in a feature triplet ƒi. The triplets may be concatenated with the pose of the probe (comprising the position P, rotation R and translation d), to form the feature vector ƒ as illustrated in FIG. 5.” [0071]);
and estimating the 3D probe pose based on the ASM/identifier representation of the virtual ultrasound image via an ASM-pose mapping model, wherein the ASM-pose mapping model that maps the ASM/label representation to a 3D probe pose(“extracting a feature vector from each of the simulated ultrasound images and from the ultrasound image; comparing the feature vector from the ultrasound image with each feature vector of the simulated ultrasound images to determine a distance or similarity for each simulated ultrasound image; and selecting at least one candidate image, the at least one candidate image comprising a subset of the simulated ultrasound images that best matches the ultrasound image, based on the distance or similarity; identifying the pose of the probe from the at least one candidate image” [Claim 1], “To perform an efficient search, it is possible (but not essential) to only search for feature vectors that have a similar number of triplets (corresponding with vessel sections) to the input ƒ1. Feature vectors may be grouped in lookup tables FM according to their size M. The search for the best candidates ƒ* may as expressed in equation (2)” [0075], “Once a set of k possible matches {J1i, . . . , JKi} are obtained for image Ii a transition probability may be used to determine a set of simulated images from J that match the set of acquired images {I1, . . . , IN} acquired by sweeping the probe over the surface of the organ. Under these conditions, each successive acquired image will correspond with a successive pose along the path swept by the probe as it moves over the surface of the organ.” [0077]).
Ramalhinho does not specifically disclose the identifier being a label, predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses; and creating a virtual image after predicting a 3D probe pose.
However, in a similar field of endeavor, Tregidgo teaches a proposed registration approach using Content-Based Image Retrieval [Abstract]
Tregidgo also teaches the identifier being a label (“In this work, we generalise our CBIR system to include multiple labels in the vessel feature encoding which increases registration performance. In the specific case of liver imaging, this is possible by labelling different vessels as branches of the portal vein or branches of the hepatic vein.” [Pg. 1043 Col. 2])
It would have been obvious to an ordinary skilled person in the art before the effective filing
date of the claimed invention to modify the system of Ramalhinho as outlined above with the identifier being a label as taught by Tregidgo, because it enables the registration problem to be accurately initialised without tracking data [Pg. 1043 Col. 2].
Ramalhinho in view of Tregidgo does not specifically teach predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses.
However, in a similar field of endeavor, Sprung teaches A method of determining a three-dimensional motion of a movable ultrasound probe [Abstract].
Sprung also teaches predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses (“the method comprises determining, from the probe motion indicator (from the relative three-dimensional displacement and rotation between the ultrasound image frames), a probe position and orientation of the ultrasound probe. The probe position and orientation may be obtained by discrete integration of multiple probe motion indicators.” [0103], “the method comprises directly predicting the ultrasound probe motion from the stream of ultrasound images, without the input of any external tracking system, and optionally based on only the image data” [0108])
It would have been obvious to an ordinary skilled person in the art before the effective filing
date of the claimed invention to modify the system of Ramalhinho in view of Tregidgo as outlined above with predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses as taught by Sprung, because it can allow for estimating the in-plane motion quite reliably [0004].
Regarding Claim 12, Ramalhinho discloses the step of predicting comprises: generating a first trajectory of 3D coordinates of the prior 3D probe poses in the sequence (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065]);
generating a second trajectory of 3D orientations of the prior 3D probe poses in the sequence; generating the predicted 3D probe pose based on the first and the second trajectories (“The mean number of plausible paths 2 for each of the nine sweep registrations vs the number of images is shown in FIG. 8. Since the Viterbi algorithm is recursive on the number of columns in the hidden Markov model (shown in FIG. 6), results are displayed as a function of the number of images used so far in the optimisation (from 2 to 20). FIG. 8 therefore shows the number of kinematically possible paths for N images (i.e. with a non-zero probability, based on the constraints defined above). The number of plausible trajectories found by the algorithm converges to 1 if enough images are used (N=17 in this case).” [0085], multiple trajectories are predicted as being the potentially correct trajectory as more images are acquired the projected trajectory narrows to the closest matching trajectory),
wherein the virtual ultrasound image is created based on a 3D model for the target organ in accordance with the predicted 3D probe pose (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065]).
Regarding Claim 13, Ramalhinho discloses the step of extracting the at least one ASM from a virtual ultrasound image comprises: identifying each 2D structure in the virtual ultrasound image corresponding to a 3D anatomical structure in the 3D model; generating a mask for the 2D structure to create a corresponding ASM(“FIG. 1 is a sequence of steps 30 according to an embodiment of the invention, for determining a probe pose corresponding with an ultrasound image by registering the ultrasound image with volumetric scan data. At step 31, the volumetric scan data is processed to determine a plurality of simulated ultrasound images corresponding with different poses of the probe (e.g. at least one of position, orientation, depth/deformation). At step 32, a feature vector is extracted from each of the simulated ultrasound images, and from the ultrasound image. The feature vector may comprise a position and size of each vessel intersection with the respective image. The feature vector may be obtained by segmentation of the images into vessels and not-vessels. At step 33, the feature vector from each simulated ultrasound image is compared with the feature vector from the ultrasound image to determine a distance or similarity value.” [0054-0057]) ;
Ramalhinho does not specifically disclose assigning a label for the 3D anatomical structure retrieved from the 3D model to the ASM to generate the ASM/label pair.
However, in a similar field of endeavor, Tregidgo teaches assigning a label for the 3D anatomical structure retrieved from the 3D model to the ASM to generate the ASM/label pair ((“In this work, we generalise our CBIR system to include multiple labels in the vessel feature encoding which increases registration performance. In the specific case of liver imaging, this is possible by labelling different vessels as branches of the portal vein or branches of the hepatic vein.” [Pg. 1043 Col. 2])
It would have been obvious to an ordinary skilled person in the art before the effective filing
date of the claimed invention to modify the system of Ramalhinho as outlined above with assigning a label for the 3D anatomical structure retrieved from the 3D model to the ASM to generate the ASM/label pair as taught by Tregidgo, because it enables the registration problem to be accurately initialised without tracking data [Pg. 1043 Col. 2].
Regarding Claim 14, Ramalhinho discloses further comprising refining the estimated 3D probe pose by updating the ASM/label representation based on the estimated 3D probe pose to generate an updated ASM/label representation for the ultrasound image, wherein the updating comprises: creating a new virtual ultrasound image based on the estimated 3D probe pose in accordance with a 3D model for the target organ, the new virtual ultrasound image providing ground truth ASM/label pairs (“FIG. 2 illustrates a sequence of steps 40, according to an embodiment of the invention, for determining a sequence of probe poses corresponding with a sequence of ultrasound images obtained by sweeping the probe over tissue, such as an organ, by registering the sequence of ultrasound images with volumetric scan data. At step 41, the volumetric scan data is processed to determine a plurality of simulated ultrasound images corresponding with different poses of the probe (e.g. at least one of position, orientation, depth/deformation). At step 42, a feature vector is extracted from each of the simulated ultrasound images, and from each of the sequence of ultrasound images. The feature vector may comprise a position and size of each vessel intersection with the respective image. The feature vector may be obtained by segmentation of the images into vessels and not-vessels.” [0060-0062]);
extracting one or more virtual ASM/label pairs from the new virtual ultrasound image; for each ASM/label pair generated based on the ultrasound image, Identifying a corresponding virtual ASM/label pair, revising the ASM/label pair from the ultrasound image if it satisfies at least one predetermined criterion with respect to the virtual ASM/label pair; and generating an update ASM/label representation based on the revised ASM/label pair (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065], “In the example described herein, it is implicit that the organ does not deform. In some embodiments, the set of simulated ultrasound images obtained from the volumetric scan may be parameterised to include deformation (e.g. in the y direction). In some embodiments the depth d parameter may represent deformation of the organ in a direction normal to the surface of the organ (rather than a simple translation without deformation). Higher accuracies may be achievable with parameterisation including deformation.” [0096]).
Regarding Claim 15, Ramalhinho discloses the step of revising according to at least one predetermined criterion comprises: if an overlap between the ASM in the ASM/label pair and the ASM in the virtual ASM/label pair is not acceptable, removing the ASM/label pair; if the overlap between the ASM in the ASM/label pair and the ASM in the virtual ASM/label pair is acceptable, replacing the label from the ASM/label pair with the label from the virtual ASM/label pair pair (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065], “The mean number of plausible paths 2 for each of the nine sweep registrations vs the number of images is shown in FIG. 8.” [0085], “In the example described herein, it is implicit that the organ does not deform. In some embodiments, the set of simulated ultrasound images obtained from the volumetric scan may be parameterised to include deformation (e.g. in the y direction). In some embodiments the depth d parameter may represent deformation of the organ in a direction normal to the surface of the organ (rather than a simple translation without deformation). Higher accuracies may be achievable with parameterisation including deformation.” [0096], as shown in Fig.8 the more ultrasound images analyzed the less possibilities for the potential number of paths (3D probe poses)).
PNG
media_image1.png
345
417
media_image1.png
Greyscale
Regarding Claim 16, Ramalhinho discloses further comprising refining the estimated 3D probe pose to generate an updated 3D probe pose based on the updated ASM/label representation, wherein the step of refining the estimated 3D probe pose comprises: obtaining a new 3D probe pose based on the updated ASM/label representation via the ASM-pose mapping model; if the new 3D probe pose and the estimated 3D probe pose satisfy a configured condition, outputting the new 3D probe pose as the updated 3D probe pose; and if the new 3D probe pose and the estimated 3D probe pose do not satisfy the configured condition, outputting the estimated 3D probe pose as the updated 3D probe pose (“At step 43, the feature vector from each simulated ultrasound image is compared with the feature vector from each of the sequence of ultrasound images to determine a distance or similarity value. At step 44, candidate simulated images are selected that best match each of the sequence of ultrasound images, based on the distance or similarity. At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0063-0065]).
Regarding Claim 17, Ramalhinho discloses that the configured condition defines that a deviation between the new 3D probe pose and the 3D probe pose is at an acceptable range (“At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0065], “imposing a transition probability penalty when a probe path direction deviates from an initial direction by more than a threshold amount.” [0024]).
Regarding Claim 18, Ramalhinho discloses further comprising adding the updated 3D probe pose to the sequence of previous 3D probe poses of the ultrasound probe (“At step 45, a probe path is identified by determining from the candidate images which is most likely to match each ultrasound image in the sequence of ultrasound images using a transition probability between two candidate images. The transition probability may be based on kinematic assumptions about the movement of the probe over time. A hidden Markov model may be used to determine the simulated images that are most likely to correspond with the sequence of ultrasound images.” [0065], as shown in Fig.8 the more ultrasound images analyzed the less possibilities for the potential number of paths (3D probe poses) and the sequence is optimized for the highest probability sequence).
PNG
media_image1.png
345
417
media_image1.png
Greyscale
Regarding Claim 19, Ramalhinho discloses the ASM-pose mapping model is created prior to the medical procedure by: retrieving a 3D model of the target organ; generating a plurality of virtual 3D probe poses in connection with an ultrasound probe; creating a virtual ultrasound image with respect to each of the plurality of virtual 3D probe poses; obtaining an ASM-pose pairing for each of the plurality of virtual 3D probe poses, where the ASM-pose pairing includes a virtual 3D probe pose and an ASM/label representation for the virtual ultrasound image created with respect to the virtual 3D probe pose; and establishing the ASM-pose mapping model for mapping an ASM/label representation to a 3D probe pose based on the ASM-pose pairings (“FIG. 1 is a sequence of steps 30 according to an embodiment of the invention, for determining a probe pose corresponding with an ultrasound image by registering the ultrasound image with volumetric scan data. At step 31, the volumetric scan data is processed to determine a plurality of simulated ultrasound images corresponding with different poses of the probe (e.g. at least one of position, orientation, depth/deformation). At step 32, a feature vector is extracted from each of the simulated ultrasound images, and from the ultrasound image. The feature vector may comprise a position and size of each vessel intersection with the respective image. The feature vector may be obtained by segmentation of the images into vessels and not-vessels. At step 33, the feature vector from each simulated ultrasound image is compared with the feature vector from the ultrasound image to determine a distance or similarity value. At step 34, a candidate image is selected as the best match, based on the distance or similarity. At step 35, the pose of the probe is identified from the candidate image” [0054-0059]).
Regarding Claim 20, Ramalhinho discloses the step of establishing comprises: generating training data based on the ASM-pose pairings; obtaining, via machine learning based on the training data, the ASM-pose mapping model to learn relationships between ASM/label representations and 3D probe poses. (“The feature vector may be extracted using a convolutional neural network. The convolutional neural network may have been trained to distinguish between ultrasound images.” [0029-0030]).
Response to Arguments
Applicant's arguments filed 04/15/2026 have been fully considered but they are not persuasive.
Regarding the U.S.C. 103 rejection of Claim 1 the applicant argues the following:
Claim 1, for example, as amended recites, "predicting the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses, resulting in a predicted 3D probe pose." By contrast, Ramalhinho uses a hidden Markov model with transition probabilities to determine which pre-computed simulated images best match acquired ultrasound images. See Ramalhinho [0065]. Using a hidden Markov model to determine which pre-computed simulated images best match is not the same as predicting "the 3D probe pose based on extrapolating from the sequence of previous 3D probe poses." Yan does not cure this deficiency. Accordingly, claim 1 is patentable over the cited art.
However, it is noted that under broadest reasonable interpretation of Claim 1, all that is required for the limitation in question is that the predicted 3D pose is based on previous 3D poses. The term “extrapolate” without further specificity is viewed as being a generic placeholder word alongside based on.
In view of the prior art, Ramalhinho discloses that the predicted 3D pose is based on matching of images from previously acquired 3d poses which constrain the potential matches ((“Once a set of k possible matches {J1i, . . . , JKi} are obtained for image Ii a transition probability may be used to determine a set of simulated images from J that match the set of acquired images {I1, . . . , IN} acquired by sweeping the probe over the surface of the organ. Under these conditions, each successive acquired image will correspond with a successive pose along the path swept by the probe as it moves over the surface of the organ. This imposes a kinematic constraint on the set of images selected from J to match the acquired images {I1, . . . , IN} because solutions that require very high acceleration and/or velocity are very unlikely to be correct.” [Ramalhinho 0077], “This can be formulated as a hidden Markov model, as shown in FIG. 6. Nodes in FIG. 6 represent a probability of each acquired image Ii matching a candidate simulated image Jki, and edges represent a probability associated with an input kinematic prior. The kinematic prior may be selected to constrain the selected images to match a condition of the kinematics of the acquisition. For example, it may be assumed that there is smoothness in the acquisition, and/or it may be assumed that the probe follows a continuous path along a direction normal to the imaging plane without moving backwards.” [Ramalhinho 0078], “During optimisation, a constraint may be implemented to reject candidate simulated image sets that do not fulfil specific kinematic conditions. For example, a sweep direction may be defined as the difference between the first two probe positions Pk2−Pk1 in the candidate simulated image set. The probability P(Ii|Jki) may be set to 0 (or reduced by a predetermined amount or ratio) if the angle between Pki+i−Pki and the sweep direction is above 90 degrees (or some other predetermined angle or variance).” [Ramalhinho 0082]);
however for compactness of the prosecution the applicants alleged requirement of the extrapolation requiring the exact previous poses is also taught by Sprung (“the method comprises determining, from the probe motion indicator (from the relative three-dimensional displacement and rotation between the ultrasound image frames), a probe position and orientation of the ultrasound probe. The probe position and orientation may be obtained by discrete integration of multiple probe motion indicators.” [Sprung 0103], “the method comprises directly predicting the ultrasound probe motion from the stream of ultrasound images, without the input of any external tracking system, and optionally based on only the image data” [Sprung 0108]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's
disclosure (US 20230240790 A1).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN MALDONADO whose telephone number is 703-756-1421. The examiner can normally be reached 8:00 am-4:00 pm PST M-Th Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christopher Koharski can be reached on (571) 272-7230. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Steven Maldonado/
Patent Examiner, Art Unit 3797
/ANNE M KOZAK/
Supervisory Patent Examiner, Art Unit 3797
Prosecution Insights