DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 10 is objected to because of the following informalities: “The method of 6…” should read “The method of claim 6…”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2 and 5 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites the limitation “the plurality of ultrasound transducers”. There is insufficient antecedent basis for this limitation in the claim.
Claim 5 recites the phrase "…using an optimizer like Covariance Matrix Adaptation Evolution Strategy", which renders the claim indefinite because it is unclear what would or would not be considered to be “like” Covariance Matrix Adaptation Evolution Strategy.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 and 14-15 are rejected under 35 U.S.C. 101 because they are directed to an abstract idea without significantly more.
Claim 1
Step 1: Claim 1 falls into the statutory category of method claims.
Step 2A-I: The claim recites the abstract idea of a mathematical algorithm comprising the following steps:
Determining the registration of a 3D image dataset of patient as a registration that maximizes a similarity between the measured set of measured ultrasound A mode signals and a simulated set of simulated ultrasound A mode signals generated from the 3D image dataset of the patient.
Step 2A-II: The claim language does not integrate the recited mathematical algorithm into a practical application because the mere performance of the algorithm does not itself improve the ability of healthcare professionals to examine patients.
Step 2B: The claim recites the following additional element:
Acquiring the ultrasound data comprising a measured set of measured ultrasound A mode signals.
However, the above additional element is merely a data gathering step required for the performance of the recited mathematical algorithm and thus does not amount to significantly more.
Furthermore, claims 3 and 5-11 are also rejected by virtue of their dependence from claim 1 and because they do not set forth any further additional elements that integrate the recited mathematical algorithm into a practical application or amount to significantly more.
Claim 2
Step 1: Claim 2 falls into the statutory category of method claims.
Step 2A-I: The claim recites an abstract idea as it inherits the limitations of claim 1. The claim also recites a further mathematical algorithm comprising the following step:
Determining the simulated set of simulated ultrasound A mode signals for a virtual position of an ultrasound transducer used for measuring the measured ultrasound A mode signals relative to the 3D image data
Step 2A-II: The claim language does not integrate the recited mathematical algorithm into a practical application because the mere performance of the algorithm does not itself improve the ability of healthcare professionals to examine patients.
Step 2B: The claim recites the following additional elements:
Wherein the ultrasound transducer…
…comprises a plurality of ultrasound detectors,
Wherein each ultrasound detected of the plurality of ultrasound transducers is operable to measure one of the measured ultrasound A mode signals in the measured set.
However, additional elements (a) and (b) do not go beyond that which is well-understood, routine, and conventional in the art (see US 20210015434 A1, Claim 12 — “The method of claim 1, wherein the one or more rangefinders comprises an ultrasound transducer and detector.”; see US 20180160937 A1, Claim 6 — “The system of claim 5, wherein the operation of the computer system generating the position information comprises: the computer system performs following equations (1) and (2) to calculate a position of the at least one ultrasound transducer relative to the ultrasound detector…”). Additional element (c) constitutes mere data gathering required for the performance of the recited mathematical algorithm.
Claim 4
Step 1: Claim 4 falls into the statutory category of method claims.
Step 2A-I: The claim recites an abstract idea because it inherits the limitations of claim 1. Additionally, the claim recites a further mathematical algorithm comprising the following steps:
Generating an ultrasound model including acoustic tissue properties from the 3D image dataset; and
Generating the simulated set of simulated ultrasound A mode signals by simulating a propagation of an ultrasound signal through the ultrasound model.
Step 2A-II: The claim language does not integrate the recited mathematical algorithm into a practical application because the mere performance of the algorithm does not itself improve the ability of healthcare professionals to examine patients.
Step 2B: The claim does not recite any additional elements.
Claim 12
Step 1: Claim 12 falls into the statutory category of method claims.
Step 2A-I: The claim recites and abstract idea as it inherits the limitations of claim 1. Additionally, the claim recites a mental process comprising the following steps:
Grouping simulated ultrasound A mode signals from the simulated set of simulated ultrasound A mode signals into at least one simulated group;
Grouping measured ultrasound A mode signals from the measured set of measured ultrasound A mode signals into at least one measured group corresponding to the at least one simulated group.
The claim also recites a further mathematical algorithm comprising the following steps:
Calculating a simulated group signal for each of the at least one simulated group;
Calculating a measured group signal for each of the measured groups.
A combination of abstract ideas is an abstract idea (See MPEP 2106.05(I) – "Adding one abstract idea (math) to another abstract idea (encoding and decoding) does not render the claim non-abstract").
Step 2A-II: The claim language does not integrate the recited abstract ideas into a practical application because the mere performance thereof does not itself improve the ability of healthcare professionals to examine patients.
Step 2B: The claim does not recite and additional elements.
Claim 14
Step 1: Claim 14 falls into the statutory of apparatus claims.
Step 2A-I: The claim recites a mathematical algorithm comprising the following steps:
Determining the registration of the 3D image dataset of the patient as a registration that maximizes a similarity between the measured set of the measured ultrasound A mode signals and a simulated set of simulated ultrasound A mode signals generated from the 3D image dataset of the patient.
Step 2A-II: The claim language does not integrate the recited mathematical algorithm into a practical application because the mere performance of the algorithm does not itself improve the ability of healthcare professionals to examine patients.
Step 2B: The claim recites the following additional elements:
A computer comprising:
A processor;
A non-transitory computer readable storage medium; and
A program stored on the non-transitory computer readable storage medium
Acquiring the ultrasound data comprising a measured set of measured ultrasound A mode signals.
However, additional elements (a)-(d) are all generic computer components configured to perform generic computer functions for the performance of the recited mathematical algorithm and thus does not amount to significantly more. Additional element (e) is mere data gathering required for the performance of the recited abstract idea.
Claim 15
Step 1: Claim 15 falls into the statutory category of apparatus claims.
Step 2A-I: The claim recites a mathematical algorithm comprising the following step:
Determining the registration of the 3D image dataset of the patient as a registration that maximizes a similarity between the measured set of the measured ultrasound A mode signals and a simulated set of simulated ultrasound A mode signals generated from the 3D image dataset of the patient.
Step 2A-II: The claim language does not integrate the recited mathematical algorithm into a practical application because the mere performance of the algorithm does not itself improve the ability of healthcare professionals to examine patients.
Step 2B: The claim recites the following additional elements:
A non-transitory computer readable storage medium storing a program
Acquiring the ultrasound data comprising a measured set of measured ultrasound A mode signals
However, additional element (a) is a generic computer component configured to perform generic computer functions for the performance of the recited mathematical algorithm and thus does not amount to significantly more. Additional element (b) is mere data gathering required for the performance of the recited mathematical algorithm.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 3-4, 6, and 14-15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lee et al. (US 20160125605 A1, hereinafter Lee).
Claim 1, 14, and 15
Lee discloses a data processing method, a corresponding computer comprising a processor ([0171] — “In this case, the image processing unit 500 may correspond to one or a plurality of processors.”) and a corresponding non-transitory computer-readable storage medium storing a program which would cause the computer to perform the method ([0173] — “The storage unit 700 may include, for example, a high-speed random access memory…”; programs are stored in memory) of determining a registration (Fig. 8) of a 3D image dataset of a patient taken at a first point in time (Fig. 8, #S100 — Tomography data is 3D image data) with ultrasound data of the patient taken at a second point in time later than the first point in time (Fig. 8, #S170; Fig. 8, #S100 — That the tomography image has already been computerized means it was taken before the ultrasound), the method comprising: acquiring the ultrasound data comprising a measured set ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; multiple measurements made successively in real time constitutes a set) of measured ultrasound A mode signals ([0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”); and determining the registration of the 3D image dataset of patient as a registration that maximizes a similarity ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”) between the measured set of measured ultrasound A mode signals ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”) and a simulated set of simulated ultrasound A mode signals generated from the 3D image dataset of the patient ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”).
Claim 3
Lee discloses wherein the determining the registration comprises: calculating a plurality of simulated sets of ultrasound A mode signals for different relative positions between the 3D image dataset and the ultrasound data ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”; different simulated sets of ultrasound A mode signals between the 3D image dataset an the ultrasound dataset (different virtual 3D ultrasound images) would have different relative positions); and using the simulated set out of the plurality of simulated sets for which the similarity is the largest ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”).
Claim 4
Lee discloses generating an ultrasound model including acoustic tissue properties from the 3D image dataset ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; ultrasound images of a body inherently include tissue data); and generating the simulated set of simulated ultrasound A mode signals by simulating a propagation of an ultrasound signal through the ultrasound model ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”; a virtual 3D ultrasound image inherently includes simulated ultrasound signals).
Claim 6
Lee discloses wherein the determining the registration further comprises optimizing the similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals in a multi-dimensional search space ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; maximizing a correlation of an ultrasound image to a virtual 3D (multi-dimensional) ultrasound image must include a multi-dimensional search space during the maximization).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 2 is rejected under 35 USC 103 as being unpatentable over Lee in view of Wein et al. (Simulation and Fully Automatic Multimodal Registration of Medical Ultrasound, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2007. MICCAI 2007. Lecture Notes in Computer Science, vol 4791. Springer, Berlin, Heidelberg; hereinafter Wein).
Claim 2
This claim is rejected under 35 USC 112(b). For the sake of compact prosecution, the limitation “wherein each ultrasound detector of the plurality of ultrasound transducers” will be assumed to mean “wherein each ultrasound detector of the plurality of ultrasound detectors”.
Lee fails to disclose determining the simulated set of simulated ultrasound A mode signals for a virtual position of an ultrasound transducer used for measuring the measured ultrasound A mode signals relative to the 3D image data, wherein the ultrasound transducer comprises a plurality of ultrasound detectors, wherein each ultrasound detector of the plurality of ultrasound transducers is operable to measure one of the measured ultrasound A mode signals in the measured set.
Wein discloses determining the simulated set of simulated ultrasound A mode signals for a virtual position of an ultrasound transducer used for measuring the measured ultrasound A mode signals relative to the 3D image data (Page 3 — “For a linear array probe, the integral equation 6 can be computed efficiently by traversing the columns in the simulated ultrasound image from top to bottom while updating the transmitted intensity based on the interpolated CT intensity and gradient values”; integral equation 6 determines the intensity I(x) of a simulated ultrasound wave for a virtual position x of the signal source, the ultrasound transducer (the linear array probe)), wherein the ultrasound transducer comprises a plurality of ultrasound detectors (Page 3 — “For a linear array probe, the integral equation 6 can be computed efficiently by traversing the columns in the simulated ultrasound image from top to bottom while updating the transmitted intensity based on the interpolated CT intensity and gradient values”; a linear array probe has multiple detector elements), wherein each ultrasound detector of the plurality of ultrasound transducers is operable to measure one of the measured ultrasound A mode signals in the measured set (Page 3 — “For a linear array probe, the integral equation 6 can be computed efficiently by traversing the columns in the simulated ultrasound image from top to bottom while updating the transmitted intensity based on the interpolated CT intensity and gradient values”; each detector in an ultrasound linear array probe is operable to measure signals). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to determine the simulated set of simulated ultrasound A mode signals for a virtual position of an ultrasound transducer as disclosed by Wein in combination with the method disclosed by Lee in order to more accurately simulate the operating conditions of an ultrasound machine.
Claim 5 is rejected under 35 USC 103 as being unpatentable over Lee in view of Steinberg et al. (US 20200405399 A1, hereinafter Steinberg).
Claim 5
Lee discloses wherein the determining the registration further comprises maximizing the similarity ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”) between the measured set of measured ultrasound A mode signals ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”) and the simulated set of simulated ultrasound A mode signals ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”).
Lee fails to disclose using an optimizer like Covariance Matrix Adaptation Evolution Strategy.
Steinberg discloses using an optimizer like Covariance Matrix Adaptation Evolution Strategy for 3D registration ([0431] — “A fine-tuned 3D-2D co-registration follows. That is performed using known techniques such as CMA-ES (covariance matrix adaptation evolution strategy).”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to use an optimizer like Covariance Matrix Adaptation Evolution Strategy as disclosed by Steinberg with the method disclosed by Lee for the advantage of increasing registration robustness.
Claims 7-8 are rejected under 35 USC 103 as being unpatentable over Lee in view of Xing et al. (M3VR—A multi-stage, multi-resolution, and multi-volumes-of-interest volume registration method applied to 3D endovaginal ultrasound, PLOS One. 2019 Nov 21. Accessed via PubMed, hereinafter Xing) and Steinberg.
Claim 7
Lee discloses determining the registration based on a calculated registration for which a similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals is maximized ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”; a correlation ratio is a calculation which makes the corresponding registration a calculated registration).
Lee fails to disclose wherein the determining the registration further comprises forming a thinned-out measured subset of measured ultrasound A mode signals from the measured set of measured ultrasound A mode signals; forming a thinned-out simulated subset of simulated ultrasound A mode signals from the simulated ultrasound A mode signals, the thinned-out simulates subset of simulated ultrasound A mode signals corresponding to the thinned-out measured subset of measured ultrasound A mode signals; calculating an intermediate registration that maximizes a similarity between the thinned-out measured subset of measured ultrasound A mode signals and the thinned-out simulated subset of simulated ultrasound A mode signals; and setting a limited search space around the calculated intermediate registration; and determining the registration based on a calculated registration for which a similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals is maximized within the limited search space.
Xing discloses forming thinned-out subsets of data for 3D registration (Abstract – “Multiple sub-volumes of interest can then be selected as target alignment regions to achieve confident consistency across the volume. Finally, a multi-resolution rigid registration is performed on these sub-volumes associated with different weights in the cost function.”) and calculating an intermediate registration (Fig. 3 — The algorithm performs a first registration in Stage 1 and a second registration in Stage 2) that maximizes a similarity between the subsets (Methods, Stage 2 — “The weighted multi-regional registration is calculated by optimizing the cost function in Eq (5), which minimizes the weighted sum similarity ϕ between multiple VOIs (𝑉FPB,𝑉FPM1 and 𝑉FPM2) in the fixed volume VF and the registered moving volume Tθ(VM)”; maximizing and minimizing are mathematically equivalent processes and whether a similarity metric is optimized via maximization or optimized via minimization is dependent only on the choice of metric, i.e. the minimization process taught by Xing serves the same purpose as the maximization process taught by Lee in claim 1 for their respective choices of similarity metric). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to form thinned-out subsets as disclosed by Xing for the set of measured ultrasound A mode signals disclosed by Lee and the set of simulated ultrasound A mode signals disclosed by Lee, the thinned-out simulated subset of simulated ultrasound A mode signals corresponding to the thinned-out measured subset of measured ultrasound A mode signals (for registration to be performed the subsets must correspond with one another), in order to calculate an intermediate registration that maximizes a similarity as disclosed by Xing between the thinned-out measured subset of measured ultrasound A mode signals and the thinned-out simulated subset of simulated ultrasound A mode signals disclosed by Lee and Xing for the advantage of improving processing time while preserving model robustness.
Lee and Xing still fail to disclose setting a limited search space around the calculated intermediate registration and determining the registration based on a calculated registration for which a similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals is maximized within the limited search space.
Steinberg discloses setting a limited search space around a 2D/3D registration of medical image data ([0082]-[0085] — “Registering the intraoperative 2D radiographic image of the targeted skeletal portion to the 3D image data of the targeted skeletal portion, the registering including: using the computer processor: using the obtained deep-learning data to limit a search space in which a 2D projection from the 3D image...”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to limit the search space as disclosed by Steinberg around the calculated intermediate registration disclosed by Lee and Xing and to determine the registration based on a calculated registration for which a similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals is maximized as disclosed by Lee within the limited search space disclosed by Steinberg in order to improve processing time.
Claim 8
Lee discloses wherein the determining the registration comprises: tracking a position of an ultrasound transducer used for measuring the measured set of measured ultrasound A mode signals relative to the patient ([0154] — “Accordingly, rough position registration may be performed using only a rotation and translation method without deformation inside the object through rigid registration.”; that position registration may be performed means that the transducer position was tracked) and determining the registration as a registration for which a similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals is maximized ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”).
Lee fails to disclose setting a limited search space based on the tracked position of the ultrasound transducer and determining the registration as a registration for which a similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals is maximized within the limited search space.
Steinberg discloses setting a limited search space around a 2D/3D registration of medical image data ([0082]-[0085] — “Registering the intraoperative 2D radiographic image of the targeted skeletal portion to the 3D image data of the targeted skeletal portion, the registering including: using the computer processor: using the obtained deep-learning data to limit a search space in which a 2D projection from the 3D image...”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to set a limited search space as disclosed by Steinberg based on the tracked position of the ultrasound transducer taught by Lee and to determine the registration as a registration for which a similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals is maximized as disclosed by Lee within the limited search space disclosed by Steinberg in order to improve registration accuracy.
Claim 9 is rejected under 35 USC 103 as being unpatentable over Lee in view of Steinberg, Xing, and Han (US 20140247977 A1).
Claim 9
Lee fails to disclose registering an atlas of the patient with the 3D image dataset; and determining a position of the 3D image dataset relative to the patient based on the registered atlas, wherein the setting the limited search space comprises setting the limited search space based on the determined position of the 3D image dataset relative to the patient.
Han discloses registering an atlas of the patient with the 3D image dataset (Claim 4 — “…wherein the performing step comprises: the processor registering the subject image with a plurality of the atlas images to map the points of the subject image to the points of the atlas images, thereby associating the registered subject image points with the labels that are associated with the atlas image points that were mapped to the registered subject image points; and the processor generating the first data by classifying a plurality of the subject image points as to whether those subject image points belong to the structure based on the labels associated with those subject image points…”; subject images are 3D, see [0022] — “It should be understood that the images processed using the techniques described herein can be take any of a number of forms. In various exemplary embodiments, the images can be medical images such as CT images. However, it should be understood that images of different types can be employed. For example, image types such as magnetic resonance (MR) images and ultrasound images could also be processed using the techniques described herein.”) and determining a position of the 3D image dataset relative to the patient based on the registered atlas (Claim 4 — “…wherein the performing step comprises: the processor registering the subject image with a plurality of the atlas images to map the points of the subject image to the points of the atlas images, thereby associating the registered subject image points with the labels that are associated with the atlas image points that were mapped to the registered subject image points…”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to register an atlas of the patient with the 3D image dataset and determine a position of the 3D image dataset relative to the patient based on the registered atlas as disclosed by Han in the method disclosed by Lee, Xing, and Steinberg in order to better register 3D images with the shape of the patient’s body.
Lee and Han still fail to disclose wherein the setting the limited search space comprises setting the limited search space based on the determined position of the 3D image dataset relative to the patient.
Steinberg discloses setting a limited search space around a 2D/3D registration of medical image data ([0082]-[0085] — “Registering the intraoperative 2D radiographic image of the targeted skeletal portion to the 3D image data of the targeted skeletal portion, the registering including: using the computer processor: using the obtained deep-learning data to limit a search space in which a 2D projection from the 3D image...”) in the method of claim 8. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to set the limited search space, disclosed by Steinberg in the method of claim 8 disclosed by Lee, Xing, and Steinberg, based on the determined position of the 3D image dataset relative to the patient disclosed by Han in order to improve registration accuracy to the shape of the patient’s body.
Claim 10 is rejected under 35 USC 103 as being unpatentable over Lee in view of Vilsmeier et al. (US 20130315463 A1, hereinafter Vilsmeier) and Steinberg.
Claim 10
Lee fails to disclose wherein the determining the registration comprises: registering an atlas of the patient with the 3D image dataset of the patient; determining a surface of the patient based on the registered atlas; and setting a limited search space based on the determined surface of the patient.
Vilsmeier discloses registering an atlas of the patient with the 3D image dataset of the patient ([0037] — “Performing a first local, elastic registration of the MRI image data set and the less distorted or undistorted image data set in an area containing definitively rigid anatomical structures as determined by means of the anatomical atlas and the immediate surroundings of said rigid structures.”; MRI image data may be in the 3D image dataset, see Lee, Fig. 8, #S110 — “Select Magnetic Resonance Image data on similar object”) and determining a surface of the patient based on the registered atlas ([0038] — “Performing a local, elastic registration of the MRI image data set and the less distorted or undistorted image data set on the basis of cutting edges of structures cut by a surface, in particular a curved surface, within the body region, wherein said surface is determined by means of the anatomical atlas.”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to register an atlas of the patient with the 3D image dataset of the patient and to determine a surface of the patient based on the registered atlas as disclosed by Vilsmeier with the method disclosed by Lee in order to improve distortion correction in the registration process.
Lee and Vilsmeier still fail to disclose setting a limited search space based on the determined surface of the patient.
Steinberg discloses setting a limited search space around a 2D/3D registration of medical image data ([0082]-[0085] — “Registering the intraoperative 2D radiographic image of the targeted skeletal portion to the 3D image data of the targeted skeletal portion, the registering including: using the computer processor: using the obtained deep-learning data to limit a search space in which a 2D projection from the 3D image...”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to set a limited search space as disclosed by Steinberg based on the determined surface of the patient disclosed by Lee and Vilsmeier to improve processing time.
Claim 11 is rejected under 35 USC 103 as being unpatentable over Lee in view of Ros et al. ("The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 3234-3243; hereinafter Ros) and Han.
Claim 11
Lee discloses the similarity between the measured set of measured ultrasound A mode signals and the simulated set of simulated ultrasound A mode signals being calculated from measured ultrasound A mode signals in the measured set corresponding to the simulated ultrasound A mode signals in the simulated set ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”).
Lee fails to disclose wherein the simulated set of simulated ultrasound A mode signals is smaller than the measured set of measured ultrasound A mode signals and the simulated ultrasound A mode signals to be calculated are set based on marks in the 3D image dataset or labels in an atlas matched to the 3D image dataset of the patient.
Ros discloses using a smaller set of simulated images than a set of measured images in the field of computer vision (Section 4.2, Training on Real and Synthetic Data — “However, here we employ the Balanced Gradient Contribution (BGC) that was first introduced in [30]. It consists of building batches with images from both domains (synthetic and real), given a fixed ratio. Real images dominate the distribution, while synthetic images are used as a sophisticated regularization term. Thus, statistics of both domains are considered during the whole procedure, creating a model which is accurate for both. In section 5 we show that extending real data with synthetic images using this technique leads to a systematic boost in segmentation accuracy.”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to use a simulated set of simulated ultrasound A mode signals disclosed by Lee that is smaller than the measured (real) set as disclosed by Ros of measured ultrasound A mode signals in order to improve model accuracy and generality.
Lee and Ros still fails to disclose wherein the simulated ultrasound A mode signals to be calculated are set based on marks in the 3D image dataset or labels in an atlas matched to the 3D image dataset of the patient.
Han discloses wherein the simulated ultrasound A mode signals to be calculated are set based on marks in the 3D image dataset or labels in an atlas matched to the 3D image dataset of the patient (Claim 4 — “…the processor registering the subject image with a plurality of the atlas images to map the points of the subject image to the points of the atlas images, thereby associating the registered subject image points with the labels that are associated with the atlas image points that were mapped to the registered subject image points…”; the subject images are understood to be the 3D image dataset, see [0022] — “It should be understood that the images processed using the techniques described herein can be take any of a number of forms. In various exemplary embodiments, the images can be medical images such as CT images. However, it should be understood that images of different types can be employed. For example, image types such as magnetic resonance (MR) images and ultrasound images could also be processed using the techniques described herein.”; simulated ultrasound signals to be calculated are based on the labeled 3D image dataset (disclosed by Lee in claim 1) which may be matched to labels in an atlas disclosed by Han). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have the simulated ultrasound A mode signals to be calculated disclosed by Lee to be set based on marks in the 3D image dataset or labels in an atlas matched to the 3D image dataset of the patient as disclosed by Han in order to improve registration accuracy.
Claim 12 is rejected under 35 USC 103 as being unpatentable over Lee in view of Xing and Park et al. (US 20230371921 A1, hereinafter Park).
Claim 12
Lee discloses wherein the similarity in the determining the registration comprises a similarity between simulated ultrasound A mode signals and measured ultrasound A mode signals ([0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; [0080] — “Exemplary modes of the ultrasound image may include an amplitude mode (A-mode), a brightness mode (B-mode), a Doppler mode (D-mode), an elastography mode (E-mode), and a motion mode (M-mode).”).
Lee fails to disclose wherein the determining the registration comprises grouping simulated ultrasound A mode signals from the simulated set of simulated ultrasound A mode signals into at least one simulated group; calculating a simulated group signal for each of the at least one simulated group; grouping measured ultrasound A mode signals from the measured set of measured ultrasound A mode signals into at least one measured group corresponding to the at least one simulated group; and calculating a measured group signal for each of the measured groups, and wherein the similarity in the determining the registration comprises a similarity between the simulated group signals and the measured group signals.
Xing discloses forming groups of data for 3D registration (Abstract – “Multiple sub-volumes of interest can then be selected as target alignment regions to achieve confident consistency across the volume. Finally, a multi-resolution rigid registration is performed on these sub-volumes associated with different weights in the cost function.”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to group as disclosed by Xing simulated ultrasound A mode signals from the simulated set of simulated ultrasound A mode signals disclosed by Lee into at least one simulated group and to group as disclosed by Xing measured ultrasound A mode signals from the measured set of measured ultrasound A mode signals disclosed by Lee into at least one measured group corresponding to the at least one simulated group (for registration to be performed the subsets must correspond with one another) in order to improve processing time of the registration process.
Lee and Xing together disclose calculating a simulated group signal for each of the at least one simulated group (Lee, [0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; a signal in a group constitutes a group signal; Lee calculates each of the simulated ultrasound A mode signals (virtual 3D ultrasound images) as per claim 1; hence, Lee calculates a simulated group signal for each of the at least one simulated group); calculating a measured group signal for each of the measured groups (Lee, [0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”; a signal in a group constitutes a group signal; Lee calculates each of the measured ultrasound A mode signals (ultrasound image obtained in real time) since ultrasound images are calculated from unprocessed signals; hence, Lee calculates a measured group signal for each of the at least one measured group), and wherein the similarity in the determining the registration comprises a similarity between the simulated group signals and the measured group signals (Lee, [0051] — “As another method, a method in which a simulation is performed on a computerized tomography image to fuse a virtual 3D ultrasound image, and then non-rigid registration of maximizing a correlation ratio between an ultrasound image obtained in real time and a virtual 3D ultrasound image is derived, has been used.”).
Prior Art
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure:
Roundhill et al. (US 20230054610 A1), Contextual Multiplanar Reconstruction of Three-Dimensional Ultrasound Imaging Data and Associated Devices, Systems, and Methods
Veronesi et al. (US 20220317294 A1), System and Method for Anatomically Aligned Multi-Planar Reconstruction Views for Ultrasound Imaging
Stopp et al. (US 20200193622 A1), Optical Tracking
Neben et al. (US 20200129151 A1), Methods and Apparatuses for Ultrasound Imaging Using Different Image Formats
Seip et al. (US 20160317129 A1), System and Method for Ultrasound and Computed Tomography Image Registration for Sonothrombolysis Treatment
Zhao et al. (US 20150201910 A1), 2D-3D Registration Method to Compensate for Organ Motion During an Interventional Procedure
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN JAMES STEAR whose telephone number is (571)272-8334. The examiner can normally be reached 7:30-5:30 EST/EDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Arleen Vazquez can be reached at (571) 272-2619. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN JAMES STEAR/Examiner, Art Unit 2857
/ARLEEN M VAZQUEZ/Supervisory Patent Examiner, Art Unit 2857
Prosecution Insights