Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This non-final office action on merits is in response to the Patent Application filed on 05/20/2025.
Status of claims
Amendments to claims 1, 3-4, 6-8, and 11-17 are acknowledged and have been carefully considered. Claims 1-17 are pending and considered below. This application is a 371 of PCT/EP2023/080903 filed on 11/07/2023, which claims the benefit of EP Application Number EP22290062.3 filed on 11/21/2022.
Information Disclosure Statement
The information disclosure statement (IDS) filed on 05/20/2025 has been acknowledged. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
Under step 1, the analysis is based on MPEP 2106.03, and claims 1-15 are drawn to a computer-implemented method, and claim 16 is drawn to a system. Thus, each claim, on its face, is directed to one of the statutory categories (i.e., useful process, machine, manufacture, or composition of matter) of 35 U.S.C. §101.
Also, the claimed invention is directed to nonstatutory subject matter. Claim 17 does not fall within at least one of the four categories of patent eligible subject matter because the claim is directed to a computer code per se.
Step 2A Prong One
Claim 1 recites the limitations of comparing the first acquired image to a plurality of reference images by determining a first plurality of confidence metrics using a first calculation method; and wherein each respective confidence metric of the first plurality of confidence metrics indicates how closely the first acquired image matches a respective reference image of the plurality of reference images. These limitations, as drafted, are processes that, under their broadest reasonable interpretations, cover performance of the limitations in the mind or by using a pen and paper. The claim encompasses a user visually comparing an acquired medical image to reference images, evaluating the degree of similarity between the images, and assigning or determining relative confidence values indicating how closely the images match in their mind or by using a pen and paper. Thus, the claim recites a mental process which is an abstract idea.
Under Step 2A Prong Two
The claimed limitations, as per method claim 1, include the steps of:
receiving a first acquired image of a patient, wherein the first acquired image is a medical image acquired using a medical imaging device;
comparing the first acquired image to a plurality of reference images by determining a first plurality of confidence metrics using a first calculation method; and
outputting the first plurality of confidence metrics for the first acquired image, wherein each respective confidence metric of the first plurality of confidence metrics indicates how closely the first acquired image matches a respective reference image of the plurality of reference images.
Examiner Note: underlined elements indicate additional elements of the claimed invention identified as performing the steps of the claimed invention.
The judicial exception expressed in claim 1 is not integrated into a practical application. The claim recites the additional elements of receiving a first acquired image of a patient, wherein the first acquired image is a medical image acquired using a medical imaging device; and outputting the first plurality of confidence metrics for the first acquired image. These limitations are recited at a high level of generality (i.e., as a general means of data acquisition and presenting results), and amounts to data gathering and outputting and displaying information, which are forms of insignificant extra-solution activities. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
Therefore, under step 2A, the claims are directed to the abstract idea, and require further analysis under Step 2B.
Under step 2B
For claim 1, under step 2B, the additional elements of receiving a first acquired image of a patient, wherein the first acquired image is a medical image acquired using a medical imaging device; and outputting the first plurality of confidence metrics for the first acquired image have been evaluated. The computer implemented method performs a general function of receiving patient data for image comparison and analysis, which represents a well-understood, routine, and conventional activity in the field of medical image processing and medical image systems. The specification discloses that the computer is used in its ordinary capacity as a data input device and does not describe any improvement to the computer itself or to the functioning of the overall computer system (see page 10, lines 19-22). Also noted in Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354, 119 USPQ2d 1739, 1742 (Fed. Cir. 2016), merely collecting information for analysis without a technological improvement does not add significantly more to an abstract idea. The use of the computer implemented method is no more than collecting information before performing analysis, comparing acquired images to reference images and determining confidence metrics indicating image similarity, and outputting the results and does not integrate the abstract idea into a practical application. Therefore, the claim does not recite an inventive concept and is not patent eligible.
Claims 2, 4-6, and 13-14 recite no further additional elements, and only further narrow the abstract idea. The previously identified additional elements, individually and as a combination, do not integrate the narrowed abstract idea into a practical application for reasons similar to those explained above, and do not amount to significantly more than the narrowed abstract idea for reasons similar to those explained above.
Claims 3, 7-12, and 15-17 recite the additional elements of labelling the first acquired image as matching the respective reference image (claims 3, 7, and 9), outputting a first plurality of confidence metrics for each acquired image (claim 8), labelling the acquired image with the confidence metric that indicates the strongest match as the acquired image matching the reference image (claim 10), displaying each acquired image and the corresponding matching reference image such that the user can view each acquired image and each corresponding matching reference image, and optionally displaying the respective confidence metric of each acquired image such that the user can view each acquired image, each corresponding matching reference image and each respective confidence metric, and optionally receiving a user input (claim 11), outputting information indicating that an abnormality has been detected (claim 12), receive, as an input, the first or each acquired image (claim 15), a processor (claim 16), and by a computer (claim 17). However, these additional elements amount to implementing an abstract idea on a generic computing device, mere data gathering, or insignificant application (i.e., an insignificant extra-solution activity)). As such, this additional element, when considered individually or in combination with the prior devices, does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea.
Thus, as the dependent claims remain directed to a judicial exception, and as the additional elements of the claims do not amount to significantly more, the dependent claims are not patent eligible.
Therefore, the claims here fail to contain any additional element(s) or combination of additional elements that can be considered as significantly more and the claims are rejected under 35 U.S.C. 101 for lacking eligible subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-6, 8, and 12-17 are rejected under 35 U.S.C. 103 as being unpatentable over Zabair et al. (U.S. Patent Publication 2013/0163837 A1), referred to hereinafter as Zabair, in view of Rom et al. (International Publication Number WO2015092567A1), referred to hereinafter as Rom.
Regarding claim 1, Zabair teaches a computer implemented method of medical imaging comprising (Zabair, [0006] “Echocardiography is described below in detail as an illustrative example of an application of the present invention. As described below, the invention can be applied to compare a plurality of images of a heart at rest with a plurality of images of the heart under stress. Various useful information can be gleaned from the comparison. The systems and methods of the present invention can also be applied to other image sequence processing subjects.”):
comparing the first acquired image, by determining a first plurality of confidence metrics using a first calculation method (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”); and
outputting the first plurality of confidence metrics for the first acquired image, wherein each respective confidence metric of the first plurality of confidence metrics indicates how closely the first acquired image matches a respective reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”).
Zabair fails to explicitly teach receiving a first acquired image of a patient, wherein the first acquired image is a medical image acquired using a medical imaging device; and a plurality of reference images.
Rom teaches receiving a first acquired image of a patient, wherein the first acquired image is a medical image acquired using a medical imaging device (Rom, pages 10-11, “One example of a set of stages is provided in the FIG. 1 which shows a flow diagram of a stress-echo process 100 (hereinafter the process 100 for the sake of clarity) performed in accordance with embodiments of the present system. The process 100 may include first through fourth stages 101 through 107, respectively. The first stage 101 is a rest stage (RS), the second stage 103 is a peak heart rate stage (PHR) (which may also be referred to as a peak exercise stage), and the third stage 105 is a post exercise stage. During each of the first through third stages 101 -105, respectively, echo information is acquired from an ultrasonic probe (e.g., a probe) of the system and thereafter stored in a memory of the system.”); and
a plurality of reference images (Rom, page 31, “In accordance with embodiments of the present system, during portions of the stress-echo process, reference images (e.g., frames) corresponding to a selected image plane may be generated from image information reconstructed from the acquired echo information. Then post-acquisition (e.g., post exam), reference images which are desired for an evaluation may be reconstructed from the stored echo information and rendered on a display of the system.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to modify the stress echocardiography image processing system of Rom with the image comparison and confidence and similarity scoring techniques taught by Zabair in order to improve the accuracy and reliability of matching acquired cardiac ultrasound images to corresponding reference images during stress echocardiography evaluation. Rom teaches acquiring cardiac ultrasound images during multiple stress echo stages and generating reference images for evaluation purposes, while Zabair teaches comparing image regions, computing similarity measures, computing confidence measures for multiple candidate matches, and identifying a superior image match based on the computed measures. A person of ordinary skill in the art would have recognized that Zabair’s matching techniques could be beneficially applied within Rom’s stress echo imaging workflow to facilitate improved image comparison and matching between acquired images and stored reference images.
Further, a person of ordinary skill in the art would have been motivated to incorporate Zabair’s similarity and confidence measure calculations into Rom’s system because stress echocardiography involves comparing images acquired at different stages of stress to corresponding reference views in order to evaluate cardiac function and abnormalities. Applying similarity measures and confidence measures, as taught by Zabair, would have predictably improved the ability of Rom’s system to determine how closely an acquired image corresponds to a reference image, thereby improving workflow efficiency, and improving consistency of image evaluation. The combination merely applies known image comparison and confidence scoring techniques to a known medical imaging workflow to yield predictable results.
Additionally, Zabair expressly teaches computing similarity measures representing the degree of similarity between image regions and computing confidence measures associated with potential image matches, while Rom teaches generating and displaying reference images reconstructed from acquired echo information for evaluation. A person of ordinary skill in the art would have understood that combining these teachings would allow the system to output confidence metrics indicating how closely an acquired image matches a respective reference image. The modification would have involved the predictable use of prior art elements according to their established functions and would have been well within the ordinary skill in the art.
Regarding claim 2, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach wherein determining each respective confidence metric of the first plurality of confidence metrics comprises determining the probability that the first acquired image shows the same view of the patient as the respective reference image of the plurality of reference images (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”, and
Rom, page 31, “In accordance with embodiments of the present system, during portions of the stress-echo process, reference images (e.g., frames) corresponding to a selected image plane may be generated from image information reconstructed from the acquired echo information. Then post-acquisition (e.g., post exam), reference images which are desired for an evaluation may be reconstructed from the stored echo information and rendered on a display of the system.”, and Rom, pages 23-24, “During act 327, the process may determine whether an image plane from the plurality of image planes has been selected for viewing. This image plane may be known as a selected image plane or a view and may be selected by the process and/or the user. Accordingly, if it is determined that an image plane has been selected, the process may continue to act 353. However, if it is determined that the image plane has not been selected, the process may continue to act 331. In accordance with some embodiments, the process may generate and render a request for the user to enter or otherwise select an image plane from the plurality of image planes to be a selected image plane. Thereafter, a user may select, for example, an image plane from the M image planes to be the selected image plane. In accordance with some embodiments, the user may select an image plane from the M image planes using a menu selection item or graphic representation from a menu rendered as a user interface (Ul) on a display of the system. For example, the operator may select an image when the image has sufficient image quality to visualize a desired anatomical structure desired for the view.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to modify the stress echocardiography image processing system of Rom with the similarity and confidence image matching techniques taught by Zabair in order to determine confidence or probability measures indicating whether an acquired cardiac ultrasound image corresponds to the same view as a reference image. Rom teaches generating reference images corresponding to selected image planes or views during a stress echocardiography process and selecting image planes based on visualization of desired anatomical structures associated with the view, while Zabair teaches comparing image regions, computing similarity measures and confidence measures for multiple candidate matches, and combining the similarity and confidence measures to identify a superior match between image data. A person of ordinary skill in the art would have recognized that applying Zabair’s confidence and similarity scoring techniques within Rom’s selected image-plane and view framework would improve the reliability, consistency, and efficiency of identifying corresponding cardiac views during stress echocardiography evaluation. The combination merely applies known image similarity and confidence scoring techniques within a known cardiac ultrasound imaging workflow to achieve predictable results including improved view matching and image comparison accuracy.
Regarding claim 4, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach wherein the comparing step further comprises: determining a second plurality of confidence metrics using a second calculation method, wherein the second calculation method calculates the confidence metrics in a different way to the first calculation method; and calculating a plurality of combined confidence metrics, wherein each respective confidence metric of the plurality of combined confidence metrics indicates how closely the first acquired image matches a respective reference image based on the respective confidence metric of the plurality of first confidence metrics and the respective confidence metric of the plurality of second confidence metrics (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to utilize multiple different image comparison metrics and combine the resulting confidence information when determining correspondence between medical images, as taught by Zabair. Zabair teaches comparing image data using different types of calculated measures, including a similarity measure representing a degree of textual similarity between image regions and a confidence measure representing smoothness and motion coherence associated with the image displacement. Zabair further teaches combining the similarity measures and confidence measures to identify a superior image match. A person of ordinary skill in the art would have recognized that the similarity measures and confidence measures represent different calculation methods for evaluating image correspondence and that combining the resulting metrics would improve the reliability and accuracy of image matching determinations by incorporating multiple types of image comparison information into the matching process. The combination merely applies known image similarity and confidence evaluation techniques within a known medical image comparison workflow to achieve predictable results including improved image matching accuracy and robustness.
Regarding claim 5, Zabair and Rom teach the invention in claim 4, as discussed above, and further teach wherein determining each respective confidence metric of the second plurality of confidence metrics comprises: determining the degree of similarity between the first acquired image and the respective reference image of the plurality of reference images, or determining the correspondence between anatomical features detected in the first acquired image and the respective reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to determine confidence metrics for medical image matching based on degrees of similarity between acquired images and reference images, as taught by Zabair. Zabair teaches comparing image regions, computing similarity measures representing a degree of similarity between image data, and using the similarity measures together with confidence measures to identify superior image matches. A person of ordinary skill in the art would have recognized that determining image similarity is a known and useful technique for evaluating correspondence between medical images and that utilizing similarity confidence metrics would improve the reliability and accuracy of identifying matching images within a medical imaging workflow. The modification merely applies known image similarity and confidence evaluation techniques within a known medical image comparison process to achieve predictable results including improved image matching and correspondence determination.
Regarding claim 6, Zabair and Rom teach the invention in claim 4, as discussed above, and further teach wherein the comparing step further comprises determining a third plurality of confidence metrics using a third calculation method, wherein the third calculation method calculates the confidence metrics in a different way compared to first and second calculation methods, and calculating a plurality of combined confidence metrics, wherein each respective combined confidence metric indicates how closely the first acquired image matches a respective reference image based on the respective confidence metric of the plurality of first confidence metrics, the respective confidence metric of the plurality of second confidence metrics and the respective confidence metric of the third plurality of confidence metrics, and optionally wherein determining each respective confidence metric of the second plurality of confidence metrics comprises determining the degree of similarity between the first acquired image and a respective reference image of the plurality of reference images, and determining each respective confidence metric of the third plurality of confidence metrics comprises determining the correspondence between anatomical features detected in the first acquired image and each respective reference image of the plurality of reference images (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.” and Zabair [0225] “In an embodiment, the ML estimation is not simply carried out in terms of estimating the motion that gives the maximum similarity measure within the search window, but instead we apply Singh's approach [A. Singh. Image-flow computation: An estimation-theoretic framework and a unified perspective. CGVIP: Image Understanding, 65(2):152-177, 1992.]. In such a formulation, the similarity measure, ECD 2bis , is computed for each possible displacement, (u, v), of the template block from the rest frame, Wr, within the search window in the stress frame, Ws. Then a confidence measure is assigned to each possible displacement by computing a response, RCD 2bis , derived from ECD2 bis , which represents the likelihood of such a displacement.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to utilize multiple different image comparison calculations and combine the resulting confidence information when determining correspondence between medical images, as taught by Zabair. Zabair teaches computing different types of image comparison information including similarity measures representing degrees of similarity between image regions, confidence measures associated with motion coherence and smoothness, and likelihood response measures derived from the similarity calculations for possible image displacements. Zabair further teaches combining the similarity measures and confidence measures to identify a superior image match. A person of ordinary skill in the art would have recognized that the different similarity, confidence, and likelihood calculations represent different calculation methods for evaluating image correspondence and that combining the resulting metrics would improve the robustness, and accuracy of medical image matching determinations by incorporating multiple forms of image comparison information into the matching process. The combination merely applies known image similarity analysis, confidence evaluation, and likelihood estimation techniques within a known medical image comparison workflow to achieve predictable results including improved image matching and correspondence determination.
Regarding claim 8, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach wherein the first acquired image is one of a plurality of acquired images, wherein each of the plurality of acquired images are medical images acquired using a medical imaging device, the method further comprising (Rom, pages 10-11, “One example of a set of stages is provided in the FIG. 1 which shows a flow diagram of a stress-echo process 100 (hereinafter the process 100 for the sake of clarity) performed in accordance with embodiments of the present system. The process 100 may include first through fourth stages 101 through 107, respectively. The first stage 101 is a rest stage (RS), the second stage 103 is a peak heart rate stage (PHR) (which may also be referred to as a peak exercise stage), and the third stage 105 is a post exercise stage. During each of the first through third stages 101 -105, respectively, echo information is acquired from an ultrasonic probe (e.g., a probe) of the system and thereafter stored in a memory of the system.” and Rom, page 31, “In accordance with embodiments of the present system, during portions of the stress-echo process, reference images (e.g., frames) corresponding to a selected image plane may be generated from image information reconstructed from the acquired echo information. Then post-acquisition (e.g., post exam), reference images which are desired for an evaluation may be reconstructed from the stored echo information and rendered on a display of the system.”);
receiving the plurality of acquired images of the patient (Rom, pages 10-11, “One example of a set of stages is provided in the FIG. 1 which shows a flow diagram of a stress-echo process 100 (hereinafter the process 100 for the sake of clarity) performed in accordance with embodiments of the present system. The process 100 may include first through fourth stages 101 through 107, respectively. The first stage 101 is a rest stage (RS), the second stage 103 is a peak heart rate stage (PHR) (which may also be referred to as a peak exercise stage), and the third stage 105 is a post exercise stage. During each of the first through third stages 101 -105, respectively, echo information is acquired from an ultrasonic probe (e.g., a probe) of the system and thereafter stored in a memory of the system.”),
comparing each acquired image of the plurality of acquired images to a plurality of reference images (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”; and
Rom, page 31, “In accordance with embodiments of the present system, during portions of the stress-echo process, reference images (e.g., frames) corresponding to a selected image plane may be generated from image information reconstructed from the acquired echo information. Then post-acquisition (e.g., post exam), reference images which are desired for an evaluation may be reconstructed from the stored echo information and rendered on a display of the system.”); and
outputting a first plurality of confidence metrics for each acquired image; wherein the comparing step comprises determining, for each acquired image, a first plurality of confidence metrics using a first calculation method, and wherein each respective confidence metric of the first plurality of confidence metrics for the acquired image indicates how closely the acquired image matches a respective reference image of the plurality of reference images (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.” and
Rom, page 31, “In accordance with embodiments of the present system, during portions of the stress-echo process, reference images (e.g., frames) corresponding to a selected image plane may be generated from image information reconstructed from the acquired echo information. Then post-acquisition (e.g., post exam), reference images which are desired for an evaluation may be reconstructed from the stored echo information and rendered on a display of the system.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to apply the image comparison and confidence matching techniques of Zabair to the plurality of cardiac ultrasound images acquired during the stress echocardiography workflow of Rom in order to compare multiple acquired images to corresponding reference images and determine confidence metrics for each acquired image. Rom teaches acquiring and storing multiple ultrasound images and reference image frames during various stages of a stress echocardiography examination, including rest, peak heart rate, and post exercise stages, while Zabair teaches comparing image data, computing similarity measures and confidence measures for image comparisons, and combining the measures to identify superior image matches. A person of ordinary skill in the art would have recognized that applying Zabair’s similarity and confidence image comparison techniques to each of the plurality of acquired ultrasound images in Rom’s stress echocardiography workflow would improve the reliability and automation of identifying corresponding image views and evaluating image correspondence across multiple acquired frames. The combination merely applies known image comparison and confidence evaluation techniques within a known multi image medical imaging workflow to achieve predictable results including automated comparison of multiple acquired cardiac ultrasound images to reference images and generation of confidence metrics indicating the closeness of corresponding image matches.
Regarding claim 12, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach further comprising: determining the presence of an abnormality in one of the acquired images (Zabair [0238] “In an embodiment, rest and stress segmentations derived using one or more of the methodologies described above are used to classify the behavior of the image subject. In an embodiment, the subject is a heart and the behavior is classified as normal or abnormal. In an embodiment, a particular class of abnormal behavior is identified. In an embodiment, the classifications aide diagnosis of cardiac disease. In an embodiment, the classification highlights subjects for further review by experts or otherwise.”); and
outputting information indicating that an abnormality has been detected (Zabair [0241] “In an embodiment, datasets are classified into normal (clinical scoring scheme value of 1) or abnormal (clinical scoring scheme values 2-4) depending on the characteristics measurable from the rest and/or stress segmentations. Classification can be carried out using a variety of features.” and Zabair [0271] “Accurate determination of normal/abnormal hearts means that clinicians can then focus on the ones that are flagged as abnormal on the system and this can potentially significantly reduce the time spent on analysis.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to incorporate abnormality detection and abnormality indication functionality into the combined medical image comparison and stress echocardiography system of Rom and Zabair, as taught by Zabair. Zabair teaches processing rest and stress cardiac image segmentations to classify cardiac behavior as normal or abnormal, identifying particular classes of abnormal behavior, and classifying datasets as normal or abnormal based on characteristics measurable from rest and stress segmentations. Zabair further teaches flagging abnormal hearts on the system so clinicians can focus review on the identified abnormal cases. A person of ordinary skill in the art would have recognized that determining the presence of abnormalities in acquired cardiac ultrasound images and outputting indications of detected abnormalities would improve diagnostic efficiency, reduce clinician review time, and improve evaluation of stress echocardiography results. The modification merely applies known abnormality classification and notification techniques within a known cardiac image analysis workflow to achieve predictable results including improved cardiac abnormality detection and clinical decision support.
Regarding claim 13, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach wherein the reference images correspond to the patient being in a rest state, and the first or each acquired image correspond to the patient being in a stress state (Zabair [0152] “In a particular embodiment, the subject invention is applied to stress echocardiography images. In stress echocardiography, a heart (or typically the left ventricle) is imaged using ultrasound at rest and then after administration of a stress stimulus. Clinical analysis is typically 2D and focuses on visualizing these two sequences of images (rest and stressed) side by side. A sick heart typically moves differently under stress than a healthy one and the clinician will grade this motion by eye and from this decide whether the heart is abnormal or not. This interpretation is subjective. Automating this task is difficult. Methods developed for analyzing rest images do not work well on stress images. Embodiments of the subject invention are directed to automating processing of the stressed images using results from processing the higher quality rest images.” Zabair [0153] “Thus, in an embodiment, the invention involves a prediction of a stress sequence segmentation given a rest sequence segmentation of the left ventricle. In a further embodiment, the prediction is then updated by either using more traditional segmentation algorithms such as level sets or snakes or using image derived information from the rest sequence to inform the segmentation of the stress sequence as described above.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to configure the image comparison and matching methods of the combined Rom and Zabair system such that the reference images correspond to a patient in a rest state and the acquired images correspond to a patient in a stress state, as taught by Zabair. Zabair teaches stress echocardiography imaging in which ultrasound images of the heart are acquired at rest and after administration of a stress stimulus, and further teaches processing and analyzing stressed image sequences using information derived from higher quality rest image sequences, including predicting stress sequence segmentation from rest sequence segmentation. A person of ordinary skill in the art would have recognized that using rest state images as reference images and stress-state images as acquired and comparison images would improve the accuracy and reliability of cardiac image comparison and abnormality evaluation during stress echocardiography procedures. The modification merely applies known stress and rest cardiac image comparison techniques within a known medical image processing workflow to achieve predictable results including improved image matching and stress image analysis.
Regarding claim 14, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach wherein the reference images correspond to standard anatomical views, and/or wherein the reference images and the first or each acquired image are cardiac ultrasound images (Rom, page 24-25, “During act 353, the process may generate image information corresponding with the (e.g., single) selected image plane in accordance with the acquired ultrasound echo information. Accordingly, the process may obtain the ultrasound echo information stored in a memory of the system in the rich data format and reconstruct corresponding image information using any suitable method. After completing act 353, the process may continue to act 359. During act 359, the process may render the generated image information corresponding with the selected image plane on a user interface of the system such as a display. The generated image information may then be viewed in real-time during the Stress exam or after the stress exam has been performed. As described further herein, the image information can be in rich data format, thereby allowing for optimization of the images after the stress exam is performed. A user may select any of the rendered images and perform desired operations upon these images at this point. For example, these operations may be used to improve the visual quality of the image so as to better visualize the anatomic structures of the image. These operations may include operations to adjust image quality such as gain, compression, dynamic range, and Transmit Gain Compensation (TGC), edge enhancements, smoothing, speckle reduction, zooming based on acoustic lines (e.g., scan converting to a higher resolution), and thresholding, turbulence, baseline, etc. in a case wherein stress color flow imaging is being performed. Thereafter, the process may generate corresponding images based on the performed operations having a format (e.g., a compressed format) which is different from the format of the ultrasound echo information which is stored in the rich data format. The generated images may be stored in a memory of the system for later use. For example, the generated images may be stored in a compressed format that is lossy or lossless, such as run length encoded (RLE), MPEG, JPEG, BMP, AVI, etc. After completing act 359, the process may continue to act 363”., and Rom, pages 23-24, “During act 327, the process may determine whether an image plane from the plurality of image planes has been selected for viewing. This image plane may be known as a selected image plane or a view and may be selected by the process and/or the user. Accordingly, if it is determined that an image plane has been selected, the process may continue to act 353. However, if it is determined that the image plane has not been selected, the process may continue to act 331. In accordance with some embodiments, the process may generate and render a request for the user to enter or otherwise select an image plane from the plurality of image planes to be a selected image plane. Thereafter, a user may select, for example, an image plane from the M image planes to be the selected image plane. In accordance with some embodiments, the user may select an image plane from the M image planes using a menu selection item or graphic representation from a menu rendered as a user interface (Ul) on a display of the system. For example, the operator may select an image when the image has sufficient image quality to visualize a desired anatomical structure desired for the view.).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to configure the reference images and acquired images of the combined Rom and Zabair system as cardiac ultrasound images corresponding to anatomical views, including standard anatomical views, as taught by Rom. Rom teaches generating and rendering ultrasound image information corresponding to selected image planes or views during a stress echocardiography examination, where the selected views are used to visualize desired anatomical cardiac structures. Rom further teaches that users may select image planes from a plurality of image planes based on image quality and visualization of anatomical structures, and that the ultrasound image information may be processed and displayed during or after the stress exam. A person of ordinary skill in the art would have recognized that using cardiac ultrasound images corresponding to selected anatomical views would improve consistency and clinical usefulness when performing image comparison and matching operations during stress echocardiography analysis. The modification merely applies known cardiac ultrasound imaging views and anatomical visualization techniques within a known medical image comparison workflow to achieve predictable results including improved image matching, anatomical evaluation, and stress echocardiography analysis.
Regarding claim 15, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach wherein calculating the plurality of first confidence metrics comprises running a first prediction algorithm, the first prediction algorithm comprising a trained machine learning algorithm for calculating the probability that an input image matches a standard anatomical view, and the first prediction algorithm is configured to (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”, and
Zabair [0119] In an embodiment, the method 401 proceeds to a step 441 wherein the subject of the plurality is classified based one or more results of the step 431. In an embodiment, the subject is classified as normal or abnormal. In an embodiment, the subject is classified into established categories. Various classification techniques can be used. In an embodiment, a supervised classification technique is used wherein a machine learning technique learns a function for classification from training data, which includes measurements or calculations upon which to base the classification (inputs) paired with their corresponding classes (outputs). In an embodiment, the training data includes one more calculations described in the step 431 above. Various learning techniques can be used, such as a naive Bayes' classifier and a random forests classifier. The naive Bayes' classifier is based on Bayes' theorem and makes strong independence assumptions about the inputs. A naïve Bayes classifier also assumes that all the inputs are equally powerful in their ability to distinguish between classes. In an embodiment, random forests can produce a very accurate classifier because of its ability to estimate the importance of variables in the classification (i.e., all inputs are not thought to be equally important as in the naive Bayes' classifier). In an embodiment, one or more results from the step 431 are used as inputs to the trained classifier to produce a classification for the subject”),
receive, as an input, the first or each acquired image (Rom, pages 10-11, “One example of a set of stages is provided in the FIG. 1 which shows a flow diagram of a stress-echo process 100 (hereinafter the process 100 for the sake of clarity) performed in accordance with embodiments of the present system. The process 100 may include first through fourth stages 101 through 107, respectively. The first stage 101 is a rest stage (RS), the second stage 103 is a peak heart rate stage (PHR) (which may also be referred to as a peak exercise stage), and the third stage 105 is a post exercise stage. During each of the first through third stages 101 -105, respectively, echo information is acquired from an ultrasonic probe (e.g., a probe) of the system and thereafter stored in a memory of the system.”), and
determine, as an output, based on the input, the plurality of first confidence metrics corresponding to the first or each acquired image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”); and
optionally wherein the comparing step further comprises determining a second plurality of confidence metrics using a second calculation method, wherein the second calculation method calculates the confidence metrics in a different way to the first calculation method, wherein calculating the plurality of second confidence metrics comprises running a second prediction algorithm, wherein the second prediction algorithm comprises a trained machine learning algorithm for calculating the degree of similarity between two images or for determining the correspondence between anatomical features detected in two images, and the second prediction algorithm is configured to: receive, as an input, the first or each acquired image, and determine, as an output, based on the input, the plurality of second confidence metrics corresponding to the first or each acquired image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”, and
Zabair [0119] In an embodiment, the method 401 proceeds to a step 441 wherein the subject of the plurality is classified based one or more results of the step 431. In an embodiment, the subject is classified as normal or abnormal. In an embodiment, the subject is classified into established categories. Various classification techniques can be used. In an embodiment, a supervised classification technique is used wherein a machine learning technique learns a function for classification from training data, which includes measurements or calculations upon which to base the classification (inputs) paired with their corresponding classes (outputs). In an embodiment, the training data includes one more calculations described in the step 431 above. Various learning techniques can be used, such as a naive Bayes' classifier and a random forests classifier. The naive Bayes' classifier is based on Bayes' theorem and makes strong independence assumptions about the inputs. A naïve Bayes classifier also assumes that all the inputs are equally powerful in their ability to distinguish between classes. In an embodiment, random forests can produce a very accurate classifier because of its ability to estimate the importance of variables in the classification (i.e., all inputs are not thought to be equally important as in the naive Bayes' classifier). In an embodiment, one or more results from the step 431 are used as inputs to the trained classifier to produce a classification for the subject”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to utilize trained machine learning algorithms together with similarity and confidence image comparison techniques in a stress echocardiography imaging workflow in order to determine confidence metrics indicating correspondence between acquired medical images and reference image views. Rom teaches acquiring cardiac ultrasound images during multiple stages of a stress echocardiography examination, thereby providing acquired medical image inputs for processing, while Zabair teaches supervised machine learning classification techniques trained using image measurements and calculations, wherein trained classifiers receive inputs and generate classification outputs based on the training data. Zabair further teaches computing similarity measures and confidence measures for image comparisons and combining the measures to identify superior image matches. A person of ordinary skill in the art would have recognized that applying Zabair’s trained machine learning and similarity-based confidence evaluation techniques to the acquired cardiac ultrasound images of Rom would improve the automation, reliability, and accuracy of determining image correspondence and image view matching within a medical imaging workflow. Further, it would have been obvious to utilize an additional similarity prediction algorithm, as taught by Zabair’s similarity measure calculations, to determine degrees of similarity between image data and generate corresponding confidence metrics for image matching. The combination merely applies known machine learning classification techniques and known similarity/confidence-based image comparison techniques within a known cardiac ultrasound imaging workflow to achieve predictable results including automated image comparison and confidence-based image matching.
Regarding claim 16, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach a system for implementing the computer implemented method of claim 1 the system comprising: a processor configured to carry out the method of any preceding claim (Zabair [0272] “Aspects of the invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Such program modules can be implemented with hardware components, software components, or a combination thereof. Moreover, those skilled in the art will appreciate that the invention may be practiced with a variety of computer-system configurations, including multiprocessor systems, microprocessor-based or programmable-consumer electronics, minicomputers, mainframe computers, and the like. Any number of computer-systems and computer networks are acceptable for use with the present invention.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to implement the combined image comparison and stress echocardiography techniques of Rom and Zabair using a processor-based computer system, as taught by Zabair. Zabair teaches that the disclosed image processing operations may be implemented using computer-executable instructions, program modules, hardware components, software components, or combinations thereof executed by various computer system configurations including microprocessor systems and multiprocessor systems. A person of ordinary skill in the art would have recognized that implementing the disclosed medical image comparison and confidence scoring operations using a processor would have been a predictable and routine use of known computer technology to automate and improve medical image processing tasks. The modification merely involves applying known processor implementation techniques to known image comparison methods to achieve predictable results including automated image matching, confidence evaluation, and image analysis.
Regarding claim 17, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the computer implemented method of claim 1 (Zabair [0272] “Aspects of the invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Such program modules can be implemented with hardware components, software components, or a combination thereof. Moreover, those skilled in the art will appreciate that the invention may be practiced with a variety of computer-system configurations, including multiprocessor systems, microprocessor-based or programmable-consumer electronics, minicomputers, mainframe computers, and the like. Any number of computer-systems and computer networks are acceptable for use with the present invention.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to provide the disclosed image comparison and stress echocardiography methods of Rom and Zabair in the form of a computer program product comprising executable instructions. Zabair teaches that aspects of the disclosed invention may be implemented as computer executable instructions and program modules executed by a computer system to perform the disclosed image processing operations. A person of ordinary skill in the art would have understood that encoding the disclosed image comparison, similarity scoring, and confidence determination operations into software instructions executable by a computer represents a conventional and well known implementation technique in the field of medical image processing. The claimed computer program product therefore amounts to the predictable use of known software implementation methods to perform known image analysis operations and would have been obvious to implement with a reasonable expectation of success.
Claims 3, 7, and 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Zabair et al. (U.S. Patent Publication 2013/0163837A1), referred to hereinafter as Zabair, in view of Rom et al. (International Publication Number WO2015092567A1), referred to hereinafter as Rom, further in view of Upton et al. (U.S. Patent Publication 2020/0388391 A1), referred to hereinafter as Upton.
Regarding claim 3, Zabair and Rom teach the invention in claim 1, as discussed above, and further teach further comprising: determining whether the first acquired image matches a respective reference image based on the respective confidence metric (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.” and
Rom, page 31, “In accordance with embodiments of the present system, during portions of the stress-echo process, reference images (e.g., frames) corresponding to a selected image plane may be generated from image information reconstructed from the acquired echo information. Then post-acquisition (e.g., post exam), reference images which are desired for an evaluation may be reconstructed from the stored echo information and rendered on a display of the system.”); and
in response to determining that the first acquired image matches the respective reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”).
Zabair and Rom fail to explicitly teach labelling the first acquired image as matching the respective reference image.
Upton teaches labelling the first acquired image as matching the respective reference image (Upton [0032] “The outcome data may be used to label the corresponding reference data sets. The labels may distinguish between different classifications. Each label may, for example comprise a classification indicating a presence or an absence of a condition or a disease. Each label may comprise a grade indicating a severity of a condition, for example comprising an indication of the severity of stenosis.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to modify the stress echocardiography image comparison system of Rom with the similarity and confidence image matching techniques of Zabair and the image labeling techniques of Upton in order to determine whether an acquired cardiac ultrasound image matches a corresponding reference image based on confidence metrics and to label the acquired image accordingly. Rom teaches generating reference images corresponding to selected image planes during stress echocardiography evaluation, while Zabair teaches comparing image data, computing similarity measures and confidence measures for candidate matches, and identifying superior matches based on the computed confidence and similarity information. Upton teaches labeling corresponding image or reference datasets based on determined classifications. A person of ordinary skill in the art would have recognized that once a superior image match is identified using Zabair’s confidence and similarity comparison techniques within Rom’s reference-image framework, it would have been obvious to associate or label the acquired image with the corresponding matched reference image using Upton’s labeling techniques in order to improve organization, tracking, retrieval, and evaluation of corresponding cardiac image data. The combination merely applies known image matching and data labeling techniques within a known medical imaging workflow to achieve predictable results including automated identification and labeling of matching cardiac ultrasound images.
Regarding claim 7, Zabair and Rom teach the invention in claim 4, as discussed above, and further teach further comprising: determining whether the first acquired image matches a respective reference image based on the respective combined confidence metric (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”, and Rom, page 31, “In accordance with embodiments of the present system, during portions of the stress-echo process, reference images (e.g., frames) corresponding to a selected image plane may be generated from image information reconstructed from the acquired echo information. Then post-acquisition (e.g., post exam), reference images which are desired for an evaluation may be reconstructed from the stored echo information and rendered on a display of the system.”); and
in response to determining that the first acquired image matches the respective reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”).
Zabair and Rom fail to explicitly teach labelling the first acquired image as matching the respective reference image.
Upton teaches labelling the first acquired image as matching the respective reference image (Upton [0032] “The outcome data may be used to label the corresponding reference data sets. The labels may distinguish between different classifications. Each label may, for example comprise a classification indicating a presence or an absence of a condition or a disease. Each label may comprise a grade indicating a severity of a condition, for example comprising an indication of the severity of stenosis.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to modify the stress echocardiography image comparison system of Rom with the combined similarity and confidence image matching techniques taught by Zabair and the image labeling techniques taught by Upton in order to determine whether an acquired cardiac ultrasound image matches a corresponding reference image based on combined confidence metrics and to label the acquired image accordingly. Rom teaches generating reference images corresponding to selected image planes during stress echocardiography evaluation, while Zabair teaches comparing image data, computing similarity measures and confidence measures for candidate matches, and combining the similarity measures and confidence measures to identify a superior match between image data. Upton teaches labeling corresponding image or reference datasets based on determined classifications. A person of ordinary skill in the art would have recognized that combining multiple similarity and confidence measures, as taught by Zabair, within Rom’s cardiac ultrasound reference-image framework would improve the reliability and accuracy of image matching determinations, and that once a superior match is identified, it would have been obvious to associate or label the acquired image with the corresponding matched reference image using Upton’s labeling techniques in order to improve organization, tracking, retrieval, and evaluation of corresponding cardiac image data. The combination merely applies known image matching, confidence scoring, and data labeling techniques within a known medical imaging workflow to achieve predictable results including automated identification and labeling of matching cardiac ultrasound images.
Regarding claim 9, Zabair and Rom teach the invention in claim 8, as discussed above, and further teach the method further comprising: determining whether each acquired image matches a respective reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”); and
in response to determining that an acquired image matches a respective reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”).
Zabair and Rom fail to explicitly teach labelling the acquired image as matching the respective reference image.
Upton teaches labelling the acquired image as matching the respective reference image (Upton [0032] “The outcome data may be used to label the corresponding reference data sets. The labels may distinguish between different classifications. Each label may, for example comprise a classification indicating a presence or an absence of a condition or a disease. Each label may comprise a grade indicating a severity of a condition, for example comprising an indication of the severity of stenosis.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to incorporate the image labeling techniques taught by Upton into the image matching system of Zabair in order to label acquired images based on identified matching reference images. Zabair teaches comparing image data, computing similarity measures and confidence measures for candidate matches, and identifying a superior match between image regions, thereby determining whether an acquired image matches a respective reference image. Upton teaches labeling corresponding reference datasets based on determined classifications and associating labels with image data. A person of ordinary skill in the art would have recognized that once a superior image match is identified using Zabair’s similarity and confidence comparison techniques, it would have been obvious to label or associate the acquired image with the corresponding matched reference image using Upton’s labeling techniques in order to improve organization, tracking, retrieval, and evaluation of corresponding medical image data. The combination applies known image matching and data labeling techniques within a known medical image processing workflow to achieve predictable results including automated identification and labeling of matching medical images.
Regarding claim 10, Zabair, Rom, and Upton teach the invention in claim 9, as discussed above, and further teach further comprising; in response to determining that multiple acquired images match the same reference image, labelling the acquired image with the confidence metric that indicates the strongest match as the acquired image matching the reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”), and
Upton [0032] “The outcome data may be used to label the corresponding reference data sets. The labels may distinguish between different classifications. Each label may, for example comprise a classification indicating a presence or an absence of a condition or a disease. Each label may comprise a grade indicating a severity of a condition, for example comprising an indication of the severity of stenosis.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to modify the image comparison and labeling techniques of Zabair and Upton such that, when multiple candidate acquired image matches correspond to the same reference image, the acquired image associated with the strongest confidence or similarity measure is selected and labeled as the matching reference image. Zabair teaches comparing multiple candidate image displacements, computing similarity measures and confidence measures for the candidate matches, and identifying a superior match based on the combined similarity and confidence information. Upton teaches labeling corresponding image or reference datasets based on determined classifications. A person of ordinary skill in the art would have recognized that when multiple possible image matches are identified, selecting the strongest or superior match based on confidence metrics prior to labeling would improve the reliability and accuracy of image association and evaluation. The combination applies known confidence-based match selection and known image labeling techniques within a known medical image processing workflow to achieve predictable results including automated selection and labeling of the strongest corresponding medical image match.
Regarding claim 11, Zabair, Rom, and Upton teach the invention in claim 9, as discussed above, and further teach further comprising: displaying each acquired image and the corresponding matching reference image such that the user can view each acquired image and each corresponding matching reference image, and optionally displaying the respective confidence metric of each acquired image such that the user can view each acquired image, each corresponding matching reference image and each respective confidence metric, and optionally receiving a user input, and using the user input to determine whether at least one of the acquired images that has been labelled as matching a reference image should be accepted as a match, rejected as a match or re- labelled to match another reference image (Zabair [0145] “At a step 615, the template is compared to each of the possible displacements of the template in the search window. In an embodiment, a superior match is identified between one of the possible displacements and the template. In an embodiment, texture information in the template is compared with texture information in each of the possible displacements. In an embodiment, identifying the superior match comprises computing a similarity measure for each of the possible displacements, wherein the similarity measure represents a degree of textual similarity between the template set and the displacement. In an embodiment, the similarity measure is based on Rayleigh distributed speckle in the template set and the displacement. In an embodiment, identifying the superior match further comprises computing a confidence measure for each of the possible displacements, wherein the confidence measure measures the smoothness and/or motion coherence of motion leading to the displacement. In an embodiment, the similarity measures and confidence measures are combined to identify the superior match.”), and Zabair [0241] “In an embodiment, datasets are classified into normal (clinical scoring scheme value of 1) or abnormal (clinical scoring scheme values 2-4) depending on the characteristics measurable from the rest and/or stress segmentations. Classification can be carried out using a variety of features.” and Rom, pages 23-24, “During act 327, the process may determine whether an image plane from the plurality of image planes has been selected for viewing. This image plane may be known as a selected image plane or a view and may be selected by the process and/or the user. Accordingly, if it is determined that an image plane has been selected, the process may continue to act 353. However, if it is determined that the image plane has not been selected, the process may continue to act 331. In accordance with some embodiments, the process may generate and render a request for the user to enter or otherwise select an image plane from the plurality of image planes to be a selected image plane. Thereafter, a user may select, for example, an image plane from the M image planes to be the selected image plane. In accordance with some embodiments, the user may select an image plane from the M image planes using a menu selection item or graphic representation from a menu rendered as a user interface (Ul) on a display of the system. For example, the operator may select an image when the image has sufficient image quality to visualize a desired anatomical structure desired for the view.”).
It would have been obvious to a person of ordinary skill in the art at the time of the invention to incorporate the similarity and confidence image matching techniques of Zabair into the graphical user interface and image view selection workflow of Rom in order to display acquired medical images together with corresponding matching reference images and associated confidence information for user review. Zabair teaches computing similarity measures and confidence measures for image comparisons and combining the measures to identify superior image matches, while Rom teaches displaying selected medical image planes within a graphical user interface and allowing user interaction and image selection through the interface. A person of ordinary skill in the art would have recognized that presenting the matched image results and associated confidence information within Rom’s user interface would improve the usability, review, and evaluation of medical image matching results by allowing users to visually inspect corresponding image matches and interact with displayed image data. Further, it would have been obvious to allow user input relating to the displayed image matches, such as selecting or reviewing image views, as taught by Rom, in order to provide additional user control and verification of image matching operations. The combination merely applies known graphical user interface interaction techniques and known confidence image matching techniques within a known medical imaging workflow to achieve predictable results including improved visualization and review of matched medical images.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Krishnan et al. (International Publication No. WO 2005096226 A2) teaches systems and methods are provided for automatically processing medical images to identify anatomical structures, determine imaging views or poses, and assess diagnostic image quality using extracted image features.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYRA R LAGOY whose telephone number is (703)756-1773. The examiner can normally be reached Monday - Friday, 8:00 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kambiz Abdi can be reached at (571)272-6702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.L./Examiner, Art Unit 3685
/KAMBIZ ABDI/Supervisory Patent Examiner, Art Unit 3685
Prosecution Insights