DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims and Rejections
Claims 16-37 were pending. Claim 33 was cancelled, and claims 38 and 39 are newly added. Claims 16 and 37 have been amended.
Claims 16-32 and 34-39 are pending.
Claim Interpretation
Claims recite the newly amended limitation of “when a predetermined number of ultrasound images have been provided” which in an interpretation it may be construed as a conditional limitation due the conditional limitation of “when” where the limitations followed by the conditional limitations may not be given a full weight in light of the below decisions as for considering the other case scenario of “when a predetermined number of ultrasound images” not being “have been provided”.
In the recent Ex parte Gopalan decision, the PTAB addressed a claim where all of the features were recited in a conditional manner. A first step of “identifying … an outlier” was performed if “traffic is outside of a prediction interval.” A second step of “identifying” was performed “only when a count of outliers … is greater than or equal to two, and exceeds an anomaly threshold.” These were the only two elements of the independent claim. Thus, if the traffic is never outside Gopalan’s prediction interval, then the steps of the method are never performed.
However, the PTAB distinguished Schulhauser and noted that this construction “would render the entire claim meaningless.” Gopalan at p. 5. The Board went on to state, “Although each of these steps is conditional, they are integrated into one method or path and do not cause the claim to diverge into two methods or paths, as in Schulhauser. Thus, we conclude that the broadest reasonable interpretation of claim 1 requires the performance of both steps…” Id. at p. 6.”
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 39 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 39 recites “wherein the quantification requires execution of at least one million instructions.” It is not clear what is meant by the “quantification” requiring execution of “instructions” or what a single instruction would be. Applicant’s disclosure refers to quantification requiring at least a millions calculations. “This requires millions or billions of calculations to generate a quantification analysis.” ([0076]). Applicant also refers to datapoints. “[T]he ultrasound quantification system is configured to process many thousands or millions of datapoints to process and analyze the plurality of ultrasound images using the quantification algorithm, as well as to provide input to the user, receive input from the user, and potentially modify the quantification.” ([0076]). However, an “instruction” is executed by a processor and can include an operation other than calculating a datapoint. However, Applicant does not explain what is considered to an instruction and what is not an instruction. As such, one having ordinary skill in the art would be unable to interpret the metes and bounds of the claims.
In addition to the above, claim 39 does not recite what is executing the at least one million instructions or if the execution can be performed by multiple components in which the total number of instructions performed by multiple components exceeds one million instructions.
Lastly, claim 39 recites “a method for ultrasound quantification, the method comprising….” The term comprising is open-ended. It is not clear if the quantification, as recited in claim 39, refers to a single measurement, the processes to obtain the single measurement, all measurements for a single subject, all processes to obtain those measurements, or all processes/measurements for multiple subjects.
Accordingly, claim 39 is rejected as being indefinite.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 16-32 and 34-39 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claims 16 and 37 recite or similarly recite:
when a predetermined number of ultrasound images have been provided, automatically selecting, by the processor, an ultrasound image subset from the plurality of ultrasound images without requiring a user input to select the ultrasound image subset; (claims 16 and 37)
automatically performing, by the processor, quantification associated with an anatomy of the subject using the ultrasound image subset, wherein the quantification is performed without requiring a user input to activate the quantification; (claims 16 and 37)
determining, by the processor, whether the quantification satisfies a predetermined quality threshold, wherein the notification is output to the display only when the quantification satisfies the predetermined quality threshold; (claim 24)
wherein automatically selecting the ultrasound image subset comprises performing, by the processor, an image quality check on the plurality of ultrasound images, wherein the ultrasound image subset comprises ultrasound images satisfying the image quality check or checking for viability; (claims 30 and 36)
identifying a plurality of cardiac beats (or overlapping cardiac beats) in the plurality of ultrasound images, wherein the ultrasound image subset comprises ultrasound images representative of the plurality of cardiac beats; (claims 31 and 33)
tracking, by the processor, a number of the plurality of ultrasound images provided to the quantification pipeline; (claim 34)
Claim limitations [a], [d], and [e], as drafted and under their broadest reasonable interpretations, recite a mathematical concept and/or a mental process. (MPEP 2106.04(a)(2)(I) (see, e.g., Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (although the claims did not recite a particular mathematical formula, the court held “[w]ithout additional limitations, a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.”)). The claim limitations are a mathematical concept because selecting ultrasound images and performing an image quality check analyzing ultrasound images, wherein image analysis requires applying intensity values to various thresholds and/or segmenting images. The claim limitation also recites a mental process because it can be performed in a human mind. (see MPEP § 2106.04(a)(2)(III)). Examples of mental processes include “observations, evaluations, judgments, and opinions.” (Id). In this case, selecting, by the processor, an ultrasound image subset from the plurality of ultrasound images without requiring a user input to select the ultrasound image subset is a mental process. NOTE: Although the claim limitations recites “without requiring user input,” the claim limitation may still be considered a mental process. (see MPEP § 2106.04(a)(2)(III)(C), example 3: “An example of a case in which a computer was used as a tool to perform a mental process is Mortgage Grader, 811 F.3d. at 1324, 117 USPQ2d at 1699… The Federal Circuit determined that these claims were directed to the concept of ‘anonymous loan shopping’, which was a concept that could be “performed by humans without a computer.” 811 F.3d. at 1324, 117 USPQ2d at 1699. Another example is Berkheimer v. HP, Inc., 881 F.3d 1360, 125 USPQ2d 1649 (Fed. Cir. 2018), in which the patentee claimed methods for parsing and evaluating data using a computer processing system. The Federal Circuit determined that these claims were directed to mental processes of parsing and comparing data, because the steps were recited at a high level of generality and merely used computers as a tool to perform the processes. 881 F.3d at 1366, 125 USPQ2d at 1652-53.”
Claim limitation [b] and [f], as drafted and under their broadest reasonable interpretation, recite a mathematical concept. (MPEP 2106.04(a)(2)(I) (see, e.g., Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (although the claims did not recite a particular mathematical formula, the court held “[w]ithout additional limitations, a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.”)). The claim limitations are mathematical concept because the claim limitations require a quantification associated with an anatomy of the subject using the ultrasound image subset or counting a number of images.
Claim limitation [c], as drafted and under its broadest reasonable interpretation, recites a mathematical concept. (MPEP 2106.04(a)(2)(I) (see, e.g., Digitech Image Techs., LLC v. Electronics for Imaging, Inc., 758 F.3d 1344, 1350, 111 USPQ2d 1717, 1721 (Fed. Cir. 2014) (although the claims did not recite a particular mathematical formula, the court held “[w]ithout additional limitations, a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.”)). The claim limitation is a mathematical concept because the claim limitation requires comparing a quantification to a threshold.
The next question is to consider whether the claims integrate the judicial exception into a practical application. A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. (MPEP 2106.04(d)). In this case, some additional elements/steps to consider include:
obtaining, by an ultrasound transducer probe, a plurality of ultrasound images while a user is performing a live ultrasound scan of a subject; (claims 16 and 37)
continuously providing, by a processor, the plurality of ultrasound images to a quantification pipeline during the live ultrasound scan; (claims 16 and 37)
outputting to a display, by the processor, a notification for the user that the quantification is available for review while the display is in a live scan mode; (claims 16 and 37)
wherein the plurality of ultrasound images comprise a video or two- dimensional apical 4-chamber images; (claims 17 and 18)
receiving, by the processor, a user input selecting the notification or editing the quantification or deleting the quantification or returning to the live scan mode or resetting the user input; (claims 19, 21-23, 28)
wherein the review mode displays the quantification as one or more labeled images; (claim 20)
storing, by the processor in a memory, the ultrasound image subset as an image loop or a numerical value derived from the quantification; (claims 25 and 26)
wherein the notification comprises an indicator of a quality of the quantification; (claim 29)
outputting to the display, by the processor, a progress indicator indicating a percentage of the plurality of cardiac beats that has been obtained; (claim 32)
triggering the processor to perform the quantification in response to the automatic selection of the ultrasound image subset; (claim 35).
images spanning at least two cardiac beats (claim 38);
executed instructions being at least one millions instructions (claim 39).
Here, the judicial exception is not integrated into a practical application. The additional elements/steps (1), (2), (11), and (12) recite the insignificant pre-solution activity of obtaining images and providing them to the algorithm. (MPEP 2106.04(d)(I), which also refers to MPEP 2106.05(g)). In order to evaluate images, it is first necessary to acquire the images and provide the images for evaluation. The additional elements/steps (3) and (5)-(10) recite insignificant post-solution activity of either displaying measurements/quantifications, storing data after the quantification, notifying the user about the quantifications, or receiving user inputs about the quantifications. Triggering analysis after receiving the subset of images for analysis is also insignificant post-solution activity. (MPEP 2106.04(d)(I), which also refers to MPEP 2106.05(g)). Additional step/element (4) generally links the use of a judicial exception to a particular technological environment or field of use.
The claims do not include additional elements/steps that are sufficient to amount to significantly more than the judicial exception. A shared quality of the additional elements and/or steps is that they do not recite any meaningful limitation that transforms the judicial exception into a patent-eligible application. (MPEP 2106.05(II)). As explained above, each of the additional element/steps (1)-(3) and (5)-(10) recite insignificant extra-solution activity (pre-solution or post-solution) that do not impose meaningfully limits on the claim. Additional step/element (4) only generally links the use of a judicial exception to a particular technological environment or field of use. Moreover, additional elements (1)-(9) are well-understood, routine, conventional activities that are known to the industry. (MPEP 2106.05(A)). (see, e.g., AASE, EDGE USER GUIDE, and ROTHBERG discussed in the Section 103 rejections below). Accordingly, the additional elements do not recite any meaningful limitation that transforms the judicial exception into a patent-eligible application.
Each of claims 16-37, when viewed as a whole, does not amount to significantly more than the judicial exception. Claims 16-37 are rejected under 35 U.S.C. 101 for lacking patent-eligible subject matter.
RESPONSE TO APPLICANT’S ARGUMENTS:
Applicant first argues that claim 16 does not recite a mathematical concept because the claim is only based on or involves a mathematical concept. As explained above, claim 16 necessarily relies upon mathematical calculations when performing image analysis and when performing a quantification.
Applicant also argues that claim 16 does not recite a mental process because it cannot be practically performed in a human mind. However, the Federal Circuit has explained that “methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas....” (MPEP 2106.04(a), III citing CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). Moreover, courts do not “distinguish between claims that recite mental processes performed by humans and claims that recite mental processes performed on a computer. As the Federal Circuit has explained, ‘[c]ourts have examined claims that required the use of a computer and still found that the underlying, patent-ineligible invention could be performed via pen and paper or in a person's mind.’” Versata Dev. Group v. SAP Am., Inc., 793 F.3d 1306, 1335, 115 USPQ2d 1681, 1702 (Fed. Cir. 2015).” (MPEP 2106.04(a), III).
Applicant also argues that the obtaining images using an ultrasound probe during a live scan is not “insufficient pre-solution activity.” However, according to the MPEP, “[t]he term ‘extra-solution activity’ can be understood as activities incidental to the primary process or product that are merely a nominal or tangential addition to the claim. Extra-solution activity includes both pre-solution and post-solution activity. An example of pre-solution activity is a step of gathering data for use in a claimed process ....” (MPEP 2106.05(9)). To be clear, obtaining ultrasound images during a live scan is merely data gathering that is incidental to the primary process/analysis of the images.
Applicant also argues that the claimed invention is integrated into a practical application because, for example, the claimed invention recites a particular machine (i.e., ultrasound probe) or effect a particular treatment. However, the claimed invention does not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. (MPEP 2106.04(d)). The claims do not recite anything particular about the ultrasound probe. (MPEP 2106.05(b)). An ultrasound probe is a well-known and conventional device for acquiring medical image data, and ultrasound scans are well-known procedures used for many medical investigations.
Applicant also argues that the claimed invention recites additional elements that amount to significantly more than the exception itself. However, as explained, “[a] shared quality of the additional elements and/or steps is that they do not recite any meaningful limitation that transforms the judicial exception into a patent-eligible application. (MPEP 2106.05(11)). As explained above, each of the additional element/steps (1)-(3) and (5)-(10) recite insignificant extra-solution activity (pre-solution or post-solution) that do not impose meaningfully limits on the claim. Additional step/element (4) only generally links the use of a judicial exception to a particular technological environment or field of use. Moreover, additional elements (1)-(9) are well-understood, routine, conventional activities that are known to the industry.”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 16-19, 21, 24-26, 30, 34-37, and 39 are rejected are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2023/0255602 A1 (hereinafter “AASE”) and U.S. Patent Appl. Publ. No. 2022/0370046 (hereinafter “NEUMANN”) or, alternatively, Smistad et al. “Real-time automatic ejection fraction and foreshortening detection using deep learning.” IEEE transactions on ultrasonics, ferroelectrics, and frequency control 67.12 (2020): 2595-2604 (hereinafter “SMISTAD”).
AASE teaches systems and methods for automatic measurements of ultrasound images. (Title and Abstract). “During a patient exam workflow where ultrasound images of a patient are acquired, the automatic measurement system 124 may perform a variety of tasks in order to generate the automatic measurements.” ([0030]). “Some measurements performed by the automatic measurement system 124 may be performed in the background while images of the patient are still being obtained by the ultrasound system, with little to no input from the user.” ([0029]). One example workflow is for echocardiograms. (e.g., [0036] and claim 9).
With respect to claim 16, AASE teaches a method for automated ultrasound quantification (see, e.g., Abstract, [0029]; see also Figure 1 and [0027], [0029] describing that the ultrasound imaging system 100 includes a probe 106 and an automatic measurement system 124). AASE teaches a method that includes:
with an ultrasound transducer probe, obtaining a plurality of ultrasound images while a user is performing a live ultrasound scan of a subject (see, e.g., [0023] describing data from probe 106 is processed into images in “real-time”);
continuously providing, by a processor, the plurality of ultrasound images to a quantification pipeline during the live ultrasound scan (see, e.g., “During a patient exam workflow where ultrasound images of a patient are acquired, the automatic measurement system 124 may perform a variety of tasks in order to generate the automatic measurements. This may include obtaining images from the computing device 122 and/or image archive 128, performing one or more processing algorithms (e.g., identifying the scan plane of each saved image, segmenting one or more anatomical features in one or more ultrasound images, measuring features within and/or across one or more ultrasound images)….” (emphasis added) ([0030]). “Some measurements performed by the automatic measurement system 124 may be performed in the background while images of the patient are still being obtained by the ultrasound system, with little to no input from the user.” (emphasis added) ([0029]));
selecting an ultrasound image subset from the plurality of ultrasound images without requiring a user input to select the ultrasound image subset (see, e.g., “The automatic measurements may be performed while the scanner (e.g., ultrasound imaging system) is being utilized to obtain the ultrasound images as part of the patient exam, for example the performance of the automatic measurements may be triggered by a determination that each ultrasound image dictated by the automatic measurement has been obtained (e.g., the proper anatomical features, in the proper planes, have been acquired).” (emphasis added) ([0013]); see also [0043] describing that the images may be “selected automatically in response to a user request to initiate the one or more automatic measurements” and [0051] describing that the background measurement dictates “three images” and the background measurement is automatically started after acquiring the third image).
automatically performing, by the processor, quantification associated with an anatomy of the subject using the ultrasound image subset, wherein the quantification is performed without requiring a user input to activate the quantification (see, e.g., “Thus, according to embodiments disclosed herein, one or more automatic ultrasound measurements may be performed on ultrasound images acquired as part of a patient exam in the background while active imaging is still ongoing, such that measurements are performed automatically without user input.” (emphasis added) ([0013]));
outputting to a display, by the processor, a notification for the user that the quantification is available for review while the display is in a live scan mode (see, e.g., “[M]ethod 300 includes at 326 displaying a notification that the background measurement is available. Because image acquisition of the patient exam is still ongoing, the user may wish to wait until the exam workflow is complete/all images and/or cine loops have been acquired before reviewing the measurement, and thus the notification that the background measurement is available may allow the user to decide when the background measurement result should be reviewed.” (emphasis added) ([0055]; see also [0029])).
While AASE explicitly teaches enabling the user to select ultrasound images for subsequent background measurements and suggests that some tasks may be automated (see, e.g., [0030]: “At least some of the tasks may be initiated without operator intervention, and may be triggered by certain conditions, such as particular anatomical features being visible in the ultrasound images.”), AASE does not explicitly teach when a predetermined number of ultrasound images have been provided, automatically selecting, by the processor, an ultrasound image subset from the plurality of ultrasound images without requiring a user input to select the ultrasound image subset.
In the same field of endeavor, NEUMANN is concerned with automating processes for a technician during an ultrasound scan. “During the scan, the sonographer manually selects ‘key images,’ which are then stored with an identified view label (e.g., ‘left kidney transversal’, ‘common bile duct’, . . . ). For some of these key images, the sonographer further measures certain quantities of interest (e.g., kidney length, width, and/or height, lumen diameter, gallbladder wall thickness, . . . ). Measurement is typically done after the user found a proper view (e.g., high image quality, organ coverage, . . . ), then ‘freezes’ the image, and finally places measurement calipers on the image. Key image selection, view identification, image ‘freezing’, and measurement caliper placement are done manually on-the-fly while scanning. These manual steps are prone to user error and may be inconsistent across sonographers (e.g., typos, different abbreviations, inter-user variability). The quality of the results depends on user experience and skill. Given the number of images, the overall process can be inefficient and tedious.” ([0002]).
To address these issues, NEUMANN teaches automatically classifying views during the ultrasound scan and performing measurements of those views. “A patient is scanned with the ultrasound scanner. The scanning results in a sequence of frames of ultrasound data representing the patient. A view is classified for at least some of the frames of ultrasound data of the sequence. The classified views from multiple of the frames of ultrasound data are gated, where the gating of the classified views rejects a first one of the classified views and passes a second one of the classified views. Structure in the frames of ultrasound data are automatically measured. The frames where measurement is performed are the frames that pass the gating of the classified views.” ([0005]).
With respect to the automatic selection, by a processor, when a predetermined number of ultrasound images have been provided, NEUMANN specifically discusses how manual selection of images can be prone to error. “Key image selection, view identification, image “freezing”, and measurement caliper placement are done manually on-the-fly while scanning. These manual steps are prone to user error and may be inconsistent across sonographers (e.g., typos, different abbreviations, inter-user variability).” ([0002]). NEUMANN then specifically teaches that manual selection can be avoided. “Manually selecting a key view can be replaced by filtered view classification results. Manually ‘freezing’ a frame may be replaced by combination of multiple filtering modules (e.g. view classification filters+measurement-filters (image quality, stability, . . . )).” ([0017]). It is clear that NEUMANN teaches an automatic selection process. “Bad results that may have been produced are filtered out before the user sees them. ‘Flickering’ (e.g., frequent change of view labels in case a classifier cannot properly differentiate one organ from another) is avoided by exploiting temporal and/or other information.” ([0017]).
As to NEUMANN’s workflow, NEUMANN teaches acquiring a real-time stream of ultrasound frames. ([0022] and [0029]). NEUMANN teaches evaluating multiple frames within a temporal window, ([0046] and [0048]), in which some views are accepted by a classification gate and others are rejected. ([0045] and [0048]). NEUMANN’s “temporal filtering approach” teaches that automatic selection occurs when a predetermined number of ultrasound images have been provided. “The view label vt is stored in the FIFO memory queue with capacity n. If the FIFO memory is at capacity (n) and at least m (m≤n) of the stored labels are identical (e.g. all of them are “spleen long”), then the gate is opened, and vt (e.g., “spleen long”) is output.” ([0048]). NEUMANN also teaches a “sequence-level classification….” ([0051]). “If the memory is at capacity, all n images currently in memory are combined (e.g. concatenated). The view classification algorithm is run on the combined image.” ([0052]).
After reaching the predetermined number of images, measurements are performed on the accepted views. ([0029] and [0037]). “The results from the classification (i.e., view label) are passed to one or more (up to Nview) filter modules 24, which apply gates 25 to pass or reject the view label. The view labels that pass the filters 24 and gates 25 are displayed 29A. The measurement algorithm 28 is applied to images 22 associated with passed view labels.” ([0029]). See also [0037]: “The frames on which the measurement is performed may be only the frames with a view passing the limitation of act 17, all frames, or another sub-set of frames.”
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE processor to automatically select, when a predetermined number of ultrasound images have been provided, the images that will be measured in the background. One of ordinary skill in the art would have been motivated to configure the AASE system to further automate tasks (as suggested in AASE) to avoid inconsistent measurements, as taught by NEUMANN, and enhance the user-friendly experience for the technician. There would have been a reasonable expectation of success as NEUMANN teaches that classifying (i.e., selecting) images from the ultrasound scan for further measurement can be performed during a live scan.
Nevertheless, Applicant alleges that NEUMANN does not teach automatically selecting the ultrasound images after a predetermined number of ultrasound images have been provided. (see page 14 of Response). While Examiner disagrees, the motivation to make this modification is also found elsewhere. For example, SMISTAD teaches a real-time automatic ejection fraction and foreshortening detection system. (Title and Abstract). SMISTAD specifically addresses the challenges to acquire ultrasonic measurements during echocardiograms. “Volume and ejection fraction (EF) measurements of the left ventricle (LV) in 2-D echocardiography are associated with a high uncertainty not only due to interobserver variability of the manual measurement, but also due to ultrasound acquisition errors such as apical foreshortening.” (Abstract). To address
PNG
media_image1.png
200
400
media_image1.png
Greyscale
these challenges, SMISTAD teaches a “real-time pipeline [that] allows for a continuous acquisition and measurement workflow without user interaction, and has the potential to significantly reduce the time spent on the analysis and measurement error due to foreshortening, while providing quantitative volume measurements in the everyday echo lab.” (Abstract).
For the SMISTAD pipeline, “[t]he different processing steps are spread out over four main threads to enable concurrent processing in real time.” (p.2598, right column, G. Real-Time Pipeline). See Figure 3 of SMISTAD here. Thread 1 outputs images to Thread 2 for classification. (Id). Thread 2 classifies the images and outputs A4C and A2C images to different image buffers. (Id). Thread 3 searches these image buffers for ED and ES frames. Each new frame is automatically processed and A4C/A2C images are added to the buffers. (p.2598, right column, G. Real-Time Pipeline). Prior to moving to Thread 4, however, SMISTAD teaches that the network requires a predetermined number of ultrasound images in order to accurately predict end-diastole (ED) and end-systole (ES). “Based on our experience, this network needs a sequence of sufficient length to accurately predict ED and ES, thus the minimum of N was set to 30 frames.” (p.2597, right column, top paragraph). The maximum number for N was set to 80 frames. (Id). In SMISTAD’s specific example, Thread 3 “first runs the timing network on the most recent 80 frames, or less, of the current image view buffer. If the timing algorithm is able to find an ED and ES frame, the segmentation network is executed for both these two frames and the result is stored.” (p.2598, right column, G. Real-Time Pipeline). Lastly, Thread 4 uses the ED and ES ultrasound images to perform the measurements. (pp.2598-2599, right column, G. Real-Time Pipeline).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify AASE’s automatic measurement method to include SMISTAD’s live-frame buffering and automatic selection technique. One having ordinary skill in the art would have been motivated to use the classification and buffering technique, as taught in SMISTAD, because it would automate the process of selecting the images to be measured and provide reliable measurements. There would have been a reasonable expectation of success as SMISTAD teaches that the automatic selection technique can be used during an echocardiogram.
With respect to claim 17, AASE teaches that the plurality of ultrasound images comprises two- dimensional apical 4-chamber images. (see, e.g., Apical 4-chamber images are one of the standard images acquired during an echocardiogram. [0047]: “When an image is saved, the image may be entered as input to the anatomy view model, which may output the anatomy view of the image (e.g., apical 4-chamber view, parasternal long axis view, etc.).” See also Figures 4-6 that each include a thumbnail of an apical 4-chamber image labeled “4CH.” “Each saved image is entered as input to an anatomy view model, which outputs the anatomy view of each image. In the current example, a 4-chamber view (4CH), a 2-chamber view (2CH), and a long axis view (LA) have been saved and identified by the anatomy view model.” ([0061])).
With respect to claim 18, AASE teaches that the plurality of ultrasound images comprises a video. (see, e.g., “[A] series of ultrasound images (e.g., multiple frames) may be saved as a cine loop,… Further, while 2D images and/or cine loops are described and illustrated herein, in some examples, 3D images and/or 3D video loops (e.g., 4D images) may be acquired and saved as part of the patient exam.” ([0045]; see also [0043] and [0046])).
With respect to claim 19, AASE teaches receiving, by the processor, a user input selecting the notification (see, e.g., “The notification may include a control button/link that the user may select to view the result of the background measurement.” ([0055])) and transitioning, by the processor, the display from the live scan mode to a review mode that displays the quantification for review by the user (see, e.g., “The results of the measurements may be displayed, for example on display device 118, at the operator's request.” ([0029]); see also [0060]: “For example, the user may select a control button/link on the measurement results notification to bring up the results (while still more images are being acquired and recorded for the exam), and then the user may subsequently edit and/or approve the background measurement. In other examples, the user may wait until the exam is complete (e.g., all images and/or cine loops for the exam have been acquired and saved), at which point the user may select a notification of “unapproved auto-measurements available” in order to review the results of the background measurement(s).”).
With respect to claim 21 (depending from claim 19), AASE teaches receiving, by the processor, a user input to edit the quantification; and automatically updating, by the processor, the quantification based on the user input. (see, e.g., “Further, in some examples, when the user is viewing the results of a background measurement, the user may be presented with an option to override the selection of the images and/or cine loops used to perform the background measurement (e.g., select one or more different images/cine loops for the measurement), and the measurement may be performed again with the new image(s).” (emphasis added) ([0059]); see also [0039]: “If a new image is obtained, the automatic measurement (and its associated processing) may be performed again.”).
With respect to claim 24, AASE teaches determining, by the processor, whether the quantification satisfies a predetermined quality threshold (see, e.g., “In the case that an automatic measurement is generated with low confidence or cannot be generated, a notification may be output via the interface 200, which may prompt the ultrasound operator or clinician to reacquire one or more ultrasound images (e.g., due to the initial image being low quality)….” ([0039]); see also [0053]: “However, if the background measurement cannot be performed or if the result of the background measurement is not rational or plausible, the background measurement may not be determined to have been performed with confidence.”), wherein the notification is output to the display only when the quantification satisfies the predetermined quality threshold (see, e.g., “If, on the other hand, the background measurement is performed with confidence at 322, method 300 includes at 326 displaying a notification that the background measurement is available.” ([0055]); see also [0068]: “In a first example of the method, outputting the notification comprises determining, with the computing device, that the background measurement was executed with confidence, and in response, outputting the notification indicating that the background measurement is available for review.”).
With respect to claim 25, AASE teaches further comprising storing, by the processor in a memory, the ultrasound image subset as an image loop (see, e.g., “When execution of the background measurement is complete, all aspects of the background measurement may be saved in memory for later retrieval, including the images/cine loops used for the measurement, segmentation of the images/cine loop, and measurement results (which may include values, plots, visualizations of measured features, etc.).” ([0050]; see also [0059]).
With respect to claim 26, AASE teaches further comprising storing, by the processor in a memory, a numerical value derived from the quantification, wherein the numerical value is stored in the memory in association with the ultrasound image subset. (see, e.g., “When execution of the background measurement is complete, all aspects of the background measurement may be saved in memory for later retrieval, including the images/cine loops used for the measurement, segmentation of the images/cine loop, and measurement results (which may include values, plots, visualizations of measured features, etc.).” ([0050]; see also [0059]).
With respect to claim 30, AASE teaches automatically selecting the ultrasound image subset comprises performing, by the processor, an image quality check on the plurality of ultrasound images, wherein the ultrasound image subset comprises ultrasound images satisfying the image quality check. (see, e.g., “At 314, method 300 includes checking parameters of each saved image and/or cine loop of the exam as each image is saved and scanning proceeds.” ([0046]). “The parameters that may be checked further include…the image quality of each image and/or cine loop, anatomical consistency across the image(s) and/or cine loop(s) needed for the background measurement, and/or initial image analysis rationality.” ([0047]).
With respect to claim 34, AASE teaches wherein automatically selecting the ultrasound image subset comprises tracking, by the processor, a number of the plurality of ultrasound images provided to the quantification pipeline, wherein the ultrasound image subset comprises a predetermined number of ultrasound images. (see, e.g., “The automatic measurements may be performed while the scanner (e.g., ultrasound imaging system) is being utilized to obtain the ultrasound images as part of the patient exam, for example the performance of the automatic measurements may be triggered by a determination that each ultrasound image dictated by the automatic measurement has been obtained (e.g., the proper anatomical features, in the proper planes, have been acquired).” (emphasis added) ([0013]); see also [0051] describing that the background measurement dictates “three images” and the background measurement is automatically started after acquiring the third image).
With respect to claim 35, AASE teaches further comprising triggering the processor to perform the quantification in response to the automatic selection of the ultrasound image subset. (see, e.g., “The automatic measurements may be performed while the scanner (e.g., ultrasound imaging system) is being utilized to obtain the ultrasound images as part of the patient exam, for example the performance of the automatic measurements may be triggered by a determination that each ultrasound image dictated by the automatic measurement has been obtained (e.g., the proper anatomical features, in the proper planes, have been acquired).” (emphasis added) ([0013]); see also [0051] describing that the background measurement dictates “three images” and the background measurement is automatically started after acquiring the third image).
With respect to claim 36, AASE teaches checking, by the processor, results of the quantification for viability and discarding the results of the quantification when the results are not viable. (see, e.g., “In the case that an automatic measurement is generated with low confidence or cannot be generated, a notification may be output via the interface 200, which may prompt the ultrasound operator or clinician to reacquire one or more ultrasound images (e.g., due to the initial image being low quality) or acquire an image needed for the automatic measurement that has yet to be acquired. “ ([0039]). “[I]f the background measurement cannot be performed or if the result of the background measurement is not rational or plausible, the background measurement may not be determined to have been performed with confidence.” ([0053]; see also [0043]: “…(images that are not saved by the operator may be temporarily stored in a first-in first-out buffer and thus eventually be automatically deleted)”).
NOTE: The broadest reasonable interpretation of “discarded” that is consistent with the specification includes rejecting the results for not being viable. Applicant does not clearly define “discarded.” The term is used only once in the disclosure and it does not provide a clear meaning: “The results of the quantification algorithm are then checked for viability, and if considered viable, are kept, or else they are discarded.” ([0056] of Applicant’s disclosure). Contrast to the term deletion, which is described as a positive act by the user throughout the disclosure. (original claim 5, [0009], [0050], [0051], [0054]). One definition of “discard” is “[t]o reject as being no longer wanted or needed; to cast aside, get rid of; to abandon.” Oxford University Press. (n.d.). Discard, v. In Oxford English dictionary. Retrieved May 26, 2026, from https://doi.org/10.1093/OED/3979832178. Rejecting does not necessarily include “deleting.” For example, the system may include the results stored somewhere without reporting the results to the user, as disclosed in AASE.
With respect to claim 37, AASE teaches a system for automated ultrasound quantification (see, e.g., Figure 1 and [0027], [0029] describing that the ultrasound imaging system 100 includes a probe 106 and an automatic measurement system 124). The system includes:
an ultrasound transducer probe configured to obtain a plurality of ultrasound images while a user is performing a live ultrasound scan of a subject (see, e.g., [0023] describing data from probe 106 is processed into images in “real-time”);
a processor (see, e.g., “Automatic measurement system 124 may be a computing device having resources (e.g., memory, processors) allocated to performing automated measurements on ultrasound images.” ([0029]) configured to:
continuously provide the plurality of ultrasound images to a quantification pipeline during the live ultrasound scan (see, e.g., “During a patient exam workflow where ultrasound images of a patient are acquired, the automatic measurement system 124 may perform a variety of tasks in order to generate the automatic measurements. This may include obtaining images from the computing device 122 and/or image archive 128, performing one or more processing algorithms (e.g., identifying the scan plane of each saved image, segmenting one or more anatomical features in one or more ultrasound images, measuring features within and/or across one or more ultrasound images)….” (emphasis added) ([0030]). “Some measurements performed by the automatic measurement system 124 may be performed in the background while images of the patient are still being obtained by the ultrasound system, with little to no input from the user.” (emphasis added) ([0029]));
select an ultrasound image subset from the plurality of ultrasound images without requiring a user input to select the ultrasound image subset (see, e.g., “The automatic measurements may be performed while the scanner (e.g., ultrasound imaging system) is being utilized to obtain the ultrasound images as part of the patient exam, for example the performance of the automatic measurements may be triggered by a determination that each ultrasound image dictated by the automatic measurement has been obtained (e.g., the proper anatomical features, in the proper planes, have been acquired).” (emphasis added) ([0013]); see also [0051] describing that the background measurement dictates “three images” and the background measurement is automatically started after acquiring the third image).
automatically perform quantification associated with an anatomy of the subject using the ultrasound image subset, wherein the quantification is performed without requiring a user input to activate the quantification (see, e.g., “Thus, according to embodiments disclosed herein, one or more automatic ultrasound measurements may be performed on ultrasound images acquired as part of a patient exam in the background while active imaging is still ongoing, such that measurements are performed automatically without user input.” (emphasis added) ([0013]));
output, to a display, a notification for the user that the quantification is available for review while the display is in a live scan mode (see, e.g., “[M]ethod 300 includes at 326 displaying a notification that the background measurement is available. Because image acquisition of the patient exam is still ongoing, the user may wish to wait until the exam workflow is complete/all images and/or cine loops have been acquired before reviewing the measurement, and thus the notification that the background measurement is available may allow the user to decide when the background measurement result should be reviewed.” (emphasis added) ([0055]; see also [0029])).
While AASE explicitly teaches enabling the user to select ultrasound images for subsequent background measurements and suggests that some tasks may be automated (see, e.g., [0030]: “At least some of the tasks may be initiated without operator intervention, and may be triggered by certain conditions, such as particular anatomical features being visible in the ultrasound images.”), AASE does not explicitly teach automatically selecting, by the processor, an ultrasound image subset from the plurality of ultrasound images without requiring a user input to select the ultrasound image subset.
In the same field of endeavor, NEUMANN is concerned with automating processes for a technician during an ultrasound scan. “During the scan, the sonographer manually selects ‘key images,’ which are then stored with an identified view label (e.g., ‘left kidney transversal’, ‘common bile duct’, . . . ). For some of these key images, the sonographer further measures certain quantities of interest (e.g., kidney length, width, and/or height, lumen diameter, gallbladder wall thickness, . . . ). Measurement is typically done after the user found a proper view (e.g., high image quality, organ coverage, . . . ), then ‘freezes’ the image, and finally places measurement calipers on the image. Key image selection, view identification, image ‘freezing’, and measurement caliper placement are done manually on-the-fly while scanning. These manual steps are prone to user error and may be inconsistent across sonographers (e.g., typos, different abbreviations, inter-user variability). The quality of the results depends on user experience and skill. Given the number of images, the overall process can be inefficient and tedious.” ([0002]).
To address these issues, NEUMANN teaches automatically classifying views during the ultrasound scan and performing measurements of those views. “A patient is scanned with the ultrasound scanner. The scanning results in a sequence of frames of ultrasound data representing the patient. A view is classified for at least some of the frames of ultrasound data of the sequence. The classified views from multiple of the frames of ultrasound data are gated, where the gating of the classified views rejects a first one of the classified views and passes a second one of the classified views. Structure in the frames of ultrasound data are automatically measured. The frames where measurement is performed are the frames that pass the gating of the classified views.” ([0005]).
With respect to the automatic selection, by a processor, when a predetermined number of ultrasound images have been provided, NEUMANN specifically discusses how manual selection of images can be prone to error. “Key image selection, view identification, image “freezing”, and measurement caliper placement are done manually on-the-fly while scanning. These manual steps are prone to user error and may be inconsistent across sonographers (e.g., typos, different abbreviations, inter-user variability).” ([0002]). NEUMANN then specifically teaches that manual selection can be avoided. “Manually selecting a key view can be replaced by filtered view classification results. Manually ‘freezing’ a frame may be replaced by combination of multiple filtering modules (e.g. view classification filters+measurement-filters (image quality, stability, . . . )).” ([0017]). It is clear that NEUMANN teaches an automatic selection process. “Bad results that may have been produced are filtered out before the user sees them. ‘Flickering’ (e.g., frequent change of view labels in case a classifier cannot properly differentiate one organ from another) is avoided by exploiting temporal and/or other information.” ([0017]).
As to NEUMANN’s workflow, NEUMANN teaches acquiring a real-time stream of ultrasound frames. ([0022] and [0029]). NEUMANN teaches evaluating multiple frames within a temporal window, ([0046] and [0048]), in which some views are accepted by a classification gate and others are rejected. ([0045] and [0048]). NEUMANN’s “temporal filtering approach” teaches that automatic selection occurs when a predetermined number of ultrasound images have been provided. “The view label vt is stored in the FIFO memory queue with capacity n. If the FIFO memory is at capacity (n) and at least m (m≤n) of the stored labels are identical (e.g. all of them are “spleen long”), then the gate is opened, and vt (e.g., “spleen long”) is output.” ([0048]). NEUMANN also teaches a “sequence-level classification….” ([0051]). “If the memory is at capacity, all n images currently in memory are combined (e.g. concatenated). The view classification algorithm is run on the combined image.” ([0052]).
After reaching the predetermined number of images, measurements are performed on the accepted views. ([0029] and [0037]). “The results from the classification (i.e., view label) are passed to one or more (up to Nview) filter modules 24, which apply gates 25 to pass or reject the view label. The view labels that pass the filters 24 and gates 25 are displayed 29A. The measurement algorithm 28 is applied to images 22 associated with passed view labels.” ([0029]). See also [0037]: “The frames on which the measurement is performed may be only the frames with a view passing the limitation of act 17, all frames, or another sub-set of frames.”
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE processor to automatically select, when a predetermined number of ultrasound images have been provided, the images that will be measured in the background. One of ordinary skill in the art would have been motivated to configure the AASE system to further automate tasks (as suggested in AASE) to avoid inconsistent measurements, as taught by NEUMANN, and enhance the user-friendly experience for the technician. There would have been a reasonable expectation of success as NEUMANN teaches that classifying (i.e., selecting) images from the ultrasound scan for further measurement can be performed during a live scan.
Nevertheless, Applicant alleges that NEUMANN does not teach automatically selecting the ultrasound images after a predetermined number of ultrasound images have been provided. (see page 14 of Response). While Examiner disagrees, the motivation to make this modification is also found elsewhere. For example, SMISTAD teaches a real-time automatic ejection fraction and foreshortening detection system. (Title and Abstract). SMISTAD specifically addresses the challenges to acquire ultrasonic measurements during echocardiograms. “Volume and ejection fraction (EF) measurements of the left ventricle (LV) in 2-D echocardiography are associated with a high uncertainty not only due to interobserver variability of the manual measurement, but also due to ultrasound acquisition errors such as apical foreshortening.” (Abstract). To address these challenges, SMISTAD teaches a “real-time pipeline [that] allows for a continuous acquisition and measurement workflow without user interaction, and has the potential to significantly reduce the time spent on the analysis and measurement error due to foreshortening, while providing quantitative volume measurements in the everyday echo lab.” (Abstract).
For the SMISTAD pipeline, “[t]he different processing steps are spread out over four main threads to enable concurrent processing in real time.” (p.2598, right column, G. Real-Time Pipeline). See Figure 3 of SMISTAD here. Thread 1 outputs images to Thread 2 for classification. (Id). Thread 2 classifies the images and outputs A4C and A2C images to different image buffers. (Id). Thread 3 searches these image buffers for ED and ES frames. Each new frame is automatically processed and A4C/A2C images are added to the buffers. (p.2598, right column, G. Real-Time Pipeline). Prior to moving to Thread 4, however, SMISTAD teaches that the network requires a predetermined number of ultrasound images in order to accurately predict end-diastole (ED) and end-systole (ES). “Based on our experience, this network needs a sequence of sufficient length to accurately predict ED and ES, thus the minimum of N was set to 30 frames.” (p.2597, right column, top paragraph). The maximum number for N was set to 80 frames. (Id). In SMISTAD’s specific example, Thread 3 “first runs the timing network on the most recent 80 frames, or less, of the current image view buffer. If the timing algorithm is able to find an ED and ES frame, the segmentation network is executed for both these two frames and the result is stored.” (p.2598, right column, G. Real-Time Pipeline). Lastly, Thread 4 uses the ED and ES ultrasound images to perform the measurements. (pp.2598-2599, right column, G. Real-Time Pipeline).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify AASE’s automatic measurement method to include SMISTAD’s live-frame buffering and automatic selection technique. One having ordinary skill in the art would have been motivated to use the classification and buffering technique, as taught in SMISTAD, because it would automate the process of selecting the images to be measured and provide reliable measurements. There would have been a reasonable expectation of success as SMISTAD teaches that the automatic selection technique can be used during an echocardiogram.
With respect to claim 39 (and in light of the Section 112(b) rejection above), the procedures performed by AASE as modified by NEUMANN or SMISTAD would necessarily include the quantification requiring execution of at least one million instructions. (see MPEP 2112). Firstly, AASE describes many of the same echocardiographic procedures and measurements as described in Applicant’s disclosure. (e.g., compare [0004] of Applicant’s disclosure and [0012] of AASE). Segmenting is an example of computationally intensive process and acquiring measurements can require segmentation. While AASE does not explicitly teach that the quantification requires at least a million instructions, one having ordinary skill in the art would understand that the recited processes would require executing at least a million instructions. (see, e.g., Awasthi et al. “LVNet: Lightweight model for left ventricle segmentation for short axis views in echocardiographic imaging.” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control 69.6 (2022): 2115-2128 (hereinafter AWASTHI). AWASTHI describes computations that are necessary for LV segmentation during cardiac ultrasounds. (Abstract). Table II compares “the floating point operations required for running a single instance of the model.” The smallest number in Table II is 1.56 gigaflops (or 1.56 billion operations).
Accordingly, the modified AASE system would necessarily execute at least one million instructions when performing the quantification. (see MPEP 2112, IV).
Claims 22 and 23 are rejected are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2023/0255602 A1 (hereinafter “AASE”) and U.S. Patent Appl. Publ. No. 2022/0370046 (hereinafter “NEUMANN”) or, alternatively, Smistad et al. “Real-time automatic ejection fraction and foreshortening detection using deep learning.” IEEE transactions on ultrasonics, ferroelectrics, and frequency control 67.12 (2020): 2595-2604 (hereinafter “SMISTAD”) as applied to claim 19 above, and further in view of Edge User Guide, FUJIFILM Sonosite, P15200-04 06/2019, 2019 (hereinafter “EDGE USER GUIDE”).
With respect to claim 22 (depending from claim 19), AASE does not explicitly teach that receiving, by the processor, a user input to delete the quantification and deleting, by the processor, the quantification in response to the user input. However, AASE teaches that the use may choose to not accept a background measurement. “After the measurements are presented to the user, the user may confirm whether each measurement should be saved in the patient exam, e.g., the user may indicate if one or more of the results is accurate, based on expertise. If a measurement is not deemed to be accurate, the user may decide not to confirm the results and the results may not be saved in the patient exam.” ([0044] of AASE).
In the same field of endeavor, EDGE USER GUIDE teaches how to operate the portable Edge ultrasound system. “The Edge ultrasound system is a portable, software-controlled device using all-digital architecture. The system has multiple configurations and feature sets used to acquire and display high-resolution, real-time ultrasound images.” (page 1, Chapter 1: Getting Started). The system is capable of performing cardiac imaging applications. “You can assess the heart, cardiac valves, great vessels, surrounding anatomical structures, overall cardiac performance, and heart size for the presence or absence of pathology.” (page 13, Chapter 1: Getting Started). “You can perform basic measurements in any imaging mode and can save the image with the measurements displayed.” (page 43, Chapter 4: Measurements and Calculations). At least for one function, EDGE USER GUIDE teaches enabling the user to delete a measurement. “To delete, select Delete on-screen.” (Id). EDGE USER GUIDE also teaches, for general calculations, one can “can display, repeat, and delete measurements from a calculation. Some measurements can be deleted directly from the patient report pages.” (page 47, Chapter 4: Measurements and Calculations) (see also page 48 teaching how to “delete a saved measurement”).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system so that the user could, upon entering an input, delete the quantification analysis, as taught in EDGE USER GUIDE. One of ordinary skill in the art would have been motivated to add this feature because AASE teaches that measurements may not be confirmed by the user or saved. One would have wanted to include the ability to delete measurements that the user does not confirm and to avoid confusion for those who later review the report. There would have been a reasonable expectation of success as user interfaces often display options in the form of icons that enable a user to delete a measurement, as taught in EDGE USER GUIDE.
With respect to claim 23 (depending from claim 19), AASE does not explicitly teach receiving, by the processor, a user input to return to the live scan mode and transitioning, by the processor, the display from the review mode to the live scan mode. However, AASE suggests that the user may return to live imaging after approving a background measurement. “The user may view the results of the background measurement at any point during or after the exam, e.g., by selecting the control button/link displayed as part of the notification that the background measurement is available to review.” ([0055]) see also [0060]: “While the background measurements are performed during the exam while images of the patient are still being acquired, the user may choose to review the results of the background measurement(s) at any suitable time once the results are available.”)
In the same field of endeavor, EDGE USER GUIDE teaches how to operate the portable Edge ultrasound system. “The Edge ultrasound system is a portable, software-controlled device using all-digital architecture. The system has multiple configurations and feature sets used to acquire and display high-resolution, real-time ultrasound images.” (page 1, Chapter 1: Getting Started). The system is capable of performing cardiac imaging applications. “You can assess the heart, cardiac valves, great vessels, surrounding anatomical structures, overall cardiac performance, and heart size for the presence or absence of pathology.” (page 13, Chapter 1: Getting Started). During an exam, the technician populates a “patient report” with calculation results. (page 72, Chapter 4: Measurements and Calculations). “You can display the patient report at any time during the exam.” (emphasis added) (Id). After displaying the patient report, “[t]o exit the patient report and return to imaging, select Done.” (emphasis added) (Id).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system so that the user could, upon entering an input, return to performing the ultrasound of the subject, as taught in EDGE USER GUIDE. One of ordinary skill in the art would have been motivated to add this feature because AASE teaches the ability to review background measurements at any time during an exam, which include in the middle of examination. One would have wanted to include a user input so that the user could return to the exam. There would have been a reasonable expectation of success as user interfaces often display options in the form of icons that enable a user to return to a previous task, as taught in EDGE USER GUIDE.
Claims 20 and 29 are rejected are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2023/0255602 A1 (hereinafter “AASE”) and U.S. Patent Appl. Publ. No. 2022/0370046 (hereinafter “NEUMANN”) or, alternatively, Smistad et al. “Real-time automatic ejection fraction and foreshortening detection using deep learning.” IEEE transactions on ultrasonics, ferroelectrics, and frequency control 67.12 (2020): 2595-2604 (hereinafter “SMISTAD”) as applied to claims 16 and 19 above, and further in view of U.S. Patent Appl. Publ. No. 2019/0130554 A1 (hereinafter “ROTHBERG”).
With respect to claim 20 (depending from claim 19), AASE does not explicitly teach wherein the review mode displays the quantification as one or more labeled images.
PNG
media_image2.png
766
590
media_image2.png
Greyscale
In the same field of endeavor, ROTHBERG teaches systems and methods for automatically performing measurements during ultrasound imaging. (Title and Abstract). Figure 20 illustrates an example of a GUI that is displayed to the user following an automatic measurement. ([0102]). “[T]he GUI 1300 includes a segmentation 2016 performed during the automatic measurement. For example, the segmentation 2016 in FIG. 20 may constitute identification of two landmark points, namely the endpoints of the long axis of the long ventricle in an image taken in parasternal long axis view of the heart, and formation of a line between the two landmark points.” ([0102]). “The GUI 1300 further includes a result 2018 of the automatic measurement (in FIG. 20, an ejection fraction value, 60%) and the offline quality value 2020 calculated on the sequence of images (in FIG. 20, 75%). The GUI 1300 further shows an edit button 2022, selection of which may cause the computing device to enter a manual mode from which an operator may annotate/segment image(s) in the sequence of images, correct/alter automatically performed segmentations (e.g., segmentation 2016) and thereby manually perform measurements based on the sequence of images.” ([0102]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system so that the system displays to the user the quantification analysis as one or more labeled images, as taught in ROTHBERG. One of ordinary skill in the art would have been motivated to display the quantification analysis on labeled images so that the user could determine whether to save the measurement, obtain a new automatically acquired measurement, or delete the measurement and enter manual mode to obtain a new one. There would have been a reasonable expectation of success as user interfaces can be programmed to display different graphical objects with ultrasound images, as taught in ROTHBERG.
With respect to claim 29, AASE does not explicitly teach that automatically notifying the user via a user interface that a quantification of the provided plurality of ultrasound images is available for review further comprises wherein the notification comprises an indicator of a quality of the quantification.
In the same field of endeavor, ROTHBERG teaches systems and methods for automatically performing measurements during ultrasound imaging. (Title and Abstract). As discussed above, Figure 20 illustrates an example of a GUI that is displayed to the user following an automatic measurement. ([0102]). ROTHBERG teaches displaying the result of the automatic measurement along with a quality value. In Figure 20, the result is the ejection fraction (EF) at 60% and the quality value (QC) at 75%.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system so that the system displays an indicator of confidence in or quality of the quantification analysis, as taught in ROTHBERG. One of ordinary skill in the art would have been motivated to display the confidence in or quality of the quantification analysis so that the user could determine whether to save the measurement, obtain a new automatically acquired measurement, or delete the measurement and enter manual mode to obtain a new one. There would have been a reasonable expectation of success as user interfaces can be programmed to display different graphical objects with ultrasound images, as taught in ROTHBERG.
Claim 31 is rejected are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2023/0255602 A1 (hereinafter “AASE”) and U.S. Patent Appl. Publ. No. 2022/0370046 (hereinafter “NEUMANN”) or, alternatively, Smistad et al. “Real-time automatic ejection fraction and foreshortening detection using deep learning.” IEEE transactions on ultrasonics, ferroelectrics, and frequency control 67.12 (2020): 2595-2604 (hereinafter “SMISTAD”) as applied to claim 16 above, and further in view of Lane, Elisabeth S., et al. “Multibeat echocardiographic phase detection using deep neural networks.” Computers in Biology and Medicine 133 (2021): 104373 (hereinafter “LANE”).
With respect to claim 31, AASE does not explicitly teach that automatically selecting the ultrasound image subset comprises identifying a plurality of cardiac beats in the plurality of ultrasound images, wherein the ultrasound image subset comprises ultrasound images representative of the plurality of cardiac beats.
In the same field of endeavor, LANE teaches using deep neural networks to create an “automated model [that] can identify multiple end-systolic and end-diastolic frames in echocardiographic videos of arbitrary length….” (Abstract). LANE emphasizes that longer recordings would reduce the importance of information from a single cardiac cycle and allow for better quantification. (see, e.g., discussion beginning at p.1, Section 1.1). “In clinical practice, longer recordings would allow for probing of physiological reactions after intervention, where detecting a subtle change in the mean value of a clinical maker, amongst much larger background beat-to-beat variability, is essential.” (p.2, Section 1.3, left column). “Measurements taken from only one heartbeat may result in test-retest variability. Therefore, it would be impossible to reliably conclude whether a patient’s condition has deteriorated over time. Such variability and inaccuracy can be reduced by averaging measurements over several heartbeats, from the same acquisition.” (pp.2-3, Section 1.3, left column).
LANE’s proposed system identifying a plurality of cardiac beats in the plurality of ultrasound images. “[T]he proposed model’s removal of all pre-processing steps and its capacity to identify multiple heartbeats in one long video is, however, an indisputable advantage.” (p.5, Section 3.1, right column). “The proposed framework was tested on A4C views; however, it is the authors’ belief that the utilised deep learning approaches could be applied to other echocardiographic views (no view-specific assumptions were made during the model developments).” (p.8, Conclusion, left column).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system to include identifying a plurality of cardiac beats in the plurality of ultrasound images, wherein the ultrasound image subset comprises ultrasound images representative of the plurality of cardiac beats. One would have been motivated to identify and store ultrasound image data sets with multiple cardiac beats in order to improve quantification of cardiac parameters as taught in LANE. (see, e.g., p.1, I. Introduction (left column) and p.2, Section 1.1 (left column)). There would have been a reasonable expectation of success as LANE teaches an automated model can be generated to identify ED and ES from ultrasound image data.
Claims 27 and 28 are rejected are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2023/0255602 A1 (hereinafter “AASE”) and U.S. Patent Appl. Publ. No. 2022/0370046 (hereinafter “NEUMANN”) or, alternatively, Smistad et al. “Real-time automatic ejection fraction and foreshortening detection using deep learning.” IEEE transactions on ultrasonics, ferroelectrics, and frequency control 67.12 (2020): 2595-2604 (hereinafter “SMISTAD”) as applied to claim 16 above, and further in view of a translation of JP 2021083955 A (hereinafter “CANON”).
With respect to claim 27, AASE does not explicitly teach wherein the notification comprises a number indicating how many cardiac beats have been quantified, and wherein the number is continuously updated during the live ultrasound scan.
In the same field of endeavor, CANON concerns an ultrasonic imaging system for acquiring measurements during echocardiographic examinations. (p.1, lines 21-26). “Examples of methods for measuring and analyzing ultrasonic images include two-dimensional WMT (Wall Motion Tracking) and three-dimensional WMT that analyze the wall motion of the myocardium, and Auto_EF (Automated Ejection) that automatically calculates the left ventricular ejection fraction. Fraction) and the like.” (p.1, lines 24-26). “Auto_EF performs pattern recognition by comparing the actual morphology of the heart with the features registered in the pre-constructed database (appearance of the heart, left ventricular endocardium, etc.), and searches for a heart having a similar pattern. This is a method of detecting the endocardium of the left ventricle and calculating the left ventricular volume at the end of diastole (EDV), the left ventricular volume at the end of systole (ESV), the ejection fraction of the left ventricle (EF), etc. as quantitative values of each heartbeat.” (p.1, lines 28-32).
CANON teaches acquiring and quantifying, in the background, image data that includes multiple heartbeats. “The processing function 176 is a function of executing a process of acquiring a quantitative value from a plurality of frames of ultrasonic image data saved by the memory control function 174 according to a save instruction of the reception function 173 in the background of live display by the display control function 172. including. For example, the processing function 176 executes a process of acquiring a quantitative value in each heartbeat from a plurality of ultrasonic image data over a plurality of heartbeats.” (p.7, lines 47-51).
CANON teaches a user interface that includes a “start button Q” wherein a user can select which type of examination to perform. “The display control function 172 causes the display 40 to display a user interface for accepting the type of cross section to be processed, for example, a ‘4ch (chamber)’ button, a ‘2ch’ button, and a ‘3ch’ button as an element of the start button Q.” (p.9, lines 9-12). In response to the user selecting the type of cross section, the system “starts a process of acquiring a quantitative value of each heartbeat in the background of the live display by the display control function 172 according to the processing start instruction received in step ST7 (step ST7). That is, the quantitative value is acquired in parallel with the live display by the ultrasonic scan.” (p.9, lines 19-22). The system processes each heartbeat from a series of heartbeats. “Here, when a plurality of heartbeats (for example, 4 heartbeats) are set as processing targets in advance, a quantitative value may be acquired for each heartbeat of the plurality of heartbeats, or the plurality of heartbeats may be acquired via the input interface 30.” (p.9, lines 226-28).
PNG
media_image3.png
565
454
media_image3.png
Greyscale
Notably, CANON teaches displaying a number that indicates how many cardiac beats have been quantified and that number is continuously updated during the live ultrasound scan. “The display control function 172 calculates the progress (%) of the process started in step ST7, and displays the progress of the process on the display 40 (step ST8).” (p.9, lines 34-35). CANON teaches displaying a “degree of progress, which is shown as “XX.X%” in Figure 6. “Here, ‘XX.X’ means a numerical value of ‘0.00’ to ‘100.0’. Over time, the progress of Auto_ef will increase.” (p.9, lines 44-46) (see also p.11, lines 42-60, describing the different variations: “Further, for example, when the quantitative value is acquired for each of the plurality of heartbeats, the display control function 172 displays the quantitative value in blue on the thumbnail image in the order in which the acquisition of the quantitative value is completed, and acquires the quantitative value from the latest heartbeat.”).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system to display a number that indicates how many cardiac beats have been quantified and that number is continuously updated during the live ultrasound scan. One of ordinary skill in the art would have been motivated to provide a user-friendly interface so that the user knows how many of the heartbeats have been processed. There would have been a reasonable expectation of success as CANON teaches that an ultrasound system can update the user as to the progress of the processing during the ultrasound exam.
With respect to claim 28, AASE does not explicitly teach receiving, by the processor, a user input selecting the notification, wherein the number resets to zero in response to the user input.
CANON teaches notifying the user of the degree of progress (Tm in Figure 6) and notifying the user of “a representative value U as a processing result.” (p.10, line29; see also Figure 7). “In FIG. 7, the representative value U includes ‘58.8%’ of the left ventricular ejection fraction (EF) based on the quantitative values of multiple heartbeats measured by Auto_ef.” (p.10, lines 32-33). The representative value U is in the same position as the information Tm for the degree of progress. (Compare Figures 6 and 7). CANON further teaches enabling the user to approve or disapprove of the calculated representative value. “[T]he operator can confirm the validity of the quantitative value, correct the contour line, reset the heartbeat to be processed, perform the approval operation, and the like.” (p.12, lines 13-16). Accordingly, the notification that begins with the progress of the processing (i.e., number indicating how many cardiac beats have been quantified) transitions to a representative value that can be approved or disapproved.
While CANON does not teach explicitly transitioning from disapproving a measurement to a new analysis, thereby resetting the progress to zero, the very purpose of the examination is to obtain measurements. If the technician is not satisfied by a measurement, the next step would be to begin the process again until the technician is satisfied with the measurement. (see, e.g., AASE at [0060]: “…and then the user may subsequently edit and/or approve the background measurement”).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system to enable a user to disapprove the representative value by restarting the analysis process of a series of heartbeats (e.g., the notification would include queries “Accept?” or “Take new measurement?” or “Restart measuring process?”). One of ordinary skill in the art would have been motivated to configure the system to allow someone to restart the process in order to acquire better measurements. There would have been a reasonable expectation of success as CANON teaches that an ultrasound system can update the user as to the progress of the processing during the ultrasound exam.
Claim 32 are rejected are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2023/0255602 A1 (hereinafter “AASE”) and U.S. Patent Appl. Publ. No. 2022/0370046 (hereinafter “NEUMANN”) or, alternatively, Smistad et al. “Real-time automatic ejection fraction and foreshortening detection using deep learning.” IEEE transactions on ultrasonics, ferroelectrics, and frequency control 67.12 (2020): 2595-2604 (hereinafter “SMISTAD”), and Lane, Elisabeth S., et al. “Multibeat echocardiographic phase detection using deep neural networks.” Computers in Biology and Medicine 133 (2021): 104373 (hereinafter “LANE”) as applied to claim 31 above, and further in view of a translation of JP 2021083955 A (hereinafter “CANON”).
With respect to claim 32, AASE does not explicitly teach outputting to the display, by the processor, a progress indicator indicating a percentage of the plurality of cardiac beats that has been obtained.
As discussed above with respect to claim 27, CANON teaches displaying a number that indicates how many cardiac beats have been quantified and that number is continuously updated during the live ultrasound scan. “The display control function 172 calculates the progress (%) of the process started in step ST7, and displays the progress of the process on the display 40 (step ST8).” (p.9, lines 34-35). CANON teaches displaying a “degree of progress, which is shown as “XX.X%” in Figure 6. “Here, ‘XX.X’ means a numerical value of ‘0.00’ to ‘100.0’. Over time, the progress of Auto_ef will increase.” (p.9, lines 44-46).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the AASE system to display a progress indicator indicating a percentage of the plurality of cardiac beats that has been obtained. One of ordinary skill in the art would have been motivated to provide a user-friendly interface so that the user knows how many of the heartbeats have been processed. There would have been a reasonable expectation of success as CANON teaches that an ultrasound system can update the user as to the progress of the processing during the ultrasound exam.
Claim 38 is rejected are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2023/0255602 A1 (hereinafter “AASE”) and U.S. Patent Appl. Publ. No. 2022/0370046 (hereinafter “NEUMANN”) or, alternatively, Smistad et al. “Real-time automatic ejection fraction and foreshortening detection using deep learning.” IEEE transactions on ultrasonics, ferroelectrics, and frequency control 67.12 (2020): 2595-2604 (hereinafter “SMISTAD”) as applied to claim 16 above, and further in view of U.S. Patent Appl. Publ. No. 2021/0000449 A1 (hereinafter “DEO”).
Neither AASE nor NEUMANN nor SMISTAD does not explicitly teach that the predetermined number of ultrasound images comprises a number of ultrasound images covering at least two cardiac beats. Nevertheless, the ultrasound images are clearly acquired continuously over an extended period of time as the images are buffered.
In the same field of endeavor, DEO teaches that an echocardiogram is “typically a collection of up to 70 videos of the heart taken over multiple cardiac cycles and focusing on different viewpoints, requiring ˜45 min by a skilled sonographer.” ([0026]). DEO suggests using automated computer vision for analyzing the images. “Automated image interpretation falls under the discipline of computer vision, which is a branch of machine learning where computers learn to mimic human vision.” ([0029]). DEO notes that there is variability in the cardiac cycles from the image data. “Given the extent of this variability and the sheer amount of multidimensional information in each study that often goes unused, embodiments can use an automated learning approach to assist human interpretation. In some embodiments, beat-to-beat variability can be addressed by averaging across multiple heart beats, such as every heartbeat.” ([0029]). “The corresponding segmented images can be used to compute standard measurements of cardiac structure, e.g., averaging across multiple cardiac cycles within a video and multiple videos for a given view.” ([0038]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify NEUMANN or SMISTAD to acquire a number of ultrasound images that cover at least two cardiac beats, as taught in DEO. One would be motivated to acquire data over multiple cardiac beats in order to be able to average the data of multiple cardiac beats and address beat-to-beat variability, as taught in DEO. There would have been a reasonable expectation of success as DEO describes the same echo procedures performed by AASE.
RESPONSE TO APPLICANT’S ARGUMENTS:
Applicant’s arguments with respect to the Section 103 rejections based on AASE and NEUMANN have been considered but are not persuasive. Applicant specifically argues that NEUMANN does not teach automatic selection, by a processor, when a predetermined number of ultrasound images have been provided.
Examiner disagrees. As explained above, NEUMANN teaches acquiring a real-time stream of ultrasound frames. ([0022] and [0029]). NEUMANN teaches evaluating multiple frames within a temporal window, ([0046] and [0048]), in which some views are accepted by a classification gate and others are rejected. ([0045] and [0048]). NEUMANN’s “temporal filtering approach” teaches that automatic selection occurs when a predetermined number of ultrasound images have been provided. “The view label vt is stored in the FIFO memory queue with capacity n. If the FIFO memory is at capacity (n) and at least m (m≤n) of the stored labels are identical (e.g. all of them are “spleen long”), then the gate is opened, and vt (e.g., “spleen long”) is output.” ([0048]). NEUMANN also teaches a “sequence-level classification….” ([0051]). “If the memory is at capacity, all n images currently in memory are combined (e.g. concatenated). The view classification algorithm is run on the combined image.” ([0052]).
After reaching the predetermined number of images, measurements are performed on the accepted views. ([0029] and [0037]). “The results from the classification (i.e., view label) are passed to one or more (up to Nview) filter modules 24, which apply gates 25 to pass or reject the view label. The view labels that pass the filters 24 and gates 25 are displayed 29A. The measurement algorithm 28 is applied to images 22 associated with passed view labels.” ([0029]). See also [0037]: “The frames on which the measurement is performed may be only the frames with a view passing the limitation of act 17, all frames, or another sub-set of frames.”
Nevertheless, Examiner has provided an alternative rejection based on SMISTAD. SMISTAD teaches a real-time automatic ejection fraction and foreshortening detection system. (Title and Abstract). SMISTAD specifically addresses the challenges to acquire ultrasonic measurements during echocardiograms. “Volume and ejection fraction (EF) measurements of the left ventricle (LV) in 2-D echocardiography are associated with a high uncertainty not only due to interobserver variability of the manual measurement, but also due to ultrasound acquisition errors such as apical foreshortening.” (Abstract). To address these challenges, SMISTAD teaches a “real-time pipeline [that] allows for a continuous acquisition and measurement workflow without user interaction, and has the potential to significantly reduce the time spent on the analysis and measurement error due to foreshortening, while providing quantitative volume measurements in the everyday echo lab.” (Abstract).
For the SMISTAD pipeline, “[t]he different processing steps are spread out over four main threads to enable concurrent processing in real time.” (p.2598, right column, G. Real-Time Pipeline). See Figure 3 of SMISTAD here. Thread 1 outputs images to Thread 2 for classification. (Id). Thread 2 classifies the images and outputs A4C and A2C images to different image buffers. (Id). Thread 3 searches these image buffers for ED and ES frames. Each new frame is automatically processed and A4C/A2C images are added to the buffers. (p.2598, right column, G. Real-Time Pipeline). Prior to moving to Thread 4, however, SMISTAD teaches that the network requires a predetermined number of ultrasound images in order to accurately predict end-diastole (ED) and end-systole (ES). “Based on our experience, this network needs a sequence of sufficient length to accurately predict ED and ES, thus the minimum of N was set to 30 frames.” (p.2597, right column, top paragraph). The maximum number for N was set to 80 frames. (Id). In SMISTAD’s specific example, Thread 3 “first runs the timing network on the most recent 80 frames, or less, of the current image view buffer. If the timing algorithm is able to find an ED and ES frame, the segmentation network is executed for both these two frames and the result is stored.” (p.2598, right column, G. Real-Time Pipeline). Lastly, Thread 4 uses the ED and ES ultrasound images to perform the measurements. (pp.2598-2599, right column, G. Real-Time Pipeline).
Prior Art Made of Record
Prior art that is made of record but not relied upon in the current rejections includes the following:
US-20170153321-A1 teaches a method that includes “automatically obtaining an elasticity image by using a preset number of consistent and consecutive images and providing the obtained elasticity image.” (Abstract).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON P GROSS whose telephone number is (571)272-1386. The examiner can normally be reached Monday-Friday 9:00-5:00CT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne M. Kozak can be reached at (571) 270-5284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON P GROSS/Examiner, Art Unit 3797
/SERKAN AKAR/Primary Examiner, Art Unit 3797
Prosecution Insights