DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 11/17/2025 have been fully considered but they are not persuasive.
Applicant argues, see Applicant’s arguments pages 9-11, that the combination of Canfield and Berkey does not teach the full scope of the independent claims and comprises hindsight reasoning. Although Examiner agrees with Applicant’s assertion that [0054] of Berkey teaches that the user can select which anatomy to label, this allows the user to focus on one or more of the plurality of anatomical structures anticipated to be present based on the view. The method does not depend on this user selection step; by presenting the menu of anticipated landmark anatomical structures to the user, Berkey shows that a determination of which structures are expected to appear in a view is made. Nevertheless, [0052] of Berkey teaches that the labeling is applied automatically. Thus, Examiner upholds the combination of Canfield and Berkey to teach this limitation.
Applicant argues, see Applicant’s argument pages 11-12, that Solem does not teach the claimed temporal smoothing of the average position of the labels. Whether the images used for the determination of the average position in Solem are training images of a plurality of human faces, as Applicant argues, is irrelevant. Solem receives a plurality of image frames and determines the average position of a given structure; this is analogous to what is performed in the instant application. Additionally, it is also irrelevant that the average location of the facial feature is determined for a future image. The determination of the average position is made in light of the images that have already been received by the system. Regarding the “temporal smoothing”, [0029] teaches that the label of the tracked structure (landmark point 605) is iteratively adjusted based on the mean position determined from the images. This iterative adjustment comprises a temporal smoothing. Examiner upholds the use of Solem.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “ultrasound device” in claims 1, 13, & 23
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
Page 5 states “The ultrasound imaging device 110 is any ultrasound device operable to acquire ultrasound images of a patient, and may be, in at least some embodiments for example, a handheld ultrasound imaging device”.
Therefore, “ultrasound device” has been interpreted to be any device capable of obtaining ultrasound images of a patient.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4, & 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Canfield (US 2018/0103912) in view of Berkey (US 2013/0184584).
Regarding claim 1, Canfield teaches an ultrasound system, comprising:
an ultrasound imaging device (ultrasound probe 10, [0014]) configured to acquire ultrasound images of a patient ([0017]);
anatomical structure recognition and labeling circuitry (neural network model 80, [0019]) implemented at least partially by machine learning circuitry (deep learning, [0005] & [0019]), the machine learning circuitry including at least one artificial neural network (deep learning neural net models, [0019]) trained to recognize whether an ultrasound image represents a particular view of an organ or other tissue or body feature ([0019]) as prescribed by a clinical standard (two-chamber view, three-chamber view, four-chamber view, long axis view, or short axis view, [0019]), wherein the anatomical structure recognition and labeling circuitry is operable according to executable instructions (software, [0019] & [0026]) stored in a memory (digital memory, [0019]), and when the executable instructions are executed by one or more processors (computer or processor, [0023]), the anatomical structure recognition and labeling circuitry is configured to:
receive the acquired ultrasound images (heart image 100, [0019]) from the ultrasound imaging device ([0019]);
automatically recognize, by the machine learning circuity, a view of the received ultrasound images (“the neural net model was trained to identify…the view of the anatomy seen in the ultrasound image”, [0019]); and
automatically recognize, by the machine learning circuity, one or more anatomical structures in the received ultrasound images (“the neural net model was trained to identify…the anatomy in an ultrasound image”, [0019]); and
a display (touchscreen display 60, [0019]) configured to display one or more of the received ultrasound images ([0019] & Figure 2).
However, Canfield fails to disclose generating an output, by the machine learning circuitry, that automatically determines a label for an anatomical structure in the received ultrasound images, wherein the machine learning circuitry is configured to use the recognized view when generating the output, the output from the machine learning circuitry indicating the anatomical structure, and when automatically determining the label, the machine learning circuitry is restricted to a defined set of labels associated with the recognized view; automatically modifying one or more of the received ultrasound images to label the anatomical structure with the label determined from the defined set of labels; and displaying the labeled anatomical structure.
Berkey teaches:
generating an output ([0052]), by the machine learning circuitry (labeling module 12, [0045]), that automatically determines a label for an anatomical structure in the received ultrasound images (step 312, [0052]), wherein the machine learning circuitry is configured to use the recognized view when generating the output ([0052] & [0054]), the output from the machine learning circuitry indicating the anatomical structure ([0052]), and when automatically determining the label, the machine learning circuitry is restricted to a defined set of labels associated with the recognized view (menu of anticipated landmark anatomical structures to appear in the ultrasound image, [0054]);
automatically modifying one or more of the received ultrasound images to label the anatomical structure with the label determined from the defined set of labels ([0088]-[0092]); and
displaying the labeled anatomical structure ([0088]-[0092]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield to include generating an output, by the machine learning circuitry, that automatically determines a label for an anatomical structure in the received ultrasound images, wherein the machine learning circuitry is configured to use the recognized view when generating the output, the output from the machine learning circuitry indicating the anatomical structure, and when automatically determining the label, the machine learning circuitry is restricted to a defined set of labels associated with the recognized view; automatically modifying one or more of the received ultrasound images to label the anatomical structure with the label determined from the defined set of labels; and displaying the labeled anatomical structure, as taught by Berkey. The system of Canfield teaches recognizing anatomical structures, so labeling and displaying the structures communicates this information to a user, allowing the user to understand the relative anatomy of a given ultrasound image. Additionally, providing labels based on a given view simplifies operation of the system and eliminates the potential for false labeling. For example, if an image is known to be of the heart, there would be no advantages in including anatomical structures relating to the liver in the group of labels, as this can only result in structures being provided with the wrong label.
Regarding claim 2, Canfield in view of Berkey teach the ultrasound system of claim 1, and Berkey further teaches that the display is configured to display information that identifies the anatomical structure at a position in the received ultrasound images which corresponds to a position of the anatomical structure ([0088]-[0092]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield such that the display is configured to display information that identifies the anatomical structure at a position in the received ultrasound images which corresponds to a position of the anatomical structure, as taught by Berkey. This eases understanding of a patient’s relative anatomy for a user, as the labels are intuitively placed next to the anatomical structures they are labeling.
Regarding claim 4, Canfield in view of Berkey teach the ultrasound system of claim 1, and Canfield further teaches that the anatomical structure recognition and labeling circuitry is configured to automatically recognize the view of each one of the received ultrasound images before a next one of the ultrasound images is received from the ultrasound imaging device ([0009]).
Paragraph [0009] teaches that the identification of anatomy and subsequent annotation of the image takes place in real time. Paragraph [0021] and Figure 3 teaches that this annotation includes identifying the view of the image. Therefore, because this occurs in real time, this inherently teaches that a view is recognized before a next image is received.
Regarding claim 13, Canfield teaches a method, comprising:
receiving, by anatomical structure recognition and labeling circuitry (neural network model 80, [0019]), ultrasound images acquired by an ultrasound imaging device (ultrasound probe 10, [0014]) ([0017]), wherein the anatomical structure recognition and labeling circuitry is implemented at least partially by machine learning circuitry (deep learning, [0005] & [0019]), the machine learning circuitry including at least one artificial neural network (deep learning neural net models, [0019]) trained to recognize whether an ultrasound image represents a particular view of an organ or other tissue or body feature ([0019]) as prescribed by a clinical standard (two-chamber view, three-chamber view, four-chamber view, long axis view, or short axis view, [0019]);
automatically recognizing, by the machine learning circuitry, a view of the received ultrasound images (“the neural net model was trained to identify…the view of the anatomy seen in the ultrasound image”, [0019]);
automatically recognizing, by the machine learning circuitry, one or more anatomical structures in the received ultrasound images (“the neural net model was trained to identify…the anatomy in an ultrasound image”, [0019]); and
displaying the acquired ultrasound images ([0019] & Figure 2).
However, Canfield fails to disclose generating an output, by the machine learning circuitry, that automatically determines a label for an anatomical structure in the received ultrasound images, wherein the machine learning circuitry is configured to use the recognized view when generating the output, the output from the machine learning circuitry indicating the anatomical structure, and when automatically determining the label, the machine learning circuitry is restricted to a defined set of labels associated with the recognized view, wherein labels in the defined set of labels identify different anatomical structures; automatically modifying one or more of the received ultrasound images to label the anatomical structure with the label determined from the defined set of labels; and displaying one or more of the received ultrasound images with the labeled anatomical structure.
Berkey teaches:
generating an output ([0052]), by the machine learning circuitry (labeling module 12, [0045]), that automatically determines a label for an anatomical structure in the received ultrasound images (step 312, [0052]), wherein the machine learning circuitry is configured to use the recognized view when generating the output ([0052] & [0054]), the output from the machine learning circuitry indicating the anatomical structure ([0052]), and when automatically determining the label, the machine learning circuitry is restricted to a defined set of labels associated with the recognized view (menu of anticipated landmark anatomical structures to appear in the ultrasound image, [0054]), wherein labels in the defined set of labels identify different anatomical structures ([0052] & [0054]);
automatically modifying one or more of the received ultrasound images to label the anatomical structure with the label determined from the defined set of labels ([0088]-[0092]); and
displaying one or more of the received ultrasound images with the labeled anatomical structure ([0088]-[0092]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the method taught by Canfield to include generating an output, by the machine learning circuitry, that automatically determines a label for an anatomical structure in the received ultrasound images, wherein the machine learning circuitry is configured to use the recognized view when generating the output, the output from the machine learning circuitry indicating the anatomical structure, and when automatically determining the label, the machine learning circuitry is restricted to a defined set of labels associated with the recognized view, wherein labels in the defined set of labels identify different anatomical structures; automatically modifying one or more of the received ultrasound images to label the anatomical structure with the label determined from the defined set of labels; and displaying one or more of the received ultrasound images with the labeled anatomical structurel; as taught by Berkey. The system of Canfield teaches recognizing anatomical structures, so labeling and displaying the structures communicates this information to a user, allowing the user to understand the relative anatomy of a given ultrasound image. Additionally, providing labels based on a given view simplifies operation of the system and eliminates the potential for false labeling. For example, if an image is known to be of the heart, there would be no advantages in including anatomical structures relating to the liver in the group of labels, as this can only result in structures being provided with the wrong label.
Regarding claim 14, Canfield in view of Berkey teach the method of claim 13, and Berkey further teaches that the displaying the received ultrasound images with the labeled one or more anatomical structures includes displaying information that identifies the anatomical structure at a position in the received ultrasound images which corresponds to a position of the anatomical structure ([0088]-[0092]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the method taught by Canfield such that the displaying the received ultrasound images with the labeled one or more anatomical structures includes displaying information that identifies the anatomical structure at a position in the received ultrasound images which corresponds to a position of the anatomical structure, as taught by Berkey. This eases understanding of a patient’s relative anatomy for a user, as the labels are intuitively placed next to the anatomical structures they are labeling.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Canfield in view of Berkey, as applied to claim 1, above, in further view of Solem (US 2014/0355821).
Regarding claim 5, Canfield in view of Berkey teach the ultrasound system of claim 1, and Berkey further teaches that the anatomical structure recognition and labeling circuitry is further configured to:
determine a positioning of the label for the anatomical structure in each ultrasound image of a plurality of sequentially acquired ultrasound images of the patient ([0088]-[0092]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield such that the anatomical structure recognition and labeling circuitry is further configured to: determine a positioning of the label for the anatomical structure in each ultrasound image of a plurality of sequentially acquired ultrasound images of the patient, as taught by Berkey. The system of Canfield teaches recognizing anatomical structures, so labeling the structures allows the user to understand the relative anatomy of a given ultrasound image.
However, Canfield in view of Berkey fail to disclose while automatically modifying the sequentially acquired ultrasound images of the patient to label the anatomical structure, averaging the positioning of the label with a positioning of the label in one or more previous ultrasound images of the sequentially acquired ultrasound images to temporally smooth the positioning of the label when displaying the sequentially acquired ultrasound images.
Solem teaches while automatically modifying the sequentially acquired images of the patient to label the anatomical structure (landmark points, [0022]) (block 210, [0018]), averaging the positioning of the label (average location for each landmark point, [0022]) with a positioning of the label in one or more previous images of the sequentially acquired images to temporally smooth the positioning of the label ([0029]) when displaying the sequentially acquired images ([0029] & Figure 6).
Paragraph [0029] teaches that landmark point 605 can be iteratively adjusted (i.e., temporally smoothed) to locations 605B & 605C based on the mean position.
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield in view of Berkey to include while automatically modifying the sequentially acquired images of the patient to label the anatomical structure, averaging the positioning of the label with a positioning of the label in one or more previous images of the sequentially acquired images to temporally smooth the positioning of the label when displaying the sequentially acquired images, as taught by Solem. Labeling the anatomical structures based on their average positions results in more accurate labeling by smoothing out any potential outliers.
Claims 6-8 & 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Canfield in view of Berkey, as applied to claims 1 & 13, above, in further view of Schneider (US 2017/0273669).
Regarding claim 6, Canfield in view of Berkey teach the ultrasound system of claim 1.
However, Canfield in view of Berkey fail to disclose ultrasound image grading circuitry configured to: receive the acquired ultrasound images from the ultrasound imaging device; automatically grade an image quality of the received ultrasound images; and assign an image quality grade to the received ultrasound images.
Schneider teaches ultrasound image grading circuitry (quality scoring module 115, [0033]) configured to:
receive the acquired ultrasound images ([0034]) from the ultrasound imaging device (probe 12, [0028]);
automatically grade an image quality of the received ultrasound images ([0033]-[0034]); and
assign an image quality grade to the received ultrasound images ([0033]-[0034]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield in view of Berkey to include ultrasound image grading circuitry configured to: receive the acquired ultrasound images from the ultrasound imaging device; automatically grade an image quality of the received ultrasound images; and assign an image quality grade to the received ultrasound images, as taught by Schneider. Scoring the image quality of an image provides a way for the user to maximize image quality, helping to obtain images of the highest possible quality.
Regarding claims 7-8, Canfield in view of Berkey and Schneider teach the ultrasound system of claim 6, and Schneider further teaches that the display (display device 118, [0034]) is configured to display an indication (indicator 206, [0050]) of the image quality grade (quality score 136, [0034]) of the received ultrasound images, wherein the indication of the image quality grade includes a number indicating the image quality grade ([0046] & Figure 5).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield in view of Berkey such that the display is configured to display an indication of the image quality grade of the received ultrasound images, wherein the indication of the image quality grade includes a number indicating the image quality grade, as taught by Schneider. As the image quality has already been graded, this communicates this information to the user, allowing them to optimize the quality of the image.
Regarding claim 16, and Canfield in view of Berkey teach the method of claim 13.
However, Canfield in view of Berkey fail to disclose automatically grading, by ultrasound image grading circuitry, an image quality of the received ultrasound images.
Schneider teaches automatically grading, by ultrasound image grading circuitry (quality scoring module 115, [0033]), an image quality of the received ultrasound images ([0033]-[0034]), and assigning an image quality grade to the received ultrasound images ([0033]-[0034]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the method taught by Canfield in view of Berkey to include automatically grading, by ultrasound image grading circuitry, an image quality of the received ultrasound images, and assigning an image quality grade to the received ultrasound images, as taught by Schneider. Scoring the image quality of an image provides a way for the user to maximize image quality, helping to obtain images of the highest possible quality.
Regarding claim 17, Canfield in view of Berkey and Schneider teach the method of claim 16, and Schneider further teaches displaying an indication (indicator 206, [0050]) of the image quality grade (quality score 136, [0034]) of the received ultrasound images (Figure 5).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the method taught by Canfield in view of Berkey to include displaying an indication of the image quality grade of the received ultrasound images, as taught by Schneider. As the image quality has already been graded, this communicates this information to the user, allowing them to optimize the quality of the image.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Canfield in view of Berkey and Schneider, as applied to claim 7, above, in further view of Davidson (US 2015/0313445).
Regarding claim 9, Canfield in view of Berkey and Schneider teach the ultrasound system of claim 7.
However, Canfield in view of Berkey and Schneider fail to disclose that the indication of the image quality grade is an integer number from 1 to 5.
Davidson teaches that the indication of the image quality grade is an integer number from 1 to 5 ([0110]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield in view of Berkey and Schneider such that the indication of the image quality grade is an integer number from 1 to 5 move, as taught by Davidson. This simplifies the image quality grade into a singular, easy-to-understand value, allowing the user to quickly and easily understand the relative quality of a given image.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Canfield in view of Berkey and Schneider, as applied to claim 6, above, in further view of Abolmaesumi (US 2020/0069292) and Cadieu (US 2020/0245976).
Regarding claim 10, Canfield in view of Berkey and Schneider teach the ultrasound system of claim 6.
However, Canfield in view of Berkey and Schneider fail to disclose that the ultrasound image grading circuitry is further configured to: provide feedback to a user based on the image quality grade, the feedback configured to guide the user toward obtaining a selected view of the one or more anatomical structures.
Abolmaesumi teaches that the ultrasound image grading circuitry (computer-implemented image analyzer 14, [0042]) is further configured to:
provide feedback (feedback, [0044]) to a user based on the image quality grade (quality assessment value, [0044]), the feedback configured to guide the user toward obtaining a selected view of the one or more anatomical structures ([0044]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield in view of Berkey and Schneider such that the ultrasound image grading circuitry is further configured to: provide feedback to a user based on the image quality grade, the feedback configured to guide the user toward obtaining a selected view of the one or more anatomical structures, as taught by Abolmaesumi. If a particular image does not surpass a quality threshold, this feedback provides the necessary steps to ensure that an acceptable image is obtained.
However, Canfield in view of Berkey, Schneider, and Abolmaesumi fail to disclose providing to the user an indication of a user motion of the ultrasound imaging device in order to obtain a selected view.
Cadieu teaches providing to the user an indication of a user motion (recommended movement and spatial orientation indicator 180, [0019]) of the ultrasound imaging device (ultrasound imaging probe 120, [0019]) in order to obtain a selected view ([0019]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield in view of Berkey, Schneider, and Abolmaesumi to include providing to the user an indication of a user motion of the ultrasound imaging device in order to obtain a selected view, as taught by Cadieu. If a particular image does not surpass a quality threshold, instructing the user to reposition the imaging device ensures that an acceptable image is obtained.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Canfield in view of Berkey and Schneider, as applied to claim 6, above, in further view of Abolmaesumi.
Regarding claim 11, Canfield in view of Berkey and Schneider teach the ultrasound system of claim 6.
However, Canfield in view of Berkey and Schneider fail to disclose that the ultrasound image grading circuitry is implemented at least partially by machine learning circuitry including at least one artificial neural network.
Abolmaesumi teaches that the ultrasound image grading circuitry (computer-implemented image analyzer 14, [0042]) is implemented at least partially by machine learning circuitry including at least one artificial neural network (deep learning neural network, [0046]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Canfield in view of Berkey and Schneider such that the ultrasound image grading circuitry is implemented at least partially by machine learning circuitry including at least one artificial neural network, as taught by Abolmaesumi. Employing machine learning to grade image quality standardizes this step, providing consistent data.
Claims 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Canfield in view of Berkey, as applied to claims 1 & 13, above, in further view of Cadieu.
Regarding claim 21, Canfield in view of Berkey teach the ultrasound system of claim 1.
However, Canfield in view of Berkey fail to disclose that the recognized view is one of a suprasternal, subcostal, short- and long-axis parasternal, 2-chamber apical, 3-chamber, 4-chamber apical, or 5-chamber apical view of a heart.
Cadieu teaches that the recognized view is one of a subcostal, short- and long-axis parasternal, 2-chamber apical, 3-chamber, 4-chamber apical, or 5-chamber apical view of a heart ([0017]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the recognized view taught by Canfield to be one of a subcostal, short- and long-axis parasternal, 2-chamber apical, 3-chamber, 4-chamber apical, or 5-chamber apical view of a heart, as taught by Cadieu. For ultrasound examinations of the heart, these views are the most and generally known and used, giving the combined invention a wide breadth of practicality.
Regarding claim 22, Canfield in view of Berkey teach the method of claim 13.
However, Canfield in view of Berkey fail to disclose that the recognized view is one of a suprasternal, subcostal, short- and long-axis parasternal, 2-chamber apical, 3-chamber, 4-chamber apical, or 5-chamber apical view of a heart.
Cadieu teaches that the recognized view is one of a subcostal, short- and long-axis parasternal, 2-chamber apical, 3-chamber, 4-chamber apical, or 5-chamber apical view of a heart ([0017]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the recognized view taught by Canfield to be one of a subcostal, short- and long-axis parasternal, 2-chamber apical, 3-chamber, 4-chamber apical, or 5-chamber apical view of a heart, as taught by Cadieu. For ultrasound examinations of the heart, these views are the most and generally known and used, giving the combined invention a wide breadth of practicality.
Claims 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Berkey in view of Canfield and Solem.
Regarding claim 23, Berkey teaches an ultrasound system, comprising:
an ultrasound imaging device (ultrasound probe 16, [0036]) configured to acquire a sequence of ultrasound images of a patient ([0036]);
anatomical structure recognition and labeling circuitry (ultrasound image labeling system 200, [0042]) operable according to executable instructions (code to implement the present disclosure, [0112]) stored in a memory (memory 26, [0042]), and when the executable instructions are executed by one or more processors (processor 24, [0042]), the anatomical structure recognition and labeling circuitry is configured to:
receive ultrasound images in the sequence of ultrasound images from the ultrasound imaging device (step 404, [0055]);
generate an output ([0052]), by the machine learning circuitry (labeling module 12, [0045]), that automatically determines a label for an anatomical structure in the received ultrasound images (step 312, [0052]), wherein the machine learning circuitry is configured to use the recognized view when generating the output ([0052] & [0054]), the output from the machine learning circuitry indicating the anatomical structure ([0052]), and when automatically determining the label, the machine learning circuitry is restricted to a defined set of labels associated with the recognized view (menu of anticipated landmark anatomical structures to appear in the ultrasound image, [0054]), wherein labels in the defined set of labels identify different anatomical structures ([0052] & [0054]);
select, from the defined set of labels, a label for an anatomical structure in the ultrasound images ([0052]), wherein the selection of the label is restricted to the defined set of labels ([0049] & [0054]);
determine, in the ultrasound images of the patient, a positioning of the label for the anatomical structure ([0088]-[0092]); and
automatically modify the ultrasound images to label the anatomical structure at the positioning with the label determined from the defined set of labels ([0088]-[0092]); and
a display (display screen 14, [0036]) configured to display the ultrasound images with the labeled anatomical structure ([0037]).
However, Berkey fails to disclose automatically recognizing a view of the ultrasound images and one or more anatomical structures in the ultrasound images.
Canfield teaches automatically recognizing a view of the ultrasound images (“the neural net model was trained to identify…the view of the anatomy seen in the ultrasound image”, [0019]) and one or more anatomical structures in the ultrasound images (“the neural net model was trained to identify…the view of the anatomy seen in the ultrasound image”, [0019]).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modify the ultrasound system taught by Berkey to include automatically recognizing a view of the ultrasound images and one or more anatomical structures in the ultrasound images, as taught by Canfield. Automating the view and anatomical structure recognition steps can increase the accuracy of the recognition.
However, Berkey in view of Canfield fail to disclose averaging the positioning of the label with a positioning of the label in one or more previous ultrasound images of the sequence of ultrasound images to temporally smooth the positioning of the label when displaying the sequence of ultrasound images.
Solem teaches averaging the positioning of the label (average location for each landmark point, [0022]) with a positioning of the label in one or more previous images of the sequence of images to temporally smooth the positioning of the label ([0029]) when displaying the sequence of images ([0029] & Figure 6).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modified the ultrasound system taught by Berkey in view of Canfield to include averaging the positioning of the label with a positioning of the label in one or more previous ultrasound images of the sequence of ultrasound images to temporally smooth the positioning of the label when displaying the sequence of ultrasound images, as taught by Solem. Labeling the anatomical structures based on their average positions results in more accurate labeling by smoothing out any potential outliers.
Regarding claim 24, Berkey in view of Canfield and Solem teach the ultrasound system of claim 23, and Berkey further teaches that the display is configured to display the sequence of ultrasound images modified with the label that identifies the anatomical structure ([0088]-[0092]).
Because Solem teaches determining a temporally smoothed positioning, the combination of Berkey and Solem would display the image is modified of labels at the temporally smoothed positioning.
Regarding claim 25, and Berkey in view of Canfield and Solem teach the ultrasound system of claim 23, and Canfield further teaches that the anatomical structure recognition and labeling circuitry (neural network model 80, [0019]) is configured to automatically recognize the view of each ultrasound image in the sequence of ultrasound images before a next one of the ultrasound images is received from the ultrasound imaging device ([0009] & [0019]).
Paragraph [0009] teaches that the neural net model analyzes images in real time. Paragraph [0019] teaches that the neural network recognizes the view of the image.
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modify the ultrasound system taught by Berkey such that the anatomical structure recognition and labeling circuitry is configured to automatically recognize the view of each ultrasound image in the sequence of ultrasound images before a next one of the ultrasound images is received from the ultrasound imaging device, as taught by Canfield. This allows the system to continue to convey accurate recognition information even when imaging at higher frame rates.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Berkey in view of Canfield and Solem, as applied to claim 23, above, in further view of Schneider.
Regarding claim 26, Berkey in view of Canfield and Solem teach the ultrasound system of claim 23.
However, Berkey in view of Canfield and Solem fail to disclose ultrasound image grading circuitry operable according to the executable instructions in the memory, such that execution of the executable instructions by the one or more processors causes the ultrasound image grading circuitry to: automatically grade an image quality of the ultrasound images; assign an image quality grade to the received ultrasound images; and automatically modify the ultrasound images to include an indication of the image quality grade.
Schneider teaches ultrasound image grading circuitry (quality scoring module 115, [0033]) operable according to the executable instructions (program code, [0027]) in the memory (memory 116, [0033]), such that execution of the executable instructions by the one or more processors (one or more processors 114, [0033]) causes the ultrasound image grading circuitry to:
automatically grade an image quality of the ultrasound images ([0033]-[0034]);
assign an image quality grade (quality score 136, [0034]) to the received ultrasound images ([0033]-[0034]); and
automatically modify the ultrasound images to include an indication (indicator 206, [0050]) of the image quality grade ([0050] & Figure 5).
It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have modify the ultrasound system taught by Berkey, Canfield, and Solem to include ultrasound image grading circuitry operable according to the executable instructions in the memory, such that execution of the executable instructions by the one or more processors causes the ultrasound image grading circuitry to: automatically grade an image quality of the ultrasound images; and assign an image quality grade to the received ultrasound images; and automatically modify the ultrasound images to include an indication of the image quality grade, as taught by Schneider. Scoring the image quality of an image provides a way for the user to maximize image quality, helping to obtain images of the highest possible quality.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADAM KOLKIN whose telephone number is (571)272-5480. The examiner can normally be reached Monday-Friday 1:00PM-10:00PM EDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Keith Raymond can be reached on (572)-270-1790. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADAM D. KOLKIN/Examiner, Art Unit 3798
/KEITH M RAYMOND/Supervisory Patent Examiner, Art Unit 3798