DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s argument on Page 11 regarding the objection to the drawings has been fully considered. However, in Fig. 18, element 920 refers to the graphic card, whereas in the specification 920 is the GPU, and 92 is the graphics card (element numbers are correct in Fig. 17). The objection to the drawings is maintained.
Applicant’s argument on Pages 11-13 regarding the rejection of Claim 18 under 35 U.S.C. 102(a)(1) as being anticipated by Wels has been fully considered but is not persuasive.
On Pages 11-12, applicant argues that “the term ‘or’ used in the objected technical feature of pending claim 18 is not dedicated to depict an alternative, but rather to cover all possible alternatives (and there are only two of them); either the ‘required landmark(s) of the list of at least two required landmarks’ are present in ‘the current ultrasound image’ or they are absent from ‘the current ultrasound image,’ that the claimed subject matter is concerned with both possible alternatives without exclusion, and that Wels “is only concerned with detecting a selected landmark in the image.” However, while the claimed limitations mentions the absence of a landmark, it does not explicitly require the ultrasound image to have an absence of landmarks. See applicant’s disclosure [0082]: “in Figure 19(d) the absence of a landmark is pathological (because this landmark must be present and visible in a physiological development), whereas in the example in Figure 20(a), the absence of any landmark in this region is physiological (i.e., the presence of any anatomical landmark in this region would be abnormal).” For this reason, the claim limitation is interpreted as an alternative and Wels still reads on the claim limitations.
On Page 12, applicant argues that Wels does not disclose any kind of examination landmark database, and further argues that because the landmark annotation database 208 of Wels is not the same as the claimed examination landmark database and does not offer the ability to verify whether the at least one identified required landmark in the current image is present in the examination landmark database. However, the landmark annotation database 208 of Wels reads on the claims examination landmark database because when the landmark is automatically detected 408, it is stored as a new landmark annotation, updates the current landmark, or is dismissed, as in [0033]-[0034]. Updating the current landmark or storing it as a new landmark is interpreted as verifying whether the at least one identified required landmark in the current image is present in an examination landmark database, comprising landmarks identified during previous iterations; wherein said examination landmarks database is empty at a first iteration when any ultrasound images had been received yet.
On Page 13, applicant argues that Wels’ condition of keeping or dismissing a landmark annotation with respect to said landmark annotation database 208 has no common matter with the claimed condition concerned by the potential absence of “at least one of the at least one identified landmark in the current image.” However, as discussed above, the ‘absence’ of the landmark is not necessarily required by the claim and may be interpreted as an alternative to detecting the presence of landmarks. Therefore, Wels still reads on the claim limitations.
Regarding the rejection of all remaining corresponding claims, applicant’s argument submitted on Pages 13-15 relies on the supposed deficiencies with respect to the rejection of parent Claim 18. Applicant’s argument is moot for the same reasons detailed above.
While differences between the prior art and the instant application are appreciated, they are not embodied in the claims in such a ways as to differentiate.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “920” has been used to designate both GPU (Fig. 17) and Graphics card (Fig. 18). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 18 and 32 are objected to because of the following informalities: minor error in antecedent basis. The claims should be amended to “[…] detect[ing] and identify[ing] [[the]] a presence of at least one required landmark of the list of at least two required landmarks or [[the]] an absence of landmarks in the current ultrasound image […]” in order to establish proper antecedent basis. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 18-34 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding Claims 18 and 32, the claims recite “[…] detect and identify […] or the absence of landmarks in the current ultrasound image […]” but then continue to only recite the steps with the identified landmarks, as an ‘absence’ would require no identification of the landmarks. The claim is indefinite as it is unclear whether the “absence of landmarks” is considered the “identified landmark,” as the claim continues. It is suggested applicant amend the claims in order to effectively establish which limitations are considered the “identified required landmark.”
Claims not explicitly addressed above are rejected as depending from a rejected claim and failing to cure deficiencies of the parent claim.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 18-21, 23-25, and 28-34 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wels et al. (US 20140219548).
Regarding Claim 18, Wels teaches a device for guiding a user in ultrasound assessment of an organ to perform a diagnostic or screening evaluation of said organ during a medical examination, (Abstract “A […] system for on-line learning of landmark detection models for end-user specific diagnostic image reading is disclosed,” [0017] “The user interface controller 202 controls an interactive 3D medical image viewer to be displayed on an end user device 212 of an end user,” and [0019] “The 3D medical images may be acquired using any imaging modality, such as […] ultrasound”), said ultrasound assessment being based on ultrasound images resulting from at least a partial reflection of ultrasound by said organ, ([0019] “the 3D medical image database 206 stores 3D medical images received by the system 200. The 3D medical images may be acquired using any imaging modality, such as […] ultrasound,” where the limitation is how ultrasound is utilized.), said device comprising:
a) at least one input configured to receive:
i) a current ultrasound image ([0019] “the 3D medical image database 206 stores 3D medical images received by the system 200.”);
ii) a list of at least two required landmarks used to perform diagnostic or screening evaluation of said organ ([0014] “viewing planes 102, 104, and 106 are oriented based on four anatomical landmarks associated with the spinal disk” and [0017] “The user interface controller 202 can control the interactive 3D medical image viewer to display detection results for automatically detected landmarks, and store landmark annotations entered by the end user in the landmark annotation database 208.”); and
b) at least one processor, ([0016] “The system 200 of FIG. 2 can be implemented on one or more computers well-known computer processors, memory units, storage devices, computer software, and other components, and provides fully automatic detection of user-defined landmarks with on-site knowledge acquisition and on-site detection model generation.”), configured to, iteratively:
i) detect and identify the presence of at least one required landmark of the list of at least two required landmarks or the absence of landmarks in the current ultrasound image ([0017] “The user interface controller can also control the machine learning module 204 to perform automatic landmark detection is response to a user selection in the interactive 3D medical image viewer invoking automatic landmark detection,” [0029] “At step 408, the selected landmark is automatically detected in the 3D medical image using the trained landmark detection model corresponding to the selected landmark,” and Claim 16 “the computer program instructions when executed by a processor cause the processor to perform operations comprising: […] determining a current landmark detection result for the selected landmark in the 3D medical image by at least one of: automatically detecting the selected landmark in the 3D medical image using a stored landmark detection model corresponding to the selected landmark, and receiving a manual annotation of the selected landmark in the 3D medical image.”);
ii) verify whether the at least one identified required landmark in the current image is present in an examination landmark database, comprising landmarks identified during previous iterations ([0028] “At step 406, it is determined if a landmark detection model for the selected landmark is reliable,” [0029] “At step 408, the selected landmark is automatically detected in the 3D medical image using the trained landmark detection model corresponding to the selected landmark,” [0032] “At step 414, it is determined if the current landmark should be kept. In an exemplary implementation, the determination of whether to keep the current landmark can depend on user input received via the interactive 3D medical image viewer. In this case, the user input may be received in response to providing the end user an option to keep or dismiss the current landmark. If it is determined that the current landmark should not be kept, the method proceeds to step 416. If it is determined that the current landmark should be kept, the method proceeds to step 418.”); wherein said examination landmarks database is empty at a first iteration when any ultrasound images had been received yet ([0034] “At step 418, the current landmark is stored and the landmark detection model corresponding to the current landmark is updated. In particular, the current landmark detection or annotation result is stored as a new landmark annotation in the landmark annotation database 208. The machine learning module 204 then updates the landmark detection module corresponding stored in the detector model database 210 based on the new landmark annotation stored in the landmark annotation database 208.” Where because the model is based on the detected landmarks, at the first iteration the landmark annotation database 208, interpreted as the examination landmarks database, will be empty. See also [0021], where the machine learning classifier used to train the landmark detection model for each landmark may be any type.); and
iii) if at least one of the at least one identified landmark in the current image is not present in the examination landmark database:
1) trigger storage of the current ultrasound image in an image database ([0020] “The landmark annotation database 208 stores landmark annotations associated with 3D medical images.”);
2) trigger storage in the examination landmark database of the at least one identified required landmark which was not yet comprised in the examination landmarks database ([0034] “If no detection model exists for the landmark in the detector model database 210, the machine learning module 204 can generate an initial landmark detection model based in part on the new landmark annotation and store the landmark detection model in the detector model database 210.”); and
3) verify that all of the at least two required landmarks of the list have been stored in the examination landmark database and if at least one of the required landmarks of the list is missing from the examination landmark database, trigger the reception of at least one additional current ultrasound image ([0031] “At step 412, it is determined whether refinement of the current landmark is necessary. In an exemplary implementation the current landmark (the landmark detection result of step 408 or the landmark annotation result of step 410) is displayed to the end user in the interactive 3D medical image viewer and an a user input is received indicating whether refinement of the current landmark is necessary. If it is determined that refinement of the current landmark is necessary, the method returns to step 410. In this case a landmark annotation of the user received at step 410 corresponds to a refinement of the current landmark detection/annotation result. If it is determined that no refinement of the current landmark is necessary, the method proceeds to step 414.”).
Regarding Claim 19, Wels teaches all limitations of Claim 18, as discussed above. Furthermore, Wels teaches wherein, for the detection and identification, the at least one processor is further configured to localize the identified required landmarks in the current ultrasound image ([0018] “This allows a user to manually annotate a target landmark in 3D medical image data and refine landmarks automatically detected in 3D medical image data” and [0021] “The machine learning module 204 performs machine learning to train detection models for 3D object position detection. In an embodiment of the present invention, the detection model for a particular landmark is a 3D translation detector trained with Haar-like features to detect the position of a 3D bounding box centered at the landmark position.”).
Regarding Claim 20, Wels teaches all limitations of Claim 18, as discussed above. Furthermore, Wels teaches wherein the at least one processor is further configured to, before the detection and identification of the required landmarks, detect a region of interest (ROI) on the current ultrasound image comprising all the anatomical landmarks present in the current ultrasound image, ([0027] “a custom view can be defined by viewing planes oriented to three or more landmarks. Accordingly, in a possible embodiment the landmark selection can be a selection of a particular view (either new or previously defined), and the view can define a set of landmarks to be detected.”), using a machine learning approach ([0024] “the machine learning module 204 performs operations such as pre-processing of the medical image data”).
Regarding Claim 21, Wels teaches all limitations of Claim 20, as discussed above. Furthermore, Wels teaches wherein the at least one processor is further configured to localize the identified required landmarks by using the machine learning approach receiving as input the region of interest of the current ultrasound image ([0018] “This allows a user to manually annotate a target landmark in 3D medical image data and refine landmarks automatically detected in 3D medical image data” and [0021] “The machine learning module 204 performs machine learning to train detection models for 3D object position detection. In an embodiment of the present invention, the detection model for a particular landmark is a 3D translation detector trained with Haar-like features to detect the position of a 3D bounding box centered at the landmark position.”).
Regarding Claim 23, Wels teaches all limitations of Claim 18, as discussed above. Furthermore, Wels teaches wherein the at least one processor is further configured to select, for performing the detection and identification of the presence or the absence of at least one required landmark in the current ultrasound image, a machine learning model previously trained on ultrasound images comprising at least part of the identified required landmarks comprised in the current ultrasound image ([0020] “The landmark annotation database 208 and the 3D medical image database 206 are linked to each other such that each the landmark annotation in the landmark annotation database 208 is linked to the corresponding 3D medical image in which the landmark was annotated/detected in the 3D medical image database 206” and [0026] “a 3D medical image is opened”).
Regarding Claim 24, Wels teaches all limitations of Claim 19, as discussed above. Furthermore, Wels teaches wherein the at least one processor is further configured to evaluate the quality of each localized landmark ([0031] “At step 412, it is determined whether refinement of the current landmark is necessary.” Where because refinement is understood as making small changes, it is interpreted that determining whether refinement is necessary is evaluating the quality of the localized landmarks.).
Regarding Claim 25, Wels teaches all limitations of Claim 24, as discussed above. Furthermore, Wels teaches wherein the at least one processor is further configured to trigger storage of a localized landmark in the examination landmarks database only if the localized landmark has at least a minimal predefined quality ([0031] “At step 412, it is determined whether refinement of the current landmark is necessary. In an exemplary implementation the current landmark (the landmark detection result of step 408 or the landmark annotation result of step 410) is displayed to the end user in the interactive 3D medical image viewer and an a user input is received indicating whether refinement of the current landmark is necessary. If it is determined that refinement of the current landmark is necessary, the method returns to step 410. In this case a landmark annotation of the user received at step 410 corresponds to a refinement of the current landmark detection/annotation result. If it is determined that no refinement of the current landmark is necessary, the method proceeds to step 414.”).
Regarding Claim 28, Wels teaches all limitations of Claim 18, as discussed above. Furthermore, Wels teaches wherein the at least one processor is configured to iteratively perform operations of: detecting and identify the presence of at least one required landmark of the list of at least two required landmarks or the absence of landmarks in the current ultrasound image, ([0017] “The user interface controller can also control the machine learning module 204 to perform automatic landmark detection is response to a user selection in the interactive 3D medical image viewer invoking automatic landmark detection,” [0029] “At step 408, the selected landmark is automatically detected in the 3D medical image using the trained landmark detection model corresponding to the selected landmark,” and Claim 16 “the computer program instructions when executed by a processor cause the processor to perform operations comprising: […] determining a current landmark detection result for the selected landmark in the 3D medical image by at least one of: automatically detecting the selected landmark in the 3D medical image using a stored landmark detection model corresponding to the selected landmark, and receiving a manual annotation of the selected landmark in the 3D medical image.”), verifying whether the at least one identified required landmark in the current image is present in an examination landmark database, ([0028] “At step 406, it is determined if a landmark detection model for the selected landmark is reliable,” [0029] “At step 408, the selected landmark is automatically detected in the 3D medical image using the trained landmark detection model corresponding to the selected landmark,” [0032] “At step 414, it is determined if the current landmark should be kept. In an exemplary implementation, the determination of whether to keep the current landmark can depend on user input received via the interactive 3D medical image viewer. In this case, the user input may be received in response to providing the end user an option to keep or dismiss the current landmark. If it is determined that the current landmark should not be kept, the method proceeds to step 416. If it is determined that the current landmark should be kept, the method proceeds to step 418.”), triggering storage of the current ultrasound image in an image database, ([0020] “The landmark annotation database 208 stores landmark annotations associated with 3D medical images.”), triggering storage in the examination landmark database of the at least one identified required landmark which was not yet comprised in the examination landmarks database, ([0034] “If no detection model exists for the landmark in the detector model database 210, the machine learning module 204 can generate an initial landmark detection model based in part on the new landmark annotation and store the landmark detection model in the detector model database 210.”), verifying that all of the at least two required landmarks of the list have been stored in the examination landmark database, ([0031] “At step 412, it is determined whether refinement of the current landmark is necessary. In an exemplary implementation the current landmark (the landmark detection result of step 408 or the landmark annotation result of step 410) is displayed to the end user in the interactive 3D medical image viewer and an a user input is received indicating whether refinement of the current landmark is necessary. If it is determined that refinement of the current landmark is necessary, the method returns to step 410. In this case a landmark annotation of the user received at step 410 corresponds to a refinement of the current landmark detection/annotation result. If it is determined that no refinement of the current landmark is necessary, the method proceeds to step 414.”), on an ordered time sequence of ultrasound images (Fig. 4 and [0015] “Embodiments of the present invention work on a database of 3D medical images and allow end users to set up and tune detector models for detecting various landmarks in the 3D medical images.” Where because the users work with multiple 3D medical images, which may be ultrasound images, as in [0019], which are captured sequentially over a period of time, the limitation is taught by Wels.).
Regarding Claim 29, Wels teaches all limitations of Claim 25, as discussed above. Furthermore, Wels teaches wherein:
a) the at least one processor is configured to select a portion of the time sequence of ultrasound images comprising a highest number of localized landmarks having at least a minimal predefined quality ([0032] “At step 414, it is determined if the current landmark should be kept” and [0034] “At step 418, the current landmark is stored and the landmark detection model corresponding to the current landmark is updated. In particular, the current landmark detection or annotation result is stored as a new landmark annotation in the landmark annotation database 208. The machine learning module 204 then updates the landmark detection module corresponding stored in the detector model database 210 based on the new landmark annotation stored in the landmark annotation database 208. The machine learning module can also update a confidence associated with the landmark detection model corresponding to the landmark.” Where the highest number of localized landmarks would be used to update the landmark detection model for the most accuracy.); and
b) wherein the at least one processor is configured to iteratively perform operations of: detecting and identify the presence of at least one required landmark of the list of at least two required landmarks or the absence of landmarks in the current ultrasound image, ([0017] “The user interface controller can also control the machine learning module 204 to perform automatic landmark detection is response to a user selection in the interactive 3D medical image viewer invoking automatic landmark detection,” [0029] “At step 408, the selected landmark is automatically detected in the 3D medical image using the trained landmark detection model corresponding to the selected landmark,” and Claim 16 “the computer program instructions when executed by a processor cause the processor to perform operations comprising: […] determining a current landmark detection result for the selected landmark in the 3D medical image by at least one of: automatically detecting the selected landmark in the 3D medical image using a stored landmark detection model corresponding to the selected landmark, and receiving a manual annotation of the selected landmark in the 3D medical image.”), verifying whether the at least one identified required landmark in the current image is present in an examination landmark database, ([0028] “At step 406, it is determined if a landmark detection model for the selected landmark is reliable,” [0029] “At step 408, the selected landmark is automatically detected in the 3D medical image using the trained landmark detection model corresponding to the selected landmark,” [0032] “At step 414, it is determined if the current landmark should be kept. In an exemplary implementation, the determination of whether to keep the current landmark can depend on user input received via the interactive 3D medical image viewer. In this case, the user input may be received in response to providing the end user an option to keep or dismiss the current landmark. If it is determined that the current landmark should not be kept, the method proceeds to step 416. If it is determined that the current landmark should be kept, the method proceeds to step 418.”), triggering storage of the current ultrasound image in an image database, ([0020] “The landmark annotation database 208 stores landmark annotations associated with 3D medical images.”), triggering storage in the examination landmark database of the at least one identified required landmark which was not yet comprised in the examination landmarks database, ([0034] “If no detection model exists for the landmark in the detector model database 210, the machine learning module 204 can generate an initial landmark detection model based in part on the new landmark annotation and store the landmark detection model in the detector model database 210.”), verifying that all of the at least two required landmarks of the list have been stored in the examination landmark database, ([0031] “At step 412, it is determined whether refinement of the current landmark is necessary. In an exemplary implementation the current landmark (the landmark detection result of step 408 or the landmark annotation result of step 410) is displayed to the end user in the interactive 3D medical image viewer and an a user input is received indicating whether refinement of the current landmark is necessary. If it is determined that refinement of the current landmark is necessary, the method returns to step 410. In this case a landmark annotation of the user received at step 410 corresponds to a refinement of the current landmark detection/annotation result. If it is determined that no refinement of the current landmark is necessary, the method proceeds to step 414.”), on an ordered time sequence of ultrasound images (Fig. 4 and [0015] “Embodiments of the present invention work on a database of 3D medical images and allow end users to set up and tune detector models for detecting various landmarks in the 3D medical images.” Where because the users work with multiple 3D medical images, which may be ultrasound images, as in [0019], which are captured sequentially over a period of time, the limitation is taught by Wels.).
Regarding Claim 30, Wels teaches all limitations of Claim 18, as discussed above. Furthermore, Wels teaches wherein triggering storage in the image database and in the examination landmarks database is executed if the at least one identified landmark in the current image is not present in the examination landmark database at least a predefined number of times and triggering the reception of at least one additional current ultrasound image is executed if at least one of the required landmarks of the list is not present in the examination database at least said predefined number of times ([0032] “At step 414, it is determined if the current landmark should be kept” and [0035] “pre-stored existing landmark detection modules can be relied on for automated landmark detection until a critical amount of new data and annotations are received,” where following step 416 of dismissing the landmark, the processor would start again with step 402 with another 3D medical image.).
Regarding Claim 31, Wels teaches all limitations of Claim 18, as discussed above. Furthermore, Wels teaches wherein, the list of predefined landmarks further comprises at least one group of at least two coupled required landmarks; and the at least one processor is further configured to verify that, after detection and identification, if at least one coupled required landmark of one group is identified also the at least one other coupled required landmark of said group has been identified in the current image or otherwise trigger the reception of at least one additional current ultrasound image (Fig. 4, where following step 416 of dismissing the landmark, the processor would start again with step 402 with another 3D medical image, and [0027] “in a possible embodiment the landmark selection can be a selection of a particular view (either new or previously defined), and the view can define a set of landmarks to be detected.”).
Regarding Claims 32 and 34, Wels teaches a method for guiding a user in ultrasound assessment of an organ to perform a diagnostic or screening evaluation of said organ during a medical examination, (Abstract “A method and system for on-line learning of landmark detection models for end-user specific diagnostic image reading is disclosed,” [0017] “The user interface controller 202 controls an interactive 3D medical image viewer to be displayed on an end user device 212 of an end user,” and [0019] “The 3D medical images may be acquired using any imaging modality, such as […] ultrasound”), said ultrasound assessment being based on ultrasound images resulting from at least a partial reflection of ultrasound by said organ, ([0019] “the 3D medical image database 206 stores 3D medical images received by the system 200. The 3D medical images may be acquired using any imaging modality, such as […] ultrasound,” where the limitation is how ultrasound is utilized.), said method comprising iteratively:
a) receiving:
i) a current ultrasound image ([0019] “the 3D medical image database 206 stores 3D medical images received by the system 200.”);
ii) a list of at least two required landmarks used to perform diagnostic or screening evaluation of said organ ([0014] “viewing planes 102, 104, and 106 are oriented based on four anatomical landmarks associated with the spinal disk” and [0017] “The user interface controller 202 can control the interactive 3D medical image viewer to display detection results for automatically detected landmarks, and store landmark annotations entered by the end user in the landmark annotation database 208.”);
b) detecting and identifying the presence of at least one required landmark of the list of at least two required landmarks or the absence of landmarks in the current ultrasound image ([0017] “The user interface controller can also control the machine learning module 204 to perform automatic landmark detection is response to a user selection in the interactive 3D medical image viewer invoking automatic landmark detection” and [0029] “At step 408, the selected landmark is automatically detected in the 3D medical image using the trained landmark detection model corresponding to the selected landmark.”);
c) verifying whether the identified at least one landmark in the current image is present in an examination landmark database comprising landmarks identified during previous iterations ([0032] “At step 414, it is determined if the current landmark should be kept. In an exemplary implementation, the determination of whether to keep the current landmark can depend on user input received via the interactive 3D medical image viewer. In this case, the user input may be received in response to providing the end user an option to keep or dismiss the current landmark. If it is determined that the current landmark should not be kept, the method proceeds to step 416. If it is determined that the current landmark should be kept, the method proceeds to step 418.”); wherein said examination landmarks database is empty at a first iteration when none of the ultrasound images had been yet received ([0034] “At step 418, the current landmark is stored and the landmark detection model corresponding to the current landmark is updated. In particular, the current landmark detection or annotation result is stored as a new landmark annotation in the landmark annotation database 208. The machine learning module 204 then updates the landmark detection module corresponding stored in the detector model database 210 based on the new landmark annotation stored in the landmark annotation database 208.” Where because the model is based on the detected landmarks, at the first iteration the landmark annotation database 208, interpreted as the examination landmarks database, will be empty.); and
d) if at least one of the identified landmarks in the current image are not present in the examination landmark database:
i) triggering storage of the current ultrasound image in an image database ([0020] “The landmark annotation database 208 stores landmark annotations associated with 3D medical images.”);
ii) triggering storage in the examination landmark database of the at least one identified required landmark which was not yet comprised in the examination landmarks database ([0034] “If no detection model exists for the landmark in the detector model database 210, the machine learning module 204 can generate an initial landmark detection model based in part on the new landmark annotation and store the landmark detection model in the detector model database 210.”);
iii) verifying that all of the at least two required landmarks of the list have been stored in the examination landmark database and if at least one of the required landmarks of the list is missing from the examination landmark database, trigger the reception of at least one additional current ultrasound image ([0031] “At step 412, it is determined whether refinement of the current landmark is necessary. In an exemplary implementation the current landmark (the landmark detection result of step 408 or the landmark annotation result of step 410) is displayed to the end user in the interactive 3D medical image viewer and an a user input is received indicating whether refinement of the current landmark is necessary. If it is determined that refinement of the current landmark is necessary, the method returns to step 410. In this case a landmark annotation of the user received at step 410 corresponds to a refinement of the current landmark detection/annotation result. If it is determined that no refinement of the current landmark is necessary, the method proceeds to step 414.”).
Furthermore, the cited actions are computer implemented, which necessitate associated computer-readable media, as in [0038] (“The above-described methods for on-site learning of landmark detection models and automatic landmark detection, as well as the system and end user device of FIG. 2, may be implemented on a computer using well-known computer processors, memory units, storage devices, computer software, and other components.”).
Regarding Claim 33, Wels teaches all limitations of Claim 32, as discussed above. Furthermore, Wels teaches wherein, the detection and identification, further comprises the localization of the identified required landmarks in the current ultrasound image ([0018] “This allows a user to manually annotate a target landmark in 3D medical image data and refine landmarks automatically detected in 3D medical image data” and [0021] “The machine learning module 204 performs machine learning to train detection models for 3D object position detection. In an embodiment of the present invention, the detection model for a particular landmark is a 3D translation detector trained with Haar-like features to detect the position of a 3D bounding box centered at the landmark position.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 22 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Wels et al. (US 20140219548) in view of Bjaerum (US 20170086785).
Regarding Claim 22, Wels teaches all limitations of Claim 21, as discussed above. However, Wels does not explicitly teach wherein the at least one processor is further configured to compare an area of the region of interest to a predefined threshold and proceed to the localization of the identified required landmarks in the region of interest only if the area of the region of interest exceeds the predefined threshold, otherwise to trigger the reception of at least one additional current ultrasound image.
In an analogous providing tactile feedback via a probe of medical imaging system field of endeavor, Bjaerum teaches a device for guiding a user in an ultrasound assessment of an organ to perform a diagnostic or screening evaluation of said organ during a medical examination, (Abstract “systems are provided for providing tactile feedback via a probe of an imaging system” and [0004] “a method comprises acquiring ultrasound data with a probe, generating an image based on the ultrasound data, determining whether the probe is in a desired position to acquire a desired region of interest, and providing a first tactile feedback through the probe in response to a determination that the probe is in the desired position.”), wherein the at least one processor, ([0012] “system controller 116”), is further configured to compare an area of the region of interest to a predefined threshold and proceed to the localization of the identified required landmarks in the region of interest only if the area of the region of interest exceeds the predefined threshold, ([0028] “At 206, the method includes determining if the probe is in the desired position to acquire the desired ROI and scan plane. Determining whether the probe is in the desired position to acquire the desired region of interest may include performing image analysis of the generated image to determine whether the generated image substantially matches an expected image for the desired region of interest. As used herein, “substantially matches” means that the generated image from the ultrasound data matches the desired image (which may be a 2D model or 3D geometrical model) by a predetermined threshold percentage (e.g., generated image matches model by 90%). Additionally, as used herein, “expected image” may be based on one or more of stored image data and/or a geometrical model, wherein the geometrical model is a computerized anatomical 3D model of the image at the desired position. For example, image analysis may consist of detecting anatomical landmarks in the acquired ultrasound image, and comparing or matching the landmarks to a geometrical 3D model (and/or stored image data) of the anatomical structure being imaged.”), otherwise to trigger the reception of at least one additional current ultrasound image ([0028] “If the analyzed data (i.e., generated image) differs from the expected image, either by the probe being in the wrong location, the probe being held at the wrong orientation, or both, then the system determines the probe position to be incorrect and the method moves to 207.”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify Wels with the teachings of Bjaerum because the modification eases the burden on a user of “manually finding the correct position on a patient via viewing the display images alone, which may be difficult, time-consuming, and result in less accurate positioning, especially for unskilled users,” as taught by Bjaerum in [0003].
Regarding Claim 26, Wels teaches all limitations of Claim 18, as discussed above. Furthermore, Bjaerum teaches wherein, when all of the at least two identified required landmarks of the list have been acquired, the at least one processor is further configured to notify the user that all required landmarks of the list have been acquired and to stop triggering the reception of additional current ultrasound images ([0035] “Alternative methods for signaling to the user that the required image and/or video has been stored and it is now safe to move the probe may include audio or visual feedback on the probe and/or display device” and [0036] “At 222, the method includes determining whether the protocol is complete. […] the system controller, […] may determine the protocol is complete.”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify Wels with the teachings of Bjaerum because the modification minimizes extra accumulation of the images during examination, ensuring an accurate and fast-tracked imaging procedure.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Wels et al. (US 20140219548) in view of Sharma et al. (US 20200258216).
Regarding Claim 27, Wels teaches all limitations of Claim 18, as discussed above. However, Wels does not explicitly teach wherein the detection and identification of the at least one required landmark is performed by the at least one processor using a machine learning approach comprising at least one Convolutional Neural Network.
In an analogous automatic view planning for image acquisition field of endeavor, Sharma teaches a device for guiding a user in an ultrasound assessment of an organ to perform a diagnostic or screening evaluation of said organ during a medical examination, (Abstract “Systems and methods are described for automatically identifying an anatomical landmark in a medical image according to local preferences associated with a particular clinical site. A medical image for performing a medical procedure is received. An anatomical landmark is identified in the medical image using a pre-trained machine learning algorithm. Feedback relating to the identified anatomical landmark is received from a user associated with a particular clinical site. The feedback is received during a normal workflow for performing the medical procedure” and [0019] “Workstation 102 may receive medical images of patient 106 from one or more medical imaging systems 104 to perform the medical procedure. Medical imaging system 104 may be of any modality, such as, e.g., […] ultrasound (US)”), wherein the detection and identification of the at least one required landmark is performed by the at least one processor using a machine learning approach comprising at least one Convolutional Neural Network ([0027] “At step 204, one or more anatomical landmarks are identified in the medical image using a pre-trained machine learning algorithm. The anatomical landmarks may be any object of interest in the medical image. […] The pre-trained machine learning algorithm may be any suitable machine learning algorithm for identifying anatomical landmarks in a medical image, such as, e.g., convolutional neural networks”).
It would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to modify Wels with the teachings of Sharma because the modification offers the advantages of excellent feature extraction from the images, ability to leverage spatial relationships within the images, and high accuracy.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Smith is cited for teaching a device for guiding a user in ultrasound assessment of an organ to perform a diagnostic or screening evaluation of said organ during a medical examination, ([0070]), said ultrasound assessment being based on ultrasound images resulting from at least a partial reflection of ultrasound by said organ, ([0070]), said device comprising: at least one input configured to receive: a current ultrasound image ([0070] and [0071]); a list of at least two required landmarks used to perform diagnostic or screening evaluation of said organ (Fig. 23, [0076], and [0122]); and at least one processor, ([0067]), configured to, iteratively: detect and identify the presence of at least one required landmark of the list of at least two required landmarks or the absence of landmarks in the current ultrasound image ([0080]); verify whether the at least one identified required landmark in the current image is present in an examination landmark database, comprising landmarks identified during previous iterations ([0134]); and if at least one of the at least one identified landmark in the current image is not present in the examination landmark database: trigger storage of the current ultrasound image in an image database ([0071]); trigger storage in the examination landmark database of the at least one identified required landmark which was not yet comprised in the examination landmarks database ([0122]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARIA CHRISTINA TALTY whose telephone number is (571)272-8022. The examiner can normally be reached M-Th 8:30-5:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mike Carey can be reached at (571) 270-7235. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARIA CHRISTINA TALTY/Examiner, Art Unit 3797
/MICHAEL J CAREY/Supervisory Patent Examiner, Art Unit 3795
Prosecution Insights