Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Amendments
Claims 3 and 7 have been amended to correct for the previous objections which are therefore withdrawn.
Response to Arguments
Applicants’ arguments with respect to claim(s) 1-18 have been considered but are moot because the new grounds of rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 6-7, 10-12, 14-17 are rejected under 35 U.S.C. 103 as being unpatentable by Geode (US 20230011031 A1) in view of Higa (US 20240054645 A1).
As per claim 1, Geode teaches the claimed:
1. A method for processing a medical image, the method comprising:
receiving the medical image;
(Geode [0005]: “receiving a first medical image of a patient from a first medical imaging modality”)
determining at least one annotational data element to be displayed in common with the received medical image, the at least one annotational data element being related to the received medical image or parts of the received medical image;
(Geode [0033]: “In some embodiments, the medical image application 112 may be employed by a medical provider (e.g., a medical doctor, a radiologist, etc.) to annotate medical images and/or to associate clinical data with medical images.”
Geode teaches the medical image application that is determined, by the medical provider, to include annotation data into the related medical images.)
determining a region of interest in the received medical image;
(Geode [0034]: “The medical recommendation application 114 may also automatically receive (e.g., from the medical image application 112) and/or generate one or more ROIs 118 on the medical images”)
calculating an appearance of the at least one annotational data element in reply to the determined region of interest; and
(Geode [0053]: “… in another embodiment, a system may be configured to aggregate and organize specific images (e.g., FOV images) and their annotated regions of interest for inclusion into structured output (e.g., the CCD structured output (i.e., a standard format)) while maintaining relationships with the data (e.g., images and associated textual data) with external systems via metadata layers of the annotations.”
Geode [0069]: “The method 900 may include, at action 910, generating first annotation data related to the first ROI”.
Geode teaches the system configuring to organize and aggregate different medical images with their annotated ROIs into a structured output, thus calculating the output of the at least one annotational data element for the determined ROIs.)
outputting the calculated appearance of the at least one annotational data element in common with the received medical image by an output device, wherein,
(Geode [0037]: “As disclosed in FIG. 2, a display 150 may include an image 152 and annotations. More specifically, display 150 includes several regions of interest (ROIs) associated with image 152. In some embodiments, an ROI may include, but is not limited to, a feature contained in a medical image”
Geode [0055]: “In one embodiment, the format and type of an output of a system may be a combination of annotation regions of interest captured in a FOV from a combination of encounters”
Geode [0090]: “The user interface 1012 may include any device configured to allow a user to interface with the computer system 1000. For example, the user interface 1012 may include a display, such as an LCD, LED, or other display, that is configured to present video, text, application user interfaces, and other data as directed by the processor 1002.”)
Geode alone does not explicitly teach the remaining limitations.
However, Geode in combination with Higa teaches the claimed:
the calculating includes, calculating a position for outputting the at least one annotational data element in relation to the received medical image, such that the at least one annotational data element is not, or only slightly overlapping the determined region of interest.
(Higa [0108]: “The superimposed-position determination unit 208 superimposes pieces of graphic information on the basis of the region information such that the pieces of graphic information do not overlap the regions of interest … An annotation that is added while the ultrasound image is displayed is also displayed so as not to overlap graphic information as described above to thereby allow the annotation to be easily viewable.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention of the superimposed position determination unit as taught by Higa with the system of Geode in order to allow graphic information, such as annotations, to be placed in a way such that it does not overlap or block the view of the actual image displayed.
As per claim 14, this claim is similar in scope to limitations recited in claim 1 and thus is rejected under the same rationale.
As per claim 3, Geode teaches the claimed:
3. The method of claim 1, further comprising:
displaying the received medical image after the determining the at least one annotational data element and before the determining the region of interest.
(Geode, FIG 5, shows examples of the medical images (502, 506, 508, 510) that include annotations such as lines or arrows, without determining a region of interest. Geode also states in paragraph [0059], “FIG. 5 is a screenshot (e.g., a presentation to a user interface) depicting a plurality of images”. Therefore, Geode teaches the displaying of the medical images after determining at least one annotational data element and before determining the region of interest.
PNG
media_image1.png
519
740
media_image1.png
Greyscale
)
As per claim 4, Geode teaches the claimed:
4. The method of claim 3, wherein the determining the region of interest is based on a user interaction with the displayed medical image.
(Geode [0062]: “According to various embodiments, a system may enable an end-user to use annotation tools for drawing a region of interest (ROI) on various images”)
As per claim 6, Geode teaches the claimed:
6. The method of claim 1, wherein the calculating includes at least one of,
adapting a size of the at least one annotational data element,
replacing an expression by an abbreviation,
adapting text wrapping,
adapting a transparency level of the at least one annotational data element,
displaying only outlines or contours of the at least one annotational data element,
substituting the at least one annotational data element by a thumbnail, or
applying at least one of a shading structure, a shadowing structure or a dashed graphical structure.
(Geode [0065]: “… in some embodiments two or more medical images may additionally or alternatively be simultaneously rendered and displayed (with visual annotations) overlaying one another (e.g., on top of one another with a potentially adjustable level of transparency to allow for one or more images to be seen through the top image) to give additional context and for comparison purposes.”)
As per claim 7, Geode teaches the claimed:
7. The method of claim 1, wherein the calculating includes,
determining an output area for displaying the at least one annotational data element,
(Geode [0049]: “A system may be configured to provide a presentation layer (e.g., via a computer screen) or portal that allows for real-time communication with data (e.g., images)”.
Geode [0052]: “A system may further be configured to group one or more annotations into discrete elements, wherein the one or more annotations may be displayed with one or more images”.
Geode teaches the system that is configured with a presentation layer that defines where things are displayed and annotations being displayed with one or more images which requires placement logic. Thus, the system determines where the annotations are placed and displayed relative to the image, which is the output area.)
wherein the output area is at least one of an area outside the received medical image, an area outside an area representing a body or a part of a body in the received medical image, or an area outside the determined region of interest.
(Geode [0051]: “Annotated regions of interest (ROI) may maintain a relationship with an original image”
Geode [0052]: “A system may further be configured to group one or more annotations into discrete elements, wherein the one or more annotations may be displayed with one or more images”.
Geode teaches the annotations are tied to the ROIs, but can also exist as separate discrete elements. Therefore, if the annotations are separate elements from the ROIs, they can be rendered outside the ROI boundary.)
As per claim 10, Geode teaches the claimed:
10. The method of claim 1, wherein the medical image is received from at least one of an ultrasonic imaging modality, an X-ray modality, a positron emission tomography apparatus or a magnetic resonance tomograph.
(Geode [0067]: “the first medical imaging modality may include a computation tomography (CT) imaging modality, a magnetic resonance imaging (MRI) imaging modality”.)
As per claim 11, Geode teaches the claimed:
11. The method of claim 1, wherein the at least one annotational data element comprises at least one of,
at least one graphical element, a textual element or a numerical element,
meta data relating to a patient,
a configuration of a medical imaging modality for capturing the medical image,
data specifying a date, or
data relating to a purpose for capturing the medical image.
(Geode in FIG. 6 and 7 shows annotational data elements such as lines, arrows, text, labels, circles, and codes, which are graphical and textual elements.
PNG
media_image2.png
515
752
media_image2.png
Greyscale
PNG
media_image3.png
252
771
media_image3.png
Greyscale
)
As per claim 12, Geode teaches the claimed:
12. The method of claim 1, wherein the determining the at least one annotational data element is based on at least one of metadata or supplementary data provided in association with the received medical image.
(Geode [0051]: “In one specific embodiment, a system may process and aggregate images including annotations (e.g., annotated regions of interest) across image specialties and modalities (e.g., radiology, pathology, surgery and oncology), wherein the annotations (e.g., metadata) may maintain relationships across one or more images within a data set.”)
As per claim 15, Geode teaches the claimed:
15. A non-transitory computer readable medium including program code sections, when executed by a computing unit, cause the computer unit to perform the method of claim 1.
(Geode [0066]: “In these and other embodiments, the method 900 may be performed by one or more processors based on one or more computer-readable instructions stored on one or more non-transitory computer-readable media.”)
As per claim 16, Geode teaches the claimed:
16. The method of claim 4, wherein the determining the region of interest is based on at least one of monitoring a pointer position, an eye-tracking of the user, or a semantic analysis of the displayed medical image.
(Geode [0062]: “According to various embodiments, a system may enable an end-user to use annotation tools for drawing a region of interest (ROI) on various images, such as image 504”.
Geode teaches the user using annotation tools to draw a region of interest, which this region of interest is determined monitoring the tool (pointer position), as shown in FIG 5.)
As per claim 17, Geode teaches the claimed:
17. The method of claim 3, wherein the determining the region of interest comprises executing a segmentation algorithm to identify an area of a body, a predetermined part of the body or an organ or a structure within the displayed medical image.
(Geode [0081]: “In this example, the calculations performed to identify a specific ROI that contains anatomy that is defined by properties such as blood flow, abnormal anatomy, cellular structure, morphology (structural, physiological or spatial heterogeneity)”.)
Claims 2, 8, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Geode in view of Higa and in further view of Reicher (US 20180060533 A1).
As per claim 2, Geode alone does not explicitly teach the remaining claim limitations.
However, Geode in combination with Reicher teaches the claimed:
2. The method of claim 1, wherein the at least one annotational data element is first displayed at an initial position in common with the received medical image and the calculating includes adapting the initial position to a target position.
(Reicher [0048]: “In some embodiments, the reporting application 65 include one more tools for adding an annotation to a displayed medical image, such as a toolbar that includes icons associated with different shapes and sizes of marks that a reader can select (drag and drop) and add to a displayed image.”
Reicher [0144]: “In some embodiments, the rules also specify what values may be added to an annotation. For example, a rule may specify particular categories of values that may be added to an annotation, such as location, lesion characteristics, measurements, diagnosis, and the like.”
Reicher teaches that rules can be added to annotations, this indicates that since rules can be added to change locations of the annotations, there was an initial position of the annotation, and the rules that indicate a change in location will result in the annotation adapting the initial position to a target position, according to the rules.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the annotation rules as taught by Reicher with the system of Geode in order to update the annotation to specify what values to add or change, such as location or other characteristics of the annotation.
As per claim 8, Geode alone does not explicitly teach the claimed limitations.
However, Geode in combination with Reicher teaches the claimed:
8. The method of claim 1, wherein the outputting is based on at least one of predetermined properties for the at least one annotational data element or the received medical image.
(Reicher [0005]: “The set of functions also includes updating the first annotation displayed within the medical image to display the first annotation in a first manner … Updating the first annotation may include updating a color of the first annotation, a size of the first annotation, an animation of the first annotation, a graphic of the first annotation, or a combination thereof.”
Reicher teaches the updating of an annotation, which includes updating the annotation’s size, color, animation, etc. This updating implies there is an initial or predetermined property of the annotations. Therefore, without updating and leaving the annotations as is, the outputting of the annotation data elements will be based on an initial or predetermined property.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to update annotations as taught by Reicher with the system of Geode in order to ensure the displayed information reflects what the user desires and to make updates in response to any changes to the images and its presentation.
As per claim 18, Geode alone does not explicitly teach the claimed limitations.
However, Geode in combination with Reicher teaches the claimed:
18. The method of claim 6, wherein the adapting the size of the at least one annotational data element adapts a font size of a text.
(Reicher [0103]: “For example, when an annotation is mapped to a structured report, the reporting application 65 may update an annotation displayed within a medical image, such as by updating the color, size, font, animation, or graphic of the annotation”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to update annotations as taught by Reicher with the system of Geode in order to ensure the displayed information reflects what the user desires and to make updates in response to any changes to the images and its presentation.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Geode in view of Higa and in further view of Vaillant (US 20230021332 A1).
As per claim 5, Geode alone does not explicitly teach the claimed limitations.
However, Geode in combination with Vaillant teaches the claimed:
5. The method of claim 1, wherein the calculating includes,
calculating the position for outputting the at least one annotational data element in relation to the received medical image, based on a spatial distance between an initial position of the at least one annotational data element and the determined region of interest.
(Vaillant [0069]: “… determine a distance between the projected first virtual point and the projected second virtual point on the second image, display the annotation on the second image via the display at the determined 2D location in response to the distance being less than a threshold distance, and display the second image via the display without the annotation in response to the distance being greater than or equal to the threshold distance.”
Vaillant teaches the displaying of the annotation if the threshold is less than a certain distance, indicating that if there is enough distance display the annotation, if not then don’t display it because it will overlap the region of interest.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to calculate distance of annotation data elements as taught by Vaillant with the system of Geode in order to minimize overlap of the annotation on the images and improve the visibility of relevant content.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Geode in view of Higa in view of Kleinferchner (US 20160275060 A1) and in further view of Reicher (US 20180060533 A1).
As per claim 9, Geode alone does not explicitly teach the claimed limitations.
However, Geode in combination with Kleinferchner and Reicher teaches the claimed:
9. The method of claim 1, wherein the calculating includes,
determining a ranking for multiple annotational data elements, and wherein the calculating is based on the determined ranking.
(Kleinferchner [0019]: “An “annotation” refers to an element of electronic content (e.g. a piece of text, an image, or a piece of video) and can contain one or more of the following: … an importance level that ranks the annotation relative to other annotations”
Reicher [0144]: “In some embodiments, the rules also specify what values may be added to an annotation. For example, a rule may specify particular categories of values that may be added to an annotation, such as location, lesion characteristics, measurements, diagnosis, and the like.”
Kleinferchner teaches the annotation that contains an importance level, and Reicher teaches the annotation rules that can be used to specify certain categories, such as measurements. Therefore, using the importance level of Kleinferchner and creating a set of rules of Reicher to calculate the output of the annotations on the images based on the importance level measurement, teaches determining a ranking for multiple annotation data elements and calculating them based on those ranking.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the importance level of annotation data elements as taught by Kleinferchner with the system of Geode in order to prioritize crucial and more relevant annotations to improve clarity and usability for the user, and to reduce cognitive overload of information that isn’t as important. Also to use the annotation rules as taught by Reicher with the system of Geode in order to update the annotation to specify what values to add or change, such as location or other characteristics of the annotation.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Geode in view of Higa and in further view of O’Conner (US 20210375435 A1).
As per claim 13, Geode alone does not explicitly teach the claimed limitations.
However, Geode in combination with O’Conner teaches the claimed:
13. The method of claim 1, wherein the appearance of an annotation data element is adapted in reply to a zoom level of the medical image.
(O’Conner [0040]: “… the position and/or shape of annotation may be automatically adjusted based on the position and/or shape of a corresponding anatomical feature in the underlying image … The bounding box may define the position (e.g., centroid), size and overall basic dimensions of the anatomical feature (e.g., the largest possible height and width based on the underlying image). The annotation shape may be automatically positioned and scaled based on the bounding box,”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to automatically adjust annotations as taught by O’Conner with the system of Geode in order to ensure the displayed images and annotation data elements are scalable and to be able to make updates to the annotation data elements in response to any changes to the images.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA SUO whose telephone number is (571)272-8387. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA JUNGWOOK SUO/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights