DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claim 20 recites “A computer program product comprising a computer readable medium storing instructions that are executable to perform a method . . ..” Applicant's disclosure does not define the claimed “computer readable medium.” The United States Patent and Trademark Office (USPTO) is obliged to give claims their broadest reasonable interpretation consistent with the specification during proceedings before the USPTO (see In re Zletz, 893 F.2d 319 Fed. Cir. 1989). The broadest reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent (see MPEP 2111.01). Thus, the plain meaning Applicant’s “”computer readable medium” in the claim and disclosure provides an open ended listing of computer-readable mediums fails to limit the claim to only non-transitory tangible media, and therefore is non-statutory (see 1351 Off. Gaz. Pat. Office 212 (February 23, 2010)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-12, 14-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (“ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models”) in view of Zhang et al. (“Multimodal Chain-of-Thought Reasoning in Language Models”). Both references were provided through Applicant’s IDS.
Regarding Claim 1, Wang teaches A medical information processing apparatus (“ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models” Wang Title.) comprising processing circuitry configured to:
input medical image data (Wang Fig. 1:
PNG
media_image1.png
180
220
media_image1.png
Greyscale
) generated by a medical imaging apparatus (X-ray machine) to a first generative model (Wang Fig. 1:
PNG
media_image2.png
412
366
media_image2.png
Greyscale
) that is configured to generate one or more findings (Wang Fig. 1:
PNG
media_image3.png
362
276
media_image3.png
Greyscale
) based on the medical image (Wang Figs. 1, 7);
inpute.g., Fig. 1: “What medicine should I take?”), and the generated one or more findings ( “The descriptions, served as a link between visual and linguistic information, are combined as inputs to a large language model (LLM).” Wang Fig. 1.) to a second generative model (Wang Fig. 1
PNG
media_image4.png
170
94
media_image4.png
Greyscale
) that is configured to output answer information (Wang Fig. 1, the answer about medication) related to the medical image data (the medication is suggested for the suspected condition in the X-ray image as shown Fig. 1); and
display, on a display device, the answer information (e.g., answer about the medication in Fig. 1) and at least part of the medical image (X-ray image in Wang Figs. 1, 7) generated by the medical imaging apparatus (X-ray machine).
Wang does not explicit disclose, but Zhang teaches also inputting the medical image data to the second generative model (
Zhang 4.1 Framework Overview:
PNG
media_image5.png
78
642
media_image5.png
Greyscale
The same Xvision is provided for both the first and second model, wherein the second model is the answer inference model.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s answer inference model with Wang. One of ordinary skill in the art would be motivated to provide stronger AI results. “We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information.” Zhang Abstract.
Claims 17 and 19-20 are substantially similar to Claim 1. The rejection analyses based on Wang in view of Zhang for Claim 1 is also applied to Claims 17 and 19-20.
In addition, Claim 17 recites “A medical imaging diagnostic apparatus comprising: a scanner configured to scan a patient to obtain a medical image data set; . . .” ( Wang 1 Introduction:
PNG
media_image6.png
136
466
media_image6.png
Greyscale
).
In addition, Claim 19 recites “A medical information processing method comprising: . . .” (Wang Fig. 1 shows a process.).
In addition, Claim 20 recites “A computer program product comprising a computer readable medium storing instructions that are executable to perform a method comprising: . . .” (“ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models” Wang Title.).
Regarding Claim 2, Wang further teaches The apparatus of claim 1, wherein the generated one or more findings comprise at least one of:
at least part of a radiology report;
position information representing position of one or more landmarks, or one or more anatomical or other features of interest;
segmentations of one or more anatomical or other features of interest (Wang Fig. 1
PNG
media_image7.png
118
304
media_image7.png
Greyscale
);
measurements of one or more anatomical or other features of interest (Wang Fig. 1
PNG
media_image8.png
118
206
media_image8.png
Greyscale
);
one or more selected parts of the medical image; or
one or more enhanced images or derived images obtained based on the medical image.
Regarding Claim 3, Wang further teaches The apparatus of claim 1,
wherein the generated one or more findings (Wang Fig. 1) comprise at least one of text (Wang Fig. 1:
PNG
media_image3.png
362
276
media_image3.png
Greyscale
), co-ordinates, at least one segmentation mask, or at least one image.
Regarding Claim 4, Wang further teaches The apparatus of claim 1, wherein the apparatus comprises a display system configured to display the answer information and at least part of the medical image on the same screen (Wang Fig. 7, where a user’s interactions with the system are displayed).
Regarding Claim 7, Wang further teaches The apparatus of claim 1, wherein the processing circuitry or the second generative model is configured to use a predetermined format (Prompt text format) for the generated one or more findings (as shown in Wang Fig. 1).
Regarding Claim 8, Wang further teaches The apparatus of claim 1, wherein the processing circuitry or the second generative model is configured to include or exclude information of type of information from the at least one finding based on expected accuracy of the information or type of information (
Wang 2.2. Vision-Language Model:
PNG
media_image9.png
358
668
media_image9.png
Greyscale
, wherein the score indicates the expected accuracy, which is used to include or exclude information used for the model).
Regarding Claim 9, Wang further teaches The apparatus of claim 1, wherein the processing circuity is configured to vary the medical image data (Wang Fig. 7 showing two different X-ray input images) and/or to vary other inputs to the first generative model (Wang Fig. 1) and/or to use a plurality of different or differently-trained first models (Wang Fig. 1:
PNG
media_image2.png
412
366
media_image2.png
Greyscale
) thereby to obtain a plurality of findings (Wang Fig. 1).
Regarding Claim 10, Wang further teaches The apparatus of claim 9, wherein the plurality of findings comprises an ensemble of multiple predictions for a given item (patient’s lung) (Wang Fig. 1:
PNG
media_image2.png
412
366
media_image2.png
Greyscale
).
Regarding Claim 11, Wang further teaches The apparatus of claim 9, wherein the plurality of findings (Wang Fig. 1:
PNG
media_image2.png
412
366
media_image2.png
Greyscale
) have at least some different information content (three different information contents) (Fig. 7, where different findings for different patients.).
Regarding Claim 12, Wang further teaches The apparatus of claim 1, wherein the processing circuitry is configured to generate a plurality of possible questions and corresponding answers from the generated findings (Fig. 7, where multiple questions are asked and multiple answers are provided.).
Regarding Claim 14, Wang further teaches The apparatus of claim 1, wherein the medical image data comprises a multi-modal data set that includes semantic or other additional information (Wang Fig. 1
PNG
media_image10.png
208
160
media_image10.png
Greyscale
), and
the inputting of the medical image to the first generative model (Wang Fig. 1
PNG
media_image10.png
208
160
media_image10.png
Greyscale
PNG
media_image2.png
412
366
media_image2.png
Greyscale
) comprises inputting the multi-modal data set to the first generative model (e.g., “medical exam image” and complaint text).
Regarding Claim 15, Wang further teaches The apparatus of claim 1, wherein one or both of the first generative model or second generative model comprises at least one of: a GPT model or other transformer network; or a large language model (LLM) (ChatGPT, Wang Title, Abstract.).
Regarding Claim 16, Wang in view of Zhang teaches The apparatus of claim 1,
wherein inputting of the medical image to the first generative model comprises inputting a vector representation of the medical image data to the first generative model (
Zhang 4.2 Model Architecture:
PNG
media_image11.png
238
996
media_image11.png
Greyscale
); and/or
wherein inputting the medical image data, question information, and the generated one or more findings to the second generative model comprises inputting at least one of the medical image, question information, and the generated one or more findings as a vector representation.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s answer inference model l with Wang. One of ordinary skill in the art would be motivated to provide stronger AI results and/or to reduce the amount of input for a machine learning model. “We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information.” Zhang Abstract.
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zhang as applied to Claim 4, in further view of Zhou et al. (US 20220130499 A1).
Regarding Claim 5, Wang in view of Zhang teaches The apparatus of claim 4.
Wang in view of Zhang does not explicitly disclose, but Zhou teaches wherein the processing circuitry is configured to select part of the medical image based on the answer information, and to display the selected part of the medical image based on the answer information (
“The answer unit 108 can generate an answer to the query 112. The answer unit 108 generates an answer token by token based at least in part on the user joint representation and the image-text representation 500. In some embodiments of the present invention, the answer unit 108 can alter the image 110 to highlight a target object of the query 112. The answer unit 108 can select an object based on the user joint representation to determine. The answer unit 108 can further visually alter the object for highlighting purposes on a user's graphical user interface. For example, the answer unit 108 can alter the image pixels to change a color of the object, add a border to the object alter the image pixels of the balance of the image 110 (e.g., blur the rest of the image 110). This allows a potential user to feel confident that a generated answer is in response to the query 112.” Zhou ¶ 42.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhou’s highlighted image with Wang in view of Zhang. One of ordinary skill in the art would be motivated to provide the user with more informative answers with pictures and/or highlights.
Regarding Claim 6, Wang in view of Zhang and Zhou teaches The apparatus of claim 4, wherein the processing circuitry is configured to select one or more display parameters based on the answer information, and to display at least part of the medical image in accordance with the selected one or more display parameters (
“The answer unit 108 can generate an answer to the query 112. The answer unit 108 generates an answer token by token based at least in part on the user joint representation and the image-text representation 500. In some embodiments of the present invention, the answer unit 108 can alter the image 110 to highlight a target object of the query 112. The answer unit 108 can select an object based on the user joint representation to determine. The answer unit 108 can further visually alter the object for highlighting purposes on a user's graphical user interface. For example, the answer unit 108 can alter the image pixels to change a color of the object, add a border to the object alter the image pixels of the balance of the image 110 (e.g., blur the rest of the image 110). This allows a potential user to feel confident that a generated answer is in response to the query 112.” Zhou ¶ 42.
The display parameters include the highlight related parameter(s).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhou’s highlighted image with Wang in view of Zhang. One of ordinary skill in the art would be motivated to provide the user with more informative answers with pictures and/or highlights.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zhang as applied to Claim 12, in further view of Berg (US 20250298721 A1).
Regarding Claim 13, Wang in view of Zhang teaches The apparatus of claim 12.
Wang in view of Zhang does not explicitly disclose, but Berg teaches wherein the processing circuitry is configured to select one or more of the possible questions (Berg ¶ 98: suggested follow-up questions) based on how well they match the question information (Berg ¶ 98: initial input query), and to output the answers (Wang Figs. 1, 7) that correspond to the selected one or more questions (
“Each path may include a follow up question. The follow up questions are suggested paths that the user may select as next steps following the initial input query. The paths may be presented to the user as suggestions. The user may also decide to enter their own follow up question.” Berg ¶ 98.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Berg’s recommended questions with Wang in view of Zhang. One of ordinary skill in the art would be motivated to make it easier for a user by guiding the user with inquiries.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zhang as applied to Claim 17, in further view of Reicher et al. (US 20230335261 A1).
Regarding Claim 18, Wang in view Zhang teaches A medical imaging diagnostic apparatus according to claim 17.
Wang in view Zhang does not explicitly disclose, but Reicher teaches wherein the processing circuitry is configured to obtain at least one stored previously-obtained medical image data set (earlier version of medical images) and input the previously-obtained medical image data set to the first generative model and/or to the second generative model for comparison between the medical image data set and the previously-obtained medical image data set (
“In some embodiments, annotations and descriptions from serial imaging exams in the same patient are tracked to compare the annotation and descriptions in the same anatomic region, thus increasing the efficiency of developing AI algorithms that compare medical images to assess changes over time.” Reicher ¶ 66.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Reicher’s comparison of medical images in time series with Wang in view of Zhang. One of ordinary skill in the art would be motivated to better investigate a patient’s condition’s progression. “In some embodiments, annotations and descriptions from serial imaging exams in the same patient are tracked to compare the annotation and descriptions in the same anatomic region, thus increasing the efficiency of developing AI algorithms that compare medical images to assess changes over time.” Reicher ¶ 66.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611
Prosecution Insights