DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of claims 1-9 and 16-20 in the reply filed on April 10, 2026 is acknowledged.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1,2,3, 6, 8, 9, 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20200161005 to Lyman in view of “Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework” to Zhang.
Regarding claim 1, Lyman discloses a computer-implemented method for processing digital medical images to infer metadata from those images (paragraph 71, 117-118; system 108 can process medical image data 410 to generate inference data (metadata)), the method comprising:
receiving a plurality of digital medical images (paragraph 71, 118; see Fig. 6a; medical scan image data 410 can be plural of images; system 108 receives medical images 410)
receiving a prompt, the prompt being a request for a specific type of metadata to be inferred from the plurality of digital medical images (paragraph 121, 124, 125; user is prompted to edit scan category and this edited scan category results in request to generate interference data 1110 using selected inference function 1105 based on the scan category; in Fig. 6a medical scan image 410 is input to the inference function 1105 to generate inference data 1110; paragraph 118; inference data 1110 output by inference function 1105 relates to specific type of metadata such as “medical codes”, “diagnosis data”);
determining, using a trained model, at least one feature descriptor from the plurality of digital medical images (paragraph 142, 144, 145; model trained based model parameter data 1355 to determine output feature vectors (feature descriptor) from the medical scan images in Fig. 7a) based on the prompt (paragraph 142; output feature vector include diagnosis data which can be inferred data based on user prompt); and
providing for output the at least one feature descriptor for each of the plurality of digital medical images (paragraph 145; output feature vectors based on medical images 410).
However Lyman does not disclose determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images.
Zhang discloses determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images (page 02, second column, last three paragraph; MMIF is foundation model which includes CCPViT which extracts feature (feature descriptor) from 2D medical images (biomedical data)).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Lyman as taught by Zhang to provide foundation model for determining features.
The motivation to combine the references is to provide highly precise foundation model that uses both image and text modality when trained can provide encoded features of the image and eliminate the misalignment of label data (page 02, second column; page 13, first column).
Regarding claim 2, Lyman discloses the method of claim 1, wherein the plurality of digital medical images include at least one of whole slide image (WSI), hematoxylin and eosin (H&E) stains, immunohistochemistry (IHC) slides, immunofluorescent slides, or CT scans (paragraph 71; medical scan image includes CT scan).
Regarding claim 3, Lyman discloses the computer-implemented method of claim 1, wherein the metadata comprises any combination of supplemental medical images, structured diagnostic reports, unstructured free text reports, genomic data, proteomic data, treatment data, responses, or diagnoses (paragraph 121; inference data 1110 includes metadata such as diagnosis data 440 and related “text for medical reports”).
Regarding claim 6, Lyman discloses the computer-implemented method of claim 1, further comprising: determining, using a content-based retrieval system, a collection of related digital medical images or cases based on the metadata associated with each of the digital medical images or cases (paragraph 248; metadata such as DICOM Id of metadata in scan data (digital medical images) used to query medical picture archive system 2620 (content-based retrieval system) to fetch corresponding scan (collection of image)).
Regarding claim 8, Lyman discloses the computer-implemented method of claim 1, further comprising: determining, using a downstream task model, output targets that were not contained within metadata types based on the at least one feature descriptor for each of the plurality of digital medical images (paragraph 145, 148-149, 150; “abnormality classification” based on another model is downstream task model that determines classification of abnormality (output target) that is not contained in metadata such as specific type of metadata such as “medical codes”, “diagnosis data”; classification is based on matrix 1371 determined from output vector (feature) of the images).
Regarding claim 9, Lyman discloses the computer-implemented method of claim 8, wherein the output targets include any combination of learning markers for drug response, building a model to replicate an existing biomarker, learning a novel biomarker from test data or other ground-truth indicators, or predicting additional disease states or diagnostics (paragraph 145, 148-149, 150; “abnormality classification” based on another model is downstream task model that determines/predicts classification of abnormality (output target); paragraph 80-81, 150; abnormality classification includes populating classification data 445 that includes “Stable Disease”, or “Progressive Disease” as disease states).
Regarding claim 16, Lyman discloses a system for processing digital medical images to infer metadata from those images (paragraph 121, 124, 125; request to generate inference data 1110 using selected inference function 1105 based on the scan category; in Fig. 6a medical scan image 410 is input to the inference function 1105 to generate inference data 1110), the system comprising:
at least one memory storing instructions; and at least one processor configured to execute the instructions to perform operations comprising (paragraph 58; processor and memory storing instructions for execution):
receiving a plurality of digital medical images (paragraph 71, 118; see Fig. 6a; medical scan image data 410 can be plural of images; system 108 receives medical images 410);
receiving a prompt, the prompt being a request for a specific type of metadata to be inferred from the plurality of digital medical images (paragraph 121, 124, 125; user is prompted to edit scan category and this edited scan category results in request to generate interference data 1110 using selected inference function 1105 based on the scan category; in Fig. 6a medical scan image 410 is input to the inference function 1105 to generate inference data 1110; paragraph 118; inference data 1110 output by inference function 1105 relates to specific type of metadata such as “medical codes”, “diagnosis data”);
determining, using a trained model, at least one feature descriptor from the plurality of digital medical images (paragraph 142, 144, 145; model trained based model parameter data 1355 to determine output feature vectors (feature descriptor) from the medical scan images in Fig. 7a) based on the prompt (paragraph 142; output feature vector include diagnosis data which can be inferred data based on user prompt); and
providing for output the at least one feature descriptor for each of the plurality of digital medical images (paragraph 145; output feature vectors based on medical images 410).
However Lyman does not disclose determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images.
Zhang discloses determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images (page 02, second column, last three paragraph; MMIF is foundation model which includes CCPViT which extracts feature from 2D medical images (biomedical data)).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Lyman as taught by Zhang to provide foundation model for determining features.
The motivation to combine the references is to provide highly precise foundation model that uses both image and text modality when trained can provide encoded features of the image and eliminate the misalignment of label data (page 02, second column; page 13, first column).
Regarding claim 17, see rejection of claim 2.
Regarding claim 18, see rejection of claim 3.
Regarding claim 19, see rejection of claim 8.
Regarding claim 20, see rejection of claim 9.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20200161005 to Lyman in view of “Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework” to Zhang further in view of US 20210065859 to McKinney.
Regarding claim 5, Zhang discloses a trained foundation model (page 02, second column, last three paragraph; MMIF is foundation model which includes CCPViT which extracts feature from 2D medical images (biomedical data)). However Lyman does not disclose the computer-implemented method of claim 1, further comprising: receiving free text; and providing for output, from the trained model, a structured synoptic diagnostic report based on the plurality of digital medical images and free text.
McKinney discloses further comprising: receiving free text; and providing for output, from the trained model, a structured synoptic diagnostic report based on the plurality of digital medical images and free text (paragraph 59; trained model 700; paragraph 63, 67-68; based on free-text 904 and images 906, structured label is output which includes diagnosis presence/absence (synoptic diagnostic report)).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Lyman as taught by McKinney to provide free text as input to the model to generate diagnostic report.
The motivation to combine the references is to generate from unstructured free-text associated with images, a structured label data for the images that clearly indicate present/absence of diagnosis for plurality of conditions as output of the model (paragraph 10-14).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20200161005 to Lyman in view of “Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework” to Zhang further in view “Content Repurposing Platform Utilizing Metadata Extracted from Rich Media” to Kojima.
Regarding claim 7, Lyman does not disclose the computer-implemented method of claim 6, further comprising: receiving content-based constraints, the content-based constraints comprising instructions to include or exclude specific types of metadata, or attributes of that metadata, to a query for content retrieval.
Kojima discloses receiving content-based constraints, the content-based constraints comprising instructions to include or exclude specific types of metadata, or attributes of that metadata, to a query for content retrieval (page 950; second column last two paragraph; page 956; second column, second paragraph; query condition (content-based constraints) is received that includes metadata used to search metadata database in order to acquire “related medical images” (content retrieval)).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Lyman as taught by Kojima to provide query instructions for retrieving contents.
The motivation to combine the references is to provide automated retrieval of contents such as medical images by using a metadata database and querying this database using metadata which can reduce data preparation time (page 956, first column under Experimental Results and second column).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20200161005 to Lyman in view of “Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework” to Zhang further in view of US 20170344711 to Liu.
Regarding claim 4, Zhang discloses using a trained foundation model (page 02, second column, last three paragraph; MMIF is foundation model which includes CCPViT which extracts feature from 2D medical images (biomedical data)).
However Lyman in view of Zhang does not disclose the computer-implemented method of claim 1, further comprising:
receiving at least one query constraint, the query constraints including judgments or hypotheses from a clinician or expert; and
providing for output, from the trained model, metadata estimations that are consistent with the at least one query constraint.
Liu discloses receiving at least one query constraint, the query constraints including judgments or hypotheses from a clinician or expert; and providing for output, from the trained model, metadata estimations that are consistent with the at least one query constraint (paragraph 15-19, 35; user query are correlated to expert keywords associated with “real doctors' knowledge” (judgments or hypotheses from a clinician or expert); Expert knowledge graph model 140 provides for output “medical conditions” as metadata estimation that are consistent with the expert keywords (query constraint)).
It would have been obvious to one of ordinary skill in the art at the time of the invention was made to modify the system of Lyman as taught by Liu to provide query constraint to model to output metadata estimation.
The motivation to combine the references is to provide automated and accurate prediction such as diagnosis of user input query using a model that can be refined and scaled accordingly based on new knowledge (paragraph 15-18).
Other Prior Art Cited
14. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220358651 to Matsuki discloses diagnosis system.
US 20190066835 to Lyman discloses x-ray diagnosis system.
US 20230056923 to Hsieh discloses medical image processing.
US 11922348 to Lyman discloses medical scan image processing.
JP 2022126373 to Suzuki discloses medical image diagnosing system.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENIYAM MENBERU whose telephone number is (571) 272-7465. The examiner can normally be reached on Monday-Friday, 10:00am-6:30pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi Sarpong can be reached on (571) 270-3438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the customer service office whose telephone number is (571) 272-2600. The group receptionist number for TC 2600 is (571) 272-2600.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
For more information about the PAIR system, see <http://pair-direct.uspto.gov/>. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Patent Examiner
Beniyam Menberu
/BENIYAM MENBERU/Primary Examiner, Art Unit 2681
06/12/2026
Prosecution Insights