DETAILED ACTION
This Office Action is responsive to Amendments and Arguments filed on May 13th, 2026. Claims 5 and 14 are cancelled, claims 1, 3, 6-8, 10, 12, 15 and 17-19 are pending and have been examined; hence this action is made FINAL.
Any objection/rejection not mentioned in this Office Action has been withdrawn by the Examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on August 22nd, 2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendments and Arguments
With regard to rejections made under 35 U.S.C. 103, Applicant argues, "Zhang relates to document processing to generate more accurate and informed document representations. In particular, the Office Action cites paragraphs [0045], [0058], and [0066] of Zhang… While the cited portions of Zhang may refer to ground truth data and other prediction data for training the document processing model, the cited portions of Zhang do not teach or suggest at least the training data is extracted 'using one or more AI (artificial intelligence) based systems' as required in amended claim 1. The cited portions of Zhang merely refer to ground truth data and other prediction data and are silent with regards to how such data is obtained.
Further, the cited portions of Zhang do not teach or suggest at least 'combining the token sequences with corresponding user-generated summaries' as recited in amended claim 1. Specifically, the cited portions of Zhang do not teach or suggest that the ground truth data is 'user-generated summaries.' The cited portions of Zhang are silent with regards to such 'user-generated summaries.'
In addition, the cited portions of Zhang do not teach or suggest that 'the large language model is constrained to a radiology domain' as recited in amended claim 1. The cited portions of Zhang merely disclose a document processing model and are silent with regards to the document processing model being 'constrained to a radiology domain,'" (starting on page 8 of Remarks).
Applicant’s argument is not persuasive. Zhang teaches the elements of now cancelled claims 5 and 14, which appear in independent claims 1, 10 and 15. Those elements are directed to the training and fine-tuning of an artificial intelligence or machine learning system, narrowed only by their field of endeavor. As the applicants are aware, a reference outside of the field reference outside of the field of endeavor is reasonably pertinent if a person of ordinary skill would have consulted it and applied its teachings when faced with the problem that the inventor was trying to solve.”
The independent claims as amended are taught by the combination of Jhaveri, Zhang and Batman, as the references establish methods for training a machine learning system that would have been obvious to a person having ordinary skill in the art of machine learning, and secondarily concerned with training that system to operate on medical information. The claims offer no novel or unexpected outcomes over the elements as taught by the references.
Further, Specification paragraph [0029] recites, “In one embodiment, the LLM may be constrained by supervised fine-tuning to convert training tabulated data to text files or token sequences, which are combined with corresponding ground truth user-generated summaries to form input-output pairs,” which aligns with the ground-truth training methods as taught by Zhang wherein “the models may be trained on ground truth data, manually generated document representations, or other sets of data” (paragraph [0075]). Accordingly, the rejections under 35 U.S.C. 103 are maintained. Further details are provided below.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
Claim 10 recites “means for receiving one or more prompts…” “means for generating a response…” “means for outputting the response,” “means for receiving a question…” “means for generating an additional response…” and “means for outputting the additional response.” Claim 14 recites “means for converting training tabulated data to token sequences,” “means for combining the token sequences with corresponding ground truth summaries,” and “means for fine-tuning the large language model…”. Because these claim limitations are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof (see paragraphs [0032], [0067] and [0074]).
PLEASE NOTE: This is NOT a rejection. Please do not address this section as a rejection. Should Applicant disagree with the INTERPRETATION, applicant may: (1) amend the claim limitation(s) to avoid them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6, 8, 10, 15, 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2021/0287783 to Jhaveri (hereinafter, "Jhaveri") in view of U.S. Patent Application Publication 2022/0230714 to Batman et al. (hereinafter, "Batman"), further in view of U.S. Patent Application Publication 2023/0222285 to Zhang et al. (hereinafter, “Zhang”).
Regarding claims 1, 10 and 15, Jhaveri teaches a method, apparatus and computer readable medium comprising receiving one or more prompts comprising 1) patient data retrieved from one or more patient databases and 2) instructions, the patient data comprising tabulated measurements and one or more medical images (paragraph [0148], "The workflow tracker dashboard may be a graphical user interface that facilitates display of patient medical information, as further shown and described with respect to FIGS. 11-13, including: medical history; acquired patient images; detailed acquisition information related to acquired patient images (e.g., where and when the images were acquired, the protocol used in acquisition, who acquired the images, the ordering provider, etc.); contact information for all verified care providers; location information for all current users; completed steps of a current user's workflow with regard to the patient (e.g., images reviewed, patient added to the worklist, etc.); and a patient worklist updated in real-time based on the workflow of current users that may include next steps," paragraphs [0150]-[0151], "Further, verified care providers may send requests to the via the communication thread for various information related to the patient care, including patient medical history, care guidelines, predicted future patient state, recommended lab tests, etc… In some examples, the messages may be automatically generated by a virtual healthcare assistant (VHA) using a data mining algorithm as the patient's worklist is continually updated in real-time. The messages sent may be stored and executed on the server system, a cloud, and/or a remote device," and paragraph [0153], "At 1016, method 1000 includes outputting the communication thread and/or dashboard for display when prompted. In an example, the prompt may include an explicit request to view the communication thread and/or workflow tracker dashboard for the patient, entered by selection of an appropriate link/control button on the workflow tracker dashboard or selection of the patient's communication thread from a collaborative interface, as indicated at 1018."); and
outputting the response (paragraph [0153], "For example, a message button may be displayed via the collaborative interface, and selection of the message button may trigger display of the communication thread and/or workflow tracker dashboard for that patient. In another example, a patient link may be selected to launch the communication thread or dashboard from a collaborative system interface.").
Jhaveri does not explicitly teach “transforming the tabulated measurements into a sequence of tokens using a large language model,” “encoding the one or more medical images into embeddings using a machine learning based encoder network,” “aligning the embeddings with a text embedding function of the large language model,” or “generating the response based on the sequence of tokens and the aligned embeddings using the large language model,” and thus, Zhang is introduced.
Zhang teaches generating a response summarizing the patient data based on the instructions by: transforming the tabulated measurements into a sequence of tokens using a large language model (paragraph [0111], "The textual input contents can be tokenized by a WordPiece tokenizer," and paragraph [0105], "The block type can include the semantic type of the content presented in the block, such as header, paragraph, images, list (bullet-items), and table. In some implementations, the document tokenizer can define 14 different block types (e.g., header, paragraph, list, table, image, caption, and padding block)."),
encoding the one or more medical images into embeddings using a machine learning based encoder network (paragraph [0112], "The image contents can be first fed to a convolutional neural network (CNN), followed by a transformation fully-connected (FC) layer to align the resulting visual embedding to the same size of the textual token embedding."),
aligning the embeddings with a text embedding function of the large language model (paragraph [0112], "The image contents can be first fed to a convolutional neural network (CNN), followed by a transformation fully-connected (FC) layer to align the resulting visual embedding to the same size of the textual token embedding."), and
generating the response based on the sequence of tokens and the aligned embeddings using the large language model (paragraph [0123], "In some implementations, the document-level model 208 can also be a visual-linguistic transformer. The resulting document-level representation can be used to generate document summaries.").
Jhaveri and Zhang are considered analogous because they are each concerned with information retrieval and reporting. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Jhaveri with the teachings of Zhang for the purpose of improving record accessibility. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Jhaveri also does not explicitly teach training a language model “wherein the large language model is constrained to a radiology domain by: extracting training tabulated clinical measurements from training medical images using one or more AI (artificial intelligence) based systems, converting the training tabulated clinical measurements to token sequences, combining the token sequences with corresponding user-generated summaries,” or “fine-tuning the large language model based on the combined token sequences and corresponding user-generated summaries,” thus, Zhang is further referenced.
Zhang teaches extracting training tabulated clinical measurements from training medical images using one or more AI (artificial intelligence) based systems (paragraph [0037], "For example, the system or method can first parse a document into blocks whose content can be textual, tabular, or multimedia (e.g. images) using a proprietary tool. The system can include a hierarchical framework to encode the blocks. The encoder model can encode each block with a multimodal transformer in the lower-level and aggregates the block-level representations and connections utilizing a specifically designed transformer at the higher-level. The system can be trained on hierarchical pre-training objectives, where the lower-level model, or block-level, can be trained with the standard masked language modeling (MLM) loss and the multimodal alignment loss (i.e. image-text matching), and the higher-level model, or document-level, can be trained with three layout-aware objectives: (1) block-order predictions, (2) masked block predictions (i.e. block MLM), and (3) image fitting predictions."),converting the training tabulated clinical measurements to token sequences (paragraph [0045], "The systems or methods may use a tokenizer for the data processing. In some implementations, the system may use a block tokenizer. The block tokenizer can process block positions, block types (e.g., header, image and text, paragraph, etc.), block attributes (e.g., bold, italic, font size, etc.), and images. For example, the systems and methods may utilize a go/block-tokenizer for block and layout processing. The tokenizer may be able to characterize the data into sub-groups (e.g., header, lists, paragraph, image, table, etc.) based on block type."),combining the token sequences with corresponding user-generated summaries (paragraph [0058], "Ground truth data and other prediction data may be used to evaluate the HVL transformer prediction. The training may be run iteratively to allow the model to make accurate predictions. "), andfine-tuning the large language model based on the combined token sequences and corresponding user-generated summaries (paragraph [0066], "In some implementations, the method can include evaluating a pre-training loss function that evaluates a difference between the prediction output and ground truth data associated with the masked training block and the plurality of additional masked training blocks. The method can include adjusting one or more parameters of the machine-learned semantic document encoding model based at least in part on the pre-training loss function.").
Jhaveri and Zhang are considered analogous because they are each concerned with information retrieval and reporting. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the learning model of Jhaveri with the training methods of Zhang for the purpose of improving machine learning performance. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
The combination of Jhaveri and Zhang does not explicitly teach “receiving a question relating to the response from a user,” “generating an additional response responding to the question using the large language model,” or “outputting the additional response,” and thus, Batman is introduced.
Batman teaches receiving a question relating to the response from a user (paragraph [0012], "Thus, if the second care provider requests information about the patients beyond what is summarized in the handoff report, the handoff system can provide the requested information.");generating an additional response responding to the question using the large language model (paragraph [0012], "The workflow system may generate answers to the inquiries based on the patient data and may provide the answers to the clinical device."); andoutputting the additional response (paragraph [0012], "Thus, if the second care provider requests information about the patients beyond what is summarized in the handoff report, the handoff system can provide the requested information.").
Jhaveri, Zhang and Batman are considered analogous because they are each concerned with processing medical information. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the reporting system of Jhaveri and Zhang with the recurrent, additional responses of Batman for the purpose of improving user experience and summary quality. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Regarding claims 6 and 17, Jhaveri teaches method and computer readable medium wherein the patient data is retrieved from a plurality of patient databases (paragraph [0040], "Thus, the user may easily find current (in real-time) and previous information related to patient care from collaborative care providers/users as well as identify where the information came from thereby saving time and increasing operational efficiency. The infrastructure 200 includes a hospital information system (HIS) 204, a radiology information system (RIS) 206, a picture archiving and communication system (PACS) 208, an interface unit 210, a data center 212, and a workstation 214.").
Regarding claims 8 and 19, Jhaveri teaches a method and computer readable medium wherein the one or more patient databases comprise at least one of EHR (electronic health record), EMR (electronic medical record), PHR (personal health record), HIS (health information system), RIS (radiology information system), PACS (picture archiving and communication system), and LIMS (laboratory information management system) (paragraph [0040], "Thus, the user may easily find current (in real-time) and previous information related to patient care from collaborative care providers/users as well as identify where the information came from thereby saving time and increasing operational efficiency. The infrastructure 200 includes a hospital information system (HIS) 204, a radiology information system (RIS) 206, a picture archiving and communication system (PACS) 208, an interface unit 210, a data center 212, and a workstation 214.").
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Jhaveri, Zhang and Batman as applied to claims 1 and 10 above, and further in view of "Automated Quantification of CT Patterns Associated with COVID-19 from Chest CT" by Chaganti et al. (hereinafter, "Chaganti").
Regarding claims 3 and 12, the combination of Jhaveri, Zhang and Batman does not teach “the patient data comprises information automatically extracted from medical images using an artificial intelligence based system,” and thus, Chaganti is introduced.
Chaganti teaches the patient data comprises information automatically extracted from medical images using an artificial intelligence based system (see Materials and Methods, Lung and Lobe Segmentation, "Next, the lung ROI image is resampled to a 2-mm isotropic volume and fed into a Deep Image-to-Image Network (DI2IN) (16) to generate the lung segmentation.").
Jhaveri, Zhang, Batman and Chaganti are considered analogous because they are each concerned with processing and reporting information. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Jhaveri, Zhang and Batman with the teachings of Chaganti for the purpose of improving record accessibility. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Claims 7, 9, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Jhaveri, Zhang and Batman as applied to claims 1, 10 and 15 above, and further in view of "Artificial Intelligence for Multimodal Data Integration in Oncology" by Lipkova et al. (hereinafter, "Lipkova").
Regarding claims 7 and 18, the combination of Jhaveri, Zhang and Batman does not teach “the patient data comprises unstructured data using different nomenclature,” and thus, Lipkova is introduced.
Lipkova teaches the patient data comprises unstructured data using different nomenclature (see Challenges and Clinical Adoption, Alignment of diverse modalities, "This refers to the integration of data from different scales, time points, or measurements. Often an acquisition of one modality results in the destruction of the sample, preventing collection of multiple measurements from the same system... Here, cross-modal autoencoders can be used to enable integration and translation between arbitrary modalities.").
Jhaveri, Zhang, Batman and Lipkova are considered analogous because they are each concerned with processing and reporting information. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Jhaveri, Zhang and Batman with the teachings of Lipkova for the purpose of improving record accessibility. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Regarding claims 9 and 20, the combination of Jhaveri, Zhang and Batman does not teach “the large language model is constrained to a specific medical domain,” however, Lipkova teaches the large language model is constrained to a specific medical domain (Introduction, "AI models are able to integrate complementary information and clinical context from diverse data sources to provide more accurate patient predictions (Figure 1A) (Hosny et al., 2018)… In this review, we summarize AI methods and strategies for multimodal data fusion, outline prospective on AI driven exploration through multimodal associations and interpretability methods, and conclude with directions for AI adoption in precision oncology.").
Jhaveri, Zhang, Batman and Lipkova are considered analogous because they are each concerned with processing and reporting information. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Jhaveri, Zhang and Batman with the teachings of Lipkova for the purpose of improving language model accuracy. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Patent 9,003,319 to Linthicum et al.
U.S. Patent 9,715,576 to Hayter, II.
U.S. Patent 12,136,481 to Baker et al.
U.S. Patent Application Publication 2014/0324469 to Reiner.
U.S. Patent Application Publication 2019/0147988 to Sharifi Sedeh et al.
U.S. Patent Application Publication 2019/0287012 to Celikyilmaz et al.
U.S. Patent Application Publication 2020/0335187 to Lefkofsky et al.
U.S. Patent Application Publication 2021/0398650 to Baker et al.
U.S. Patent Application Publication 2022/0310219 to Sanchez, Jr. et al.
U.S. Patent Application Publication 2023/0187031 to White et al.
U.S. Patent Application Publication 2023/0237773 to Li et al.
U.S. Patent Application Publication 2023/0259544 to Sotudeh Gharebagh et al.
U.S. Patent Application Publication 2024/0290065 to Park et al.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEAN T SMITH whose telephone number is (571)272-6643. The examiner can normally be reached Monday - Friday 8:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, PIERRE-LOUIS DESIR can be reached at (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEAN THOMAS SMITH/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
Prosecution Insights