DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-20 are pending.
This communication is in response to the communication filed July 21, 2023.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite systems and methods for making medical decisions based on multimodal data, which are statutory categories of inventions.
Specifically, independent claim 1 recites: receive respective types of patient data and generate respective encoded representations of the types of patient data, wherein the types of patient data include at least a first type of patient data comprising a description of symptoms experienced by a patient, and a second type of patient data comprising one or more of a test result of the patient, a medical history of the patient, or demographic information about the patient…receive the encoded representations and generate an output based on the encoded representations, wherein the output includes at least one of a medical decision, a medical summary, or a medical questionnaire associated with the patient.
Independent claim 14 recites encoding…multiple types of patient data into respective encoded representations, wherein the multiple types of patient data include at least at least a first type of patient data comprising a description of symptoms experienced by a patient, and a second type of patient data comprising one or more of a test result of the patient, a medical history of the patient, or demographic information about the patient; and generating an output based on the encoded representations, wherein the output is generated…and comprises at least one of a medical decision, a medical summary, or a medical questionnaire associated with the patient.
The limitations directed to encoding data and generating encoded data are interpreted as being grouped within the mental processes grouping of abstract ideas, because they may involve using a computation model and mapping textual data to respective standardized alphanumeric codes. The totality of the limitations, including the limitations directed to receiving data and generating an output of the data including a medical decision, summary, or questionnaire are interpreted as being grouped within the certain methods of organizing human activity grouping of abstract ideas because the claims involve a series of steps for receiving data, analyzing it based on various encoding and decoding, and outputting the results as a medical decision. The claims are interpreted to recite concepts relating to tracking or organizing medical information. The processes involve managing human behavior in the healthcare space. See MPEP 2106.04. Accordingly, the claims recite an abstract idea.
The dependent claims further recite limitations directed to the abstract idea stated above. The dependent claims describe types of patient data used, types of medical images used, receiving medical questions and generating medical answers, determining highest reward value of a medical decision, obtaining additional patient data, implement a large language model, and recommending a medical scan. The claims are interpreted to recite concepts relating to tracking or organizing medical information. The processes involve managing human behavior in the healthcare space. See MPEP 2106.04. Accordingly, the claims recite an abstract idea. The claims recite additional elements that are not interpreted as part of the abstract idea, and are addressed below.
The judicial exception is not integrated into a practical application. Integration into a practical application requires an additional element or a combination of additional elements in the claim to apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the exception. The additional elements include computers, storage devices, various kinds of artificial neural networks, and implementation and training of neural networks.
The claims merely use the additional elements as tools to perform abstract ideas and generally link the use of a judicial exception to a particular technological environment. The use of the additional elements as tools to implement the abstract idea and generally to link the use of the abstract idea to a particular technological environment does not render the claim patent eligible, because it requires no more than a computer performing functions that correspond to acts required to carry out the abstract idea. Specifically, the computers may include storage devices, processors, and artificial neural networks functioning to receive, encode, decode, generate, process, and output data and may be any suitable device capable of executing the functions of the claims (specification p. 13).
The additional elements do not show an improvement to the functioning of a computer or to any other technology, rather the additional elements perform general computing functions and do not indicate how the particular combination improves any technology or provides a technical solution to a technical problem. See Apple v. Ameranth, 842 F.3d 1229, 1240 (Fed. Cir. 2016). The additional elements do not use the exception to affect a particular treatment or prophylaxis for a disease, do not apply the exception using particular machines, and do not effect a transformation or reduction of a particular article to a different state or thing, rather the computer elements are generally stated as to their structure and function and are only used to make decisions instead of directly providing specific treatment or prophylaxis. Therefore, the additional elements do not impose any meaningful limits on practicing the abstract idea and the additional limitations are not indicative of materializing into a practical application. Accordingly, the claim is directed to an abstract idea.
Generic computer elements recited as performing generic computer functions that are well-understood, routine, or conventional activities amount to no more than implementing the abstract idea with a computerized system (Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network and performing repetitive calculations); Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) ("The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims."); See MPEP 2106.05(d) and July 2015 Update: Section IV). Here, the claim limitations directed to receiving and outputting data are similar to receiving and sending information over a network and the limitations directed to encoding and decoding are similar to a computer performing repetitive calculations.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because, as discussed above with respect to integration of the abstract idea into a practical application, the additional elements of using computers, storage devices, and neural networks to perform the steps of the claims amount to no more than using computer related devices to automate or implement the abstract idea of making medical decisions based on a multimodal data.
The use of a computer or processor to merely automate or implement the abstract idea cannot provide significantly more than the abstract idea itself. (See MPEP 2106.05(f) where mere instructions to apply an exception does not render an abstract idea patent eligible). There is no indication that the additional limitations alone or in combination improves the functioning of a computer or any other technology, improves another technology or technical field, or effects a transformation or reduction of a particular article to a different state or thing. Moreover, the claim limitations may be implemented by any suitable device capable of executing the functions of the claims (specification p. 13). The neural networks are recited at a high level of generality and perform functions related to basic computer functions as stated above. Therefore, the claims are not patent eligible.
In conclusion, the claims are directed to the abstract idea of making medical decisions based on a multimodal data. The claims do not provide an inventive concept, because the claims do not recite additional elements or a combination of elements that amount to significantly more than the judicial exception of the claims. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology, and the collective functions merely provide conventional computer implementation. Therefore, whether taken individually or as an order combination, the claims are nonetheless rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yerebakan et al. US20250322641 in view of Sharma et al. US20210330269.
As per claim 1, Yerebakan teaches
a system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to (Yerebakan par. 6, 225 teaches using computer-program product, computer-readable storage medium.)
implement an artificial neural network (ANN), wherein the ANN comprises: (Yerebakan par. 75-78 teaches inputting obtained text strings into a trained machine learning model. A trained machine learning model can comprise a neural network, a support vector machine, a decision tree and/or a Bayesian network, and/or the trained function can be based on k-means clustering, Qlearning, genetic algorithms and/or association rules. In particular, a neural network can be a deep neural network, a convolutional neural network or a convolutional deep neural network. Furthermore, a neural network can be an adversarial network, a deep adversarial network and/or a generative adversarial network.)
multiple encoder neural networks configured to receive respective types of patient data and generate respective encoded representations of the types of patient data, (Yerebakan par. 75, 187, 325 teaches the target medical image study and each of the candidate medical image studies are respectively comprise one or more attributes each having an attribute value comprising a text string indicating content of the medical image study. Where one or more of the first attribute values AV or second attribute values AV comprises a text string, generating the first vector FV or the second vector SV, respectively, may comprise encoding the text string into a vector representation.)
a second type of patient data comprising one or more of a test result of the patient, a medical history of the patient, or demographic information about the patient; and (Yerebakan par. 81 teaches non-image data may comprise information not directly related to the image data such as personal information of the patient such as gender, age, weight, insurance details, information about the treating physician)
generate an output based on the encoded representations, wherein the output includes at least one of a medical decision, a medical summary, or a medical questionnaire associated with the patient (Yerebakan par. 387, 545 teaches recommendations for follow up examinations and treatment options, interpreted as medical decisions).
Yerebakan teaches patient data and decoding, but does not specifically teach the following limitations met by Sharma,
wherein the types of patient data include at least a first type of patient data comprising a description of symptoms experienced by a patient, and (Sharma par. 28 teaches patient data including patient symptoms)
a decoder neural network configured to receive the encoded representations and (Sharma par. 49 teaches training a decoder network. The decoder network comprises a plurality of layers that recode features to generate reconstructed data representing a reconstruction of the training input data input into encoder network.)
It would have been obvious to one of ordinary skill in the art at the time the invention was filed to modify the systems and methods as taught by Yerebakan to use patient symptom data and decoder neural network to receive encoded representations as taught by Sharma with the motivation to improve predictions and diagnostics in healthcare. Visual assessment of the imaging by a radiologist to determine the qualitative information is subjective and often narrowly focused. Accordingly, such quantitative and qualitative information from the imaging data is currently underutilized for patient management of patients suspected of, or confirmed as, having COVID-19 (Sharma par. 3-4).
As per claim 2, Yerebakan and Sharma teach all the limitations of claim 1 and further teach wherein the types of patient data further include a third type of patient data comprising one or more medical images of the patient (Yerebakan par. 97, 221, 227 teaches using patient data including medical images).
As per claim 3, Yerebakan and Sharma teach all the limitations of claim 2 and further teach wherein the one or more medical images of the patient include multiple mammogram images of the patient that correspond to different views of a breast area of the patient, and wherein the output generated by the decoder neural network includes an indication of whether a medical abnormality exists in the breast area (Yerebakan par. 97, 221, 227 teaches using imaging systems for medical images including mammography, where feature vectors may be visual features, such as a visual manifestation of a medical abnormality, a pattern, an anatomy, a medical landmark and so forth as indicated by the image of the respective medical image series.).
As per claim 4, Yerebakan and Sharma teach all the limitations of claim 1 and further teach wherein the multiple encoder neural networks include a first encoder neural network and a second encoder neural network, the first encoder neural network configured to encode the first type of patient data into at least a first vector representing features of the first type of patient data, the second encoder neural network configured to encode the second type of patient data into at least a second vector representing features of the second type of patient data (Yerebakan par. 95-99 teaches the target medical image series, a first feature vector, obtaining, for the respective one of the candidate medical image series comprised in the reference medical study, a second feature vector, and determining a comparability metric indicative of the degree of comparability between the first feature vector and the respective second feature vector of the candidate medical image series. Feature vectors may be determined by encoding image data and non-image data associated with the respective medical image series using a particular encoding algorithm.).
As per claim 5, Yerebakan and Sharma teach all the limitations of claim 4 and further teach wherein the first encoder neural network and the second encoder neural network are trained jointly via contrastive learning and, during the training, first training data and second training data provided respectively to the first encoder neural network and the second encoder neural network are treated as a positive pair if the first training data and second training data belong to a same patient, and as a negative pair if the first training data and second training belong to different patients (Yerebakan par. 135-137 teaches various trained functions and networks. A triplet loss function is a loss function for machine learning algorithms where a reference (anchor) input is compared to a positive (truthy) input and a negative (false) input. The distance from the baseline (anchor) input to the positive input is minimized, and the distance from the reference input to the negative input is maximized. Transferred to the present case, the positive input could be a slice with verified similarity to a given reference slice. The similarity can, for instance, be verified by a user or be due to a verified adjacency of the slice to the reference slice in an image study. The latter has the advantage that training data can be obtained rather easily. The negative input may for instance be a slice that is less similar to the reference slice than the positive input. In particular, the negative input may be a slice that is not related to the reference slice, e.g., a slice from a different series or patient. A contrastive loss function, interpreted as contrastive learning, may be used instead of the triplet loss function.).
As per claim 6, Yerebakan and Sharma teach all the limitations of claim 1 and further teach wherein the decoder neural network includes a transformer neural network (Yerebakan par. 129, 543 teaches deep distance metric learning networks transforming data, interpreted as a transformer neural network).
As per claim 7, Yerebakan and Sharma teach all the limitations of claim 6 and further teach wherein the respective encoded representations of the types of patient data are concatenated into a combined representation and provided to the decoder neural network, the combined representation including separators that distinguish the respective encoded representations of the types of patient data (Yerebaken par. 276-279, 327 teaches each training text string may represent a concatenation of a plurality individual such text strings of different appropriate attributes of an image file. In any case, each training text string may be labelled with a body region to which the training text string corresponds. For example, in some examples, the training text string may be labelled with the actual body region represented by the medical imaging data of the file from which the training text strings originates, the actual body region being determined by an expert practitioner. Third vector may be generated for each of these three attribute values. Then, each of these three third vectors may be concatenated to generate the first vector FV or second vector SV as appropriate).
As per claim 8, Yerebakan and Sharma teach all the limitations of claim 1 and further teach wherein the decoder neural network is configured to implement a large language model (LLM) pre-trained for predicting the output based on the respective encoded representations of the types of patient data (Yerebakan par. 86, 544 teaches applying a model based prediction algorithm for disease progression or tumor growth with or without treatment based on at least some image element entries comprised in the region of interest. One evaluation function may comprise applying a model or predictive algorithm to image element entries belonging to a identified lesion or tumor to predict its structural development. The algorithm may further include additional data like patient specific data, e.g., age, gender, genetic information, or the like, which may be requested from the medical information system.).
As per claim 9, Yerebakan and Sharma teach all the limitations of claim 8 and further teach wherein, when executed by the one or more computers, the instructions stored in the one or more storage devices further cause the one or more computers to receive a medical question and generate an answer to the medical question based on the LLM (Yerebakan par. 471 teaches medical question and answer in a clinician environment using a suitably executed query, for a particular feature of interest (e.g., a particular anatomical feature), and one or more target image patches annotated with one or more words A-D matching the query may be extracted and, e.g., displayed to a radiologist.).
As per claim 10, Yerebakan and Sharma teach all the limitations of claim 1 and further teach wherein the decoder neural network is configured to determine a distribution of actions associated with respective reward values and select, from the distribution, a first action associated with a highest reward value as the medical decision for the patient (Yerebakan par. 149-151 teaches a policy network may be seen as a trained function following a certain learned policy for iteratively performing a plurality of actions in order to fulfill the learned task. The policy of the learned policy network may be defined as the ability to select a next action to select a candidate location to iteratively find a location corresponding to the region of interest such that the long-term reward is favored, interpreted as determining a distribution of actions.).
As per claim 11, Yerebakan and Sharma teach all the limitations of claim 10 and further teach wherein the decoder neural network is trained to learn the distribution of actions via reinforcement learning (Yerebakan par. 149-151 teaches implementing the learned policy network as policy networks thus may be seen as one embodiment of reinforcement learnt trained functions.).
As per claim 12, Yerebakan and Sharma teach all the limitations of claim 10 and further teach wherein, when executed by the one or more computers, the instructions stored in the one or more storage devices further cause the one or more computers to obtain additional patient data based on the first action selected by the decoder neural network, encode the additional patient data into an additional representation using at least one of the multiple encoder neural networks, and determine, using the decoder neural network, a second action for the patient based at least on the additional representation (Yerebakan par. 149-151 teaches iterative processes, where any data received after a first data and any action after a first action may be interpreted as additional data and a second action, respectively).
As per claim 13, Yerebakan and Sharma teach all the limitations of claim 1 and further teach wherein the medical decision associated with the patient includes a recommendation of a medical scan procedure for the patient (Yerebakan par. 545 teaches recommendations for follow up examinations, interpreted as medical scan procedure).
As per claim 14, Yerebakan and Sharma teaches a method, comprising: encoding, using respective encoder neural networks, multiple types of patient data into respective encoded representations, wherein the multiple types of patient data include at least at least a first type of patient data comprising a description of symptoms experienced by a patient, and a second type of patient data comprising one or more of a test result of the patient, a medical history of the patient, or demographic information about the patient; and generating an output based on the encoded representations, wherein the output is generated using a decoder neural network and comprises at least one of a medical decision, a medical summary, or a medical questionnaire associated with the patient (see claim 1 rejection).
As per claim 15, Yerebakan and Sharma teach all the limitations of claim 14 and further teach wherein the multiple types of patient data further include a third type of patient data comprising one or more medical images of the patient (see claim 2 rejection).
As per claim 16, Yerebakan and Sharma teach all the limitations of claim 15 and further teach wherein the one or more medical images of the patient include multiple mammogram images of the patient that correspond to different views of a breast area of the patient, and wherein the output generated by the decoder neural network includes an indication of whether a medical abnormality exists in the breast area (see claim 3 rejection).
As per claim 17, Yerebakan and Sharma teach all the limitations of claim 14 and further teach wherein the decoder neural network includes a transformer neural network (see claim 6 rejection).
As per claim 18, Yerebakan and Sharma teach all the limitations of claim 17 and further teach wherein the respective encoded representations of the multiple types of patient data are concatenated into a combined representation and provided to the decoder neural network, the combined representation including separators that distinguish the respective encoded representations of the multiple types of patient data (see claim 7 rejection).
As per claim 19, Yerebakan and Sharma teach all the limitations of claim 14 and further teach wherein the decoder neural network is configured to implement a large language model (LLM) pre-trained for predicting the output based on the respective encoded representations of the multiple types of patient data, and wherein generating the output based on the encoded representations comprises receiving a medical question and generating an answer to the medical question based on the LLM (see claims 8 and 9 rejections).
As per claim 20, Yerebakan and Sharma teach all the limitations of claim 14 and further teach wherein the output generated by the decoder neural network includes a recommendation of a medical scan procedure for the patient (see claim 13 rejection).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAY M. PATEL whose telephone number is (571)272-6793 and email is jay.patel2@uspto.gov. The examiner can normally be reached on Monday-Friday 8AM-4:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Peter H. Choi can be reached on (469)295-9171. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAY M. PATEL/Primary Examiner, Art Unit 3686
Prosecution Insights