DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claims 1-20 are drawn to a method, a system and a non-transitory computer-readable storage medium, which is/are statutory categories of invention (Step 1: YES).
Independent claim 1 recites determining a feature in the image; determining by inputting the feature and at least a portion of the patient data as input, a confidence level associated with a first disease; determining based on the confidence level being higher than a threshold, a recommendation for screening of the patient based on the first disease.
Independent claim 10 recites determining a feature in the image; determining, by inputting the feature and at least a portion of the patient data as input, a confidence level associated with a first disease; determining, based on the confidence level being higher than a threshold, a recommendation for screening of the patient based on the first disease.
Independent claim 16 recites determine a feature in the image; determine, by inputting the feature and at least a portion of the EMR data as input, a confidence level associated with a first disease; and determine, based on the confidence level being higher than a threshold, a recommendation for screening of the patient based on the first disease.
The respective dependent claims 2-9, 11-15, and 17-20, but for the inclusion of the additional elements specifically addressed below, provide recitations further limiting the invention of the independent claim(s).
Said recited limitations, as drafted, under their broadest reasonable interpretation, cover certain methods of organizing human activity, as reflected in the specification, which states that having “detecting potential diseases based on retinal images and patient health records, and providing recommendations for further screening related to the detected diseases” (see: specification paragraph 2). If a claim limitation, under its broadest reasonable interpretation, covers managing personal behavior or relationships or interactions between people, then it falls within the “Certain Methods of Organizing Human Activity” grouping of abstract ideas. The present claims cover certain methods of organizing human activity because they address problems where “a primary care doctor may not be able to review the retinal scans for signs of disease” and “manual analysis of retinal scans of a patient by a retina specialist, in addition to adding to a cost and complexity of a health screening, may also fail to flag early signs of a disease because the retina specialist may not be familiar with a patient's overall medical history” (see: specification paragraph 3). These problems are addressed by an invention “to screen a patient for a host of potential diseases automatically” (see: specification paragraph 4), and particularly, the recited limitations address these problems “to a disease identification system programmed or otherwise configured to generate a recommendation for further screening” (see: specification paragraph 17). Accordingly, the claims recite an abstract idea(s) (Step 2A Prong One: YES).
This judicial exception is not integrated into a practical application. The claims are abstract but for the inclusion of the additional elements including an “by the processor…by the processor and…to a machine learning (ML) model…by the processor and…” (claim 1), “by the processor…by the processor, a training dataset…updating, by the processor, the ML model by re-training with the augmented training dataset…” (claim 4), “the ML model is trained, based on a training dataset…” (claim 6), “the training dataset…” (claim 7), “the ML model comprises an expert system…” (claim 9), “memory; a processor; and computer-executable instructions stored in the memory and executable by the processor to perform operations comprising:…to a machine learning (ML) model…” (claim 10), “wherein the ML model is trained, based on a training dataset…” (claim 11), “the training dataset…” (claim 12), “a training dataset…and updating the ML model by re-training with the augmented training dataset” (claim 13), “the ML model…in a training dataset…” (claim 14), “A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed, cause one or more processors to:…to a machine learning (ML) model…” (claim 16), “the ML model is trained based on a training dataset…,” (claim 17), and “the ML model…in a training dataset…” (claim 18), which are additional elements that are recited at a high level of generality (e.g., the “processor” is configured through no more than a statement than that functions are performed “by” said processor; the “memory” and “processor” are configured through no more than a statement than that “instructions” stored in said memory are “executable” by said processor to perform operations; the “machine learning (ML) model” is “trained” through no more than a statement than that training is “based on a training dataset” to produce desired results, and that “re-training” is performed “with the augmented training dataset”; the “non-transitory computer-readable storage medium” is configured through no more than a statement than that stored “instructions” stored by said medium “cause” one or more processors to perform functions “when executed”) such that they amount to no more than mere instruction to apply the exception using generic computer elements. See: MPEP 2106.05(f).
The claims recite the additional elements of “receiving, by a processor, an image of a retina of an eye of a patient; receiving, by the processor and from an electronic medical record (EMR) of the patient, patient data corresponding to the patient…” (claim 1), “wherein the anonymized patient data and the corresponding images of the retina are extracted from an electronic medical records (EMR) system” (claim 8), “receiving an image of a retina of an eye of a patient; receiving, from an electronic medical record (EMR) of the patient, patient data corresponding to the patient…” (claim 10), and “receive, from an optical imaging device, an image of a retina of an eye of a patient; access, from an electronic medical record (EMR) storage, EMR data of the patient…” (claim 16), which are nominal or tangential additions to the abstract idea(s) and amount to extra-solution activity concerning mere data gathering. The addition of an insignificant extra-solution activity limitation does not impose meaningful limits on the claims such that is it not nominally or tangentially related to the invention. In the claimed context, these claimed additional elements are incidental to the performance of the recited abstract idea(s) as outlined in the recitations above. Similarly, the claims recite the additional elements of “providing, by the processor and to an output device, an output indicating the recommendation…” (claim 1) and “providing, to the EMR of the patient, an output indicating the recommendation…” (claim 10), which are considered insignificant post-solution activity concerning an insignificant application, and similarly, the addition of an insignificant extra-solution activity does not impose meaningful limits on the claims such that is it not nominally or tangentially related to the invention. In the claimed context, these claimed additional elements are incidental to the performance of the recited abstract idea(s) as outlined in the recitations above. See: MPEP 2106.05(g).
The combination of these additional elements is no more than mere instructions to apply the exception using generic computer elements and limitations directed toward extra-solution activity. Accordingly, even in combination, these additional elements do not integrate the abstract idea(s) into a practical application because they do not impose any meaningful limits on practicing the abstract idea(s). Accordingly, the claims are directed to an abstract idea(s) (Step 2A Prong Two: NO).
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea(s) into a practical application, using the additional elements to perform the abstract idea(s) amounts to no more than mere instructions to apply the exception using generic components. Mere instructions to apply an exception using generic components cannot provide an inventive concept. See MPEP 2106.05(f).
Further, the claimed additional elements directed toward extra-solution activity, identified above, are not sufficient to amount to significantly more than the judicial exception because they are generic components that are configured to perform well-understood, routine, and conventional activities previously known to the industry. See: MPEP 2106.05(d). Said additional elements are recited at a high level of generality and provide conventional functions that do not add meaningful limits to practicing the abstract idea(s). The originally filed specification supports this conclusion:
Paragraph 23, where “In some examples, the EMR system 110 may be connected to a clinical device 114 via a network 116. The clinical device 114 can include a computing device, such as a device including at least one processor configured to perform operations. In some cases, the operations are stored in memory in an executable format. Examples of computing devices include a personal computer, a tablet computer, a smart television (TV), a mobile device, a mobile phone, or an Internet of Things (loT) device. In some examples, the clinical device 114 may be operated by the operator 104, and may receive the image(s) 108 captured by the optical imaging device 106…In examples, the clinical device 114 or the optical imaging device 106 may store the image(s) 108 of the eye(s) of the patient 102 in the EMR system 110 in association with the EMR data 112 of the patient 102.”
Paragraph 24, where “In examples, the network 116 may represent one or more communication networks. Examples of communication networks include at least one wired interface (e.g., an ethernet interface, an optical cable interface, etc.) and/or at least one wireless interface (e.g., a BLUETOOTH interface, a WI-FI interface, a near-field communication (NFC) interface, a LongTerm Evolution (LTE) interface, a New Radio (NR) interface, etc.). In some examples, data or other signals may be transmitted between elements of FIG. 1 over a wide area network (WAN), such as the Internet. In some cases, the data may include one or more data packets (e.g., Internet Protocol (IP) data packets), datagrams, or a combination thereof.”
Paragraph 25, where “In various examples, the clinical device 114 may be connected, via the network 116, to a remote computing device 118, such as a server implemented on a cloud platform. In examples, the clinical device 114 may upload, to the remote computing device 118, the image(s) 108 captured by the optical imaging device 106. The remote computing device 118 may implement an image analysis system 120 that receives, as an input, the image(s) 108 captured by the optical imaging device 106…In other examples, the optical imaging device 106 may be in direct communication with the remote computing device 118 to upload the image(s) 108...”
Paragraph 34, where “In examples, the remote computing device 118 may also implement an EMR data extractor component 122. The EMR data extractor component 122 may process the EMR data 112 to extract data of interest…”
Paragraph 36, where “…receive, from the clinical device 114 and/or directly from the EMR system 110, the EMR data 112 associated with the patient 102…”
Paragraph 38, where “…where the recommendation may be output, via a user interface, to the operator 104…”
Paragraph 42, where “…Additionally, although FIG. 1 illustrates the optical imaging device 106, the EMR system 110, the clinical device 114, and the remote computing device 118 as separate entities, in some implementations, one or more of these entities may be correspond to the same computing device.”
Paragraph 46, where “…the EMR data extractor 122 may issue a query to the EMR system 110 (e.g., to a database storing EMR information) requesting anonymized data, and may receive, in response, the EMR data 206, and associations with corresponding data of a same patient in the retinal images 204…”
Paragraph 71, where “…In examples, the recommendation may be provided to the patient and/or a healthcare provider caring for the patient, and may be added to the electronic health record(s) of the patient.”
Paragraph 80, where “…The memory 602 may include volatile and nonvolatile memory and/or removable and non-removable media implemented in any type of technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data.”
Paragraph 81, where “As described herein, the processor(s) 604, can be a single processing unit or a number of processing units, and can include single or multiple processing cores, comprising a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or both CPU and GPU, or other processing unit known in the art.)…”
Paragraph 82, where “The device(s) 600 can also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 6 by removable storage 606 and non-removable storage 608. Tangible computer-readable media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. The memory 602, removable storage 606, and non-removable storage 608 are all examples of computer-readable storage media. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Discs (DVDs), Content-Addressable Memory (CAM), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the device(s) 600. Any such tangible computer-readable media can be part of the device(s) 600.”
Paragraph 84, where “As illustrated in FIG. 6, the device(s) 600 can also include one or more wired or wireless transceiver(s) 614. For example, the transceiver(s) 614 can include a Network Interface Card (NIC), a network adapter, a LAN adapter, or a physical, virtual, or logical address to connect to the various base stations or networks (e.g., the network 116) contemplated herein, for example, or the various user devices and servers. To increase throughput when exchanging wireless data, the transceiver(s) 614 can utilize Multiple-Input/Multiple-Output (MIMO) technology. The transceiver(s) 614 can include any sort of wireless transceivers capable of engaging in wireless, Radio Frequency (RF) communication. The transceiver(s) 614 can also include other wireless modems, such as a modem for engaging in Wi-Fi, WiMAX, Bluetooth, or infrared communication.”
The claims recite the additional elements directed to pre-solution and post-solution activity, as recited and indicated above, each of which amount to extra-solution activity. The specification (e.g., as excerpted above) does not indicate that the additional element(s) provide anything other than well‐understood, routine, and conventional functions when claimed in a merely generic manner (as they are presently). See: MPEP 2106.05(g).
Further, the concepts of receiving or transmitting data over a network, such as using the Internet to gather data, storing and retrieving information in memory, extracting data, and presenting information have been identified by the courts as well-understood, routine, and conventional activities. See: MPEP 2106.05(d)(II).
Viewing the limitations as an ordered combination, the claims simply instruct the additional elements to implement the concept described above in the identification of abstract idea(s) with routine, conventional activity specified at a high level of generality in a particular technological environment. Hence, the claims as a whole, considering the additional elements individually and as an ordered combination, do not amount to significantly more than the abstract idea(s) (Step 2B: NO).
Dependent claim(s) 2-9, 11-15, and 17-20, when analyzed as a whole, considering the additional elements individually and/or as an ordered combination, are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitation(s) fail(s) to establish that the claim(s) is/are not directed to an abstract idea(s) without significantly more. These claims fail to remedy the deficiencies of their parent claims above, and are therefore rejected for at least the same rationale as applied to their parent claims above, and incorporated herein.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8 and 10-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over WO 2023/214890 to Rezaei in view of U.S. Patent Application Publication 2020/0388386 to Sharma.
As per claim 1, Rezaei teaches a method, comprising:
receiving, by a processor (see: Rezaei, Fig. 1; and paragraph 48, is met by the processing system 1002 may have processing facilities represented by one or more processors 1004. memory 1006, and other components typically present in such computing environments. In the exemplary embodiment illustrated the memory 1006 stores information accessible by processor 1004, the information comprising instructions 1008 that may be executed by the processor 1004 and data 1010 that may be retrieved, manipulated or stored by the processor 1004), an image of a retina of an eye of a patient (see: Rezaei, Fig. 2A; and paragraph 56, is met by at input stage 2002 one or more fundus images are received - for example a collection of fundus photographs of an individual);
receiving, by the processor and, patient data corresponding to the patient (see: Rezaei, Fig. 2A; and paragraph 73, is met by meta-information of an individual associated with the one or more fundus images is received and that meta-information includes gender, ethnicity, HbAlc, TCHDL, etc. Some of the meta-information is categorical data such as gender, ethnicity, deprivation value, medicine, and other is numerical data such as age, HbAlc, etc);
determining, by the processor, a feature in the image (see: Rezaei, Fig. 2A; and paragraph 67-69 and 73, is met by good quality images with related labels pass through a plurality of Al models. These Al models include sets of risk contributing factor (RCF) CNNs 201 0a to 201 On that are trained to detect indicators of: glycaemic control, blood pressure, cholesterol, and exposure to smoking. These indicators include, but are not limited to: drusen appearance, clustering, and/or location; pigmentation change in density and/or location; arteriovenous crossing; change in arteriovenous crossing calibre and/or thickness change; arteriovenous tortuosity; retinal oedema size and/or pattern; and/or microaneurysms concentration. Moreover, these outputted features are then concatenated to form a one-dimensional array (i.e. the individual-level fundus image feature vector) at 2016);
determining, by the processor and by inputting the feature and at least a portion of the patient data as input to a machine learning (ML) model, a confidence level associated with a first disease (see: Rezaei, Fig. 2A; and paragraph 74-75, is met by the processing pipelines of fundus images 2016 and meta-information 2020 are completed, the individual-level fundus image feature vector and meta-information vector are concatenated together to form an individual feature vector at step 2022 and that the individual feature vector is processed by a CVD risk prediction neural network model 2024 utilizing a fully connected neural network (FCNN). See also Figs. 4A-D);
determining, by the processor and based on the confidence level being higher than a threshold, a recommendation for screening of the patient based on the first disease (see: Rezaei, Fig. 4D; and paragraph 32 and 94, is met by instances in which the CVD risk is above a threshold (in this example 15%), a referral recommendation 4076 is included in the report - for example recommending a consultation with a cardiologist); and
providing, by the processor and to an output device, an output indicating the recommendation (see: Rezaei, Fig. 4D; and paragraph 89, is met by exemplary individual report; may be sent to a healthcare provider for further follow ups).
Rezaei fails to specifically teach receiving the patient data from an electronic medical record (EMR) of the patient; however, Sharma teaches patient data and/or medical image may be obtained from memory, such as a computerized patient record or a picture archiving and communications system (see: Sharma, paragraph 32 and 42).
It would have been obvious to one of ordinary skill in the art at the time the invention was field to modify the receiving of the metadata as taught by Rezaei to include obtaining patient data and/or medical image from a computerized patient record or a picture archiving and communications system as taught by Sharma with the motivation of providing patient-specific optimization (see: Sharma, paragraph 45).
As per claim 2, Rezaei and Sharma teach the invention as claimed, see discussion of claim 1, and further teach:
wherein the feature comprises at least one of: a brightness level of an optic disc of the retina, a diameter of blood vessels of the retina, a topology of the blood vessels of the retina, an edema of the optic disc, or an arteriovenous ratio (AVR) (see: Rezaei, paragraph 20, is met by a brightness adjustment process may be performed on the one or more fundus images; paragraph 24 and 67, is met by the CNNs may look for "localized" signs of biological changes and physiological changes (e.g. microaneurysms, oedema, etc.) changes, and or "global" changes in an image that could indicate presence of glycaemic control, blood pressure, cholesterol, and exposure to smoking (e.g. pigmentary changes in the peripapillary region, arterial/venous crossing deformations, vascular tortuosity changes, vascular calibre changes, etc.). In examples the signs may include, but not be limited to: drusen appearance, clustering, and/or location; pigmentation change in density and/or location; arteriovenous crossing; change in arteriovenous crossing calibre and/or thickness change; arteriovenous tortuosity; retinal oedema size and/or pattern; and/or microaneurysms concentration).
As per claim 3, Rezaei and Sharma teach the invention as claimed, see discussion of claim 1, and further teach:
wherein the patient data comprises at least one of: an age of the patient, a sex of the patient, a race of the patient, a smoking status of the patient, a blood pressure measurement of the patient, or one or more medical test results associated with the patient (see: Rezaei, paragraph 73 is met by gender, ethnicity, HbAlc, TCHDL, etc. Some of the meta-information is categorical data such as gender, ethnicity, deprivation value, medicine, etc, and other is numerical data such as age, HbAlc, etc.).
As per claim 4, Rezaei and Sharma teach the invention as claimed, see discussion of claim 1, and further teach:
receiving, by the processor, follow-up information indicating whether the patient was diagnosed with the first disease; augmenting, by the processor, a training dataset to include a data point comprising the follow-up information, the feature, and at least the portion of the patient data; and updating, by the processor, the ML model by re-training with the augmented training dataset (see: Rezaei, Fig. 4D; and paragraph 30, 32, 76, and 94, is met by instances in which the CVD risk is above a threshold (in this example 15%), and the training data includes labels for each individual as to whether they encounter a CVD event (e.g., heart failure) after the fundus images been taken and meta-information has been recorded. Therefore, we can measure the Al model predicted risk with the truth using the cross-entropy loss…the back-propagation method is used to calculate the gradients…the parameters are updated at the negative gradients direction).
As per claim 5, Rezaei and Sharma teach the invention as claimed, see discussion of claim 1, and further teach:
wherein the first disease comprises one of: obstructive sleep apnea (OSA), anemia, heart disease, kidney disease, multiple sclerosis (MS), or Alzheimer's disease (see: Rezaei, Fig. 4D; and paragraph 94, is met by kidney disease).
As per claim 6, Rezaei and Sharma teach the invention as claimed, see discussion of claim 1, and further teach:
wherein the ML model is trained, based on a training dataset, to identify, based on the image and the patient data as inputs, the confidence level associated with the first disease (see: Rezaei, Fig. 4D; and paragraph 30, 32, 76, and 94, is met by instances in which the CVD risk is above a threshold (in this example 15%), and the training data includes labels for each individual as to whether they encounter a CVD event (e.g., heart failure) after the fundus images been taken and meta-information has been recorded. Therefore, we can measure the Al model predicted risk with the truth using the cross-entropy loss).
As per claim 7, Rezaei and Sharma teach the invention as claimed, see discussion of claim 6, and further teach:
wherein the training dataset includes anonymized patient data and corresponding images of the retina associated with a plurality of patients, and an indication of normal health or one or more diseases associated with each respective patient (see: Rezaei, paragraph 54, is met by Because not all measurements are related to the CVD risk, irrelevant columns were discarded according to the expert advice. As a result, 35 columns corresponding to 21 fields remained, including: age, sex, ethnicity, deprivation score, family history, smoking, systolic blood pressure, BMI, TC/HDL, HbAlc, state of diabetes (Y /N), diabetic type, atrial fibrillation, anti hypertensives, antithrombotic medication, lipid lowering medication, eGFR, metolazone prior 6 months, lipids in prior 6 months, LLD prior 6 months, anticoagulation medication prior 6 months, antiplay prior 6 months, CVD event and date, etc.; and paragraph 96-99, is met by disclose that the UK Biobank data was used).
As per claim 8, Rezaei and Sharma teach the invention as claimed, see discussion of claim 7, and further teach:
wherein the anonymized patient data (see: Rezaei, Fig. 2A; and paragraph 73, is met by meta-information of an individual associated with the one or more fundus images is received and that meta-information includes gender, ethnicity, HbAlc, TCHDL, etc. Some of the meta-information is categorical data such as gender, ethnicity, deprivation value, medicine, and other is numerical data such as age, HbAlc, etc; paragraph 54, is met by Because not all measurements are related to the CVD risk, irrelevant columns were discarded according to the expert advice. As a result, 35 columns corresponding to 21 fields remained, including: age, sex, ethnicity, deprivation score, family history, smoking, systolic blood pressure, BMI, TC/HDL, HbAlc, state of diabetes (Y /N), diabetic type, atrial fibrillation, anti hypertensives, antithrombotic medication, lipid lowering medication, eGFR, metolazone prior 6 months, lipids in prior 6 months, LLD prior 6 months, anticoagulation medication prior 6 months, antiplay prior 6 months, CVD event and date, etc.; and paragraph 96-99, is met by disclose that the UK Biobank data was used) and the corresponding images of the retina are extracted (see: Rezaei, Fig. 2A; and paragraph 67-69 and 73, is met by good quality images with related labels pass through a plurality of Al models. These Al models include sets of risk contributing factor (RCF) CNNs 201 0a to 201 On that are trained to detect indicators of: glycaemic control, blood pressure, cholesterol, and exposure to smoking. These indicators include, but are not limited to: drusen appearance, clustering, and/or location; pigmentation change in density and/or location; arteriovenous crossing; change in arteriovenous crossing calibre and/or thickness change; arteriovenous tortuosity; retinal oedema size and/or pattern; and/or microaneurysms concentration. Moreover, these outputted features are then concatenated to form a one-dimensional array (i.e. the individual-level fundus image feature vector) at 2016).
Rezaei fails to specifically teach receiving the patient data and images from an electronic medical records (EMR) system; however, Sharma teaches patient data and/or medical image may be obtained from memory, such as a computerized patient record or a picture archiving and communications system (see: Sharma, paragraph 32 and 42).
It would have been obvious to one of ordinary skill in the art at the time the invention was field to modify the receiving of the metadata and images as taught by Rezaei to include obtaining patient data and/or medical image from a computerized patient record or a picture archiving and communications system as taught by Sharma with the motivation of providing patient-specific optimization (see: Sharma, paragraph 45).
As per claim 10, Rezaei teaches a system, comprising:
memory; a processor; and computer-executable instructions stored in the memory and executable by the processor to perform operations comprising (see: Rezaei, Fig. 1; and paragraph 48, is met by the processing system 1002 may have processing facilities represented by one or more processors 1004. memory 1006, and other components typically present in such computing environments. In the exemplary embodiment illustrated the memory 1006 stores information accessible by processor 1004, the information comprising instructions 1008 that may be executed by the processor 1004 and data 1010 that may be retrieved, manipulated or stored by the processor 1004):
receiving an image of a retina of an eye of a patient (see: Rezaei, Fig. 2A; and paragraph 56, is met by at input stage 2002 one or more fundus images are received - for example a collection of fundus photographs of an individual);
receiving patient data corresponding to the patient (see: Rezaei, Fig. 2A; and paragraph 73, is met by meta-information of an individual associated with the one or more fundus images is received and that meta-information includes gender, ethnicity, HbAlc, TCHDL, etc. Some of the meta-information is categorical data such as gender, ethnicity, deprivation value, medicine, and other is numerical data such as age, HbAlc, etc);
determining a feature in the image (see: Rezaei, Fig. 2A; and paragraph 67-69 and 73, is met by good quality images with related labels pass through a plurality of Al models. These Al models include sets of risk contributing factor (RCF) CNNs 201 0a to 201 On that are trained to detect indicators of: glycaemic control, blood pressure, cholesterol, and exposure to smoking. These indicators include, but are not limited to: drusen appearance, clustering, and/or location; pigmentation change in density and/or location; arteriovenous crossing; change in arteriovenous crossing calibre and/or thickness change; arteriovenous tortuosity; retinal oedema size and/or pattern; and/or microaneurysms concentration. Moreover, these outputted features are then concatenated to form a one-dimensional array (i.e. the individual-level fundus image feature vector) at 2016);
determining, by inputting the feature and at least a portion of the patient data as input to a machine learning (ML) model, a confidence level associated with a first disease (see: Rezaei, Fig. 2A; and paragraph 74-75, is met by the processing pipelines of fundus images 2016 and meta-information 2020 are completed, the individual-level fundus image feature vector and meta-information vector are concatenated together to form an individual feature vector at step 2022 and that the individual feature vector is processed by a CVD risk prediction neural network model 2024 utilizing a fully connected neural network (FCNN). See also Figs. 4A-D);
determining, based on the confidence level being higher than a threshold, a recommendation for screening of the patient based on the first disease (see: Rezaei, Fig. 4D; and paragraph 32 and 94, is met by instances in which the CVD risk is above a threshold (in this example 15%), a referral recommendation 4076 is included in the report - for example recommending a consultation with a cardiologist); and
providing, to the EMR of the patient, an output indicating the recommendation (see: Rezaei, Fig. 4D; and paragraph 89, is met by exemplary individual report; may be sent to a healthcare provider for further follow ups).
Rezaei fails to specifically teach receiving the patient data from an electronic medical record (EMR) of the patient; however, Sharma teaches patient data and/or medical image may be obtained from memory, such as a computerized patient record or a picture archiving and communications system (see: Sharma, paragraph 32 and 42).
It would have been obvious to one of ordinary skill in the art at the time the invention was field to modify the receiving of the metadata as taught by Rezaei to include obtaining patient data and/or medical image from a computerized patient record or a picture archiving and communications system as taught by Sharma with the motivation of providing patient-specific optimization (see: Sharma, paragraph 45).
As per claim 11, Rezaei and Sharma teach the invention as claimed, see discussion of claim 10, and further teach:
wherein the ML model is trained, based on a training dataset, to identify, based on the image and the patient data as inputs, the confidence level associated with the first disease (see: Rezaei, Fig. 4D; and paragraph 30, 32, 76, and 94, is met by instances in which the CVD risk is above a threshold (in this example 15%), and the training data includes labels for each individual as to whether they encounter a CVD event (e.g., heart failure) after the fundus images been taken and meta-information has been recorded. Therefore, we can measure the Al model predicted risk with the truth using the cross-entropy loss).
As per claim 12, Rezaei and Sharma teach the invention as claimed, see discussion of claim 11, and further teach:
wherein the training dataset includes anonymized patient data and corresponding images of the retina associated with a plurality of patients, and an indication of normal health or one or more diseases (see: Rezaei, paragraph 54, is met by Because not all measurements are related to the CVD risk, irrelevant columns were discarded according to the expert advice. As a result, 35 columns corresponding to 21 fields remained, including: age, sex, ethnicity, deprivation score, family history, smoking, systolic blood pressure, BMI, TC/HDL, HbAlc, state of diabetes (Y /N), diabetic type, atrial fibrillation, anti hypertensives, antithrombotic medication, lipid lowering medication, eGFR, metolazone prior 6 months, lipids in prior 6 months, LLD prior 6 months, anticoagulation medication prior 6 months, antiplay prior 6 months, CVD event and date, etc.; and paragraph 96-99, is met by disclose that the UK Biobank data was used).
As per claim 13, Rezaei and Sharma teach the invention as claimed, see discussion of claim 10, and further teach:
receiving follow-up information indicating whether the patient was diagnosed with the first disease; augmenting a training dataset to include a data point comprising the follow-up information, the feature, and at least the portion of the patient data; and updating the ML model by re-training with the augmented training dataset (see: Rezaei, Fig. 4D; and paragraph 30, 32, 76, and 94, is met by instances in which the CVD risk is above a threshold (in this example 15%), and the training data includes labels for each individual as to whether they encounter a CVD event (e.g., heart failure) after the fundus images been taken and meta-information has been recorded. Therefore, we can measure the Al model predicted risk with the truth using the cross-entropy loss…the back-propagation method is used to calculate the gradients…the parameters are updated at the negative gradients direction).
As per claim 14, Rezaei and Sharma teach the invention as claimed, see discussion of claim 10, and further teach:
wherein the ML model is based at least in part on determining, in a training dataset, a correlation between the first disease and the feature or the patient data (see: Rezaei, paragraph 68, is met by each CNN 2011 configured to produce a probability of the feature it is trained to look at; paragraph 76, is met by The model is trained using Adam optimizer with the back propagation algorithm, and crossentropy loss function to depict the predicted value with the target. The training data includes labels for each individual as to whether they encounter a CVD event (e.g., heart failure) after the fundus images been taken and meta-information has been recorded).
As per claim 15, Rezaei and Sharma teach the invention as claimed, see discussion of claim 10, and further teach:
wherein the first disease is one of: obstructive sleep apnea (OSA), anemia, heart disease, kidney disease, multiple sclerosis (MS), or Alzheimer's disease (see: Rezaei, Fig. 4D; and paragraph 94, is met by kidney disease).
As per claim 16, Rezaei teaches a non-transitory computer-readable storage medium storing processor-executable instructions that, when executed, cause one or more processors to (see: Rezaei, Fig. 1; and paragraph 48, is met by the processing system 1002 may have processing facilities represented by one or more processors 1004. memory 1006, and other components typically present in such computing environments. In the exemplary embodiment illustrated the memory 1006 stores information accessible by processor 1004, the information comprising instructions 1008 that may be executed by the processor 1004 and data 1010 that may be retrieved, manipulated or stored by the processor 1004):
receive, from an optical imaging device, an image of a retina of an eye of a patient (see: Rezaei, Fig. 2A; and paragraph 56, is met by at input stage 2002 one or more fundus images are received - for example a collection of fundus photographs of an individual);
access data of the patient (see: Rezaei, Fig. 2A; and paragraph 73, is met by meta-information of an individual associated with the one or more fundus images is received and that meta-information includes gender, ethnicity, HbAlc, TCHDL, etc. Some of the meta-information is categorical data such as gender, ethnicity, deprivation value, medicine, and other is numerical data such as age, HbAlc, etc);
determine a feature in the image (see: Rezaei, Fig. 2A; and paragraph 67-69 and 73, is met by good quality images with related labels pass through a plurality of Al models. These Al models include sets of risk contributing factor (RCF) CNNs 201 0a to 201 On that are trained to detect indicators of: glycaemic control, blood pressure, cholesterol, and exposure to smoking. These indicators include, but are not limited to: drusen appearance, clustering, and/or location; pigmentation change in density and/or location; arteriovenous crossing; change in arteriovenous crossing calibre and/or thickness change; arteriovenous tortuosity; retinal oedema size and/or pattern; and/or microaneurysms concentration. Moreover, these outputted features are then concatenated to form a one-dimensional array (i.e. the individual-level fundus image feature vector) at 2016);
determine, by inputting the feature and at least a portion of the EMR data as input to a machine learning (ML) model, a confidence level associated with a first disease (see: Rezaei, Fig. 2A; and paragraph 74-75, is met by the processing pipelines of fundus images 2016 and meta-information 2020 are completed, the individual-level fundus image feature vector and meta-information vector are concatenated together to form an individual feature vector at step 2022 and that the individual feature vector is processed by a CVD risk prediction neural network model 2024 utilizing a fully connected neural network (FCNN). See also Figs. 4A-D); and
determine, based on the confidence level being higher than a threshold, a recommendation for screening of the patient based on the first disease (see: Rezaei, Fig. 4D; and paragraph 32 and 94, is met by instances in which the CVD risk is above a threshold (in this example 15%), a referral recommendation 4076 is included in the report - for example recommending a consultation with a cardiologist).
Rezaei fails to specifically teach receiving the patient data from an electronic medical record (EMR) storage, EMR data; however, Sharma teaches patient data and/or medical image may be obtained from memory, such as a computerized patient record or a picture archiving and communications system (see: Sharma, paragraph 32 and 42).
It would have been obvious to one of ordinary skill in the art at the time the invention was field to modify the receiving of the metadata as taught by Rezaei to include obtaining patient data and/or medical image from a computerized patient record or a picture archiving and communications system as taught by Sharma with the motivation of providing patient-specific optimization (see: Sharma, paragraph 45).
As per claim 17, Rezaei and Sharma teach the invention as claimed, see discussion of claim 16, and further teach:
wherein the ML model is trained based on a training dataset comprising anonymized patient data and corresponding images of the retina associated with a plurality of patients, and an indication of normal health or one or more diseases of respective patients (see: Rezaei, paragraph 54, is met by Because not all measurements are related to the CVD risk, irrelevant columns were discarded according to the expert advice. As a result, 35 columns corresponding to 21 fields remained, including: age, sex, ethnicity, deprivation score, family history, smoking, systolic blood pressure, BMI, TC/HDL, HbAlc, state of diabetes (Y /N), diabetic type, atrial fibrillation, anti hypertensives, antithrombotic medication, lipid lowering medication, eGFR, metolazone prior 6 months, lipids in prior 6 months, LLD prior 6 months, anticoagulation medication prior 6 months, antiplay prior 6 months, CVD event and date, etc.; and paragraph 96-99, is met by disclose that the UK Biobank data was used).
As per claim 18, Rezaei and Sharma teach the invention as claimed, see discussion of claim 16, and further teach:
wherein the ML model is based at least in part on determining, in a training dataset, a correlation between the first disease and the feature or the EMR data (see: Rezaei, paragraph 68, is met by each CNN 2011 configured to produce a probability of the feature it is trained to look at; paragraph 76, is met by The model is trained using Adam optimizer with the back propagation algorithm, and crossentropy loss function to depict the predicted value with the target. The training data includes labels for each individual as to whether they encounter a CVD event (e.g., heart failure) after the fundus images been taken and meta-information has been recorded).
As per claim 19, Rezaei and Sharma teach the invention as claimed, see discussion of claim 16, and further teach:
wherein: the EMR data comprises at least one of: an age of the patient, a blood pressure measurement of the patient, or one or more medical test results associated with the patient, and the feature comprises at least one of: a brightness level of an optic disc of the retina, a diameter of blood vessels of the retina, an edema of the optic disc, or an arteriovenous ratio (AVR) (see: Rezaei, paragraph 73 is met by gender, ethnicity, HbAlc, TCHDL, etc. Some of the meta-information is categorical data such as gender, ethnicity, deprivation value, medicine, etc, and other is numerical data such as age, HbAlc, etc.).
As per claim 20, Rezaei and Sharma teach the invention as claimed, see discussion of claim 16, and further teach:
wherein the first disease comprises one of: obstructive sleep apnea (OSA), anemia, heart disease, kidney disease, multiple sclerosis (MS), or Alzheimer's disease (see: Rezaei, Fig. 4D; and paragraph 94, is met by kidney disease).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over WO 2023/214890 to Rezaei in view of U.S. Patent Application Publication 2020/0388386 to Sharma further in view of U.S. Patent Application Publication 2024/0029885 to Holden.
As per claim 9, Rezaei and Sharma teach the invention as claimed, see discussion of claim 1, and further teach:
wherein the ML model indicating rules correlating the feature and the patient data with a probability of occurrence of the first disease (see: Rezaei, Fig. 2A; and paragraph 74-75, is met by the processing pipelines of fundus images 2016 and meta-information 2020 are completed, the individual-level fundus image feature vector and meta-information vector are concatenated together to form an individual feature vector at step 2022 and that the individual feature vector is processed by a CVD risk prediction neural network model 2024 utilizing a fully connected neural network (FCNN). See also Figs. 4A-D).
Rezaei fails to specifically teach that the neural network model comprises an expert system; however, Holden teaches expert/AI system (hereafter referred to simply as the “expert system”) can include a traditional rules-based expert system and/or a machine learning system, the machine learning system including neural networks and/or knowledge graphs (see: Holden, paragraph 5, 44, and 61).
It would have been obvious to one of ordinary skill in the art at the time the invention was field to modify the neural network model as taught by Rezaei to include an expert/AI system as taught by Sharma with the motivation of to select an appropriate course of action (see: Sharma, paragraph 45).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT A SOREY whose telephone number is (571)270-3606. The examiner can normally be reached Monday through Friday, 8am to 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fonya Long can be reached at (571) 270-5096. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT A SOREY/Primary Examiner, Art Unit 3682