DETAILED ACTION
Response to Arguments
The amendments filed 11/17/2025 have been entered and made of record.
Applicant's amendments and corresponding arguments filed 11/17/2025 have been fully considered, but are moot in view of the new ground(s) of rejection because the Applicant has substantially amended at least independent claims 1, 9 and 16:
Re Claim 1, the newly added limitation “process a whole slide image associated with the patient using a convolutional neural network trained to output one or more of a probability of an input whole slide image being microbiome-high and a probability of the input whole slide image being microbiome-low” has been considered, however, which is rejected by KUNZ as modified by SEGAL, and further in view of a new reference Poore (US 20230332249 A1), because:
as discussed in the previous Office Action, KUNZ as modified by SEGAL discloses process a whole slide image associated with the patient using a convolutional neural network trained to output the quantities of the microbiomes, and determining the level of the quantities of the microbiomes of an input whole slide image, by comparing to a bias, or threshold (see SEGAL: e.g., -- a fully-connected neural network, each of the neurons in a particular layer is connected to and provides input value to those in the next layer. These input values are then summed and this sum compared to a bias, or threshold. If the value exceeds the threshold for a particular neuron, that neuron then holds a positive value which can be used as input to neurons in the next layer of neurons….. [0138] The machine learning procedure used according to some embodiments of the present invention is a trained machine learning procedure. A machine learning procedure can be trained according to some embodiments of the present invention by feeding a machine learning training program with microbiome data of a cohort of subjects from which the quantities of the metabolite have been determined by blood tests… t least a portion of the decision rules relate to one or more microbes in the microbiome. A simple decision rule may be a threshold for the amount of a particular microbes, but more complex rules, relating to more than one microbes are also contemplated. The machine learning training program also accumulates data at the leaves of the trees. The structures of the trees, the decision rules for the split nodes, and the data at the leaves are all selected by the machine learning training program, automatically and typically without user intervention, such that the microbiome data at the root of the trees provide the quantities of the metabolite as determined by blood tests at the leaves of the trees. --, in [0136]-[0139], and, Table. 1, -- analyzing the amount of each of the corresponding microbes set forth in Table 1 in the fecal microbiome of the subject,--, in [0146]-[0147], and [0385]-[0386]);
regarding to newly added limitation the Neural Network “output a probability of {the level of the quantities of} microbiomes on the whole slide image”, KUNZ as modified by SEGAL further disclose Neural Network “output a probability of {the level of the quantities of} microbiomes on the whole slide image” (see KUNZ: e.g., -- to train a machine learning system to predict the presence of any gene amplification from the training set in addition to the cause of death. Step 510 may utilize techniques described in step 410 in addition to training the machine learning to identify particular genes. The system may train a machine learning system to create a vector, where each potential value of the vector represents the chances of the individual having a particular cardiac Arrhythmia gene. The machine learning system may include a corresponding classification layer that outputs a vector of probabilities indicating particular percentage chances that particular genes exist in the individual. The classifier may be responsible for assigning probabilities of each potential Arrhythmia gene.--, in [0110]-[0112] {herein “outputs a vector of probabilities indicating…. chances that particular genes exist in the individual”, and, “assigning probabilities of each potential Arrhythmia gene”, particular expression levels of potential gene, which are biomarkers associated with particular “microbiomes”}; also see SEGAL: e.g., -- Logistic regression may also predict the probability of occurrence for each data point. Logistic regressions also include a multinomial variant. The multinomial logistic regression model is a regression model which generalizes logistic regression by allowing more than two discrete outcomes……. a Bayes optimal classifier algorithm is employed to apply the maximum a posteriori hypothesis to a new record in order to predict the probability of its classification, as well as to calculate the probabilities from each of the other hypotheses obtained from a training set and to use these probabilities as weighting factors for future predictions of the subject's blood contents (particularly the metabolites and optionally and preferably their quantity).--, in [0131]-[0132]; also see: -- The confidence level of the metabolite quantity can be affirmed by conducting a hypothesis test… ypical decision criteria include a choice of a test statistic and significance level (denoted algebraically as “alpha”) to be applied to the analysis. Many different test statistics can be used in hypothesis testing, including mean, variance and the like. A p-value can be calculated and be compared to the significance level. The p-value is quantitative assessment of the probability of observing a value of the test statistic that is either as extreme as or more extreme than the calculated value of the test statistic.--, in [0197] {apparently, herein SEGAL’s neural network using the training set of slide image data to output the probabilities of quantities level of microbiome of subject [e.g., quantity of gut microbiome, bacterial population, metabolite quantity..etc.,], in statistic classes, such as claimed probabilities of being microbiome-high, or being microbiome-low, “the desired, quantity of the particular metabolite can be fed to a machine learning procedure (that has been trained using microbiome data and that is associated with the particular metabolite) in a manner that the machine learning procedure propagates backwards to solve the inverse problem and to provide a set of recommended amounts of microbes (FIG. 14). …..the altering is carried out by increasing a bacterial population whose level is predicted to being below the level in a healthy subject. Table 1 provides examples of bacterial populations which positively and negatively correlate with a particular metabolite” in [0213]-[216], and [0412]});
KUNZ as modified by SEGAL however do not explicitly disclose neural network output a probability of the tested sample being microbiome-high and a probability of the being microbiome-low {and specially applied in the diagnosis and treatment process of bladder cancer, as specific as used in the same as the current Application};
Poore discloses a neural network output a probability of the tested sample being microbiome-high and a probability of the being microbiome-low {and specially applied in the diagnosis and treatment process of bladder cancer, as specific as used in the same as the current Application} (see Poore: e.g., -- TCGA cancer microbiome. FIG. 8a illustrates a table of TCGA study abbreviations. FIG. 8b illustrates PCA of Voom-normalized data, where greyscale-colors represent sequencing platform of the sample and each dot denotes a cancer microbiome sample. FIG. 8c illustrates PCA of the data following consecutive Voom-SNM supervised normalization, as labelled by sequencing platform…. [0100] FIGS. 9a-h illustrate performance metrics discriminating between and within TCGA types of cancer using microbial abundances. FIGS. 9a-f illustrate examples from the heatmaps in FIGS. 4f-h. A greyscale-color gradient (top) denotes the probability threshold at any point along the ROC and PR curves. An inset confusion matrix is shown using a 50% probability threshold cutoff, which can be used to calculate sensitivity, specificity, precision, recall, positive predictive value, negative predictive values, and so forth at the corresponding point on the ROC and PR curves. FIGS. 9g-h illustrate linear regressions of model performance, specifically AUROC (FIG. 9g) and AUPR (FIG. 9h), for discriminating between types of cancer in a one-cancer-type-versus-all-others manner, as a function of minority class size. Performances are shown for models using microorganisms detected in primary tumors, with the greatest number of samples (n=13,883) and types of cancer (n=32) to compare.--, in [0099]-[0100]; and, -- ML model inspection. Selecting the data type (for example, all likely contaminants removed), cancer type (for example, invasive breast carcinoma), and comparison of interest (for example, tumor versus normal) will automatically update the ROC and PR curves, as well as the confusion matrix (using a probability cutoff threshold of 50%)--, in [0104]-[0105]; and, -- (3) an ML model was built on all the remaining samples in the subsample and applied on the left-out sample to make a prediction with a certain probability; (4) steps 2-3 were repeated until all samples had been iterated through; (5) using the list of observed classes and list of predicted classes along with their probabilities,--, in [0208]; and also see: -- the invention provides a method of diagnosing metastatic cancer, wherein the cancer is adrenocortical cancer, bladder cancer, brain cancer (lower grade glioma; glioblastoma), breast cancer, cervical cancer, cholangiocarcinoma, colon cancer..etc., --, in [0059], -- greyscale-red (high) to greyscale-blue (low) for distinguishing between TCGA primary tumors (FIG. 4f), between tumor and normal samples (FIG. 4g), and between stage I and stage IV cancers (FIG. 4h). “NA” may indicate that not enough samples (e.g., fewer than 20) were available in any ML class for model training.--, in [0094[);
KUNZ (as modified by SEGAL) and Poore are combinable as they are in the same field of endeavor: neural network applied in assessments of level/ amount of microbiomes. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify KUNZ (as modified by SEGAL)’s system using Poore’s teachings by including neural network output a probability of the tested sample being microbiome-high and a probability of the being microbiome-low {and specially applied in the diagnosis and treatment process of bladder cancer, as specific as used in the same as the current Application} to KUNZ (as modified by SEGAL)’s assessments of level/ amount of microbiomes in order to calculate sensitivity, specificity, precision, recall, positive predictive value, negative predictive values for cancer treatment processes and the prognosis of the patient (see POORE: e.g. in [0005]-[0007], [0059], [0094], [0099]-[0100], [0104]-[0105], and [0208]);
Therefore, amended claims 1-20 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5-6, 8-10, 13-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over KUNZ (US 20230062811 A1, claims priority of US-Provisional-Application US 63236330 20210824, contents and disclosures applied in this Office Action found the corresponding and exactly the same contexts in US-Provisional-Application US 63236330 20210824), and in view of SEGAL (US 20220102000 A1), and further in view of a new reference Poore (US 20230332249 A1).
Re Claim 1, KUNZ discloses a computer system for assessing a patient’s response to a cancer treatment wherein the patient has cancer (see KUNZ: e.g., -- [0042] Specifically, FIG. 1A illustrates an electronic network 120 that may be connected to servers at hospitals, laboratories, and/or doctors' offices, etc. For example, physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125, etc., may each be connected to an electronic network 120, such as the Internet, through one or more computers, servers, and/or handheld mobile devices. According to an exemplary embodiment of the present disclosure, the electronic network 120 may also be connected to server systems 110, which may include processing devices that are configured to implement a tissue viewing platform 100, which includes a slide analysis tool 101 for determining specimen property or image property information pertaining to digital pathology image(s), and using machine learning to classify a specimen, according to an exemplary embodiment of the present disclosure.
[0043] The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may create or otherwise obtain images of one or more patients' cytology specimen(s), histopathology specimen(s), slide(s) of the cytology specimen(s), digitized images of the slide(s) of the histopathology specimen(s), or any combination thereof. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytology information, etc. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may transmit digitized slide images and/or patient-specific information to server systems 110 over the electronic network 120. Server systems 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125. Server systems 110 may also include processing devices for processing images and data stored in the one or more storage devices 109. Server systems 110 may further include one or more machine learning tool(s) or capabilities. For example, the processing devices may include a machine learning tool for a tissue viewing platform 100,--, in [0042]-[0043], and,
-- [0054] The training image intake module 132 may create or receive a dataset comprising one or more training images corresponding to either or both of images of a human and/or animal tissue and images that are graphically rendered. For example, the training images may be received from any one or any combination of the server systems 110, physician servers 121, and/or laboratory information systems 125. This dataset may be kept on a digital storage device. The training slide matching module 133 may intake training data related to a cause of death. For example, training slide module 133 training data may include receiving one or more images (e.g., WSIs) of a deceased human or animal. Further, the training data may include information such as age, ethnicity, and ancillary test results. The training data may also include biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information can also be ingested, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB). The training slide module 133 may intake full WSIs, or may intake one or more tiles of WSIs. The training slide module 133 may include the ability to break an inputted WSI into tiles to perform further analysis of individual tiles of a WSI. The training slide matching module 133 may utilize convolutional neural network (“CNN”), a Graph Neural Network (GNN), CoordConv, Capsule network, Random Forest Support Vector Machine, Transformer trained directly with the appropriate loss function in order to help provide training for the machine learning techniques described herein. Training slide module may also be used to train a machine learning module to determine a cause of death, predicts the fields on an autopsy report, cardiac arrhythmogenic genes, contributing factors of cardiovascular disease, liver toxins, and/or cause of miscarriage. The slide background module 134 may analyze images of tissues and determine a background within a digital pathology image. It is useful to identify a background within a digital pathology slide to ensure tissue segments are not overlooked.--, in [0054]; and,
-- [0058] Next, data ingested may be inserted into a salient region detection module 204, as described in greater detail below. A salient region detection module 204, as further described below, may be used to identify salient regions to be analyzed for each digital image. A salient region may be an image or one or more areas of an image that are considered relevant to a pathologist determining a diagnosis or treatment. A salient region may be dependent on what type of analysis is being performed on a slide. For example, when performing analysis of a whole slide image to determine which drug to utilize to target a cancer, the areas of a whole slide image that includes the cancerous region or the areas of the cancer that have the most immune cells (e.g., tumor infiltrating lymphoctyes) contained within the cancer may be considered the salient regions. A salient region may be specific to a particular application of the invention. Further, the salient region may be particular to specific organs or may depend on specific toxins. For example, for certain toxins, the exact morphology of the necrotic regions may be relevant to a diagnosis and thus be considered salient regions in an image. In another example, within kidney tissues, the salient region may be in and around the glomerulus. The detection of a salient region may be done manually by a user or may be done automatically using AI/ML. An entire image or specific image regions may be identified as salient. The entire disclosure of U.S. Non-Provisional application Ser. No. 17/313,617 filed May 6, 2021 is hereby incorporated herein by reference in its entirety.
[0059] Next, the digital whole slide images from the data ingestion 202, which may or not have had a salient region identified, may be provided to a pathology inference module 206. A pathology inference module, as further described below, may be used to infer one or more fields in one or more of an autopsy report, toxin report, cardiovascular report, or cause of death using machine learning and computer vision from the one or more digital image(s). The pathology inference module may incorporate spatial information from disparate regions in an image.--, in [0058]-[0059]; -- [0112] At step 552 the system (e.g., the intake module 136 of slide analysis tool 101) may receive digital images (e.g., H&E whole slide images) of pathology specimens from a deceased human/animal may be received into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).
[0113] In one embodiment, the system (e.g., the intake module 136 of slide analysis tool 101) may receive a gross description. The system may further receive deceased human/animal information (e.g., age, ethnicity, ancillary test results, etc.) that may be ingested to stratify and split the system for machine learning. The system may additionally ingest biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB).--, in [0112]-[0113], and [0123]-[0125]; and,
-- By solely examining small tissue samples, a human body may have significantly less lab processes performed on it, leaving the body in a more dignified state. Another useful aspect of this embodiment is that it may be used by drug companies to help determine cause of death during clinical studies of humans and animals. This may provide more accurate information to the drug companies and help studies become safer and more efficient.
[0100] FIG. 5A may provide an example of how to train an embodiment of the pathology inference module 206 that may be used for cardiac (heart) assessment for early deaths and is capable of predicting cardiac particular genes. FIG. 5B may provide an example of how to use an embodiment of the pathology inference module 206 that may be used for cardiac (heart) assessment for early deaths and is capable of predicting cardiac particular genes.--, [0098]-[0101], and [0118]-[0120]);
the system comprising:
a processor; and
a computer-readable storage medium storing computer-readable instructions which, when executed by a processor (see KUNZ: e.g.,, --[0164] As shown in FIG. 11, device 1100 may include a central processing unit (CPU) 1120. CPU 1120 may be any type of processor device including, for example, any type of special purpose or a general-purpose microprocessor device. As will be appreciated by persons skilled in the relevant art, CPU 1120 also may be a single processor in a multi-core/multiprocessor system, such system operating alone, or in a cluster of computing devices operating in a cluster or server farm. CPU 1120 may be connected to a data communication infrastructure 1110, for example a bus, message queue, network, or multi-core message-passing scheme.
[0165] Device 1100 may also include a main memory 1140, for example, random access memory (RAM), and also may include a secondary memory 1130. Secondary memory 1130, for example a read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive.--, in [0163]-[0165]), cause the processor to:
execute a convolutional neural network to detect a level of microbiome in a whole slide image associated with the patient (see KUNZ: e.g., --[0054] The training image intake module 132 may create or receive a dataset comprising one or more training images corresponding to either or both of images of a human and/or animal tissue and images that are graphically rendered. For example, the training images may be received from any one or any combination of the server systems 110, physician servers 121, and/or laboratory information systems 125. This dataset may be kept on a digital storage device. The training slide matching module 133 may intake training data related to a cause of death. For example, training slide module 133 training data may include receiving one or more images (e.g., WSIs) of a deceased human or animal. Further, the training data may include information such as age, ethnicity, and ancillary test results. The training data may also include biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information can also be ingested, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB). The training slide module 133 may intake full WSIs, or may intake one or more tiles of WSIs. The training slide module 133 may include the ability to break an inputted WSI into tiles to perform further analysis of individual tiles of a WSI. The training slide matching module 133 may utilize convolutional neural network (“CNN”), a Graph Neural Network (GNN), CoordConv, Capsule network, Random Forest Support Vector Machine, Transformer trained directly with the appropriate loss function in order to help provide training for the machine learning techniques described herein. Training slide module may also be used to train a machine learning module to determine a cause of death, predicts the fields on an autopsy report, cardiac arrhythmogenic genes, contributing factors of cardiovascular disease, liver toxins, and/or cause of miscarriage. The slide background module 134 may analyze images of tissues and determine a background within a digital pathology image.--, in [0054]-[0058]; and --[0127] At step 660, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI from a liver biopsy). The output may be displayed as a list of any toxins found. Further, the system may rank the list based on highest amount of toxin present and/or by more dangerous toxin present. Outputting the toxins may further include saving the information to electronic storage such as digital evidence or forensic system, or displaying the results to a pathology.
0128] FIG. 7 may provide an example of how to use an embodiment of the pathology inference module 206 that may be used for infection detection analysis.
[0129] Occult infection at death can be ascertained during an autopsy by examining blood and tissue cultures that assess for the growth of fungal or bacterial organisms, or by direct sampling and visualization of tissue containing these organisms. The embodiments disclosed herein and further described in method 750 may be configured to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria. This may obviate the reliance on examining time-consuming and imprecise blood and tissue cultures for establishing a cause of infection.
[0130] The machine learning module utilized in FIG. 7 may be trained based on techniques described in relation to FIG. 4A. Specifically, the machine learning module may be trained in this embodiment to determine, based on H&E WSI, whether the presence of an infection is included. H&E slides with labeled infections may be fed to the machine learning model of FIG. 4A to train the system to output one or more infections found on a WSI. Looking for infection may include searching for the presence of fungi, bacteria, or mycobacteria.--, in [0127]-[0130] {“assess for the growth of fungal or bacterial organisms”, and “to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria”, those are microbiomes}; and,
--[0136] At step 760, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI). The output may be displayed as a list of any infectious found. Further, the system may rank the list based on highest amount of infectious disease present and/or by more dangerous toxin present.--, in [0136]);
KUNZ however does not explicitly disclose based on an output of the convolutional neural network, categorize the whole slide image as one of microbiome-low and microbiome-high;
SEGAL discloses based on an output of the convolutional neural network, categorize the whole slide image as one of microbiome-low and microbiome-high (see SEGAL: e.g., -- a fully-connected neural network, each of the neurons in a particular layer is connected to and provides input value to those in the next layer. These input values are then summed and this sum compared to a bias, or threshold. If the value exceeds the threshold for a particular neuron, that neuron then holds a positive value which can be used as input to neurons in the next layer of neurons….. [0138] The machine learning procedure used according to some embodiments of the present invention is a trained machine learning procedure. A machine learning procedure can be trained according to some embodiments of the present invention by feeding a machine learning training program with microbiome data of a cohort of subjects from which the quantities of the metabolite have been determined by blood tests… t least a portion of the decision rules relate to one or more microbes in the microbiome. A simple decision rule may be a threshold for the amount of a particular microbes, but more complex rules, relating to more than one microbes are also contemplated. The machine learning training program also accumulates data at the leaves of the trees. The structures of the trees, the decision rules for the split nodes, and the data at the leaves are all selected by the machine learning training program, automatically and typically without user intervention, such that the microbiome data at the root of the trees provide the quantities of the metabolite as determined by blood tests at the leaves of the trees. --, in [0136]-[0139], and, Table. 1, -- analyzing the amount of each of the corresponding microbes set forth in Table 1 in the fecal microbiome of the subject,--, in [0146]-[0147], and [0385]-[0386]);
KUNZ and SEGAL are combinable as they are in the same field of endeavor: neural network applied in assessments of level/ amount of microbiomes. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify KUNZ’s system using SEGAL’s teachings by including based on an output of the convolutional neural network, categorize the whole slide image as one of microbiome-low and microbiome-high to KUNZ’s assessments of level/ amount of microbiomes in order to the quantity of microbiomes a trained machine learning procedure with amount of a plurality of microbes of a microbiome of the subject (see SEGAL: e.g. in abstract, Table. 1, [0136]-[0139], [0146]-[0147], and [0385]-[0386]);
KUNZ as modified by SEGAL discloses process a whole slide image associated with the patient using a convolutional neural network trained to output the quantities of the microbiomes, and determining the level of the quantities of the microbiomes of an input whole slide image, by comparing to a bias, or threshold (see SEGAL: e.g., -- a fully-connected neural network, each of the neurons in a particular layer is connected to and provides input value to those in the next layer. These input values are then summed and this sum compared to a bias, or threshold. If the value exceeds the threshold for a particular neuron, that neuron then holds a positive value which can be used as input to neurons in the next layer of neurons….. [0138] The machine learning procedure used according to some embodiments of the present invention is a trained machine learning procedure. A machine learning procedure can be trained according to some embodiments of the present invention by feeding a machine learning training program with microbiome data of a cohort of subjects from which the quantities of the metabolite have been determined by blood tests… t least a portion of the decision rules relate to one or more microbes in the microbiome. A simple decision rule may be a threshold for the amount of a particular microbes, but more complex rules, relating to more than one microbes are also contemplated. The machine learning training program also accumulates data at the leaves of the trees. The structures of the trees, the decision rules for the split nodes, and the data at the leaves are all selected by the machine learning training program, automatically and typically without user intervention, such that the microbiome data at the root of the trees provide the quantities of the metabolite as determined by blood tests at the leaves of the trees. --, in [0136]-[0139], and, Table. 1, -- analyzing the amount of each of the corresponding microbes set forth in Table 1 in the fecal microbiome of the subject,--, in [0146]-[0147], and [0385]-[0386]);
regarding to newly added limitation the Neural Network “output a probability of {the level of the quantities of} microbiomes on the whole slide image”, KUNZ as modified by SEGAL further disclose Neural Network “output a probability of {the level of the quantities of} microbiomes on the whole slide image” (see KUNZ: e.g., -- to train a machine learning system to predict the presence of any gene amplification from the training set in addition to the cause of death. Step 510 may utilize techniques described in step 410 in addition to training the machine learning to identify particular genes. The system may train a machine learning system to create a vector, where each potential value of the vector represents the chances of the individual having a particular cardiac Arrhythmia gene. The machine learning system may include a corresponding classification layer that outputs a vector of probabilities indicating particular percentage chances that particular genes exist in the individual. The classifier may be responsible for assigning probabilities of each potential Arrhythmia gene.--, in [0110]-[0112] {herein “outputs a vector of probabilities indicating…. chances that particular genes exist in the individual”, and, “assigning probabilities of each potential Arrhythmia gene”, particular expression levels of potential gene, which are biomarkers associated with particular “microbiomes”}; also see SEGAL: e.g., -- Logistic regression may also predict the probability of occurrence for each data point. Logistic regressions also include a multinomial variant. The multinomial logistic regression model is a regression model which generalizes logistic regression by allowing more than two discrete outcomes……. a Bayes optimal classifier algorithm is employed to apply the maximum a posteriori hypothesis to a new record in order to predict the probability of its classification, as well as to calculate the probabilities from each of the other hypotheses obtained from a training set and to use these probabilities as weighting factors for future predictions of the subject's blood contents (particularly the metabolites and optionally and preferably their quantity).--, in [0131]-[0132]; also see: -- The confidence level of the metabolite quantity can be affirmed by conducting a hypothesis test… ypical decision criteria include a choice of a test statistic and significance level (denoted algebraically as “alpha”) to be applied to the analysis. Many different test statistics can be used in hypothesis testing, including mean, variance and the like. A p-value can be calculated and be compared to the significance level. The p-value is quantitative assessment of the probability of observing a value of the test statistic that is either as extreme as or more extreme than the calculated value of the test statistic.--, in [0197] {apparently, herein SEGAL’s neural network using the training set of slide image data to output the probabilities of quantities level of microbiome of subject [e.g., quantity of gut microbiome, bacterial population, metabolite quantity..etc.,], in statistic classes, such as claimed probabilities of being microbiome-high, or being microbiome-low, “the desired, quantity of the particular metabolite can be fed to a machine learning procedure (that has been trained using microbiome data and that is associated with the particular metabolite) in a manner that the machine learning procedure propagates backwards to solve the inverse problem and to provide a set of recommended amounts of microbes (FIG. 14). …..the altering is carried out by increasing a bacterial population whose level is predicted to being below the level in a healthy subject. Table 1 provides examples of bacterial populations which positively and negatively correlate with a particular metabolite” in [0213]-[216], and [0412]});
KUNZ as modified by SEGAL however do not explicitly disclose neural network output a probability of the tested sample being microbiome-high and a probability of the being microbiome-low {and specially applied in the diagnosis and treatment process of bladder cancer, as specific as used in the same as the current Application};
Poore discloses a neural network output a probability of the tested sample being microbiome-high and a probability of the being microbiome-low {and specially applied in the diagnosis and treatment process of bladder cancer, as specific as used in the same as the current Application} (see Poore: e.g., -- TCGA cancer microbiome. FIG. 8a illustrates a table of TCGA study abbreviations. FIG. 8b illustrates PCA of Voom-normalized data, where greyscale-colors represent sequencing platform of the sample and each dot denotes a cancer microbiome sample. FIG. 8c illustrates PCA of the data following consecutive Voom-SNM supervised normalization, as labelled by sequencing platform…. [0100] FIGS. 9a-h illustrate performance metrics discriminating between and within TCGA types of cancer using microbial abundances. FIGS. 9a-f illustrate examples from the heatmaps in FIGS. 4f-h. A greyscale-color gradient (top) denotes the probability threshold at any point along the ROC and PR curves. An inset confusion matrix is shown using a 50% probability threshold cutoff, which can be used to calculate sensitivity, specificity, precision, recall, positive predictive value, negative predictive values, and so forth at the corresponding point on the ROC and PR curves. FIGS. 9g-h illustrate linear regressions of model performance, specifically AUROC (FIG. 9g) and AUPR (FIG. 9h), for discriminating between types of cancer in a one-cancer-type-versus-all-others manner, as a function of minority class size. Performances are shown for models using microorganisms detected in primary tumors, with the greatest number of samples (n=13,883) and types of cancer (n=32) to compare.--, in [0099]-[0100]; and, -- ML model inspection. Selecting the data type (for example, all likely contaminants removed), cancer type (for example, invasive breast carcinoma), and comparison of interest (for example, tumor versus normal) will automatically update the ROC and PR curves, as well as the confusion matrix (using a probability cutoff threshold of 50%)--, in [0104]-[0105]; and, -- (3) an ML model was built on all the remaining samples in the subsample and applied on the left-out sample to make a prediction with a certain probability; (4) steps 2-3 were repeated until all samples had been iterated through; (5) using the list of observed classes and list of predicted classes along with their probabilities,--, in [0208]; and also see: -- the invention provides a method of diagnosing metastatic cancer, wherein the cancer is adrenocortical cancer, bladder cancer, brain cancer (lower grade glioma; glioblastoma), breast cancer, cervical cancer, cholangiocarcinoma, colon cancer..etc., --, in [0059], -- greyscale-red (high) to greyscale-blue (low) for distinguishing between TCGA primary tumors (FIG. 4f), between tumor and normal samples (FIG. 4g), and between stage I and stage IV cancers (FIG. 4h). “NA” may indicate that not enough samples (e.g., fewer than 20) were available in any ML class for model training.--, in [0094[);
KUNZ (as modified by SEGAL) and Poore are combinable as they are in the same field of endeavor: neural network applied in assessments of level/ amount of microbiomes. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify KUNZ (as modified by SEGAL)’s system using Poore’s teachings by including neural network output a probability of the tested sample being microbiome-high and a probability of the being microbiome-low {and specially applied in the diagnosis and treatment process of bladder cancer, as specific as used in the same as the current Application} to KUNZ (as modified by SEGAL)’s assessments of level/ amount of microbiomes in order to calculate sensitivity, specificity, precision, recall, positive predictive value, negative predictive values for cancer treatment processes and the prognosis of the patient (see POORE: e.g. in [0005]-[0007], [0059], [0094], [0099]-[0100], [0104]-[0105], and [0208]);
KUNZ as modified by SEGAL and Poore further disclose in response to an output of the convolutional neural network based on the whole slide image associated with the patient, categorize the whole slide image associated with the patient as one of microbiome-low and microbiome-high (see KUNZ: e.g., -- to train a machine learning system to predict the presence of any gene amplification from the training set in addition to the cause of death. Step 510 may utilize techniques described in step 410 in addition to training the machine learning to identify particular genes. The system may train a machine learning system to create a vector, where each potential value of the vector represents the chances of the individual having a particular cardiac Arrhythmia gene. The machine learning system may include a corresponding classification layer that outputs a vector of probabilities indicating particular percentage chances that particular genes exist in the individual. The classifier may be responsible for assigning probabilities of each potential Arrhythmia gene.--, in [0110]-[0112] {herein “outputs a vector of probabilities indicating…. chances that particular genes exist in the individual”, and, “assigning probabilities of each potential Arrhythmia gene”, particular expression levels of potential gene, which are biomarkers associated with particular “microbiomes”}; also see SEGAL: e.g., -- Logistic regression may also predict the probability of occurrence for each data point. Logistic regressions also include a multinomial variant. The multinomial logistic regression model is a regression model which generalizes logistic regression by allowing more than two discrete outcomes……. a Bayes optimal classifier algorithm is employed to apply the maximum a posteriori hypothesis to a new record in order to predict the probability of its classification, as well as to calculate the probabilities from each of the other hypotheses obtained from a training set and to use these probabilities as weighting factors for future predictions of the subject's blood contents (particularly the metabolites and optionally and preferably their quantity).--, in [0131]-[0132]; also see: -- The confidence level of the metabolite quantity can be affirmed by conducting a hypothesis test… ypical decision criteria include a choice of a test statistic and significance level (denoted algebraically as “alpha”) to be applied to the analysis. Many different test statistics can be used in hypothesis testing, including mean, variance and the like. A p-value can be calculated and be compared to the significance level. The p-value is quantitative assessment of the probability of observing a value of the test statistic that is either as extreme as or more extreme than the calculated value of the test statistic.--, in [0197] {apparently, herein SEGAL’s neural network using the training set of slide image data to output the probabilities of quantities level of microbiome of subject [e.g., quantity of gut microbiome, bacterial population, metabolite quantity..etc.,], in statistic classes, such as claimed probabilities of being microbiome-high, or being microbiome-low, “the desired, quantity of the particular metabolite can be fed to a machine learning procedure (that has been trained using microbiome data and that is associated with the particular metabolite) in a manner that the machine learning procedure propagates backwards to solve the inverse problem and to provide a set of recommended amounts of microbes (FIG. 14). …..the altering is carried out by increasing a bacterial population whose level is predicted to being below the level in a healthy subject. Table 1 provides examples of bacterial populations which positively and negatively correlate with a particular metabolite” in [0213]-[216], and [0412]});
based on the categorization of the whole slide image associated with the patient, determine a characteristic of cancer associated with the patient (see KUNZ: e.g., --[0043] The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may create or otherwise obtain images of one or more patients' cytology specimen(s), histopathology specimen(s), slide(s) of the cytology specimen(s), digitized images of the slide(s) of the histopathology specimen(s), or any combination thereof. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytology information, etc. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may transmit digitized slide images and/or patient-specific information to server systems 110 over the electronic network 120. Server systems 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125. Server systems 110 may also include processing devices for processing images and data stored in the one or more storage devices 109. Server systems 110 may further include one or more machine learning tool(s) or capabilities. For example, the processing devices may include a machine learning tool for a tissue viewing platform 100,--, in [0042]-[0043], and,
-- [0054] The training image intake module 132 may create or receive a dataset comprising one or more training images corresponding to either or both of images of a human and/or animal tissue and images that are graphically rendered. For example, the training images may be received from any one or any combination of the server systems 110, physician servers 121, and/or laboratory information systems 125. This dataset may be kept on a digital storage device. The training slide matching module 133 may intake training data related to a cause of death. For example, training slide module 133 training data may include receiving one or more images (e.g., WSIs) of a deceased human or animal. Further, the training data may include information such as age, ethnicity, and ancillary test results. The training data may also include biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information can also be ingested, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB). The training slide module 133 may intake full WSIs, or may intake one or more tiles of WSIs. The training slide module 133 may include the ability to break an inputted WSI into tiles to perform further analysis of individual tiles of a WSI. The training slide matching module 133 may utilize convolutional neural network (“CNN”), a Graph Neural Network (GNN), CoordConv, Capsule network, Random Forest Support Vector Machine, Transformer trained directly with the appropriate loss function in order to help provide training for the machine learning techniques described herein. Training slide module may also be used to train a machine learning module to determine a cause of death, predicts the fields on an autopsy report, cardiac arrhythmogenic genes, contributing factors of cardiovascular disease, liver toxins, and/or cause of miscarriage. The slide background module 134 may analyze images of tissues and determine a background within a digital pathology image. It is useful to identify a background within a digital pathology slide to ensure tissue segments are not overlooked.--, in [0054]; and,
-- [0058] Next, data ingested may be inserted into a salient region detection module 204, as described in greater detail below. A salient region detection module 204, as further described below, may be used to identify salient regions to be analyzed for each digital image. A salient region may be an image or one or more areas of an image that are considered relevant to a pathologist determining a diagnosis or treatment. A salient region may be dependent on what type of analysis is being performed on a slide. For example, when performing analysis of a whole slide image to determine which drug to utilize to target a cancer, the areas of a whole slide image that includes the cancerous region or the areas of the cancer that have the most immune cells (e.g., tumor infiltrating lymphoctyes) contained within the cancer may be considered the salient regions. A salient region may be specific to a particular application of the invention. Further, the salient region may be particular to specific organs or may depend on specific toxins. For example, for certain toxins, the exact morphology of the necrotic regions may be relevant to a diagnosis and thus be considered salient regions in an image. In another example, within kidney tissues, the salient region may be in and around the glomerulus. The detection of a salient region may be done manually by a user or may be done automatically using AI/ML. An entire image or specific image regions may be identified as salient. The entire disclosure of U.S. Non-Provisional application Ser. No. 17/313,617 filed May 6, 2021 is hereby incorporated herein by reference in its entirety.
[0059] Next, the digital whole slide images from the data ingestion 202, which may or not have had a salient region identified, may be provided to a pathology inference module 206. A pathology inference module, as further described below, may be used to infer one or more fields in one or more of an autopsy report, toxin report, cardiovascular report, or cause of death using machine learning and computer vision from the one or more digital image(s). The pathology inference module may incorporate spatial information from disparate regions in an image.--, in [0058]-[0059]).
Re Claim 2, KUNZ as modified by SEGAL and Poore further disclose wherein the whole slide image associated with the patient comprises a hematoxylin and eosin (H&E)-stained pathology slides (see KUNZ: e.g., -- [0058] Next, data ingested may be inserted into a salient region detection module 204, as described in greater detail below. A salient region detection module 204, as further described below, may be used to identify salient regions to be analyzed for each digital image. A salient region may be an image or one or more areas of an image that are considered relevant to a pathologist determining a diagnosis or treatment. A salient region may be dependent on what type of analysis is being performed on a slide. For example, when performing analysis of a whole slide image to determine which drug to utilize to target a cancer, the areas of a whole slide image that includes the cancerous region or the areas of the cancer that have the most immune cells (e.g., tumor infiltrating lymphoctyes) contained within the cancer may be considered the salient regions. A salient region may be specific to a particular application of the invention. Further, the salient region may be particular to specific organs or may depend on specific toxins. For example, for certain toxins, the exact morphology of the necrotic regions may be relevant to a diagnosis and thus be considered salient regions in an image. In another example, within kidney tissues, the salient region may be in and around the glomerulus. The detection of a salient region may be done manually by a user or may be done automatically using AI/ML. An entire image or specific image regions may be identified as salient. The entire disclosure of U.S. Non-Provisional application Ser. No. 17/313,617 filed May 6, 2021 is hereby incorporated herein by reference in its entirety.
[0059] Next, the digital whole slide images from the data ingestion 202, which may or not have had a salient region identified, may be provided to a pathology inference module 206. A pathology inference module, as further described below, may be used to infer one or more fields in one or more of an autopsy report, toxin report, cardiovascular report, or cause of death using machine learning and computer vision from the one or more digital image(s). The pathology inference module may incorporate spatial information from disparate regions in an image.--, in [0058]-[0059]; and, -- [0112] At step 552 the system (e.g., the intake module 136 of slide analysis tool 101) may receive digital images (e.g., H&E whole slide images) of pathology specimens from a deceased human/animal may be received into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).--, in [0058]-[0059]; and --[0127] At step 660, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI from a liver biopsy). The output may be displayed as a list of any toxins found. Further, the system may rank the list based on highest amount of toxin present and/or by more dangerous toxin present. Outputting the toxins may further include saving the information to electronic storage such as digital evidence or forensic system, or displaying the results to a pathology.
0128] FIG. 7 may provide an example of how to use an embodiment of the pathology inference module 206 that may be used for infection detection analysis.
[0129] Occult infection at death can be ascertained during an autopsy by examining blood and tissue cultures that assess for the growth of fungal or bacterial organisms, or by direct sampling and visualization of tissue containing these organisms. The embodiments disclosed herein and further described in method 750 may be configured to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria. This may obviate the reliance on examining time-consuming and imprecise blood and tissue cultures for establishing a cause of infection.
[0130] The machine learning module utilized in FIG. 7 may be trained based on techniques described in relation to FIG. 4A. Specifically, the machine learning module may be trained in this embodiment to determine, based on H&E WSI, whether the presence of an infection is included. H&E slides with labeled infections may be fed to the machine learning model of FIG. 4A to train the system to output one or more infections found on a WSI. Looking for infection may include searching for the presence of fungi, bacteria, or mycobacteria.--, in [0127]-[0130] {“assess for the growth of fungal or bacterial organisms”, and “to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria”, those are microbiomes}; and,
--[0136] At step 760, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI). The output may be displayed as a list of any infectious found. Further, the system may rank the list based on highest amount of infectious disease present and/or by more dangerous toxin present.--, in [0136]).
Re Claim 5, KUNZ as modified by SEGAL and Poore further disclose wherein the instructions further cause the processor to determine a survival statistic of the patient based on the output of the convolutional neural network (see SEGAL: e.g., -- According to still another embodiment, the microbial composition of any of the aspects of the present invention is devoid (or comprises only trace quantities) of fecal material (e.g., fiber).
[0253] The probiotic bacteria may be in any suitable form, for example in a powdered dry form. In addition, the probiotic microorganism may have undergone processing in order for it to increase its survival. For example, the microorganism may be coated or encapsulated in a polysaccharide, fat, starch, protein or in a sugar matrix. Standard encapsulation techniques known in the art can be used. For example, techniques discussed in U.S. Pat. No. 6,190,591, which is hereby incorporated by reference in its entirety, may be used.--, in [0252]-[0253]).
Re Claim 6, KUNZ as modified by SEGAL and Poore further disclose wherein microbiome-low is associated with a microbiome level of less than a median microbiome level of a cohort and microbiome-high is associated with a microbiome level of greater than the median microbiome level of the cohort (see SEGAL: e.g., -- a fully-connected neural network, each of the neurons in a particular layer is connected to and provides input value to those in the next layer. These input values are then summed and this sum compared to a bias, or threshold. If the value exceeds the threshold for a particular neuron, that neuron then holds a positive value which can be used as input to neurons in the next layer of neurons….. [0138] The machine learning procedure used according to some embodiments of the present invention is a trained machine learning procedure. A machine learning procedure can be trained according to some embodiments of the present invention by feeding a machine learning training program with microbiome data of a cohort of subjects from which the quantities of the metabolite have been determined by blood tests… t least a portion of the decision rules relate to one or more microbes in the microbiome. A simple decision rule may be a threshold for the amount of a particular microbes, but more complex rules, relating to more than one microbes are also contemplated. The machine learning training program also accumulates data at the leaves of the trees. The structures of the trees, the decision rules for the split nodes, and the data at the leaves are all selected by the machine learning training program, automatically and typically without user intervention, such that the microbiome data at the root of the trees provide the quantities of the metabolite as determined by blood tests at the leaves of the trees. --, in [0136]-[0139], and, Table. 1, -- analyzing the amount of each of the corresponding microbes set forth in Table 1 in the fecal microbiome of the subject,--, in [0146]-[0147], and [0385]-[0386]).
Re Claim 8, KUNZ as modified by SEGAL and Poore further disclose wherein the instructions further cause the processor to determine a pathologic complete response (pCR) based on the output of the convolutional neural network see SEGAL: e.g., --et another envisaged example is Illumina or Solexa sequencing, e.g. by using the Illumina Genome Analyzer technology, which is based on reversible dye-terminators. DNA molecules are typically attached to primers on a slide and amplified so that local clonal colonies are formed. Subsequently one type of nucleotide at a time may be added, and non-incorporated nucleotides are washed away. Subsequently, images of the fluorescently labeled nucleotides may be taken and the dye is chemically removed from the DNA, allowing a next cycle. Yet another example is the use of Applied Biosystems' SOLiD technology, which employs sequencing by ligation. This method is based on the use of a pool of all possible oligonucleotides of a fixed length, which are labeled according to the sequenced position. Such oligonucleotides are annealed and ligated. Subsequently, the preferential ligation by DNA ligase for matching sequences typically results in a signal informative of the nucleotide at that position. Since the DNA is typically amplified by emulsion PCR, the resulting bead, each containing only copies of the same DNA molecule, can be deposited on a glass slide resulting in sequences of quantities and lengths comparable to Illumina sequencing. A further method is based on Helicos' Heliscope technology, wherein fragments are captured by polyT oligomers tethered to an array. At each sequencing cycle, polymerase and single fluorescently labeled nucleotides are added and the array is imaged.--, in [0091]).
Re Claims 9-10, 13-15, claims 9-10, 13-15 are the corresponding method claim to claims 1-2, 5-6, and 8 respectively. Claims 9-10, 13-15 thus are rejected for the similar reasons for claims 1-2, 5-6, and 8. See above discussions with regard to claims 1-2, 5-6, and 8 respectively. Furthermore, KUNZ as modified by SEGAL and Poore further disclose a method of assessing a patient’ s response to a cancer treatment wherein the patient has cancer (see KUNZ: e.g., -- [0042] Specifically, FIG. 1A illustrates an electronic network 120 that may be connected to servers at hospitals, laboratories, and/or doctors' offices, etc. For example, physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125, etc., may each be connected to an electronic network 120, such as the Internet, through one or more computers, servers, and/or handheld mobile devices. According to an exemplary embodiment of the present disclosure, the electronic network 120 may also be connected to server systems 110, which may include processing devices that are configured to implement a tissue viewing platform 100, which includes a slide analysis tool 101 for determining specimen property or image property information pertaining to digital pathology image(s), and using machine learning to classify a specimen, according to an exemplary embodiment of the present disclosure.
[0043] The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may create or otherwise obtain images of one or more patients' cytology specimen(s), histopathology specimen(s), slide(s) of the cytology specimen(s), digitized images of the slide(s) of the histopathology specimen(s), or any combination thereof. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytology information, etc. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may transmit digitized slide images and/or patient-specific information to server systems 110 over the electronic network 120. Server systems 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125. Server systems 110 may also include processing devices for processing images and data stored in the one or more storage devices 109. Server systems 110 may further include one or more machine learning tool(s) or capabilities. For example, the processing devices may include a machine learning tool for a tissue viewing platform 100,--, in [0042]-[0043], and,
-- [0054] The training image intake module 132 may create or receive a dataset comprising one or more training images corresponding to either or both of images of a human and/or animal tissue and images that are graphically rendered. For example, the training images may be received from any one or any combination of the server systems 110, physician servers 121, and/or laboratory information systems 125. This dataset may be kept on a digital storage device. The training slide matching module 133 may intake training data related to a cause of death. For example, training slide module 133 training data may include receiving one or more images (e.g., WSIs) of a deceased human or animal. Further, the training data may include information such as age, ethnicity, and ancillary test results. The training data may also include biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information can also be ingested, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB). The training slide module 133 may intake full WSIs, or may intake one or more tiles of WSIs. The training slide module 133 may include the ability to break an inputted WSI into tiles to perform further analysis of individual tiles of a WSI. The training slide matching module 133 may utilize convolutional neural network (“CNN”), a Graph Neural Network (GNN), CoordConv, Capsule network, Random Forest Support Vector Machine, Transformer trained directly with the appropriate loss function in order to help provide training for the machine learning techniques described herein. Training slide module may also be used to train a machine learning module to determine a cause of death, predicts the fields on an autopsy report, cardiac arrhythmogenic genes, contributing factors of cardiovascular disease, liver toxins, and/or cause of miscarriage. The slide background module 134 may analyze images of tissues and determine a background within a digital pathology image. It is useful to identify a background within a digital pathology slide to ensure tissue segments are not overlooked.--, in [0054]; and,
-- [0058] Next, data ingested may be inserted into a salient region detection module 204, as described in greater detail below. A salient region detection module 204, as further described below, may be used to identify salient regions to be analyzed for each digital image. A salient region may be an image or one or more areas of an image that are considered relevant to a pathologist determining a diagnosis or treatment. A salient region may be dependent on what type of analysis is being performed on a slide. For example, when performing analysis of a whole slide image to determine which drug to utilize to target a cancer, the areas of a whole slide image that includes the cancerous region or the areas of the cancer that have the most immune cells (e.g., tumor infiltrating lymphoctyes) contained within the cancer may be considered the salient regions. A salient region may be specific to a particular application of the invention. Further, the salient region may be particular to specific organs or may depend on specific toxins. For example, for certain toxins, the exact morphology of the necrotic regions may be relevant to a diagnosis and thus be considered salient regions in an image. In another example, within kidney tissues, the salient region may be in and around the glomerulus. The detection of a salient region may be done manually by a user or may be done automatically using AI/ML. An entire image or specific image regions may be identified as salient. The entire disclosure of U.S. Non-Provisional application Ser. No. 17/313,617 filed May 6, 2021 is hereby incorporated herein by reference in its entirety.
[0059] Next, the digital whole slide images from the data ingestion 202, which may or not have had a salient region identified, may be provided to a pathology inference module 206. A pathology inference module, as further described below, may be used to infer one or more fields in one or more of an autopsy report, toxin report, cardiovascular report, or cause of death using machine learning and computer vision from the one or more digital image(s). The pathology inference module may incorporate spatial information from disparate regions in an image.--, in [0058]-[0059]; -- [0112] At step 552 the system (e.g., the intake module 136 of slide analysis tool 101) may receive digital images (e.g., H&E whole slide images) of pathology specimens from a deceased human/animal may be received into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).
[0113] In one embodiment, the system (e.g., the intake module 136 of slide analysis tool 101) may receive a gross description. The system may further receive deceased human/animal information (e.g., age, ethnicity, ancillary test results, etc.) that may be ingested to stratify and split the system for machine learning. The system may additionally ingest biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB).--, in [0112]-[0113], and [0123]-[0125]; and,
-- By solely examining small tissue samples, a human body may have significantly less lab processes performed on it, leaving the body in a more dignified state. Another useful aspect of this embodiment is that it may be used by drug companies to help determine cause of death during clinical studies of humans and animals. This may provide more accurate information to the drug companies and help studies become safer and more efficient.
[0100] FIG. 5A may provide an example of how to train an embodiment of the pathology inference module 206 that may be used for cardiac (heart) assessment for early deaths and is capable of predicting cardiac particular genes. FIG. 5B may provide an example of how to use an embodiment of the pathology inference module 206 that may be used for cardiac (heart) assessment for early deaths and is capable of predicting cardiac particular genes.--, [0098]-[0101], and [0118]-[0120]).
Re Claims 16-17, and 20, claims 16-17, and 20 are the corresponding medium claim to claims 1-2, and 5 respectively. Claims 16-17, and 20 thus are rejected for the similar reasons for claims 1-2, and 5. See above discussions with regard to claims 1-2, and 5 respectively. Furthermore, KUNZ as modified by SEGAL and Poore further disclose at least one machine-readable non-transitory medium comprising a plurality of instructions, executed on a computing device, to facilitate the computing device to perform the method (see KUNZ: e.g.,, --[0164] As shown in FIG. 11, device 1100 may include a central processing unit (CPU) 1120. CPU 1120 may be any type of processor device including, for example, any type of special purpose or a general-purpose microprocessor device. As will be appreciated by persons skilled in the relevant art, CPU 1120 also may be a single processor in a multi-core/multiprocessor system, such system operating alone, or in a cluster of computing devices operating in a cluster or server farm. CPU 1120 may be connected to a data communication infrastructure 1110, for example a bus, message queue, network, or multi-core message-passing scheme.
[0165] Device 1100 may also include a main memory 1140, for example, random access memory (RAM), and also may include a secondary memory 1130. Secondary memory 1130, for example a read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive.--, in [0163]-[0165]; and, --[0054] The training image intake module 132 may create or receive a dataset comprising one or more training images corresponding to either or both of images of a human and/or animal tissue and images that are graphically rendered. For example, the training images may be received from any one or any combination of the server systems 110, physician servers 121, and/or laboratory information systems 125. This dataset may be kept on a digital storage device. The training slide matching module 133 may intake training data related to a cause of death. For example, training slide module 133 training data may include receiving one or more images (e.g., WSIs) of a deceased human or animal. Further, the training data may include information such as age, ethnicity, and ancillary test results. The training data may also include biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information can also be ingested, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB). The training slide module 133 may intake full WSIs, or may intake one or more tiles of WSIs. The training slide module 133 may include the ability to break an inputted WSI into tiles to perform further analysis of individual tiles of a WSI. The training slide matching module 133 may utilize convolutional neural network (“CNN”), a Graph Neural Network (GNN), CoordConv, Capsule network, Random Forest Support Vector Machine, Transformer trained directly with the appropriate loss function in order to help provide training for the machine learning techniques described herein. Training slide module may also be used to train a machine learning module to determine a cause of death, predicts the fields on an autopsy report, cardiac arrhythmogenic genes, contributing factors of cardiovascular disease, liver toxins, and/or cause of miscarriage. The slide background module 134 may analyze images of tissues and determine a background within a digital pathology image.--, in [0054]-[0058]; and --[0127] At step 660, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI from a liver biopsy). The output may be displayed as a list of any toxins found. Further, the system may rank the list based on highest amount of toxin present and/or by more dangerous toxin present. Outputting the toxins may further include saving the information to electronic storage such as digital evidence or forensic system, or displaying the results to a pathology.
0128] FIG. 7 may provide an example of how to use an embodiment of the pathology inference module 206 that may be used for infection detection analysis.
[0129] Occult infection at death can be ascertained during an autopsy by examining blood and tissue cultures that assess for the growth of fungal or bacterial organisms, or by direct sampling and visualization of tissue containing these organisms. The embodiments disclosed herein and further described in method 750 may be configured to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria. This may obviate the reliance on examining time-consuming and imprecise blood and tissue cultures for establishing a cause of infection.
[0130] The machine learning module utilized in FIG. 7 may be trained based on techniques described in relation to FIG. 4A. Specifically, the machine learning module may be trained in this embodiment to determine, based on H&E WSI, whether the presence of an infection is included. H&E slides with labeled infections may be fed to the machine learning model of FIG. 4A to train the system to output one or more infections found on a WSI. Looking for infection may include searching for the presence of fungi, bacteria, or mycobacteria.--, in [0127]-[0130] {“assess for the growth of fungal or bacterial organisms”, and “to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria”, those are microbiomes}; and,
--[0136] At step 760, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI). The output may be displayed as a list of any infectious found. Further, the system may rank the list based on highest amount of infectious disease present and/or by more dangerous toxin present.--, in [0136]).
Claims 3-4, 7, 11-12, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over KUNZ as modified by SEGAL and Poore, and further in view of Colley (US 20210090694 A1).
Re Claim 3, KUNZ as modified by SEGAL and Poore however do not explicitly disclose wherein the cancer is bladder cancer, head and neck cancer, or ovarian cancer,
Colley discloses {determine a characteristic of} the cancer is bladder cancer, head and neck cancer, or ovarian cancer {based on the whole slide image} (see Colley: e.g., -- [0137] Cancer immunotherapies, such as checkpoint blockade therapy and cancer vaccines, have shown striking clinical success in a wide range of malignancies, particularly those with melanoma, lung, bladder, and colorectal cancers. Recently, the Food & Drug Administration (FDA) announced approval of checkpoint blockade to treat cancers with a specific genomic indication known as microsatellite instability (MSI). For the first time, the FDA has recognized the use of a genomic profile, rather than an anatomical tumor type (e.g., endometrial or gastric tumor types), as a criterion in the drug approval process. There are currently only a handful of FDA approved checkpoint blockade antibodies. Based on results from ongoing clinical trials, checkpoint blockade antibodies appear poised to make a major impact in tumors with microsatellite instability. However, challenges in the assessment and use of MSI are considerable.--, in [0137], and, -- [0212] Notably, clinical trial databases and websites often express the clinical trial information using free text (i.e., unstructured data). For example, one trial on clinicaltrials.gov is a Phase I/II clinical trial using the drugs sapacitabine and olaparib. According to the study description, “the FDA (the U.S. Food and Drug Administration) has approved Olaparib as a treatment for metastatic HER2 negative breast cancer with a BRCA mutation. Olaparib is an inhibitor of PARP (poly [adenosine diphosphate-ribose] polymerase), which means that it stops PARP from working. PARP is an enzyme (a type of protein) found in the cells of the body. In normal cells when DNA is damaged, PARP helps to repair the damage. The FDA has not approved Sapacitabine for use in patients including people with this type of cancer. Sapacitabine and drugs of its class have been shown to have antitumor properties in many types of cancer, e.g., leukemia, lung, breast, ovarian, pancreatic and bladder cancer. Sapacitabine may help to stop the growth of some types of cancers. In this research study, the investigators are evaluating the safety and effectiveness of Olaparib in combination with Sapacitabine in BRCA mutant breast cancer.” The trial has fourteen inclusion criteria and twenty exclusion criteria, each described using free text. One inclusion criteria for the clinical trial is “Documented germline mutation in BRCA1 or BRCA2 that is predicted to be deleterious or suspected deleterious (known or predicted to be detrimental/lead to loss of function).--, in [0212], [0362], and [1813]-[1814]);
KUNZ (as modified by SEGAL and Poore) and Colley are combinable as they are in the same field of endeavor: neural network applied in assessments of level/ amount of microbiomes. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify KUNZ (as modified by SEGAL and Poore)’s system using COLLEY’s teachings by including {determine a characteristic of} the cancer is bladder cancer, head and neck cancer, or ovarian cancer {based on the whole slide image} to KUNZ (as modified by SEGAL and Poore)’s determine a characteristic of the cancer in order to improve treatment and diagnosis and the efficacy of medicines ingested by the patient and immunology of bladder cancer based on the information derived from microbiome, such as the viruses and bacteria of a patient (see COLLEY: e.g. in [1768]).
Re Claim 4, KUNZ as modified by SEGAL, Poore and COLLEY further disclose wherein the convolutional neural network is trained based on Formalin-Fixed Paraffin-Embedded (FFPE) diagnostic images sourced from a bladder cohort (see COLLEY: e.g., -- [2758] In some examples, an objective of the present techniques is to combine RNA seq data across many different datasets, overcoming the technical differences in sample collection methods used by many labs today. As noted above, different sources of bias can affect RNA seq datasets, these include biases based on tissue type, e.g., fresh, frozen or formalin fixed, paraffin embedded (FFPE).--, in [2758]).
Re Claim 7, KUNZ as modified by SEGAL. Poore and COLLEY further disclose determine if the patient is responding to the cancer treatment based on the categorization of the whole slide image associated with the patient (see KUNZ: e.g., -- [0058] Next, data ingested may be inserted into a salient region detection module 204, as described in greater detail below. A salient region detection module 204, as further described below, may be used to identify salient regions to be analyzed for each digital image. A salient region may be an image or one or more areas of an image that are considered relevant to a pathologist determining a diagnosis or treatment. A salient region may be dependent on what type of analysis is being performed on a slide. For example, when performing analysis of a whole slide image to determine which drug to utilize to target a cancer, the areas of a whole slide image that includes the cancerous region or the areas of the cancer that have the most immune cells (e.g., tumor infiltrating lymphoctyes) contained within the cancer may be considered the salient regions. A salient region may be specific to a particular application of the invention. Further, the salient region may be particular to specific organs or may depend on specific toxins. For example, for certain toxins, the exact morphology of the necrotic regions may be relevant to a diagnosis and thus be considered salient regions in an image. In another example, within kidney tissues, the salient region may be in and around the glomerulus. The detection of a salient region may be done manually by a user or may be done automatically using AI/ML. An entire image or specific image regions may be identified as salient. The entire disclosure of U.S. Non-Provisional application Ser. No. 17/313,617 filed May 6, 2021 is hereby incorporated herein by reference in its entirety.
[0059] Next, the digital whole slide images from the data ingestion 202, which may or not have had a salient region identified, may be provided to a pathology inference module 206. A pathology inference module, as further described below, may be used to infer one or more fields in one or more of an autopsy report, toxin report, cardiovascular report, or cause of death using machine learning and computer vision from the one or more digital image(s). The pathology inference module may incorporate spatial information from disparate regions in an image.--, in [0058]-[0059]; and, -- [0112] At step 552 the system (e.g., the intake module 136 of slide analysis tool 101) may receive digital images (e.g., H&E whole slide images) of pathology specimens from a deceased human/animal may be received into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).--, in [0058]-[0059]); and
continue treating the patient with the cancer treatment in response to determining the patient is responding to the cancer treatment see KUNZ: e.g., -- [0058] Next, data ingested may be inserted into a salient region detection module 204, as described in greater detail below. A salient region detection module 204, as further described below, may be used to identify salient regions to be analyzed for each digital image. A salient region may be an image or one or more areas of an image that are considered relevant to a pathologist determining a diagnosis or treatment. A salient region may be dependent on what type of analysis is being performed on a slide. For example, when performing analysis of a whole slide image to determine which drug to utilize to target a cancer, the areas of a whole slide image that includes the cancerous region or the areas of the cancer that have the most immune cells (e.g., tumor infiltrating lymphoctyes) contained within the cancer may be considered the salient regions. A salient region may be specific to a particular application of the invention. Further, the salient region may be particular to specific organs or may depend on specific toxins. For example, for certain toxins, the exact morphology of the necrotic regions may be relevant to a diagnosis and thus be considered salient regions in an image. In another example, within kidney tissues, the salient region may be in and around the glomerulus. The detection of a salient region may be done manually by a user or may be done automatically using AI/ML. An entire image or specific image regions may be identified as salient. The entire disclosure of U.S. Non-Provisional application Ser. No. 17/313,617 filed May 6, 2021 is hereby incorporated herein by reference in its entirety.
[0059] Next, the digital whole slide images from the data ingestion 202, which may or not have had a salient region identified, may be provided to a pathology inference module 206. A pathology inference module, as further described below, may be used to infer one or more fields in one or more of an autopsy report, toxin report, cardiovascular report, or cause of death using machine learning and computer vision from the one or more digital image(s). The pathology inference module may incorporate spatial information from disparate regions in an image.--, in [0058]-[0059]; and, -- [0112] At step 552 the system (e.g., the intake module 136 of slide analysis tool 101) may receive digital images (e.g., H&E whole slide images) of pathology specimens from a deceased human/animal may be received into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).--, in [0058]-[0059]; also see Colley: e.g., -- [0137] Cancer immunotherapies, such as checkpoint blockade therapy and cancer vaccines, have shown striking clinical success in a wide range of malignancies, particularly those with melanoma, lung, bladder, and colorectal cancers. Recently, the Food & Drug Administration (FDA) announced approval of checkpoint blockade to treat cancers with a specific genomic indication known as microsatellite instability (MSI). For the first time, the FDA has recognized the use of a genomic profile, rather than an anatomical tumor type (e.g., endometrial or gastric tumor types), as a criterion in the drug approval process. There are currently only a handful of FDA approved checkpoint blockade antibodies. Based on results from ongoing clinical trials, checkpoint blockade antibodies appear poised to make a major impact in tumors with microsatellite instability. However, challenges in the assessment and use of MSI are considerable.--, in [0137], and, -- [0212] Notably, clinical trial databases and websites often express the clinical trial information using free text (i.e., unstructured data). For example, one trial on clinicaltrials.gov is a Phase I/II clinical trial using the drugs sapacitabine and olaparib. According to the study description, “the FDA (the U.S. Food and Drug Administration) has approved Olaparib as a treatment for metastatic HER2 negative breast cancer with a BRCA mutation. Olaparib is an inhibitor of PARP (poly [adenosine diphosphate-ribose] polymerase), which means that it stops PARP from working. PARP is an enzyme (a type of protein) found in the cells of the body. In normal cells when DNA is damaged, PARP helps to repair the damage. The FDA has not approved Sapacitabine for use in patients including people with this type of cancer. Sapacitabine and drugs of its class have been shown to have antitumor properties in many types of cancer, e.g., leukemia, lung, breast, ovarian, pancreatic and bladder cancer. Sapacitabine may help to stop the growth of some types of cancers. In this research study, the investigators are evaluating the safety and effectiveness of Olaparib in combination with Sapacitabine in BRCA mutant breast cancer.” The trial has fourteen inclusion criteria and twenty exclusion criteria, each described using free text. One inclusion criteria for the clinical trial is “Documented germline mutation in BRCA1 or BRCA2 that is predicted to be deleterious or suspected deleterious (known or predicted to be detrimental/lead to loss of function).--, in [0212], [0362], and [1813]-[1814]). See the similar obviousness and motivation statements for the references combination as addressed above for claim 3.
Re Claims 11-12, claims 11-12 are the corresponding method claim to claims 3-4, respectively. Claims 11-12 thus are rejected for the similar reasons for claims 3-4. See above discussions with regard to claims 3-4 respectively. Further, KUNZ as modified by SEGAL, Poore and Colley further disclose a method of assessing a patient’ s response to a cancer treatment wherein the patient has cancer (see KUNZ: e.g., -- [0042] Specifically, FIG. 1A illustrates an electronic network 120 that may be connected to servers at hospitals, laboratories, and/or doctors' offices, etc. For example, physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125, etc., may each be connected to an electronic network 120, such as the Internet, through one or more computers, servers, and/or handheld mobile devices. According to an exemplary embodiment of the present disclosure, the electronic network 120 may also be connected to server systems 110, which may include processing devices that are configured to implement a tissue viewing platform 100, which includes a slide analysis tool 101 for determining specimen property or image property information pertaining to digital pathology image(s), and using machine learning to classify a specimen, according to an exemplary embodiment of the present disclosure.
[0043] The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may create or otherwise obtain images of one or more patients' cytology specimen(s), histopathology specimen(s), slide(s) of the cytology specimen(s), digitized images of the slide(s) of the histopathology specimen(s), or any combination thereof. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytology information, etc. The physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125 may transmit digitized slide images and/or patient-specific information to server systems 110 over the electronic network 120. Server systems 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician servers 121, hospital servers 122, clinical trial servers 123, research lab servers 124, and/or laboratory information systems 125. Server systems 110 may also include processing devices for processing images and data stored in the one or more storage devices 109. Server systems 110 may further include one or more machine learning tool(s) or capabilities. For example, the processing devices may include a machine learning tool for a tissue viewing platform 100,--, in [0042]-[0043], and,
-- [0054] The training image intake module 132 may create or receive a dataset comprising one or more training images corresponding to either or both of images of a human and/or animal tissue and images that are graphically rendered. For example, the training images may be received from any one or any combination of the server systems 110, physician servers 121, and/or laboratory information systems 125. This dataset may be kept on a digital storage device. The training slide matching module 133 may intake training data related to a cause of death. For example, training slide module 133 training data may include receiving one or more images (e.g., WSIs) of a deceased human or animal. Further, the training data may include information such as age, ethnicity, and ancillary test results. The training data may also include biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information can also be ingested, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB). The training slide module 133 may intake full WSIs, or may intake one or more tiles of WSIs. The training slide module 133 may include the ability to break an inputted WSI into tiles to perform further analysis of individual tiles of a WSI. The training slide matching module 133 may utilize convolutional neural network (“CNN”), a Graph Neural Network (GNN), CoordConv, Capsule network, Random Forest Support Vector Machine, Transformer trained directly with the appropriate loss function in order to help provide training for the machine learning techniques described herein. Training slide module may also be used to train a machine learning module to determine a cause of death, predicts the fields on an autopsy report, cardiac arrhythmogenic genes, contributing factors of cardiovascular disease, liver toxins, and/or cause of miscarriage. The slide background module 134 may analyze images of tissues and determine a background within a digital pathology image. It is useful to identify a background within a digital pathology slide to ensure tissue segments are not overlooked.--, in [0054]; and,
-- [0058] Next, data ingested may be inserted into a salient region detection module 204, as described in greater detail below. A salient region detection module 204, as further described below, may be used to identify salient regions to be analyzed for each digital image. A salient region may be an image or one or more areas of an image that are considered relevant to a pathologist determining a diagnosis or treatment. A salient region may be dependent on what type of analysis is being performed on a slide. For example, when performing analysis of a whole slide image to determine which drug to utilize to target a cancer, the areas of a whole slide image that includes the cancerous region or the areas of the cancer that have the most immune cells (e.g., tumor infiltrating lymphoctyes) contained within the cancer may be considered the salient regions. A salient region may be specific to a particular application of the invention. Further, the salient region may be particular to specific organs or may depend on specific toxins. For example, for certain toxins, the exact morphology of the necrotic regions may be relevant to a diagnosis and thus be considered salient regions in an image. In another example, within kidney tissues, the salient region may be in and around the glomerulus. The detection of a salient region may be done manually by a user or may be done automatically using AI/ML. An entire image or specific image regions may be identified as salient. The entire disclosure of U.S. Non-Provisional application Ser. No. 17/313,617 filed May 6, 2021 is hereby incorporated herein by reference in its entirety.
[0059] Next, the digital whole slide images from the data ingestion 202, which may or not have had a salient region identified, may be provided to a pathology inference module 206. A pathology inference module, as further described below, may be used to infer one or more fields in one or more of an autopsy report, toxin report, cardiovascular report, or cause of death using machine learning and computer vision from the one or more digital image(s). The pathology inference module may incorporate spatial information from disparate regions in an image.--, in [0058]-[0059]; -- [0112] At step 552 the system (e.g., the intake module 136 of slide analysis tool 101) may receive digital images (e.g., H&E whole slide images) of pathology specimens from a deceased human/animal may be received into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).
[0113] In one embodiment, the system (e.g., the intake module 136 of slide analysis tool 101) may receive a gross description. The system may further receive deceased human/animal information (e.g., age, ethnicity, ancillary test results, etc.) that may be ingested to stratify and split the system for machine learning. The system may additionally ingest biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB).--, in [0112]-[0113], and [0123]-[0125]; and,
-- By solely examining small tissue samples, a human body may have significantly less lab processes performed on it, leaving the body in a more dignified state. Another useful aspect of this embodiment is that it may be used by drug companies to help determine cause of death during clinical studies of humans and animals. This may provide more accurate information to the drug companies and help studies become safer and more efficient.
[0100] FIG. 5A may provide an example of how to train an embodiment of the pathology inference module 206 that may be used for cardiac (heart) assessment for early deaths and is capable of predicting cardiac particular genes. FIG. 5B may provide an example of how to use an embodiment of the pathology inference module 206 that may be used for cardiac (heart) assessment for early deaths and is capable of predicting cardiac particular genes.--, [0098]-[0101], and [0118]-[0120]).
Re Claims 18-19, claims 18-19 are the corresponding medium claim to claims 3-4, respectively. Claims 18-19 thus are rejected for the similar reasons for claims 3-4. See above discussions with regard to claims 3-4 respectively. Further, KUNZ as modified by SEGAL, Poore and Colley further disclose at least one machine-readable non-transitory medium comprising a plurality of instructions, executed on a computing device, to facilitate the computing device to perform the method (see KUNZ: e.g.,, --[0164] As shown in FIG. 11, device 1100 may include a central processing unit (CPU) 1120. CPU 1120 may be any type of processor device including, for example, any type of special purpose or a general-purpose microprocessor device. As will be appreciated by persons skilled in the relevant art, CPU 1120 also may be a single processor in a multi-core/multiprocessor system, such system operating alone, or in a cluster of computing devices operating in a cluster or server farm. CPU 1120 may be connected to a data communication infrastructure 1110, for example a bus, message queue, network, or multi-core message-passing scheme.
[0165] Device 1100 may also include a main memory 1140, for example, random access memory (RAM), and also may include a secondary memory 1130. Secondary memory 1130, for example a read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive.--, in [0163]-[0165]; and, --[0054] The training image intake module 132 may create or receive a dataset comprising one or more training images corresponding to either or both of images of a human and/or animal tissue and images that are graphically rendered. For example, the training images may be received from any one or any combination of the server systems 110, physician servers 121, and/or laboratory information systems 125. This dataset may be kept on a digital storage device. The training slide matching module 133 may intake training data related to a cause of death. For example, training slide module 133 training data may include receiving one or more images (e.g., WSIs) of a deceased human or animal. Further, the training data may include information such as age, ethnicity, and ancillary test results. The training data may also include biomarkers such as genomic/epigenomic/transcriptomic/proteomic/microbiome information can also be ingested, e.g., point mutations, fusion events, copy number variations, microsatellite instabilities (MSI), or tumor mutation burden (TMB). The training slide module 133 may intake full WSIs, or may intake one or more tiles of WSIs. The training slide module 133 may include the ability to break an inputted WSI into tiles to perform further analysis of individual tiles of a WSI. The training slide matching module 133 may utilize convolutional neural network (“CNN”), a Graph Neural Network (GNN), CoordConv, Capsule network, Random Forest Support Vector Machine, Transformer trained directly with the appropriate loss function in order to help provide training for the machine learning techniques described herein. Training slide module may also be used to train a machine learning module to determine a cause of death, predicts the fields on an autopsy report, cardiac arrhythmogenic genes, contributing factors of cardiovascular disease, liver toxins, and/or cause of miscarriage. The slide background module 134 may analyze images of tissues and determine a background within a digital pathology image.--, in [0054]-[0058]; and --[0127] At step 660, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI from a liver biopsy). The output may be displayed as a list of any toxins found. Further, the system may rank the list based on highest amount of toxin present and/or by more dangerous toxin present. Outputting the toxins may further include saving the information to electronic storage such as digital evidence or forensic system, or displaying the results to a pathology.
0128] FIG. 7 may provide an example of how to use an embodiment of the pathology inference module 206 that may be used for infection detection analysis.
[0129] Occult infection at death can be ascertained during an autopsy by examining blood and tissue cultures that assess for the growth of fungal or bacterial organisms, or by direct sampling and visualization of tissue containing these organisms. The embodiments disclosed herein and further described in method 750 may be configured to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria. This may obviate the reliance on examining time-consuming and imprecise blood and tissue cultures for establishing a cause of infection.
[0130] The machine learning module utilized in FIG. 7 may be trained based on techniques described in relation to FIG. 4A. Specifically, the machine learning module may be trained in this embodiment to determine, based on H&E WSI, whether the presence of an infection is included. H&E slides with labeled infections may be fed to the machine learning model of FIG. 4A to train the system to output one or more infections found on a WSI. Looking for infection may include searching for the presence of fungi, bacteria, or mycobacteria.--, in [0127]-[0130] {“assess for the growth of fungal or bacterial organisms”, and “to assess H&E WSI for the presence of fungi, bacteria, or mycobacteria”, those are microbiomes}; and,
--[0136] At step 760, the system (e.g., the output interface 138 of slide analysis tool 101) may output whether one or more toxins were present from a WSI (e.g., an H&E WSI). The output may be displayed as a list of any infectious found. Further, the system may rank the list based on highest amount of infectious disease present and/or by more dangerous toxin present.--, in [0136]).
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662