DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-20 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim 12 is rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Bauchet et al. (US PGPub US2023/0143701 A1, hereby referred to as “Bauchet”).
Consider Claim 12.
Bauchet teaches:
12. A computer-implemented method for providing a first machine learning model, the computer-implemented method comprising: (Bauchet: abstract, One or more methods of predicting expression levels. At least one of the methods includes preprocessing image data representing at least one biological image of a patient to generate preprocessed image data representing at least one preprocessed biological image of the patient; and applying a trained machine learning model to the preprocessed image data to predict, based at least partially on the at least one preprocessed biological image, an expression level of a biological indicator. [0024]-[0027], Figure 1; [0024] FIG. 1 is a block diagram illustrating an example system 100 for predicting expression levels. The system 100 includes computer processors 110. The computer processors 110 include computer-readable memory 111 and computer readable instructions 112. The system 100 also includes a machine learning system 150. The machine learning system 150 includes a machine learning model 120. The machine learning system 150 may be separate from or integrated with the computer processors 110.)
12. receiving a training dataset including a training medical image and related non-imaging data; (Bauchet: [0027] When executing the computer-executable instructions 112, the computer processors 110 are configured to receive image data 160 and generate, based on the image data 160, data representing a predicted expression level 163 of a target biological indicator (for example, a biomarker). The image data 160 can include data representing at least one biological image of a patient. In some implementations, the at least one biological image is an image of an H&E stained biological sample (for example, a tumor biopsy sample). H&E slide image files can include several resolutions of the H&E slide (for example, from 5 times of magnification to 20 times of magnification of cells). In some implementations, the image data 160 received by the computer processors 110 represents the images of cells having 20 times of magnification. Using higher resolution images can result in increased prediction accuracy of machine learning models (for example, the machine learning model 120). When executing the computer-executable instructions 112, the computer processors 110 are configured to preprocess the image data 160. In some implementations, preprocessing the image data 160 includes reducing the size of the image by segmenting certain tissues regions of the image from other portion of the image (for example, segmenting tumor tissue regions from other regions that include normal tissue and artifacts) using automatic image thresholding. In some implementations, Otsu's method is used to segment the tissue regions of the image, which can include applying an Otsu thresholding algorithm that returns a single intensity threshold that separates pixels into two classes, a foreground and a background. The algorithm can exhaustively search for the threshold that minimizes the intra-class variance, defined as a weighted sum of variances of the two classes according to the following formulation:
σω 2(t)=ω0(t)σ0 2(t)+ω1(t)σ1 2(t)
in which weights ω0 and ω1 are the probabilities of the two classes separated by a threshold t, and σ02 and σ12 are the variances of these two classes.)
12. determining output data by applying a second machine learning model to the training medical image, wherein the second machine learning model includes the first machine learning model; (Bauchet: [0027]-[0032], [0029] The machine learning system 150 applies machine learning techniques to train the machine learning model 120 that, when applied to the input data, generates indications of whether the input data items have the associated property or properties, such as probabilities that the input data items have a particular Boolean property, an estimated value of a scalar property, or an estimated value of a vector (i.e., ordered combination of multiple scalars). [0030] As part of the training of the machine learning model 120, the machine learning system 150 can form a training set of input data by identifying a positive training set of input data items that have been determined to have the property in question, and, in some implementations, forms a negative training set of input data items that lack the property in question. [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data.)
12. modifying a parameter of the second machine learning model based on a difference between the related non-imaging data and the output data; (Bauchet: [0024], Figure 1, [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data. [0032] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120 with the feature vectors of the positive training set and the negative training set serving as the inputs. Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—are used in some implementations. The machine learning model 120, when applied to the feature vector extracted from the input data item, outputs an indication of whether the input data item has the property in question, such as a Boolean yes/no estimate, a scalar value representing a probability, a vector of scalar values representing multiple properties, or a nonparametric distribution of scalar values representing different ad no a priori fixed numbers of multiple properties, which may be represented either explicitly or implicitly in a Hilbert or similar infinite dimensional space.)
12. extracting the first machine learning model from the second machine learning model; (Bauchet: [0041] FIG. 3 is a flow diagram illustrating an example architecture 300 for predicting expression levels using a virtual staining machine learning model 380. In the illustrated implementation, architecture 300 includes similar components as the architecture 200 discussed previously with reference to FIG. 2 , and also includes a virtual staining machine learning model 380. In some implementations, the machine learning model 380 includes a conditional cycleGAN model, which can refer to a deep learning model that is capable of learning to map one image domain to another image domain based on paired image data. In the illustrated implementation, the preprocessing module 220 also receives data representing IHC patient images 310 a (for example, images of a patients “special” IHC stains, as described previously) and performs the Otsu segmenting and image tiling (and in some implementations, sub-tiling/patching) discussed previously to generate data representing preprocessed IHC image tiles 310 b (and, in some implementations, sub-tiles). The virtual staining machine learning model 380 is configured to receive the data representing the preprocessed H&E image tiles 210 b and the preprocessed IHC image tiles 310 b to virtually IHC stain the preprocessed H&E image tiles 210 b using the preprocessed IHC image tiles 310 b. As indicated previously, virtual IHC staining can include discovering correlations between an expression of a target protein exhibited in the special IHC stain of the IHC image and the cell morphology exhibited by the H&E image. The virtually stained IHC image tiles 310 c are then received by the machine learning model 240 to generate the data indicating the predicted expression level 210 c. Because the expression level of target proteins can be derived from the IHC images, virtually staining the preprocessed H&E image tiles 210 b can increase the prediction accuracy of the machine learning model 240. [0042] In some implementations, the virtual staining machine learning model 380 is trained by providing the model 380 data representing pairs of H&E images and IHC images so that the model 380 can learn how to transform values of the pixels of the H7E image to different values such that the resulting image approximates an IHC stained image while reserving the cell morphology exhibited by the original H&E images. Each of the approximation and reserving can be measured by loss functions, which can facilitate maximizing the similarity between virtually stained IHC images and real IHC images while minimizing inconsistencies of the cell morphology between the virtually stained IHC images and the original H&E images. If an H&E image and an IHC image are available for the same patient, and pixels of the H&E image and the IHC images can be registered (for example, through an image registration process), the virtual staining model 380 can be trained using methods such as U-nets. If the images are not matched for a patient, or the pixels cannot be registered, the model 380 can be trained using methods like CycleGAN.)
12. and providing the first machine learning model. (Bauchet: [0032] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120 with the feature vectors of the positive training set and the negative training set serving as the inputs. Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—are used in some implementations. The machine learning model 120, when applied to the feature vector extracted from the input data item, outputs an indication of whether the input data item has the property in question, such as a Boolean yes/no estimate, a scalar value representing a probability, a vector of scalar values representing multiple properties, or a nonparametric distribution of scalar values representing different ad no a priori fixed numbers of multiple properties, which may be represented either explicitly or implicitly in a Hilbert or similar infinite dimensional space. [0045] At block 420, a machine learning model is applied to the preprocessed image data to predict an expression level of a target biomarker based on the biological image. [0046] At block 430, the predicted expression level is compared with an expression level threshold to determine whether to perform an IHC screening test for the patient. In some implementations, if the predicted expression level exceeds the expression level threshold, the IHC screening test is performed. If the predicted expression level does not exceed the expression level threshold, the IHC screening test is not performed. For example, the IHC screening test can be performed by a medical expert trained to determined IHC expression proportion scored based on IHC stains. [0047] At block 440, based on the results of the IHC screening test, it is determined whether the patient should be enrolled in a clinical trial.)
Claim 12 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by Teichmann et al. (US PGPub US2022/0319000 A1, hereby referred to as “Teichmann”).
Consider Claim 12.
Teichmann teaches:
12. A computer-implemented method for providing a first machine learning model, the computer-implemented method comprising: (Teichmann: abstract, Various example embodiments pertain to processing images that depict tissue samples using a neural network algorithm. The neural network algorithm includes multiple encoder branches that are copies of each other that share the same parameters. The encoder branches can, accordingly, be referred to as Siamese copies of each other. [0015]-[0016], [0095]-[0100], Figures 2 and 3, [0095] FIG. 2 schematically illustrates a system 60 according to various examples. The system 60 includes a microscope 69 that can provide the input image 90. The input image 90 can be provided to a device 65 of the system 60. The device 65 can include a processor 62 in the memory 63. The device 65 can receive the input image 90 via an interface 64, e.g., from the microscope 69 or from a database 68 such as a picture archiving system (PACS).)
12. receiving a training dataset including a training medical image and related non-imaging data; (Teichmann: [0095] FIG. 2 schematically illustrates a system 60 according to various examples. The system 60 includes a microscope 69 that can provide the input image 90. The input image 90 can be provided to a device 65 of the system 60. The device 65 can include a processor 62 in the memory 63. The device 65 can receive the input image 90 via an interface 64, e.g., from the microscope 69 or from a database 68 such as a picture archiving system (PACS). [0096] The processor 62 can load and execute program code from the memory 63. Upon loading and executing the program code, the processor 62 can perform techniques as described herein such as: performing digital pathology using a NN 100; training the NN 100; obtaining input images 90, e.g., via the interface 64 from the microscope 69 or from the database 68; applying the NN to the input images 90 to infer one or more semantic histopathology features, e.g., MSI, cancer grading, clinical pathway prediction, Gleason scoring and others. [0097] FIG. 3 schematically illustrates aspects with respect to the NN 100. The NN 100 includes multiple encoder branches 101.)
12. determining output data by applying a second machine learning model to the training medical image, (Teichmann: [0082] Then, the NN receives the tiles as input. The NN can include an encoder-decoder architecture. Encoder branches are tasked with processing each of the k tiles. For each tile, a respective encoder branch produces a latent feature data structure such as a feature vector which contains information about the features found in the respective tile. Those latent data structures are then merged by a decoder branch whose purpose it is to produce a joined prediction of the semantic histopathology feature based on the k feature vectors produced by the encoder branches.)
12. wherein the second machine learning model includes the first machine learning model; (Teichmann: [0086] As a general rule, the decoder branch can include one or more of the following layers to implement said aggregating of the latent data structures of the multiple tiles: an averaging layer, a max-pooling layer, a 1-D convolution layer, a fully-connected layer, or a layer implementing a matrix product with a weight kernel. [0087] For instance, the decoder branch could be implemented by an average or max pooling of all k feature vectors, followed by one or more fully-connected layers. Alternatively the merging of the feature vectors can be learned using a 1D-convolution, a fully connected layer or a matrix product with a weight kernel.)
12. modifying a parameter of the second machine learning model based on a difference between the related non-imaging data and the output data; (Teichmann: [0082], [0084] The weights of the encoder branches may be shared between the encoder branches, enabling efficient learning and processing of each layer of the network. This means that the multiple encoder branches processing the multiple tiles can share the same parameters. i.e., the same weights can be used for all encoder branches. This can be enforced during the training, by changing the weights in a shared manner. [0085] Sometimes, a concept of sharing parameters between multiple encoder branches is referred to as Siamese copies. [0097] FIG. 3 schematically illustrates aspects with respect to the NN 100. The NN 100 includes multiple encoder branches 101. [0098] The multiple encoder branches 101 each include multiple layers. They encode the input, i.e., reduce the dimensionality of the respective data structures. The encoder branches 101 process multiple tiles 91-1-91-2 of an input image 90 that depicts a tissue sample. Thereby, for each encoder branch, a respective latent feature data structure 102-1-102-4 is obtained. In the illustrated examples 2-D vectors are obtained. [0099] These latent feature data structures 102-1-102-4 are then processed by a decoder branch 111, to obtain a merged latent feature data structure 112—here, a 3-D matrix—that incorporates information from all latent feature data structures 102-1-102-4.)
12. extracting the first machine learning model from the second machine learning model; (Teichmann: [0100]-[0102], [0103] FIG. 4 is a flowchart of a method according to various examples. The method of FIG. 4 could be executed by a device such as the device 65. More specifically, the method of FIG. 4 could be executed by the processor 62 upon loading respective program code from the memory 63. [0104] At box 3005, a training of the NN for predicting one or more semantic histopathology features associated with the tissue sample based on an input image of the tissue sample is performed. For example, the NN 100 can be trained. [0105] As a general rule, the training is based on a training input image that depicts a tissue sample. A training input image used during training can be similar to the input image used during inference (cf. FIG. 1, input image 90). The same imaging modality and the same configuration of the imaging modality can be used. [0106] Then, a ground-truth label for a semantic histopathology feature is obtained. The ground-truth label can be obtained for the training input image as a whole. A single ground-truth label can be sufficient. [0107] Specifically, it is not required to obtain ground-truth labels for multiple tiles (as would be required in some reference implementations). [0108] It is then possible to perform an end-to-end training of the NN based on the ground-truth label and the training input image. Specifically, end-to-end training denotes a process in which multiple encoder branches and the decoder branch are jointly trained. I.e., weights are jointly adjusted for the multiple encoder branches and the decoder branch based on a single loss value.)
12. and providing the first machine learning model. (Teichmann: [0125]-[0131], [0125] FIG. 8 is a flowchart of a method according to various examples. The method of FIG. 8 illustrates aspects with respect to training of a NN that can infer a semantic histopathology feature associated with a tissue sample. The method of FIG. 8 can be used to train the NN used in FIG. 5. The method of FIG. 8 could be used to train the NN 100 that has been discussed above in connection with FIG. 3. FIG. 8 can, accordingly, implement box 3005 of the method according to FIG. 4. [0126] At box 3205, a training input image is obtained. The training input image is obtained from a microscope or from an imaging database. The training input image depicts a tissue sample. Box 3205 can correspond to box 3105. [0127] Then, at box 3210, a ground-truth labels obtained for the training input image. Specifically, the ground-truth labels can indicate a semantic histopathology feature associated with the tissue sample. Manual annotation would be possible. [0128] Then, at box 3215, multiple tiles are determined for the training input image. Box 3215 can be implemented using the same selection process also used for box 3112 of the method of FIG. 5. [0129] As will be appreciated, the ground-truth labels determined at box 3210 does not need to be determined individually for each tile. This has the advantage of significantly reduced complexity of the annotation process. [0130]-[0131]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-6 and 8-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bauchet et al. (US PGPub US2023/0143701 A1, hereby referred to as “Bauchet”), in view of Teichmann et al. (US PGPub US2022/0319000 A1, hereby referred to as “Teichmann”)
Consider Claims 1 and 13.
Bauchet teaches:
1. A computer-implemented method for providing a similar medical image, the computer-implemented method comprising: / 13. An image providing system for providing a similar medical image, the image providing system comprising: a memory storing computer-executable instructions; and at least one processor configured to execute the computer-executable instructions to cause (Bauchet: abstract, One or more methods of predicting expression levels. At least one of the methods includes preprocessing image data representing at least one biological image of a patient to generate preprocessed image data representing at least one preprocessed biological image of the patient; and applying a trained machine learning model to the preprocessed image data to predict, based at least partially on the at least one preprocessed biological image, an expression level of a biological indicator. [0024]-[0027], Figure 1; [0024] FIG. 1 is a block diagram illustrating an example system 100 for predicting expression levels. The system 100 includes computer processors 110. The computer processors 110 include computer-readable memory 111 and computer readable instructions 112. The system 100 also includes a machine learning system 150. The machine learning system 150 includes a machine learning model 120. The machine learning system 150 may be separate from or integrated with the computer processors 110.)
1. receiving a first medical image related to a first patient; / 13. the image providing system to receive a first medical image related to a first patient, (Bauchet: [0027] When executing the computer-executable instructions 112, the computer processors 110 are configured to receive image data 160 and generate, based on the image data 160, data representing a predicted expression level 163 of a target biological indicator (for example, a biomarker). The image data 160 can include data representing at least one biological image of a patient. In some implementations, the at least one biological image is an image of an H&E stained biological sample (for example, a tumor biopsy sample). H&E slide image files can include several resolutions of the H&E slide (for example, from 5 times of magnification to 20 times of magnification of cells). In some implementations, the image data 160 received by the computer processors 110 represents the images of cells having 20 times of magnification. Using higher resolution images can result in increased prediction accuracy of machine learning models (for example, the machine learning model 120). When executing the computer-executable instructions 112, the computer processors 110 are configured to preprocess the image data 160. In some implementations, preprocessing the image data 160 includes reducing the size of the image by segmenting certain tissues regions of the image from other portion of the image (for example, segmenting tumor tissue regions from other regions that include normal tissue and artifacts) using automatic image thresholding. In some implementations, Otsu's method is used to segment the tissue regions of the image, which can include applying an Otsu thresholding algorithm that returns a single intensity threshold that separates pixels into two classes, a foreground and a background. The algorithm can exhaustively search for the threshold that minimizes the intra-class variance, defined as a weighted sum of variances of the two classes according to the following formulation:
σω 2(t)=ω0(t)σ0 2(t)+ω1(t)σ1 2(t)
in which weights ω0 and ω1 are the probabilities of the two classes separated by a threshold t, and σ02 and σ12 are the variances of these two classes.)
1. determining a first feature vector by applying a first machine learning model to the first medical image, / 13. determine a first feature vector by applying a first machine learning model to the first medical image, (Bauchet: [0027]-[0032], [0029] The machine learning system 150 applies machine learning techniques to train the machine learning model 120 that, when applied to the input data, generates indications of whether the input data items have the associated property or properties, such as probabilities that the input data items have a particular Boolean property, an estimated value of a scalar property, or an estimated value of a vector (i.e., ordered combination of multiple scalars). [0030] As part of the training of the machine learning model 120, the machine learning system 150 can form a training set of input data by identifying a positive training set of input data items that have been determined to have the property in question, and, in some implementations, forms a negative training set of input data items that lack the property in question. [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data.)
1. the first machine learning model having been trained based on training data sets, each of the training datasets including a training medical image and related non-imaging data; / 13. the first machine learning model having been trained based on training datasets, and each of the training datasets including a training medical image and related non-imaging data, (Bauchet: [0030]-[0033], [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data. [0032] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120 with the feature vectors of the positive training set and the negative training set serving as the inputs. Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—are used in some implementations. The machine learning model 120, when applied to the feature vector extracted from the input data item, outputs an indication of whether the input data item has the property in question, such as a Boolean yes/no estimate, a scalar value representing a probability, a vector of scalar values representing multiple properties, or a nonparametric distribution of scalar values representing different ad no a priori fixed numbers of multiple properties, which may be represented either explicitly or implicitly in a Hilbert or similar infinite dimensional space.)
1. receiving a plurality of second feature vectors, the plurality of second feature vectors being a result of applying the first machine learning model to a plurality of second medical images; / 13. receive a plurality of second feature vectors, the plurality of second feature vectors being a result of applying the first machine learning model to a plurality of second medical images, (Bauchet: [0024], Figure 1, [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data. [0032] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120 with the feature vectors of the positive training set and the negative training set serving as the inputs. Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—are used in some implementations. The machine learning model 120, when applied to the feature vector extracted from the input data item, outputs an indication of whether the input data item has the property in question, such as a Boolean yes/no estimate, a scalar value representing a probability, a vector of scalar values representing multiple properties, or a nonparametric distribution of scalar values representing different ad no a priori fixed numbers of multiple properties, which may be represented either explicitly or implicitly in a Hilbert or similar infinite dimensional space.)
1. determining, based on the first feature vector, a similar feature vector from the plurality of second feature vectors; / 13. determine, based on the first feature vector, a similar feature vector from the plurality of second feature vectors, (Bauchet: [0041] FIG. 3 is a flow diagram illustrating an example architecture 300 for predicting expression levels using a virtual staining machine learning model 380. In the illustrated implementation, architecture 300 includes similar components as the architecture 200 discussed previously with reference to FIG. 2 , and also includes a virtual staining machine learning model 380. In some implementations, the machine learning model 380 includes a conditional cycleGAN model, which can refer to a deep learning model that is capable of learning to map one image domain to another image domain based on paired image data. In the illustrated implementation, the preprocessing module 220 also receives data representing IHC patient images 310 a (for example, images of a patients “special” IHC stains, as described previously) and performs the Otsu segmenting and image tiling (and in some implementations, sub-tiling/patching) discussed previously to generate data representing preprocessed IHC image tiles 310 b (and, in some implementations, sub-tiles). The virtual staining machine learning model 380 is configured to receive the data representing the preprocessed H&E image tiles 210 b and the preprocessed IHC image tiles 310 b to virtually IHC stain the preprocessed H&E image tiles 210 b using the preprocessed IHC image tiles 310 b. As indicated previously, virtual IHC staining can include discovering correlations between an expression of a target protein exhibited in the special IHC stain of the IHC image and the cell morphology exhibited by the H&E image. The virtually stained IHC image tiles 310 c are then received by the machine learning model 240 to generate the data indicating the predicted expression level 210 c. Because the expression level of target proteins can be derived from the IHC images, virtually staining the preprocessed H&E image tiles 210 b can increase the prediction accuracy of the machine learning model 240. [0042] In some implementations, the virtual staining machine learning model 380 is trained by providing the model 380 data representing pairs of H&E images and IHC images so that the model 380 can learn how to transform values of the pixels of the H7E image to different values such that the resulting image approximates an IHC stained image while reserving the cell morphology exhibited by the original H&E images. Each of the approximation and reserving can be measured by loss functions, which can facilitate maximizing the similarity between virtually stained IHC images and real IHC images while minimizing inconsistencies of the cell morphology between the virtually stained IHC images and the original H&E images. If an H&E image and an IHC image are available for the same patient, and pixels of the H&E image and the IHC images can be registered (for example, through an image registration process), the virtual staining model 380 can be trained using methods such as U-nets. If the images are not matched for a patient, or the pixels cannot be registered, the model 380 can be trained using methods like CycleGAN.)
1. selecting the similar medical image from the plurality of second medical images, wherein the similar medical image is related to the similar feature vector; / 13. select the similar medical image from the plurality of second medical images, wherein the similar medical image is related to the similar feature vector, (Bauchet: [0042] In some implementations, the virtual staining machine learning model 380 is trained by providing the model 380 data representing pairs of H&E images and IHC images so that the model 380 can learn how to transform values of the pixels of the H7E image to different values such that the resulting image approximates an IHC stained image while reserving the cell morphology exhibited by the original H&E images. Each of the approximation and reserving can be measured by loss functions, which can facilitate maximizing the similarity between virtually stained IHC images and real IHC images while minimizing inconsistencies of the cell morphology between the virtually stained IHC images and the original H&E images. If an H&E image and an IHC image are available for the same patient, and pixels of the H&E image and the IHC images can be registered (for example, through an image registration process), the virtual staining model 380 can be trained using methods such as U-nets. If the images are not matched for a patient, or the pixels cannot be registered, the model 380 can be trained using methods like CycleGAN. [0043] FIG. 4 is a flowchart illustrating an example method 400 for predicting expression levels. In some implementations, the system 100 described earlier with reference to FIG. 1 performs one or more blocks of the method 400. The method 400 includes preprocessing image data (block 410) and predicting an expression level (block 420). In some implementations, the method 400 includes determining whether to perform an IHC screening test (block 430) and determining whether to enroll a patient in a clinical trial (block 440). [0044] At block 410, image data representing a biological image of a patient (for example, an image of an H&E stain) is preprocessed. This can include applying an Otsu thresholding algorithm to segment certain tissues exhibited by the biological image from other components of the medical image. Additionally, or alternatively, preprocessing can include separating the image into a plurality of image tiles. As previously indicated, the image tiles can further be separated into patches (sub-tiles) centered at cell nuclei.)
1. and providing the similar medical image. / 13. and provide the similar medical image. (Bauchet: [0032] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120 with the feature vectors of the positive training set and the negative training set serving as the inputs. Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—are used in some implementations. The machine learning model 120, when applied to the feature vector extracted from the input data item, outputs an indication of whether the input data item has the property in question, such as a Boolean yes/no estimate, a scalar value representing a probability, a vector of scalar values representing multiple properties, or a nonparametric distribution of scalar values representing different ad no a priori fixed numbers of multiple properties, which may be represented either explicitly or implicitly in a Hilbert or similar infinite dimensional space. [0045] At block 420, a machine learning model is applied to the preprocessed image data to predict an expression level of a target biomarker based on the biological image. [0046] At block 430, the predicted expression level is compared with an expression level threshold to determine whether to perform an IHC screening test for the patient. In some implementations, if the predicted expression level exceeds the expression level threshold, the IHC screening test is performed. If the predicted expression level does not exceed the expression level threshold, the IHC screening test is not performed. For example, the IHC screening test can be performed by a medical expert trained to determined IHC expression proportion scored based on IHC stains. [0047] At block 440, based on the results of the IHC screening test, it is determined whether the patient should be enrolled in a clinical trial.)
Even if Bauchet does not specifically teach providing a similar medical image
Teichmann teaches:
1. A computer-implemented method for providing a similar medical image, the computer-implemented method comprising: / 13. An image providing system for providing a similar medical image, the image providing system comprising: a memory storing computer-executable instructions; and at least one processor configured to execute the computer-executable instructions to cause (Teichmann: abstract, Various example embodiments pertain to processing images that depict tissue samples using a neural network algorithm. The neural network algorithm includes multiple encoder branches that are copies of each other that share the same parameters. The encoder branches can, accordingly, be referred to as Siamese copies of each other. [0015]-[0016], [0095]-[0100], Figures 2 and 3, [0095] FIG. 2 schematically illustrates a system 60 according to various examples. The system 60 includes a microscope 69 that can provide the input image 90. The input image 90 can be provided to a device 65 of the system 60. The device 65 can include a processor 62 in the memory 63. The device 65 can receive the input image 90 via an interface 64, e.g., from the microscope 69 or from a database 68 such as a picture archiving system (PACS).)
1. receiving a first medical image related to a first patient; / 13. the image providing system to receive a first medical image related to a first patient, (Teichmann: [0095] FIG. 2 schematically illustrates a system 60 according to various examples. The system 60 includes a microscope 69 that can provide the input image 90. The input image 90 can be provided to a device 65 of the system 60. The device 65 can include a processor 62 in the memory 63. The device 65 can receive the input image 90 via an interface 64, e.g., from the microscope 69 or from a database 68 such as a picture archiving system (PACS). [0096] The processor 62 can load and execute program code from the memory 63. Upon loading and executing the program code, the processor 62 can perform techniques as described herein such as: performing digital pathology using a NN 100; training the NN 100; obtaining input images 90, e.g., via the interface 64 from the microscope 69 or from the database 68; applying the NN to the input images 90 to infer one or more semantic histopathology features, e.g., MSI, cancer grading, clinical pathway prediction, Gleason scoring and others. [0097] FIG. 3 schematically illustrates aspects with respect to the NN 100. The NN 100 includes multiple encoder branches 101.)
1. determining a first feature vector by applying a first machine learning model to the first medical image, / 13. determine a first feature vector by applying a first machine learning model to the first medical image, (Teichmann: [0082] Then, the NN receives the tiles as input. The NN can include an encoder-decoder architecture. Encoder branches are tasked with processing each of the k tiles. For each tile, a respective encoder branch produces a latent feature data structure such as a feature vector which contains information about the features found in the respective tile. Those latent data structures are then merged by a decoder branch whose purpose it is to produce a joined prediction of the semantic histopathology feature based on the k feature vectors produced by the encoder branches.)
1. the first machine learning model having been trained based on training data sets, each of the training datasets including a training medical image and related non-imaging data; / 13. the first machine learning model having been trained based on training datasets, and each of the training datasets including a training medical image and related non-imaging data, (Teichmann: [0086] As a general rule, the decoder branch can include one or more of the following layers to implement said aggregating of the latent data structures of the multiple tiles: an averaging layer, a max-pooling layer, a 1-D convolution layer, a fully-connected layer, or a layer implementing a matrix product with a weight kernel. [0087] For instance, the decoder branch could be implemented by an average or max pooling of all k feature vectors, followed by one or more fully-connected layers. Alternatively the merging of the feature vectors can be learned using a 1D-convolution, a fully connected layer or a matrix product with a weight kernel.)
1. receiving a plurality of second feature vectors, the plurality of second feature vectors being a result of applying the first machine learning model to a plurality of second medical images; / 13. receive a plurality of second feature vectors, the plurality of second feature vectors being a result of applying the first machine learning model to a plurality of second medical images, (Teichmann: [0082], [0084] The weights of the encoder branches may be shared between the encoder branches, enabling efficient learning and processing of each layer of the network. This means that the multiple encoder branches processing the multiple tiles can share the same parameters. i.e., the same weights can be used for all encoder branches. This can be enforced during the training, by changing the weights in a shared manner. [0085] Sometimes, a concept of sharing parameters between multiple encoder branches is referred to as Siamese copies. [0097] FIG. 3 schematically illustrates aspects with respect to the NN 100. The NN 100 includes multiple encoder branches 101.)
1. determining, based on the first feature vector, a similar feature vector from the plurality of second feature vectors; / 13. determine, based on the first feature vector, a similar feature vector from the plurality of second feature vectors, (Teichmann: [0098] The multiple encoder branches 101 each include multiple layers. They encode the input, i.e., reduce the dimensionality of the respective data structures. The encoder branches 101 process multiple tiles 91-1-91-2 of an input image 90 that depicts a tissue sample. Thereby, for each encoder branch, a respective latent feature data structure 102-1-102-4 is obtained. In the illustrated examples 2-D vectors are obtained. [0099] These latent feature data structures 102-1-102-4 are then processed by a decoder branch 111, to obtain a merged latent feature data structure 112—here, a 3-D matrix—that incorporates information from all latent feature data structures 102-1-102-4.)
1. selecting the similar medical image from the plurality of second medical images, wherein the similar medical image is related to the similar feature vector; / 13. select the similar medical image from the plurality of second medical images, wherein the similar medical image is related to the similar feature vector, (Teichmann: [0100]-[0102], [0103] FIG. 4 is a flowchart of a method according to various examples. The method of FIG. 4 could be executed by a device such as the device 65. More specifically, the method of FIG. 4 could be executed by the processor 62 upon loading respective program code from the memory 63. [0104] At box 3005, a training of the NN for predicting one or more semantic histopathology features associated with the tissue sample based on an input image of the tissue sample is performed. For example, the NN 100 can be trained. [0105] As a general rule, the training is based on a training input image that depicts a tissue sample. A training input image used during training can be similar to the input image used during inference (cf. FIG. 1, input image 90). The same imaging modality and the same configuration of the imaging modality can be used. [0106] Then, a ground-truth label for a semantic histopathology feature is obtained. The ground-truth label can be obtained for the training input image as a whole. A single ground-truth label can be sufficient. [0107] Specifically, it is not required to obtain ground-truth labels for multiple tiles (as would be required in some reference implementations). [0108] It is then possible to perform an end-to-end training of the NN based on the ground-truth label and the training input image. Specifically, end-to-end training denotes a process in which multiple encoder branches and the decoder branch are jointly trained. I.e., weights are jointly adjusted for the multiple encoder branches and the decoder branch based on a single loss value.)
1. and providing the similar medical image. / 13. and provide the similar medical image. (Teichmann: [0125] FIG. 8 is a flowchart of a method according to various examples. The method of FIG. 8 illustrates aspects with respect to training of a NN that can infer a semantic histopathology feature associated with a tissue sample. The method of FIG. 8 can be used to train the NN used in FIG. 5. The method of FIG. 8 could be used to train the NN 100 that has been discussed above in connection with FIG. 3. FIG. 8 can, accordingly, implement box 3005 of the method according to FIG. 4. [0126] At box 3205, a training input image is obtained. The training input image is obtained from a microscope or from an imaging database. The training input image depicts a tissue sample. Box 3205 can correspond to box 3105. [0127] Then, at box 3210, a ground-truth labels obtained for the training input image. Specifically, the ground-truth labels can indicate a semantic histopathology feature associated with the tissue sample. Manual annotation would be possible. [0128] Then, at box 3215, multiple tiles are determined for the training input image. Box 3215 can be implemented using the same selection process also used for box 3112 of the method of FIG. 5. [0129] As will be appreciated, the ground-truth labels determined at box 3210 does not need to be determined individually for each tile. This has the advantage of significantly reduced complexity of the annotation process. [0130] Then, at box 3220, the training can be performed. This is based on the ground-truth label, as well as the training input image. [0131])
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Bauchet’s machine learning method and system for predicting expression levels with the teachings of Teichmann for an ANN-based digital pathology The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the trained machine learning model of Bauchet to further predict using Teichmann’s NN-based algorithm for tissue sample parameters. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Bauchet, while the teaching of Teichmann continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of more accurately depicting and identifying tissue samples using a neural network architecture. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 12.
Bauchet teaches:
12. A computer-implemented method for providing a first machine learning model, the computer-implemented method comprising: (Bauchet: abstract, One or more methods of predicting expression levels. At least one of the methods includes preprocessing image data representing at least one biological image of a patient to generate preprocessed image data representing at least one preprocessed biological image of the patient; and applying a trained machine learning model to the preprocessed image data to predict, based at least partially on the at least one preprocessed biological image, an expression level of a biological indicator. [0024]-[0027], Figure 1; [0024] FIG. 1 is a block diagram illustrating an example system 100 for predicting expression levels. The system 100 includes computer processors 110. The computer processors 110 include computer-readable memory 111 and computer readable instructions 112. The system 100 also includes a machine learning system 150. The machine learning system 150 includes a machine learning model 120. The machine learning system 150 may be separate from or integrated with the computer processors 110.)
12. receiving a training dataset including a training medical image and related non-imaging data; (Bauchet: [0027] When executing the computer-executable instructions 112, the computer processors 110 are configured to receive image data 160 and generate, based on the image data 160, data representing a predicted expression level 163 of a target biological indicator (for example, a biomarker). The image data 160 can include data representing at least one biological image of a patient. In some implementations, the at least one biological image is an image of an H&E stained biological sample (for example, a tumor biopsy sample). H&E slide image files can include several resolutions of the H&E slide (for example, from 5 times of magnification to 20 times of magnification of cells). In some implementations, the image data 160 received by the computer processors 110 represents the images of cells having 20 times of magnification. Using higher resolution images can result in increased prediction accuracy of machine learning models (for example, the machine learning model 120). When executing the computer-executable instructions 112, the computer processors 110 are configured to preprocess the image data 160. In some implementations, preprocessing the image data 160 includes reducing the size of the image by segmenting certain tissues regions of the image from other portion of the image (for example, segmenting tumor tissue regions from other regions that include normal tissue and artifacts) using automatic image thresholding. In some implementations, Otsu's method is used to segment the tissue regions of the image, which can include applying an Otsu thresholding algorithm that returns a single intensity threshold that separates pixels into two classes, a foreground and a background. The algorithm can exhaustively search for the threshold that minimizes the intra-class variance, defined as a weighted sum of variances of the two classes according to the following formulation:
σω 2(t)=ω0(t)σ0 2(t)+ω1(t)σ1 2(t)
in which weights ω0 and ω1 are the probabilities of the two classes separated by a threshold t, and σ02 and σ12 are the variances of these two classes.)
12. determining output data by applying a second machine learning model to the training medical image, wherein the second machine learning model includes the first machine learning model; (Bauchet: [0027]-[0032], [0029] The machine learning system 150 applies machine learning techniques to train the machine learning model 120 that, when applied to the input data, generates indications of whether the input data items have the associated property or properties, such as probabilities that the input data items have a particular Boolean property, an estimated value of a scalar property, or an estimated value of a vector (i.e., ordered combination of multiple scalars). [0030] As part of the training of the machine learning model 120, the machine learning system 150 can form a training set of input data by identifying a positive training set of input data items that have been determined to have the property in question, and, in some implementations, forms a negative training set of input data items that lack the property in question. [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data.)
12. modifying a parameter of the second machine learning model based on a difference between the related non-imaging data and the output data; (Bauchet: [0024], Figure 1, [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data. [0032] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120 with the feature vectors of the positive training set and the negative training set serving as the inputs. Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—are used in some implementations. The machine learning model 120, when applied to the feature vector extracted from the input data item, outputs an indication of whether the input data item has the property in question, such as a Boolean yes/no estimate, a scalar value representing a probability, a vector of scalar values representing multiple properties, or a nonparametric distribution of scalar values representing different ad no a priori fixed numbers of multiple properties, which may be represented either explicitly or implicitly in a Hilbert or similar infinite dimensional space.)
12. extracting the first machine learning model from the second machine learning model; (Bauchet: [0041] FIG. 3 is a flow diagram illustrating an example architecture 300 for predicting expression levels using a virtual staining machine learning model 380. In the illustrated implementation, architecture 300 includes similar components as the architecture 200 discussed previously with reference to FIG. 2 , and also includes a virtual staining machine learning model 380. In some implementations, the machine learning model 380 includes a conditional cycleGAN model, which can refer to a deep learning model that is capable of learning to map one image domain to another image domain based on paired image data. In the illustrated implementation, the preprocessing module 220 also receives data representing IHC patient images 310 a (for example, images of a patients “special” IHC stains, as described previously) and performs the Otsu segmenting and image tiling (and in some implementations, sub-tiling/patching) discussed previously to generate data representing preprocessed IHC image tiles 310 b (and, in some implementations, sub-tiles). The virtual staining machine learning model 380 is configured to receive the data representing the preprocessed H&E image tiles 210 b and the preprocessed IHC image tiles 310 b to virtually IHC stain the preprocessed H&E image tiles 210 b using the preprocessed IHC image tiles 310 b. As indicated previously, virtual IHC staining can include discovering correlations between an expression of a target protein exhibited in the special IHC stain of the IHC image and the cell morphology exhibited by the H&E image. The virtually stained IHC image tiles 310 c are then received by the machine learning model 240 to generate the data indicating the predicted expression level 210 c. Because the expression level of target proteins can be derived from the IHC images, virtually staining the preprocessed H&E image tiles 210 b can increase the prediction accuracy of the machine learning model 240. [0042] In some implementations, the virtual staining machine learning model 380 is trained by providing the model 380 data representing pairs of H&E images and IHC images so that the model 380 can learn how to transform values of the pixels of the H7E image to different values such that the resulting image approximates an IHC stained image while reserving the cell morphology exhibited by the original H&E images. Each of the approximation and reserving can be measured by loss functions, which can facilitate maximizing the similarity between virtually stained IHC images and real IHC images while minimizing inconsistencies of the cell morphology between the virtually stained IHC images and the original H&E images. If an H&E image and an IHC image are available for the same patient, and pixels of the H&E image and the IHC images can be registered (for example, through an image registration process), the virtual staining model 380 can be trained using methods such as U-nets. If the images are not matched for a patient, or the pixels cannot be registered, the model 380 can be trained using methods like CycleGAN.)
12. and providing the first machine learning model. (Bauchet: [0032] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120 with the feature vectors of the positive training set and the negative training set serving as the inputs. Different machine learning techniques—such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps—are used in some implementations. The machine learning model 120, when applied to the feature vector extracted from the input data item, outputs an indication of whether the input data item has the property in question, such as a Boolean yes/no estimate, a scalar value representing a probability, a vector of scalar values representing multiple properties, or a nonparametric distribution of scalar values representing different ad no a priori fixed numbers of multiple properties, which may be represented either explicitly or implicitly in a Hilbert or similar infinite dimensional space. [0045] At block 420, a machine learning model is applied to the preprocessed image data to predict an expression level of a target biomarker based on the biological image. [0046] At block 430, the predicted expression level is compared with an expression level threshold to determine whether to perform an IHC screening test for the patient. In some implementations, if the predicted expression level exceeds the expression level threshold, the IHC screening test is performed. If the predicted expression level does not exceed the expression level threshold, the IHC screening test is not performed. For example, the IHC screening test can be performed by a medical expert trained to determined IHC expression proportion scored based on IHC stains. [0047] At block 440, based on the results of the IHC screening test, it is determined whether the patient should be enrolled in a clinical trial.)
Even if Bauchet does not specifically teach “wherein the second machine learning model includes the first machine learning model”
Teichmann teaches:
12. A computer-implemented method for providing a first machine learning model, the computer-implemented method comprising: (Teichmann: abstract, Various example embodiments pertain to processing images that depict tissue samples using a neural network algorithm. The neural network algorithm includes multiple encoder branches that are copies of each other that share the same parameters. The encoder branches can, accordingly, be referred to as Siamese copies of each other. [0015]-[0016], [0095]-[0100], Figures 2 and 3, [0095] FIG. 2 schematically illustrates a system 60 according to various examples. The system 60 includes a microscope 69 that can provide the input image 90. The input image 90 can be provided to a device 65 of the system 60. The device 65 can include a processor 62 in the memory 63. The device 65 can receive the input image 90 via an interface 64, e.g., from the microscope 69 or from a database 68 such as a picture archiving system (PACS).)
12. receiving a training dataset including a training medical image and related non-imaging data; (Teichmann: [0095] FIG. 2 schematically illustrates a system 60 according to various examples. The system 60 includes a microscope 69 that can provide the input image 90. The input image 90 can be provided to a device 65 of the system 60. The device 65 can include a processor 62 in the memory 63. The device 65 can receive the input image 90 via an interface 64, e.g., from the microscope 69 or from a database 68 such as a picture archiving system (PACS). [0096] The processor 62 can load and execute program code from the memory 63. Upon loading and executing the program code, the processor 62 can perform techniques as described herein such as: performing digital pathology using a NN 100; training the NN 100; obtaining input images 90, e.g., via the interface 64 from the microscope 69 or from the database 68; applying the NN to the input images 90 to infer one or more semantic histopathology features, e.g., MSI, cancer grading, clinical pathway prediction, Gleason scoring and others. [0097] FIG. 3 schematically illustrates aspects with respect to the NN 100. The NN 100 includes multiple encoder branches 101.)
12. determining output data by applying a second machine learning model to the training medical image, (Teichmann: [0082] Then, the NN receives the tiles as input. The NN can include an encoder-decoder architecture. Encoder branches are tasked with processing each of the k tiles. For each tile, a respective encoder branch produces a latent feature data structure such as a feature vector which contains information about the features found in the respective tile. Those latent data structures are then merged by a decoder branch whose purpose it is to produce a joined prediction of the semantic histopathology feature based on the k feature vectors produced by the encoder branches.)
12. wherein the second machine learning model includes the first machine learning model; (Teichmann: [0086] As a general rule, the decoder branch can include one or more of the following layers to implement said aggregating of the latent data structures of the multiple tiles: an averaging layer, a max-pooling layer, a 1-D convolution layer, a fully-connected layer, or a layer implementing a matrix product with a weight kernel. [0087] For instance, the decoder branch could be implemented by an average or max pooling of all k feature vectors, followed by one or more fully-connected layers. Alternatively the merging of the feature vectors can be learned using a 1D-convolution, a fully connected layer or a matrix product with a weight kernel.)
12. modifying a parameter of the second machine learning model based on a difference between the related non-imaging data and the output data; (Teichmann: [0082], [0084] The weights of the encoder branches may be shared between the encoder branches, enabling efficient learning and processing of each layer of the network. This means that the multiple encoder branches processing the multiple tiles can share the same parameters. i.e., the same weights can be used for all encoder branches. This can be enforced during the training, by changing the weights in a shared manner. [0085] Sometimes, a concept of sharing parameters between multiple encoder branches is referred to as Siamese copies. [0097] FIG. 3 schematically illustrates aspects with respect to the NN 100. The NN 100 includes multiple encoder branches 101. [0098] The multiple encoder branches 101 each include multiple layers. They encode the input, i.e., reduce the dimensionality of the respective data structures. The encoder branches 101 process multiple tiles 91-1-91-2 of an input image 90 that depicts a tissue sample. Thereby, for each encoder branch, a respective latent feature data structure 102-1-102-4 is obtained. In the illustrated examples 2-D vectors are obtained. [0099] These latent feature data structures 102-1-102-4 are then processed by a decoder branch 111, to obtain a merged latent feature data structure 112—here, a 3-D matrix—that incorporates information from all latent feature data structures 102-1-102-4.)
12. extracting the first machine learning model from the second machine learning model; (Teichmann: [0100]-[0102], [0103] FIG. 4 is a flowchart of a method according to various examples. The method of FIG. 4 could be executed by a device such as the device 65. More specifically, the method of FIG. 4 could be executed by the processor 62 upon loading respective program code from the memory 63. [0104] At box 3005, a training of the NN for predicting one or more semantic histopathology features associated with the tissue sample based on an input image of the tissue sample is performed. For example, the NN 100 can be trained. [0105] As a general rule, the training is based on a training input image that depicts a tissue sample. A training input image used during training can be similar to the input image used during inference (cf. FIG. 1, input image 90). The same imaging modality and the same configuration of the imaging modality can be used. [0106] Then, a ground-truth label for a semantic histopathology feature is obtained. The ground-truth label can be obtained for the training input image as a whole. A single ground-truth label can be sufficient. [0107] Specifically, it is not required to obtain ground-truth labels for multiple tiles (as would be required in some reference implementations). [0108] It is then possible to perform an end-to-end training of the NN based on the ground-truth label and the training input image. Specifically, end-to-end training denotes a process in which multiple encoder branches and the decoder branch are jointly trained. I.e., weights are jointly adjusted for the multiple encoder branches and the decoder branch based on a single loss value.)
12. and providing the first machine learning model. (Teichmann: [0125] FIG. 8 is a flowchart of a method according to various examples. The method of FIG. 8 illustrates aspects with respect to training of a NN that can infer a semantic histopathology feature associated with a tissue sample. The method of FIG. 8 can be used to train the NN used in FIG. 5. The method of FIG. 8 could be used to train the NN 100 that has been discussed above in connection with FIG. 3. FIG. 8 can, accordingly, implement box 3005 of the method according to FIG. 4. [0126] At box 3205, a training input image is obtained. The training input image is obtained from a microscope or from an imaging database. The training input image depicts a tissue sample. Box 3205 can correspond to box 3105. [0127] Then, at box 3210, a ground-truth labels obtained for the training input image. Specifically, the ground-truth labels can indicate a semantic histopathology feature associated with the tissue sample. Manual annotation would be possible. [0128] Then, at box 3215, multiple tiles are determined for the training input image. Box 3215 can be implemented using the same selection process also used for box 3112 of the method of FIG. 5. [0129] As will be appreciated, the ground-truth labels determined at box 3210 does not need to be determined individually for each tile. This has the advantage of significantly reduced complexity of the annotation process. [0130] Then, at box 3220, the training can be performed. This is based on the ground-truth label, as well as the training input image. [0131])
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Bauchet’s machine learning method and system for predicting expression levels with the teachings of Teichmann for an ANN-based digital pathology The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the trained machine learning model of Bauchet to further predict using Teichmann’s NN-based algorithm for tissue sample parameters. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Bauchet, while the teaching of Teichmann continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of more accurately depicting and identifying tissue samples using a neural network architecture. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 14.
The combination of Bauchet and Teichmann teaches:
14. A non-transitory computer program product comprising instructions that, when executed by a computer, cause the computer to carry out the method of claim 1./ 15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to carry out the method of claim 1. (Teichmann: [0125] FIG. 8 is a flowchart of a method according to various examples. The method of FIG. 8 illustrates aspects with respect to training of a NN that can infer a semantic histopathology feature associated with a tissue sample. The method of FIG. 8 can be used to train the NN used in FIG. 5. The method of FIG. 8 could be used to train the NN 100 that has been discussed above in connection with FIG. 3. FIG. 8 can, accordingly, implement box 3005 of the method according to FIG. 4. [0126] At box 3205, a training input image is obtained. The training input image is obtained from a microscope or from an imaging database. The training input image depicts a tissue sample. Box 3205 can correspond to box 3105. [0127] Then, at box 3210, a ground-truth labels obtained for the training input image. Specifically, the ground-truth labels can indicate a semantic histopathology feature associated with the tissue sample. Manual annotation would be possible. [0128] Then, at box 3215, multiple tiles are determined for the training input image. Box 3215 can be implemented using the same selection process also used for box 3112 of the method of FIG. 5. [0129] As will be appreciated, the ground-truth labels determined at box 3210 does not need to be determined individually for each tile. This has the advantage of significantly reduced complexity of the annotation process. [0130] Then, at box 3220, the training can be performed. This is based on the ground-truth label, as well as the training input image. [0131] )
Consider Claims 2 and 16.
The combination of Bauchet and Teichmann teaches:
2. The computer-implemented method according to claim 1, wherein the first medical image is a whole-slide image./ 16. The computer-implemented method according to claim 1, wherein the first medical image is a histopathology slide image. (Teichmann: [0100]-[0102], [0103] FIG. 4 is a flowchart of a method according to various examples. The method of FIG. 4 could be executed by a device such as the device 65. More specifically, the method of FIG. 4 could be executed by the processor 62 upon loading respective program code from the memory 63. [0104] At box 3005, a training of the NN for predicting one or more semantic histopathology features associated with the tissue sample based on an input image of the tissue sample is performed. For example, the NN 100 can be trained. [0105] As a general rule, the training is based on a training input image that depicts a tissue sample. A training input image used during training can be similar to the input image used during inference (cf. FIG. 1, input image 90). The same imaging modality and the same configuration of the imaging modality can be used. [0106] Then, a ground-truth label for a semantic histopathology feature is obtained. The ground-truth label can be obtained for the training input image as a whole. A single ground-truth label can be sufficient. [0107] Specifically, it is not required to obtain ground-truth labels for multiple tiles (as would be required in some reference implementations). [0108] It is then possible to perform an end-to-end training of the NN based on the ground-truth label and the training input image. Specifically, end-to-end training denotes a process in which multiple encoder branches and the decoder branch are jointly trained. I.e., weights are jointly adjusted for the multiple encoder branches and the decoder branch based on a single loss value.)
Consider Claims 3.
The combination of Bauchet and Teichmann teaches:
3. The computer-implemented method of claim 1, wherein the related non-imaging data comprises at least one of the following categories: omics data, diagnosis data, treatment history, and demographic data. (Teichmann: [0089] According to various examples disclosed herein, the inferred at least one histopathology feature can be used in various use cases. For instance, it would be possible to implement a decision-support functionality. Here, the at least one semantic histopathology feature can be used in order to support decision-making of a clinician or a practitioner. For instance, a suggestion could be made regarding a suggested treatment of the patient. The suggested treatment could be helpful to favorably affect the clinical pathway of the patient. [0090] Alternatively or additionally, in a further use case, it would be possible to perform a diagnosis of a pathology of the patient based on the at least one semantic histopathology feature. [0091] Alternatively or additionally, in yet a further use case, it would be possible to generate a clinical report based on the at least one histopathology feature that has been inferred. The clinical report can be generated based on one or more templates that defines the framework of the clinical report. The templates can then be adjusted in order to include details of a diagnosis based on the at least one histopathology feature.)
Consider Claims 4.
The combination of Bauchet and Teichmann teaches:
4. The computer-implemented method of claim 3, wherein the related non-imaging data comprises the omics data, wherein the omics data comprises at least one of genomic data, transcriptomic data, proteomic data, or metabolomic data. (Teichmann: [0092] Alternatively or additionally, and yet a further use case, it would be possible to perform a “rule-in” test. Here, additional diagnostic actions—e.g., genomic-based tumor analysis—could be selectively recommended based on the at least one semantic histopathology feature. FIG. 1 schematically illustrates an input image that can be input to a NN according to various examples. The input image 90 is a WSI. For instance, the dimension of the input image 90 could be tens of thousands of pixels in each one of the X dimension and the Y dimension. [0093] The input sample depicts a tissue sample. The tissue sample could be obtained from a biopsy; a small slice can be prepared and placed on a fixative such as formalin. Then, the input image 90 can be acquired.)
Consider Claims 5.
The combination of Bauchet and Teichmann teaches:
5. The computer-implemented method claim 1, wherein the first machine learning model is trained by training a second machine learning model including the first machine learning model, the second machine learning model is trained to predict non-imaging data based on medical images, and the second machine learning model is a classifier. / 11. The computer-implemented method according to claim 10, wherein the similar medical image includes a plurality of second patches, a second patch of the plurality of second patches is related to the similar feature vector, and the providing the similar medical image includes displaying an association between the first patch and the second patch. (Bauchet: [0028] In some implementations, preprocessing the image data includes segmenting each tissue region into a plurality of tiles. Each of the tiles can correspond to a discrete portion of the original image. In some implementations, the original image includes a size of up to 40000×50000 pixels, which is segmented into tiles sizes of 1024×1024. In some implementations, the tiles may be further segmented into sub-tiles (patches) having sizes of, for example, 27×27. These sub-tiles can be centered around nuclei identified in the tiles using segmentation methods such as the Otsu thresholding algorithm discussed previously. In some implementations, preprocessing the image data includes receiving an IHC image of the patient, and applying a virtual staining machine learning model to virtually stain the H&E image using the IHC image, as discussed later with reference to FIG. 3. [0034] In some implementations, the machine learning model 120 includes a neural network. In some implementations, the neural network includes a convolutional neural network (CNN) architecture. A CNN generally describes a class of deep neural networks and can include a shared-weights architecture and translation invariance characteristics. In some implementations, the machine learning model includes a deep recurrent attention model (DRAM), which can refer to a neural network that includes a built-in degree of translation invariance, but the amount of computations performed can be controlled independently from the input image size. In some implementations, a DRAM includes a deep recurrent neural network trained with reinforcement learning to attend to the most relevant areas of large input patches (for example, the segmented tissue regions of the preprocessed image data discussed previously). In some implementations, the machine learning model 120 is developed using multiple-instances learning (MIL). MTh can refer to a type of supervised learning in which, instead of receiving a set of instances which are individually labeled, the machine learning model 120 receives a set of labeled “bags,” each containing many instances. Each bag can either be labeled negative, indicating that all of the instances in it are negative, or positive, indicating that the bag includes at least one instance which is positive. From the collection of labeled bags, the machine learning model 120 can attempt to either induce a concept that will label individual instances correctly or learn how to label bags without inducing the concept.)
Consider Claims 6 and 17.
The combination of Bauchet and Teichmann teaches:
6. The computer-implemented method of claim 1, wherein the first machine learning model includes a feature unit and an attention unit, wherein the attention unit is based on an attention mechanism, and the determining a first feature vector includes determining a plurality of sub-features by applying the feature unit to the first medical image, and determining the first feature vector by applying the attention unit to the plurality of sub-features. / 17. The computer-implemented method of claim 5, wherein the first machine learning model includes a feature unit and an attention unit, wherein the attention unit is based on an attention mechanism, and the determining a first feature vector includes determining a plurality of sub-features by applying the feature unit to the first medical image, and determining the first feature vector by applying the attention unit to the plurality of sub-features. (Bauchet: [0028] In some implementations, preprocessing the image data includes segmenting each tissue region into a plurality of tiles. Each of the tiles can correspond to a discrete portion of the original image. In some implementations, the original image includes a size of up to 40000×50000 pixels, which is segmented into tiles sizes of 1024×1024. In some implementations, the tiles may be further segmented into sub-tiles (patches) having sizes of, for example, 27×27. These sub-tiles can be centered around nuclei identified in the tiles using segmentation methods such as the Otsu thresholding algorithm discussed previously. In some implementations, preprocessing the image data includes receiving an IHC image of the patient, and applying a virtual staining machine learning model to virtually stain the H&E image using the IHC image, as discussed later with reference to FIG. 3 . While both H&E stains and IHC stains refer to stains used in histopathology, IHC images can refer to images of “special” IHC stains that are more often used to identify expression of a specific protein, which can inform decisions of treating a patient (whereas H&E stains refer to stains that may be more often used for investigating underlying cellular and tissue structures). H&E stains may often be used as a tool for cancer diagnosis and can be available for many patients diagnosed with solid tumor, and the virtual staining machine learning model can be used to discover correlations between cell morphology exhibited in H&E stains with expression of a target protein in IHC stains. When executing the computer-executable instructions 112, and once the image data is preprocessed, the computer processors 110 are configured to apply the machine learning model 120 to the preprocessed image data. Teichmann: [0048] According to one or more example embodiments, computer processing devices may be described as including various functional units that perform various operations and/or functions to increase the clarity of the description. However, computer processing devices are not intended to be limited to these functional units. For example, in one or more example embodiments, the various operations and/or functions of the functional units may be performed by other ones of the functional units. Further, the computer processing devices may perform the operations and/or functions of the various functional units without sub-dividing the operations and/or functions of the computer processing units into these various functional units.)
Consider Claim 8.
The combination of Bauchet and Teichmann teaches:
8. The computer-implemented method according to claim 6, wherein the first machine learning model is a Deep Local Feature Network. (Bauchet: [0034] In some implementations, the machine learning model 120 includes a neural network. In some implementations, the neural network includes a convolutional neural network (CNN) architecture. A CNN generally describes a class of deep neural networks and can include a shared-weights architecture and translation invariance characteristics. In some implementations, the machine learning model includes a deep recurrent attention model (DRAM), which can refer to a neural network that includes a built-in degree of translation invariance, but the amount of computations performed can be controlled independently from the input image size. In some implementations, a DRAM includes a deep recurrent neural network trained with reinforcement learning to attend to the most relevant areas of large input patches (for example, the segmented tissue regions of the preprocessed image data discussed previously). In some implementations, the machine learning model 120 is developed using multiple-instances learning (MIL). MTh can refer to a type of supervised learning in which, instead of receiving a set of instances which are individually labeled, the machine learning model 120 receives a set of labeled “bags,” each containing many instances. Each bag can either be labeled negative, indicating that all of the instances in it are negative, or positive, indicating that the bag includes at least one instance which is positive. From the collection of labeled bags, the machine learning model 120 can attempt to either induce a concept that will label individual instances correctly or learn how to label bags without inducing the concept. [0035] The machine learning model 120 is configured to predict an expression level of a biological indicator (for example, a target biomarker) based on the biological image of the preprocessed image data 160 (that is, using the preprocessed image data). For example, if a clinical trial is directed to immunotherapy for non-small cell lung cancer, a target biomarker can include programmed death ligand 1 (PD-L1). Accordingly, in some implementations, the machine learning model 120 is configured to predict an expression level (for example, an expression score from 0 to 100) of PD-L1 exhibited by the patient based on the biological image. [0036] To train the machine learning model 120 to predict expression levels of a biological indicator, the machine learning system 150 can apply the machine learning model 120 to a training set that includes biological images of a patient (for example, images of H&E stained slides) and known expression levels of the target biological indicator. The known expression levels of the target biological indicator can be manually produced by medical experts based on analyzing IHC stains. The machine learning model 120 can learn to associate certain features of the biological images with the known expression levels. Based on these associations, when the machine learning model 120 is applied to a new biological image, the machine learning model can predict an expression level of the target biological indicator using the learned associations. In some implementations, when training the machine learning model 120, the machine learning system 150 uses a directional loss function, which can optimize the model parameters to take advantage of continuous proportion scores while focusing on the target to be predicted. In some implementations, the direction loss function pushes the predicted values closed to the true values but also forces the prediction to yield the same biomarker status (that is, whether or not a target biomarker is present) as the true one. In some implementations, it achieves this goal by pushing an error more heavily if it yields a wrong biomarker status prediction.)
Consider Claims 9 and 19.
The combination of Bauchet and Teichmann teaches:
9. The computer-implemented method according to claim 1, further comprising: determining a segmentation of a tumor within the first medical image, wherein the determining a first feature vector includes applying the first machine learning model to a part of the first medical image corresponding to the segmented tumor. / 19. The computer-implemented method according to claim 5, further comprising: determining a segmentation of a tumor within the first medical image, wherein the determining a first feature vector includes applying the first machine learning model to a part of the first medical image corresponding to the segmented tumor. (Bauchet: [0027] When executing the computer-executable instructions 112, the computer processors 110 are configured to receive image data 160 and generate, based on the image data 160, data representing a predicted expression level 163 of a target biological indicator (for example, a biomarker). The image data 160 can include data representing at least one biological image of a patient. In some implementations, the at least one biological image is an image of an H&E stained biological sample (for example, a tumor biopsy sample). H&E slide image files can include several resolutions of the H&E slide (for example, from 5 times of magnification to 20 times of magnification of cells). In some implementations, the image data 160 received by the computer processors 110 represents the images of cells having 20 times of magnification. Using higher resolution images can result in increased prediction accuracy of machine learning models (for example, the machine learning model 120). When executing the computer-executable instructions 112, the computer processors 110 are configured to preprocess the image data 160. In some implementations, preprocessing the image data 160 includes reducing the size of the image by segmenting certain tissues regions of the image from other portion of the image (for example, segmenting tumor tissue regions from other regions that include normal tissue and artifacts) using automatic image thresholding. In some implementations, Otsu's method is used to segment the tissue regions of the image, which can include applying an Otsu thresholding algorithm that returns a single intensity threshold that separates pixels into two classes, a foreground and a background. The algorithm can exhaustively search for the threshold that minimizes the intra-class variance, defined as a weighted sum of variances of the two classes according to the following formulation:
σω 2(t)=ω0(t)σ0 2(t)+ω1(t)σ1 2(t)
in which weights ω0 and ω1 are the probabilities of the two classes separated by a threshold t, and σ02 and σ12 are the variances of these two classes. Teichmann: [0068] Example semantic histopathology features that could be inferred by the NN can be selected from the group consisting of: Gleason scoring, cancer grading, cancer stage estimation, clinical pathway prediction, sub-tumor classification, metastasis evaluation, microsatellite instability (MSI) or stability. [0069] For example, a cancer grading can be in accordance with a predefined grading system. Examples would include the Bloom-Richardson score or TNM classification of malignant tumors. Classification systems are available for brain tumors, breast tumors, prostate cancer (Gleason scoring), and other kind of tumors. [0070] A metastasis describes the spread of cancer cells into other areas of the body. This can often be detected by analyzing a tissue sample that is acquired remote from the source cancer site.)
Consider Claims 10 and 20.
The combination of Bauchet and Teichmann teaches:
10. The computer-implemented method according to claim 1, further comprising: dividing the first medical image into a plurality of first patches, wherein the determining a first feature vector includes applying the first machine learning model to a first patch of the plurality of first patches. / 20. The computer-implemented method according to claim 5, further comprising: dividing the first medical image into a plurality of first patches, wherein the determining a first feature vector includes applying the first machine learning model to a first patch of the plurality of first patches. (Bauchet: [0027] When executing the computer-executable instructions 112, the computer processors 110 are configured to receive image data 160 and generate, based on the image data 160, data representing a predicted expression level 163 of a target biological indicator (for example, a biomarker). The image data 160 can include data representing at least one biological image of a patient. In some implementations, the at least one biological image is an image of an H&E stained biological sample (for example, a tumor biopsy sample). H&E slide image files can include several resolutions of the H&E slide (for example, from 5 times of magnification to 20 times of magnification of cells). In some implementations, the image data 160 received by the computer processors 110 represents the images of cells having 20 times of magnification. Using higher resolution images can result in increased prediction accuracy of machine learning models (for example, the machine learning model 120). When executing the computer-executable instructions 112, the computer processors 110 are configured to preprocess the image data 160. In some implementations, preprocessing the image data 160 includes reducing the size of the image by segmenting certain tissues regions of the image from other portion of the image (for example, segmenting tumor tissue regions from other regions that include normal tissue and artifacts) using automatic image thresholding. In some implementations, Otsu's method is used to segment the tissue regions of the image, which can include applying an Otsu thresholding algorithm that returns a single intensity threshold that separates pixels into two classes, a foreground and a background. The algorithm can exhaustively search for the threshold that minimizes the intra-class variance, defined as a weighted sum of variances of the two classes according to the following formulation:
σω 2(t)=ω0(t)σ0 2(t)+ω1(t)σ1 2(t)
in which weights ω0 and ω1 are the probabilities of the two classes separated by a threshold t, and σ02 and σ12 are the variances of these two classes. Teichmann: [0068] Example semantic histopathology features that could be inferred by the NN can be selected from the group consisting of: Gleason scoring, cancer grading, cancer stage estimation, clinical pathway prediction, sub-tumor classification, metastasis evaluation, microsatellite instability (MSI) or stability. [0069] For example, a cancer grading can be in accordance with a predefined grading system. Examples would include the Bloom-Richardson score or TNM classification of malignant tumors. Classification systems are available for brain tumors, breast tumors, prostate cancer (Gleason scoring), and other kind of tumors. [0070] A metastasis describes the spread of cancer cells into other areas of the body. This can often be detected by analyzing a tissue sample that is acquired remote from the source cancer site.)
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Bauchet et al. (US PGPub US2023/0143701 A1, hereby referred to as “Bauchet”), in view of Teichmann et al. (US PGPub US2022/0319000 A1, hereby referred to as “Teichmann”) further in view of Dillman et al. (US PGPub US 2023/0237649, hereby referred to as “Dillman”).
Consider Claims 7 and 18.
The combination of Bauchert and Teichmann teaches:
7. The computer-implemented method according to claim 6, wherein the determining a first feature vector further comprises: reducing dimensionality of the plurality of sub-features by at least one of principal component analysis or applying learnable bottleneck layers. / 18. The computer-implemented method according to claim 17, wherein the determining a first feature vector further comprises: reducing dimensionality of the plurality of sub-features by at least one of principal component analysis or applying learnable bottleneck layers. (Bauchert: [0031] The machine learning system 150 extracts feature values from the input data of the training set, the features being variables deemed potentially relevant to whether or not the input data items have the associated property or properties. An ordered list of the features for the input data is herein referred to as the feature vector for the input data. In some implementations, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principle component analysis (PCA), learned deep features from a neural network, or the like) to reduce the amount of data in the feature vectors for the input data to a smaller, more representative set of data.)
Even if the combination does not teach: applying learnable bottleneck layers:
Dillman teaches:
Dillman teaches:
1. A computer-implemented method for providing a similar medical image, the computer-implemented method comprising: / 13. An image providing system for providing a similar medical image, the image providing system comprising: a memory storing computer-executable instructions; and at least one processor configured to execute the computer-executable instructions to cause (Dillman: abstract, Embodiments provide a deep learning framework to accurately segment liver and spleen using a convolutional neural network with both short and long residual connections to extract their radiomic and deep features from multiparametric MRI. Embodiments will provide an “ensemble” deep learning model to quantify biopsy derived liver fibrosis stage and percentage using the integration of multiparametric MRI radiomic and deep features, MRE data, as well as routinely available clinical data. Embodiments will provide a deep learning model to quantify MRE-derived liver stiffness using multiparametric MRI, radiomic and deep features and routinely-available clinical data. [0056] C.1. Overview. A conceptual overview of embodiments of the current disclosure incorporating three aims is shown in FIG. 2 . For patients with chronic liver diseases (CLD), embodiments will utilize multiparametric MRI 202, MR elastography (MRE) 204, and correlative histologic data. In Aim 1 200, embodiments will provide a deep learning framework to accurately segment liver and spleen using SLRes-U-Net 206 to extract their radiomic and deep features from multiparametric MRI 202. The SLRes-U-Net simultaneously takes multiparametric MRI 202 (e.g., T1-, T2-, and diffusion-weighted images) as inputs and jointly segments liver and spleen 208. Based on the segmentations, such embodiments will run a well-established PyRadiomics pipeline 210 to extract radiomic features 212 as well as implement a pre-trained very deep convolutional neural network 214 (CNN, e.g., GoogLeNet, ResNet) to extract deep features 216. In Aim 2 218, embodiments will provide an “ensemble” deep learning model (LFNet) 220 to quantify biopsy-derived liver fibrosis stage and percentage 222 using the integration of multiparametric MRI 202 radiomic and deep features 212, 216, MRE data 204, as well as routinely-available clinical data 224. Such outputs 222 may be communicated to the user via computer display, electronic messaging, print-out, or any other known mechanism for communication. In Aim 3 226, embodiments will provide a deep learning model (LSNet) 228 to quantify MRE-derived liver stiffness 230 using multiparametric MRI 202, radiomic and deep features 212, 216 and routinely-available clinical data 224. By decoding each model, embodiments will identify, validate, and disseminate a series of the most discriminative imaging and clinical features to the community. Outputs 232 from Aim 2 and/or Aim 3 may include a decision support system and/or an AI Diagnosis Report for clinical radiology. Such outputs 232 may be communicated to the user via computer display, electronic messaging, print-out, or any other known mechanism for communication. The techniques will enhance our abilities to assess CLD in a quantitative, noninvasive, patient-friendly manner as well as to provide more patient-centric, precision medicine.)
1. receiving a first medical image related to a first patient; / 13. the image providing system to receive a first medical image related to a first patient, (Dillman: [0056] C.1. Overview. A conceptual overview of embodiments of the current disclosure incorporating three aims is shown in FIG. 2 . For patients with chronic liver diseases (CLD), embodiments will utilize multiparametric MRI 202, MR elastography (MRE) 204, and correlative histologic data. In Aim 1 200, embodiments will provide a deep learning framework to accurately segment liver and spleen using SLRes-U-Net 206 to extract their radiomic and deep features from multiparametric MRI 202. The SLRes-U-Net simultaneously takes multiparametric MRI 202 (e.g., T1-, T2-, and diffusion-weighted images) as inputs and jointly segments liver and spleen 208. Based on the segmentations, such embodiments will run a well-established PyRadiomics pipeline 210 to extract radiomic features 212 as well as implement a pre-trained very deep convolutional neural network 214 (CNN, e.g., GoogLeNet, ResNet) to extract deep features 216. In Aim 2 218, embodiments will provide an “ensemble” deep learning model (LFNet) 220 to quantify biopsy-derived liver fibrosis stage and percentage 222 using the integration of multiparametric MRI 202 radiomic and deep features 212, 216, MRE data 204, as well as routinely-available clinical data 224. Such outputs 222 may be communicated to the user via computer display, electronic messaging, print-out, or any other known mechanism for communication. In Aim 3 226, embodiments will provide a deep learning model (LSNet) 228 to quantify MRE-derived liver stiffness 230 using multiparametric MRI 202, radiomic and deep features 212, 216 and routinely-available clinical data 224. By decoding each model, embodiments will identify, validate, and disseminate a series of the most discriminative imaging and clinical features to the community. Outputs 232 from Aim 2 and/or Aim 3 may include a decision support system and/or an AI Diagnosis Report for clinical radiology. Such outputs 232 may be communicated to the user via computer display, electronic messaging, print-out, or any other known mechanism for communication. The techniques will enhance our abilities to assess CLD in a quantitative, noninvasive, patient-friendly manner as well as to provide more patient-centric, precision medicine.)
1. determining a first feature vector by applying a first machine learning model to the first medical image, / 13. determine a first feature vector by applying a first machine learning model to the first medical image, (Dillman: [0097] As shown in FIG. 10 , exemplary LSNet 228 includes four input channels, including three imaging channels (T1-weighted (1102), T2-weighted (1104), diffusion-weighted (1106)) and one clinical channel 1108. Each imaging channel 1102, 1104 & 1106 further includes two subchannels, for radiomic and deep features respectively. Radiomic subchannel contains an input layer 1110 handling one-dimensional radiomic feature vector, a fully-connected layer 1116, a batch normalization layer 1120, and a dropout layer 1122. Deep subchannel contains an input layer 1112 handling two-dimensional deep feature maps, a convolutional layer 1118, a batch normalization layer 1120, a dropout layer 1122, and a flatten layer 1124. The structure of clinical channel 1108 may be same as the radiomic subchannel, as both radiomic and clinical features can be vectorized. The radiomic and deep subchannels may be fused 1126 to summarize the latent information from all imaging data; this output may be further fused 1128 with latent information from clinical data. Embodiments may have a soft-max output layer 1114 a for LS classification and a linear regression output layer 114 b for LS prediction (kPa). [0098] D. Summary. Embodiments may result in internally and externally validated prognostic models for quantifying LF and LS. Exemplary DL techniques may be employed for the prediction of other important clinical outcomes in CLD (inflammation, onset of portal hypertension and related complications, time to transplant/transplant listing, mortality, etc.) as well as to other organs and diseases. [0099] E. DeepLiverNet (904)—A machine learning model that can categorically classify the severity of liver stiffening using both anatomic T2-weighted MR images and clinical data for pediatric and young adult patients with known or suspected chronic liver disease)
1. the first machine learning model having been trained based on training data sets, each of the training datasets including a training medical image and related non-imaging data; / 13. the first machine learning model having been trained based on training datasets, and each of the training datasets including a training medical image and related non-imaging data, (Dillman: [0095]-[0097], [0095] DeepLiverNet contained two separate input channels 910, 912 for imaging 906 and clinical data 908, respectively. For the imaging channel 910, transfer learning layers 914 were first designed by reusing a pre-trained very deep CNN model (VGG-19) for T2-weighted MRI deep feature extraction. It was followed by adaptive learning layers 916 to learn the latent imaging features unique to the severity of LS. The clinical channel 912 was designed to capture the latent clinical features. Then, fusion layers 918 were employed to integrate the latent imaging and clinical features. Lastly, a soft-max classifier 920 was used to predict the outcome. The DL model was trained using a stochastic gradient descent algorithm. Rotation and shift-based data augmentation methods were utilized to enlarge the training samples by 10 times. Internal 10-fold cross-validation with 178 examinations shows an AUROC of 0.80 (95% CI: 0.79, 0.81) using deep features and AUROC of 0.86 (95% CI: 0.85, 0.87) when combined with clinical features. External validation of the DL model with an independent dataset consisting of 95 MRI examinations achieved an AUROC of 0.77. Saliency maps (Grad-CAM)70, 72 also were created to show areas of deep feature discrimination (FIGS. 9A-C provide saliency maps showing areas of greatest deep feature discrimination).)
1. receiving a plurality of second feature vectors, the plurality of second feature vectors being a result of applying the first machine learning model to a plurality of second medical images; / 13. receive a plurality of second feature vectors, the plurality of second feature vectors being a result of applying the first machine learning model to a plurality of second medical images, (Dillman: [0097] C.7.3.1. LSNet model design. FIG. 10 provides architecture of the exemplary LSNet model 228 for liver stiffness quantification 230. Such outputs 230 may be communicated to the user via computer display, electronic messaging, print-out, or any other known mechanism for communication. LSNet 228 is a multi-channel multi-task DL model that uses multiparametric radiomic 212 and deep features 216 as well as clinical data 224 as inputs, and that can classify a given patient into one of two groups (e.g., no/mild vs. moderate/severe [≥3 kPa] liver stiffening) as well as predict his/her (kPa). As shown in FIG. 10, exemplary LSNet 228 includes four input channels, including three imaging channels (T1-weighted (1102), T2-weighted (1104), diffusion-weighted (1106)) and one clinical channel 1108. Each imaging channel 1102, 1104 & 1106 further includes two subchannels, for radiomic and deep features respectively. Radiomic subchannel contains an input layer 1110 handling one-dimensional radiomic feature vector, a fully-connected layer 1116, a batch normalization layer 1120, and a dropout layer 1122. Deep subchannel contains an input layer 1112 handling two-dimensional deep feature maps, a convolutional layer 1118, a batch normalization layer 1120, a dropout layer 1122, and a flatten layer 1124. The structure of clinical channel 1108 may be same as the radiomic subchannel, as both radiomic and clinical features can be vectorized. The radiomic and deep subchannels may be fused 1126 to summarize the latent information from all imaging data; this output may be further fused 1128 with latent information from clinical data. Embodiments may have a softmax output layer 1114 a for LS classification and a linear regression output layer 114 b for LS prediction (kPa).)
1. determining, based on the first feature vector, a similar feature vector from the plurality of second feature vectors; / 13. determine, based on the first feature vector, a similar feature vector from the plurality of second feature vectors, (Dillman: [0097] As shown in FIG. 10 , exemplary LSNet 228 includes four input channels, including three imaging channels (T1-weighted (1102), T2-weighted (1104), diffusion-weighted (1106)) and one clinical channel 1108. Each imaging channel 1102, 1104 & 1106 further includes two subchannels, for radiomic and deep features respectively. Radiomic subchannel contains an input layer 1110 handling one-dimensional radiomic feature vector, a fully-connected layer 1116, a batch normalization layer 1120, and a dropout layer 1122. Deep subchannel contains an input layer 1112 handling two-dimensional deep feature maps, a convolutional layer 1118, a batch normalization layer 1120, a dropout layer 1122, and a flatten layer 1124. The structure of clinical channel 1108 may be same as the radiomic subchannel, as both radiomic and clinical features can be vectorized. The radiomic and deep subchannels may be fused 1126 to summarize the latent information from all imaging data; this output may be further fused 1128 with latent information from clinical data. Embodiments may have a soft-max output layer 1114 a for LS classification and a linear regression output layer 114 b for LS prediction (kPa). [0098] D. Summary. Embodiments may result in internally and externally validated prognostic models for quantifying LF and LS. Exemplary DL techniques may be employed for the prediction of other important clinical outcomes in CLD (inflammation, onset of portal hypertension and related complications, time to transplant/transplant listing, mortality, etc.) as well as to other organs and diseases. [0099] E. DeepLiverNet (904)—A machine learning model that can categorically classify the severity of liver stiffening using both anatomic T2-weighted MR images and clinical data for pediatric and young adult patients with known or suspected chronic liver disease)
1. selecting the similar medical image from the plurality of second medical images, wherein the similar medical image is related to the similar feature vector; / 13. select the similar medical image from the plurality of second medical images, wherein the similar medical image is related to the similar feature vector, (Dillman: [0088] 1) First, a diverse model library is built. The diversity plays a key role, and it is a necessary and sufficient condition in building a powerful stacking ensemble model.165, 169, 170 Each of input data types (MRE-derived LS 204, multiparametric MRI radiomic 212 and deep features 216, and routinely-available clinical data 224) may be used to create multiple unique ML models (810, 812, 814, 816, 818, 820 & 822). The model library 826 that may consist of a diverse set of multiple traditional ML models, including SVM (810),45 ANN (818),30 random forest (820),44 logistic regression (812),43 Ridge (814)171 and least absolute shrinkage and selection operator (LASSO) (822).172 Multiple same type of models may be trained with different hyperparameter settings and training datasets; and then 2) the multiple ML classifiers from the model library 826 are integrated using a DL model. Multi-channel, multi-task DNN 824 may be applied as a fusion model. The number of channels may be designed based on the number of models in model library 826. Each input channel may contain several neural network blocks. The multiple input channels may be eventually fused into one output channel through a fusion block. Each block may include a fully-connected layer, a batch normalization layer, and a dropout regularization layer. Followed by the fusion block, a softmax output layer may be used to predict fibrosis stage (F0-4); and a linear regression layer may be used to quantify fibrosis percentage (0-100%). [0089] C.7. Aim 3 (226). A DL model (LSNet) 228 to quantify MRE-derived LS 230 using multiparametric MRI radiomic 212 and deep features 216 as well as clinical features 224.)
1. and providing the similar medical image. / 13. and provide the similar medical image. (Dillman: [0056] In Aim 2 218, embodiments will provide an “ensemble” deep learning model (LFNet) 220 to quantify biopsy-derived liver fibrosis stage and percentage 222 using the integration of multiparametric MRI 202 radiomic and deep features 212, 216, MRE data 204, as well as routinely-available clinical data 224. Such outputs 222 may be communicated to the user via computer display, electronic messaging, print-out, or any other known mechanism for communication. In Aim 3 226, embodiments will provide a deep learning model (LSNet) 228 to quantify MRE-derived liver stiffness 230 using multiparametric MRI 202, radiomic and deep features 212, 216 and routinely-available clinical data 224. By decoding each model, embodiments will identify, validate, and disseminate a series of the most discriminative imaging and clinical features to the community. Outputs 232 from Aim 2 and/or Aim 3 may include a decision support system and/or an AI Diagnosis Report for clinical radiology. Such outputs 232 may be communicated to the user via computer display, electronic messaging, print-out, or any other known mechanism for communication. The techniques will enhance our abilities to assess CLD in a quantitative, noninvasive, patient-friendly manner as well as to provide more patient-centric, precision medicine. [0087] C.6.3.1. LFNet model (220) design. LFNet 220 is designed in an embodiment to be a two-level ensemble model (FIG. 7 ), combining the predictive power of both state-of-the-art DL and traditional ML. FIG. 7 is a block diagram illustrating architecture of an exemplary ensemble LFNet model 220 for liver fibrosis prediction 222. Each input data type (MRE-derived LS 204, multiparametric MRI radiomic 212 and deep features 216, and routinely-available clinical data 224) may be used to create multiple unique ML models (810, 812, 814, 816, 818, 820 & 822). The output of these models is then integrated using a multi-task deep neural network 824. The output 222 of the deep neural network 824 will include both predicted histologic liver fibrosis stage (F0-F4) and fibrosis percentage (0-100%). [0088] 1) First, a diverse model library is built. The diversity plays a key role, and it is a necessary and sufficient condition in building a powerful stacking ensemble model.165, 169, 170 Each of input data types (MRE-derived LS 204, multiparametric MRI radiomic 212 and deep features 216, and routinely-available clinical data 224) may be used to create multiple unique ML models (810, 812, 814, 816, 818, 820 & 822).)
7. The computer-implemented method according to claim 6, wherein the determining a first feature vector further comprises: reducing dimensionality of the plurality of sub-features by at least one of principal component analysis or applying learnable bottleneck layers. / 18. The computer-implemented method according to claim 17, wherein the determining a first feature vector further comprises: reducing dimensionality of the plurality of sub-features by at least one of principal component analysis or applying learnable bottleneck layers. (Dillman: [0053] B.5. Illuminating the “black box” nature of DL methods. Despite DL's many practical successes, there is still skepticism regarding its clinical adoption emanating from its ‘black box’ nature. The inability to understand a model (with millions of model parameters) can lead to mistrust and limit confidence in the method, and, thus, is it may present a barrier to the clinical translation of such techniques.64-66 There has been increasing effort in making DL methods more transparent. In theory, the DL model compresses input data as if by squeezing the information through a bottleneck, retaining only the features most relevant to the learning task.67,68 The compression process is pronounced at the DL model's deeper layers where information relevant to the output labels is preserved at the expense of gradually “forgetting” input information. In practice, methods have been proposed to decompose model decisions in terms of inputs.69)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the combination of Bauchet and Teichmann for a NN-based machine learning method and system for identifying similar image datasets in digital pathology with the deep learning framework of Dillman as they are all directed towards the same field of endeavor. The determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the trained machine learning model of the combination of Bauchet and Teichmann to further leverage the deep learning framework for accurate segmentation of features of interest as proposed by Dillman. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of the combination of Bauchet and Teichmann, while the teaching of Dillman continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of more accurately segmenting features of interest. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
July 9, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights