DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group I claims 1-7 and 12 in the reply filed on 12/01/2025 is acknowledged. Accordingly, claims 8-11 and 13-14 are withdrawn.
Examiner notes
Regarding 101 abstract idea, the claim recites determining corresponding scores, and combining model output which could be abstract idea. However, the claim calculates scores, requires multiple trained models, map the input into a latent space, compares the latent representation to dataset fingerprints derived from the training data and uses the resulted scores to control how the trained model outputs are combined. Therefore, claim 1 is eligible under 101 abstract idea because the claim recites improvement to a technical machine learning process, which makes it a practical application.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-7 and 12 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sjolund et al. (US 20190332900 A1).
Regarding claim 1.
Sjolund teaches a prediction system (see ¶ 2, “the present disclosure pertains to the generation, training, and use of neural networks adapted for processing imaging data from among multiple types and mode variations of imaging modalities, including with uses for medical imaging processing and radiotherapy treatment system operations.”, also ¶ 152, “the model may be the model adapted to produce an output (e.g., an inference, prediction, etc.) from subsequent medical imaging data, according to the training that is performed with the fused latent representation.”) for applying multiple trained models to an input instance (see ¶ 35, “an image processing model (e.g., a neural network model) that learns a general representation of multimodal imaging data in an unsupervised manner. This model is designed to be insensitive to missing imaging modalities, and thus is adaptable to apply trained concepts to images of a second modality output even if trained on images from a first modality output.”, also see ¶ 77, encoder, also see ¶ 136, “FIG. 6 illustrates a data flow diagram of an exemplary encoding process performed with an encoder model, used in connection with training and use of a modality-agnostic imaging processing model. As indicated above (e.g., in FIG. 4), multiple encoders may be used as part of a single fusion model.”, also see ¶¶ 80, 94 and 106, i.e. multiple encoders correspond to multiple trained models),
wherein the multiple trained models are trained on respective training datasets (see ¶ 136, “FIG. 6 illustrates a data flow diagram of an exemplary encoding process performed with an encoder model, used in connection with training and use of a modality-agnostic imaging processing model. As indicated above (e.g., in FIG. 4), multiple encoders may be used as part of a single fusion model.”, also see ¶¶ 80, 94 and 106, i.e. multiple encoders correspond to multiple trained models), comprising: a data interface configured to access the multiple trained models in the form of a combined model (see ¶ 136, “FIG. 6 illustrates a data flow diagram of an exemplary encoding process performed with an encoder model, used in connection with training and use of a modality-agnostic imaging processing model. As indicated above (e.g., in FIG. 4), multiple encoders may be used as part of a single fusion model.”, i.e. wherein single fusion model corresponds to combined model, also see Figures 3 and 4, items 320 and 420 show multiple trained models forming a combined model sharing data which corresponds to data interface access),
the combined model defining a latent space (see ¶ 77, “Encoding. In the encoding operation (320), each input datum (310) is mapped to a shared latent representation”, also see ¶ 79, “Downstream tasks. Each of the one or several downstream tasks (370) takes the unified representation as input (360) and produces corresponding output (380).”, also see ¶ 88, “the fused latent representation should produce all outputs as accurately as possible.”), a trained model being configured to determine a model output for the input instance by determining a representation of the input instance in the latent space and determining the model output therefrom, the combined model further comprising respective dataset fingerprints of the multiple trained models, a dataset fingerprint of a trained model characterizing latent space representations of training instances of the training dataset of the trained model (see figures 3 and 4, items 340 and 430 with the latent space representation z.sub.i which are the outputs of the encoders are the dataset fingerprints; also see ¶ 94, “each encoder (420) for a given input (410) outputs (430) a confidence value v.sub.i in addition to the latent space representation z.sub.i. Both of these values are used as inputs to a, possibly learnable, function g.sub.θi (440) which outputs a confidence weighting w.sub.i=g.sub.θi (z.sub.i, v.sub.i) (450)”, also see ¶ 10, "the mapping conserves respective latent variables corresponding to a spatial representation of the respective latent representations", also see ¶ 107-108, “the latent variable has a particular distribution. For example, assume E(X) has a multivariate normal distribution N (μ.sub.1, Σ.sub.1)”, also see ¶ 123, “the fusion operator a is a function that takes the output from the encoders, and “fuses” them, creating the fused latent representation. In the specific language of variational auto-encoders with Gaussian latent variables, every encoder Ei outputs a pair Ei(xi)=(μi, Σi) where μi is the mean, containing the information we're interested in, and Σi is the covariance, indicating the inaccuracy of the result. The fusion is thus a mapping taking any number of such pairs and returning a pair (μ, Σ).”, i.e. where encoder outputs/latent variables learn from data and are treated as characteristics of training data for each encoder),
a processor subsystem configured to provide a combined model output for the multiple trained models (see figures 3 and 4, a combined model output for the multiple trained models) by: obtaining an input instance (see figures 3 and 4, items 310 and 410, input instances);
determining correspondence scores (see ¶ 94, confidence weighting) between the input instance and the multiple trained models, a correspondence score between the input instance and a trained model indicating a correspondence between the input instance and the training dataset of the trained model, the correspondence score being based on a representation of the input instance in the latent space and the dataset fingerprint of the trained model (see figure 4, items z, v, w, and see ¶ 94, "each encoder (420) for a given input (410) outputs (430) a confidence value v.sub.i in addition to the latent space representation z.sub.i. Both of these values are used as inputs to a, possibly learnable, function g.sub.θi (440) which outputs a confidence weighting w.sub.i=g.sub.θi (z.sub.i, v.sub.i) (450). In another example, it the fusion operation may be defined as z{circumflex over ()}=(w, z). It is reasonable to require that the confidence weights are nonnegative and sum to unity ", i.e. with the confidence weights being the correspondence scores, the weight is based on the input instance as it is function of the outputs of the encoders; also see ¶ 96, "a soft-max weighting of the latent space representations"; also see ¶ 103, "forming the weighted sum");
determining model outputs of one or more of the multiple trained models for the input instance (see figures 3 and 4, items 320 and 420, and ¶ 94, "each encoder (420) for a given input (410) outputs (430) a confidence value Vi in addition to the latent space representation", i.e. the latent space representation is the model output);
combining the model outputs into the combined model output according to the determined correspondence scores of the respective trained models (see figures 3 and 4, and ¶ 94, "each encoder (420) for a given input (410) outputs (430) a confidence value v.sub.i in addition to the latent space representation z.sub.i. Both of these values are used as inputs to a, possibly learnable, function g.sub.θi (440) which outputs a confidence weighting w.sub.i=g.sub.θi (z.sub.i, v.sub.i) (450). In another example, it the fusion operation may be defined as z{circumflex over ()}=(w, z). It is reasonable to require that the confidence weights are nonnegative and sum to unity", i.e. the resulting fused representation being the combine model output according to the confidence weighting/value).
Regarding claim 2.
Sjolund teaches the system as in claim 1,
Sjolund further teaches wherein the input instance comprises one or more of an image, a stack of multiple images, and time-series sensor data, for example, of physiological measurements (see ¶ 50, “the image data 152 may include one or more MRI image (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric MRI, 4D cine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), Computed Tomography (CT) images (e.g., 2D CT, Cone beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), Positron Emission Tomography (PET) images, PET-CT images, X-ray images, fluoroscopic images, radiotherapy portal images, Single-Photo Emission Computed Tomography (SPECT) images, Elastography images, Photoacoustic images, Magnetoencephalography (MEG) images, Electroencephalography (EEG) images, or computer generated synthetic images (e.g., pseudo-CT images) and the like. Further, the image data 152 may also include or be associated with medical image processing data, for instance, training images, and ground truth images, contoured images, and dose images.”).
Regarding claim 3.
Sjolund teaches the system as in claim 1,
Sjolund further teaches wherein the dataset fingerprint of the trained model comprises multiple cluster centroids in the latent space, a cluster centroid representing a cluster of training input instances, the processor subsystem being configured to determine a correspondence score between the input instance and the trained model based on similarity values between the input instance the multiple cluster centroids (see ¶ 142, “The latent representations encoded should share similar features in the latent space before fusion, otherwise the fusion of latent vectors become meaningless. The similarity of latent variables thus may be maximized by minimizing the variance of latent vectors among all models, which contributes to the cost function.”).
Regarding claim 4.
Sjolund teaches the system as in claim 1,
Sjolund further teaches wherein the dataset fingerprint of the trained model comprises a generative model, the processor subsystem being configured to determine the correspondence score between the input instance and the trained model based on a likelihood of the latent space representation being generated by the generative model (see ¶ 109, “To solve this, prior approaches have proposed the variational auto-encoder (VAE). This makes the generator probabilistic, which allows a stable computation of the Kullback-Leibler distance. In particular, prior approaches have proposed that the encoder could return a normal distribution, in which case it is sufficient if the output has two parts, μ and Σ, representing the mean and covariance.”, also see ¶ 111, “variational auto-encoder may be tailored for images to exhibit spatial correlations in the latent space.”, i.e. a variational autoencoder (VAE) is a type of generative model).
Regarding claim 5.
Sjolund teaches the system as in claim 1,
Sjolund further teaches wherein the correspondence score between the input instance and the trained model is further based on the input instance and/or a model output of the trained model for the input instance, the dataset fingerprint of the trained model further characterizing training instances and/or training outputs (see figure 4, items z, v, w, and see ¶ 94, "each encoder (420) for a given input (410) outputs (430) a confidence value v.sub.i in addition to the latent space representation z.sub.i. Both of these values are used as inputs to a, possibly learnable, function g.sub.θi (440) which outputs a confidence weighting w.sub.i=g.sub.θi (z.sub.i, v.sub.i) (450). In another example, it the fusion operation may be defined as z{circumflex over ()}=(w, z). It is reasonable to require that the confidence weights are nonnegative and sum to unity i.e. with the confidence weights being the correspondence scores, the score is based on the input instance as it is function of the outputs of the encoders; also see ¶ 96, "a soft-max weighting of the latent space representations"; also see ¶ 103, "forming the weighted sum").
Regarding claim 6.
Sjolund teaches the system as in claim 1,
Sjolund further teaches wherein the processor subsystem is configured to combine the determined model outputs into the combined model output by applying a trainable combination model to the determined correspondence scores and model outputs (see figures 3 and 4, and ¶ 94, "each encoder (420) for a given input (410) outputs (430) a confidence value v.sub.i in addition to the latent space representation z.sub.i. Both of these values are used as inputs to a, possibly learnable, function g.sub.θi (440) which outputs a confidence weighting w.sub.i=g.sub.θi (z.sub.i, v.sub.i) (450). In another example, it the fusion operation may be defined as z{circumflex over ()}=(w, z). It is reasonable to require that the confidence weights are nonnegative and sum to unity", i.e. the resulting fused representation being the combine model output according to the confidence weighting/value, also see ¶ 83, “a more general approach whereby the fusion can be learned”).
Regarding claim 7.
Sjolund teaches the system as in claim 1,
Sjolund further teaches wherein the processor subsystem is further configured to determine a reliability score of the combined model output based on the determined correspondence scores, the reliability score indicating a correspondence between the input instance and the combined training datasets of the multiple trained models (see ¶ 103, “where, further, ƒ.sub.a may be the pixelwise maximum and ƒ.sub.b may be a Gaussian process regression model. To arrive at a single estimate {circumflex over (z)} of the latent space representation, the arguably simplest way is by forming the weighted sum {circumflex over (z)}=w.sub.1z.sub.1+w.sub.2z.sub.2.”, i.e. weighted sum corresponds to reliability score input instance and the combined training datasets of the multiple trained models).
Regarding Claim 12,
Claim 12 is directed to a method. Claim 12 recites: computer-implemented method to perform a process that has limitations similar to the limitations of claim 1. Thus, claim 12 is rejected with the same rationale applied against claim 1.
Related prior arts:
PLUMBLEY et al. (US 20210117869 A1) teaches ensemble may be generated by training a plurality of models based on a plurality of datasets associated with compounds; calculating model performance statistics for each of the plurality of trained models; selecting and storing a set of optimal trained model(s) from the trained models based on the calculated model performance statistics; and forming one or more ensemble models, each ensemble model comprising multiple models from the set of optimal trained model(s).
GOODSITT et al. (US 20210192282 A1) teaches FIG. 7 is a flow diagram of an exemplary process 700 for building an ensemble model for tagging datasets. the data model may be configured to generate data matching statistical and content characteristics of a training dataset.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IMAD M KASSIM whose telephone number is (571)272-2958. The examiner can normally be reached 10:30AM-5:30PM, M-F (E.S.T.).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J. Huntley can be reached at (303) 297 - 4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IMAD KASSIM/Primary Examiner, Art Unit 2129