DETAILED ACTION
Status of Claims
This is a Final Office Action in response to the arguments and/or amendments filed on 22 June 2026.
Claim(s) 1, 31, and 32 is/are amended. Claim(s) 36 is/are new.
Claim(s) 1-33 and 36 is/are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-33 and 36 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims not listed below are rejected for dependency.
Claim 1 recites “wherein the single architecture deep learning model processes the plurality of dissimilar input feature modalities using a single loss function.” The identified limitation would be unclear to one of ordinary skill in the art.
One of ordinary skill in the art understands the term “loss function” in the machine learning arts to refer to an evaluation operation of the output of a machine learning model during training. The output of the machine learning model is input into a loss function which in turn outputs metrics of the distance between the model’s outputs and the target outputs. One of ordinary skill in the art would not understand a loss function to be used during inference where the model is in practical use.
The present claims do not recite any training operations and appear to relate to a fully trained machine learning model being used for inference.
It would be unclear to one of ordinary skill in the art what it means for the claimed model to “processes the plurality of dissimilar input feature modalities using a single loss function”. The limitation appears to literally require a superfluous calculation of a loss function during inference. However, the limitation could also refer to the model using a single loss function during the model’s training. Both the literal interpretation and the training interpretation would be plausible to one of ordinary skill in the art, rendering the scope of the claim ambiguous. As such, one of ordinary skill in art would not be able to determine the boundaries of the claim and as such the claim is indefinite. Claims 31 and 32 are similarly rejected.
For the purposes of examination, the limitation will be interpreted as requiring that the “single architecture deep learning model” was trained via a training step including a single loss function.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-33 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1, which is representative of claims 31 and 32, recites: a
receiving,
each respective segment corresponding to a different respective input feature modality,
each respective segment including one or more respective preprocessing [steps]
each respective segment including a respective plurality of isolated processing [steps]
the predefined configuring parameters specifying, for each of the three or more segments,
dynamically configuring,
wherein the plurality of dissimilar input feature modalities include at least three of: structured data, time-series data, imaging data, genomic data, or categorical data;
for each of the three or more segments:
preprocessing,
processing,
generating respective segment outputs, wherein the respective segment outputs have different dimensional profiles comprising at least two of: a one-dimensional output, a two-dimensional output, or a three-dimensional output;
processing,
processing,
processing,
The preceding recitation of the claim has had strikethroughs applied to the additional elements beyond the abstract idea to more clearly demonstrate the limitations setting forth the abstract idea. The remaining limitations describe a concept of analyzing multi-modal data to generate clinically relevant information. This concept describes a mental process that an analyst should follow to generate a cross-modality analysis of dissimilar data, similar to the “mental process that a neurologist should follow when testing a patient for nervous system malfunctions” given in MPEP 2106.04(a)(2)(II)(C) as an example of managing personal behavior in the methods of organizing human activity sub-grouping. Therefore, the claims are determined to set forth a method of organizing human activity, and as such are determined to recite an abstract idea.
MPEP 2106, reflecting the 2019 PEG, directs examiners at Step 2A Prong Two to consider whether the additional elements of the claims integrate a recited abstract idea into a practical application.
Claim 1 recites the additional element of one or more processors which are used to implement various steps of the method. Claim 31 recites the additional element of a computing system comprising: one or more processors and one or more memories. Claim 32 recites the additional element of a non-transitory computer-readable medium having stored thereon instructions which are executed via one or more processors. These additional elements are recited at an extremely high level of generality, and are interpreted as generic computing devices used to implement the abstract idea. Per MPEP 2106.05(f), implementing an abstract idea on a generic computing device does not integrate an abstract idea into a practical application in Step 2A Prong Two, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea on a generic computer. As such, these additional elements do not integrate the abstract idea into a practical application.
The claims further recite the additional elements of preprocessing using the one or more respective preprocessing layers, processing via the respective plurality of isolated processing layers, processing using one or more combination layers, processing using one or more fully-connected layers, processing using one or more final output layers, and a respective architecture corresponding to processing layers wherein the single architecture deep learning model processes the plurality of dissimilar input feature modalities using a single loss function. The various layers are interpreted as high level portions of a neural network. Interpreted as such, these additional elements provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP 2106.05(f). These additional elements also merely indicate a field of use or technological environment in which the judicial exception is performed, as this limitation merely confines the use of the abstract idea to a particular technological environment involving neural networks. As such, these additional elements do not integrate the abstract idea into a practical application.
The claims further recite an additional element of reshaping the respective segment outputs having different dimensional profiles to a common dimension. This additional element does not appear to constitute a technical improvement, based on the lack of technical explanation of the specification regarding the additional element. This additional element does not implement the abstract idea with a particular machine or manufacturer. This additional element does not effect a transformation of a particular article, based on 1) the generality of the change, 2) the generality of the segment outputs, 3) the nature of the change being a mere rearrangement of data, and 4) the absence of a physical article. Instead, this additional element appears to be an insignificant manipulation of gathered information. As such, this additional element is considered insignificant extra-solution activity. As such, this additional element does not integrate the abstract idea into a practical application.
There are no further additional elements. When considered as a combination, the additional elements only generally link the abstract idea and insignificant extra-solution activity to a technological environment involving computing devices implementing neural networks. Per MPEP 2106.04(d), the courts have identified generally linking the use of a judicial exception to a particular technological environment as insufficient to integrate a judicial exception into a practical application. Therefore the combination of additional elements does not integrate the abstract idea into a practical application. Therefore the claims are determined to be directed to an abstract idea.
At Step 2B of the Mayo/Alice analysis, examiners are to consider whether the additional elements amount to significantly more than the abstract idea.
As previously noted, the claims recite additional elements which may be interpreted as generic computing devices used to implement the abstract idea. However, per MPEP 2106.05(f), implementing an abstract idea on a generic computing does not add significantly more in Step 2B, similar to how the recitation of the computer in the claim in Alice amounted to mere instructions to apply the abstract idea on a generic computer. As such, these additional elements do not amount to significantly more.
As previously noted, the claims recite additional elements of various processing layers. These additional elements merely indicate a field of use or technological environment in which the judicial exception is performed, as this limitation merely confines the use of the abstract idea to a particular technological environment involving neural networks. As such, these additional elements do not amount to significantly more.
As previously noted, the claims recite an additional element of reshaping the respective segment outputs having different dimensional profiles to a common dimension which was considered insignificant extra-solution activity under Prong Two. At Step 2B, the conventionality of an additional element may be considered in evaluating whether it is insignificant extra-solution activity. Before the priority date of the claimed invention, Nguyen et al. (US 10496884 B1) notes that “Tensor reshaping is a functionality that is available in typical neural network libraries as it is needed to modify C×H×W tensor dimensions” Column 24, Lines 15-17. This further indicates that the identified additional element is insignificant extra-solution activity. As such, this additional element does not amount to significantly more.
There are no further additional elements. When considered as a combination, the additional elements only generally link the abstract idea and insignificant extra-solution activity to a technological environment involving computing devices implementing neural networks. Per MPEP 2106.05, the courts have identified generally linking the use of a judicial exception to a particular technological environment as insufficient to amount to significantly more than a judicial exception. Therefore the combination of additional elements does not amount to significantly more than the abstract idea. Therefore, when considered individually and as an ordered combination, the additional elements of the independent claims do not amount to significantly more than the judicial exception. Thus the independent claims are not patent eligible.
Dependent claims 3-30 and 33 further describe the abstract idea, but these claims continue to recite an abstract idea, albeit a narrowed one. Claims 3-21, 23, 25, 26, and 28-30 recite no further additional elements. The previously identified additional elements, individually and as a combination, do not either integrate the abstract idea into a practical application or amount to significantly more than the abstract idea, for the same reasons as articulated above. Dependent claim 2, 22, 24, and 27 further describes the additional element of the layers, but this additional element continues to amount to no more than an instruction to implement the abstract idea with a generic computing device. The additional elements of these claims, when considered either individually or as a combination, do not either integrate the abstract idea into a practical application or amount to significantly more than the abstract idea, for the same reasons as articulated above. Dependent claim 33 describes receiving the configuration parameters in a file. This additional element only generally links the abstract idea to a technological environment of a computer with a file system, and as such, when considered either individually or as a combination with the other additional elements, does not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea. Thus as the dependent claims remain directed to a judicial exception, and as the additional elements of the claims do not amount to significantly more, the dependent claims are not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5-18, 24, and 31-33 are rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt et al. (US 2021/0145404 A1) in view of Mahmood et al. (US 2022/0367053 A1).
Regarding Claim 1, 31, and 32: Fornwalt discloses a computer-implemented method for configuring a single architecture deep learning model to process a plurality of dissimilar input feature modalities, the method comprising:
receiving, via one or more processors, predefined configuration parameters corresponding to [two] segments, each respective segment corresponding to a different respective input feature modality, each respective segment including one or more respective preprocessing layers, and each respective segment including a respective plurality of isolated processing layers, the predefined configuration parameters specifying, for each of the [two] segments, a respective architecture, one or more respective processing layers, and one or more connections to other segments; dynamically configuring, via one or more processors, each of the [two] or more segments based on the predefined configuration parameters (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051] and Fig. 1).
where the plurality of dissimilar input feature modalities include at least [two] of: structured data, time-series data, imaging data, genomic data, or categorical data for each of the [two] segments (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. … the tabular EHR data layer (Tab). See at least [0051] and Fig. 1).
for each of the [two] segments: preprocessing, via one or more processors, the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate respective preprocessing outputs, and processing, via one or more processors, the respective preprocessing outputs via the respective plurality of isolated processing layers by: generating respective segment outputs, wherein the respective segment outputs have different dimensional profiles comprising at least two of: a one-dimensional output, a two-dimensional output, or a three dimensional output; processing, via one or more processors using the one or more combination layers, the respective segment outputs to generate a combination by: reshaping the respective segment outputs having the different dimensional profiles to a common dimension (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. The convolutional layer (Cony) is shown in the top box with a solid outline and the tabular EHR data layer (Tab) is shown in the bottom box with a dashed outline. The convolutional layer consists of Convolutional Neural Networks (CNN), Batch Normalizations (Batch Norm.), rectified linear units (ReLU), and a three-dimensional Maximum Pooling layer (3D Max Pool). The tabular layer consists of a fully connected layer (Dense) with sigmoid activations and a Drop Out layer. See at least [0051] and Fig. 1. Also: The first model 300 can include a flatten layer 344. See at least [0126]. Examiner’s Note: Flatten layer reshapes its 5x4x2x16 input to a 640 input, which has the same dimensionality as the output of Tab 3).
adding or reducing a weight of at least one of the respective segment outputs; processing, via one or more processors using one or more fully-connected layers, the combination output to generate, based on the set of interactions between the outputs that were captured, a fully-connected output (The tabular layer consists of a fully connected layer (Dense) with sigmoid activations and a Drop Out layer. See at least [0051] and Fig. 1).
processing, via one or more processors using one or more final output layers, the fully-connected output to generate a modeling output providing clinically relevant information (The mortality prediction is output as a risk score that is associated with a predicted mortality of a patient. See at least [0051] and Fig. 1);
wherein the single architecture deep learning model processes the plurality of dissimilar input features modalities using a single loss function (The RMSProp algorithm was used to train the networks. See at least [0090]. Also: As the DNN was trained, the loss (binary cross-entropy) on the validation set was evaluated at each epoch. If the validation loss did not decrease for more than 10 epochs the training was stopped and the performance, in AUC, of the test set was reported. See at least [0091]).
Fornwalt does not expressly disclose the use of three or more segments.
Mahmood teaches a neural network processing system which uses three or more segments (This Experiment demonstrates Pathomic Fusion, an algorithm that fuses different types of data (e.g., histology image, cell graph, and genomic features) into a multimodal tensor. See at least [0071]. Also: The joint multimodal tensor computed by the matrix outer product of these feature vectors would capture important unimodal, bimodal and trimodal interactions of all features of these three modalities, shown in FIG. 5. See at least [0086 and Fig. 5).
Fornwalt provides a multi-modal neural network system that uses separate subnetworks to analyze echocardiographic video and electronic health record data before combining those results for another subnetwork to produce a final result, upon which the claimed invention’s use of a third type of data in a third subnetwork can be seen as an improvement. However, Mahmood demonstrates that the prior art already knew of multi-modal neural network system that that uses separate subnetworks to analyze three separate modalities, including genomic data, to produce a final result. One of ordinary skill in the art could have easily applied the techniques of Mahmood to the system of Fornwalt to incorporate additional information from a genomic modality. Further, one of ordinary skill in the art would have recognized that such an application of Mahmood would have resulted in an improved system which would have more information for producing final risk estimates. As such, the application of Mahmood, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood.
Regarding Claim 2: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the respective plurality of isolated processing layers implement at least one of: i) an artificial neural network; ii) a convolutional neural network; or iii) a Sequence neural network (The convolutional layer (Cony) is shown in the top box with a solid outline and the tabular EHR data layer (Tab) is shown in the bottom box with a dashed outline. See at least [0051]).
Regarding Claim 3: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein receiving the predefined configuration parameters corresponding to the [two] or more segments includes receiving at least one of: i) a digital image or a digital video input; ii) a time series input; iii) an electronic health records data input; iv) an electrocardiogram input; v) an echocardiogram input; vi) a computed tomography input; vii) a magnetic resonance imaging input; viii) a tabular data input; or ix) a natural language string input (For training the DNN, a dataset of 723,754 clinically-acquired videos of the heart consisting of .sup.?45 million images was leveraged. See at least [0065]).
Regarding Claim 5: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein at least one of the plurality of dissimilar input feature modalities includes tabular data; and wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs includes preprocessing at least one of the plurality of dissimilar input feature modalities via: i) categorical feature processing; ii) numerical feature processing; or iii) feature censoring (All continuous variables were cleaned to remove physiologically out-of-limit values (manually defined by a cardiologist), which were presumed to reflect input errors, and set as missing. Eight categorical variables were identified in the echocardiography measurements that each reported five valvular regurgitation and stenosis severity levels (including not assessed) and converted them to forty one-hot encoded binary variables. An ordinal variable reporting diastolic function was also identified and coded it as 1 for normal, 0 for dysfunction (but no grade reported), and 1, 2 and 3 for diastolic dysfunction grades I, II, and III, respectively. For non-echocardiography-derived measurements, such as LDL, HDL, blood pressure, heart rate (if not taken at the study), weight, and height measurements, the most recent past measurement was retrieved, within a 1-year window, relative to the echocardiogram acquisition date. See at least [0108]).
Regarding Claim 6: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein at least one of the plurality of dissimilar input feature modalities includes image and/or video data; and wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs includes preprocessing at least one of the plurality of dissimilar input feature modalities via: i) image size normalization; il) image augmentation; or iii) a Fourier space transformation (For training the DNN, a dataset of 723,754 clinically-acquired videos of the heart consisting of .sup.?45 million images was leveraged. See at least [0065]. Also: Since each video from a view group could potentially have different dimensions, all videos were normalized to the most common row and column dimension pairs of its corresponding view. Each frame was cropped or padded with zeros to match the most common dimensions among the view group, keeping the beam-formed image centered. See at least [0112]).
Regarding Claim 7: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses receiving an expected format; and wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs is based on the expected format (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051] and Fig. 1. Also: the process 100 can be implemented as instructions (e.g., computer readable instructions) on at least one memory, and executed by one or more processors coupled to the at least one memory. See at least [0098]. Also: The process 100 predicts a risk score for the patient based on a neural network. See at least [0098]. Also: As the DNN was trained, the loss (binary cross-entropy) on the validation set was evaluated at each epoch. See at least [0091]. Also: See at least [0106]-[0111]).
Regarding Claim 8: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the expected format includes one or more of: (i) a single numeric value; (ii) a range of numeric values; or (iii) a Boolean value (At 608, the process 600 can receive a risk score from the trained model. In some embodiments, risk score can be the risk score 520 in FIG. 19. In some embodiments, the risk score can be a mortality risk score. See at least [0165]).
Regarding Claim 9: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the combination output contributes to one or more loss functions that include one or more of: i) a regression function; ii) a binary classification function; iii) a multi-class classification function; or iv) a survival modeling function (As the DNN was trained, the loss (binary cross-entropy) on the validation set was evaluated at each epoch. See at least [0091] and Fig. 1).
Regarding Claim 10: Fornwalt in view of Mahmood makes obvious the above limitations. The limitation wherein the regression function is a linear activation function included in one or more neurons of a dense output layer further limits a limitation claimed in the alternative. As Fornwalt in view of Mahmood teaches an alternative to the further limited option, Fornwalt in view of Mahmood continues to make obvious the claimed invention.
Regarding Claim 11: Fornwalt in view of Mahmood makes obvious the above limitations. The limitation wherein the regression function includes one or more of: i) a mean squared error loss function; ii) a mean squared logarithmic error loss function; iii) a mean absolute error loss function; or iv) a Huber loss function further limits a limitation claimed in the alternative. As Fornwalt in view of Mahmood teaches an alternative to the further limited option, Fornwalt in view of Mahmood continues to make obvious the claimed invention.
Regarding Claim 12: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the binary classification function is a sigmoid activation function included in one or more neurons of a dense output layer (The tabular layer consists of a fully connected layer (Dense) with sigmoid activations and a Drop Out layer. The input video dimensions were 150?109?60 pixels, and the output dimension of every layer are shown. See at least [0051] and Fig. 1).
Regarding Claim 13: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the binary classification function includes one or more of: i) a binary cross entropy loss function; ii) a Kullback-Leibler divergence loss function; iii) a Hinge function loss function; iv) a squared Hinge loss function; or v) a focal loss function (As the DNN was trained, the loss (binary cross-entropy) on the validation set was evaluated at each epoch. See at least [0091]).
Regarding Claim 14: Fornwalt in view of Mahmood makes obvious the above limitations. The limitation wherein the multi-class classification function is a softmax activation function included in one or more neurons of a dense output layer further limits a limitation claimed in the alternative. As Fornwalt in view of Mahmood teaches an alternative to the further limited option, Fornwalt in view of Mahmood continues to make obvious the claimed invention.
Regarding Claim 15: Fornwalt in view of Mahmood makes obvious the above limitations. The limitation wherein the multi-class classification function includes one or more of: i) a categorical cross entropy loss function; ii) Kullback-Leibler divergence loss function; iii) a Sparse multiclass cross-entropy loss function; iv) a focal loss function; or v) a negative log-likelihood function further limits a limitation claimed in the alternative. As Fornwalt in view of Mahmood teaches an alternative to the further limited option, Fornwalt in view of Mahmood continues to make obvious the claimed invention.
Regarding Claim 16: Fornwalt in view of Mahmood makes obvious the above limitations. The limitation wherein the survival modeling function is a linear activation function included in one or more neurons of a dense output layer further limits a limitation claimed in the alternative. As Fornwalt in view of Mahmood teaches an alternative to the further limited option, Fornwalt in view of Mahmood continues to make obvious the claimed invention.
Regarding Claim 17: Fornwalt in view of Mahmood makes obvious the above limitations. The limitation wherein the survival modeling function includes a Cox-proportional hazard loss function further limits a limitation claimed in the alternative. As Fornwalt in view of Mahmood teaches an alternative to the further limited option, Fornwalt in view of Mahmood continues to make obvious the claimed invention.
Regarding Claim 18: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the respective plurality of isolated processing layers include one or more binary classification layers; and wherein the one or more final output layers include a binary classification output layer (The cardiologists' responses were binary, and the Machine's response was continuous. 0.5 was set as the threshold for the Machine's response prior to performing the final comparison experiment. See at least [0097]).
Regarding Claim 24: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the respective plurality of isolated processing layers include one or more survival modeling layers and the one or more final output layers; and wherein the one or more combination layers include a survival output layer (The process 100 predicts a risk score for the patient based on a neural network. See at least [0098] and Fig. 1. Also: As the DNN was trained, the loss (binary cross-entropy) on the validation set was evaluated at each epoch. See at least [0091]. Also: See at least [0106]-[0111]. Examiner’s note: The specification does not define the terms “survival modeling layers”, “survival loss function”, or “survival output layer” and these terms are not terms of art. Therefore the broadest reasonable interpretation of these limitations include any respective modeling layer, loss function, and output layer which processes data relating to patient survival).
Regarding Claim 33: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein receiving, via the one or more processors, the predefined configuration parameters includes receiving the predefined configuration parameters in a structured configuration file (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051] and Fig. 1)
Claims 4, 19, 20, 23, and 25-30 are rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt et al. (US 2021/0145404 A1) in view of Mahmood et al. (US 2022/0367053 A1), and further in view of Vaid et al. (US 2023/0309967 A1).
Regarding Claim 4: Fornwalt in view of Mahmood makes obvious the above limitations. Fornwalt does not appear to disclose wherein at least one of the plurality of dissimilar input feature modalities includes electrocardiogram time series data; and wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs includes preprocessing at least one of the plurality of dissimilar input feature modalities via: i) an electrocardiogram lead recalibration; ii) an electrocardiogram lead amplitude standardization; iii) an electrocardiogram lead augmentation; iv) an electrocardiogram data restructuring; v) an electrocardiogram lead rescaling; or vi) a Fourier space transformation
However, Vaid teaches wherein at least one of the plurality of dissimilar input feature modalities includes electrocardiogram time series data (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]). Vaid further teaches wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs includes preprocessing at least one of the plurality of dissimilar input feature modalities via: i) an electrocardiogram lead recalibration; ii) an electrocardiogram lead amplitude standardization; iii) an electrocardiogram lead augmentation; iv) an electrocardiogram data restructuring; v) an electrocardiogram lead rescaling; or vi) a Fourier space transformation (Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. Each vector may correspond to a lead, with each XML file containing data for leads I, II, and V.sub.1-V.sub.6. The length of these vectors can vary. For example, these vectors can extend to five seconds (2,500 samples) or ten seconds (5,000 samples) of recorded information for each lead in addition to longer rhythm strip recordings. To avoid potential artifacts causes by extending 2,500 samples to 5,000 samples, the diagnostic platform may restrict each sample to only the first five seconds of its recording. Furthermore, the ECG data may not include—or the diagnostic platform may simply not consider—data for leads III, aVF, aVL, or aVR. These leads may be considered to have no additional information as the data can be derived from linear transformations of the vectors representing the other leads. As such, these leads may not be included in the model developed by the diagnostic platform. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 19: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the plurality of dissimilar input feature modalities include electronic health record features (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]); and wherein the combination output is a probability from 0 and 1 representing a future risk to a patient (At 108, the process 100 can receive a risk score from the trained neural network. The risk score can be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the mortality risk score can be an all-cause mortality risk score. See at least [0103]). Fornwalt does not appear to disclose wherein the plurality of dissimilar input feature modalities include electrocardiogram features.
However, Vaid teaches wherein the dissimilar input feature modalities include electrocardiogram (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 20: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the plurality of dissimilar input feature modalities include echocardiogram features and electronic health record features (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]); and wherein the combination output is a probability from 0 and 1 representing a future risk to a patient (At 108, the process 100 can receive a risk score from the trained neural network. The risk score can be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the mortality risk score can be an all-cause mortality risk score. See at least [0103]). Fornwalt does not appear to disclose wherein the dissimilar input feature modalities include electrocardiogram features.
However, Vaid teaches wherein the dissimilar input feature modalities include electrocardiogram features (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 23: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the plurality of dissimilar input feature modalities include electronic healthcare record features (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]); and wherein the combination output is a numerical value indicating a physiological aspect of a patient (At 108, the process 100 can receive a risk score from the trained neural network. The risk score can be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the mortality risk score can be an all-cause mortality risk score. See at least [0103]). Fornwalt does not appear to disclose wherein the dissimilar input feature modalities include electrocardiogram features.
However, Vaid teaches wherein the dissimilar input feature modalities include electrocardiogram features (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 25: Fornwalt in view of Mahmood makes obvious the above limitations. As previously noted, Fornwalt discloses wherein the plurality of dissimilar input feature modalities include electronic healthcare record features (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]); and wherein the combination output is a numerical value (At 108, the process 100 can receive a risk score from the trained neural network. The risk score can be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the mortality risk score can be an all-cause mortality risk score. See at least [0103]). Fornwalt does not appear to disclose wherein the dissimilar input feature modalities include electrocardiogram features.
However, Vaid teaches wherein the dissimilar input feature modalities include electrocardiogram features (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 26: Fornwalt in view of Mahmood makes obvious the above limitations. As previously noted, Fornwalt discloses wherein the plurality of dissimilar input feature modalities include genomic data features and electronic healthcare record features (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]. Also: The electronic health record dataset can include values of a number of parameters including demographic parameters, vitals parameters… The demographic parameters can include age, sex, and smoking status. The vitals parameters can include height, weight, heart rate, diastolic blood pressure, and systolic blood pressure. See at least [0007]); and wherein the combination output is a numerical value (At 108, the process 100 can receive a risk score from the trained neural network. The risk score can be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the mortality risk score can be an all-cause mortality risk score. See at least [0103]). Fornwalt does not appear to disclose wherein the dissimilar input feature modalities include electrocardiogram features.
However, Vaid teaches wherein the dissimilar input feature modalities include electrocardiogram features (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 27: Fornwalt in view of Mahmood makes obvious the above limitations. Fornwalt does not expressly disclose wherein the respective plurality of isolated processing layers include one or more multiclass classification layers and the one or more final output layers; and wherein the one or more combination layers include a multiclass classification output layer. However, Vaid teaches processing layers include one or more multiclass classification layers; and wherein the one or more combination layers include a multiclass classification output layer (In other embodiments, the model is a multiclass classification model that is trained to output more detailed predictions. In contrast to the aforementioned binary classification model, the multiclass classification model may be trained to distinguish between different states or severities of right ventricular dysfunction. Thus, the multiclass classification model may learn to distinguish between mild, moderate, and severe right ventricular dysfunction. Additionally or alternatively, the multiclass classification model may learn to distinguish between different forms of right ventricular dysfunction. As an example, with sufficient training data, the multiclass classification model may learn to distinguish between indicators of RVSD and indicators of RVD. While the multiclass classification model could serve as a screening tool like the binary classification model, it could also serve as a treating tool. The predictions output by the multiclass classification tool could be used to identify appropriate “next steps.” Those “next steps” could involve enrolling in a treatment program, scheduling examination by appropriate healthcare professional, etc. See at least [0066]).
Fornwalt and Mahmood suggests a system which takes patient health data and generates patient risk information, upon which the claimed invention’s use of multiclassification techniques can be seen as an improvement. However, Vaid demonstrates that the prior art already knew of multiclassification analysis in neural networks. One of ordinary skill in the art could have easily applied the techniques of Vaid to the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such an application of Vaid would’ve resulted in an improved system which could provide risk information for different conditions. As such, the application of Vaid and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 28: Fornwalt in view of Mahmood and Vaid makes obvious the above limitations. Additionally, Fornwalt discloses wherein the plurality of dissimilar input feature modalities include electronic healthcare record features (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]); and wherein the combination output is a respective probability from 0 to 1 corresponding to a plurality of heart failure classifications (At 108, the process 100 can receive a risk score from the trained neural network. The risk score can be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the mortality risk score can be an all-cause mortality risk score. See at least [0103]). Fornwalt does not appear to disclose wherein the dissimilar input feature modalities include electrocardiogram features.
However, Vaid teaches wherein the dissimilar input feature modalities include electrocardiogram features (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 29: Fornwalt in view of Mahmood and Vaid makes obvious the above limitations. Additionally, Fornwalt discloses wherein the plurality of dissimilar input feature modalities includes a structured data set (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]); wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs includes preprocessing the structured data using at least one of (i) categorical feature processing, (ii) numerical feature processing, or (iii) feature censoring (manually defined by a cardiologist), which were presumed to reflect input errors, and set as missing. Eight categorical variables were identified in the echocardiography measurements that each reported five valvular regurgitation and stenosis severity levels (including not assessed) and converted them to forty one-hot encoded binary variables. An ordinal variable reporting diastolic function was also identified and coded it as 1 for normal, 0 for dysfunction (but no grade reported), and 1, 2 and 3 for diastolic dysfunction grades I, II, and III, respectively. For non-echocardiography-derived measurements, such as LDL, HDL, blood pressure, heart rate (if not taken at the study), weight, and height measurements, the most recent past measurement was retrieved, within a 1-year window, relative to the echocardiogram acquisition date. See at least [0108]). Fornwalt does not appear to disclose wherein the plurality of dissimilar input feature modalities includes a time series data set or wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs includes preprocessing the time-series data using at least one of (i) electrocardiogram lead recalibration, (ii) electrocardiogram lead amplitude standardization, (iii) electrocardiogram lead augmentation, (iv) electrocardiogram data restructuring, (v) electrocardiogram lead rescaling, or (vi) Fourier transformation.
Vaid teaches wherein the plurality of dissimilar input feature modalities includes a time series data set (ECG waveform data (or simply “ECG data”). See at least [0039]. Also: Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. See at least [0092]) and wherein preprocessing the plurality of dissimilar input feature modalities using the one or more respective preprocessing layers to generate the respective preprocessing outputs includes preprocessing the time-series data using at least one of (i) electrocardiogram lead recalibration, (ii) electrocardiogram lead amplitude standardization, (iii) electrocardiogram lead augmentation, (iv) electrocardiogram data restructuring, (v) electrocardiogram lead rescaling, or (vi) Fourier transformation (Waveform data within XML files is normally formatted as one-dimensional collections (also called “vectors”) of integers samples at a rate of 500 hertz (“Hz”), for example. Each vector may correspond to a lead, with each XML file containing data for leads I, II, and V.sub.1-V.sub.6. The length of these vectors can vary. For example, these vectors can extend to five seconds (2,500 samples) or ten seconds (5,000 samples) of recorded information for each lead in addition to longer rhythm strip recordings. To avoid potential artifacts causes by extending 2,500 samples to 5,000 samples, the diagnostic platform may restrict each sample to only the first five seconds of its recording. Furthermore, the ECG data may not include—or the diagnostic platform may simply not consider—data for leads III, aVF, aVL, or aVR. These leads may be considered to have no additional information as the data can be derived from linear transformations of the vectors representing the other leads. As such, these leads may not be included in the model developed by the diagnostic platform. See at least [0092]).
Fornwalt and Mahmood suggest a multi-modal neural network system that uses separate subnetworks to analyze diagnostic information to produce a clinical result, which differs from the claimed invention by the substitution of one of Fornwalt and Mahmood’s modalities for a ECG modality and associated processing. Vaid demonstrates that the prior art already knew of using ECG modality and associated processing in a multi-modal neural network system to produce clinical results. One of ordinary skill in the art could have trivially substituted Vaid’s modality into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would use ECG data to generate clinical result information. As such, the identified substitution, and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Vaid.
Regarding Claim 30: Fornwalt in view of Mahmood and Vaid makes obvious the above limitations. Additionally, Fornwalt discloses wherein the structured data is a tabular data set (The convolutional layer (Cony) is shown in the top box with a solid outline and the tabular EHR data layer (Tab) is shown in the bottom box with a dashed outline. See at least [0051]).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt et al. (US 2021/0145404 A1) in view of Mahmood et al. (US 2022/0367053 A1), and further in view of Arienzo et al. (US 2020/0193592 A1).
Regarding Claim 21: Fornwalt in view of Mahmood makes obvious the above limitations. As previously noted, Fornwalt discloses wherein the plurality of dissimilar input feature modalities include electronic healthcare record features (FIG. 1 is a neural network architecture 10 for mortality prediction from echocardiography videos and electronic health record (EHR) data. See at least [0051]); and wherein the combination output is a probability from 0 and 1 representing a patient’s future risk (At 108, the process 100 can receive a risk score from the trained neural network. The risk score can be associated with a risk of a clinical outcome for the patient. In some embodiments, the risk score can be a mortality risk score. In some embodiments, the mortality risk score can be an all-cause mortality risk score. See at least [0103]).
Fornwalt does not explicitly disclose a modality of retinal scan image features or where the risk is for onset diabetes.
However, Arienzo teaches using a neural network to analyze retinal scan image features to predict a risk for onset diabetes (The inventors have developed techniques for using a captured image of a person's retina fundus to determine the person's predisposition to certain diseases. For example, the appearance the person's retina fundus may indicate whether the person is at risk for various conditions such as diabetes. See at least [0169]. Also: FIGS. 5A and 5B are block diagrams of portions 500a and 500b forming an exemplary convolutional neural network (CNN) configured to extract data from a captured image of a person's retina fundus. In the illustrative embodiment of FIGS. 5A and 5B, portion 500a may be operatively coupled to portion 500b, such as with an output of portion 500a coupled to an input of portion 500b.S See at least [0096]).
Fornwalt and Mahmood suggests a system which takes patient health data and generates patient risk information, which differs from the claim invention by the substitution of Fornwalt’s echocardiogram data for retinal scan data, and Fornwalt’s all mortality risk for a risk of diabetes. However, Arienzo demonstrates that the prior art already knew of using a neural network to evaluate retinal scan data to determine risk of diabetes. One of ordinary skill in the art could have trivially substituted these elements into the system of Fornwalt and Mahmood, and further one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in an improved system which would use retinal image data and electronic health record data to generate diabetes risk outlooks. As such, the identified substitution and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Arienzo.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt et al. (US 2021/0145404 A1) in view of Mahmood et al. (US 2022/0367053 A1), and further in view of Semturs et al. (US 2020/0311933 A1).
Regarding Claim 22: Fornwalt in view of Mahmood makes obvious the above limitations. Additionally, Fornwalt discloses wherein the respective plurality of isolated processing layers include one or more binary classification layers and the one or more final output layers (The process 100 predicts a risk score for the patient based on a neural network. See at least [0098] and Fig. 1. Also: As the DNN was trained, the loss (binary cross-entropy) on the validation set was evaluated at each epoch. See at least [0091]. Also: See at least [0106]-[0111]).
However, Fornwalt does not appear to disclose wherein the one or more combination layers include a regression output layer.
Semturs teaches wherein the layers include a regression output layer (The system 100 then processes the convolutional output through one or more fully-connected layers to generate the model output for the neural network. For example, when the model output is a regression output the output layer of the neural network can be a linear layer that directly generates the regressed value of the property of the blood. See at least [0030]. Also: When the model generates regression outputs, the system can train the model to minimize a regression loss function, such as mean squared error loss function, a mean absolute error loss function, a Huber Loss function or the like. See at least [0050]).
Fornwalt and Mahmood suggests a system which takes patient health data and generates patient risk information, upon which the claimed invention’s use of regression analysis to generate the output can be seen as an improvement. However, Semturs demonstrates that the prior art already knew of incorporating regression analysis into a neural network to generate output predictions. One of ordinary skill in the art could have easily applied the techniques of Semturs to the system of Fornwalt and Mahmood to generate the outputs of Fornwalt via regression. Further, one of ordinary skill in the art would have recognized that such an application of Semturs would have resulted in an improved system which would be robust against extreme values in the data. As such, the application of Semturs and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Semturs.
Claim 36 is rejected under 35 U.S.C. 103 as being unpatentable over Fornwalt et al. (US 2021/0145404 A1) in view of Mahmood et al. (US 2022/0367053 A1), and further in view of Surendran et al. (US 2022/0343137 A1)
Regarding Claim 36: Fornwalt in view of Mahmood makes obvious the above limitations. Fornwalt does not expressly disclose wherein reshaping the respective segments outputs comprises padding outputs having a lower dimension up to a highest dimension of the respective segment outputs using zero-padding. However, Surendran teaches wherein reshaping the respective segments outputs comprises padding outputs having a lower dimension up to a highest dimension of the respective segment outputs using zero-padding (a processing logic may determine whether tensor parameters of said two sum reduction operations have a same dimension. If said two tensors have different dimensions, a processing logic may zero pad a tensor of said two tensors that has a smaller dimension, to match a shape of larger tensor. In at least one embodiment, zero padding a tensor may refer to a process of increasing a dimension of said tensor to reach a given dimension by adding additional elements having a value of zero to a copy of said tensor. See at least [0084]).
Fornwalt and Mahmood suggests a multi-modal machine learning system which reshapes intermediate processing results to generate patient risk information, which differs from the shaped invention by the substitution of Fornwalt’s dimensional reduction reshaping for a dimensional increase reshaping. However, Surendran demonstrates that the prior art already knew of increasing a tensor’s dimension with zero padding. One of ordinary skill in the art could have trivially substituted Surendran’s reshaping technique into the system of Fornwalt and Mahmood. Further, one of ordinary skill in the art would have recognized that such a substitution would have predictably resulted in a system which would operate its analysis in the dimensional space of the largest intermediate processing result. As such, the identified substitution and the claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention in view of the disclosures of Fornwalt and the teachings of Mahmood and Surendran.
Response to Arguments
Applicant’s Argument Regarding 112(a) and 112(b) Rejections of claims 1-33: The amended claims no longer recite “tuning” by “pruning or trimming,” nor do they recite outputs that are “compatible and combinable” prior to reshaping.
Examiner’s Response: Applicant's amendments filed 22 June 2026 have been fully considered and they resolve the identified rejections. The prior rejections under 112(a) and 112(b) are withdrawn.
Applicant’s Argument Regarding 101 Rejections of claims 1-33:
The as-filed specification identified a specific technical problem with conventional multimodal machine learning approaches. Conventional techniques require training and operating separate models for each data modality … Furthermore, “a single loss function may not effectively be used for all features in a traditional sense … The specification discloses a specific technical solution to these problems: a single-architecture deep learning model that processes dissimilar modalities through isolated segments and combines their outputs.
The combination layer processing recited in the amended claims imposes a meaningful limitation on any alleged abstract idea. The reshaping of outputs having different dimensional profiles to a common dimension, combined with weight adjustment, describes a specific technical mechanism for combining fundamentally incompatible data types within a single architecture. … This is not merely “apply it on a generic computer”; rather it is a particular way of solving the dimensional incompatibility problem that enables the architectural improvement described throughout the specification.
The precedential Desjardins decision supports the eligibility of the claims as amended. In particular, the as-filed specification discloses that the claimed architecture eliminates the need for 10 or more separate ensemble models, enables a single loss function across all modalities, and maps non-linear relationships across modalities resulting in significant improvements in model performance.
The reshaping limitation is not extra-solution activity; it is central to the claimed improvement. Without reshaping outputs of different dimensional profiles to a common dimension, the single architecture combination of dissimilar modalities would not be possible (see paragraph [0063] of the as-filed specification).
Applicant further notes that new dependent claim 36 recites that the shaping comprises “padding outputs having a lower dimension up to a highest dimension of the respective segment outputs using zero-padding,” providing additional specificity to the combination layer processing.
Nguyen establishes only that generic tensor reshaping functionality exists in neural network libraries. It does not establish that the specified ordered combination recited in the amended claims was well-understood, routine, or conventional.
Examiner’s Response: Applicant's arguments filed 22 June 2026 have been fully considered but they are not persuasive.
One of ordinary skill in the art would simply not recognize applicant as having solved the articulated problem. Applicant’s asserted solution amounts to instructions to not have the problem. For example, note the disclosure’s statement that “a single loss function may not effectively be used for all features.” The disclosure fails to provide any technical technique for using a single loss function to train models with multiple features. Per MPEP 2106.05(a), “If it is asserted that the invention improves upon conventional functioning of a computer, or upon conventional technology or technological processes, a technical explanation as to how to implement the invention should be present in the specification.” Here, one of ordinary skill in the art would reject the notion that Applicant’s claim constitutes a technical improvement.
The reshaping does not constitute a “particular way of solving the dimensional incompatibility problem.” It is a high-level description of a basic tensor operation. The 101 rejection provides as part of the Step 2B analysis, evidence which states that “Tensor reshaping is a functionality that is available in typical neural network libraries as it is needed to modify C×H×W tensor dimensions.” Again, one of ordinary skill in the art would reject the notion that this constitutes a technical improvement.
Applicant’s argument here appears largely untethered to the actual claim language. And again, it appears unfounded to assert that the disclosure “enables a single loss function” when it does not describe how to actually overcome that problem. Finally, the claims are not analogous to those in Desjardins as one of ordinary skill in the art would not regard there to be any improvement to machine learning present.
The as-filed disclosure at [0063] does not appear to provide any support for Applicant’s assertion that “[w]ithout reshaping outputs of different dimensional profiles to a common dimension, the single architecture combination of dissimilar modalities would not be possible.”
Examiner notes that Claim 36 stands as not rejected under 101. This feature is considered am additional element which poses a meaningful limitation on the implementation of the abstract idea, and as such, integrates the abstract idea into a practical application. Thus claim 36 is eligible at Step 2A, Prong 2.
Examiner notes that neither the Mayo/Alice test nor the Berkheimer memo requires that the totality of the claims be demonstrated to be conventional. The Nguyen reference establishes that reshaping vectors is a conventional neural network operation. With that element no longer plausibly a technical improvement, Applicant relies on the sum of the architecture to be the improvement. But again, there is no technical explanation of how to implement that architecture beyond a rough sketch of its arrangement. Given the lack of technical details of implementation, one of ordinary skill in the art would not consider it to be a technical improvement.
Additional Considerations
The prior art made of record and not relied upon that is considered pertinent to applicant’s disclosure can be found in the PTO-892 Notice of References Cited.
Dubovsky et al. (US 2020/0074247 A1) describes jointly training multiple subnetworks of a neural network architecture with a single loss function ([0006]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Bion A Shelden whose telephone number is (571)270-0515. The examiner can normally be reached M-F, 12pm-10pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kambiz Abdi can be reached at (571) 272-6702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Bion A Shelden/Primary Examiner, Art Unit 3685 2026-08-21
Prosecution Insights