Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed April 14, 2026. The status of the claims are as follows:
Claims 1-20 are pending;
Claims 1, 11, and 20 have been amended;
In view of Applicant’s amendment and remarks filed April 14, 2026, the rejection of claims 1-20 under 35 U.S.C. § 101 is withdrawn.
Claims 1-20 are rejected under 35 U.S.C. § 103 for the reasons set forth in this Office action.
The amendment to independent claims 1, 11, and 20 introduced limitations directed to jointly training the recited neural network to obtain a first set of parameters, receiving user-specific training data, and subsequently fine-tuning the prediction neural network beginning from the first set of parameters using the user-specific training data. The amendment therefore necessitated the additional prior-art teachings and new ground of rejection presented in this Office action.
Response to Amendments
The Office action is in response to Applicant’s communication filed April 14, 2026 in response to the Office action mailed January 14, 2026. Applicant’s remarks and amendments to the claims have been considered with the results set forth below.
Response to Arguments
Regarding prior rejection under 35 U.S.C. § 101
Applicant argues that claims 1-20 do not recite a judicial exception, or alternatively, that the claims integrate the recited mathematical concepts into a practical application.
Applicant particularly relies on the amended limitations requiring jointly training a convolutional neural network, an aggregation neural network, and an output neural network to produce a prediction neural network having a first set of parameters, followed by fine-tuning the prediction neural network using user-specific training data. Applicant further argues that the claimed combination provides a particular technical implementation for processing multimodal temporal data and repeatedly generating updated predictions as the applicable time window changes.
Based on the amended claims considered as a whole and in view of the particular ordered combination recited, the Examiner finds these arguments persuasive.
Accordingly, the rejection of claims 1-20 under 35 U.S.C. § 101 is withdrawn.
Regarding prior rejection under 35 U.S.C. § 103
Applicant argues that Cheung, Dang, and Tan do not teach or suggest the limitations newly added to independent claims 1, 11, and 20, including:
“jointly training (1) a convolutional neural network comprising a stack of convolutional layers and a normalization layer following each convolutional layer except for a final convolutional layer, (2) an aggregation neural network, and (3) an output neural network to produce a prediction neural network with a first set of parameters, wherein the prediction neural network is trained to make a prediction of a time-changing entity;”
“receiving, from a user, user-specific training data;“ and
“performing fine tuning of the prediction network by training the prediction
network with the first set of parameters on the user-specific training data;”
Applicant further argues that Tan does not teach or suggest:
“a normalization layer following each convolutional layer except a final convolutional layer”
Applicant’s arguments are persuasive to the extent that the previously applied combination of Cheung, Dang, and Tan did not expressly address the complete sequence newly added to claims 1, 11, and 20.
Regarding Applicant’s argument concerning Joint Training:
Applicant’s argument is not persuasive to the extent Applicant contends that the references fail to teach joint training generally.
Dang expressly teaches network parameters that are “jointly trained with the entire model”. Cheung also teaches supervised training of a predictive neural-network model using training examples and corresponding outcomes. Thus, Cheung in view of Dang teaches or suggests jointly or end-to-end training components of a prediction model.
However, Applicant is correct that the former rejection did not fully address the complete amended sequence comprising both an initial joint-training stage and a subsequent user-specific fine-tuning stage.
Regarding Applicant’s argument concerning User-Specific Fine-Tuning:
Applicant argues that Cheung’s disclosure of using a validation set to “fine tune” hyperparameters does not teach fine-tuning an already-trained prediction neural network using user-specific training data.
This argument is persuasive with respect to Cheung considered alone.
Cheung’s cited disclosure concerns selection or adjustment of model hyperparameters, including the number of layers, the number of neurons, the number of convolutional filters, and dropout probability. The disclosure does not itself teach subsequently training the already-trained prediction neural network on data specific to a user while beginning from the parameter values obtained during the initial training stage.
Dang’s disclosure of jointly training parameters with the entire model also does not, by itself, teach the subsequently performed user-specific personalization now recited.
The former rejection therefore did not establish that Cheung, Dang, and Tan teach the complete amended fine-tuning limitation.
Applicant’s argument does not, however, establish patentability over the prior art as a whole. The new rejection below applies additional prior art that teaches personalizing or further training an existing machine-learning model using user-specific training data. The additional reference is relied upon for the portion of the amended limitation not taught by Cheung and Dang.
When the teachings are combined as set forth in the new rejection, the resulting method begins with an already-trained prediction model having trained parameter values and subsequently trains that existing model using user-specific data. The parameters present in the existing model at the beginning of the subsequent training correspond to the claimed “first set of parameters”.
Accordingly, Applicant’s argument is persuasive to a deficiency in the former rejection but is not persuasive against the new combination presented in this Office action.
Regarding Applicant’s argument concerning Normalization Following Each Convolutional Layer Except the Final Convolutional Layer:
Applicant argues that Tan reaches batch normalization following the convolutional layers disclosed within Tan’s residual blocks but does not teach omitting normalization after the final convolutional layer.
This argument is persuasive with respect to the rejection as previously stated.
Tan teaches normalization following the convolutional layers within its disclosed residual-block structures. Tan does not expressly teach the claimed arrangement in which normalization follows the intermediate convolutional layers but does not follow the final convolutional layer of the convolutional-neural-network stack.
The prior Office action further stated that leaving the final convolution unnormalized would have been a routine design choice. Upon reconsideration, the prior rationale did not sufficiently establish that normalized and unnormalized final convolutional layers were known interchangeable alternatives that perform the same function or produced the same result.
Accordingly, the prior design-choice rationale is withdrawn.
Applicant’s argument nevertheless does not establish patentability over the prior art as a whole. The new rejection below applies additional prior art as to the deficiency in the former Cheung-Dang-Tan rejection but is not persuasive against the new combination presented in this Office action.
Regarding Applicant’s argument concerning Tan’s Final Layer:
Applicant additionally asserts that Tan affirmatively teaches a normalization layer following the “final convolution layer”.
This argument is not fully persuasive.
The cited portions of Tan show normalization following the last convolutional layer within particular residual blocks. Applicant has not established that the last convolutional layer illustrated within each residual block is necessarily the final convolutional layer of Tan’s complete neural-network architecture.
Nevertheless, because Tan does not expressly disclose the claimed exception for the final convolutional layer, Tan is not relied upon in the present rejection as teaching that exception.
Regarding Applicant’s argument concerning Desing-Choice:
Applicant argues that removing a normalization layer is not a mere rearrangement of known components and may affect network operation.
This argument is persuasive with respect to the previous design-choice rationale.
The new rejection does not rely on a conclusion that omission of the final normalization layer is merely a matter of design choice. Instead, the new rejection relies on an express prior-art teaching of a convolutional architecture in which normalization is applied to the intermediate convolutional layers but omitted after the final output-generating convolutional layer.
Accordingly, Applicant’s argument concerning the inadequacy of the former design-choice rationale do not overcome the new rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Patrick Cheung (US20210056413A1) in view of Xuan-Hong Dang (US11915123B2) in further view of Min Wu (US20210315470A1) and further in view of Mark Sandler (US20200104706A1).
Regarding claim 1, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach a system comprising: one or more computers, and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
“jointly training (1) a convolutional neural network comprising a stack of convolutional layers and (3) an output neural network to produce a prediction neural network with a first set of parameters, wherein the prediction neural network is trained to make a prediction of a time-changing entity;” – Cheung teaches this limitation in part. Cheung teaches a cross-modal prediction neural network that includes convolutional feature-processing components and a final fully connected output network:
“Scalars from
Ν
x
c
o
n
v
filters are then concatenated to form a compact predictive feature vector which is then fed to a final connected network at stage S162 followed by a sigmoid function at stage S163 to produce prediction 117a.” (Cheung, p. 7, ¶[0096])
Cheung further teaches jointly training the components of the prediction model by adjusting the model weights using a loss function:
“The AXCNN was implemented using a deep learning library utilizing an Adadelta optimizer with the default parameter values and batch size of 256 for training the model. Binary cross-entropy was used as the loss function to adjust the weights. Training was stopped when no further improvement on the validation loss is found after 25 epochs.” (Cheung, p. 8, ¶[0102])
Cheung also teaches that the resulting prediction network is trained to predict a future, time-changing condition of a patient:
“AXCNN was used to predict if the HF patient would be readmitted within the next 30 days.” (Cheung, p. 7, ¶[0098])
Thus, Cheung teaches jointly training convolutional and output neural-network components to produce a prediction neural network having a first set of learned parameters and trained to predict a time-changing entity, namely, the future readmission status of a patient.
“receiving, by a hardware accelerator executing the prediction network, an input sequence of multi-modal feature vectors characterizing the entity over a time window, wherein each multi-modal feature vector in the input sequence corresponds to a different time interval during the time window;” – Cheung teaches this limitation. Cheung teaches execution using processor hardware that may include an FPGA or ASIC:
“The processor 91 may be any hardware device capable of executing instructions stored in memory or storage or otherwise processing data. As such, the processor 91 may include a microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other similar devices.” (Cheung, p. 8, ¶[0108])
Cheung further teaches receiving multimodal temporal data, including:
“Those resampled time series which have more than 100 points were truncated to keep the last 100 measurement points while others which have fewer
than 100 points were zero-padded to maintain 100 points in length.” (Cheung, p. 8, ¶[0099])
Cheung further teaches that the physiological measurements are resampled according to recurring time intervals:
“vital sign measurements and laboratory test results were resampled with backward-filled for every hour and every 4 hours respectively” (Cheung, p. 8, ¶[0099])
Thus, Cheung teaches receiving, by hardware capable of executing or accelerating the prediction network, a sequence of multimodal feature data characterizing the patient over a time window, wherein the feature data corresponds to respective time intervals.
“generating a prediction of the entity using the prediction network by: processing, by the hardware accelerator, the input sequence of multimodal feature vectors using the convolutional neural network to generate a latent sequence that comprises a plurality of latent feature vectors;” – Cheung teaches this limitation. Cheung teaches applying convolutional filters to input sequences to generate respective feature-map outputs:
“where
N
x
c
o
n
v
is the number of filters and f is a non-linear activation function. A max pooling operation is applied to each filter to extract a scalar y(j)=MAX(Y(:, j)).” (Cheung, p. 7, ¶[0096])
The convolutional feature maps generated before max pooling correspond to latent feature representations generated from the input sequence.
Cheung further teaches a stack of convolutional stages:
“a two-stage convolutional neural network 150b employs a convolutional module including two stacked convolutional neural network stages S151 and S152 (FIG. SA) applying convolution and max pooling” (Cheung, p. 6, ¶[0087])
Thus, Cheung teaches processing the input sequence using a convolutional neural network to generate latent feature representations.
“processing, by the hardware accelerator, the latent sequence of latent feature vectors using ” – Cheung teaches this limitation in part. Cheung teaches concatenating feature outputs to form a compact predictive feature vector:
“Scalars from
Ν
x
c
o
n
v
filters are then concatenated to form a compact predictive feature vector” (Cheung, p. 7, ¶[0096])
Thus, Cheung teaches aggregating latent feature outputs to generate an aggregated predictive feature vector.
“and processing, by the hardware accelerator, the aggregated feature vector using the output neural network to generate a prediction that characterizes the entity after the time window;” – Cheung teaches this limitation. Cheung teaches processing the aggregated predictive feature vector using a fully connected network and output function:
“Scalars from
Ν
x
c
o
n
v
filters are then concatenated to form a compact predictive feature vector which is then fed to a final connected network at stage S162 followed by a sigmoid function at stage S163 to produce prediction 117a.” (Cheung, p. 7, ¶[0096])
Cheung further teaches that the prediction characterizes a future condition occurring after the time window used to generate the input:
“AXCNN was used to predict if the HF patient would be readmitted within the next 30 days.” (Cheung, p. 7, ¶[0098])
Thus, Cheung teaches processing an aggregated feature vector using an output neural network to generate a prediction characterizing the entity after the observed time window.
“adjusting the time window to incorporate the new time interval by removing a multi-model feature vector corresponding to the oldest time interval in the input sequence ” – Cheung teaches this limitation in part. Cheung teaches maintaining a fixed-length window containing the most recent measurement points:
“Those resampled time series which have more than 100 points were truncated to keep the last 100 measurement points while others which have fewer than 100 points were zero-padded to maintain 100 points in length.” (Cheung, p. 8, ¶[0099])
Maintaining only the last 100 measurement points teaches excluding measurement points older than the bounded time window.
Cheung does not teach these limitations and/or portions of:
“and a normalization layer following each convolutional layer except a final convolutional layer, (2) an aggregation neural network,”
“receiving, from a user, user-specific training data;”
“performing fine tuning of the prediction network by training the prediction network with the first set of parameters on the user-specific training data;”
“the aggregation neural network”
“receiving, by the hardware accelerator, an additional multi-modal feature vector corresponding to a new time interval and in response:”
“and adding the additional multi-modal feature vector corresponding to the new time interval”
“and reprocessing, by the hardware accelerator executing the prediction network, the input sequence with the adjusted time window to generate an updated prediction.”
Dang, however, teaches these limitations and/or portions of:
“(2) an aggregation neural network” – Dang teaches an asynchronous recurrent aggregation neural network:
“Asynchronous RNN (AsyncLSTM) that iteratively fuses encoded features from multiple different data modalities,” (Dang, col. 4, lines 5-6)
Dang further teaches that the AsyncLSTM parameters are jointly trained with the entire model:
“are the network parameters jointly trained with the entire model.” (Dang, col. 6, lines 2-3)
Thus, Dang teaches an aggregation neural network whose parameters are jointly trained with the other components of the complete prediction model.
“the aggregation neural network” – Dang teaches using the AsyncLSTM to fuse encoded multimodal feature representations and aggregate the representations into a single vector:
“Encodings from the text data (114) and encodings from the time series data (132) are combined in a concatenation layer (150) to aggregate the encoding into a single vectors” (Dang, col. 5, lines 24-27)
Thus, Dang teaches processing latent multimodal feature vectors using an aggregation neural network to generate an aggregated feature vector.
“receiving, by the hardware accelerator, an additional multi-modal feature vector corresponding to a new time interval and in response:” – Dang teaches receiving and processing time-series data at successive time intervals:
“At each timestep I in the series sequence (220), AsyncLSTM searches through the representative text sequence” (Dang, col. 6, lines 6-8)
Dang further discloses:
“With respect to the time series data, an input sample (220) at each time point t is a sequence of m values” (Dang, col. 5, lines 54-56)
In view of Cheung’s fixed-length window containing the last 100 measurement points, Dang’s timestep-wise advancement teaches or suggests adding data for the newly encountered time interval while the oldest interval is removed to preserve the fixed window length.
“and reprocessing, by the hardware accelerator executing the prediction network, the input sequence with the adjusted time window to generate an updated prediction.” – Dang teaches that its aggregation neural network performs reasoning at every successive timestep:
“AsyncLSTM incorporates information learned in the textual domain to every step it performs reasoning on in the time series” (Dang, col. 6, lines 4-6)
Dang further teaches forwarding each updated state to the next cell in the AsyncLSTM and producing an output representing behavior of the time-series data for a defined period. Thus, Dang teaches repeatedly processing the temporally updated information as the sequence advances.
Dang does not teach these limitations and/or portions of:
“and a normalization layer following each convolutional layer except a final convolutional layer”
“receiving, from a user, user-specific training data;”
“performing fine tuning of the prediction network by training the prediction network with the first set of parameters on the user-specific training data;”
Wu, however, teaches these limitations and/or portions of:
“and a normalization layer following each convolutional layer except a final convolutional layer” – Wu teaches a convolutional neural-network architecture in which normalization is applied following the intermediate convolutional layers but is omitted following the final convolutional layer:
“layer normalization may be applied to all the convolutional and transposed-convolutional layers except the final ECG generation layer.” (Wu, pg. 10, ¶[0120])
Wu further describes the final ECG-generation layer as a convolutional layer at the end of the ECG-inference pipeline. Thus, Wu expressly teaches a stack of convolutional layers having normalization applied to the convolutional layers except the final convolutional layer.
Wu does not teach these limitations and/or portions of:
“receiving, from a user, user-specific training data;”
“performing fine tuning of the prediction network by training the prediction network with the first set of parameters on the user-specific training data;”
Sandler, however, teaches these remaining limitations and/or portions of:
“receiving, from a user, user-specific training data;” – Sandler teaches receiving training examples from a user computing device and training a model using data specific to that user:
“the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user specific data received from the user computing device 102.” (Sandler, pg. 8, ¶[0110])
Sandler characterizes this process as personalizing the model. Sandler also teaches locally personalizing the model:
“the user computing device 102 can implement the model trainer 160 to personalize the models 120 based on user-specific data.” (Sandler, pg. 8-9, ¶[0113])
Thus, Sandler teaches receiving, from a user, user-specific training data.
“performing fine tuning of the prediction network by training the prediction network with the first set of parameters on the user-specific training data;” – Sandler teaches obtaining a previously trained neural network having a first set of learned parameters:
“a computing system can obtain a machine-learned model ( e.g., neural network) that has been previously trained on a first training dataset to perform a first task.” (Sandler, pg. 3, ¶[0043])
Sandler further teaches:
“The machine-learned model can include a first set of learnable parameters ( e.g., weights or other parameters).” (Sandler, pg. 3, ¶[0043])
Sandler teaches fine-tuning the previously trained model while reusing its existing parameter values:
“a small patch can be used to fine-tune a model in order for the model to be repurposed to a different task” (Sandler, pg. 4, ¶[0058])
“re-training or re-purposing a machine-learned model to solve very different types of problems while reusing a significant portion (e.g., 95% or greater) of the existing parameter values.” (Sandler, pg. 4, ¶[0058])
Sandler further teaches that the already-provided model may be trained or personalized on user-specific data received from the user computing device. (Sandler ¶¶ [0110] and [0113])
Thus, Sandler teaches fine-tuning a previously trained neural network beginning from the reusing its existing first set of learned parameters, wherein the fine-tuning is performed using user-specific training data.
It would have been obvious to a person of ordinary skill in the art before the effective filling data to modify Cheung’s prediction neural network according to the teachings of Dang, Wu, and Sandler.
Dang teaches a jointly trained AsyncLSTM that iteratively fuses encoded features from multiple modalities and models temporal relationships among successive inputs. Incorporating Dang’s aggregation neural network into Cheung’s prediction network would have enabled Cheung’s convolutionally generated features to be temporally and multimodality aggregated before being supplied to the output network.
It further would have been obvious to maintain Cheung’s fixed-length window of the last 100 measurement points as the combined model advances through the successive timesteps taught by Dang by removing the oldest feature vector, adding the newly received feature vector, and reprocessing the adjusted sequence. This would have been predictably generated an updated prediction reflecting the most recent available data while preserving the fixed input length.
Wu teaches a trainable convolutional architecture in which normalization follows the intermediate convolutional layers but not the final output-generating convolutional layer. Appling Wu’s arrangement to the convolutional neural network of the Cheung-Dang combination would have predictably normalized intermediate feature representations while permitting the final convolutional layer to produce its output without having a subsequent normalization transformation.
Sandler teaches personalizing an already-trained machine-learned model using user-specific training examples while retaining and reusing the model’s existing learned parameters. Applying Sandler’s technique to the trained Cheung-Dang-Wu prediction network would have enabled the generally trained network to be adapted to a particular user without retraining from randomly initialized parameters.
Each modification applies a known neural-network technique according to its established function. Accordingly, the combination would have predictably produced a jointly trained multimodal temporal prediction network having the claimed normalization architecture, capable of user-specific fine-tuning and repeated execution on an advancing fixed-length time window. A person of ordinary skill would therefore have had a reasonable expectation of success.
Regarding claim 2, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 1,
wherein the stack convolutional layers that each have a respective one- dimensional kernel. – Cheung teaches this limitation. Cheung teaches a plurality/stack of convolutional layers and explicitly identifies kernel size for those layers, in the context of applying convolution to time series / sequences:
“a two-stage convolutional neural network 150b employs a convolutional module including two stacked convolutional neural network stages S151 and S152 (FIG. SA) applying convolution and max pooling” (Cheung, p. 6, ¶[0087])
“each time series is considered as a channel input … where Cl and C2 are the numbers of convolutional filters in stages S151 and S152, Size is the kernel size …” (Cheung, p. 6, ¶[0086])
Cheung also teaches using specific filter widths (i.e., kernel widths) for the CNN:
“The three filter widths were set to 3, 4 and 5 with 20 filters each for CNN.” (Cheung, p. 8, ¶[0101])
And Cheung explains that its convolution is performed on a sequence of inputs (i.e., 1-D along the sequence dimension):
“Convolution stage S143 is performed on the sequence of inputs …” (Cheung, p. 6, ¶[0081])
Thus, Cheung expressly teaches (i) multiple/stacked convolutional stages, and (ii) each such convolution uses a kernel size / filter width, applied to a sequence/time-series input.
Regarding claim 3, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 1,
wherein the aggregation neural network is a recurrent neural network. – Cheung does not teach this limitation. Cheung teaches an “aggregation” stage combining feature vectors from multiple networks, but does not specify that the aggregation network is an RNN. Dang, however, expressly teaches using a recurrent neural network (RNN) to analyze encoded multi-modal sets:
“leverage an artificial recurrent neural network (RNN) to analyze the encoded first and second data sets, including iteratively and asynchronously fuse the first and second encodings” (Dang, col. 18, lines 42-45)
This maps directly to the claim limitation because Dang’s RNN is used to fuse / combine (i.e., aggregate) the encodings/vectors from multiple modalities (first and second data sets).
It would have been obvious to a person having ordinary skill in the art to implement Cheung’s aggregation neural network as the RNN taught by Dang, because Dang expressly teaches RNN-based fusion of multimodal encodings/vectors and RNNs are a known design choice for modeling temporal dependencies and aggregating sequential/multimodal representations, yielding predictable results.
Regarding claim 4, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 1,
wherein the output neural network comprises one or more fully-connected layers followed by an output layer. – Cheung teaches this limitation. Cheung discloses an output/prediction network having a fully connected stage followed by a sigmoid function stage (i.e., an output layer):
“cross-modal convolutional neural network 160a has a neural architecture employing a module including … a fully connected stage S162 and a sigmoid function stage S163 … to produce a prediction 117a” (Cheung, p. 7, ¶[0094])
“Scalars from
Ν
x
c
o
n
v
filters are then concatenated to form a compact predictive feature vector which is then fed to a final connected network at stage S162 followed by a sigmoid function at stage S163 to produce prediction 117a.” (Cheung, p. 7, ¶[0096])
These passages teach that the output neural network includes at least one fully-connected layer followed by an output layer that produces the prediction.
Cheung expressly teaches the limitation of claim 4. Therefore, claim 4 is unpatentable for the reasons set forth for claim 1.
Regarding claim 5, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 1, the operations further comprising:
obtaining data characterizing the entity from a plurality of different data streams; - Cheung does not teach this limitation. Dang, however, teaches obtaining input data from multiple different streams/modalities. For example, Dang teaches different modalities are provided as different feeds/streams, including a textual stream and a time-series stream:
“the first data feed,
f
e
e
d
A
(452A), represents a time-stamped textual modality, … and the second data feed,
f
e
e
d
B
(452B), represents time series data modality.” (Dang, col. 8, lines 58-62)
“Input … includes time series data from the second input … and textual data from the first input” (Dang, col. 5, 51-53)
generating the multi-modal feature vectors in the input sequence by converting the data characterizing the entity into a standardized format. – Cheung does not teach this limitation. Dang, however, teaches converting heterogeneous, different-format multimodal inputs into vectors (i.e., encoding into a common vector format):
“the multi-modal data set comprising data in different formats from two or more modalities … encode the first data set into first encodings comprising one or more first vectors and encode the second data set into second encodings comprising one or more second vectors;” (Dang, col. 19, lines 16-18, 22-25)
“The first data feed manager functions to encode the first data set into a first set of vectors.” (Dang, col. 1, lines 34-35)
Thus, Dang expressly teaches the limitations of claim 5. Therefore, claim 5 is unpatentable for the reasons set forth for claim 1.
Regarding claim 6, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 5, wherein generating each multi-modal feature vector comprises:
identifying respective features of each of a plurality of feature types that characterize the entity during the corresponding time interval for the multi-modal feature vector; - Cheung does not teach this limitation. Dang, however, teaches identifying/deriving modality-specific (feature-type-specific) vectors on a per-time-interval basis. For example, Dang discloses that the model receives multiple modalities (text and time series), and that the textual items are collected/processed at time period t, i.e., during a corresponding time interval:
“with n being the total number of news stories collected at the time stamp t. Each vector is represented for a textual article, such as news story, or a sequence of related words.” (Dang, col. 5, lines 4-7)
“With respect to the time series data, an input sample (220) at each time point t is a sequence of m values …” (Dang, col. 5, lines 54-56)
Dang also makes clear these are different modalities being encoded/handled as vectors (i.e., distinct feature types being identified/extracted per interval):
“the AsynchLSTM receives two modalities of data, including … text data, and … numerical time series data…” (Dang, col. 5, lines 33-37)
These disclosures teach that, for a given time period/time point t (corresponding time interval), the system processes distinct feature types/modalities (text/time-series) into corresponding vectors (i.e., identifying respective features of each of the plurality of feature types for that interval).
for each feature type, adding the identified respective features of the feature type to one or more entries of the multi-modal feature vector that correspond to the feature type. – Cheung teaches this limitation. Cheung teaches producing and retaining separate feature vectors corresponding to different data types (feature types), and then combining them at an upper layer:
“at a lower neural network layer … input the encoded data … to produce an encoded feature vector, (2) input the embedded data … to output an embedded feature vector, and (3) input the sampled data into a sampled neural network to output an sampled feature vector, and at an upper neural network layer, ( 4) input at least two of the encoded feature vector, the embedded feature vector and the sampled feature vector into a convolutional neural network…” (Cheung, p.2, ¶[0009])
This disclosure supports the claimed concept of, for each feature type, placing the identified features into corresponding “entries” (i.e., structured components/slots) of a multimodal vector representation - here, the encoded feature vector / embedded feature vector / sampled feature vector corresponding to different feature types – prior to combination/fusion.
Given Dang’s per-time-interval modality-specific vector formation (text vectors at time stamp t; time-series sample at time point t) and Cheung’s structured per-data feature vector organization (encoded/embedded/sampled feature vectors), it would have been obvious to combine Dang’s modality/time-interval identification with Cheung’s per-feature-type vector assembly so that, for each time interval, the respective feature-type features are placed into corresponding entries/components of the multimodal feature vector to preserve semantic alignment for downstream fusion/prediction.
Regarding claim 7, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 1,
wherein . – Cheung teaches this limitation in part. Cheung expressly teaches training using many training inputs and corresponding ground truth labels. For example, Cheung’s explicit “training” disclosure:
“batch size of 256 for training the model. Binary cross-entropy was used as the loss function to adjust the weights. Training was stopped when no further improvement on the validation loss is found after 25 epochs.” (Cheung, p. 8, ¶[0102])
Cheung frames the prediction target as 30-day readmission:
“predict if the HF patient would be readmitted within the next 30 days.” (Cheung, p. 7, ¶[0098])
And gives readmissions occurring within 30 days after discharge as a counted outcome:
“The AXCNN was applied to a 30-day unplanned readmission data for heart failure (HF) collected from a large hospital system in Arizona, United States. The dataset consisted of patient encounter information for 6730 HF patients of age 18 or over (mean: 72.7, std:
14.4), 60% are males, between October 2015 and June 2017. Among them 853 patients have at least a readmission within 30 days after discharge” (Cheung, p. 7, ¶[0098])
Cheung does not teach this portion of the claim limitation:
“… the convolutional neural network, the aggregation neural network, and the output neural network have been jointly trained on … “
Dang, however, teaches this limitation:
“… the convolutional neural network, the aggregation neural network, and the output neural network have been jointly trained on … “ – Dang expressly discloses joint training of network parameters with the overall model:
“are the network parameters jointly trained with the entire model.” (Dang, col. 6, lines 2-3)
This provides direct support for the “jointly trained” concept (i.e., end-to-end training across model components).
Cheung teaches training a predictive neural network using a training set with corresponding ground truth labels/outcomes. Dang teaches that network parameters may be “jointly trained with the entire model”. Therefore, it would have been obvious to a POSITA to jointly train the convolutional neural network, aggregation neural network, and output neural network of the combined Cheung/Dang system using training input sequences and corresponding ground truth outcomes, as such end-to-end and supervised training of multi-component neural architectures was a known technique yielding predictable results (i.e., coordinated optimization of the feature extractor, fusion/aggregation, and prediction head).
Regarding claim 8, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 1,
wherein the entity is a financial asset – Cheung does not teach this limitation. Dang, however, teaches this limitation. Dang expressly applies its multimodal system to a stock / financial market use case:
“In the venue of financial market use case … adjusted close price of a stock.” (Dang, col. 9, lines 43-44)
“… a third data feed (506) is in time series data in the form of stock exchange data.” (Dang, col. 10, lines 12-13)
These disclosures teach that the entity being characterized/predicted is a financial asset (e.g., a stock) and that stock-market data (stock exchange data) is an input modality.
and wherein each multi-modal feature vector comprises technical analysis features and sentiment analysis features. – Cheung does not teach this limitation. Dang, however, teaches technical/price-derived features used to compute a market prediction, including explicit price-differential computations:
“A market predication is computed via a second order differential of an adjusted close price of a stock.” (Dang, col. 9, lines 45-46)
“… time series data directed to the price of the stock” (Dang, col. 9, lines 48-49)
“… stock exchange data.” (Dang, col. 10, line 13)
These teachings correspond to technical-analysis-type inputs/features derived from stock price time series (adjusted close price; price changes; stock exchange data).
In addition, Dang expressly defines and uses market sentiment (optimistic/pessimistic) in the financial embodiment:
“Market sentiment for day t+1 is optimistic … Market sentiment for t+1 is pessimistic …” (Dang, col. 9, lines 52-54)
Dang expressly teaches the limitations of claim 8. Therefore, claim 8 is unpatentable for the reasons set forth for claim 1.
Regarding claim 9, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 8,
wherein each multi-modal feature vector further comprises fundamental analysis features. – Cheung does not teach this limitation. Dang, however, teaches incorporating “quarterly reports” (a fundamental-information modality) into the multimodal vector representation and expressly discloses that quarterly reports are provided as an input modality in the financial-market embodiment:
“As shown, quarterly reports are represented in a first feed (502) and news articles are represented in a second feed (504). Both the quarterly reports and the news articles are textual data, and are received by textual module (512) …” (Dang, col. 9-10, lines 65-67, 1-2)
Dang further teaches that this textual input (including quarterly reports) is converted into a representation vector (i.e., a feature vector representation):
“The textual module (512) learns semantic dependencies among words present in the first feed (502) and the second feed (504) and aggregates them into a representative vector for each document …” (Dang, col. 10, lines 3-6)
Dang also teaches that outputs from the financial-market architecture fuse information associated with the financial time-series behaviors/performance with the news-related content:
“Output (574) … is directed at news filtering … in association with stock time-series behaviors and performance (572). More specifically, the news (570) and the performance behavior (572) are fused into the output (574).” (Dang, col. 10, lines 16-21)
It would have been obvious to a person of ordinary skill in the art before the effective filing data to include quarterly-report-derived features (fundamental information) in the multi-modal feature vectors of the Cheung-Dang-Wu-Sandler combination because Dang teaches predicting financial-asset behavior using multimodal financial information, and quarterly reports were a known source of information concerning an asset’s underlying financial condition. The modification would have predictably provided additional financial information for use in generating the prediction.
Regarding claim 10, Cheung in view of Dang in further view of Wu and further in view of Sandler, teach the system of claim 8,
wherein the prediction characterizes a predicted trading behavior of the financial asset at an end of a next time interval after the end of the time window. – Cheung does not teach this limitation. Dang, however, teaches predicting future (next-interval) behavior of a financial asset (e.g., stock market sentiment / behavior) and expressly teaches predicting next-day/next-interval market behavior (market sentiment) for a stock:
“Market sentiment for day t+1 is optimistic … Market sentiment for t+1 is pessimistic …” (Dang, col. 9, lines 52-54)
This passage teaches a prediction directed to a financial asset/market at a next time interval (t+1) following the current time t (i.e., after the time window ending at time t).
Dang also characterizes the output as a behavior label for the time series over a defined period of time:
“… produces output (170) in the form of a label of behavior of the time series data for a define period of time.” (Dang, col. 5, lines 28-30)
These teachings correspond to the “prediction characterizes a predicted training behavior … at the end of a next time interval” since Dang teaches predicting a labeled behavior outcome for time series data and explicitly provides a next-interval (t+1) sentiment prediction in a financial-market context.
Dang expressly teaches the limitations of claim 10. Therefore, claim 10 is unpatentable for the reasons set forth for claim 8 and subsequently, claim 1.
Regarding claims 11-19
Independent claims 11 is the method counterpart of system claim 1 and recites substantially the same substantive operations including:
jointly training the convolutional neural network, aggregation neural network, and output neural network to produce a prediction neural network having a first set of parameters;
training the prediction neural network to make a prediction of a time-changing entity;
receiving user-specific training data;
fine-tuning the prediction neural network, beginning from the first set of parameters, on the user-specific training data;
processing a time-windowed sequence of multimodal feature vectors using the convolutional neural network having normalization following each convolutional layer except the final convolutional layer;
aggregating the resulting latent feature vectors;
generating a prediction using the output neural network; and
advancing the time window and reprocessing the adjusted sequence to generate an updated prediction.
Accordingly, the teachings and rationales discussed above regarding claim 1 apply equally to claim 11.
In particular, Cheung teaches the core multimodal temporal prediction pipeline, including convolutional processing, an output neural network, bounded time-series inputs, prediction of a future condition, and implementation of computing hardware. Dang teaches a recurrent aggregation neural network that iteratively fuses multimodal encoded features over a temporal sequence and whose parameters are jointly trained with the entire model. Wu teaches applying normalization following the intermediate convolutional layers while omitting normalization following the final output-generating convolutional layer. Sandler teaches receiving user-specific training data and subsequently fine-tuning or personalizing an already-trained machine-learning model while beginning from and reusing the model’s existing trained parameter values.
For the reasons set forth above regarding claim 1, it would have been obvious to combine these teachings to perform the method recited in claim 11. Expressing the substantially identical neural-network operations as method steps rather than as operations performed by a system does not materially alter the prior-art analysis.
Claims 12-19 correspond respectively to system claims 2-9 and recite substantially the same additional limitations in method form:
claim 12 corresponds to claim 2;
claim 13 corresponds to claim 3;
claim 14 corresponds to claim 4;
claim 15 corresponds to claim 5;
claim 16 corresponds to claim 6;
claim 17 corresponds to claim 7;
claim 18 corresponds to claim 8; and
claim 19 corresponds to claim 9;
Accordingly, the teachings and rationales discussed above regarding claims 2-9 apply equally to claims 12-19, respectively. Claims 11-19 are therefore rejected under 35 U.S.C. § 103 as being unpatentable over Cheung in view of Dang, further in view of Wu, and further in view of Sandler.
Regarding claim 20
Independent claim 20 is directed to a non-transitory computer-readable storage medium storing instructions that, when executed by one or more computers, cause performance of substantially the same neural-network training, fine-tuning, temporal processing, aggregation, prediction, time-window adjustment, and reprocessing operations recited in independent claims 1 and 11.
Claim 20 expressly includes the amended limitations requiring:
jointly training the convolutional neural network, aggregation neural network, and output neural network to produce a prediction neural network having a first set of parameters;
training the prediction neural network to make a prediction of a time-changing entity;
receiving user-specific training data; and
fine-tuning the prediction neural network, beginning from the first set of parameters, on the user-specific training data;
For the reasons set forth above regarding claims 1 and 11, Cheung in view of Dang, further in view of Wu, and further in view of Sandler teaches or suggests the substantive operations recited in claim 20.
Cheung additionally teaches the claimed non-transitory computer-readable storage medium. In particular, Cheung expressly recites:
“A non-transitory machine-readable storage medium encoded with instructions for execution by a processor” (Cheung, pg. 10, claim 9)
Thus, Cheung teaches the claimed non-transitory storage-medium and stored-instructions limitations, while the substantive operations performed upon execution are taught or suggested by Cheung in view of Dang, further in view of Wu, and further in view of Sandler for the reasons discussed above regarding claims 1 and 11.
The recitation of the substantially identical operations as stored computer-executable instructions does not materially distinguish the claimed subject matter from the corresponding system and method implementations. Accordingly, claim 20 is rejected under 35 U.S.C. § 103 as being unpatentable over Cheung in view of Dang, further in view of Wu, and further in view of Sandler.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Paul Coleman whose telephone number is (571)272-4687. The examiner can normally be reached Mon-Fri.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL COLEMAN/ Examiner, Art Unit 2126
/DAVID YI/ Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights