DETAILED ACTION
Examiner Remarks
In light of Applicant’s Remarks and Amendments submitted on 09/03/2025, Examiner has withdrawn the objections to the Specification and the Rejections under §112(b). Furthermore, Applicant’s Remarks and more specifically paras. 0030-0040 of Applicant’s Specification recites a technical explanation of the recited improvement; namely, transforming a numerically based model into one that is able to also process textual data in a separate manner. And after examining Applicant’s Amended claims, the claims reflect this disclosed improvement in technology by reciting that representations of the historical time series data and the timestamped data are provided as separate inputs to the encoder of the trained model.
Accordingly, Examiner agrees with Applicant’s Remarks that Applicant’s invention provides an improvement to machine learning dealing with encoder architectures and withdraws the rejection under §101.
Response to Arguments
Applicant argues that the prior art of Lee does not teach the amended claim limitations of separately providing representations of the historical time series data and the time-stamped data as separate inputs to the encoder of the trained model and argues that Lee only discloses “just concatenating the text and time series embeddings into one joint vector before applying an encoder on top of the concatenated vector without, distinguishing which part is time or text.” See pg., 11 of Applicant’s Remarks submitted on 09/03/2025.
Respectfully, Examiner disagrees. As Lee details on pg., 2, “we introduce MoAT, a multi-modal augmented time series forecasting framework that addresses data scarcity issues by employing both sample-wise and feature-wise multi-modal augmentation.” (Emphasis added). After obtaining the patch embeddings for the time series and text data, sample-wise augmentation is employed which passes the two modalities as independent training samples to the encoder of a generic Transformer. As Lee details on pg., 5, “[t]o employ the sample-wise augmentation, we use embeddings from the two modalities as independent training samples. Specifically, we pass them separately to the shared encoder, preventing any explicit interactions between patches across modalities...[w]e refer to these embeddings as single-modal representations as they originate from individual modalities.”(Emphasis added). This is reflected by equation (2) as detailed herein:
z
^
t
i
m
e
i
=
Encoder(
z
t
i
m
e
i
+
W
t
i
m
e
p
o
s
)
∈
R
N
d
and
z
^
t
e
x
t
i
=
Encoder(
z
t
e
x
t
i
+
W
t
e
x
t
p
o
s
)
∈
R
N
d
Accordingly, Lee teaches the amended claim limitation of separately providing representations of the historical time series data and the time-stamped data as separate inputs to the encoder of the trained model and the 103 rejection has not been withdrawn.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. GR 20240100192, filed on 03/14/2024.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 06/03/2025 and 09/25/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-9, 11, 13-14, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, Geon, et al. "MoAT: Multi-Modal Augmented Time Series Forecasting." (2023)(“Lee”) in view of Crabtree et al., US 20180373766 A1(“Crabtree”).
Regarding claim 1, Lee teaches a method for predicting a next value in a time series, the method comprising:
[receiving a request for a value] related to a time series(Lee, pg., 7, “We evaluate the performance of MoAT across six multi-modal datasets: Fuel, Gold, Stock-Index, Stock-Tech, Bitcoin, and Covid. These datasets, which will be made publicly available, consist of multivariate time series data[related to a time series] ranging from daily to monthly resolution. Additionally, each dataset is accompanied by a collection of documents associated with each timestep.”);1
identifying a trained model comprising an encoder(Lee, pgs., 7-8, “We evaluate the performance of MoAT [identifying a trained model] across six multi-modal datasets…[b]y default, we used a hidden dimension of 64 and 4 attention heads for the Transformer encoder. For more specific configuration information, refer to Appendix B.4[comprising an encoder].”);
retrieving historical time series data and time-stamped information that is related to the time series data, the time-stamped information being of a different modality to the time series data(Lee, pg., 7, “We evaluate the performance of MoAT across six multi-modal datasets: Fuel, Gold, Stock-Index, Stock-Tech, Bitcoin, and Covid. These datasets, which will be made publicly available, consist of multivariate time series data ranging from daily to monthly resolution. Additionally, each dataset is accompanied by a collection of documents associated with each timestep.” & Lee, pg., 14-15, “Bitcoin is a daily dataset consisting of Bitcoin (BTC), Ethereum (ETH), Tether (USDT), and Binance Coin (BNB) prices…spanning from November 13, 2017 to November 23, 2019. For each day, we filtered tweets about Bitcoins…that received at least 100 likes and 50 retweets[retrieving historical time series data and time-stamped information that is related to the time series data, the time-stamped information being of a different modality to the time series data].”);
separately providing representations of the historical time series data and the timestamped data as separate inputs to the encoder of the trained model; obtaining, as output from the trained a model, a prediction of the value related to the time series; and providing the prediction of the value[as a response to the request](Lee, pg., 8, see also Table 1, “As shown in Table 1[and providing the prediction of the value],MoAT achieves a significant performance advantage over both uni-modal and multi-modal baselines.” & Lee, pg., 5, “To employ the sample-wise augmentation, we use embeddings from the two modalities as independent training samples. Specifically, we pass them separately to the shared encoder, preventing any explicit interactions between patches across modalities...[w]e refer to these embeddings as single-modal representations as they originate from individual modalities: detailed herein:
z
^
t
i
m
e
i
=
Encoder(
z
t
i
m
e
i
+
W
t
i
m
e
p
o
s
)
∈
R
N
d
and
z
^
t
e
x
t
i
=
Encoder(
z
t
e
x
t
i
+
W
t
e
x
t
p
o
s
)
∈
R
N
d
[separately providing representations of the historical time series data and the timestamped data as separate inputs to the encoder of the trained model; obtaining, as output from the trained a model, a prediction of the value related to the time series]”).2
Lee does not teach: receiving a request for a value; as a response to the request.
However, Crabtree teaches:
receiving a request for a value [related to a time series](Crabtree, para. 0033, see also fig. 1, “Multiple dimension time series data store module 120 may receive streaming data from a large plurality of sensors that may be of several different types[receiving a request for a value]. Multiple dimension time series data store module 120 may also store any time series data encountered by system 100….”);3
[and providing the prediction of the value] as a response to the request(Crabtree, para. 0034, see also fig. 1,“Results of the transformative analysis process may then be combined with further client directives[as a response to the request]… which also…machine learning algorithms 130a to allow future trends and outcomes to be rapidly forecast based upon the current system derived results and choosing each a plurality of possible business decisions.”).4
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Lee with the teachings of Crabtree the motivation to do so would be to develop a data collection and extraction system that curates multimodal data for multimodal machine learning tasks(Crabtree, paras. 0003-0005, “Another
problem encountered when processing richly formatted data with machine learning is the multimodal nature of the data, for example, images, audio, and video may exist as supplementary
data. What is needed is a system that can take gather multimodal data… [a]ccordingly, the inventor has conceived, and reduced to practice, a system and method for automated scalable contextual data collection and extraction”).
Regarding claim 2, Lee in view of Crabtree teaches the method of claim 1, wherein the trained model has a transformer architecture(Lee, pg., 5, “Commonly, we introduce modality-specific learnable positional embeddings
W
t
i
m
e
p
o
s
∈
R
N
×
d
and
W
t
e
x
t
p
o
s
∈
R
N
×
d
for time series and text, respectively. In addition, we employ a vanilla Transformer encoder (denoted as “Encoder”below) equipped with multi-head attention[wherein the trained model has a transformer architecture].” ).
Regarding claim 3, Lee in view of Crabtree teaches the method of claim 1, wherein the time-stamped information is provided as input to the trained model by dividing the timestamped information into a token sequence, each token in the token sequence having a timestamp(Lee, pgs. 4-5, “Specifically, we are given a sequence
D
=
(
D
1
,
…
,
D
L
)
of sets of documents, where at each timestep t, we have a set
D
t
=
(
D
t
,
1
,
…
,
D
t
,
|
D
t
|
)
of an arbitrary number of texts. Firstly, we use the pretrained language model…to represent each text
D
t
,
j
as a
d
'
-dimensional embedding vector…[t]his yields an embedding matrix
d
t
=
d
t
,
1
,
…
,
d
t
,
|
D
t
|
∈
R
|
D
t
|
×
d
'
of texts at each timestep t[by dividing the timestamped information into a token sequence, each token in the token sequence having a timestamp].” & Lee, pg., 15, “To generate embeddings for each text, we used the pretrained language model. Specifically, we used the sentence transformer provided by Hugging Face with the pretrained model named all-mpnet-base-v2…which is trained on a 1B sentence pairs dataset using contrastive loss.”).
Regarding claim 4, Lee in view of Crabtree teaches the method of claim 1, wherein the time series comprises prices for one or more assets over a preceding time period and the different modality of the time-stamped information is text(Lee, pg., 14-15, “Bitcoin is a daily dataset consisting of Bitcoin (BTC), Ethereum (ETH), Tether (USDT), and Binance Coin (BNB) prices…spanning from November 13, 2017 to November 23, 2019[prices for one or more assets over a preceding time period]. For each day, we filtered tweets about Bitcoins…that received at least 100 likes and 50 retweets[and the different modality of the time-stamped information is text].”).
Regarding claim 5, Lee in view of Crabtree teaches the method of claim 4, wherein the timestamped information comprises news headlines(Lee, pg., 14-15, “For each day, we filtered tweets about Bitcoins…that received at least 100 likes and 50 retweets[news headlines].”).
Regarding claim 6, Lee in view of Crabtree teaches the method of claim 1, wherein the trained model was trained using supervised learning, semi-supervised learning, or both(Lee, pg. 15, “For each dataset, we split the time series and text data based on their temporal order into
train, validation, and test sets using a 6:2:2 ratio… [t]he total number of training epochs is set to 200. But, if the validation loss fails to decrease for 10 consecutive epochs, we stop the training process early… [o]ur offline prediction synthesis module is trained separately from the
main forecasting model. We utilize a Ridge regression approach, incorporating 16 weight parameters and an intercept for aggregating the 16 predictions from the decoder. During each training epoch, a Ridge regression is trained from scratch using the training set. Then, the model’s learned parameters are utilized to evaluate validation and test sets[was trained using supervised learning].”).5
Regarding claim 7, Lee in view of Crabtree teaches the method of claim 1, wherein the request is received from a client device via an application programming interface (API) and providing the response comprises sending the prediction of the value to the client device(Crabtree, para. 0033, see also fig. 1, “FIG. 1 is a diagram of an exemplary architecture of
a business operating system 100… Client access to system 105 for specific data entry, system control and for interaction with system output such as automated predictive decision making
and planning and alternate pathway simulations[and providing the response comprises sending the prediction of the value to the client device], occurs through the system's distributed, extensible high bandwidth cloud interface 110 which uses a versatile, robust web application driven interface for both input and display of client-facing information and a data store 112[wherein the request is received from a client device via an application programming interface (API)]….”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Lee with the above teachings of Crabtree for the same rationale stated at Claim 1.
Regarding claim 8, Lee in view of Crabtree teaches the method of claim 1, wherein the historical time series data comprises one or more multivariate series(Lee, pg., 4, “The problem of multivariate time series forecasting is defined as predicting the future T steps of time-series values
X
=
x
L
+
1
,
…
,
x
L
+
T
∈
R
T
×
C
, based on the past L steps of time series data Y=
x
1
,
…
,
x
L
∈
R
L
×
C
where C represents the dimension or the number of channels of the time series data[one or more multivariate series]. ”), the method further comprising:
dividing the one or more multivariate series into a plurality of univariate series(Lee, pg., 4, “The problem of multivariate time series forecasting is defined as predicting the future T steps of time-series values
X
=
x
L
+
1
,
…
,
x
L
+
T
∈
R
T
×
C
, based on the past L steps of time series data Y=
x
1
,
…
,
x
L
∈
R
L
×
C
where C represents the dimension or the number of channels of the time series data…[t]o patch time series
x
(
i
)
=
x
1
i
,
…
,
x
L
i
∈
R
L
of the i-th channel[dividing the one or more multivariate series into a plurality of univariate series]….” & Lee, pg., 13, “[W]e incorporate the concept of channel independence within the MoAT framework. In this approach, instead of mixing information across channels, we consider each channel as an independent data sample that shares the projection weight parameters and Transformer encoder weights. Thus, each input time series sample is considered a univariate time series[dividing the one or more multivariate series into a plurality of univariate series]….”);
and dividing each univariate series into patches, each patch corresponding to a time range(Lee, pgs. 4-5, see also fig. 3(a), “To patch time series
x
(
i
)
=
x
1
i
,
…
,
x
L
i
∈
R
L
of the i-th channel, we segment it into multiple (non-)overlapping patches. Precisely, given the patch length P and the stride S, we segment
x
(
i
)
into N patches, each of length P…denoted as
p
(
i
)
=
p
1
i
,
…
,
p
N
i
∈
R
N
×
P
[and dividing each univariate series into patches, each patch corresponding to a time range].”),
wherein the patches are embedded are provided as input to an encoder of the trained model(Lee, pgs. 4-5, see also fig. 3(a), “We map these patches into a d-dimensional latent space using a learnable linear projection
W
t
i
m
e
∈
R
P
×
d
i.e.,
z
t
i
m
e
(
i
)
=
p
(
i
)
W
t
i
m
e
∈
R
N
×
d