Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites “process first input time-series data associated with a first time range using an embedding generator to generate an input embedding, the input embedding including a positional embedding and a temporal embedding, wherein the positional embedding indicates a position of an input value of the first input time-series data within the first input time-series data, and wherein the temporal embedding indicates that a first time associated with the input value is included in at least one of a particular day, a particular week, a particular month, a particular year, or a particular holiday.” Generating an input embedding is a mental process that can be practically performed in the human mind, perhaps with the aid of pencil and paper. This mental process merely evaluates data and decides on different data (such as a vector) as a representation. The claim further recites “process the input embedding using a predictor to generate second predicted time-series data associated with a second time range, wherein the second time range is subsequent to at least a portion of the first time range.” This too is a mental process of judgement or evaluation that considers data (the embedding) and decides on additional data as a prediction. The claim does not recited much detail about how these processes of generating data are performed, so they can be simple evaluation and decision processes performed in the human mind.
This judicial exception is not integrated into a practical application because the only result of the claim is the limitation “provide an output to a second device, the output based on the second predicted time-series data.” Providing output is mere data gathering, which is insignificant extra-solution activity and therefore not significantly more than an abstract idea. The claim does not do anything practical with the output, so the result of the claim is merely data, which is an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the only additional elements are one or more processors, an embedding generator, and a predictor. Processors are generic computing components. They are recited at a high degree of generality, so the use of processors is merely applying the abstract idea to a generic computing environment. The embedding generator and predictor appear to be software executed by processors, so they too are generic computing components and therefore not significantly more than an abstract idea.
Claim 2 recites “detect, based on a comparison of second input time-series data and the second predicted time-series data, a change of operating mode of a monitored system.” Comparing data values is a mental process that evaluates the values and judges their similarity or difference. This judicial exception is not integrated into a practical application because the only result of the claim is information indicating a change in mode; nothing practical is performed using the information. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the only additional element is the limitation “receive second input time-series data associated with the second time range.” Receiving data is mere data gathering, which is insignificant extra-solution activity.
Claim 3 recites “generate an alert responsive to determining that the change of operating mode corresponds to an anomaly.” Determining that a change of operating mode corresponds to an anomaly is a mental process of judgement or evaluation that considers the change in operating mode (i.e. information) and decides if the change in mode is different from a typical operating mode (i.e. making a comparison). This judicial exception is not integrated into a practical application because the only result of the claim is information indicating an anomaly; nothing practical is performed using the information. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the only additional element is “generate an alert.” Generating (and outputting) an alert is mere data gathering, which is insignificant extra-solution activity.
Claim 4 recites “wherein the one or more processors are configured to, in response to determining that the change of operating mode corresponds to an anomaly, generate the output to indicate one or more features of the first input time-series data that have the greatest impact in determining the second predicted time-series data.” Indicating one or more features of the first input time-series data that have the greatest impact . . .” is a mental process of judgment or evaluation that considers the features and judges which one(s) have the most effect on the output. There are no limitations placed on how this process is performed (i.e. very little detail is provided by the claim), so it can be simply and practically performed in the human mind. As with the previous claims, there is no practical application recited and the only additional elements are generic computing components and mere data gathering, which is insignificant extra-solution activity.
Claim 5 recites “determine residual data based on a comparison of the second input time-series data and the second predicted time-series data.” Comparing is a mental process of judgement or evaluation that judges differences between data elements, and determining residual data is a mental process that decides on data values as residual data. The claim further recites “determine a risk score based on the residual data.” This too is a mental process that judges the residual data and decides on a value.
This judicial exception is not integrated into a practical application because the only result of the claim is the limitation “based on determining that the risk score is greater than a threshold, generate an output indicating detection of a change of operating mode of a monitored system.” Generating output is mere data gathering, which is insignificant extra-solution activity and therefore not significantly more than an abstract idea. The claim does not do anything practical with the output, so the result of the claim is merely data, which is an abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the only additional elements are one or more processors, which are generic computing components as detailed above.
Claim 6 recites “wherein the first input time-series data includes a plurality of sensor values generated during the first time range by a plurality of sensors.” Sensors are generic computing components recited at a high degree of generality, so they do not render the claim significantly more than an abstract idea.
Claim 7 recites “wherein the embedding generator includes a batch normalization layer configured to apply normalization to a first batch of time-series data to generate a first batch of normalized time-series data, wherein the first batch of time-series data includes the first input time-series data, and wherein the first batch of normalized time-series data includes first normalized time-series data corresponding to the first input time-series data.” Normalization is a mathematical calculation, which is an abstract idea. This judicial exception is not integrated into a practical application because the only result of the claim is normalized data, which is an abstract idea; nothing practical is performed using the normalized data. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the only additional element is a batch normalization layer which is a generic computing component recited at a high degree of generality.
Claim 8 recites “wherein the embedding generator includes a spatial attention layer configured to apply first weights to the first normalized time-series data to generate first weighted time-series data.” Applying weights to data is understood as multiplication, which is a mathematical calculation, an abstract idea. There is no practical application recited, and the only additional element is a spatial attention layer, which is a generic computing component recited at a high degree of generality.
Claim 9 recites “wherein a first batch of weighted time-series data includes a plurality of sequences of weighted time-series data, wherein a first sequence of weighted time-series data includes the first weighted time-series data, wherein the embedding generator includes a convolution layer configured to apply convolution weights to the first sequence of weighted time-series data to generate first convolved time-series data, and wherein the input embedding is based at least in part on the first convolved time-series data.” Applying convolution weights is a set of mathematical calculations (multiplication and addition), which is an abstract idea. There is no practical application recited, and the only additional elements are forms of data, which are abstract ideas.
Claim 10 recites “wherein the predictor includes: an encoder configured to process the input embedding to generate encoded data; and a decoder configured to process the encoded data to generate the second predicted time-series data.” Generating the encoded data and generating predicted time-series data are a mental processes, as described for claim 1. There is no practical application recited because the only thing produced by the claim is data. An encoder and decoder are generic computing components recited at a high degree of generality, so they do not render the claim significantly more than an abstract idea.
Claim 11 recites “wherein the encoder comprises a first masked multi-head attention network, wherein an input to the first masked multi-head attention network is based on the input embedding, and wherein the encoded data is based on an output of the first masked multi-head attention network.” A masked multi-head attention network is a generic computing component recited at a high degree of generality. The rest of the claim merely recites details about data, which are abstract ideas.
Claim 12 recites “wherein the encoder comprises a fourier transform layer, wherein an input to the fourier transform layer is based on the input embedding, and wherein the encoded data is based on an output of the fourier transform layer.” A fourier transform is a mathematical calculation; a fourier transform layer (i.e. a layer that performs a fourier transform) is a generic computing component recited at a high degree of generality. So, none of the elements of the claim are significantly more than an abstract idea. The claim does not recite a practical application because the only thing produced is data.
Claim 13 recites “wherein the decoder is further configured to process the encoded data to generate predicted time-series data associated with multiple time ranges subsequent to the first time range, and wherein the multiple time ranges include the second time range.” Generating predicted time-series data is a mental process of judgment or evaluation, as detailed for claim 1. The claim does not recite a practical application, and the only additional elements are generic computing components, as detailed above.
Claim 14 recites “wherein the one or more processors are further configured to receive one or more input values of the first input time-series data from a sensor during the first time range, wherein the one or more input values include the input value, and wherein the position of the input value indicated by the positional embedding corresponds to a position of receipt of the input value relative to receipt of the one or more input values.” Receiving data is mere data gathering; the rest of the claim elements are simply details about data values, which are abstract ideas. The claim does not recite a practical application.
Claim 15 recites “wherein the input value is received from the sensor at the first time, and wherein the first time is included in the first time range.” This is merely details about data, which are abstract ideas.
Claim 16 recites “wherein the predictor is further configured to process the input embedding to generate predicted time-series data associated with multiple time ranges subsequent to the first time range, and wherein the multiple time ranges include the second time range.” Generating predicted data is a mental process, as detailed above. The claim does not recite a practical application because nothing is done with the predicted data. The only additional elements are generic computing components, as detailed above.
Claim 17 recites “wherein the second device includes at least one of a display device, a storage device, or a controller of a monitored system.” All of the recited elements are generic computing components, so they do not render the claim significantly more than an abstract idea.
Claim 18 recites elements substantially similar to those of claim 1, so it recites an abstract idea without significantly more for the same reasons.
Claim 19 recites “processing second input time-series data using the embedding generator to generate a second input embedding, the second input time-series data associated with the second time range; and processing the second input embedding using the predictor to generate third predicted time-series data associated with a third time range.” Generating predicted time-series data is a mental process, as explained above. The claim does not recite a practical application because nothing practical is done with the predicted data. The only additional elements are generic computing components (the embedding generator and the predictor), which do not render the claim significantly more than an abstract idea, as explained above.
Claim 20 recites elements substantially similar to those of claim 1, so recites an abstract idea without significantly more for the same reasons. The non-transitory computer readable medium storing instructions and one or more processors are generic computing components recited at a high degree of generality, so they do not render the claim significantly more than an abstract idea.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 6, 10, and 13-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Thopalli et al. (U.S. 2022/0327335, hereinafter “Thopalli”).
Regarding Claim 1, Thopalli teaches a device (figs. 1 and 2; ¶ [0023]) comprising:
one or more processors (figs. 1 and 2; ¶ [0026] and [0085]) configured to:
process first input time-series data associated with a first time range using an embedding generator to generate an input embedding, the input embedding including a positional embedding and a temporal embedding, wherein the positional embedding indicates a position of an input value of the first input time-series data within the first input time-series data, and wherein the temporal embedding indicates that a first time associated with the input value is included in at least one of a particular day, a particular week, a particular month, a particular year, or a particular holiday (figs. 2 and 4; ¶ [0028] and [0034] – [0035]—a fusion network system generates spatial embeddings and temporal embeddings from sensor or other time-series data, and fuses them into a combined embedding that includes both a positional embedding and a temporal embedding. The spatial embedding indicates a position of an input value within time-series data, such as a position in an image of a geographical location. The temporal embedding indicates a time at any time scale, such as minutes, hours, or days, as described in examples in ¶ [0027], [0040], and [0075]);
process the input embedding using a predictor to generate second predicted time-series data associated with a second time range, wherein the second time range is subsequent to at least a portion of the first time range (¶ [0029] – [0030] and [0053]—a variety of data can be predicted, including second time-series data such as wildfire boundary maps at time t+1 {i.e. a second time range subsequent to the first time range}); and
provide an output to a second device, the output based on the second predicted time-series data (¶ [0053] – [0054]—the prediction is output).
Regarding Claim 6, Thopalli wherein the first input time-series data includes a plurality of sensor values generated during the first time range by a plurality of sensors (¶ [0025] and [0034]).
Regarding Claim 10, Thopalli teaches wherein the predictor includes: an encoder configured to process the input embedding to generate encoded data; and a decoder configured to process the encoded data to generate the second predicted time-series data (fig. 3; ¶ [0031]. A decoder generating the second predicted time-series data is further described in ¶ [0060] – [0062]).
Regarding Claim 13, Thopalli teaches wherein the decoder is further configured to process the encoded data to generate predicted time-series data associated with multiple time ranges subsequent to the first time range, and wherein the multiple time ranges include the second time range (¶ [0029] – [0030] and [0053]—a variety of data can be predicted, including second time-series data such as wildfire boundary maps at time t+1 {i.e. a second time range subsequent to the first time range}. The time t+1 indicates that the process is repeated for multiple times, thus generating predicted time-series data for multiple time ranges).
Regarding Claim 14, Thopalli teaches wherein the one or more processors are further configured to receive one or more input values of the first input time-series data from a sensor during the first time range, wherein the one or more input values include the input value, and wherein the position of the input value indicated by the positional embedding corresponds to a position of receipt of the input value relative to receipt of the one or more input values (fig. 4; ¶ [0025], [0034], and [0048]—input data are received from sensors, and the positional embeddings of the input data correspond to geographical locations, i.e. positions of receipt of the input values from the sensors).
Regarding Claim 15, Thopalli teaches wherein the input value is received from the sensor at the first time, and wherein the first time is included in the first time range (¶ [0040]—input values are received in time series data during a first time range such as a first hour or day).
Regarding Claim 16, Thopalli teaches wherein the predictor is further configured to process the input embedding to generate predicted time-series data associated with multiple time ranges subsequent to the first time range, and wherein the multiple time ranges include the second time range (¶ [0029] – [0030] and [0053]—a variety of data can be predicted, including second time-series data such as wildfire boundary maps at time t+1 {i.e. a second time range subsequent to the first time range}. The time t+1 indicates that the process is repeated for multiple times, thus generating predicted time-series data for multiple time ranges subsequent to the first time range).
Regarding Claim 17, Thopalli teaches wherein the second device includes at least one of a display device, a storage device, or a controller of a monitored system (¶ [0083]).
Regarding Claim 18, Thopalli teaches a method (¶ [0026] and [0201]) comprising:
processing first input time-series data associated with a first time range using an embedding generator to generate an input embedding, the input embedding including a positional embedding and a temporal embedding, wherein the positional embedding indicates a position of an input value of the first input time-series data within the first input time-series data, and wherein the temporal embedding indicates that a first time associated with the input value is included in at least one of a particular day, a particular week, a particular month, a particular year, or a particular holiday (figs. 2 and 4; ¶ [0028] and [0034] – [0035]—a fusion network system generates spatial embeddings and temporal embeddings from sensor or other time-series data, and fuses them into a combined embedding that includes both a positional embedding and a temporal embedding. The spatial embedding indicates a position of an input value within time-series data, such as a position in an image of a geographical location. The temporal embedding indicates a time at any time scale, such as minutes, hours, or days, as described in examples in ¶ [0027], [0040], and [0075]);
processing the input embedding using a predictor to generate second predicted time-series data associated with a second time range (¶ [0029] – [0030], and [0053]—a variety of data can be predicted, including second time-series data such as wildfire boundary maps at time t+1 {i.e. a second time range subsequent to the first time range}); and
providing an output to a second device, the output based on the second predicted time-series data (¶ [0053] – [0054]—the prediction is output).
Regarding Claim 19, Thopalli teaches:
processing second input time-series data using the embedding generator to generate a second input embedding, the second input time-series data associated with the second time range (¶ [0034] – [0035]—the embedding can process time-series data, which clearly includes multiple time ranges including a second time range); and
processing the second input embedding using the predictor to generate third predicted time-series data associated with a third time range (¶ [0053] – [0054]—output at time t+1 for a time series is clearly repeated for multiple time ranges, thus predicting third time-series data associated with a third time range that follows the second time range).
Regarding Claim 20, Thopalli teaches a non-transitory computer-readable medium storing instructions (¶ [0026] and [0110]) that, when executed by one or more processors, cause the one or more processors to:
process first input time-series data associated with a first time range using an embedding generator to generate an input embedding, the input embedding including a positional embedding and a temporal embedding, wherein the positional embedding indicates a position of an input value of the first input time-series data within the first input time-series data, and wherein the temporal embedding indicates that a first time associated with the input value is included in at least one of a particular day, a particular week, a particular month, a particular year, or a particular holiday (figs. 2 and 4; ¶ [0028] and [0034] – [0035]—a fusion network system generates spatial embeddings and temporal embeddings from sensor or other time-series data, and fuses them into a combined embedding that includes both a positional embedding and a temporal embedding. The spatial embedding indicates a position of an input value within time-series data, such as a position in an image of a geographical location. The temporal embedding indicates a time at any time scale, such as minutes, hours, or days, as described in examples in ¶ [0027], [0040], and [0075]);
process the input embedding using a predictor to generate second predicted time-series data associated with a second time range (¶ [0029] – [0030], and [0053]—a variety of data can be predicted, including second time-series data such as wildfire boundary maps at time t+1 {i.e. a second time range subsequent to the first time range}); and
provide an output to a second device, the output based on the second predicted time-series data (¶ [0053] – [0054]—the prediction is output).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Thopalli, as applied to claim 1, above, in view of Li, Zhihan, et al. (“Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding,” Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021; hereinafter “Li”).
Regarding Claim 2, Thopalli teaches wherein the one or more processors are further configured to: receive second input time-series data associated with the second time range (¶ [0034] – [0035]—the embedding can process time-series data, which clearly includes multiple time ranges including a second time range).
Thopalli does not specifically teach the one or more processors are configured to: detect, based on a comparison of second input time-series data and the second predicted time-series data, a change of operating mode of a monitored system. However, Li teaches detecting, based on a comparison of second input time-series data and a second predicted time-series data, a change of operating mode of a monitored system (section 3.3, “Inference” portion—a machine learning system uses MCMC imputation to predict a second time-series data. An anomaly is detected when the imputed time-series data differs from the actual observed data).
All of the claimed elements were known in Thopalli and Li and could have been combined by known methods with no change in their respective functions. It therefore would have been obvious to a person of ordinary skill in the art at the time of filing of the applicant’s invention to combine the anomaly detection of Li with the time-series data of Thopalli to yield the predictable result of the one or more processors being configured to: detect, based on a comparison of second input time-series data and the second predicted time-series data, a change of operating mode of a monitored system. One would be motivated to make this combination for the purpose of reducing labor time and costs for detecting anomalies in complex systems (Li, section 1).
Regarding Claim 3, Thopalli/Li teaches wherein the one or more processors are configured to generate an alert responsive to determining that the change of operating mode corresponds to an anomaly (Li, section 1 and section 3.3, “Inference” portion).
Regarding Claim 4, Thopalli/Li teaches wherein the one or more processors are configured to, in response to determining that the change of operating mode corresponds to an anomaly, generate the output to indicate one or more features of the first input time-series data that have the greatest impact in determining the second predicted time-series data (Li, section 3.4 and algorithm 1—the anomalous dimensions of the MTS embedding are determined to interpret which dimensions have the greatest impact on determining the anomaly).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Thopalli, as applied to claim 1, above, in view of Kesavan et al. (U.S. 2023/0071606, hereinafter “Kesavan”).
Regarding Claim 5, Thopalli does not specifically teach wherein the one or more processors are further configured to:
receive second input time-series data associated with the second time range; determine residual data based on a comparison of the second input time-series data and the second predicted time-series data;
determine a risk score based on the residual data; and
based on determining that the risk score is greater than a threshold, generate an output indicating detection of a change of operating mode of a monitored system.
However, Kesavan teaches one or more processors configured to: receive second input time-series data associated with a second time range; determine residual data based on a comparison of the second input time-series data and second predicted time-series data; determine a risk score based on the residual data; and based on determining that the risk score is greater than a threshold, generate an output indicating detection of a change of operating mode of a monitored system (¶ [0018]—a residual is used to determine anomalies in a monitored system, as further described in ¶ [0094]; an anomaly is a change of operating mode of a system. The health score is a risk score, and detailed in ¶ [0111] because it represents the risk of failure).
All of the claimed elements were known in Thopalli and Kesavan and could have been combined by known methods with no change in their respective functions. It therefore would have been obvious to a person of ordinary skill in the art at the time of filing of the applicant’s invention to combine the residual and anomaly detection of Kesavan with the time-series data and embeddings of Thopalli to yield the predictable result of wherein the one or more processors are further configured to: receive second input time-series data associated with the second time range; determine residual data based on a comparison of the second input time-series data and the second predicted time-series data; determine a risk score based on the residual data; and based on determining that the risk score is greater than a threshold, generate an output indicating detection of a change of operating mode of a monitored system. One would be motivated to make this combination for the purpose of improving availability of a system by predicting anomalies and failures (Kesavan, Abstract and ¶ [0012]).
Claims 7-9 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Thopalli, as applied to claim 1, above, in view of Gong et al. (U.S. 2021/0004682, hereinafter “Gong”).
Regarding Claim 7, Thopalli does not specifically teach wherein the embedding generator includes a batch normalization layer configured to apply normalization to a first batch of time-series data to generate a first batch of normalized time-series data, wherein the first batch of time-series data includes the first input time-series data, and wherein the first batch of normalized time-series data includes first normalized time-series data corresponding to the first input time-series data.
However, Gong teaches wherein an embedding generator includes a batch normalization layer configured to apply normalization to a first batch of time-series data to generate a first batch of normalized time-series data, wherein the first batch of time-series data includes the first input time-series data, and wherein the first batch of normalized time-series data includes first normalized time-series data corresponding to the first input time-series data (¶ [0133] and [0161]—pre-processing of input time-series data may include normalization to generate a batch of normalized time-series data. The embedding generator is further described in ¶ [0072] – [0073]).
All of the claimed elements were known in Thopalli and Gong and could have been combined by known methods with no change in their respective functions. It therefore would have been obvious to a person of ordinary skill in the art at the time of filing of the applicant’s invention to combine the batch normalization of Gong with the time-series data and embedding generator of Thopalli to yield the predictable result of wherein the embedding generator includes a batch normalization layer configured to apply normalization to a first batch of time-series data to generate a first batch of normalized time-series data, wherein the first batch of time-series data includes the first input time-series data, and wherein the first batch of normalized time-series data includes first normalized time-series data corresponding to the first input time-series data. One would be motivated to make this combination for the purpose of improving the efficiency of sequence models that predict time-series data (Gong, ¶ [0005]).
Regarding Claim 8, Thopalli/Gong teaches wherein the embedding generator includes a spatial attention layer configured to apply first weights to the first normalized time-series data to generate first weighted time-series data (Gong, ¶ [0073] – [0076]—the embedding generator includes an attention model that applies weights to the normalized temporal and spatial data to generate weighted time-series data).
Regarding Claim 9, Thopalli/Gong teaches wherein a first batch of weighted time-series data includes a plurality of sequences of weighted time-series data, wherein a first sequence of weighted time-series data includes the first weighted time-series data, wherein the embedding generator includes a convolution layer configured to apply convolution weights to the first sequence of weighted time-series data to generate first convolved time-series data, and wherein the input embedding is based at least in part on the first convolved time-series data (Gong, ¶ [0110] – [0111]—machine-learned model 300 can include one or more convolutional neural networks with convolutional layers. ¶ [0125] explains that the embeddings may be performed by the machine-learned model 300, indicating that embedding generator includes a convolution layer that applies weights to the sequences of weighted time-series data).
Regarding Claim 12, Thopalli/Gong teaches wherein the encoder comprises a fourier transform layer, wherein an input to the fourier transform layer is based on the input embedding, and wherein the encoded data is based on an output of the fourier transform layer (Gong, ¶ [0129]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Thopalli, as applied to claim 1, above, in view of Souly et al. (U.S. 2023/0051237, hereinafter “Souly”).
Regarding Claim 11, Thopalli teaches using an attention mechanism (¶ [0041] and [0054] – [0055]—examples use a weighted attention mechanism, and Thopalli states that other attention mechanisms may be use), but does not explicitly teach wherein the encoder comprises a first masked multi-head attention network, wherein an input to the first masked multi-head attention network is based on the input embedding, and wherein the encoded data is based on an output of the first masked multi-head attention network.
However, Souly teaches an encoder comprises a first masked multi-head attention network, wherein an input to the first masked multi-head attention network is based on an input embedding, and wherein the encoded data is based on an output of the first masked multi-head attention network (fig. 2; the encoder receives an embedding of an input sequence and comprises a masked multi-head attention network that generates an output).
All of the claimed elements were known in Thopalli and Souly and could have been combined by known methods with no change in their respective functions. It therefore would have been obvious to a person of ordinary skill in the art at the time of filing of the applicant’s invention to combine the multi-head attention network of Souly with the encoder and attention mechanism of Thopalli to yield the predictable result of wherein the encoder comprises a first masked multi-head attention network, wherein an input to the first masked multi-head attention network is based on the input embedding, and wherein the encoded data is based on an output of the first masked multi-head attention network. One would be motivated to make this combination for the purpose of improving predictions by enabling the network to focus on relevant information from other portions of a sequence (Souly, ¶ [0040]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. This art includes:
Song, Huan, et al. (“Attend and diagnose: Clinical time series analysis using attention models,” Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018) teaches time series data analysis using an input embedding and a positional encoding
Gugulothu, Narendhar, et al. (“Predicting remaining useful life using time series embeddings based on recurrent neural networks,” arXiv preprint arXiv:1709.01073 (2017)) teaches predicting remaining useful life of a device using an RNN encoder-decoder for time-series embeddings of sensor data
Ferragut et al. (U.S. 2015/0106927) teaches anomaly detection using a position encoding and a time value
Kumar et al. (U.S. Patent 11,625,627) teaches an encoder with attention layers and normalization
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAL W SCHNEE whose telephone number is (571) 270-1918. The examiner can normally be reached M-F 7:30 a.m. - 6:00 p.m.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at 303-297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAL SCHNEE/Primary Examiner, Art Unit 2129