DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to pending claims 1-20 filed 6/23/2026.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 are rejected under 35 U.S.C. 103 as being unpatentable over Padhi ("Tabular transformers for modeling multivariate time series", published 2021) in view of Singh (US 20230222178 A1) in view of Arora ("Language models enable simple systems for generating structured views of heterogeneous data lakes", published 4/20/2023).
For claim 1, Padhi discloses: a system for generating synthetic training data (§3 ¶1 discloses use of GPT models for generating synthetic tabular data, such as for modeling user credit card purchases), the system comprising:
operations comprising:
receiving a first set of candidate behaviors (§3 ¶1-2: receiving credit card dataset, see §2.3: Transaction Datatset, §2.1 fig.1 showing example sequential tabular data), wherein each candidate behavior comprises plain text describing user activity (ibid, particularly fig.1);
processing the first set of candidate behaviors using a first language processing model to generate a set of representations associated with the first set of candidate behaviors in a real-valued embedding space (§3 ¶1-2: plain text data is processed via a language model that operates on language data, first via quantization and then via embedding via the TabGPT encoder, see fig.3, the encoder deep neural nets implemented via GPT generating a real-valued embedding, see fig.3), wherein the first language processing model is a deep learning model trained to map candidate behaviors to user activity representations in the real-valued embedding space (ibid: GPT models);
for each representation in the set of representations, processing the representation using a second language processing model to generate a first sequence of behavior tokens representative of a timeline of user activities (§3 ¶1-2, fig.3: representation embeddings are processed via the causal encoder model in order to generate behavior token sequence representing timeline of user purchases, see §3 ¶3), wherein the second language processing model is trained on prior user activity (§3 ¶2-3: training on prior transaction data);
using the set of representations and first sequence of behavior tokens, updating the second language processing model (§3 ¶2-3, fig.3: operating the GPT model during training phase to adjust parameters based on training data, hence, using generated representation and generated tokens, autoregressively input into the GPT model, in order to update decoder parameters); and
using the updated second language processing model, processing a second set of candidate behaviors to generate a second sequence of behavior tokens (§3 ¶4 goes over evaluation of synthetic data, hence, processing additional data, such as via process of fig.3 ¶3). Padhi does not disclose:
one or more processors; and
one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors;
using the second sequence of behavior tokens, generating notifications indicating expected user activity.
Wherein the candidate behavior includes one or more sentences.
Singh discloses: one or more processors (fig.7A:722); and
one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors (fig.7A:726);
using the second sequence of behavior tokens, generating notifications indicating expected user activity (0003, 0154: generating story notifications with synthetic data, such as display via UI of fig.4A, ).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi by incorporating the hardware architecture of Singh. Both concern the art of synthetic data generation, and the incorporation would have, provided a common hardware configuration for implementing the method (0154).
Padhi modified by Singh does not include he remaining limitations.
Arora discloses: wherein the behaviors include one or more sentences (Arora discloses general purpose structured data extraction, such as applied to financial data, natural language data, etc.; see §F, fig.5 (p.29-30) showing exemplary tabular data generation from NL language data including sentence data, Table 4 (p.13) showing application to various databases containing sentence data, hence, combination with Padhi yielding application to behaviors encoded in sentence data).
It would have been obvious before the effective filing date to one of ordinary skill in the art to modify the method of Padhi by incorporating the data extraction technique of Arora. Both concern the art of tabular and structured data processing, and the incorporation would have, according to Arora, unlock insights trapped in data lakes, including financial data (§1 ¶1-2).
Claim(s) 2, 4-6, 10, 12-13, 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Padhi ("Tabular transformers for modeling multivariate time series", published 2021) in view of Arora ("Language models enable simple systems for generating structured views of heterogeneous data lakes", published 4/20/2023).
For claim 2, Padhi discloses: a method for generating synthetic training data, the method comprising:
receiving a first set of candidate behaviors (§3 ¶1-2: receiving credit card dataset, see §2.3: Transaction Datatset, §2.1 fig.1 showing example sequential tabular data including plain text);
processing the first set of candidate behaviors using a first language processing model to generate a set of representations in an embedding space (§3 ¶1-2: plain text data is processed via a language model that operates on language data, first via quantization and then via embedding via the TabGPT encoder, see fig.3, the encoder deep neural nets implemented via GPT generating a real-valued embedding, see fig.3);
for each representation in the set of representations, processing the representation using a second language processing model to generate a first sequence of behavioral tokens representative of a timeline of user activities (§3 ¶1-2, fig.3: representation embeddings are processed via the causal encoder model in order to generate behavior token sequence representing timeline of user purchases, see §3 ¶3);
using the set of representations and first sequence of behavior tokens, updating the second language processing model (§3 ¶2-3, fig.3: operating the GPT model during training phase to adjust parameters based on training data, hence, using generated representation and generated tokens, autoregressively input into the GPT model, in order to update decoder parameters); and
using the updated second language processing model, processing a second set of candidate behaviors to generate a second sequence of behavior tokens (§3 ¶4 goes over evaluation of synthetic data, hence, processing additional data, such as via process of fig.3 ¶3).
Padhi does not disclose: wherein the behaviors include one or more sentences.
Arora discloses: wherein the behaviors include one or more sentences (Arora discloses general purpose structured data extraction, such as applied to financial data, natural language data, etc.; see §F, fig.5 (p.29-30) showing exemplary tabular data generation from NL language data including sentence data, Table 4 (p.13) showing application to various databases containing sentence data, hence, combination with Padhi yielding application to behaviors encoded in sentence data).
It would have been obvious before the effective filing date to one of ordinary skill in the art to modify the method of Padhi by incorporating the data extraction technique of Arora. Both concern the art of tabular and structured data processing, and the incorporation would have, according to Arora, unlock insights trapped in data lakes, including financial data (§1 ¶1-2).
For claim 4, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi further discloses: wherein the second language processing model performs sequential next-token prediction using a transformer algorithm (§3 ¶1-2: GPT model constitutes sequential next token prediction via transformer).
For claim 5, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi further discloses: wherein a representation is a sequential data structure (§3 ¶1-2: sequential transaction data, see also §1 ¶1-2 giving overview of sequential tabular time series data, hence, generating sequential representations, see fig.3), comprising:
a set of tokens comprising real values in the embedding space (§3 ¶1-2: plain text time series tabular data is quantized and fed into GPT encoder to generate sequential data in an embedding space); and
a set of sequential relations indicating a linear order among the set of tokens (fig.3 ¶2: prior seed data and autoregressively generated data are fed sequentially into the GPT model to generate further inferences, hence, sequential relations among tokens of the set and between fields).
For claim 6, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi further discloses: receiving a set of labelled candidate behaviors (§3 ¶1-2: credit card dataset with associated fields, see also §2.3 Transaction Dataset and §2.1 fig.1 showing exemplary data);
updating the first language processing model based on the set of labelled candidate behaviors to produce a set of labelled representations, wherein the set of labelled representations is associated with a set of outcomes (§3 ¶2: during training, generating synthetic data from the GPT model including first encoder model based on labeled behavior data set to produce likewise-labeled predicted rows, the predicted rows indicating purchase outcomes); and
using the set of labelled representations and the associated set of outcomes, training the second language processing model to predict outcomes based on input representations (ibid: the GPT model is trained to generate predictions based on generated input representations in an iterative process).
For claim 10, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi further discloses: obtaining a set of real behaviors associated with the first sequence of behavior tokens generated using the second language processing model (§3 ¶1-2, fig.3: during training, the GPT model weights are adjusted via comparison to real purchase behaviors in the training data set); and
using the set of real behaviors and the first sequences of behavior tokens, updating the second language processing model (ibid: GPT causal decoder is updated).
For claim 12, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi further discloses: wherein generating the first sequence of behavioral tokens comprises:
based on an input representation, using the second language processing model to generate a first set of candidate tokens, wherein each candidate token in the first set of candidate tokens is associated with a probability (§3 ¶1-2, fig.3: given the representation from the encoder embedding, the GPT model being a model that produces a set of likely outcomes at a softmax layer for each token, see Padhi §1 ¶2 disclosing use SOTA GPT model as found in reference Radford GPT-2, hence, Radford GPT-2, particularly §2.3 ¶1 (incorporating Radford GPT-1 as for architecture details) and Radford GPT-1 which are cited as incorporated references for disclosing overall GPT architecture, particularly §3.1 eq.2, fig.1 generating a softmax probability as output ); and
in response to selecting a candidate token from the first set of candidate tokens, recursively generating sets of candidate tokens based on the input representation and the selected candidate token (§3 ¶1-2: autoregressive training of GPT models based on encoder via selected candidate constitutes a recursive generation).
Claim 13 recites a computer readable media analogous to the method of claim 2 and is hence likewise rejected. Furthermore, Singh discloses: one or more non-transitory computer-readable media that, when executed by one or more processors, cause operations comprising (fig.7A:722, 726) the recited steps.
For claim 17, Padhi disclose the method of claim 13, as described above. Padhi further discloses: receiving a set of labelled scenario descriptions (§3 ¶1-2: credit card dataset with associated fields, see also §2.3 Transaction Dataset and §2.1 fig.1 showing exemplary data, hence, descriptions of purchase scenarios);
updating the first language processing model based on the set of labelled scenario descriptions to produce a set of labelled representations, wherein the set of labelled representations is associated with a set of outcomes (§3 ¶2: during training, generating synthetic data from the GPT model including first encoder model based on labeled behavior data set to produce likewise-labeled predicted rows, the predicted rows indicating purchase outcome); and
using the set of labelled representations and the associated set of outcomes, training the second language processing model to predict outcomes based on input representations (ibid: the GPT model is trained to generate predictions based on generated input representations in an iterative process).
Claims 15-16 recite computer media analogous to the above methods and are likewise rejected.
Claim(s) 3, 11, 14, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Padhi ("Tabular transformers for modeling multivariate time series", published 2021) in view of Arora ("Language models enable simple systems for generating structured views of heterogeneous data lakes", published 4/20/2023) in view of Dake ("University Students Behaviour Modelling Using the K‐Prototype Clustering Algorithm", published 8/7/2023).
For claim 3, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi modified by Arora does not disclose the limitations of claim 3.
Wang discloses: processing the second language processing model to generate a correspondence map, wherein the correspondence map correlates representations in the embedding space to plain text (p.5: Attention Guidance ¶2 discloses extracting attention matrix of a student model, the attention matrix being a correspondence map that correlates each representation with respective prior field tokes corresponding to plain text, see fig.1);
using the correspondence map, generating an alignment score for the first language processing model (Attention is aligned with the teacher and a KL divergence loss is calculated against a teacher benchmark, hence, alignment score ); and
based on the alignment score, updating the first language processing model (p.6 eq.12: the second model is updated based on the attention loss as a component of overall loss).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi by incorporating the attention distillation technique of Wang. Both concern the art of transformer training, and the incorporation would have, according to Wang, improve performance gap during transfer learning, i.e., such as when performing model updates, distilling a smaller model, etc., so that more of the teacher (the why or attention aspect) is transferred to the student (p.5 Attention Guidance ¶1), hence, improving machine learning efficiency (§1 ¶1).
For claim 11, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi modified by Arora further does not disclose the limitations of claim 11: using a prototype-based clustering model, processing the first or second sequence of behavioral tokens to determine clusters of user systems, wherein each first sequence of behavioral tokens is a prototype used by the prototype-based clustering model, and wherein each cluster of user systems is associated with a scenario description in first set of scenario descriptions.
Dake discloses: using a prototype-based clustering model, processing the first or second sequence of behavioral tokens to determine clusters of user systems (p.4 fig.1 gives overview of the system, with §3.4 giving an overview of the k-prototype clustering model), wherein each first element is a prototype used by the prototype-based clustering model (K-prototype algorithm (p.6 eq.2) clusters data based on a cluster centroids, these centroids being statistical incorporations of data elements, hence, each element being a prototype via incorporation of the clustering model; combination of Padhi’ s sequential tabular data yielding application to behavior sequences), and wherein each cluster of user systems is associated with a scenario description in first set of scenario descriptions (As each cluster is associated with a prototype, and each prototype is associated with features, each cluster of user systems is associated with a feature description as the average or mode scenario description of the cluster).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi modified by Arora by incorporating the prototype clustering technique of Dake. Both concern the art of human behavioral categorizing, and the incorporation would have, according to Dake, expose hidden patterns in user behavior for further action (p.2 col.2 ¶2).
Claim(s) 14 recite computer media analogous to the above methods and are likewise rejected.
For claim 20, Padhi modified by Arora disclose the method of claim 13, as described above. Padhi modified by Arora does not disclose the limitations of claim 20: using a prototype-based clustering model, processing first sequence(s) of behavioral tokens to determine clusters of user systems, wherein each first sequence of behavioral tokens is a prototype used by the prototype-based clustering model, and wherein each cluster of user systems is associated with a scenario description in first set of scenario descriptions.
Dake discloses: using a prototype-based clustering model, processing first sequence(s) of behavioral tokens to determine clusters of user systems (p.4 fig.1 gives overview of the system, with §3.4 giving an overview of the k-prototype clustering model), wherein each first sequence of behavioral tokens is a prototype used by the prototype-based clustering model (K-prototype algorithm (p.6 eq.2) clusters data based on a cluster centroids, these centroids being statistical incorporations of data elements, hence, each element being a prototype via incorporation of the clustering model; combination of Padhi’ s sequential tabular data yielding application to behavior sequences), and wherein each cluster of user systems is associated with a scenario description in first set of scenario descriptions (As each cluster is associated with a prototype, and each prototype is associated with features, each cluster of user systems is associated with a feature description as the average or mode scenario description of the cluster).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi modified by Arora by incorporating the prototype clustering technique of Dake. Both concern the art of human behavioral categorizing, and the incorporation would have, according to Dake, expose hidden patterns in user behavior for further action (p.2 col.2 ¶2).
Claim(s) 7, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Padhi ("Tabular transformers for modeling multivariate time series", published 2021) in view of Arora ("Language models enable simple systems for generating structured views of heterogeneous data lakes", published 4/20/2023) in view of Liu ("A survey on evolutionary neural architecture search", published 2021) in view of Bhanot ("The problem of fairness in synthetic healthcare data", published 2021).
For claim 7, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi further discloses: training a first candidate model for the second language processing model using a first algorithm (§3, fig.3: training a GPT causal decoder model).
Padhi modified by Arora does not disclose the remaining limitations:
generating a first fit metric associated with the first candidate model, wherein the first fit metric comprises a first bias score and a first error score;
training a second candidate model for the second language processing model using a second algorithm;
generating a second fit metric associated with the second candidate model, wherein the second fit metric comprises a second bias score and a second error score; and
using the first fit metric and the second fit metric, selecting parameters of the second language processing model through a weighted combination of the parameters of the first candidate model and the parameters of the second candidate model.
Liu discloses: generating a first fit metric associated with the first candidate model (§II.A gives overview of an evolutionary neural architecture search strategy, with §I eq.1 disclosing calculating performance parameters based on the training data, hence, generating fitness metric), wherein the first fit metric comprises a first error score (eq.1: error based on test data);
training a second candidate model for the second language processing model using a second algorithm (ibid: second candidate model is generated and its output evaluated for fitness via second algorithm, see §I eq.1 disclosing a performance evaluation based on the training data );
generating a second fit metric associated with the second candidate model (§I eq.1: fitness metric is evaluated), wherein the second fit metric comprises a second error score (eq.1: error based on test data); and
using the first fit metric and the second fit metric, selecting parameters of the second language processing model through a weighted combination of the parameters of the first candidate model and the parameters of the second candidate model (p.11 col.1 ¶1: weight inheritance between generations, such as based on selection strategies (see §V) constitutes selecting parameters of a second, child model by selectively taking a weighted (e.g., Boolean) combination of first parameter weights).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi modified by Arora by incorporating the neural architecture search technique of Liu. Both concern the art of neural network training, and the incorporation would have, according to Liu, obtain better architectures for better neural network processing (§1 ¶1).
Padhi modified by Arora modified by Liu does not disclose: wherein the first and second fitness metric comprise a respective bias score.
Bhanot discloses: wherein the first and second fitness metric comprise a respective bias score (Bhanot disclose the use of bias estimates for evaluating synthetic data, see §2.1-2.2, hence, combination with the fitness metric technique of Liu yielding a technique where bias scores are used as a secondary objective).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi modified by Liu by incorporating the bias scoring of Bhanot. Both concern the art of synthetic data generation and evaluation, and the incorporation would have, according to Bhanot, prevent inequities when using synthetic data (abs, p.3 ¶5)
Claim(s) 18 recite computer media analogous to the above methods and are likewise rejected.
Claim(s) 8 are rejected under 35 U.S.C. 103 as being unpatentable over Padhi ("Tabular transformers for modeling multivariate time series", published 2021) in view of Arora ("Language models enable simple systems for generating structured views of heterogeneous data lakes", published 4/20/2023) in view of Liu ("A survey on evolutionary neural architecture search", published 2021).
For claim 8, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi modified by Arora further discloses: training the second language processing model to be a first candidate model using a first algorithm, wherein the first candidate model generates a first output first sequence (§3, fig.3: training a GPT causal decoder model to generate an output sequence).
Padhi modified by Arora does not disclose the remaining limitations: using the set of representations and the first output first sequence, generating a first performance metric associated with the first candidate model;
training a second candidate model using a second algorithm, wherein the second candidate model generates a second output first sequence;
using the set of representations and the second output first sequence, generating a second performance metric associated with the second candidate model; and
using the first performance metric and the second performance metric, selecting the second language processing model to be one of the first candidate model and the second candidate model.
Liu discloses: using the set of representations and the first output first sequence, generating a first performance metric associated with the first candidate model (§II.A, fig.2: fitness of the various generated individuals is evaluated);
training a second candidate model using a second algorithm, wherein the second candidate model generates a second output first sequence (ibid: second candidate model is generated and its output evaluated for fitness, see §I eq.1 disclosing a performance evaluation based on the training data );
using the set of representations and the second output first sequence, generating a second performance metric associated with the second candidate model (ibid: fitness of additional models are evaluated based on output according to eq.1); and
using the first performance metric and the second performance metric, selecting the second language processing model to be one of the first candidate model and the second candidate model (fig.2, §II.A selection is made for further iterations until a stopping point is reached, see §V Population updating disclosing various strategies, e.g., elitism, discarding. ).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi modified by Arora by incorporating the neural architecture search technique of Liu. Both concern the art of neural network training, and the incorporation would have, according to Liu, obtain better architectures for better neural network processing (§1 ¶1).
Claim(s) 9, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Padhi ("Tabular transformers for modeling multivariate time series", published 2021) in view of Arora ("Language models enable simple systems for generating structured views of heterogeneous data lakes", published 4/20/2023) in view of Wang ("Attention distillation: self-supervised vision transformer students need more guidance", published 2022).
For claim 9, Padhi modified by Arora disclose the method of claim 2, as described above. Padhi modified by Arora does not disclose the limitations of claim 9.
Wang discloses: processing the second language processing model to extract an attention matrix, wherein the attention matrix is indicative of extents of consideration the second language processing model gives to each of its input features (p.5: Attention Guidance ¶2 discloses extracting attention matrix of a student model, the attention matrix being indicative of the relative attention for the input features);
comparing the attention matrix against a preset benchmark attention matrix to generate an attention score (Attention is aligned with the teacher sand a KL divergence loss is calculated against a teacher benchmark); and
updating the second language processing model based on the attention score (p.6 eq.12: the second model is updated based on the attention loss as a component of overall loss).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Padhi modified by Arora by incorporating the attention distillation technique of Wang. Both concern the art of transformer training, and the incorporation would have, according to Wang, improve performance gap during transfer learning, i.e., such as when performing model updates, distilling a smaller model, etc., so that more of the teacher (the why or attention aspect) is transferred to the student (p.5 Attention Guidance ¶1), hence, improving machine learning efficiency (§1 ¶1).
Claim(s) 19 recite computer media analogous to the above methods and are likewise rejected.
Response to Arguments
Applicant’s arguments have been fully considered. In the remarks, Applicant argued:
1. The art of record does not disclose the newly added limitations.
Applicant’s arguments have been fully considered but are moot in view of the newly cited art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Walters (US 20200012902 A1) discloses synthetic time-series data generation.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/LIANG LI/
Primary examiner AU 2143
Prosecution Insights