Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
The following action is in response to the communication(s) received on 03/29/2024.
As of the claims filed 03/29/2024:
Claims 1-20 are pending.
Claims 1, 11, and 18 are independent claims.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 04/05/2024, 04/28/2026, and 06/11/2026 were filed and in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites method, thus a process, one of the four statutory categories of patentable subject matter (Step 1). However, Claim 1 further recites:
vectorizing the sequential tabular data by embedding and concatenating vector fractions respectively corresponding to value fractions dv, positional fractions dₚ, and feature fractions df such that a representative tensor (B, S, D) is formed where B corresponds to a batch size of the sequential tabular data, S corresponds to a sequence length of the sequential tabular data, and D corresponds to an embedding dimension, wherein the continuous variables are zero padded to provide the value fraction dv, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites:
receiving sequential tabular data including continuous variables, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim 2, dependent on 1, further recites
D is represented by formula 1:
PNG
media_image1.png
22
178
media_image1.png
Greyscale
(1)., which is merely a detail of an abstract idea (vectorizing the sequential tabular data…D corresponds to an embedding dimension).
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 3, dependent on 1, further recites
the positional fractions dp are derived from tokenizing and vectorizing positional information of the sequential tabular data, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 4, dependent on 1, further recites
the feature fractions df are derived from tokenizing and vectorizing descriptive names associated with each column of the sequential tabular data, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 5, dependent on 1, further recite. no additional abstract ideas. However: Under Step 2A Prong 2, the claim recites:
embedding is provided with an existing embedding tool, as the performance of an abstract idea on a computer is not more than instructions to 'apply it' on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more. The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim 6, dependent on 1, further recite. no additional abstract ideas. However: Under Step 2A Prong 2, the claim recites:
embedding is provided by a learnable embedding layer, as the performance of an abstract idea on a computer is not more than instructions to 'apply it' on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more. The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim 7, dependent on 1, further recites
the vector fractions (dv, dp, df) are respectively weighted as (1/4, 1/4, 1/2), which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 8, dependent on 1, further recite. no additional abstract ideas. However: Under Step 2A Prong 2, the claim recites:
the sequential tabular data is feed to an attention- based neural network after vectorizing, which is merely an insignificant extra-solution activity of data transfer, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because the activity of data transfer (MPEP 2106.05(g)) cannot provide significantly more, as receiving or transmitting data over a network is well understood, routine, and conventional (MPEP 2106.05(d)(II)(i), buySAFE, Inc. v. Google, Inc). The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim 9, dependent on 1, further recites
determining an actuation signal from the regression-based prediction output,, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites:
passing the representative tensor to a plurality of attention layers to provide a regression-based prediction output, as the performance of an abstract idea on a computer is not more than instructions to 'apply it' on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application;
and controlling an actuator using the actuation signal, which is merely an insignificant extra-solution activity of controlling an actuator based on a prediction signal, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more; controlling an actuator based on a prediction signal (MPEP 2106.05(g)) cannot provide significantly more, as controlling an actuator is well understood, routine, and conventional (Wang et al., “Large Language Models for Robotics: Opportunities, Challenges, and Perspectives”, p.5 right ¶1: In addition to prompt methods, fine-tuning downstream tasks based on pre-trained LMs is also a common approach in the field of robot control.) (MPEP 2106.05(d)). The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim 10, dependent on 1, further recites
sequential tabular data includes categorical variables which are tokenized before vectorizing, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 11 recites a system, thus a machine, one of the four statutory categories of patentable subject matter (Step 1). However, Claim 11 further recites:
encode sequential tabular data to encoded data, the sequential tabular data including categorical data entries and continuous data entries, which is an evaluation or judgement that can be performed in the human mind;
the categorical data entries being tokenized and the continuous data entries being zero padded prior to vectorizing and embedding, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites:
for manufacturing data, which merely specifies the particular field of use or particular technological environment in which the abstract idea is to be performed, which by MPEP 2106.05(h) cannot integrate the abstract idea into a practical application;
non-transitory memory with computer-readable instruction, and a processor to execute the computer-readable instruction, the instruction operable to, as the performance of an abstract idea on a computer is not more than instructions to 'apply it' on a computer, which by MPEP 2106.05(f) cannot integrate an abstract idea into a practical application;
and feeding the encoded data to a transformer, which is merely an insignificant extra-solution activity of data transfer, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because the activity of data transfer (MPEP 2106.05(g)) cannot provide significantly more, as receiving or transmitting data over a network is well understood, routine, and conventional (MPEP 2106.05(d)(II)(i), buySAFE, Inc. v. Google, Inc); the particular field of use or particular technological environment (MPEP 2106.05(h)) cannot provide significantly more; implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more. The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim 12, dependent on 11, further recites
instructions are operable to perform a regression-based task, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 13, dependent on 12, further recites
the regression-based task is a prediction, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 14, dependent on 12, further recites
encoded data is represented by a tensor (B, S, D) where B is a batch size, S is a sequence length, and D is an embedding dimension, which is merely a detail of an abstract idea (encode sequential tabular data to encoded data).
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 15, dependent on 14, further recites
the embedding dimension D is derived from at least a first fraction corresponding to value embedding, and one or more additional fractions corresponding respectively to additional relational aspects, which is merely a detail of an abstract idea (encode sequential tabular data to encoded data… represented by a tensor (B,S,D)… D is an embedding dimension.).
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 16, dependent on 15, further recites
the first fraction and additional fractions are concatenated, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 17, dependent on 15, further recites
D is represented by formula (1):
PNG
media_image1.png
22
178
media_image1.png
Greyscale
(1), where dv corresponds to the value embedding fraction and di-dn correspond to the additional relational aspects, which is merely a detail of an abstract idea (encode sequential tabular data to encoded data… represented by a tensor (B,S,D)… D is an embedding dimension).
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 18 recites a method, thus a process, one of the four statutory categories of patentable subject matter (Step 1). However, Claim 18 further recites:
tokenizing and vectorizing the tabular data to vectorized data, the vectorized data comprised of a plurality of vector fractions, the continuous variables each being zero padded during vectorization, which is an evaluation or judgement that can be performed in the human mind;
embedding the vectorized data to provide embedded vectorized data, which is an evaluation or judgement that can be performed in the human mind;
and concatenating the embedded vectorized data, which is an evaluation or judgement that can be performed in the human mind.
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites:
receiving tabular data including categorical variables and continuous variables, which is merely an insignificant extra-solution activity of data gathering, which by MPEP 2106.05(g) cannot integrate an abstract idea into a practical application.
Thus, the claim is directed towards and abstract idea.
Further, the additional element(s), alone or in combination, do not provide significantly more than the abstract idea itself, because the activity of data gathering (MPEP 2106.05(g)) cannot provide significantly more, as storing and retrieving information in memory is well understood, routine, and conventional (MPEP 2106.05(d)(II)(iv)). The combination of these additional elements does not provide an inventive concept; thus, the claim remains ineligible.
Claim 19, dependent on 18, further recites
the plurality of vector fractions includes a first fraction directed to value embedding dv, a second fraction directed to positional embedding dp, and a third fraction directed to feature embedding df, which is merely a detail of an abstract idea (tokenizing and vectorizing the tabular data to vectorized data).
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim 20, dependent on 18, further recites
concatenating the embedded vectorized data is represented by a tensor (B, S, D) where B is a batch size, S is a sequence length, and D is an embedding dimension, which is represented by formula 1:
PNG
media_image2.png
22
178
media_image2.png
Greyscale
(1) where dv corresponds to a value embedding fraction and dᵢ-dₙ correspond to the additional relational aspects including dₚ which corresponds to a positional embedding fraction, and df which corresponds to feature-name embedding fraction., which is merely a detail of an abstract idea (concatenating the embedded vectorized data).
Thus, the claim recites an abstract idea under Step 2A Prong 1.Under Step 2A Prong 2, the claim recites no additional elements which could integrate the abstract idea into a practical application or provide significantly more than the abstract idea itself; thus, the claim remains ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-6, 8, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Yin et al., “TABERT: Pretraining for Joint Understanding of Textual and Tabular Data” (hereinafter Yin), in view of Ritze et al., “Web Data Commons - Web Table Corpora” (hereinafter Ritze), further in view of Somepalli et al., "SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training" (hereinafter Somepalli), further in view of Cristina," How to Implement Scaled Dot-Product Attention from Scratch in TensorFlow and Keras " (hereinafter Cristina).
Regarding Claim 1, Yin teaches:
A method of encoding data comprising: receiving sequential tabular data… (Yin [p.12 left ¶1] We collect parallel examples of tables and their surrounding NL sentences from two sources: … WDC WebTable Corpus… is a large collection of Web tables extracted from the Common Crawl Web scrape. We use its 2015 English-language relational subset, which consists of 50.8 million relational tables and their surrounding NL contexts.
[p.2 left ¶2] In this paper we present TABERT, a pretraining approach for joint understanding of NL text and (semi-)structured tabular data (§ 3). TABERT is built on top of BERT, and jointly learns contextual representations for utterances and the structured schema of DB tables (e.g., a vector for each utterance token and table column). Specifically, TABERT linearizes the structure of tables to be compatible with a Transformer-based BERT model.) (Note: the 2015 relational subset corresponds to the sequential tabular data)
…and vectorizing the sequential tabular data by embedding and concatenating vector fractions respectively corresponding to value fractions dv, positional fractions dp, and feature fractions df (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.
[p.12 right ¶2] Model We develop our text-to-SQL parser based on TranX, which translates an NL utterance into a tree-structured abstract meaning representation following user-specified grammar, before deterministically convert the generated abstract MR [meaning representation] into an SQL query. TranX models the construction process of an abstract MR (tree structured representation of an SQL query) using a transition-based system, which decomposes its generation story into a sequence of actions following the user defined grammar.
[p.5 right ¶2] Specifically, to predict a cell token si,j k ∈ si,j , its positional embedding ek and the cell representations si,j are fed into a two-layer network f(·) with GeLU activations.) (Note: the cell value corresponds to the value fraction dv; the positional embedding ek corresponds to the positional fraction dp; column name corresponds to the feature fraction df)
Yin does not explicitly recite that the sequential tabular data (via the WDC WebTable Corpus) includes continuous variables. However, Ritze further teaches:
…including continuous variables; (Ritze [p.4 fig.1 left]
PNG
media_image4.png
286
377
media_image4.png
Greyscale
) (Note: the relational table including area corresponds to the continuous variables)
Ritze and Yin are analogous to the present invention because both are from the same field of endeavor of utilizing tabular data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the continuous variables from the WDC dataset as explicitly recited in Ritze to Yin’s data encoding method, which utilizes the same WDC dataset. The motivation would be to “This page provides an overview of the corpora as well as their use cases” (Ritze, p.1 ¶1).
Yin/Ritze does not teach, but Somepalli further teaches:
such that a representative tensor (B, S, D) is formed where B corresponds to a batch size of the sequential tabular data, S corresponds to a sequence length of the sequential tabular data, and D corresponds to an embedding dimension, (Somepalli [p.5, alg.1, “x is bxnxd…”]
PNG
media_image5.png
710
1514
media_image5.png
Greyscale
) (Note: n increases the length of the input sequence as there are more features, thus corresponding to the sequence length)
Somepalli and Yin/Ritze are analogous to the present invention because both are from the same field of endeavor of applying attention-based neural network methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the representative tensor for input to the attention model from Somepalli into Yin/Ritze’s data encoding method. The motivation would be to “performs attention over both rows and columns, and it includes an enhanced embedding method.” (Somepalli, abstract).
Yin/Ritze/Somepalli does not teach, but Cristina further teaches:
wherein the continuous variables are zero padded to provide the value fraction dv. (Cristina [p.3 ¶5] Since the word embeddings are zero-padded to a specific sequence length, a padding mask needs to be introduced in order to prevent the zero tokens from being processed along with the input in both the encoder and decoder stages.)
Cristina and Yin/Ritze/Somepalli are analogous to the present invention because both are from the same field of endeavor of embedding variables for attention-based models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the zero padding for the input embeddings from Cristina into Yin/Ritze/Somepalli’s data encoding method. The motivation would be to “prevent the zero tokens from being processed along with the input in both the encoder and decoder stages” (Cristina [p.3 ¶5]).
Regarding Claim 2, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina, further teaches:
The method of claim 1, wherein D is represented by formula 1:
PNG
media_image1.png
22
178
media_image1.png
Greyscale
(1). (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.) (Note: Column name corresponds to an additional relational aspect in di.)
Regarding Claim 3, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina, further teaches:
The method of claim 1, wherein the positional fractions dₚ are derived from tokenizing and vectorizing positional information of the sequential tabular data. (Yin [p.5 right ¶2] Specifically, to predict a cell token si,j k ∈ si,j , its positional embedding ek and the cell representations si,j are fed into a two-layer network f(·) with GeLU activations.)
Regarding Claim 4, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina, further teaches:
The method of claim 1, wherein the feature fractions df are derived from tokenizing and vectorizing descriptive names associated with each column of the sequential tabular data. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a con catenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
) (Note: the column name corresponds to the feature fraction)
Regarding Claim 5, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina further teaches:
The method of claim 1, wherein embedding is provided with an existing embedding tool. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a con catenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
) (Note: the TABERT input linearization corresponds to an existing embedding tool)
Regarding Claim 6, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina, further teaches:
The method of claim 1, wherein embedding is provided by a learnable embedding layer. (Yin [p.3 fig.5]
PNG
media_image6.png
497
837
media_image6.png
Greyscale
) (Note: the per-row encoding corresponds to the learnable embedding layer)
7. The method of claim 1, wherein the vector fractions (dv, dp, df) are respectively weighted as (1/4, 1/4, 1/2).
No prior art found
Regarding Claim 8, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina, further teaches:
The method of claim 1, wherein the sequential tabular data is feed to an attention-based neural network after vectorizing. (Yin [p.3 fig.5]
PNG
media_image6.png
497
837
media_image6.png
Greyscale
) (Note: the per-row encoding corresponds to the learnable embedding layer (and thus the vectorization); the vertical self-attention layer corresponds to the attention-based neural network, which occurs after being fed the vectorized tabular data)
Regarding Claim 10, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina, further teaches:
The method of claim 1, wherein the sequential tabular data includes categorical variables which are tokenized before vectorizing. (Yin [p.3 fig.5 bottom right]
PNG
media_image7.png
370
624
media_image7.png
Greyscale
) (Note: ‘Venue… Erfurt’ in bottom right corresponds to a categorical variable, which Is vectorized in (B))
Yin/Ritze/Cristina
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Yin/Ritze/Somepalli/Cristina, in view of Shah et al., "US20210086353A1" (hereinafter Shah).
Regarding Claim 9, Yin/Ritze/Somepalli/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 1. Yin, via Yin/Ritze/Somepalli/Cristina, further teaches:
The method of claim 1, further comprising passing the representative tensor to a plurality of attention layers to provide a regression-based prediction output, (Yin [p.5 left ¶3] Given the column representation cj, TABERT is trained to predict the bag of masked (name and type) tokens from cj using a multi-label classification objective.) (Note: making a prediction is a regression-based task, thus corresponding to a regression-based prediction output)
Yin/Ritze/Somepalli/Cristina does not teach, but Shah further teaches:
determining an actuation signal from the regression-based prediction output, and controlling an actuator using the actuation signal. (Shah [0061] At block 216, the system controls actuator(s) of the robot to cause the robot to perform the action indicated by the action prediction output. For example, the system can control the actuator(s) to control the motion primitive indicated by the action prediction output.)
Shah and Yin/Ritze/Somepalli/Cristina are analogous to the present invention because both are from the same field of endeavor of attention-based models for making predictions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the actuator control based on the prediction output from Shah into Yin/Ritze/Somepalli/Cristina’s data encoder method. The motivation would be to “The free-form natural language input can direct the robot to accomplish a particular task, optionally with reference to one or more intermediary steps for accomplishing the particular task” (Shah, abstract).
Claims 11-13 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yin, in view of Ritze, further in view of Cristina.
Regarding Claim 11, Yin teaches:
A system for manufacturing data, the system comprising: non-transitory memory with computer-readable instruction, and a processor to execute the computer-readable instruction, the instruction operable to: (Yin [abstract] In this paper we present TABERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TABERT is trained on a large corpus of 26 million tables and their English contexts.) (Note: training a LM (language model) requires memory, computer-readable instruction, and a processor to execute the instruction)
encode sequential tabular data to encoded data, the sequential tabular data including
categorical data entries… (Yin [p.3 fig.5 bottom right]
PNG
media_image7.png
370
624
media_image7.png
Greyscale
) (Note: the input table corresponds to the sequential tabular data; the vertical self-attention layer corresponds to the encoded data; ‘Venue… Erfurt’ in bottom right corresponds to a categorical data entry)
…the categorical data entries being tokenized…prior to vectorizing and embedding; (Yin [p.3 fig.5 bottom right]
PNG
media_image7.png
370
624
media_image7.png
Greyscale
) (Note: ‘Venue… Erfurt’ in bottom right corresponds to a categorical variable, which Is vectorized in (B))
and feeding the encoded data to a transformer. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model.)
Yin does not explicitly recite that the sequential tabular data (via the WDC WebTable Corpus) includes continuous data entries. However, Ritze further teaches:
the sequential tabular data including… continuous data entries, (Ritze [p.4 fig.1 left]
PNG
media_image4.png
286
377
media_image4.png
Greyscale
) (Note: the relational table including area corresponds to the continuous variables; Lake name corresponds to categorical data)
Ritze and Yin are analogous to the present invention because both are from the same field of endeavor of utilizing tabular data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the continuous data entries from the WDC dataset as explicitly recited in Ritze to Yin’s data encoding method, which utilizes the same WDC dataset. The motivation would be to “This page provides an overview of the corpora as well as their use cases” (Ritze, p.1 ¶1).
Yin/Ritze does not teach, but Cristina further teaches:
and the continuous data entries being zero padded prior to vectorizing and embedding (Cristina [p.3 ¶5] Since the word embeddings are zero-padded to a specific sequence length, a padding mask needs to be introduced in order to prevent the zero tokens from being processed along with the input in both the encoder and decoder stages.) (Note: the encoder stage corresponds to the vectorizating and embedding)
Cristina and Yin/Ritze are analogous to the present invention because both are from the same field of endeavor of embedding variables for attention-based models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the zero padding for the input embeddings from Cristina into Yin/Ritze’s data encoding method. The motivation would be to “prevent the zero tokens from being processed along with the input in both the encoder and decoder stages” (Cristina [p.3 ¶5]).
Regarding Claim 12, Yin/Ritze/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 11. Yin, via Yin/Ritze/Cristina, further teaches:
The system of claim 11, wherein the instructions are operable to perform a regression-based task. (Yin [p.5 left ¶3] Given the column representation cj, TABERT is trained to predict the bag of masked (name and type) tokens from cj using a multi-label classification objective.) (Note: making a prediction corresponds to a regression-based task)
Regarding Claim 13, Yin/Ritze/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 12. Yin, via Yin/Ritze/Cristina, further teaches:
The system of claim 12, wherein the regression-based task is a prediction. (Yin [p.5 left ¶3] Given the column representation cj, TABERT is trained to predict the bag of masked (name and type) tokens from cj using a multi-label classification objective.) (Note: making a prediction corresponds to a regression-based task)
Regarding Claim 18, Yin teaches:
A method of encoding data, the method comprising: receiving tabular data including categorical variables… (Yin [p.12 left ¶1] We collect parallel examples of tables and their surrounding NL sentences from two sources: … WDC WebTable Corpus… is a large collection of Web tables extracted from the Common Crawl Web scrape. We use its 2015 English-language relational subset, which consists of 50.8 million relational tables and their surrounding NL contexts.
[p.3 fig.5 bottom right]
PNG
media_image7.png
370
624
media_image7.png
Greyscale
) (Note: ‘Venue… Erfurt’ in bottom right corresponds to a categorical variable; the 2015 relational subset corresponds to the sequential tabular data)
tokenizing and vectorizing the tabular data to vectorized data, the vectorized data comprised of a plurality of vector fractions, (Yin [p.3 fig.5 bottom right]
PNG
media_image7.png
370
624
media_image7.png
Greyscale
) (Note: ‘Venue… Erfurt’ in bottom right corresponds to a categorical variable, which Is vectorized in (B))
embedding the vectorized data to provide embedded vectorized data; and concatenating the embedded vectorized data. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.
[p.12 right ¶2] Model We develop our text-to-SQL parser based on TranX (Yin and Neubig, 2018), which translates an NL utterance into a tree-structured abstract meaning representation following user-specified grammar, before deterministically convert the generated abstract MR into an SQL query. TranX models the construction process of an abstract MR (tree structured representation of an SQL query) using a transition-based system, which decomposes its generation story into a sequence of actions following the user defined grammar.
[p.5 right ¶2] Specifically, to predict a cell token si,j k ∈ si,j , its positional embedding ek and the cell representations si,j are fed into a two-layer network f(·) with GeLU activations.)
Yin does not explicitly recite that the sequential tabular data (via the WDC WebTable Corpus) includes continuous variables. However, Ritze further teaches:
tabular data including… continuous variables; (Ritze [p.4 fig.1 left]
PNG
media_image4.png
286
377
media_image4.png
Greyscale
) (Note: the relational table including area corresponds to the continuous variables; Lake name corresponds to categorical data)
Ritze and Yin are analogous to the present invention because both are from the same field of endeavor of utilizing tabular data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the continuous variables from the WDC dataset as explicitly recited in Ritze to Yin’s data encoding method, which utilizes the same WDC dataset. The motivation would be to “This page provides an overview of the corpora as well as their use cases” (Ritze, p.1 ¶1).
Yin/Ritze does not teach, but Cristina further teaches:
the continuous variables each being zero padded during vectorization; (Cristina [p.3 ¶5] Since the word embeddings are zero-padded to a specific sequence length, a padding mask needs to be introduced in order to prevent the zero tokens from being processed along with the input in both the encoder and decoder stages.) (Note: introducing the padding mask in the encoder stage corresponds to zero padding the continuous variables during vectorization)
Cristina and Yin/Ritze are analogous to the present invention because both are from the same field of endeavor of embedding variables for attention-based models. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the zero padding for the input embeddings from Cristina into Yin/Ritze’s data encoding method. The motivation would be to “prevent the zero tokens from being processed along with the input in both the encoder and decoder stages” (Cristina [p.3 ¶5]).
Regarding Claim 19, Yin/Ritze/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 18. Yin, via Yin/Ritze/Cristina, further teaches:
The method of claim 18, wherein the plurality of vector fractions includes a first fraction directed to value embedding dv, a second fraction directed to positional embedding dp, and a third fraction directed to feature embedding df. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.
[p.12 right ¶2] Model We develop our text-to-SQL parser based on TranX (Yin and Neubig, 2018), which translates an NL utterance into a tree-structured abstract meaning representation following user-specified grammar, before deterministically convert the generated abstract MR into an SQL query. TranX models the construction process of an abstract MR (tree structured representation of an SQL query) using a transition-based system, which decomposes its generation story into a sequence of actions following the user defined grammar.
[p.5 right ¶2] Specifically, to predict a cell token si,j k ∈ si,j , its positional embedding ek and the cell representations si,j are fed into a two-layer network f(·) with GeLU activations.)
Regarding Claim 20, Yin/Ritze/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 19. Yin/Ritze/Cristina does not teach, but Somepalli further teaches:
The method of claim 19, wherein concatenating the embedded vectorized data is represented by a tensor (B, S, D) where B is a batch size, S is a sequence length, and D is an embedding dimension, (Somepalli [p.5, alg.1, “x is bxnxd…”]
PNG
media_image5.png
710
1514
media_image5.png
Greyscale
) (Note: n increases the length of the input sequence as there are more features, thus corresponding to the sequence length)
Somepalli and Yin/Ritze/Cristina are analogous to the present invention because both are from the same field of endeavor of applying attention-based neural network methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the representative tensor for input to the attention model from Somepalli into Yin/Ritze/Cristina’s data encoding method. The motivation would be to “performs attention over both rows and columns, and it includes an enhanced embedding method.” (Somepalli, abstract).
Yin, via Yin/Ritze/Cristina/Somepalli, further teaches:
which is represented by formula 1: (1) where dv corresponds to a value embedding fraction and dᵢ-dₙ correspond to the additional relational aspects including dₚ which corresponds to a positional embedding fraction, and df which corresponds to feature-name embedding fraction. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.
[p.12 right ¶2] Model We develop our text-to-SQL parser based on TranX (Yin and Neubig, 2018), which translates an NL utterance into a tree-structured abstract meaning representation following user-specified grammar, before deterministically convert the generated abstract MR into an SQL query. TranX models the construction process of an abstract MR (tree structured representation of an SQL query) using a transition-based system, which decomposes its generation story into a sequence of actions following the user defined grammar.
[p.5 right ¶2] Specifically, to predict a cell token si,j k ∈ si,j , its positional embedding ek and the cell representations si,j are fed into a two-layer network f(·) with GeLU activations.)
Claims 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Yin/Ritze/Cristina, in view of Somepalli.
Regarding Claim 14, Yin/Ritze/Cristina respectively teaches and incorporates the claimed limitations and rejections of Claim 12. Yin/Ritze/Cristina does not teach, but Somepalli further teaches:
The system of claim 12, wherein encoded data is represented by a tensor (B, S, D) where B is a batch size, S is a sequence length, and D is an embedding dimension. (Somepalli [p.5, alg.1, “x is bxnxd…”]
PNG
media_image5.png
710
1514
media_image5.png
Greyscale
) (Note: n increases the length of the input sequence as there are more features, thus corresponding to the sequence length)
Somepalli and Yin/Ritze/Cristina are analogous to the present invention because both are from the same field of endeavor of applying attention-based neural network methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the representative tensor for input to the attention model from Somepalli into Yin/Ritze/Cristina’s data encoding method. The motivation would be to “performs attention over both rows and columns, and it includes an enhanced embedding method.” (Somepalli, abstract).
Regarding Claim 15, Yin/Ritze/Cristina/Somepalli respectively teaches and incorporates the claimed limitations and rejections of Claim 14. Yin, via Yin/Ritze/Cristina/Somepalli, further teaches:
The system of claim 14, wherein the embedding dimension D is derived from at least a first fraction corresponding to value embedding, and one or more additional fractions corresponding respectively to additional relational aspects. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.) (Note: Column name corresponds to an additional relational aspect)
Regarding Claim 16, Yin/Ritze/Cristina/Somepalli respectively teaches and incorporates the claimed limitations and rejections of Claim 15. Yin, via Yin/Ritze/Cristina/Somepalli, further teaches:
The system of claim 15, wherein the first fraction and additional fractions are concatenated. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.)
Regarding Claim 17, Yin/Ritze/Cristina/Somepalli respectively teaches and incorporates the claimed limitations and rejections of Claim 15. Yin, via Yin/Ritze/Cristina/Somepalli, further teaches:
The system of claim 15, wherein D is represented by formula (1):
PNG
media_image1.png
22
178
media_image1.png
Greyscale
(1), where dv corresponds to the value embedding fraction and dᵢ-dₙ correspond to the additional relational aspects. (Yin [p.4 left ¶2] Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R2, which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s2,1 valued 2005 in R2 in Fig. 1 is encoded as
PNG
media_image3.png
61
398
media_image3.png
Greyscale
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.) (Note: Column name corresponds to an additional relational aspect)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEP HAN whose telephone number is (703)756-1346. The examiner can normally be reached Mon-Fri 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached on (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.H./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights