DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 1/7/2026 has been entered.
Response to Arguments
Applicant's arguments filed 3/10/2026 have been fully considered
35 USC § 101: These issues have been not resolved and the rejection has not been withdrawn in light of the amendments and arguments. The new amendment including “obtaining data samples for the relevant tables and/or views subsequent to the relevant tables and/or views being determined, wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views;” of the recited information in a database is insignificant extra-solution activity under MPEP 2106.05(f). Accordingly, these additional elements also do not integrate the abstract idea into a practical application. Looking at the additional elements as a combination and the claim as a whole, they still do not provide any integration into a practical application. The additional elements of “obtaining data samples for the relevant tables and/or views subsequent to the relevant tables and/or views being determined, wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views” information in a database which were indicated as insignificant extra-solution activity, are still insignificant when re-evaluated as these are well-understood, routine, and conventional (WURC). See MPEP 2106.05(d)(II) for evidence of these operations being court-identified as WURC provided at least by “iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93” and/or “i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 … OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network).”Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea above). The amendment “wherein the augmented text is based on the original query” encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. Please see 101 rejection below for more details.
35 USC § 102/35 USC § 103:
Regarding Applicant’s Argument (pages: 8-12): Examiner’s response:- Applicant’s arguments with respect to the rejection(s) of under 35 USC § 102/103 have been fully considered, upon further consideration, a new ground(s) of rejection is made in view of US 20240354309 A1; SACHETI; Arun Kumar et al. (hereinafter Sacheti). As cited below Sacheti teaches new amendments "wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views". It is important to note that this rejection is one of obviousness and not one of anticipation, hence elements from one art can be combined into a foundation of another separate art and there can be obviousness conclusions reached in the mapping and teaching of the prior arts inventions into the instant applications claim limitations. Regarding limitation "wherein the augmented text is based on the original query" The examiner points to figure 2 in conjunction to figure 5 of Singh along side paragraph 59 &71-72 which show the revisions(augmentations) to the original query. The language "based on the original query" is interpreted under Broadest reasonable interpretation as any sort of connection or link to the original query, and as shown from figure 2 in conjunction to figure 5 of Singh along side paragraph 59 &71-72 the steps of revision’s/augmentations are performing in the same flow of connection a query/multiple queries in which any of each can be interpretated as the "original query".
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. Independent Claim(s) 1,12, and 20 are directed to the abstract idea of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context.
Regarding claim 1: the limitation of “A method for facilitating usage of natural language for cyber asset management, the method comprising: determining, using a … in a first pass, relevant tables and/or views based on an original query, the original query being a natural language query received from a user…preparing an augmented context, the augmented context including the data samples for the relevant tables and/or views, wherein the augmented text is based on the original query; generating … in a second pass subsequent to the first pass, a generated query based on the augmented context; and generating a response for the user based on the generated query, the response being in natural language” as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting “large language model (LLM) electronically” to perform the ‘Determining, preparing and generating’ operations nothing in the claim element precludes the step from practically being performed in the mind. For example, these ‘Determining, preparing and generating’ limitations in the context of this claim encompasses the mental process a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context.
This judicial exception is not integrated into a practical application. In particular, claim 1 only recites the additional elements of: “a large language model (LLM)”. The “large language model (LLM)” is recited at a high-level of generality (i.e., as a generic computer components performing a generic computer functions) such that it amounts to no more than mere instructions to apply the exception using a computer under MPEP 2106.05(f). Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
The “obtaining data samples for the relevant tables and/or views subsequent to the relevant tables and/or views being determined, wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views” of the recited information in a database is insignificant extra-solution activity under MPEP 2106.05(f). Accordingly, these additional elements also do not integrate the abstract idea into a practical application. Looking at the additional elements as a combination and the claim as a whole, they still do not provide any integration into a practical application. The additional elements of “obtaining data samples for the relevant tables and/or views subsequent to the relevant tables and/or views being determined, wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views” information in a database which were indicated as insignificant extra-solution activity, are still insignificant when re-evaluated as these are well-understood, routine, and conventional (WURC). See MPEP 2106.05(d)(II) for evidence of these operations being court-identified as WURC provided at least by “iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93” and/or “i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 … OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network).”Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea above). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea.
Corresponding system claim 12 is rejected similarly as claim 1 above. Contain additional elements “A system configured to facilitate usage of natural language for cyber asset management, the system comprising: a modeler… a data sampler… an augmented context preparer… a response generator” which is recited at a high-level of generality (i.e., as a generic computer components performing a generic computer functions) such that it amounts to no more than mere instructions to apply the exception using a computer under MPEP 2106.05(f). Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Corresponding product claim 20 is rejected similarly as claim 1 above. Claim 20 contain additional elements “a non-transitory computer readable storage medium storing a program for causing a computer to execute” which is recited at a high-level of generality (i.e., as a generic computer components performing a generic computer functions) such that it amounts to no more than mere instructions to apply the exception using a computer under MPEP 2106.05(f). Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Claim 2 contains the same mental process and abstract idea as claim 1 above rejects. Claim 2 recites “wherein the relevant tables and/or views are SQL tables and/or views, and/or wherein the generated query is an SQL query” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. Additional element of “is an SQL query” can mean the human just changes the format of the search strategy. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Corresponding system claim 13 is rejected similarly as claim 2 above. Claim 3 contains the same mental process and abstract idea as claim 1 above rejects. Claim 3 recites “wherein generating the response comprises: processing, using a query engine, the generated query to retrieve query results; and converting, using a natural language converter, the query results to the response” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. Additional elements related to converting the query results, can be the user just changes the format of the final search conclusion. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Corresponding system claim 14 is rejected similarly as claim 3 above.
Claim 4 contains the same mental process and abstract idea as claim 1 above rejects. Claim 4 recites “wherein the query engine is an SQL query engine or an application programming interface (API) engine” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements of an SQL query engine or an application programming interface (API) is recited at a high-level of generality (i.e., as a generic computer components performing a generic computer functions) such that it amounts to no more than mere instructions to apply the exception using a computer under MPEP 2106.05(f). Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Claim 5 contains the same mental process and abstract idea as claim 1 above rejects. Claim 5 recites “wherein the augmented context includes the original query” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements of the augmented context including the original query still is being interpreted as the same mental process of developing the search strategy with certain specific context/criteria. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Corresponding system claim 15 is rejected similarly as claim 5 above. Claim 6 contains the same mental process and abstract idea as claim 1 above rejects. Claim 6 recites “wherein the relevant tables and/or views includes tables and/or views and/or endpoints necessary to answer the original query” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements of wherein the relevant tables and/or views includes tables and/or views and/or endpoints necessary to answer the original query is being interpreted as the same mental process of developing the search strategy with certain specific context/criteria. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Claim 7 contains the same mental process and abstract idea as claim 1 above rejects. Claim 7 recites “finding other query examples relevant to the original query, wherein the augmented context also includes the other query examples” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements of finding other query examples relevant to the original query, wherein the augmented context also includes the other query examples of the recited information is insignificant extra-solution activity under MPEP 2106.05(f). Accordingly, these additional elements also do not integrate the abstract idea into a practical application. Looking at the additional elements as a combination and the claim as a whole, they still do not provide any integration into a practical application. The additional elements of “obtaining data samples for the relevant tables and/or views subsequent to the relevant tables and/or views being finding other query examples relevant to the original query, wherein the augmented context also includes the other query examples information which were indicated as insignificant extra-solution activity, are still insignificant when re-evaluated as these are well-understood, routine, and conventional (WURC). See MPEP 2106.05(d)(II) for evidence of these operations being court-identified as WURC provided at least by “iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93” and/or “i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 … OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Corresponding system claim 16 is rejected similarly as claim 7 above. Claim 8 contains the same mental process and abstract idea as claim 1 above rejects. Claim 8 recites “wherein finding the other query examples comprises: transforming the original query into a query vector of one or more numbers; obtaining a query list comprising n queries closest in distance to the original query, n >= 1, distances being calculated based on the query vector of the original vector and query vectors associated with each query of the query list; and adding one or more queries of the query list into the augmented context” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements are merely performing mathematical calculations which still encompasses the same mental process of developing a search strategy. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Corresponding system claim 17 is rejected similarly as claim 8 above. Claim 9 contains the same mental process and abstract idea as claim 1 above rejects. Claim 9 recites “wherein the transforming the original query into the query vector comprises transforming the original query based on any one or more of embeddings, simhash, and ngrams” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements are merely performing mathematical calculations which still encompasses the same mental process of developing a search strategy. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Corresponding system claim 18 is rejected similarly as claim 9 above. Claim 10 contains the same mental process and abstract idea as claim 1 above rejects. Claim 10 recites “The method of claim 8, wherein finding the other query examples further comprises: subsequent to obtaining the query list and prior to adding the one or more queries of the query list, removing x queries from the query list, x => 1, the removed queries having x lowest entropies among the n queries of the query list” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements are merely performing mathematical calculations which still encompasses the same mental process of developing a search strategy. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Corresponding system claim 19 is rejected similarly as claim 10 above. Claim 11 contains the same mental process and abstract idea as claim 1 above rejects. Claim 11 recites “wherein removing the x queries is performed based on a pairwise comparison among the n queries of the query list” which is rejected similarly as claim 1 above and in the context of this claim encompasses the mental process of a human looking through files of data, using a search strategy in order to obtain samples for relevant tables in the data, preparing new context based on the samples and creating a new search strategy based on new context. The additional elements are merely performing mathematical calculations which still encompasses the same mental process of developing a search strategy. The additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. The elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Claim(s) 1-20 are therefore not drawn to eligible subject matter as they are directed to an abstract idea without significantly more.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 20210397610 A1; SINGH; Pranav et al. (hereinafter Singh) in view of US 20240354309 A1; SACHETI; Arun Kumar et al. (hereinafter Sacheti).
Regarding claim 1, Singh teaches A method for facilitating usage of natural language for cyber asset management, the method comprising: determining, using a large language model (LLM) electronically in a first pass, relevant tables and/or views based on an original query (Singh [0006] machine learning system for use with a digital assistant, the method comprising: obtaining training data comprising query data samples; obtaining vector representations of the query data [0008] obtaining the training data comprises obtaining pairs of text data representing queries and responses, and obtaining vector representations of the query data samples comprises converting the pairs of text data to corresponding vector representations. Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system.[0013] an encoder neural network architecture similar to, or even a duplication of, the encoder portion of the encoder-decoder neural network architecture [0040] The server device 120 in FIG. 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query placed by a user, e.g. either by voice following speech-to-text processing or by text entry. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data following receipt at the query processor 124. The query data 140 may also have associated metadata such as conversation state, user location information, and other user profile information. The query processor 124 is configured to receive the query data 140 and compute response data 142 to provide a response to the query. [0055] This may comprise clustering the vector representations within vector space, i.e. identifying clusters based on one or more clustering functions. The clustering functions may be provided by machine learning libraries such as sci-kit learn. It should be noted that clustering is typically performed in vector spaces with hundreds of dimensions (e.g., corresponding to hundreds of vector elements)—this is typically difficult to visualize in a world of 2 or 3 dimensions. In one case, a hierarchical clustering function may be used to iteratively define clusters. In one case, the vector representations may comprise vector representations of query data and vector representations of accompanying response data [0066] vectors in the clusters identified; the query indexes that correspond to these response points or vectors (e.g., response vector i and corresponding query i from the data set 510) are then grouped as per the response clusters. In certain cases, the query vectors within each group may themselves be clustered. This may be used to select one of the group as the canonical query for the group, e.g. based on a distance to the centroid of the query cluster [77-82] further elaborate on the matter [FIG.2 in conjunction with FIG.5] shows determining, using a large language model (LLM) in a first pass, relevant tables and/or views based on an original query) the original query being a natural language query received from a user; (Singh [FIG.2] shows the original query being a natural language query received from a user [0008] In certain examples, obtaining the training data comprises obtaining pairs of text data representing queries and responses, and obtaining vector representations of the query data samples comprises converting the pairs of text data to corresponding vector representations. Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system.[0040] The server device 120 in FIG. 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query placed by a user, e.g. either by voice following speech-to-text processing or by text entry. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data following receipt at the query processor 124. The query data 140 may also have associated metadata such as conversation state, user location information, and other user profile information. The query processor 124 is configured to receive the query data 140 and compute response data 142 to provide a response to the query. [0042] FIG. 2 shows an example 200 of a digital voice assistant 220, which may be implemented in one or more of the client devices 110 in FIG. 1...) obtaining data samples for the relevant tables and/or views subsequent to the relevant tables and/or views being determined; (Singh [0006] obtaining training data comprising query data samples; obtaining vector representations of the query data samples; clustering the vector representations; determining canonical queries and corresponding query groups based on the clustered vector representations, wherein corresponding query groups correspond to determined canonical queries; generating paired data samples based on determined canonical queries and selections from the corresponding query groups; and training an encoder-decoder neural network architecture using the paired data samples [0008] Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system. [0054] At block 410, vector representations of the query data samples are obtained. This may comprise obtaining a vector encoding of one or more of text data representing a query and text data representing a corresponding response. The vector encoding may be generated by a neural network encoder. [0056] At block 420, canonical queries and corresponding query groups are determined based on the clustered vector representations. This may comprise selecting clusters of vector representations relating to different queries, e.g. as determined in block 415, and determining one representative query data sample for each cluster. For example, a representative (canonical) query data sample may comprise a more frequent query data sample within the cluster and/or have a vector representation [0070] In one variation, the method 400 of FIG. 4 and/or the process illustrated in FIG. 5, further comprises filtering generated paired data samples, such as the target-source data samples [0072] augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities [69-73] elaborate on the matter) wherein the augmented text is based on the original query; (Singh [FIG.2 in conjunction with FIG.5] shows wherein the augmented text is based on the original query [0059] A digital assistant is configured to use the trained encoder-decoder neural network architecture to map data representing an initial query to data representing a revised query associated with a canonical query, for example as described with reference to FIGS. 2 and 3. The data representing the revised query that is output by the neural network architecture may be further processed to provide a response to the initial query, e.g. as described with reference to FIGS. 1 and 2. By suppling the source and target data samples as generated from the clusters in block 425, the neural network architecture “learns” to map non-standard sequences of tokens to standardized or canonical sequences of tokens that reflect better formed input for successful and accurate query [0071] the query data 512 and the response data 514 prior to vectorization and clustering. In this case, the method 400 may comprise replacing the text data for tagged named entities with a named entity type tag, such as respectively replacing the detected names of people, places and times with the defined tags <PEOPLE>, <PLACE>, and <TIME>. This helps cluster similar queries with different named entities together, e.g. the text strings “What is the weather in Toronto?” and “What is the weather in San Diego?” may both be converted into the token sequences processing.[0072] augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample [79-85] elaborate on the matter of the augmented text is based on the original query) preparing an augmented context, the augmented context including the data samples for the relevant tables and/or views; (Singh [0006] obtaining training data comprising query data samples; obtaining vector representations of the query data samples; clustering the vector representations; determining canonical queries and corresponding query groups based on the clustered vector representations, wherein corresponding query groups correspond to determined canonical queries; generating paired data samples based on determined canonical queries and selections from the corresponding query groups; and training an encoder-decoder neural network architecture using the paired data samples [0008] Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system. [0054] At block 410, vector representations of the query data samples are obtained. This may comprise obtaining a vector encoding of one or more of text data representing a query and text data representing a corresponding response. The vector encoding may be generated by a neural network encoder. [0056] At block 420, canonical queries and corresponding query groups are determined based on the clustered vector representations. This may comprise selecting clusters of vector representations relating to different queries, e.g. as determined in block 415, and determining one representative query data sample for each cluster. For example, a representative (canonical) query data sample may comprise a more frequent query data sample within the cluster and/or have a vector representation [0070] In one variation, the method 400 of FIG. 4 and/or the process illustrated in FIG. 5, further comprises filtering generated paired data samples, such as the target-source data samples [0072] augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities [69-73] elaborate on the matter) generating, using the LLM electronically in a second pass subsequent to the first pass, a generated query based on the augmented context; (Singh [FIG.3 in conjunction with FIG.5] shows generating, using the LLM in a second pass subsequent to the first pass, a generated query based on the augmented context [0006] determining canonical queries and corresponding query groups based on the clustered vector representations, wherein corresponding query groups correspond to determined canonical queries; generating paired data samples based on determined canonical queries and selections from the corresponding query groups; and training an encoder-decoder neural network architecture using the paired data samples, wherein the selections from the corresponding query groups are supplied as input sequence data and the determined canonical queries are supplied as output sequence data, wherein the digital assistant is configured to map data representing an initial query to data representing a revised query associated with one of the canonical queries, via the encoder-decoder neural network architecture, the data representing the revised query being further processed to provide a response to the initial query [0015] a second stage of clustering on the queries themselves, query groups that may be easily mapped onto canonical queries are generated [0057] At block 425, paired data samples are generated based on selected canonical queries and selections from the query group corresponding to each selected canonical query. For example, for each cluster, the canonical query (i.e., the data representation of said query in text or sequence form) may be set as a target or ground truth output data and each query in turn within the cluster may be paired with this canonical query as source input data. [0067] This filtering may thus be seen as a second stage of clustering based on the query vectors 522 as opposed to the response vectors 524... [87-92] further elaborate on the matter) and generating a response for the user based on the generated query, the response being in natural language. (Singh [0042] FIG. 2 shows an example 200 of a digital voice assistant 220, which may be implemented in one or more of the client devices 110 in FIG. 1. The digital voice assistant 200 is configured to receive audio input data 220 and generate a response 230 in one or more configured modalities. In the example, of FIG. 2, the digital voice assistant 220 may be configured to supply a response to a query as one or more of an audio output 232 (e.g., a voice response), a text response 234 and/or a visual response 236, such as one or more frames of video. Different modalities may be provided depending on the implementation of the digital voice assistant 220. In the example 200 of FIG. 2, the digital voice assistant 220 comprise an audio capture device 242, a speech-to-text system 244, a neural network pre-processor 246, a query processor 248 and a response interface 250.[0044] The speech-to-text system 244 receives audio data and translates this into a text data output. For example, the speech-to-text system 244 may comprise an automated transcription system that is based on one or more of hidden Markov models and neural network architectures (e.g., convolutional and recurrent neural network encoders). In this example, the speech-to-text system 244 provides a text data output for the neural network pre-processor 246 [45] The query processor 248 may output a response as pre-rendered data (e.g., an audio and/or video sample) or the response interface 250 may comprise a text-to-modality system, such as a text-to-speech or text-to-video system, or other vectorized information representation system, such as Musical Instrument Digital Interface (MIDI) music, Scalable Vector Graphics (SVG) graphics, or Flash animations, for providing the response 230. [78-83] elaborate on the matter [FIG.2] shows generating a response for the user based on the generated query, the response being in natural language)
Singh lacks explicitly and orderly teaching wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views; However Sacheti teaches wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views; (Sacheti [0030] a table and populate the table with values of attributes of entities that are relevant to the query, images that are relevant to the query, and so forth. Further, the query classifier 126 can include instructions in the prompt that cause the GLM 112 to provide interactive graphics that represent ways in which the table can be modified, such as identities of attributes not represented in the table (e.g., due to screen size constraints), entities not represented in the table (e.g., due to screen size constraints), and so forth. Thus, based upon the prompt, the GLM 112 can generate and display a table showing multiple entities meeting the criterion and values of attributes of those entities. The user can then provide additional interaction with respect to the interactive graphic elements, causing the GLM 112 to present an updated table (e.g., with additional values of attributes included, with values for attributes removed, and so forth). Still further, the GLM 112 can update a table based upon further queries from the user, such as “I want it to be a hybrid SUV.” In this example, the GLM 112 updates the table to exclude vehicles that are not hybrid SUVs (and have a cost under $30,000). Operation of the GLM 112 is described in further detail below. [0033] a table and populate the table with attributes of entities identified by the search system 110 and provided to the GLM 112 by the search system 110. Further, the prompt information can include instructions to the GLM 112 to generate interactive graphical elements that can be interacted with to allow for relatively quick modification of content of a table constructed by the GLM 112. Examples are presented below [37-38] elaborate on the matter [FIG.1 in conjunction with FIG.3] shows visual of overall system wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to take all prior methods and make the addition of Sacheti in order to efficiently generate and populate a table in a database system (Sacheti [0004] generating, and/or displaying appropriate information in an efficient manner in response to certain types of user input. [0006] facilitate providing a conversational chat-based interface that includes a table that includes cells that comprise content, where the table is constructed by the GLM. For instance, a table can include attributes of entities[0029] As discussed above, operation of the search engine 110 is improved based upon the GLM 112, and operation of the GLM 112 is improved based upon the search engine 110. For instance, the search engine 110 is able to provide outputs that the search engine 110 was not previously able to provide (e.g., based upon outputs generated by the GLM 112), and the GLM 112 is improved by using information obtained by the search engine 110 to generate outputs (e.g., information identified by the search engine 110 can be included as a portion of a prompt used by the GLM 112 to generate outputs). Specifically, the GLM 112 generates results based upon information obtained by the search engine 110 that have a higher likelihood of being accurate when compared to results generated by GLMs 112 that are not based upon such information, as the search engine 110 is associated with years of design to curate information sources to ensure accuracy thereof.[0041] The communications diagram 200 illustrates various advantages of the computing system 100 over conventional search systems...[FIG.1 in conjunction with FIG.3] shows visual of overall system wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views)
Corresponding product claim 20 is rejected similarly as claim 1 above. Additional Limitations: A computer-readable medium storing computer-executable instructions, the stored computer-executable instructions configured to cause one or more processors (Singh [FIG. 7B] shows a computer-readable medium storing computer-executable instructions, the stored computer-executable instructions configured to cause one or more processors [86 & 97] shows a computer-readable medium storing computer-executable instructions, the stored computer-executable instructions configured to cause one or more processors )
Regarding claim 2, Singh and Sacheti teach The method of claim 1, wherein the relevant tables and/or views are SQL tables and/or views, and/or wherein the generated query is an SQL query. (Singh [0006] determining canonical queries and corresponding query groups based on the clustered vector representations, wherein corresponding query groups correspond to determined canonical queries; generating paired data samples based on determined canonical queries and selections from the corresponding query groups; and training an encoder-decoder neural network architecture using the paired data samples, wherein the selections from the corresponding query groups are supplied as input sequence data and the determined canonical queries are supplied as output sequence data, wherein the digital assistant is configured to map data representing an initial query to data representing a revised query associated with one of the canonical queries, via the encoder-decoder neural network architecture, the data representing the revised query being further processed to provide a response to the initial query [0015] a second stage of clustering on the queries themselves, query groups that may be easily mapped onto canonical queries are generated [0057] At block 425, paired data samples are generated based on selected canonical queries and selections from the query group corresponding to each selected canonical query. For example, for each cluster, the canonical query (i.e., the data representation of said query in text or sequence form) may be set as a target or ground truth output data and each query in turn within the cluster may be paired with this canonical query as source input data. [0067] This filtering may thus be seen as a second stage of clustering based on the query vectors 522 as opposed to the response vectors 524... [87-92] further elaborate on the matter [FIG.3 in conjunction with FIG.5] shows wherein the relevant tables and/or views are SQL tables and/or views, and/or wherein the generated query is an SQL query)
Regarding claim 3, Singh and Sacheti teach The method of claim 1, wherein generating the response comprises: processing, using a query engine, the generated query to retrieve query results; and converting, using a natural language converter, the query results to the response. (Singh [0010] a first portion of text data represents an output of speech-to-text processing that is performed on audio data for a voice query and a second portion of text data comprises data for use in providing the response to the voice query, wherein the digital assistant is configured to map data representing an initial voice query to data representing a revised voice query via the trained neural machine learning model. The present disclosure is of particular benefit for voice applications, where there is a high chance of malformed input queries due to the inherent noise present in both the real-world and in speech-to-text systems. [0040] The server device 120 in FIG. 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query placed by a user, e.g. either by voice following speech-to-text processing or by text entry. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data following receipt at the query processor 124. The query data 140 may also have associated metadata such as conversation state, user location information, and other user profile information. The query processor 124 is configured to receive the query data 140 and compute response data 142 to provide a response to the query. The response data 142 may comprise text data and/or data derived from a response representation that is used to generate one or more of a text response, a voice response (e.g., via a text-to-speech system) and a visual response (e.g., via one or more display devices). For example, a user may make a request for information such as “What is the traffic like on Route 48?”, “What is the weather like today?”, or “Where can I buy the best vegan taco?” and the response data 140 may be a formatted response[0045] In FIG. 2, the query processor 248 may operate in a similar way to the query processor 124 of FIG. 1. The query processor 248 receives the mapped text data from the neural network pre-processor 246 and generates data useable to derive a response to the original query. For example, this may comprise text data that is provided to the response interface 250 for rendering into a configured and/or available modality. The query processor 248 may perform local processing to provide the response and/or make a network request, such as is illustrated in FIG. 1, to obtain data to provide the response. The query processor 248 may output a response as pre-rendered data (e.g., an audio and/or video sample) or the response interface 250 may comprise a text-to-modality system, such as a text-to-speech or text-to-video system [79-81] elaborate on processing, using a query engine, the generated query to retrieve query results and converting, using a natural language converter, the query results to the response. [FIG.2] shows using a query engine, the generated query to retrieve query results and converting, using a natural language converter, the query results to the response )
Regarding claim 4, Singh and Sacheti teach The method of claim 3, wherein the query engine is an SQL query engine or an application programming interface (API) engine. (Singh [0025] FIG. 1 is a schematic illustration showing a query processor for a digital assistant according to an example [0040] The server device 120 in FIG. 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query placed by a user, e.g. either by voice following speech-to-text processing or by text entry. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data following receipt at the query processor 124. The query data 140 may also have associated metadata such as conversation state, user location information, and other user profile information. The query processor...[0045] In FIG. 2, the query processor 248 may operate in a similar way to the query processor 124 of FIG. 1. The query processor 248 receives the mapped text data from the neural network pre-processor 246 and generates data useable to derive a response to the original query. For example, this may comprise text data that is provided to the response interface 250 for rendering into a configured and/or available modality. The query processor 248 may perform local processing to provide the response and/or make a network request, such as is illustrated in FIG. 1, to obtain data to provide the response...[0046] The query interface 342 may comprise an internal interface (such as an application programming interface or function interface) or a remote interface (such as a network interface in a distributed system). [FIG.2 and 3] show wherein the query engine is an SQL query engine or an application programming interface (API) engine)
Regarding claim 5, Singh and Sacheti teach The method of claim 1, wherein the augmented context includes the original query. (Singh [0006] map data representing an initial query to data representing a revised query associated with one of the canonical queries, via the encoder-decoder neural network architecture, the data representing the revised query being further processed to provide a response to the initial query. [0059] A digital assistant is configured to use the trained encoder-decoder neural network architecture to map data representing an initial query to data representing a revised query associated with a canonical query, for example as described with reference to FIGS. 2 and 3. The data representing the revised query that is output by the neural network architecture may be further processed to provide a response to the initial query [0072] the original text tokens may be stored in association with (e.g., together with or indexed by) the processed text portions, such as pairs 512 and 514. Following generation of the data set 550 for training the machine learning system, the original text tokens may be written back to the data samples. For example, following named entity parsing the query data 512 may comprise [[“What, is, the, weather, in, <PLACE>, ?” ], “Toronto” ], where clustering is performed on the first list element and then, within the text tokens within one of the target and source data 552, 554, the original token “Toronto” is inserted back into the set of tokens in place of <PLACE>, i.e. becomes again [“What, is, the, weather, in, “Toronto”, ?” ]. This is performed for both the source and target data samples 512, 514 with the same replacement being performed, i.e. “Toronto” is inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples. [FIG. 2 in conjunction with FIG.5] shows wherein the augmented context includes the original query)
Regarding claim 6, Singh and Sacheti teach The method of claim 1, wherein the relevant tables and/or views includes tables and/or views and/or endpoints necessary to answer the original query. (Singh [FIG.2] shows wherein the relevant tables and/or views includes tables and/or views and/or endpoints necessary to answer the original query [0040] configured to receive the query data 140 and compute response data 142 to provide a response to the query. The response data 142 may comprise text data and/or data derived from a response representation that is used to generate one or more of a text response, a voice response (e.g., via a text-to-speech system) and a visual response (e.g., via one or more display devices). For example, a user may make a request for information such as “What is the traffic like on Route 48?”, “What is the weather like today?”, or “Where can I buy the best vegan taco?” and the response data 140 may be a formatted response...[0042] FIG. 2 shows an example 200 of a digital voice assistant 220, which may be implemented in one or more of the client devices 110 in FIG. 1. The digital voice assistant 200 is configured to receive audio input data 220 and generate a response 230 in one or more configured modalities. In the example, of FIG. 2, the digital voice assistant 220 may be configured to supply a response to a query as one or more of an audio output 232 (e.g., a voice response), a text response 234 and/or a visual response 236, such as one or more frames of video. [75-82] go into detail on wherein the relevant tables and/or views includes tables and/or views and/or endpoints necessary to answer the original query [FIG.5] further shows wherein the relevant tables and/or views includes tables and/or views and/or endpoints necessary to answer the original query )
Regarding claim 7, Singh and Sacheti teach The method of claim 1, further comprising: finding other query examples relevant to the original query, wherein the augmented context also includes the other query examples. (Singh [0009] multiple different query data samples, e.g. collected from different geographical areas, to a common or shared representation that may facilitate the clustering [0065] FIG. 6A shows an example of queries that are group based on response clusters. FIG. 6A shows three clusters 601, 602, and 603 with the query data (e.g., 512) for each cluster as a list of text data. In this example, the three clusters 601 to 603 may result from an initial stage of clustering. In a subsequent iteration of clustering, clusters 601 and 603 may be combined (e.g., based on a distance between centroids or averages of each cluster and a second predefined distance threshold).[0072] In certain cases, if named entity recognition and tag replacement is performed as described above, the original text tokens may be stored in association with (e.g., together with or indexed by) the processed text portions, such as pairs 512 and 514. Following generation of the data set 550 for training the machine learning system, the original text tokens may be written back to the data samples. For example, following named entity parsing the query data 512 may comprise [[“What, is, the, weather, in, <PLACE>, ?” ], “Toronto” ], where clustering is performed on the first list element and then, within the text tokens within one of the target and source data 552, 554, the original token “Toronto” is inserted back into the set of tokens in place of <PLACE>, i.e. becomes again [“What, is, the, weather, in, “Toronto”, ?” ]. This is performed for both the source and target data samples 512, 514 with the same replacement being performed, i.e. “Toronto” is inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples. [91-95] elaborate on finding other query examples relevant to the original query, wherein the augmented context also includes the other query examples.)
Regarding claim 8, Singh and Sacheti teach The method of claim 7, wherein finding the other query examples comprises: transforming the original query into a query vector of one or more numbers; (Singh [0007] The method generates effective training data for the machine learning system by converting query data samples to corresponding vector representations and then clustering these vector representations within vector space. [0008] In certain examples, obtaining the training data comprises obtaining pairs of text data representing queries and responses, and obtaining vector representations of the query data samples comprises converting the pairs of text data to corresponding vector representations. Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system. [0061] Via a first process, such as block 410 in FIG. 4, the text data pairs 512, 514 are converted into corresponding vector pairs 522, 524. This may comprise tokenizing the text representations in the data pairs 512, 514 and applying a transformer or recurrent neural network architecture to the tokenized text representations to generate the embedding vectors....[FIG.2 in conjunction with FIG.5] shows transforming the original query into a query vector of one or more numbers [64-71] go into further detail on the matter) obtaining a query list comprising n queries closest in distance to the original query, n >= 1, distances being calculated based on the query vector of the original vector and query vectors associated with each query of the query list; (Singh [0007] corresponding vector representations and then clustering these vector representations within vector space. The clusters generated by the clustering allow queries to be grouped and within each group one member to be selected as a canonical query. This then produces, e.g., for each group, a set of queries that are paired with a canonical query, e.g., for the group. The set of queries and the canonical query for each group are then respectively used to generate source (real-world input) and target data samples (a preferred form) for the machine learning system, which allows the system to learn how to map ill-formed queries with more variation into more regularly formed queries that match commonly used requests [0054] At block 410, vector representations of the query data samples are obtained. This may comprise obtaining a vector encoding of one or more of text data representing a query and text data representing a corresponding response. The vector encoding may be generated by a neural network encoder. Although examples herein are primarily based on a text-to-vector mapping, in other examples direct audio-to-vector mappings may also be used. The neural network encoder may comprise a separate neural network architecture to the machine learning system (e.g., a system that differs from neural network architecture 346) or may comprise an encoder portion of the machine learning system (e.g., the encoder portion of the encoder-decoder neural network architecture 346). In one case, the neural network encoder may comprise a pre-trained sentence encoder, such as the Universal Sentence Encoder described in the paper of the same name by Cer et al, published on arXiv in March 2018 and incorporated herein by reference, or the Sentence-BERT encoder discussed below. The form of the vector encoding for block 410 may vary between implementation and may not form part of an inference procedure, e.g. may only be used in the creation of training data. The vector representations may comprise a predefined number of floating-point elements (e.g., 256 to 1024) that are normalized within a predefined range (e.g., 0 to 1 or −1 to 1). [0055] the vector representations may comprise vector representations of query data and vector representations of accompanying response data, in this case the vector representations of accompanying response data may be clustered within a first stage and the vector representations of query data may be clustered in a second stage, using the results of the first stage clustering (e.g., a second stage of clustering within groups defined by the first stage clustering). Clustering may comprise evaluating a distance metric within vector space, such as a cosine similarity distance, and grouping neighboring points within vector space based on the distance metric. [0064] In one example query samples (e.g., 512) may be initially grouped based on a first defined distance measure or threshold between points representing response vectors 524. This may result in a first set of clusters. The remaining points representing response vectors 524 may then be clustered using a second defined distance measure or threshold. The second distance measure may be greater than the first distance measure, but the second distance measure may be a low value to group near-identical responses. A hierarchical clustering function may use a cosine similarity distance as the distance measure for clustering. In this example, an initial tight clustering guarantees that only highly similar responses are grouped together...[64-71] go into further detail on the matter) and adding one or more queries of the query list into the augmented context. (Singh [0064] In one example query samples (e.g., 512) may be initially grouped based on a first defined distance measure or threshold between points representing response vectors 524. This may result in a first set of clusters. The remaining points representing response vectors 524 may then be clustered using a second defined distance measure or threshold. The second distance measure may be greater than the first distance measure, but the second distance measure may be a low value to group near-identical responses. A hierarchical clustering function may use a cosine similarity distance as the distance measure for clustering. In this example, an initial tight clustering guarantees that only highly similar responses are grouped together... [0071] selection from the query group and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. In the first case, the method 400 may comprise, e.g. as part of blocks 405 or 410, performing named entity recognition on one or more of the query data 512 and the response data 514. This may be performed by applying one or more functions from available machine learning processing libraries such as the spaCy library managed by ExplosionAI GmbH or the Stanford Named Entity Recognizer provided by Stanford University. Named entity recognition is a known parsing function that may be applied to unstructured text data to locate tokens (e.g., characters, words or groups of words) within the text data that relate to entities or things with consistent referents, such as people, places and times. In certain cases, named entity recognition is performed upon both the query data 512 and the response data 514 prior to vectorization and clustering. In this case, the method 400 may comprise replacing the text data for tagged named entities with a named entity type tag, such as respectively replacing the detected names of people, places and times with the defined tags <PEOPLE>, <PLACE>, and <TIME>. This helps cluster similar queries with different named entities together, e.g. the text strings “What is the weather in Toronto?” and “What is the weather in San Diego?” may both be converted into the token sequences “[What, is, the, weather, in, <PLACE>, ?]” and so mapped to a common point....[0072] In certain cases, if named entity recognition and tag replacement is performed as described above, the original text tokens may be stored in association with (e.g., together with or indexed by) the processed text portions, such as pairs 512 and 514. Following generation of the data set 550 for training the machine learning system, the original text tokens may be written back to the data samples. For example, following named entity parsing the query data 512 may comprise [[“What, is, the, weather, in, <PLACE>, ?” ], “Toronto” ], where clustering is performed on the first list element and then, within the text tokens within one of the target and source data 552, 554, the original token “Toronto” is inserted back into the set of tokens in place of <PLACE>, i.e. becomes again [“What, is, the, weather, in, “Toronto”, ?” ]. This is performed for both the source and target data samples 512, 514 with the same replacement being performed, i.e. “Toronto” is inserted ....inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples. [64-71] go into further detail on the matter)
Regarding claim 9, Singh and Sacheti teach The method of claim 8, wherein the transforming the original query into the query vector comprises transforming the original query based on any one or more of embeddings, simhash, and ngrams. (Singh [0013] obtaining vector representations comprises: obtaining text representations of at least the responses to the queries; generating embedding vectors for the text representations; and projecting the embedding vectors to a lower dimensionality vector space to output the vector representations. It has been found in tests that compressing the vector representations leads to more efficient clustering (in lower dimensionality vector space) and helps remove surface noise that can lead to poor quality clusters. In one case, generating embedding vectors for the text representations comprises: tokenizing the text representations and applying a transformer neural network architecture to the tokenized text representations to generate the embedding vectors. Hence, an encoder neural network architecture similar to, or even a duplication of, the encoder portion of the encoder-decoder neural network architecture, can be used to generate a useful embedding at a sentence level that represents, e.g., queries and their responses. Using an encoder neural network architecture in this manner results in more informative vector representations than, say, an average of the word embedding within the text representations. The “sentence embedding” from the initial encoder may then form the input to the dimensionality reduction stage. [0046] The query interface 342 may comprise a tokenizer and/or an embedding layer such that the input sequence data 344 may comprise a sequence of integers (e.g., representing token indexes) or vectors (e.g., representing embedding vectors). In certain cases, tokenization and/or embedding may form part of the neural network architecture 346 and so may be omitted by the query interface 342. The query interface 342 may comprise an internal interface (such as an application programming interface or function interface) or a remote interface (such as a network interface in a distributed system). [0048] In general, the output sequence data 348 represents a canonical data representation for the query. As shown in FIG. 3, it may be of a similar form to the input sequence data 344, e.g. comprise a sequence of integer token indexes and/or embedding vectors. The output interface 350 is configured to process the output sequence data 350 for use in providing a response to the query. In the example 300 of FIG. 3, the output interface 350 receives the output sequence data 348 and generates the modified query data 330. For example, the output interface 350 may convert the output sequence data 348 back into text data, e.g. by replacing token indexes with the corresponding text token and/or mapping embedding vectors may to selections from a dictionary of tokens.[0061] Via a first process, such as block 410 in FIG. 4, the text data pairs 512, 514 are converted into corresponding vector pairs 522, 524. This may comprise tokenizing the text representations in the data pairs 512, 514 and applying a transformer or recurrent neural network architecture to the tokenized text representations to generate the embedding vectors. Generating embedding vectors may comprise the application of a sentence encoder, such as one of the encoders described above, or the Sentence-BERT encoder described in the paper “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks” by Reimers et al (published on arXiv August 2019), which is incorporated by reference herein. Known public sentence encoder parameters may be used without separate training (e.g., pre-trained parameters based on large data sets). Although, FIG. 5 shows vector pairs...[88-94] elaborate on the transforming the original query into the query vector comprises transforming the original query based on any one or more of embeddings, simhash, and ngrams.)
Regarding claim 10, Singh and Sacheti teach The method of claim 8, wherein finding the other query examples further comprises: subsequent to obtaining the query list and prior to adding the one or more queries of the query list, removing x queries from the query list, x => 1, the removed queries having x lowest entropies among the n queries of the query list. (Singh [0017] In certain examples, generating paired data samples comprises filtering generated paired data samples. This may comprise one or more of: removing paired data samples with a canonical query whose named entity tags do not match the named entity tags in the corresponding selection from the query group; and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. By filtering (i.e., post-processing) the generated data set of source queries and canonical queries, the quality of the data set may be improved resulting in improved training. It should be noted that production implementations may include data sets with over a million training samples, so, at this scale, there may still be statistical noise and sub-optimal source-target pairs despite the operations described above; filtering thus helps remove such pairs from the data set before training.[0071] In one case, the above filtering comprises one or more of removing paired data samples with a canonical query whose named entity tags do not match the named entity tags in the corresponding selection from the query group and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. In the first case, the method 400 may comprise, e.g. as part of blocks 405 or 410, performing named entity recognition on one or more of the query data 512 and the response data 514. This may be performed by applying one or more functions from available machine learning processing libraries such as the spaCy library managed by ExplosionAI GmbH or the Stanford Named Entity Recognizer provided by Stanford University. Named entity recognition is a known parsing function that may be applied to unstructured text data to locate tokens (e.g., characters, words or groups of words) within the text data that relate to entities or things with consistent referents, such as people, places and times. In certain cases, named entity recognition is performed upon both the query data 512 and the response data 514 prior to vectorization and clustering. [0072] inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples.[0077] In a second case of post-processing, a named entity tag matching operation may be performed. This may be similar to the named entity tag matching operation that is described above and that is applied to the training data set 550. In this case, the named entity tag matching operation may comprise removing possible or candidate output sequences whose named entity tags do not match those associated with the input sequence data (e.g., as extracted from a text form of query data 320). [69-77] further elaborate on the matter)
Regarding claim 11, Singh and Sacheti teach The method of claim 10, wherein removing the x queries is performed based on a pairwise comparison among the n queries of the query list. (Singh [0017] generating paired data samples comprises filtering generated paired data samples. This may comprise one or more of: removing paired data samples with a canonical query whose named entity tags do not match the named entity tags in the corresponding selection from the query group; and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. By filtering (i.e., post-processing) the generated data set of source queries and canonical queries, the quality of the data set may be improved resulting in improved training. It should be noted that production implementations may include data sets with over a million training samples, so, at this scale, there may still be statistical noise and sub-optimal source-target pairs despite the operations described above; filtering thus helps remove such pairs from the data set before training.[0071] In one case, the above filtering comprises one or more of removing paired data samples with a canonical query whose named entity tags do not match the named entity tags in the corresponding selection from the query group and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. In the first case, the method 400 may comprise, e.g. as part of blocks 405 or 410, performing named entity recognition on one or more of the query data 512 and the response data 514. This may be performed by applying one or more functions from available machine learning processing libraries such as the spaCy library managed by ExplosionAI GmbH or the Stanford Named Entity Recognizer provided by Stanford University. Named entity recognition is a known parsing function that may be applied to unstructured text data to locate tokens (e.g., characters, words or groups of words) within the text data that relate to entities or things with consistent referents, such as people, places and times. In certain cases, named entity recognition is performed upon both the query data 512 and the response data 514 prior to vectorization and clustering. [0072] inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples.[0073] A further post-processing filter may further compare named entities within the source and target data samples 512, 514 of a given pair and use a result of this comparison to determine whether the pair should be removed. For example, a given pair may be dropped if the named entity tags in the source and the target text do not match. This ensures that any named entity tag that is needed to respond to the query is present in both the source and target data samples, e.g. if <DATE> and/or <LOCATION> occurs in the source query 552, they also need to occur in the target query 554 to remain in the training data 550.[0074] In the case, where a check on semantic distance is performed, this may comprise comparing a distance measure in vector space between the source and target data samples and removing data sample pairs that are greater than a predefined threshold. It may also comprise generating language parsing metrics for both source and target data samples and then removing data samples based on a comparison of these metrics. In yet another case, further tests may be applied to one or more of the source and target data samples and the pair dropped as a data sample if one or more of the source and target data samples ... [90-95] further elaborate on the matter)
Regarding claim 12, Singh teaches A system configured to facilitate usage of natural language for cyber asset management, the system comprising: a modeler configured to determine, using a large language model (LLM) electronically in a first pass, relevant tables and/or views based on an original query, (Singh [0006] machine learning system for use with a digital assistant, the method comprising: obtaining training data comprising query data samples; obtaining vector representations of the query data [0008] obtaining the training data comprises obtaining pairs of text data representing queries and responses, and obtaining vector representations of the query data samples comprises converting the pairs of text data to corresponding vector representations. Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system.[0013] an encoder neural network architecture similar to, or even a duplication of, the encoder portion of the encoder-decoder neural network architecture [0040] The server device 120 in FIG. 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query placed by a user, e.g. either by voice following speech-to-text processing or by text entry. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data following receipt at the query processor 124. The query data 140 may also have associated metadata such as conversation state, user location information, and other user profile information. The query processor 124 is configured to receive the query data 140 and compute response data 142 to provide a response to the query. [0055] This may comprise clustering the vector representations within vector space, i.e. identifying clusters based on one or more clustering functions. The clustering functions may be provided by machine learning libraries such as sci-kit learn. It should be noted that clustering is typically performed in vector spaces with hundreds of dimensions (e.g., corresponding to hundreds of vector elements)—this is typically difficult to visualize in a world of 2 or 3 dimensions. In one case, a hierarchical clustering function may be used to iteratively define clusters. In one case, the vector representations may comprise vector representations of query data and vector representations of accompanying response data [0066] vectors in the clusters identified; the query indexes that correspond to these response points or vectors (e.g., response vector i and corresponding query i from the data set 510) are then grouped as per the response clusters. In certain cases, the query vectors within each group may themselves be clustered. This may be used to select one of the group as the canonical query for the group, e.g. based on a distance to the centroid of the query cluster [77-82] further elaborate on the matter [FIG.2 in conjunction with FIG.5] shows determining, using a large language model (LLM) in a first pass, relevant tables and/or views based on an original query) the original query being a natural language query received from a user; (Singh [FIG.2] shows the original query being a natural language query received from a user [0008] In certain examples, obtaining the training data comprises obtaining pairs of text data representing queries and responses, and obtaining vector representations of the query data samples comprises converting the pairs of text data to corresponding vector representations. Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system.[0040] The server device 120 in FIG. 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query placed by a user, e.g. either by voice following speech-to-text processing or by text entry. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data following receipt at the query processor 124. The query data 140 may also have associated metadata such as conversation state, user location information, and other user profile information. The query processor 124 is configured to receive the query data 140 and compute response data 142 to provide a response to the query. [0042] FIG. 2 shows an example 200 of a digital voice assistant 220, which may be implemented in one or more of the client devices 110 in FIG. 1...) a data sampler configured to obtain data samples for the relevant tables and/or views subsequent to the relevant tables and/or views being determined; and an augmented context preparer configured to prepare an augmented context, the augmented context including the data samples for the relevant tables and/or views (Singh [0006] obtaining training data comprising query data samples; obtaining vector representations of the query data samples; clustering the vector representations; determining canonical queries and corresponding query groups based on the clustered vector representations, wherein corresponding query groups correspond to determined canonical queries; generating paired data samples based on determined canonical queries and selections from the corresponding query groups; and training an encoder-decoder neural network architecture using the paired data samples [0008] Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system. [0054] At block 410, vector representations of the query data samples are obtained. This may comprise obtaining a vector encoding of one or more of text data representing a query and text data representing a corresponding response. The vector encoding may be generated by a neural network encoder. [0056] At block 420, canonical queries and corresponding query groups are determined based on the clustered vector representations. This may comprise selecting clusters of vector representations relating to different queries, e.g. as determined in block 415, and determining one representative query data sample for each cluster. For example, a representative (canonical) query data sample may comprise a more frequent query data sample within the cluster and/or have a vector representation [0070] In one variation, the method 400 of FIG. 4 and/or the process illustrated in FIG. 5, further comprises filtering generated paired data samples, such as the target-source data samples [0072] augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities [69-73] elaborate on the matter) wherein the augmented text is based on the original query (Singh [FIG.2 in conjunction with FIG.5] shows wherein the augmented text is based on the original query [0059] A digital assistant is configured to use the trained encoder-decoder neural network architecture to map data representing an initial query to data representing a revised query associated with a canonical query, for example as described with reference to FIGS. 2 and 3. The data representing the revised query that is output by the neural network architecture may be further processed to provide a response to the initial query, e.g. as described with reference to FIGS. 1 and 2. By suppling the source and target data samples as generated from the clusters in block 425, the neural network architecture “learns” to map non-standard sequences of tokens to standardized or canonical sequences of tokens that reflect better formed input for successful and accurate query [0071] the query data 512 and the response data 514 prior to vectorization and clustering. In this case, the method 400 may comprise replacing the text data for tagged named entities with a named entity type tag, such as respectively replacing the detected names of people, places and times with the defined tags <PEOPLE>, <PLACE>, and <TIME>. This helps cluster similar queries with different named entities together, e.g. the text strings “What is the weather in Toronto?” and “What is the weather in San Diego?” may both be converted into the token sequences processing.[0072] augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample [79-85] elaborate on the matter of the augmented text is based on the original query) wherein the modeler is configured to generate, using the LLM electronically in a second pass subsequent to the first pass, a generated query based on the augmented context, (Singh [FIG.3 in conjunction with FIG.5] shows generating, using the LLM in a second pass subsequent to the first pass, a generated query based on the augmented context [0006] determining canonical queries and corresponding query groups based on the clustered vector representations, wherein corresponding query groups correspond to determined canonical queries; generating paired data samples based on determined canonical queries and selections from the corresponding query groups; and training an encoder-decoder neural network architecture using the paired data samples, wherein the selections from the corresponding query groups are supplied as input sequence data and the determined canonical queries are supplied as output sequence data, wherein the digital assistant is configured to map data representing an initial query to data representing a revised query associated with one of the canonical queries, via the encoder-decoder neural network architecture, the data representing the revised query being further processed to provide a response to the initial query [0015] a second stage of clustering on the queries themselves, query groups that may be easily mapped onto canonical queries are generated [0057] At block 425, paired data samples are generated based on selected canonical queries and selections from the query group corresponding to each selected canonical query. For example, for each cluster, the canonical query (i.e., the data representation of said query in text or sequence form) may be set as a target or ground truth output data and each query in turn within the cluster may be paired with this canonical query as source input data. [0067] This filtering may thus be seen as a second stage of clustering based on the query vectors 522 as opposed to the response vectors 524... [87-92] further elaborate on the matter) and wherein the system further comprises a response generator configured to generate a response for the user based on the generated query, the response being in natural language. (Singh [0042] FIG. 2 shows an example 200 of a digital voice assistant 220, which may be implemented in one or more of the client devices 110 in FIG. 1. The digital voice assistant 200 is configured to receive audio input data 220 and generate a response 230 in one or more configured modalities. In the example, of FIG. 2, the digital voice assistant 220 may be configured to supply a response to a query as one or more of an audio output 232 (e.g., a voice response), a text response 234 and/or a visual response 236, such as one or more frames of video. Different modalities may be provided depending on the implementation of the digital voice assistant 220. In the example 200 of FIG. 2, the digital voice assistant 220 comprise an audio capture device 242, a speech-to-text system 244, a neural network pre-processor 246, a query processor 248 and a response interface 250.[0044] The speech-to-text system 244 receives audio data and translates this into a text data output. For example, the speech-to-text system 244 may comprise an automated transcription system that is based on one or more of hidden Markov models and neural network architectures (e.g., convolutional and recurrent neural network encoders). In this example, the speech-to-text system 244 provides a text data output for the neural network pre-processor 246 [45] The query processor 248 may output a response as pre-rendered data (e.g., an audio and/or video sample) or the response interface 250 may comprise a text-to-modality system, such as a text-to-speech or text-to-video system, or other vectorized information representation system, such as Musical Instrument Digital Interface (MIDI) music, Scalable Vector Graphics (SVG) graphics, or Flash animations, for providing the response 230. [78-83] elaborate on the matter [FIG.2] shows generating a response for the user based on the generated query, the response being in natural language. ) Singh lacks explicitly and orderly teaching wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views; However Sacheti teaches wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views; (Sacheti [0030] a table and populate the table with values of attributes of entities that are relevant to the query, images that are relevant to the query, and so forth. Further, the query classifier 126 can include instructions in the prompt that cause the GLM 112 to provide interactive graphics that represent ways in which the table can be modified, such as identities of attributes not represented in the table (e.g., due to screen size constraints), entities not represented in the table (e.g., due to screen size constraints), and so forth. Thus, based upon the prompt, the GLM 112 can generate and display a table showing multiple entities meeting the criterion and values of attributes of those entities. The user can then provide additional interaction with respect to the interactive graphic elements, causing the GLM 112 to present an updated table (e.g., with additional values of attributes included, with values for attributes removed, and so forth). Still further, the GLM 112 can update a table based upon further queries from the user, such as “I want it to be a hybrid SUV.” In this example, the GLM 112 updates the table to exclude vehicles that are not hybrid SUVs (and have a cost under $30,000). Operation of the GLM 112 is described in further detail below. [0033] a table and populate the table with attributes of entities identified by the search system 110 and provided to the GLM 112 by the search system 110. Further, the prompt information can include instructions to the GLM 112 to generate interactive graphical elements that can be interacted with to allow for relatively quick modification of content of a table constructed by the GLM 112. Examples are presented below [37-38] elaborate on the matter [FIG.1 in conjunction with FIG.3] shows visual of overall system wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to take all prior methods and make the addition of Sacheti in order to efficiently generate and populate a table in a database system (Sacheti [0004] generating, and/or displaying appropriate information in an efficient manner in response to certain types of user input. [0006] facilitate providing a conversational chat-based interface that includes a table that includes cells that comprise content, where the table is constructed by the GLM. For instance, a table can include attributes of entities[0029] As discussed above, operation of the search engine 110 is improved based upon the GLM 112, and operation of the GLM 112 is improved based upon the search engine 110. For instance, the search engine 110 is able to provide outputs that the search engine 110 was not previously able to provide (e.g., based upon outputs generated by the GLM 112), and the GLM 112 is improved by using information obtained by the search engine 110 to generate outputs (e.g., information identified by the search engine 110 can be included as a portion of a prompt used by the GLM 112 to generate outputs). Specifically, the GLM 112 generates results based upon information obtained by the search engine 110 that have a higher likelihood of being accurate when compared to results generated by GLMs 112 that are not based upon such information, as the search engine 110 is associated with years of design to curate information sources to ensure accuracy thereof.[0041] The communications diagram 200 illustrates various advantages of the computing system 100 over conventional search systems...[FIG.1 in conjunction with FIG.3] shows visual of overall system wherein each of the relevant tables and/or views include one or more attributes, and wherein the data samples include entities that populate the relevant tables and/or views)
Regarding claim 13, Singh and Sacheti teach The system of claim 12, wherein the relevant tables and/or views are SQL tables and/or views, and/or wherein the generated query is an SQL query. (Singh [0006] determining canonical queries and corresponding query groups based on the clustered vector representations, wherein corresponding query groups correspond to determined canonical queries; generating paired data samples based on determined canonical queries and selections from the corresponding query groups; and training an encoder-decoder neural network architecture using the paired data samples, wherein the selections from the corresponding query groups are supplied as input sequence data and the determined canonical queries are supplied as output sequence data, wherein the digital assistant is configured to map data representing an initial query to data representing a revised query associated with one of the canonical queries, via the encoder-decoder neural network architecture, the data representing the revised query being further processed to provide a response to the initial query [0015] a second stage of clustering on the queries themselves, query groups that may be easily mapped onto canonical queries are generated [0057] At block 425, paired data samples are generated based on selected canonical queries and selections from the query group corresponding to each selected canonical query. For example, for each cluster, the canonical query (i.e., the data representation of said query in text or sequence form) may be set as a target or ground truth output data and each query in turn within the cluster may be paired with this canonical query as source input data. [0067] This filtering may thus be seen as a second stage of clustering based on the query vectors 522 as opposed to the response vectors 524... [87-92] further elaborate on the matter [FIG.3 in conjunction with FIG.5] shows wherein the relevant tables and/or views are SQL tables and/or views, and/or wherein the generated query is an SQL query)
Regarding claim 14, Singh and Sacheti teach The system of claim 12, wherein the response generator comprises: a query engine configured to process the generated query to retrieve query results; and a natural language converter configured to convert the query results to the response. (Singh [0010] a first portion of text data represents an output of speech-to-text processing that is performed on audio data for a voice query and a second portion of text data comprises data for use in providing the response to the voice query, wherein the digital assistant is configured to map data representing an initial voice query to data representing a revised voice query via the trained neural machine learning model. The present disclosure is of particular benefit for voice applications, where there is a high chance of malformed input queries due to the inherent noise present in both the real-world and in speech-to-text systems. [0040] The server device 120 in FIG. 1 is configured to receive query data 140 from one or more of the client devices 110. The query data 140 may comprise text data representing a query placed by a user, e.g. either by voice following speech-to-text processing or by text entry. In other cases, the query data 140 may comprise audio data or audio features that are converted to text data following receipt at the query processor 124. The query data 140 may also have associated metadata such as conversation state, user location information, and other user profile information. The query processor 124 is configured to receive the query data 140 and compute response data 142 to provide a response to the query. The response data 142 may comprise text data and/or data derived from a response representation that is used to generate one or more of a text response, a voice response (e.g., via a text-to-speech system) and a visual response (e.g., via one or more display devices). For example, a user may make a request for information such as “What is the traffic like on Route 48?”, “What is the weather like today?”, or “Where can I buy the best vegan taco?” and the response data 140 may be a formatted response[0045] In FIG. 2, the query processor 248 may operate in a similar way to the query processor 124 of FIG. 1. The query processor 248 receives the mapped text data from the neural network pre-processor 246 and generates data useable to derive a response to the original query. For example, this may comprise text data that is provided to the response interface 250 for rendering into a configured and/or available modality. The query processor 248 may perform local processing to provide the response and/or make a network request, such as is illustrated in FIG. 1, to obtain data to provide the response. The query processor 248 may output a response as pre-rendered data (e.g., an audio and/or video sample) or the response interface 250 may comprise a text-to-modality system, such as a text-to-speech or text-to-video system [79-81] elaborate on processing , using a query engine, the generated query to retrieve query results and converting, using a natural language converter, the query results to the response. [FIG.2] shows using a query engine, the generated query to retrieve query results and converting, using a natural language converter, the query results to the response )
Regarding claim 15, Singh and Sacheti teach The system of claim 12, wherein the augmented context includes the original query. (Singh [0006] map data representing an initial query to data representing a revised query associated with one of the canonical queries, via the encoder-decoder neural network architecture, the data representing the revised query being further processed to provide a response to the initial query. [0059] A digital assistant is configured to use the trained encoder-decoder neural network architecture to map data representing an initial query to data representing a revised query associated with a canonical query, for example as described with reference to FIGS. 2 and 3. The data representing the revised query that is output by the neural network architecture may be further processed to provide a response to the initial query [0072] the original text tokens may be stored in association with (e.g., together with or indexed by) the processed text portions, such as pairs 512 and 514. Following generation of the data set 550 for training the machine learning system, the original text tokens may be written back to the data samples. For example, following named entity parsing the query data 512 may comprise [[“What, is, the, weather, in, <PLACE>, ?” ], “Toronto” ], where clustering is performed on the first list element and then, within the text tokens within one of the target and source data 552, 554, the original token “Toronto” is inserted back into the set of tokens in place of <PLACE>, i.e. becomes again [“What, is, the, weather, in, “Toronto”, ?” ]. This is performed for both the source and target data samples 512, 514 with the same replacement being performed, i.e. “Toronto” is inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples. [FIG. 2 in conjunction with FIG.5] shows wherein the augmented context includes the original query)
Regarding claim 16, Singh and Sacheti teach The system of claim 12, further comprising: a query finder configured to find other query examples relevant to the original query, wherein the augmented context also includes the other query examples. (Singh [0009] multiple different query data samples, e.g. collected from different geographical areas, to a common or shared representation that may facilitate the clustering [0065] FIG. 6A shows an example of queries that are group based on response clusters. FIG. 6A shows three clusters 601, 602, and 603 with the query data (e.g., 512) for each cluster as a list of text data. In this example, the three clusters 601 to 603 may result from an initial stage of clustering. In a subsequent iteration of clustering, clusters 601 and 603 may be combined (e.g., based on a distance between centroids or averages of each cluster and a second predefined distance threshold).[0072] In certain cases, if named entity recognition and tag replacement is performed as described above, the original text tokens may be stored in association with (e.g., together with or indexed by) the processed text portions, such as pairs 512 and 514. Following generation of the data set 550 for training the machine learning system, the original text tokens may be written back to the data samples. For example, following named entity parsing the query data 512 may comprise [[“What, is, the, weather, in, <PLACE>, ?” ], “Toronto” ], where clustering is performed on the first list element and then, within the text tokens within one of the target and source data 552, 554, the original token “Toronto” is inserted back into the set of tokens in place of <PLACE>, i.e. becomes again [“What, is, the, weather, in, “Toronto”, ?” ]. This is performed for both the source and target data samples 512, 514 with the same replacement being performed, i.e. “Toronto” is inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples. [91-95] elaborate on finding other query examples relevant to the original query, wherein the augmented context also includes the other query examples.)
Regarding claim 17, Singh and Sacheti teach The system of claim 16, wherein the query finder is configured to: transform the original query into a query vector of one or more numbers; (Singh [0007] The method generates effective training data for the machine learning system by converting query data samples to corresponding vector representations and then clustering these vector representations within vector space. [0008] In certain examples, obtaining the training data comprises obtaining pairs of text data representing queries and responses, and obtaining vector representations of the query data samples comprises converting the pairs of text data to corresponding vector representations. Using query data samples in a text form means that the present technology is applicable to many different forms of dialogue and conversational agent including both text-based chatbots and voice systems that include a speech-to-text system. [0061] Via a first process, such as block 410 in FIG. 4, the text data pairs 512, 514 are converted into corresponding vector pairs 522, 524. This may comprise tokenizing the text representations in the data pairs 512, 514 and applying a transformer or recurrent neural network architecture to the tokenized text representations to generate the embedding vectors....[FIG.2 in conjunction with FIG.5] shows transforming the original query into a query vector of one or more numbers [64-71] go into further detail on the matter) obtain a query list comprising n queries closest in distance to the original query, n >= 1, distances being calculated based on the query vector of the original vector and query vectors associated with each query of the query list; (Singh [0007] corresponding vector representations and then clustering these vector representations within vector space. The clusters generated by the clustering allow queries to be grouped and within each group one member to be selected as a canonical query. This then produces, e.g., for each group, a set of queries that are paired with a canonical query, e.g., for the group. The set of queries and the canonical query for each group are then respectively used to generate source (real-world input) and target data samples (a preferred form) for the machine learning system, which allows the system to learn how to map ill-formed queries with more variation into more regularly formed queries that match commonly used requests [0054] At block 410, vector representations of the query data samples are obtained. This may comprise obtaining a vector encoding of one or more of text data representing a query and text data representing a corresponding response. The vector encoding may be generated by a neural network encoder. Although examples herein are primarily based on a text-to-vector mapping, in other examples direct audio-to-vector mappings may also be used. The neural network encoder may comprise a separate neural network architecture to the machine learning system (e.g., a system that differs from neural network architecture 346) or may comprise an encoder portion of the machine learning system (e.g., the encoder portion of the encoder-decoder neural network architecture 346). In one case, the neural network encoder may comprise a pre-trained sentence encoder, such as the Universal Sentence Encoder described in the paper of the same name by Cer et al, published on arXiv in March 2018 and incorporated herein by reference, or the Sentence-BERT encoder discussed below. The form of the vector encoding for block 410 may vary between implementation and may not form part of an inference procedure, e.g. may only be used in the creation of training data. The vector representations may comprise a predefined number of floating-point elements (e.g., 256 to 1024) that are normalized within a predefined range (e.g., 0 to 1 or −1 to 1). [0055] the vector representations may comprise vector representations of query data and vector representations of accompanying response data, in this case the vector representations of accompanying response data may be clustered within a first stage and the vector representations of query data may be clustered in a second stage, using the results of the first stage clustering (e.g., a second stage of clustering within groups defined by the first stage clustering). Clustering may comprise evaluating a distance metric within vector space, such as a cosine similarity distance, and grouping neighboring points within vector space based on the distance metric. [0064] In one example query samples (e.g., 512) may be initially grouped based on a first defined distance measure or threshold between points representing response vectors 524. This may result in a first set of clusters. The remaining points representing response vectors 524 may then be clustered using a second defined distance measure or threshold. The second distance measure may be greater than the first distance measure, but the second distance measure may be a low value to group near-identical responses. A hierarchical clustering function may use a cosine similarity distance as the distance measure for clustering. In this example, an initial tight clustering guarantees that only highly similar responses are grouped together...[64-71] go into further detail on the matter) and add one or more queries of the query list into the augmented context. (Singh [0064] In one example query samples (e.g., 512) may be initially grouped based on a first defined distance measure or threshold between points representing response vectors 524. This may result in a first set of clusters. The remaining points representing response vectors 524 may then be clustered using a second defined distance measure or threshold. The second distance measure may be greater than the first distance measure, but the second distance measure may be a low value to group near-identical responses. A hierarchical clustering function may use a cosine similarity distance as the distance measure for clustering. In this example, an initial tight clustering guarantees that only highly similar responses are grouped together... [0071] selection from the query group and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. In the first case, the method 400 may comprise, e.g. as part of blocks 405 or 410, performing named entity recognition on one or more of the query data 512 and the response data 514. This may be performed by applying one or more functions from available machine learning processing libraries such as the spaCy library managed by ExplosionAI GmbH or the Stanford Named Entity Recognizer provided by Stanford University. Named entity recognition is a known parsing function that may be applied to unstructured text data to locate tokens (e.g., characters, words or groups of words) within the text data that relate to entities or things with consistent referents, such as people, places and times. In certain cases, named entity recognition is performed upon both the query data 512 and the response data 514 prior to vectorization and clustering. In this case, the method 400 may comprise replacing the text data for tagged named entities with a named entity type tag, such as respectively replacing the detected names of people, places and times with the defined tags <PEOPLE>, <PLACE>, and <TIME>. This helps cluster similar queries with different named entities together, e.g. the text strings “What is the weather in Toronto?” and “What is the weather in San Diego?” may both be converted into the token sequences “[What, is, the, weather, in, <PLACE>, ?]” and so mapped to a common point....[0072] In certain cases, if named entity recognition and tag replacement is performed as described above, the original text tokens may be stored in association with (e.g., together with or indexed by) the processed text portions, such as pairs 512 and 514. Following generation of the data set 550 for training the machine learning system, the original text tokens may be written back to the data samples. For example, following named entity parsing the query data 512 may comprise [[“What, is, the, weather, in, <PLACE>, ?” ], “Toronto” ], where clustering is performed on the first list element and then, within the text tokens within one of the target and source data 552, 554, the original token “Toronto” is inserted back into the set of tokens in place of <PLACE>, i.e. becomes again [“What, is, the, weather, in, “Toronto”, ?” ]. This is performed for both the source and target data samples 512, 514 with the same replacement being performed, i.e. “Toronto” is inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples. [64-71] go into further detail on the matter)
Regarding claim 18, Singh and Sacheti teach The system of claim 17, wherein the query finder is configured to transform the original query into the query vector based on any one or more of embeddings, simhash, and ngrams. (Singh [0013] obtaining vector representations comprises: obtaining text representations of at least the responses to the queries; generating embedding vectors for the text representations; and projecting the embedding vectors to a lower dimensionality vector space to output the vector representations. It has been found in tests that compressing the vector representations leads to more efficient clustering (in lower dimensionality vector space) and helps remove surface noise that can lead to poor quality clusters. In one case, generating embedding vectors for the text representations comprises: tokenizing the text representations and applying a transformer neural network architecture to the tokenized text representations to generate the embedding vectors. Hence, an encoder neural network architecture similar to, or even a duplication of, the encoder portion of the encoder-decoder neural network architecture, can be used to generate a useful embedding at a sentence level that represents, e.g., queries and their responses. Using an encoder neural network architecture in this manner results in more informative vector representations than, say, an average of the word embedding within the text representations. The “sentence embedding” from the initial encoder may then form the input to the dimensionality reduction stage. [0046] The query interface 342 may comprise a tokenizer and/or an embedding layer such that the input sequence data 344 may comprise a sequence of integers (e.g., representing token indexes) or vectors (e.g., representing embedding vectors). In certain cases, tokenization and/or embedding may form part of the neural network architecture 346 and so may be omitted by the query interface 342. The query interface 342 may comprise an internal interface (such as an application programming interface or function interface) or a remote interface (such as a network interface in a distributed system). [0048] In general, the output sequence data 348 represents a canonical data representation for the query. As shown in FIG. 3, it may be of a similar form to the input sequence data 344, e.g. comprise a sequence of integer token indexes and/or embedding vectors. The output interface 350 is configured to process the output sequence data 350 for use in providing a response to the query. In the example 300 of FIG. 3, the output interface 350 receives the output sequence data 348 and generates the modified query data 330. For example, the output interface 350 may convert the output sequence data 348 back into text data, e.g. by replacing token indexes with the corresponding text token and/or mapping embedding vectors may to selections from a dictionary of tokens.[0061] Via a first process, such as block 410 in FIG. 4, the text data pairs 512, 514 are converted into corresponding vector pairs 522, 524. This may comprise tokenizing the text representations in the data pairs 512, 514 and applying a transformer or recurrent neural network architecture to the tokenized text representations to generate the embedding vectors. Generating embedding vectors may comprise the application of a sentence encoder, such as one of the encoders described above, or the Sentence-BERT encoder described in the paper “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks” by Reimers et al (published on arXiv August 2019), which is incorporated by reference herein. Known public sentence encoder parameters may be used without separate training (e.g., pre-trained parameters based on large data sets). Although, FIG. 5 shows vector pairs...[88-94] elaborate on the transforming the original query into the query vector comprises transforming the original query based on any one or more of embeddings, simhash, and ngrams.)
Regarding claim 19, Singh and Sacheti teach The system of claim 17, wherein subsequent to obtaining the query list and prior to adding the one or more queries of the query list, the query finder is further configured to remove x queries from the query list, x => 1, the removed queries having x lowest entropies among the n queries of the query list. (Singh [0017] paired data samples comprises filtering generated paired data samples. This may comprise one or more of: removing paired data samples with a canonical query whose named entity tags do not match the named entity tags in the corresponding selection from the query group; and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. By filtering (i.e., post-processing) the generated data set of source queries and canonical queries, the quality of the data set may be improved resulting in improved training. It should be noted that production implementations may include data sets with over a million training samples, so, at this scale, there may still be statistical noise and sub-optimal source-target pairs despite the operations described above; filtering thus helps remove such pairs from the data set before training.[0071] In one case, the above filtering comprises one or more of removing paired data samples with a canonical query whose named entity tags do not match the named entity tags in the corresponding selection from the query group and removing paired data samples based on a comparison of semantic distance metrics for the canonical query and the corresponding selection from the query group. In the first case, the method 400 may comprise, e.g. as part of blocks 405 or 410, performing named entity recognition on one or more of the query data 512 and the response data 514. This may be performed by applying one or more functions from available machine learning processing libraries such as the spaCy library managed by ExplosionAI GmbH or the Stanford Named Entity Recognizer provided by Stanford University. Named entity recognition is a known parsing function that may be applied to unstructured text data to locate tokens (e.g., characters, words or groups of words) within the text data that relate to entities or things with consistent referents, such as people, places and times. In certain cases, named entity recognition is performed upon both the query data 512 and the response data 514 prior to vectorization and clustering. [0072] inserted in place of <PLACE> tags in both the source and target data samples 512, 514. This may also help to augment the training data to improve accuracy, for example, as source query samples with different named entities but a common processed form may be mapped to a target query sample with a further different named entity and then different versions may be created using different versions of the original named entities. For example, all the queries in data 601 and 603 may be mapped to one target data sample to initially generate sixteen source-target pairs, and then all the original <DATE> and <LOCATION> named entities, corresponding to different dates and locations where there may be many more than sixteen entities, may be inserted into different copies of the sixteen source-target pairs to generate more than sixteen training samples.[0077] In a second case of post-processing, a named entity tag matching operation may be performed. This may be similar to the named entity tag matching operation that is described above and that is applied to the training data set 550. In this case, the named entity tag matching operation may comprise removing possible or candidate output sequences whose named entity tags do not match those associated with the input sequence data (e.g., as extracted from a text form of query data 320). [69-77] further elaborate on the matter)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ARYAN D TOUGHIRY whose telephone number is (571)272-5212. The examiner can normally be reached Monday - Friday, 9 am - 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aleksandr Kerzhner can be reached at (571) 270-1760. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ARYAN D TOUGHIRY/Examiner, Art Unit 2165