DETAILED ACTION
This action is in response to the original filing on April 12, 2024. Claims 1-20 are pending and have been considered below. Claims 1, 9, and 17 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1 – The claim is directed to a product: one or more non-transitory, computer-readable storage media…
Step 2A, Prong 1 – A judicial exception is recited in this claim as it recites a mental process (see MPEP 2106.04(a)(2)(III)):
generate, using the set of tokens, a set of candidate query string items by identifying, in the block-based data structure, an item that corresponds to a token in the set of tokens, wherein the item relates to at least one of a block title, block identifier, block content, or a block property… A human can reasonably perform generating a set of candidate query string items by identifying an item that corresponds to a token in a set of tokens within the human mind or with the aid of a pen and paper.
Step 2A, Prong 2 – The following limitations are additional elements that fail to integrate the judicial exception into a practical application:
one or more non-transitory, computer-readable storage media comprising instructions recorded thereon, wherein the instructions, when executed by at least one data processor of a computing system, cause the computing system to… a computer-readable media and processor(s) of a computing system used as a mere tool to apply an exception is a generic element for performing or applying the abstract idea using a generic computing environment (see MPEP 2106.05(f)).
generate, by a question and answer (Q&A) assistant engine of a multimodal content management system having a block-based data structure, a set of tokens comprising a set of data source tokens and a set of parameter tokens, wherein the set of tokens is generated based on a natural language prompt… to generate a set of tokens from a natural language prompt is data gathering and outputting (see MPEP 2106.05(g)).
using the set of tokens, generate a code unit executable against the block-based data structure, comprising operations to… to generate a code unit using a set of tokens is data gathering and outputting (see MPEP 2106.05(g)).
execute a trained neural network to generate, using the set of tokens, a set of candidate query string items by identifying… using a neural network as a mere tool to apply an exception is a generic element for performing or applying the abstract idea using a generic computing environment (see MPEP 2106.05(f)).
based on the set of candidate query string items, generate a set of query strings… generating a set of query strings from a set of candidate query string items is data gathering and outputting (see MPEP 2106.05(g)).
generate the code unit, wherein the code unit comprises a top N query string of the set of query strings and a wrapper… generating a code unit from the set of query strings and a wrapper is data gathering and outputting (see MPEP 2106.05(g)).
and generate and display, at a graphical user interface, a visualization comprising the code unit… generating and displaying a visualization from a code unit is data gathering and outputting (see MPEP 2106.05(g)).
Step 2B – These elements are recited at such a high level of generality that they fail to integrate the abstract idea into a practical application, since they provide nothing more than mere instructions to implement an abstract idea on a generic computer (MPEP 2106.05(f)) or only amount to data gathering and outputting (MPEP 2106.05(g)) without significantly more. These limitations, taken either alone or in combination, fail to provide an inventive concept. Thus, the claim is not patent eligible.
Claims 2-8 recite limitations which further narrow the abstract ideas of claim 1 by specifying more details of the mental process that occurs:
Regarding claim 2, specifying wherein the trained neural network is trained on two or more of: (i) block types, (ii) block dependencies, (iii) block content values, (iv) block content types, or (v) block format in this manner does not overcome the rejection of claim 1 as modifying “the trained neural network” does not make “identifying” to not be a mental process.
Regarding claim 3, describing wherein the code unit is executable against the block-based data structure to generate, at least in part, a result set responsive to the natural language prompt is still data gathering and outputting (see MPEP 2106.05(g)).
Regarding claim 4, describing wherein the code unit is executable against a set of blocks to generate a result set that comprises a first item in a first modality and a second item in a second modality is still data outputting (see MPEP 2106.05(g)).
Regarding claim 5, specifying wherein the top N query string is a first top N query string that relates to a first query executable to retrieve items in the first modality, and wherein the code unit further comprises a second top N query string that relates to a second query executable to retrieve items in the second modality in this manner does not overcome the rejection of claim 4 as modifying “the top N query string” and “the code unit” does not make “identifying” to not be a mental process.
Regarding claim 6, specifying wherein the natural language prompt is associated with or comprises an item that specifies a format of the code unit in this manner does not overcome the rejection of claim 1 as modifying “the natural language prompt” does not make “identifying” to not be a mental process. Furthermore, describing and wherein the instructions, when executed by the at least one data processor of the computing system, cause the computing system to generate at least one of the top N query string or the wrapper according to the specified format of the code unit is still data gathering and outputting (see MPEP 2106.05(g)).
Regarding claim 7, specifying wherein the format of the code unit specifies a call to an application programming interface (API) function executable against the block-based data structure in this manner does not overcome the rejection of claim 6 as modifying “the format of the code unit” does not make “identifying” to not be a mental process.
Regarding claim 8, this claim further limits the abstract idea of claim 1 to be based on a mental process: wherein the instructions, when executed by the at least one data processor of the computing system, cause the computing system to determine the top N query string by determining a predictive accuracy indicator for a particular query string… A human can reasonably perform determining “the top N query string” by determining a predictive accuracy indicator for a “particular query string” within the human mind or with the aid of a pen and paper. Furthermore, using processor(s) of a computing system as a mere tool to apply an exception is a generic element for performing or applying the abstract idea using a generic computing environment (see MPEP 2106.05(f)).
Claims 9-16 are method claims which contain similar limitations to the computer-readable medium of claims 1-8, respectively. Therefore, claims 9-16 are rejected under substantially the same rationale as claims 1-8, respectively.
Claims 17-20 are system claims which contain similar limitations to the computer-readable medium of claims 1-4, respectively. Therefore, claims 17-20 are rejected under substantially the same rationale as claims 1-4, respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 6-9, 11-12, 14-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al. (“FlowSense: A Natural Language Interface for Visual data Exploration within a Dataflow System,” 2019, hereinafter Yu) in view of Kumar et al. (US 20250217769 A1, hereinafter Kumar).
Regarding claim 1:
Regarding the limitation one or more non-transitory, computer-readable storage media comprising instructions recorded thereon, wherein the instructions, when executed by at least one data processor of a computing system, cause the computing system to: generate, by a question and answer (Q&A) assistant engine of a multimodal content management system having a block-based data structure, a set of tokens comprising a set of data source tokens and a set of parameter tokens, wherein the set of tokens is generated based on a natural language prompt, Yu teaches instructions… wherein the instructions, when executed by at least one data processor of a computing system, cause the computing system to (Page 12, Section A, ¶1 “We provide an open source repository that contains the details of the FlowSense implementation… This repository includes the grammar rules, backend API (implemented in TypeScript and Python), and integration tests,” one of ordinary skill in the art would recognize that the “repository” contains instructions which are implied to be implemented when executed by at least one data processor of a computing system which clones or pulls the repository’s contents): generate, by an … assistant engine (Abstract: “FlowSense employs a semantic parser with special utterance tagging and special utterance placeholders to generalize to different datasets and dataflow diagrams”) of a… content management system (Page 1, Col. 2, ¶1 “Dataflow visualization systems (DFVS) have been proposed to achieve larger analytical flexibility,” Page 2, Col. 1, ¶1 “With the integration of FlowSense, dataflow diagram editing becomes more intuitive in VisFlow, and consequently the user can use the DFVS more efficiently,” Page 3, Col. 1, Section 2.4, ¶1 “FlowSense uses semantic parsing to process NL input and map user queries to VisFlow functions,” Page 1, Fig. 1, Page 3, Table 1, and Page 7, Fig. 5 depict the outputs of various “VisFlow functions” including data visualization through scatterplots, charts, and maps; “VisFlow” encompasses a content management system when given its broadest reasonable interpretation of a system that manipulates content such as graphs and maps which are displayed for a user) having a block-based data structure (Page 5, Col. 1, Section 3.4, ¶1 “We… identified a common pattern with five key query components that all VisFlow functions may contain: function type, function options, source node(s), target node(s), and port specification,” Page 6, Col. 2, Section 4.4, ¶1 “Once a query is successfully completed, FlowSense performs the VisFlow function(s)… This typically results in the creation of one or more nodes… FlowSense may also update existing nodes without creating any new nodes… Additionally, a query may operate on multiple existing nodes at once,” wherein a framework that manipulates “nodes” through “queries” and “functions” encompasses a block-based data structure, wherein a “node” containing data and identifying information encompasses a block when given its broadest reasonable interpretation), a set of tokens comprising a set of data source tokens and a set of parameter tokens (Page 4, Col. 1, ¶1 “FlowSense recognizes table column names, node labels, node types, and dataset names as special utterances,” wherein “node labels… dataset names” encompass data source tokens and “column names… node types” encompass parameter tokens when given their broadest reasonable interpretations as a set of identifying tokens and tokens specifying selection criteria, respectively), wherein the set of tokens is generated based on a natural language prompt (Page 1, Col. 1, Section 1, ¶1 “Natural language interfaces (NLI) for data visualizations seek better usability of visualization solutions by introducing natural language (NL) query support,” Page 4, Col. 1, ¶1 “FlowSense extracts a special group of tokens called the special utterances from NL input. Special utterances are words that refer to entities in the dataset or the dataflow diagram. They are the arguments and operands of VisFlow functions”). However, Yu fails to teach one or more non-transitory, computer-readable storage media comprising instructions recorded thereon, wherein the instructions, when executed by at least one data processor of a computing system, cause the computing system to: generate, by a question and answer (Q&A) assistant engine of a multimodal content management system…
Kumar, in the same field of endeavor, teaches one or more non-transitory, computer-readable storage media comprising instructions recorded thereon (¶69 “The physical memory can include datastores, working memory portions, storage portions, and the like. Storage portions of the memory can include executable instructions that, when executed by the processor, cause the processor to (with assistance of working memory) instantiate an instance of a generative output application”) and a question and answer (Q&A) assistant engine (Fig. 1 – 112, 114, 116, ¶139 “the centralized generative service 112 utilizes a variety of software plugins and/or automated assistant services to provide the requested operations. The centralized generative service 112 may generate prompts, which, in this example, can be provided as input to a prompt engineering/prompt preconditioning service (such as the prompt management service 114) that, in turn, provides a modified user prompt as input to a generative output service 116”) of a multimodal system (Fig. 2 – 203-207, ¶178 “backend applications 203, 205, 207 may each provide a different platform including, without limitation, documentation platforms, issue tracking platforms, user and project directory platforms, source code management systems or platforms, project management or project tracking platforms, and other platforms”).
Regarding the limitation using the set of tokens, generate a code unit executable against the block-based data structure, comprising operations to: execute a trained neural network to generate, using the set of tokens, a set of candidate query string items by identifying, in the block-based data structure, an item that corresponds to a token in the set of tokens, wherein the item relates to at least one of a block title, block identifier, block content, or a block property, Yu teaches using the set of tokens, generate a code unit executable against the block-based data structure (Page 3, Col. 2, Section 3.1, ¶1 “By implementing the VisFlow functions, FlowSense essentially supports the building blocks of visual data exploration in VisFlow so that analyses rendered by VisFlow native interactions can be carried out with FlowSense. These functions also reflect the fundamental analytical activity defined in information visualization task taxonomies,” Page 5, Col. 2, Section 4.1, ¶1 “Special utterances have remarkable roles in executing a VisFlow function. Their tagging is performed on the fly when the user types the query,” Page 6, Col. 1, ¶1 “tagging helps analyze the basic semantic structure of a query” and Section 4.2, ¶1 “FlowSense uses keyword classification to identify the semantic meaning of words in the NL query and uses this information to decide a proper VisFlow function to execute,” Col. 2, Section 4.4, ¶1 “Once a query is successfully completed, FlowSense performs the VisFlow function(s) with the given function options. This typically results in the creation of one or more nodes… update existing nodes… operate on multiple existing nodes,” wherein a “VisFlow function” encompasses a code unit executable against the block-based data structure), comprising operations to… generate, using the set of tokens, a set of candidate query string items by identifying, in the block based data structure, an item that corresponds to a token in the set of tokens (Page 4, Fig. 2, Caption: “The five major components of a query pattern are underscored. Each component and its relevant parts in the parse tree and the dataflow diagram are highlighted by a unique color,” Col. 1, ¶1 “FlowSense extracts a special group of tokens called the special utterances from NL input. Special utterances are words that refer to entities in the dataset or the dataflow diagram,” Col. 2, Section 3.3.1, ¶3 “special utterances enable VisFlow functions to operate on elements that are important for dataflow diagram editing and visual data exploration,” wherein “relevant parts,” “entities,” or “elements” encompass a set of candidate query string items corresponding to “special utterances” or tokens of the set of tokens), wherein the item relates to at least one of a block title, block identifier, block content, or block property (Page 4, Fig. 2, Caption: “An example FlowSense query… The result of executing this query is to create a parallel coordinates plot of columns mpg, horsepower, and origin, with its input coming from the selection port of the node labelled MyChart,” wherein “MyChart,” “parallel coordinates plot,” and “column names” including “mpg” or “horsepower,” encompass a block identifier, block property or the type of “node,” and block content, respectively). However, Yu fails to teach operations to: execute a trained neural network to generate…
Kumar teaches operations to execute a trained neural network to generate outputs (Fig. 2 – 220, ¶180 “the prompt management service may include an intent recognition model that is configured to classify the user input a being directed to a class or type of inquiry. The prompt management service 220 may use natural language processing including tokenization and word embedding techniques to convert the natural language input into a multi-dimensional vector or other representation. The processed natural language input may then be provided to the intent recognition module, which has been constructed to provide an intent classification or query classification as an output. The intent recognition model may include a transformer model that has been trained using a corpus of previous user input or queries… In some cases, the intent recognition model is a bidirectional encoder representation transformer model that has been trained using a training corpus that includes, but… not limited to, previous interactions,” ¶240 “the generative service may use a trained intent recognition module to determine an intent (also referred to as an action intent) for the user input. The intent or action intent may indicate which corpus of content is relevant to the user input and a candidate set of operations required to perform the requested task… In some cases, the intent recognition module is adapted to output a suggested assistant service or plugin instead of or in addition to the action intent”).
Yu teaches based on the set of candidate query string items, generate a set of query strings (Page 4, Col. 1, Section 3.3, ¶2 “The grammar of the FlowSense semantic parser attempts to derive an input query by recursively searching for all possible matches (up to a preset limit) of the grammar rules. This procedure is called derivation,” Col. 2, Section 3.3.1, ¶1 “The FlowSense grammar consists of static grammar rules and the special utterance placeholders. The special utterance placeholders are at runtime dynamically replaced by their corresponding dataflow elements. Therefore, the FlowSense semantic parsing is independent of the dataset, the dataflow diagram, and the analytical tasks,” ¶2 “FlowSense uses the generic variable ⟨column⟩ in its grammar as a special utterance placeholder. At runtime, a real column name (e.g. “mpg”) is automatically extracted from the dataset. FlowSense identifies column names on the fly as the user types the query. “mpg” would show up as a tagged column, and then matched with ⟨column⟩ by the parser,” Section 3.3.2, ¶1 “It is possible to have ambiguity when multiple possible query derivations exist,” Page 5, Col. 2, Fig. 3, Caption: “Special utterances are identified by unique colors. (a) Query auto completion suggestions; (b) Special utterance token completion: “scatter plot” is presented after the letter “R” is entered; (c) Dropdown for handling tagging ambiguity: “mpg” are both column name and node label,” Fig. 3 depicts generating a set of query strings in a dropdown menu through “query and token auto-completion” based on the set of candidate query string items such as “mpg” or “scatterplot” using grammar-based “derivation”).
Yu further teaches generate the code unit, wherein the code unit comprises a top N query string of the set of query strings and a wrapper (Pages 4-5, Col. 2, Section 3.3.2, ¶1 “we choose to resolve certain syntactic ambiguity in the parsing phase with supervised learning on a weight vector… that gives the probability of derivations based on input utterances. Stochastic gradient descent (SGD) is employed to optimize the multiclass hinge loss objective… The objective is given by:
PNG
media_image1.png
68
617
media_image1.png
Greyscale
Page 5, Col. 1, ¶1 “In the above, x is the input query, y is the preferred derivation, and y’ is a derivation choice… The objective function has a penalty for possible choices of incorrect predictions that are within a margin of one from the correct predictions. The parser fits the training examples by giving intended derivations higher probability so that they are preferred in case of ambiguity,” wherein “a margin” determines a top N query string or “intended derivations,” Section 3.4, ¶2 “We design a broad set of variables and rules that are able to not only accept queries with a particular component order, but also their different arrangements. For instance, “Show mpg and horsepower in a scatterplot” is equivalent to “Show a scatterplot of mpg and horse power”. They both can be accepted by FlowSense. FlowSense is also able to derive multiple functions from one single query and execute their combination, e.g. “Show the cars with mpg greater than 15 in a scatterplot” infers both visualization and filtering functions,” wherein “function” encompasses the code unit, as explained above, Page 6, Col. 1, Section 4.3, ¶1 “After the parser identifies the existing key components of a query, FlowSense attempts to fill in the blanks where information is missing using default values or the diagram editing focus,” Fig. 4 depicts “Query Pattern Completion,” wherein values used to fill in “Incorrect/Missing Information” encompass and a wrapper).
Yu further teaches and generate and display, at a graphical user interface, a visualization comprising the code unit (Page 6, Col. 2, Section 4.4, ¶1 “Once a query is successfully completed, FlowSense performs the VisFlow function(s) with the given function options. This typically results in the creation of one or more nodes, e.g. the visualization function creates one plot while the highlighting function creates three nodes,” Page 1, Fig. 1, Page 3, Table 1, and Page 7, Fig. 5 depict various examples of visualization at a graphical user interface based on “function(s)” or the code unit).
Yu and Kumar are analogous art to the claimed invention as both are from the same field of endeavor of using natural language to generate structured data. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the invention, to combine the computer-readable medium, Q&A assistant engine of a multimodal system, and execution of a neural network of Kumar with the instructions comprising the operations of Yu. The motivation to do so is “to improve the accuracy or relevance of the content generated” (Kumar, ¶186).
Regarding claim 3, Yu in view of Kumar teaches the media of claim 1 (and thus the rejection of claim 1 is incorporated).
Yu further teaches wherein the code unit is executable against the block-based data structure to generate, at least in part, a result set responsive to the natural language prompt (Page 5, Col. 1, Section 3.4, ¶2 “FlowSense is also able to derive multiple functions from one single query and execute their combination,” Page 6, Col. 2, Section 4.4, ¶1 “Once a query is successfully completed, FlowSense performs the Vis Flow function(s) with the given function options. This typically results in the creation of one or more nodes, e.g. the visualization function creates one plot while the highlighting function creates three nodes… Additionally, a query may operate on multiple existing nodes at once, e.g. linking and merging two tables create edges between two nodes”).
Regarding claim 4, Yu in view of Kumar teaches the media of claim 3 (and thus the rejection of claim 3 is incorporated).
Regarding the limitation wherein the code unit is executable against a set of blocks to generate a result set that comprises a first item in a first modality and a second item in a second modality, Yu teaches wherein the code unit is executable against a set of blocks to generate a result set (Page 5, Col. 1, Section 3.4, ¶2 and Page 6, Col. 2, Section 4.4, ¶1, see claim 3 above) that comprises a first item in a first modality (Page 2, Col. 1, Section 2.1, ¶1 “we focus on dataflow systems for visualization purposes… we build FlowSense for VisFlow, as its subset flow model supports many of the low-level visual data analysis tasks… such as characterizing distribution, finding extremum…,” Page 1, Fig. 1, Page 3, Table 1, and Page 7, Fig. 5 depict various “visualizations” including graphs, plots, and maps, encompassing a first item in a first modality). However, Yu fails to teach a result set that comprises a first item in a first modality and a second item in a second modality.
Kumar teaches a result set that comprises a second item in a second modality (Fig. 2 – 260, 266, ¶189 “Each assistant service 250, 260, 270 may also include a set of software plugins that are adapted to perform specific functionality with respect to one or more of the platforms,” ¶190 “The second assistant service 260 may be directed to an issue tracking assistant… the set of plugins 266 includes a structured query plugin, which may be adapted to generate and execute structured queries based on natural language prompts… The set of plugins 266 also includes an event summary plugin, which may be used to extract and summarize a series of entries, comments, or events that occur with respect to an issue or other content item,” wherein “a series of entries, comments, or events” encompass and a second item in a second modality).
Yu and Kumar are analogous art to the claimed invention as both are from the same field of endeavor of using natural language to generate structured data. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the invention, to combine the result set of a second modality of Kumar with the result set of a first modality of Yu. The motivation to do so is “to improve the accuracy or relevance of the content generated” (Kumar, ¶186).
Regarding claim 6, Yu in view of Kumar teaches the media of claim 1 (and thus the rejection of claim 1 is incorporated).
Yu further teaches wherein the natural language prompt is associated with or comprises an item that specifies a format of the code unit (Page 4, Col. 1, Section 3.3, ¶1 “FlowSense applies a semantic parser to map an NL query to one of the VisFlow functions based on an elaborate grammar designed for these functions… Below is an example rule:
PNG
media_image2.png
43
626
media_image2.png
Greyscale
… ⟨VisualizationType⟩ stands for a phrase that describes a visualization metaphor such as scatterplot or parallel coordinates”), and wherein the instructions, when executed by the at least one data processor of the computing system (Page 12, Section A, ¶1 “We provide an open source repository that contains the details of the FlowSense implementation… This repository includes the grammar rules”), cause the computing system to generate at least one of the top N query string or the wrapper according to the specified format of the code unit (Page 5, Col. 1, ¶1 “The parser fits the training examples by giving intended derivations higher probability so that they are preferred in case of ambiguity. In particular, the rule that expands to a column special utterance will be preferred over a rule that expands to a wildcard” and Section 3.4, ¶1 “The three columns “mpg, horsepower, and origin” indicate the options (i.e. what to visualize) for the visualization function. The phrase “in a parallel coordinates plot” indicates a new visualization node of the given visualization type is to be created as the target node,” Page 4, Fig. 2, Caption: “The result of executing this query is to create a parallel coordinates plot of columns mpg, horsepower, and origin”).
Regarding claim 7, Yu in view of Kumar teaches the media of claim 6 (and thus the rejection of claim 6 is incorporated).
Regarding the limitation wherein the format of the code unit specifies a call to an application programming interface (API) function executable against the block-based data structure, Yu teaches wherein the format of the code unit (Page 4, Col. 1, Section 3.3, ¶1 “FlowSense applies a semantic parser to map an NL query to one of the VisFlow functions based on an elaborate grammar designed for these functions… ⟨VisualizationType⟩ stands for a phrase that describes a visualization metaphor such as scatterplot or parallel coordinates”) specifies a call to… a function executable against the block-based data structure (Page 6, Col. 2, Section 4.4, ¶1 “Once a query is successfully completed, FlowSense performs the VisFlow function(s) with the given function options. This typically results in the creation of one or more nodes, e.g. the visualization function creates one plot,” wherein “node” encompasses block, as explained above with respect to claim 1). However, Yu fails to teach wherein the format of the code unit specifies a call to an application programming interface (API) function…
Kumar teaches wherein a type of prompt specifies a call to an application programming interface (API) function (Fig. 1 – 114, 116, ¶155 “Output of the prompt management service 114 can be referred to as a modified prompt or a preconditioned prompt. This modified prompt can be provided to the generative output service 116 as an input. More particularly, the prompt management service 114 is configured to structure an API request to the generative output service 116. The API request can include the modified prompt as an attribute of a structured data object that serves as a body of the API request… One example of such an API request is a POST request to a Restful API endpoint served by the generative output service 116”).
Yu and Kumar are analogous art to the claimed invention as both are from the same field of endeavor of using natural language to generate structured data. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the invention, to combine the API call of Kumar with the code unit format and function executable against the block-based data structure of Yu. The motivation to do so is “to improve the accuracy or relevance of the content generated” (Kumar, ¶186).
Regarding claim 8, Yu in view of Kumar teaches the media of claim 1 (and thus the rejection of claim 1 is incorporated).
Yu further teaches wherein the instructions, when executed by the at least one data processor of the computing system, cause the computing system to determine the top N query string by determining a predictive accuracy indicator for a particular query string (Page 5, Col. 1, ¶1 “x is the input query, y is the preferred derivation, and y′ is a derivation choice… The objective function has a penalty for possible choices of incorrect predictions that are within a margin of one from the correct predictions. The parser fits the training examples by giving intended derivations higher probability so that they are preferred in case of ambiguity”).
Claims 9, 11-12, and 14-16 are method claims which contain similar limitations to the computer-readable medium of claims 1, 3-4, and 6-8, respectively. Therefore, claims 9, 11-12, and 14-16 are rejected under substantially the same rationale as claims 1, 3-4, and 6-8, respectively.
Claims 17 and 19-20 are system claims which contain similar limitations to the computer-readable medium of claims 1 and 3-4, respectively. Therefore, claims 17 and 19-20 are rejected under substantially the same rationale as claims 1 and 3-4, respectively.
Claims 2, 10, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Kumar and further in view of Wang et al. (“RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers,” 2021, hereinafter Wang).
Regarding claim 2, Yu in view of Kumar teaches the media of claim 1 (and thus the rejection of claim 1 is incorporated).
Regarding the limitation wherein the trained neural network is trained on two or more of: (i) block types, (ii) block dependencies, (iii) block content values, (iv) block content types, or (v) block format, Kumar teaches wherein the trained neural network is trained on previous queries (¶180 “The intent recognition model may include a transformer model that has been trained using a corpus of previous user input or queries… In some cases, the intent recognition model is a bidirectional encoder representation transformer model that has been trained using a training corpus that includes, but… not limited to, previous interactions,”). However, the combination of Yu and Kumar fails to teach wherein the trained neural network is trained on two or more of: (i) block types, (ii) block dependencies, (iii) block content values, (iv) block content types, or (v) block format.
Wang, in the same field of endeavor, teaches wherein a neural network is trained on two or more of: (i) block types, (ii) block dependencies, (iii) block content values, (iv) block content types, or (v) block format (Page 4, Table 1 and Col. 1, Fig. 2 and Section 4.1, ¶¶2-3 “each column has a type T ∈ {number, text}. Formally, we represent the database schema as a directed graph G = <V,E>. Its nodes V = C ∪ T are the columns and tables of the schema, each labeled with the words in its name (for columns, we prepend their type T to the label). Its edges are defined by the pre-existing database relations described in Table 1. Figure 2 illustrates an example graph (with a subset of actual edges and labels),” wherein “a type T” of a column “node” in the “directed graph” encompasses (i) block types and “edges” encompass (ii) block dependencies, Page 3, Col. 1, Section 3, ¶1 “we introduce relation-aware self-attention, a model for embedding the semi-structured input sequences in a way that jointly encodes pre-existing relational structure in the inputs,” Col. 2, ¶1 “RAT provides a way to communicate known relations to the encoder by adding their representations to the attention mechanism,” ¶3 “The RAT framework represents all the pre-existing features for each edge (i, j)… where each pij(s) is either a learned embedding for the relation R(s) if the relation holds for the corresponding edge (i.e. if (i,j) ∈ R(s)), or a zero vector of appropriate size,” Page 7, Col. 1-2, Section 5.1, ¶¶1-2 “We use the Spider dataset… for most of our experiments … Spider contains 8,659 examples (questions and SQL queries, with the accompanying schemas)… we perform most evaluations… using the development set. It contains 1,034 examples, with databases and schemas distinct from those in the training set,” Pages 7-8, Col. 2, Table 4 and Section 5.2, ¶3 “Table 4 shows an ablation study over different RAT-based relations. The ablations are run on RAT-SQL without value-based linking to avoid interference with information from the database. Schema linking and graph relations make statistically significant improvements,” suggesting that “the model” is trained to learn “graph relations” in the “database schema” consisting of “nodes” and “edges” or (i) block types, (ii) block dependencies).
Yu, Kumar, and Wang are analogous art to the claimed invention as all are in the same field of endeavor of using natural language to generate structured data. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the trained neural network of Kumar and the training data comprising block types and block dependencies of Wang with the instructions comprising the operations of Yu. The motivation to do so is to “to improve the accuracy or relevance of the content generated” (Kumar, ¶186) and “to gain significant state of the art improvement on text-to-SQL parsing… This representation learning will be beneficial in tasks beyond text-to-SQL, as long as the input has some predefined structure” (Wang, Page 9, Col. 2, Section 6, ¶2).
Claims 10 and 18 are a method and system claim, respectively, which contain similar limitations to the computer-readable medium of claim 2. Therefore, claims 10 and 18 are rejected under substantially the same rationale as claim 2.
Claims 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Kumar and further in view of Xu et al. (“In-IDE Code Generation from Natural Language: Promise and Challenges,” 2021, herein after Xu).
Regarding claim 5, Yu in view of Kumar teaches the media of claim 4 (and thus the rejection of claim 4 is incorporated).
Regarding the limitation wherein the top N query string is a first top N query string that relates to a first query executable to retrieve items in the first modality, and wherein the code unit further comprises a second top N query string that relates to a second query executable to retrieve items in the second modality, Yu teaches wherein the top N query string is a first top N query string that relates to a first query executable to retrieve items in the first modality (Page 5, Col. 1, ¶1 “x is the input query, y is the preferred derivation, and y’ is a derivation choice… The objective function has a penalty for possible choices of incorrect predictions that are within a margin of one from the correct predictions. The parser fits the training examples by giving intended derivations higher probability so that they are preferred in case of ambiguity,” Section 3.4, ¶2 “We design a broad set of variables and rules that are able to not only accept queries with a particular component order, but also their different arrangements… FlowSense is also able to derive multiple functions from one single query and execute their combination,” Page 6, Col. 2, Section 4.4, ¶1 “Once a query is successfully completed, FlowSense performs the Vis Flow function(s) with the given function options. This typically results in the creation of one or more nodes, e.g. the visualization function creates one plot while the highlighting function creates three nodes… Additionally, a query may operate on multiple existing nodes at once, e.g. linking and merging two tables create edges between two nodes,” Page 3, Table 1, Row E depicts the “Highlighting” function derived from a sample query which “creates three nodes” and displays three respective visualizations for the user, encompassing to retrieve items in the first modality). However, Yu fails to teach and wherein the code unit further comprises a second top N query string that relates to a second query executable to retrieve items in the second modality.
Kumar teaches and wherein the code unit (Fig. 2 – 260, 264, ¶185 “a decision engine, which may include code or scripts for defining a sequence of reasoning and retrieval actions, which may be used to propose and evaluate alternatives… The decision engine may also be adapted to execute one or more of the proposed actions,” ¶187 “each assistant service 250, 260, 270 includes a respective configuration module 254, 264, 274 that is adapted for the subject-matter expertise or specialized operations used for each respective assistant service… Each respective configuration module 254, 264, 274 may, for example, include distinct decision engines,” wherein “a decision engine, which may include code or scripts” encompasses the code unit) further comprises a second… query string that relates to a second query executable to retrieve items in the second modality (Fig. 7 – 750, ¶255 “the issue tracking system assistant service utilizes a structured query plugin in order to construct a structured query that can be executed on a data store (e.g., database) of the issue tracking platform… the structured query plugin uses a specific predetermined prompt that is adapted to cause a generative output engine to provide a generative output that is formatted in accordance with a specific structured query format, that may be specific to the issue tracking platform or database… The predetermined prompt text may be adapted or amended by the structured query plugin to include portions of the user input… The structured query plugin may then be configured to use the generative response to conduct a structured query on the issue tracking platform and return the requested results in the response 750. In this example, the response 750 includes a link 752, which may be selected to view the list of results (e.g., the list of issues identified using the structured query)”). However, Kumar fails to teach a second top N query string…
Xu, in the same field of endeavor, teaches a second top N query string that relates to a second query (Page 111:7, Fig. 2 and Page 111:8, ¶1 “Figure 2 illustrates the plugin’s user interface… A popup appears at the current cursor position (Figure 2a), and the user can enter a command in natural language that they would like to be realized in code… The plugin then sends the request to the underlying code generation and code retrieval systems, and displays a ranked list of results, with the top 7 code generation results at the top, followed by the top 7 code retrieval results (Figure 2b)”).
Yu, Kumar, and Xu are analogous art to the claimed invention as all are in the same field of endeavor of using natural language to generate structured data. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the second top N query string of Xu and the second query string to retrieve items in the second modality of Kumar with the first top N query string to retrieve items in the first modality of Yu. The motivation to do so is to “to improve the accuracy or relevance of the content generated” (Kumar, ¶186) and to use natural language as “a useful medium to turn ideas into code, even for experienced programmers” (Xu, Page 111:27, ¶3).
Claim 13 is a method claim which contains similar limitations to the computer-readable medium of claim 5. Therefore, claim 13 is rejected under substantially the same rationale as claim 5.
Documents Considered but Not Relied Upon
“Natural Language Interfaces to Data,” 2022, by Quamar et al., hereinafter Quamar. Quamar is a survey detailing references in which natural language processing is used to help non-technical users generate structured queries. Among these references is Wang’s “RAT-SQL.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM M LEE whose telephone number is (571)272-4761. The examiner can normally be reached Mon-Fri. 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM M LEE/
Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights