Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the communication filed on December 13, 2024.
Claims 1-20 are examined and are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Du et al (US 2005/0094137 A1), in view of Xu et al (US 12,332,896 B1).
As per claim 1, Du discloses:
- a computer-implemented method performed by one or more computing devices, comprising (a computer implemented method, Para [0007], line 1-10),
- receiving a natural language description of a requested data pipeline (processing natural language description by requesting a data pipeline, Para [0006], [0030], [0082] – [0083]),
- generating a model prompt comprising the received natural language description and a prompt template, wherein (model prompt with received natural language description, Fig. 2, item 2002, 2004, Para [0080],
- the prompt template comprises a set of action labels and a processing example (node in a graph to process set of action, such as upload an image, drop here, rotate, scale, etc., Fig. 4A-4D, Para [0251]),
- each action label in the set of action labels indicates a respective data processing action (processing data in each action, Para [0007], [0080], [0103], [0281], Fig. 4A-4D),
- the processing example includes a sample query comprising a sample natural language description of a sample data pipeline and a sample answer comprising one or more sample action labels associated with the sample natural language description of the sample data pipeline, each of the one or more sample action labels being included in the set of action labels (examples with questions and answer set (i.e., sample query and sample answer) with sample natural language description and action labels, Para [0116] – [0166], [0174] – [0183],
- generating a project template from the one or more executable nodes, wherein the project template comprises the one or more executable nodes and one or more connections associated with the one or more executable nodes (generating software code (i.e., project template), for pipeline node, Para [0030], [0037] – [0038], [0060]),
- and generating the requested data pipeline from the project template (generating a data pipeline, Para [0040], [0043]),
Du does not explicitly disclose querying a multimodal model (MM) with the model prompt; receiving an MM response from the MM, wherein the MM response comprises one or more action labels corresponding to the natural language description of the requested data pipeline in a format guided by the prompt template; identifying one or more executable nodes in an executable node library, each executable node corresponding to a respective action label in the MM response and configured to perform the data processing action associated with the respective action label. However, in the same field of endeavor Xu in an analogous art disclose querying a multimodal model (MM) with the model prompt (querying a large language model (i.e., multimodal model), Fig. 1A, column 9, line 40-50, receiving an MM response from the MM, wherein the MM response comprises one or more action labels corresponding to the natural language description of the requested data pipeline in a format guided by the prompt template (response to the question or query or input, column 4, line 60-67, column 6, line 25-35, Fig. 1A), identifying one or more executable nodes in an executable node library, each executable node corresponding to a respective action label in the MM response and configured to perform the data processing action associated with the respective action label (identifying node to response to user input query or question and perform respective action, Fig. 1A, 1C-1D, item 134, 122, column 11, line 15-30, column 15, line 40-60).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate multimodal query with the model prompt and identifying node to execute the response to the query input as taught by Xu as the means to process natural language query with description in a data pipeline in Du, (Du, Para [0006], [0030], [0082] – [0083], Xu, Fig. 1A, column 9, line 40-50). Du and Xu are analogous prior art since they both deal with processing natural language query with description in a multimodal model or generative model. A person of the ordinary skill in the art would have been motivated to make aforementioned modification to develop code for a particular action in a natural language question description in an efficient manner. This is because one aspect of Du develops software code using graphical representations of functionalities instead of writing traditional text-based code in an accurate and efficient way, as described in at least Para [0029]. Querying a multimodal data model in part of this process. However, Du doesn’t specify any particular manner in which query multimodal model and identify node to answer the natural language description query. This would have lead one of the ordinary skill in the art to seek and recognize the Querying a multimodal data model with user prompt as taught by Xu. Xu describes querying large language multimodal model using a graph construction in an efficient manner as described at least in column 7, line 25-30, as desired by Du.
As per claim 2, rejection of claim 1 is incorporated, and further Du discloses:
- wherein the generating the requested data pipeline comprises generating software code that, when executed, performs the requested data pipeline by combining, as provided in the project template, the one or more executable nodes configured to perform respective data processing actions (software code to generate data pipeline and execute in a node to perform a task (i.e., action), Abstract, line 12-18, Para [0030], [0037] – [0038], [0060], Fig. 1, 4A-4D).
As per claim 3, rejection of claim 1 is incorporated, and further Du discloses:
- the sample answer of the prompt template further comprises, for each of the one or more action labels associated with the sample natural language description, a data source label identifying a data source on which the data processing action of the action label will operate (uploading images (i.e., data) from different sources (i.e., identifying data sources), Para [0257], [0264], [0296], [0090]),
- the MM response further comprises one or more data source labels, each data source label associated with a respective one of the one or more action labels in the MM response (multimodal response with respective action, Para [0116] – [0166]).
As per claim 4, rejection of claim 3 is incorporated, and further Du discloses:
- wherein two or more action labels in the MM response are associated with identical data source labels (drop here, item 145 and 150 (two or more action labels) are associated with input image from a camera (i.e., identical data source), Fig. 4A, item 105-125, 140-165).
As per claim 5, rejection of claim 1 is incorporated, and further Du discloses:
- wherein the prompt template comprises additional processing examples, each additional processing example including an additional sample query and an additional sample answer (more than one question and answer (i.e. additional processing example with sample query and sample answer), Para [0116] – [0166], [0174] – [0183]).
As per claim 6, rejection of claim 1 is incorporated, and further Xu discloses:
- prior to receiving the natural language description: receiving, from a user, an initial natural language request (initial request prior to description, Fig. 1C, item 168, 172, Fig. 1, column 17, line 40-55),
- querying the MM with the initial natural language request to generate the natural language description of the requested data pipeline (Fig. 1, item 106, 134, searching large language model (i.e., multimodal model), column 7, line 20-30, column 9, line 55-65).
As per claim 7, rejection of claim 1 is incorporated, and further Du discloses:
- generating the set of action labels on-the-fly based on a listing of available executable nodes in the executable node library (quickly initialize a pipeline for an action available in the node library, Para [0046], [0073] – [0074], Fig. 1, item 40, Fig. 4-5).
As per claim 8, rejection of claim 1 is incorporated, and further Du discloses:
- generating a user interface to visualize the project template, wherein the user interface provides a preview mode comprising the project template, wherein the preview mode visualizes the one or more executable nodes and the one or more connections associated with the one or more executable nodes (visualizing project template, Fig. 5, item 565, preview to visualize connection of nodes, Fig. 5, Fig 4A-4D, Para [0056] – [0057], Para [0240]).
As per claim 9, rejection of claim 1 is incorporated, and further Xu discloses:
- wherein at least one of the set of action labels, one or more processing examples, or the one or more executable nodes are pre- defined in a database (nodes are predefined in a data base, Fig. 4, item 414, 428, column 25, line 35-45).
As per claim 10, rejection of claim 9 is incorporated, and further Xu discloses:
- updating the database by adding, to the database, at least one of a new action label, a new processing example, or a new executable node (modifying graph by adding or removing node or edge, column 16, line 45-58, Fig. 4).
As per claim 11, rejection of claim 9 is incorporated, and further Xu discloses:
- updating the database by deleting, from the database, at least one of the one or more action labels, the one or more processing examples, or the one or more executable nodes (modifying graph by adding or removing node or edge, column 16, line 45-58, Fig. 4).
As per claim 12, rejection of claim 1 is incorporated, and further Xu discloses:
- parsing the MM response to extract the one or more action labels (parsing unstructured data (i.e., parsing response), column 4, line 5-10, column 7, line 1-10, Fig. 1, 4-5).
- determining, for each extracted action label, a correlation between the extracted action label and the one or more executable nodes (relationship between dispersed pieces (i.e., correlation), column 2, line 55-67, column 6, line 1-15),
- selecting, from the executable node library for each extracted action label, a respective one of the one or more executable nodes correlating to the extracted action label or a default executable node based on the correlation between the extracted action label and the one or more executable nodes being low (selecting node to perm a task, column 14, line 60-67, column 16, line 45-55).
As per claim 13, rejection of claim 1 is incorporated, and further Xu discloses:
- wherein the one or more executable nodes collectively identify at least two types of data processing actions of the data pipeline (matching data with desired type of response (i.e., at lest two types of action), column 4, line 10-20, column 14, line 25-35, column 24, line 45-55, Fig. 4, 1).
As per claims 13-17 and 19,
Claims 13-17 and 19 are system claims corresponding to method claims 1, 3, 6, 8 and 12 respectively and rejected under the same reason set forth to the rejection of claims 1, 3, 6, 8 and 12 above.
As per claim 18, rejection of claim 14 is incorporated, and further Du discloses:
- updating, using a user interface, the project template based on additional information received from the user, wherein the additional information comprises one or more edits associated with at least one parameter of the one or more executable nodes or the one or more connections associated with the one or more executable nodes (edit to modify the code, Para [0005], [0008], [0032], [0061]).
As per claim 20,
Claim 20 is a computer readable medium claim corresponding to method claim 1 respectively and rejected under the same reason set forth to the rejection of claim 1 above.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMMED R UDDIN whose telephone number is (571)270-3138. The examiner can normally be reached M-F: 9:00 AM-5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Beausoliel Robert can be reached at 571-272-3645. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMMED R UDDIN/Primary Examiner, Art Unit 2167