DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to application filed on October 19, 2023.
Claims 1-20 are pending.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 6, 8-11, 14, 16-17 and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Benyamin (US 2024/0414414).
With respect to Claim 1, Benyamin discloses:
one or more computer readable storage media; (see Figure 9; memory 940)
one or more processors operatively coupled with the one or more computer readable storage media; (see Figure 9; processor 910)
and program instructions stored on the one or more computer readable storage media that, when executed by the one or more processors, direct the computing apparatus to at least: (Processor 910 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessors, and/or controllers that perform instructions stored in memory 940 to manipulate data stored in the memory. Processor instructions can configure the processor 910 to perform processes in accordance with certain embodiments of the invention, Paragraph 75)
a) receive a user request to generate a visualization; (see Figure 1; receiving a request to generate a video from a user using a user device, Abstract, lines 1-3 and Paragraph 38)
b) submit a first prompt to a foundation model to obtain code for generating an instance of the visualization requested by the user; (selecting (obtain) a template (code) based on the determined theme and input text (first prompt) using a large language model (LLM) (foundation model) used to generate the requested video, Paragraph 43; Process can request that a video (obtain code) be generated based on the determined theme and the received input text (first prompt) received by a LLM (foundation model), Paragraph 41)
c) generate an instance of the visualization using the code; (generate the requested video based on the selected template (code) at the video generation server system, Paragraphs 43 and 46)
d) submit the instance of the visualization to an image model to obtain a description of the instance; (Video generation server system 130 may transmit the generated video to the image-to-text server system 150 (image model), where the video can be described in text, Paragraph 46; see Figure 7; the extraction and describing of the at least one frame of the generated video is performed by the models (image model) deployed at image-to-text server system 150, Paragraph 48)
and e) submit a second prompt to the foundation model to obtain an evaluation of the instance of the visualization with respect to the visualization requested by the user, wherein the second prompt includes the description produced by the image model. (Process 300 evaluates (330) if the generated video is satisfactory. In numerous embodiments, this evaluation requires the LLM (foundation model) to interface with the video generation server system 130 to determine if the video is satisfactory based on the description of the generated video (from the image model) and the input text., Paragraph 46)
With respect to Claim 2, all the limitations of Claim 1 have been addressed above; and Benyamin further discloses:
wherein the program instructions further direct the computing apparatus to iteratively repeat steps b-e until the evaluation determines that the instance of the visualization satisfies the user request. (If the video generation server system 130 deems the generated video to be unsatisfactory, process 300 updates (340) the generated video. Process 300 may repeat steps 330 and 340 until the video generation server system 130 deems that the generated video is satisfactory., Paragraph 46)
With respect to Claim 3, all the limitations of Claim 2 have been addressed above; and Benyamin further discloses:
wherein the program instructions further direct the computing apparatus to display the instance of the visualization in a user interface. (The user may view (display) and determine if the selected template and generated video (instance of the visualization) are satisfactory., Paragraph 80)
With respect to Claim 6, all the limitations of Claim 1 have been addressed above; and Benyamin further discloses:
wherein the program instructions further direct the computing apparatus to:
submit a third prompt to the foundation model to obtain a second code for generating a second instance of the visualization, wherein the third prompt includes the evaluation of the instance of the visualization; (If the video generation server system 130 deems the generated video to be unsatisfactory (evaluation of the instance), process 300 updates (340) the generated video. Process 300 may repeat steps 330 and 340 until the video generation server system 130 deems that the generated video is satisfactory (submit third prompt). If the video generation server system 130 deems the generated video to be satisfactory, process 300 outputs (350) the generated video., Paragraph 46; determine if the video is satisfactory based on the description of the generated video and the input text (third prompt includes evaluation of the instance), Paragraph 46)
generate the second instance of the visualization using the second code; (see Figure 3; update generated video 340 (second code))
submit the second instance of the visualization to the image model to obtain a description of the second instance of the visualization; (The image-to-text server system (image model) can provide the description of the extracted at least one frame to the LLM and query the LLM, Paragraph 49)
submit a fourth prompt to the foundation model to obtain an evaluation of the second instance of the visualization with respect to the visualization requested by the user, wherein the fourth prompt includes the description of the second instance of the visualization; (query the LLM, which then queries the video generation server system 130 to determine if the extracted at least one frame is satisfactory. The LLM can identify portions on the at least one frame based on responses from the video generation server system 130 that need to be updated in response to the query. In numerous embodiments, the video generation server system 130 performs an analysis of the extracted at least one frame by reviewing the description provided by the image-to-text server system. (fourth prompt includes the description of the second instance of the visualization), Paragraph 49)
and display the second instance of the visualization in the user interface. (output generated video, Paragraph 46; The user may view (display) and determine if the selected template and generated video (instance of the visualization) are satisfactory., Paragraph 80
With respect to Claim 8, all the limitations of Claim 1 have been addressed above; and Benyamin further discloses:
wherein the user request is a natural language input from the user. (receiving basic text (natural language input) from a user using a user device, Paragraphs 38 and 41)
Claims 9-11 and 14 are method claims corresponding to the apparatus claims above (Claims 1-3 and 6) and, therefore, are rejected for the same reasons set forth in the rejections of Claims 1-3 and 6.
Claims 16-17 and 20 are computer readable storage media claims corresponding to the apparatus claims above (Claims 1-2 and 8) and, therefore, are rejected for the same reasons set forth in the rejections of Claims 1-2 and 8.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4, 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Benyamin (US 2024/0414414) in view of Fujitomi et al. (US 2023/0287801).
With respect to Claim 4, all the limitations of Claim 3 have been addressed above; and Benyamin further discloses:
wherein the program instructions further direct the computing apparatus to [generate] the evaluation in association with the instance of the visualization. (Process 300 evaluates (330) if the generated video is satisfactory. In numerous embodiments, this evaluation requires the LLM (foundation model) to interface with the video generation server system 130 to determine if the video is satisfactory based on the description of the generated video (from the image model) and the input text., Paragraph 46)
Benyamin does not disclose:
display the evaluation in a user interface
However, Fujitomi et al. disclose:
display the evaluation in a user interface (display a result of the performance evaluation on the display unit, Paragraph 132)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Fujitomi et al. into the teaching of Benyamin to include displaying the evaluation in a user interface in order to show/notify a user the result of an evaluation.
Claim 12 is a method claim corresponding to the apparatus claim above (Claim 4) and, therefore, is rejected for the same reasons set forth in the rejection of Claim 4.
Claim 18 is a computer readable storage media claim corresponding to the apparatus claim above (Claim 4) and, therefore, is rejected for the same reasons set forth in the rejection of Claim 4.
Claims 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Benyamin (US 2024/0414414) in view of Bales et al. (US 2017/0212829).
With respect to Claim 5, all the limitations of Claim 2 have been addressed above; and Benyamin further discloses:
wherein the program instructions further direct the computing apparatus to display the instance of the visualization in the user interface when a number of instances of the visualization reaches a [threshold]. (The user may view (display) and determine if the generated video (instance of the visualization) is satisfactory after the generated video has been iteratively updated (number of instances of the visualization) to update portions of the generated video that were deemed unsatisfactory until the LLM deems that the generated video is satisfactory (threshold)., Paragraphs 49-50 and 80)
Benyamin do not disclose:
[threshold] is a maximum value.
However, Bales et al. disclose:
[threshold] is a maximum value. (repeating the generation/updating of code for a set number of iterations (maximum value), Paragraph 96)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Bales et al. into the teaching of Benyamin to include a [threshold] is a maximum value in order to avoid potential infinite loops. (Bales et al., Paragraph 96)
Claim 13 is a method claim corresponding to the apparatus claim above (Claim 5) and, therefore, is rejected for the same reasons set forth in the rejection of Claim 5.
Claims 7, 15 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Benyamin (US 2024/0414414) in view of Paula Maddigan et al. (“Chat2VIS: Generating Data Visualizations via Natural Language Using ChatGPT, Codex and GPT-3 Large Language Models”, May 2023).
With respect to Claim 7, all the limitations of Claim 1 have been addressed above; and Benyamin does not disclose:
wherein the code comprises Python code, and wherein to generate the instance of the visualization, the program instructions direct the computing apparatus to execute the code using a Python engine.
However, Paula Maddigan et al. disclose:
wherein the code comprises Python code, and wherein to generate the instance of the visualization, the program instructions direct the computing apparatus to execute the code using a Python engine. (A user enters a NL query via a Streamlit NLI app
which is an open-source Python framework for web-based dashboards. The query is combined with a prompt script which engineers a suitable prompt for a selected dataset. The prompt is forwarded to selected LLMs, which return a Python script (python code) that is subsequently rendered within the Streamlit NLI (Python engine), Page 4, III. Methodology, Paragraph 3, lines 1-7)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Paula Maddigan et al. into the teaching of Benyamin to include wherein the code comprises Python code, and wherein to generate the instance of the visualization, the program instructions direct the computing apparatus to execute the code using a Python engine in order to be able to generate Python scripts for visualizing data based on natural language queries without explicit direction as to which type of graphs to generate. (Paula Maddigan et al., Page 4, III. Methodology, Paragraph 1, lines 1-4)
Claim 15 is a method claim corresponding to the apparatus claim above (Claim 7) and, therefore, is rejected for the same reasons set forth in the rejection of Claim 7.
Claim 19 is a computer readable storage media claim corresponding to the apparatus claim above (Claim 7) and, therefore, is rejected for the same reasons set forth in the rejection of Claim 7.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kaushal et al. (US 2024/0419711) discloses a user interface workflow for natural language querying.
Tong et al. (US 2023/0325416) discloses generation and adaptation of a visualization from one or more data sets.
Malak et al. (US 2025/0094703) discloses AI-generated data objects for data visualization.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LANNY N UNG whose telephone number is (571)270-7708. The examiner can normally be reached Mon-Thurs 6:30am-3:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bradley Teets can be reached at 571-272-3338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LANNY N UNG/ Primary Examiner, Art Unit 2197
Prosecution Insights