Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant’s arguments, see Remarks, filed 07/29/2026, Examiner concurs that Zhang is not qualifying prior art. The reference has therefore been withdrawn, and a new non-final action is provided herein. See below.
Note: The claims are not directed towards patent ineligible subject matter under 35 U.S.C. 101
Step 1: IS THE CLAIM DIRECTED TO A PROCESS, MACHINE, MANUFACTURE OR COMPOSITION OF MATTER?
Yes
Step 2A.1: IS THE CLAIM DIRECTED TO A LAW OF NATURE, A NATURAL PHENOMENON (PRODUCT OF NATURE) OR AN ABSTRACT IDEA?
No
Step 2A.2: DOES THE CLAIM RECITE ADDITIONAL ELEMENTS THAT INTEGRATE THE JUDICIAL EXCEPTION INTO A PRACTICAL APPLICATION?
Yes, if the claims are alternatively construed to be abstract in step 2A1. The claims seek to improve the accuracy of the LLM result of image-to-text supported by the specification, and reflected by the claims e.g. in spec: 0036 0037 0095 In other words, the claims enable the invention to accuracy and save resources for converting image to understanding.
Supported by the following:
In Finjan Inc. v. Blue Coat Systems, Inc., 879 F.3d 1299, 125 USPQ2d 1282 (Fed. Cir. 2018), the claimed invention was a method of virus scanning that scans an application program, generates a security profile identifying any potentially suspicious code in the program, and links the security profile to the application program. 879 F.3d at 1303-04, 125 USPQ2d at 1285-86. The Federal Circuit noted that the recited virus screening was an abstract idea, and that merely performing virus screening on a computer does not render the claim eligible. 879 F.3d at 1304, 125 USPQ2d at 1286. The court then continued with its analysis under part one of the Alice/Mayo test by reviewing the patent’s specification, which described the claimed security profile as identifying both hostile and potentially hostile operations. The court noted that the security profile thus enables the invention to protect the user against both previously unknown viruses and “obfuscated code,” as compared to traditional virus scanning, which only recognized the presence of previously-identified viruses. The security profile also enables more flexible virus filtering and greater user customization. 879 F.3d at 1304, 125 USPQ2d at 1286. The court identified these benefits as improving computer functionality, and verified that the claims recite additional elements (e.g., specific steps of using the security profile in a particular way) that reflect this improvement. Accordingly, the court held the claims eligible as not being directed to the recited abstract idea. 879 F.3d at 1304-05, 125 USPQ2d at 1286-87. This analysis is equivalent to the Office’s analysis of determining that the additional elements integrate the judicial exception into a practical application at Step 2A Prong Two, and thus that the claims were not directed to the judicial exception (Step 2A: NO).
Examples of claims that improve technology and are not directed to a judicial exception include: Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1339, 118 USPQ2d 1684, 1691-92 (Fed. Cir. 2016) (claims to a self-referential table for a computer database were directed to an improvement in computer capabilities and not directed to an abstract idea); McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 1315, 120 USPQ2d 1091, 1102-03 (Fed. Cir. 2016) (claims to automatic lip synchronization and facial expression animation were directed to an improvement in computer-related technology and not directed to an abstract idea); Visual Memory LLC v. NVIDIA Corp., 867 F.3d 1253,1259-60, 123 USPQ2d 1712, 1717 (Fed. Cir. 2017) (claims to an enhanced computer memory system were directed to an improvement in computer capabilities and not an abstract idea); Finjan Inc. v. Blue Coat Systems, Inc., 879 F.3d 1299, 125 USPQ2d 1282 (Fed. Cir. 2018) (claims to virus scanning were found to be an improvement in computer technology and not directed to an abstract idea); SRI Int’l, Inc. v. Cisco Systems, Inc., 930 F.3d 1295, 1303 (Fed. Cir. 2019) (claims to detecting suspicious activity by using network monitors and analyzing network packets were found to be an improvement in computer network technology and not directed to an abstract idea). Additional examples are provided in MPEP § 2106.05(a).
Regarding the December 5th 2025 Memo in light of September 26, 2025 Appeals Review Panel Decision in Ex parte Desjardins, Appeal 2024-000567 for Application 16/319,040, in deciding if a recited abstract idea does or does not direct the entire claim to an abstract idea, when a claim is considered as a whole:
Paragraph 21 of the Specification, which the Appellant cites, identifies improvements in training the machine learning model itself. Of course, such an assertion in the Specification alone is insufficient to support a patent eligibility determination, absent a subsequent determination that the claim itself reflects the disclosed improvement. See MPEP § 2106.05(a) (citing Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1316 (Fed. Cir. 2016)). Here, however, we are persuaded that the claims reflect such an improvement. For example, one improvement identified in the 8 Appeal2024-000567 Application 16/319,040 Specification is to "effectively learn new tasks in succession whilst protecting knowledge about previous tasks." Spec. ,r 21. The Specification also recites that the claimed improvement allows artificial intelligence (AI) systems to "us[e] less of their storage capacity" and enables "reduced system complexity." Id. When evaluating the claim as a whole, we discern at least the following limitation of independent claim 1 that reflects the improvement: "adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task." We are persuaded that constitutes an improvement to how the machine learning model itself operates, and not, for example, the identified mathematical calculation. Under a charitable view, the overbroad reasoning of the original panel below is perhaps understandable given the confusing nature of existing § 101 jurisprudence, but troubling, because this case highlights what is at stake. Categorically excluding AI innovations from patent protection in the United States jeopardizes America's leadership in this critical emerging technology. Yet, under the panel's reasoning, many AI innovations are potentially unpatentable-even if they are adequately described and nonobvious-because the panel essentially equated any machine learning with an unpatentable "algorithm" and the remaining additional elements as "generic computer components," without adequate explanation. Dec. 24. Examiners and panels should not evaluate claims at such a high level of generality.
Specifically, Ex Parte Desjardins explained the following:
Enfish ranks among the Federal Circuit's leading cases on the eligibility of technological improvements. In particular, Enfish recognized that “[m]uch of the advancement made in computer technology consists of improvements to software that, by their very nature, may not be defined by particular physical features but rather by logical structures and processes.” 822 F.3d at 1339. Moreover, because “[s]oftware can make non-abstract improvements to computer technology, just as hardware improvements can,” the Federal Circuit held that the eligibility determinations should turn on whether “the claims are directed to an improvement to computer functionality versus being directed to an abstract idea.” Id. at 1336. (Desjardins, page 8).
Further in Ex Parte Desjardins, Appeal No. 2024-000567 (PTAB September 26, 2025, Appeals Review Panel Decision) (precedential), the claimed invention was a method of training a machine learning model on a series of tasks. The Appeals Review Panel (ARP) overall credited benefits including reduced storage, reduced system complexity and streamlining, and preservation of performance attributes associated with earlier tasks during subsequent computational tasks as technological improvements that were disclosed in the patent application specification. Specifically, the ARP upheld the Step 2A Prong One finding that the claims recited an abstract idea (i.e., mathematical concept). In Step 2A Prong Two, the ARP then determined that the specification identified improvements as to how the machine learning model itself operates, including training a machine learning model to learn new tasks while protecting knowledge about previous tasks to overcome the problem of “catastrophic forgetting” encountered in continual learning systems. Importantly, the ARP evaluated the claims as a whole in discerning at least the limitation “adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task” reflected the improvement disclosed in the specification. Accordingly, the claims as a whole integrated what would otherwise be a judicial exception instead into a practical application at Step 2A Prong Two, and therefore the claims were
The claim itself does not need to explicitly recite the improvement described in the specification (e.g., “thereby increasing the bandwidth of the channel”). See, e.g., Ex Parte Desjardins, Appeal No. 2024-000567 (PTAB September 26, 2025, Appeals Review Panel Decision) (precedential), in which the specification identified the improvement to machine learning technology by explaining how the machine learning model is trained to learn new tasks while protecting knowledge about previous tasks to overcome the problem of “catastrophic forgetting,” and that the claims reflected the improvement identified in the specification. Indeed, enumerated improvements identified in the Desjardins specification included disclosures of the effective learning of new tasks in succession in connection with specifically protecting knowledge concerning previously accomplished tasks; allowing the system to reduce use of storage capacity; and the enablement of reduced complexity in the system. Such improvements were tantamount to how the machine learning model itself would function in operation and therefore not subsumed in the identified mathematical calculation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20230259709 A1 Dancewicz; Adam et al. (hereinafter Dancewicz) in view of US 20260127691 A1 Tran; Bao (hereinafter Tran).
Re claim 1, Dancewicz teaches
1. A method comprising:
identifying, by a processing device, one or more layout elements included in a schematic representation in digital content; (using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059)
generating, by the processing device, schematic layout data by filtering the content stream and identifying data points associated with the schematic representation based on the filtered content stream; and (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059)
outputting, by the processing device, a schematic understanding result based on the schematic layout data. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
extracting, by the processing device, a content stream from the digital content usable to render the schematic representation; (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Re claim 2, Dancewicz teaches
2. The method as described in claim 1, wherein the schematic representation is configured as a chart, a circuit diagram, a flow diagram, or a translation invariant image representation. (chart, table, etc… extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 3, Dancewicz teaches
3. The method as described in claim 1, wherein the schematic representation is a translation invariant image representation that contains abstractions and conveys relative component interactions. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 4, Dancewicz teaches
4. The method as described in claim 1, wherein the one or more layout elements identify a title, an axis title, a location of respective said layout elements in relation to the digital content, one or more gridlines, a data series a legend, one or more ticks describing a unit a measure used for a respective said axis, a plot area, a chart area, one or more annotations, an error bar, a trendline, a mark, shading, highlighting, a color of the respective said layout elements, one or more axis labels, or a value of the respective said layout elements. (a chart contains a variety of such examples e.g. … extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 5, Dancewicz teaches
5. The method as described in claim 1, wherein the generating includes identifying the data points as corresponding to one or more vectors of a chart configured as the schematic representation. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 6, Dancewicz teaches
6. The method as described in claim 1, wherein the filtering including filtering the content stream into a vector stream usable to render one or more vectors of the schematic representation and a text stream usable to render text associated with the schematic representation. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
Streaming (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Re claim 7, Dancewicz teaches
7. The method as described in claim 6, wherein the one or more vectors are used to plot the data points and wherein the identifying is based on the one or more vectors. (a chart with datapoints… and extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 8, Dancewicz teaches
8. The method as described in claim 1, wherein the schematic layout data includes tabular data describing values of the data points from the schematic representation. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 9, Dancewicz teaches
9. The method as described in claim 1, further comprising: detecting coordinates of the schematic representation by segmenting the digital content using at least one machine-learning model; and (neural network or machine learning model Abstract… extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
extracting the schematic representation from the digital content based on the coordinates. (coordinate per se i.e. utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7… extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059…)
Re claim 10, While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
10. The method as described in claim 1, wherein the digital content includes a plurality of layers and the generating the schematic layout data includes generating schematic layout data independently for each said layer and identifying one or connections between respective said layers. (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Re claim 11, Dancewicz teaches
11. A system comprising: one or more computer-readable storage media; and a processing device coupled to the one or more computer-readable storage media to perform operations including: (0068 computer)
segmenting a chart from digital content; (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
extracting one or more vector operations from the digital content, the one or more vector operations associated with one or more vectors included in the chart; (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
generating schematic layout data by identifying data points associated with the one or more vectors of the chart based on the one or more vector operations; and (utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7… extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059)
outputting a schematic understanding result in response to a query based at least in part on the schematic layout data using a machine-learning model. (using a machine learning model Abstract and extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
In response to a query (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Re claim 12, Dancewicz teaches
12. The system as described in claim 11, wherein the generating the schematic layout data including forming a vector stream by filtering a content stream from the digital content and the extracting is based on the filtering. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
streaming (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Re claim 13, Dancewicz teaches
13. The system as described in claim 12, wherein the filtering includes filtering the content stream into the vector stream usable to render the one or more vectors of the chart and a text stream usable to render text associated with the chart, and wherein the identifying of the data points is based on the vector stream. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 14, Dancewicz teaches
14. The system as described in claim 11, wherein the schematic layout data is configured as tabular data. (Table, chart, etc. … extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 15, Dancewicz teaches
15. One or more computer-readable storage media storing instructions that, responsive to execution by a processing device, causes the processing device to perform operations including: (0068)
identifying one or more layout elements included in a schematic representation included in digital content; (chart for instance, extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
filtering the content stream into a vector stream usable to render one or more vectors of the schematic representation and a text stream usable to render text associated with the schematic representation; (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
generating schematic layout data based on the filtering. (producing a result based on the image in a structured form… extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
extracting a content stream from the digital content usable to render the schematic representation; (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Re claim 16, Dancewicz teaches
16. The one or more computer-readable storage media as described in claim 15, wherein the schematic layout data includes tabular data describing values of data points of one or more vectors from the schematic representation. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 17, Dancewicz teaches
17. The one or more computer-readable storage media as described in claim 15, wherein the schematic representation is a chart. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 18, Dancewicz teaches
18. The one or more computer-readable storage media as described in claim 15, the operations further comprising: detecting coordinates of the schematic representation by segmenting the digital content using at least one machine-learning model; and (utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059…)
extracting the schematic representation from the digital content based on the coordinates. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
Re claim 19, Dancewicz teaches
19. The one or more computer-readable storage media as described in claim 15, the operations further comprising receiving a query and outputting a schematic understanding result is based on the query and the schematic layout data using a machine-learning model. (extracting data points expressly 0054 and as vectors thereof fig. 4b with 0039 0044 for instance the text in element 406b extracted from a chart image analogous to an understanding per se… using image to text style analysis including data comprising charts, tables, and images thereof 0005 0031 0059… also utilizing the dimensions of the data points within an image or chart 0037 with fig. 3 when extracting the layout via a neural network as in fig. 7)
While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
A query (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Re claim 20, While Dancewicz teaches a neural network to extract data from an image such as a chart, and rendering an image into a description such as with text or data points, it fails to teach real time such as an AI system in the context of a specific LLM, and accepting a user prompt to perform such rendering in real-time per se:
20. The one or more computer-readable storage media as described in claim 19, wherein the machine-learning model is a large language model (LLM). (Tran image extraction including charts tables, diagrams, Abstract 0040 0058 and assigning a description or text or structured equivalent thereof 0004 performed in real time 0101 e.g. streaming per se claim 5 under the premise of an LLM with self-attention or similar layer connection 0203 concepts to handle queries and responses as images/diagrams/charts are input within a document Claim 1)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Dancewicz to incorporate the above claim limitations as taught by Tran to allow for simple substitution of one known element for another to obtain predictable results such as using an LLM in place of a neural network well-known layer-based operations for the purposes of producing a detailed description from an user inputting an image, diagram, chart, table, etc. in a query-response real-time system, thereby rendering on-demand images and a corresponding text to extract specificity from the image if desired by the user, such as data which is visually unknown as well as preserved data which can be clearly seen and is unchanged upon rendering e.g. a simple title in graphic form, but then providing meaning to a complex image/diagram to reduce analysis time such as summarizing what the data therein to understand their significance and how they interconnect with the textual content, thus saving time to understand the response to a user input/query.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20240362208 A1 Naufel; Mark
Visual encoder to extract meaning
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL COLUCCI whose telephone number is (571)270-1847. The examiner can normally be reached on M-F 9 AM - 7 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL COLUCCI/Primary Examiner, Art Unit 2655 (571)-270-1847
Examiner FAX: (571)-270-2847
Michael.Colucci@uspto.gov
Prosecution Insights