Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/03/2026 has been entered.
Detailed Action
Applicant amended claims 1, 12, and 23 and presented claims 1-33 for reconsideration on 08/03/2026.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-33 are rejected under 35 U.S.C. 103(a) as being unpatentable over Gong et al., Pub. No.: US 2026/0030480 A1 (Gong), in view of Tchuiev et al., “DUQIM-Net: Probabilistic Object Hierarchy Representation for Multi-View Manipulation” (Tchuiev) and further in view of Messina et al., Pub. No.: US 2018/0336405 A1 (Messina).
Claim 1 . Gong teaches:
A computer-implemented method for converting visualizations in documents, wherein the computer-implemented method comprises:
receiving, by a processor set, a number of documents, wherein the number of documents comprise a number of visualizations; (Gong, ¶ 97, documents comprise textual and non-textual data: “parsing module 304 is configured to parse content items that comprise both textual data and non-textual image data…the types of non-textual image data may include any of the following: charts and graphs, such as bar charts, line graphs, pie charts, scatter plots, histograms, area charts, bubble charts, radar charts, and Gantt charts; pictures, such as photographs, illustrations, diagrams, clip art, and infographics; and videos that can be embedded or linked, along with animated GIFs”)
identifying, by the processor set using a set of machine learning models, a number of objects within the number of documents and locations for the number of objects from the number of documents; (Gong, ¶ 97, objects such as charts, graphs, etc. are identified; ¶111, “picture LMM 308, chart and plot LMM 310, document LMM 316, table LMM 318, and other LMMs that may be employed to analyze content items may generate image descriptions or textual data from content items that include non-textual image data”)
identifying, by the processor set using the set of machine learning models, the number of visualizations from the number of objects within the number of documents; (Gong, ¶¶ 97, 111, objects such as chart, graphs, etc. are visualizations)
classifying, by the processor set, the number of visualizations into different visualization types, wherein each visualization from the number of visualizations is labeled with zero or more visualization types; (note that, labeling “with zero or more visualization types” means there is no labeling; ¶ 26, “Responsive to detecting content items with non-textual image data, the RAG agent invokes an LMM that is configured to classify non-textual image data into a variety of classifications”, ¶ 107, “When the input includes chart and plot data, image classification LMM 306 uses specialized layers to recognize axis labels, grid lines, and plotted data points, distinguishing between different chart types, such as bar graphs, scatter plots, or line charts. In the case of table data, image classification LMM 306 focuses on detecting grid structures, cell contents, and numeric or textual patterns within the table, classifying the data into appropriate table-related categories”)
parsing, by the processor set using the set of machine learning models, the number of visualizations to generate a number of graphs representing the number of visualizations based on visualization types for the number of visualizations, wherein parsing each visualization comprises applying image processing to detect visual elements and coordinates of the detected visual elements within the image of the visualization; (Gong, ¶¶ 97-99, 116-120, wherein image processing is applied for identifying a visualization “such as bar charts, line graphs, pie charts, scatter plots, histograms, area charts, bubble charts, radar charts, and Gantt charts” and visual elements such as “axes, data points, bars, lines, and labels”, the identified visualization and visual elements are represented as correlated elements to be used by an LLM, the position/coordinates of the visual elements are also identified: “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels. These layers detect and extract features specific to chart visualization, such as the placement and orientation of axes, the scale of the plot, the position and shape of data markers, and any accompanying legends or labels…For example, if classification logic identifies a bar chart with labeled axes and varying bar heights, chart and plot LMM 310 classifies the data categories (based on x-axis labels) and the corresponding values (based on y-axis positions or numerical labels). Chart and plot LMM 310 then generates text that reflects the relationships between the data points, such as, "This bar chart shows sales figures for four regions, with Region A having the highest sales at $1 ,000,000, while Region D has the lowest at $300,000."”)
extracting, by the processor set, textual information associated with the number of visualizations from pages that contain the number of visualizations in the number of documents; and (Gong, ¶¶ 97-99, 116-120, textural information such as titles, labels, etc., are extracted: “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels. These layers detect and extract features specific to chart visualization, such as the placement and orientation of axes, the scale of the plot, the position and shape of data markers, and any accompanying legends or labels”; ¶ 122, “ chart and plot LMM 310 integrates recognized text, such as axis titles or labels, into the output”)
Gong did not specifically disclose but Tchuiev discloses generate a representation a number of graphs representing the number of visualizations. (Tchuiev, sec. III, wherein identified objects and their relationships in a documents are represented as vertices and edges: a matrix is generated for representing objects and their relationships in a document: “Consider an environment with multiple stacked objects. These object are stacked in a hierarchical structure…This structure can be represented as a graph…with NV vertices… and NE edges…The vertices represent the objects themselves, while the directed edges represent the direct hierarchical relationship between these objects”)
Gong ¶¶ 119, 166 discloses that “chart and plot data 364 represents images that comprise visual representations of data, such as bar charts, line graphs, pie charts, or scatter plots. This type of image data is structured to convey quantitative or categorical information visually through axes, data points, and labels. Chart and plot data 364 typically includes various elements, like gridlines, legends, and numerical values, associated with plotted points or bars. The primary focus for processing chart and plot data 364 is extracting the relationships between the data points and generating meaningful interpretations or summaries of the visualized data”. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing generate a representation a number of graphs representing the number of visualizations because doing so would provide for an alternative representation of identified visual objects and their associations as nodes and edges for achieving the same predictable result of generating meaningful interpretations or summaries of the visualized data.
Gong did not specifically disclose but Messina discloses generating, by the processor set, a machine readable table for the number of visualizations based on the number of graphs and the textual information associated with the number of visualizations, wherein generating the machine readable table comprises associating extracted textual information with corresponding detected elements represented in the number of graphs, and merging elements detected by the parsing model with the extracted textual information. (Messina, ¶ 38, wherein an identified visual and textual data are associated/merged by aggregating the identified visual and textual data; the aggregated visual and textual data is machine readable: “The two information streams, data from visual 210, 211 and textual strings (tokens) 212 are aggregated by an aggregation module 213 to produce output in the form of a file representing the extracted data from the charts 222. The aggregation module 213 reads column labels 214, row labels 215, values 216 and titles 217 from the outputs 210-212 from the shape analysis 202 and text extraction 207 modules. The aggregation module 213 then writes to an output file the read column labels 218, row labels 219, values 220 and titles 221. The output file 222 contains meta-information about title, values, labels, colors, and layout of the chart. The result can be further exploited for data mining applications, search engines, advanced semantic queries or to re-purpose the chart in a new, more meaningful or interactive, representation”)
Gong, ¶¶ 119, 166 implicitly discloses the feature by processing the visualized data to be used by an LLM for generating meaningful interpretations or summaries of the visualized data; meaning the visualized data must have been transformed to a machine readable data to be used by an LLM. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing generating, by the processor set, a machine readable table for the number of visualizations based on the number of graphs and the textual information associated with the number of visualizations, wherein generating the machine readable table comprises associating extracted textual information with corresponding detected elements represented in the number of graphs, and merging elements detected by the parsing model with the extracted textual information because doing so would provide for generating a machine readable table explicitly for achieving the same predictable result of processing a document comprising textual and non-textual data items.
Claim 12. Gong teaches:
A computer system for converting visualizations in documents, comprising: a processor set; a set of one or more computer-readable storage media; and program instructions stored on the set of one or more computer-readable storage media to cause the processor set to perform operations comprising:
receiving a number of documents, wherein the number of documents comprise a number of visualizations; (Gong, ¶ 97, documents comprise textual and non-textual data: “parsing module 304 is configured to parse content items that comprise both textual data and non-textual image data…the types of non-textual image data may include any of the following: charts and graphs, such as bar charts, line graphs, pie charts, scatter plots, histograms, area charts, bubble charts, radar charts, and Gantt charts; pictures, such as photographs, illustrations, diagrams, clip art, and infographics; and videos that can be embedded or linked, along with animated GIFs”)
identifying a number of objects within the number of documents and locations for the number of objects from the number of documents using a set of machine learning models; (Gong, ¶ 97, objects such as charts, graphs, etc. are identified; ¶111, “picture LMM 308, chart and plot LMM 310, document LMM 316, table LMM 318, and other LMMs that may be employed to analyze content items may generate image descriptions or textual data from content items that include non-textual image data”)
identifying the number of visualizations from the number of objects within the number of documents using the set of machine learning models; (Gong, ¶¶ 97, 111, objects such as chart, graphs, etc. are visualizations)
classifying the number of visualizations into different visualization types, wherein each visualization from the number of visualizations is labeled with zero or more visualization types; (Gong, ¶ 26, “Responsive to detecting content items with non-textual image data, the RAG agent invokes an LMM that is configured to classify non-textual image data into a variety of classifications”, ¶ 107, “When the input includes chart and plot data, image classification LMM 306 uses specialized layers to recognize axis labels, grid lines, and plotted data points, distinguishing between different chart types, such as bar graphs, scatter plots, or line charts. In the case of table data, image classification LMM 306 focuses on detecting grid structures, cell contents, and numeric or textual patterns within the table, classifying the data into appropriate table-related categories”)
parsing the number of visualizations to generate a number of graphs representing the number of visualizations based on visualization types for the number of visualizations using the set of machine learning models, wherein parsing each visualization comprises applying image processing to detect visual elements and coordinates of the detected visual elements within the image of the visualization; (Gong, ¶¶ 97-99, 116-120, wherein image processing is applied for identifying a visualization “such as bar charts, line graphs, pie charts, scatter plots, histograms, area charts, bubble charts, radar charts, and Gantt charts” and visual elements such as “axes, data points, bars, lines, and labels”, the identified visualization and visual elements are represented as correlated elements to be used by an LLM, the position/coordinates of the visual elements are also identified: “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels. These layers detect and extract features specific to chart visualization, such as the placement and orientation of axes, the scale of the plot, the position and shape of data markers, and any accompanying legends or labels…For example, if classification logic identifies a bar chart with labeled axes and varying bar heights, chart and plot LMM 310 classifies the data categories (based on x-axis labels) and the corresponding values (based on y-axis positions or numerical labels). Chart and plot LMM 310 then generates text that reflects the relationships between the data points, such as, "This bar chart shows sales figures for four regions, with Region A having the highest sales at $1 ,000,000, while Region D has the lowest at $300,000."”)
extracting textual information associated with the number of visualizations from pages that contain the number of visualizations in the number of documents; and (Gong, ¶¶ 97-99, 116-120, textural information such as titles, labels, etc., are extracted: “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels. These layers detect and extract features specific to chart visualization, such as the placement and orientation of axes, the scale of the plot, the position and shape of data markers, and any accompanying legends or labels”; ¶ 122, “ chart and plot LMM 310 integrates recognized text, such as axis titles or labels, into the output”)
Gong did not specifically disclose but Tchuiev discloses generate a representation a number of graphs representing the number of visualizations. (Tchuiev, sec. III, wherein identified objects and their relationships in a documents are represented as vertices and edges: a matrix is generated for representing objects and their relationships in a document: “Consider an environment with multiple stacked objects. These object are stacked in a hierarchical structure…This structure can be represented as a graph…with NV vertices… and NE edges…The vertices represent the objects themselves, while the directed edges represent the direct hierarchical relationship between these objects”)
Gong ¶¶ 119, 166 discloses that “chart and plot data 364 represents images that comprise visual representations of data, such as bar charts, line graphs, pie charts, or scatter plots. This type of image data is structured to convey quantitative or categorical information visually through axes, data points, and labels. Chart and plot data 364 typically includes various elements, like gridlines, legends, and numerical values, associated with plotted points or bars. The primary focus for processing chart and plot data 364 is extracting the relationships between the data points and generating meaningful interpretations or summaries of the visualized data”. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing generate a representation a number of graphs representing the number of visualizations because doing so would provide for an alternative representation of identified visual objects and their associations as nodes and edges for achieving the same predictable result of generating meaningful interpretations or summaries of the visualized data.
Gong did not specifically disclose but Messina discloses generating, by the processor set, a machine readable table for the number of visualizations based on the number of graphs and the textual information associated with the number of visualizations, wherein generating the machine readable table comprises associating extracted textual information with corresponding detected elements represented in the number of graphs, and merging elements detected by the parsing model with the extracted textual information. (Messina, ¶ 38, wherein an identified visual and textual data are associated/merged by aggregating the identified visual and textual data; the aggregated visual and textual data is machine readable: “The two information streams, data from visual 210, 211 and textual strings (tokens) 212 are aggregated by an aggregation module 213 to produce output in the form of a file representing the extracted data from the charts 222. The aggregation module 213 reads column labels 214, row labels 215, values 216 and titles 217 from the outputs 210-212 from the shape analysis 202 and text extraction 207 modules. The aggregation module 213 then writes to an output file the read column labels 218, row labels 219, values 220 and titles 221. The output file 222 contains meta-information about title, values, labels, colors, and layout of the chart. The result can be further exploited for data mining applications, search engines, advanced semantic queries or to re-purpose the chart in a new, more meaningful or interactive, representation”)
Gong, ¶¶ 119, 166 implicitly discloses the feature by processing the visualized data to be used by an LLM for generating meaningful interpretations or summaries of the visualized data; meaning the visualized data must have been transformed to a machine readable data to be used by an LLM. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing generating, by the processor set, a machine readable table for the number of visualizations based on the number of graphs and the textual information associated with the number of visualizations, wherein generating the machine readable table comprises associating extracted textual information with corresponding detected elements represented in the number of graphs, and merging elements detected by the parsing model with the extracted textual information because doing so would provide for generating a machine readable table explicitly for achieving the same predictable result of processing a document comprising textual and non-textual data items.
Claim 23 . Gong teaches:
A computer program product for converting visualizations in documents comprising: a set of one or more computer-readable storage media; program instructions stored in the set of one or more computer-readable storage media to perform operations comprising:
receiving, by a processor set, a number of documents, wherein the number of documents comprise a number of visualizations; (Gong, ¶ 97, documents comprise textual and non-textual data: “parsing module 304 is configured to parse content items that comprise both textual data and non-textual image data…the types of non-textual image data may include any of the following: charts and graphs, such as bar charts, line graphs, pie charts, scatter plots, histograms, area charts, bubble charts, radar charts, and Gantt charts; pictures, such as photographs, illustrations, diagrams, clip art, and infographics; and videos that can be embedded or linked, along with animated GIFs”)
identifying, by the processor set using a set of machine learning models, a number of objects within the number of documents and locations for the number of objects from the number of documents; (Gong, ¶ 97, objects such as charts, graphs, etc. are identified; ¶111, “picture LMM 308, chart and plot LMM 310, document LMM 316, table LMM 318, and other LMMs that may be employed to analyze content items may generate image descriptions or textual data from content items that include non-textual image data”)
identifying, by the processor set using the set of machine learning models, the number of visualizations from the number of objects within the number of documents; (Gong, ¶¶ 97, 111, objects such as chart, graphs, etc. are visualizations)
classifying, by the processor set, the number of visualizations into different visualization types, wherein each visualization from the number of visualizations is labeled with zero or more visualization types; (Gong, ¶ 26, “Responsive to detecting content items with non-textual image data, the RAG agent invokes an LMM that is configured to classify non-textual image data into a variety of classifications”, ¶ 107, “When the input includes chart and plot data, image classification LMM 306 uses specialized layers to recognize axis labels, grid lines, and plotted data points, distinguishing between different chart types, such as bar graphs, scatter plots, or line charts. In the case of table data, image classification LMM 306 focuses on detecting grid structures, cell contents, and numeric or textual patterns within the table, classifying the data into appropriate table-related categories”)
parsing, by the processor set using the set of machine learning models, the number of visualizations to generate a number of graphs representing the number of visualizations based on visualization types for the number of visualizations, wherein parsing each visualization comprises applying image processing to detect visual elements and coordinates of the detected visual elements within the image of the visualization; (Gong, ¶¶ 97-99, 116-120, wherein image processing is applied for identifying a visualization “such as bar charts, line graphs, pie charts, scatter plots, histograms, area charts, bubble charts, radar charts, and Gantt charts” and visual elements such as “axes, data points, bars, lines, and labels”, the identified visualization and visual elements are represented as correlated elements to be used by an LLM, the position/coordinates of the visual elements are also identified: “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels. These layers detect and extract features specific to chart visualization, such as the placement and orientation of axes, the scale of the plot, the position and shape of data markers, and any accompanying legends or labels…For example, if classification logic identifies a bar chart with labeled axes and varying bar heights, chart and plot LMM 310 classifies the data categories (based on x-axis labels) and the corresponding values (based on y-axis positions or numerical labels). Chart and plot LMM 310 then generates text that reflects the relationships between the data points, such as, "This bar chart shows sales figures for four regions, with Region A having the highest sales at $1 ,000,000, while Region D has the lowest at $300,000."”)
extracting, by the processor set, textual information associated with the number of visualizations from pages that contain the number of visualizations in the number of documents; and (Gong, ¶¶ 97-99, 116-120, textural information such as titles, labels, etc., are extracted: “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels. These layers detect and extract features specific to chart visualization, such as the placement and orientation of axes, the scale of the plot, the position and shape of data markers, and any accompanying legends or labels”; ¶ 122, “ chart and plot LMM 310 integrates recognized text, such as axis titles or labels, into the output”)
Gong did not specifically disclose but Tchuiev discloses generate a representation a number of graphs representing the number of visualizations. (Tchuiev, sec. III, wherein identified objects and their relationships in a documents are represented as vertices and edges: a matrix is generated for representing objects and their relationships in a document: “Consider an environment with multiple stacked objects. These object are stacked in a hierarchical structure…This structure can be represented as a graph…with NV vertices… and NE edges…The vertices represent the objects themselves, while the directed edges represent the direct hierarchical relationship between these objects”)
Gong ¶¶ 119, 166 discloses that “chart and plot data 364 represents images that comprise visual representations of data, such as bar charts, line graphs, pie charts, or scatter plots. This type of image data is structured to convey quantitative or categorical information visually through axes, data points, and labels. Chart and plot data 364 typically includes various elements, like gridlines, legends, and numerical values, associated with plotted points or bars. The primary focus for processing chart and plot data 364 is extracting the relationships between the data points and generating meaningful interpretations or summaries of the visualized data”. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing generate a representation a number of graphs representing the number of visualizations because doing so would provide for an alternative representation of identified visual objects and their associations as nodes and edges for achieving the same predictable result of generating meaningful interpretations or summaries of the visualized data.
Gong did not specifically disclose but Messina discloses generating, by the processor set, a machine readable table for the number of visualizations based on the number of graphs and the textual information associated with the number of visualizations, wherein generating the machine readable table comprises associating extracted textual information with corresponding detected elements represented in the number of graphs, and merging elements detected by the parsing model with the extracted textual information. (Messina, ¶ 38, wherein an identified visual and textual data are associated/merged by aggregating the identified visual and textual data; the aggregated visual and textual data is machine readable: “The two information streams, data from visual 210, 211 and textual strings (tokens) 212 are aggregated by an aggregation module 213 to produce output in the form of a file representing the extracted data from the charts 222. The aggregation module 213 reads column labels 214, row labels 215, values 216 and titles 217 from the outputs 210-212 from the shape analysis 202 and text extraction 207 modules. The aggregation module 213 then writes to an output file the read column labels 218, row labels 219, values 220 and titles 221. The output file 222 contains meta-information about title, values, labels, colors, and layout of the chart. The result can be further exploited for data mining applications, search engines, advanced semantic queries or to re-purpose the chart in a new, more meaningful or interactive, representation”)
Gong, ¶¶ 119, 166 implicitly discloses the feature by processing the visualized data to be used by an LLM for generating meaningful interpretations or summaries of the visualized data; meaning the visualized data must have been transformed to a machine readable data to be used by an LLM. It would have been obvious before the effective filling date of the claimed invention to a person having ordinary skill in the art to combine the applied references for disclosing generating, by the processor set, a machine readable table for the number of visualizations based on the number of graphs and the textual information associated with the number of visualizations, wherein generating the machine readable table comprises associating extracted textual information with corresponding detected elements represented in the number of graphs, and merging elements detected by the parsing model with the extracted textual information because doing so would provide for generating a machine readable table explicitly for achieving the same predictable result of processing a document comprising textual and non-textual data items.
Claim 2. The computer-implemented method of claim 1, wherein the number of graphs comprise a number of nodes representing elements in the number of visualizations and a number of edges between the number of nodes representing relationships between the elements in the number of visualizations and wherein the visualizations are images of charts, graphs, or diagrams embedded in unstructured documents. (Gong, ¶¶ 99, 116-120, 166, wherein data elements and their associations/relationship in structured or unstructured documents are identified; Tchuiev, sec. III, wherein identified objects and their relationships in a documents are represented as vertices and edges)
Claims 13 and 24 are rejected under the same rationale as above.
Claim 3. The computer-implemented method of claim 1, wherein parsing, by the processor set using the set of machine learning models, the number of visualizations to generate a number of graphs representing the number of visualizations based on visualization types for the number of visualizations comprises:
selecting, by the processor set, a visualization from the number of visualizations; (Gong, ¶¶ 73, 97, 116-120, an identified non-textual data item is selected: “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels”
identifying, by the processor set using a parsing model from the set of machine learning models, a number of elements for the visualization from the number of visualizations, wherein the parsing model is selected based on visualization type for the visualization; (Gong, ¶¶ 73, 97, 116-120, data elements are identified based on the type of the non-textual data “such as bar charts, line graphs, pie charts, scatter plots, histograms, area charts, bubble charts, radar charts, and Gantt charts”)
inputting, by the processor set using the parsing model, the number of elements for the visualization into a self-attention layer in the parsing model to identify relationships and dependencies between the number of elements for the visualization; and (Gong, ¶¶ 73, 97, 116-120, “transformers are composed of multiple layers containing a multi-head, self-attention mechanism and a position-wise, feed-forward network”)
generating, by the processor set using the parsing model, a number of vectors to represent relationships and dependencies between the number of elements for the visualization. (Gong, ¶¶ 73, 97, 116-120, “transformers are composed of multiple layers containing a multi-head, self-attention mechanism and a position-wise, feed-forward network. Within the architecture of transformer models, the multi-head, self-attention mechanism and position-wise, feed-forward network function in concert to process input data. The multi-head, self-attention mechanism is designed to enable parallel processing of input sequences, allowing the model to simultaneously evaluate the importance of different segments of the input relative to each other. This mechanism operates by generating multiple sets of query, key, and value vectors for each element in the input sequence through linear transformation. The relevance of each element to every other element is calculated using a scaled dot-product attention function that computes the attention scores by taking the dot product of the query vector with the key vectors, dividing each by the square root of the dimension of the key vectors to scale the scores, then applying a softmax function to obtain the weights for the value vectors”; Tchuiev, sec. III, “The detector’s encoder outputs e feature vectors with hidden size h. Per standard Transformer Encoder-Decoder architecture, these feature vectors serve as keys and values that are input into the decoder…. By passing xd through a Multi-Layer-Perceptron (MLP), each feature vector in xd is translated into a bounding-box proposal xb…”)
Claims 14 and 25 are rejected under the same rationale as above.
Claim 4. The computer-implemented method of claim 3, further comprising:
inputting, by the processor set, the number of vectors to an adjacency head from the parsing model to output an adjacency matrix for the visualization, wherein the adjacency matrix is a representation of a graph for the visualization. (Gong, ¶¶ 73, 97, 116-120, “transformers are composed of multiple layers containing a multi-head, self-attention mechanism and a position-wise, feed-forward network. Within the architecture of transformer models, the multi-head, self-attention mechanism and position-wise, feed-forward network function in concert to process input data. The multi-head, self-attention mechanism is designed to enable parallel processing of input sequences, allowing the model to simultaneously evaluate the importance of different segments of the input relative to each other. This mechanism operates by generating multiple sets of query, key, and value vectors for each element in the input sequence through linear transformation. The relevance of each element to every other element is calculated using a scaled dot-product attention function that computes the attention scores by taking the dot product of the query vector with the key vectors, dividing each by the square root of the dimension of the key vectors to scale the scores, then applying a softmax function to obtain the weights for the value vectors”, Tchuiev, sec. III and fig. 3, “The encoder output passes through an MLP for dimension reduction, is concatenated with the bound box coordinates, and then passes through the adjacency head MLP to form the adjacency matrix”)
Claims 15 and 26 are rejected under the same rationale as above.
Claim 5. The computer-implemented method of claim 4, wherein the adjacency head is a standard multi-layer perceptron (MLP) for generating a number of outputs for each input, wherein the number of outputs generated for each input correspond to one full row of the adjacency matrix. (Tchuiev, sec. III, “By passing xd through a Multi-Layer-Perceptron (MLP), each feature vector in xd is translated into a bounding-box proposal xb…”; fig. 3, “The encoder output passes through an MLP for dimension reduction, is concatenated with the bound box coordinates, and then passes through the adjacency head MLP to form the adjacency matrix”)
Claims 16 and 27 are rejected under the same rationale as above.
Claim 6. The computer-implemented method of claim 4, wherein inputting, by the processor set using the parsing model, the number of vectors to an adjacency head to output an adjacency matrix for the visualization comprises:
creating, by the processor set, a number of vector pairs using the number of vectors, wherein each vector pairs from the number of vector pairs is created by concatenating two vectors from the number of vectors; and (note that concatenating or summing two vectors are considered as mathematical operations based on the needed outcome: Tchuiev, sec. III, “The detector’s encoder outputs e feature vectors with hidden size h. Per standard Transformer Encoder-Decoder architecture, these feature vectors serve as keys and values that are input into the decoder…. By passing xd through a Multi-Layer-Perceptron (MLP), each feature vector in xd is translated into a bounding-box proposal xb…”; fig. 3, “The encoder output passes through an MLP for dimension reduction, is concatenated with the bound box coordinates, and then passes through the adjacency head MLP to form the adjacency matrix”)
inputting, by the processor set, the number of vector pairs to the adjacency head from the parsing model to output an adjacency matrix for the visualization. (Tchuiev, sec. III and fig. 3, “The encoder output passes through an MLP for dimension reduction, is concatenated with the bound box coordinates, and then passes through the adjacency head MLP to form the adjacency matrix”; note that Xe is directly utilized for creating an adjacency matrix, comprises feature vectors and the created “adjacency matrix describes the relationships between staked objects”)
Claims 17 and 28 are rejected under the same rationale as above.
Claim 7. The computer-implemented method of claim 4, wherein inputting, by the processor set using the parsing model, the number of vectors to an adjacency head to output an adjacency matrix for the visualization comprises:
creating, by the processor set, a number of vector pairs using the number of vectors, wherein each vector pair from the number of vector pairs is created by summing two vectors from the number of vectors; and (note that concatenating or summing two vectors are considered as mathematical operations based on the needed outcome: Tchuiev, sec. III, “The detector’s encoder outputs e feature vectors with hidden size h. Per standard Transformer Encoder-Decoder architecture, these feature vectors serve as keys and values that are input into the decoder…. By passing xd through a Multi-Layer-Perceptron (MLP), each feature vector in xd is translated into a bounding-box proposal xb…”; fig. 3, “The encoder output passes through an MLP for dimension reduction, is concatenated with the bound box coordinates, and then passes through the adjacency head MLP to form the adjacency matrix”)
inputting, by the processor set, the number of vector pairs to the adjacency head from the parsing model to output an adjacency matrix for the visualization. (Tchuiev, sec. III and fig. 3, “The encoder output passes through an MLP for dimension reduction, is concatenated with the bound box coordinates, and then passes through the adjacency head MLP to form the adjacency matrix”)
Claims 18 and 29 are rejected under the same rationale as above.
Claim 8. The computer-implemented method of claim 3, further comprising:
inputting, by the processor set, each element from the number of elements into a number of shape heads from the parsing model to predict shape of each element for the visualization, wherein each shape head from the number of shape heads is designed to predict a particular shape for the number of elements. (Gong, ¶ 73, wherein “transformers are composed of multiple layers containing a multi-head, self-attention mechanism and a position-wise, feed-forward network” suggests that a shape head can be used as needed; ¶106, “When provided with an image, image classification LMM 306 first extracts relevant features through a pre-trained convolutional neural network (CNN) that analyzes various aspects, such as shapes, textures, and patterns, in the image. The extracted features are then passed through connected layers designed to map the visual data to predefined categories”; ¶ 116, “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers, similar to how other models handle images, but optimized to identify structured elements like axes, data points, bars, lines, and labels. These layers detect and extract features specific to chart visualization, such as the placement and orientation of axes, the scale of the plot, the position and shape of data markers, and any accompanying legends or labels”; Tchuiev, sec. III, wherein “The detector and the adjacency head it utilizes infer the objects’ bounding box, class, and adjacency matrix, to be used by the action selection” indicates that a shape head can be used as needed)
Claims 19 and 30 are rejected under the same rationale as above.
Claim 9. The computer-implemented method of claim 3, further comprising:
inputting, by the processor set, each element from the number of elements into a color head from the parsing model to predict color of each element for the visualization, wherein the color head predicts color values for each input element. (Gong, ¶ 73, wherein “transformers are composed of multiple layers containing a multi-head, self-attention mechanism and a position-wise, feed-forward network” suggests that a color head can be used as needed; ¶¶ 106, 116, 165, “Picture data 362 is composed of pixels that form recognizable patterns, such as edges, textures, and colors that can be processed to identify objects, people, animals, or specific visual elements within the image”; ¶ 183, “The legend is a key that explains the symbols, colors, or patterns used in the chart to differentiate between different data series or categories”,¶ 189, “Color is a critical component, encompassing the entire spectrum of hues, saturation, and brightness levels, which together create the visual impression of the picture”; Tchuiev, sec. III, wherein “The detector and the adjacency head it utilizes infer the objects’ bounding box, class, and adjacency matrix, to be used by the action selection” indicates that a color head can be used as needed)
Claims 20 and 31 are rejected under the same rationale as above.
Claim 10. The computer-implemented method of claim 3, further comprising:
inputting, by the processor set, each element from the number of elements into a class head from the parsing model to predict element category of each element for the visualization. (Gong, ¶ 73, wherein “transformers are composed of multiple layers containing a multi-head, self-attention mechanism and a position-wise, feed-forward network” suggests that a class head can be used as needed; ¶¶ 116-120, “When a chart or plot is input, chart and plot LMM 310 first processes the visual data through a series of convolutional layers…if classification logic identifies a bar chart with labeled axes and varying bar heights, chart and plot LMM 310 classifies the data categories (based on x-axis labels) and the corresponding values (based on y-axis positions or numerical labels)”; Tchuiev, sec. III, wherein “The detector and the adjacency head it utilizes infer the objects’ bounding box, class, and adjacency matrix, to be used by the action selection” indicates a class head can be used as needed)
Claims 21 and 32 are rejected under the same rationale as above.
Claim 11. The computer-implemented method of claim 1, wherein the set of machine learning models comprise a number of parsing models, and wherein each parsing model from the number of parsing models is designed to perform parsing for visualizations of a particular visualization type. (Gong, ¶ 26, “Initially, a RAG agent accesses content items that include non-textual image data and/or textual data. The non-textual image data includes a table, a chart, a document, or a picture in an embodiment. Responsive to detecting content items with non-textual image data, the RAG agent invokes an LMM that is configured to classify non-textual image data into a variety of classifications. The RAG agent also detects that a content item includes nontextual image data corresponding to a particular classification. Using this classification as selection criteria, the RAG agent selects an LMM from a set of available LMMs, ensuring that the selected LMM is configured to generate text from non-textual image data that corresponds to the particular classification”; ¶¶ 103, 106-107,111, 116)
Claims 22 and 33 are rejected under the same rationale as above.
Response to Amendment and Arguments
Applicant’s arguments with respect to rejected claims have been fully considered but are not persuasive for at least the following reasons.
Applicant argues the applied references do not teach the amended feature because “Messina merely aggregates outputs produced independently by two separate processing modules. In this case, the shape-analysis module independently determines graphical components, the text-extraction module independently determines textual content, and the aggregation module subsequently reads the outputs from those independent modules…Messina simply combines two independent result sets to produce chart metadata”. Remarks, 16.
Merging operation is simply associating an identified text element with an identified visual element “For example, a bar label detected by the number of parsing models 218 may overlap with a text element containing the text "33". These two are associated together, so that the bar label reads "33"”. Spec, ¶ 81.
Messina explicitly discloses aggregating/associating a textual element, e.g., a value from a column or row label with a visual element, e.g., the column or label from which a value is obtained output the aggregated labels and textual information in a file that “can be further exploited for data mining applications, search engines, advanced semantic queries or to re-purpose the chart in a new, more meaningful or interactive, representation”.
Applicant argues the applied references do not teach “the claimed operation of creating a plurality of vector pairs, wherein each vector pair is generated by concatenating two vectors from the plurality of vectors, and providing the vector pairs as inputs to the adjacency head” because “the cited portions of Tchuiev do not disclose the claimed pairwise vector construction… claim 6 requires constructing pairwise feature representations by concatenating two vectors from the plurality of vectors that correspond to different detected visualization elements, before providing the resulting vector pairs to the adjacency head. Accordingly, the claimed concatenation is a concatenation between vector corresponding to visualization element i and vector corresponding to visualization element j… The Examiner appears to treat the two concatenation operations as equivalent because both involve concatenation. Applicant respectfully submits that the issue is not whether concatenation itself is known. Rather, the issue is what is being concatenated. Concatenating a feature vector with metadata describing the same object is materially different from constructing an explicit pairwise representation from two different visualization-element vectors. The former enriches the representation of a single object by incorporating positional information, while the latter explicitly encodes the relationship between two different visualization elements… Tchuiev derives pairwise relationships only implicitly during the subsequent matrix multiplication operation, Applicant's claimed invention forms explicit pairwise feature representations before adjacency prediction. These represent distinct neural network architectures for modeling relationships and are not interchangeable implementations of the same operation… The Examiner further asserts that concatenation is a known mathematical operation. However, obviousness cannot be established by identifying a known mathematical operation in the abstract. The rejection must demonstrate that the cited references teach or suggest the claimed computational architecture, namely constructing pairwise vector representations by concatenating two vectors from the plurality of visualization-element vectors before the adjacency head performs relationship inference. The cited references do not do so”. Remarks, 17-18.
In response, as noted in the office action, a mathematical operation such as concatenating or summing two vectors can be used based on a needed outcome. Tchuiev explicitly discloses providing two vectors to an adjacency matrix because Xe which is directly utilized for creating an adjacency matrix, comprises feature vectors and the created “adjacency matrix describes the relationships between staked objects”.
Conclusion
The prior arts made of record in PTO-326 and not relied upon are considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHSEN ALMANI whose telephone number is (571)270-7722. The examiner can normally be reached on M-F, 9:00 to 5:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ann J. Lo can be reached on 571-272-9767. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHSEN ALMANI/Primary Examiner, Art Unit 2159
