Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This Office Action corresponds to application 18/743,275 which was filed on 6/14/2024.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 5/18/2026 has been entered.
Response to Amendment
In the reply filed 5/18/2026, claims 1-9, 12-16, and 18-20 have been amended. Claims 21-23 have been cancelled and claims 24-26 have been added. Accordingly, claims 1-9, 12-16, 18-20, and 24-26 are currently pending.
Response to Arguments
Applicant’s arguments filed 5/18/2026 have been fully considered but are moot in view of new grounds of rejection.
Examiner’s note
Claims 19-20 falls within a statutory category, because paragraph 142 of the instant specification recites “The specific term “computer-readable storage medium” or “storage device” expressly excludes propagated signals per se; a computer-readable storage medium or storage device is “non-transitory” in this regard.”
Claim Objections
Claim 9 is objected to because of the following informalities: Claim 9 includes the phrase “… pertaining set of training examples”, which is interpreted as a misspelling of pretraining. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 3 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites “in the graph”. However, there are multiple graphs introduced in the parent claims and it is not clear which graph is being referenced, therefore making it indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8, and 18-20, 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US2021/0081677, previously cited in ‘892), hereinafter Wang, in view of Dines (US2025/0054327), and Betthauser et al. (US2022/0230053), hereinafter Betthauser.
Regarding Claim 1:
Wang teaches:
A method for creating and updating an index in a local computing device, comprising: receiving by the local computing device parameters of a trained graph neural network from one or more servers (Wang, figures 1-2, [0044-0045, 0092], note the computer vision system, which comprises the graph neural network, may be stored on the local computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture, which means the computing device may receive a trained graph neural network);
capturing and storing a plurality of screenshots at different respective times at the local computing device, the plurality of screenshots being a second set of images that are different than the first images (Wang, figures 1-2 and 8, [0020, 0045, 0048, 0090-0091], note receiving a collection of images; note the images may be at different respective times, e.g., video sequence of images; note the images may be captured and stored; note the computer vision system may be stored on the computing devices, e.g., local computing device; note the images are different than the ones used to train the graph neural network; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines);
using a machine-trained model to identify features associated with the plurality of screenshots (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image);
determining, based on the features, relationships among pairs of screenshots that satisfy one or more prescribed similarity tests (Wang, figures 5 and 8, [0018-0021, 0053-0054, 0071, 0093-0097], note determining relationships among nodes; note nodes are associated with the node embeddings/features of the images, therefore the relationships between the nodes would be based on the features; note determining similarity between nodes to determine the relationships, e.g., similarity test; note screenshots are images);
assigning screenshot nodes in a second graph to represent the plurality of screenshots, the second graph being generated by the local computing device and being different than the first graph (Wang, figures 1-2 and 8, [0020, 0045, 0090-0097], note creating a graph that comprises assigning a plurality of image/screenshot nodes; note the AI model may be deployed locally; note the images are different than the training images; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines);
assigning edges to pairs of screenshot nodes having the relationships determined by the determining, the edges being associated with the types of relationships (Wang, figure 8, [0003, 0019-0021, 0071, 0083, 0093-0097], note determining relationships/edges among nodes; note nodes are associated with the node embeddings of the images; note determining/deriving the edges/relationships; note the edges are for similar image, e.g., image edge type; note identifying common objects; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines);
generating a plurality of target embeddings associated with the screenshot nodes in the second graph that represent the plurality of screenshots using the trained graph neural network (Wang, figures 1-2 and 8, [0018-0022, 0045, 0090-0097], note that each image node has node embeddings, e.g., target embeddings associated with the screenshot nodes; note using the graph to train/update the graph neural network and obtain high-order relationship information and spatial information to perform segmentation functions on image content such as identifying and segmenting target objects or common objects, e.g., target embeddings; note the computer vision system, which comprises the graph neural network, may be stored on the computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines); and
storing the plurality of target embeddings in a data store as an index for use in retrieving any of the screenshots (Wang, figures 2 and 8, [0018-0022, 0048], note using the graph to train/update the graph neural network and obtain high-order relationship information and spatial information to perform segmentation functions on image content such as identifying and segmenting target objects or common objects, e.g., target embeddings; note the segmentation results are stored in the database, e.g., data store; note storing the graph neural networks in storage, which is broadly interepted as being stored as an index; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines);
capturing an additional screenshot by the local computing device (Wang, figures 1-2 and 8, [0020, 0045, 0048, 0090-0091], note receiving a collection of images; note the images may be at different respective times, e.g., video sequence of images; note the images may be captured and stored; note the computer vision system may be stored on the computing devices, e.g., local computing device; note the images are different than the ones used to train the graph neural network; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines);
creating an additional node in the second graph for the additional screenshot using the machine-trained models, and connecting the additional node to other nodes the second graph (Wang, figures 1-2 and 8, [0020-0021, 0045, 0090-0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image; note creating a graph that comprises assigning a plurality of image/screenshot nodes; note the AI model may be deployed locally; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines); and
generating an additional target embedding for the additional screenshot using the trained graph neural network, and updating the index to include the additional target embedding (Wang, figures 1-2 and 8, [0018-0022, 0045, 0090-0097], note that each image node has node embeddings, e.g., target embeddings associated with the screenshot nodes; note using the graph to train/update the graph neural network and obtain high-order relationship information and spatial information to perform segmentation functions on image content such as identifying and segmenting target objects or common objects, e.g., target embeddings; note the computer vision system, which comprises the graph neural network, may be stored on the computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture; note storing the updated graph neural networks in storage, which is broadly interepted as being stored as an index; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines),
wherein the trained graph neural network is a neural network that maps initial node embeddings into updated node embeddings (Wang, figures 1-2, [0044-0045, 0092], note the computer vision system, which comprises the graph neural network, may be stored on the local computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture, which means the computing device may receive a trained graph neural network).
While Wang teaches the creation and use of graph neural networks for images, Wang doesn’t specifically teach receiving a trained graph neural network having been generated by the one or more servers based on a first graph that represents at least a first set of first images; using a plurality of different machine-trained models; assigning different types of edges to pairs of screenshot nodes; and the trained graph neural network is different than the plurality of different machine-trained models. However, Dines is in the same field of endeavor, data analysis and information retrieval, and Dines teaches:
receiving by the local computing device parameters of a trained graph neural network from one or more servers, the trained graph neural network having been generated by the one or more servers based on a first graph that represents at least a first set of first images (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note the AI/ML models may be deep learning neural networks (DLNN), or any other suitable neural network, and graph neural networks are a type of DLNNs; note training set uses images. Also, when combined with the previously cited references the neural network would be a graph neural network as taught by Wang);
capturing and storing a plurality of screenshots at different respective times at the local computing device, the plurality of screenshots being a second set of images that are different than the first images (Dines, [0031, 0036], note task mining may include taking screenshots, which are different than the images used to train the models; note AI/ML models may be deployed locally);
using a plurality of different machine-trained models to identify features associated with the plurality of screenshots (Dines, [0031-0032], note using multiple AI/ML models analyze data. When combined with the previously cited references this would be for identifying features associated with the screenshots);
capturing an additional screenshot by the local computing device (Dines, [0031, 0036], note task mining may include taking screenshots, which are different than the images used to train the models; note AI/ML models may be deployed locally);
creating an additional node in the second graph for the additional screenshot using the plurality of different machine-trained models, and connecting the additional node to other nodes the second graph (Dines, [0031-0032, 0036], note using multiple AI/ML models analyze data; note taking multiple screenshots. When combined with the previously cited references this would be for image node and graph creation as taught by Wang);
wherein the trained graph neural network is a neural network that maps initial node embeddings into updated node embeddings, and the trained graph neural network is different than the plurality of different machine-trained models (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally, which means the trained model is the neural network that maps the initial node embeddings into updated node embeddings; note since the training is performed separately and remote, it is interepted as different than the plurality of different machine-trained models; note using multiple AI/ML models analyze data; note the AI/ML models may be deep learning neural networks (DLNN), or any other suitable neural network, and graph neural networks are a type of DLNNs; note training set uses images. Also, when combined with the previously cited references the neural network would be a graph neural network as taught by Wang).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
While Wang as modified teaches the creation and use of graph neural networks for images, Wang as modified doesn’t specifically teach assigning different types of edges to pairs of screenshot nodes having the relationships determined by the determining, the different types of edges being associated with the different respective types of relationships. However, Betthauser is in the same field of endeavor, data analysis and information retrieval, and Betthauser teaches:
the trained graph neural network having been generated by the one or more servers based on a first graph that represents at least a first set of first images (Betthauser, [0020], note the use of ImageNet and WordNet to train models. When combined with the previously cited references this would be for the pretrained graph as taught by Dines)
using a plurality of different machine-trained models to identify features associated with the plurality of screenshots (Betthauser, figure 5, [0008, 0019], note using different machine learning models for different data types. When combined with the previously cited references this would be for the images/screenshots)
assigning different types of edges to pairs of screenshot nodes having the relationships determined by the determining, the different types of edges being associated with the different respective types of relationships (Betthauser, [0020], note generating a graph neural network with different edge types for various data types, such as text and image);
storing the plurality of target embeddings in a data store as an index for use in retrieving any of the screenshots (Betthauser, figure 3, [0022, 0037, 0039, 0041], note indexing the embeddings. When combined with the previously cited references this would be for the target image embeddings for screenshots);
updating the index to include the additional target embedding (Betthauser, [0051], note updating the modes. When combined with the previously cited references this would be for the updating the graphs and corresponding index for the screenshots).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 2:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein a particular screenshot node in the second graph that describes a particular screenshot describes an entirety of contents presented on a user interface presentation at a particular time (Wang, figure 8, [0019-0021, 0092, 0097], note the nodes represent the images; note screenshots are images which may describe an entirety of contents presented on a user interface; note the image describing an entirety of contents presented on a user interface is nonfunctional descriptive material as explained in section 2111.05 of the MPEP and does not hold patentable weight) (Dines, [0031, 0036], note task mining may include taking screenshots).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
Regarding Claim 3:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein a particular screenshot node in the graph that describes a particular screenshot describes a portion of an entirety of contents presented on a user interface presentation at a particular time, the portion being less than the entirety (Wang, figure 8, [0019-0021, 0092, 0097], note the nodes represent the images; note screenshots are images which may describe a portion of contents presented on a user interface; note the image describing a portion of contents presented on a user interface is nonfunctional descriptive material as explained in section 2111.05 of the MPEP and does not hold patentable weight) (Dines, [0031, 0036], note task mining may include taking screenshots).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
Regarding Claim 4:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the second graph also includes: text nodes associated with instances of text, each instance of text being associated with at least one of the plurality of screenshots (Wang, figure 8, [0003, 0019-0021, 0071, 0083, 0093-0097], note determining relationships/edges among nodes; note nodes are associated with the node embeddings of the images; note identifying common objects; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines) (Betthauser, figures 1-2, 5, and 7, [0003, 0019-0021, 0028, 0034, 0053], note combining nodes of various embedding types such as text and image nodes); and
forth edges that connect the text nodes to the screenshot nodes that represent the plurality of screenshots (Wang, figure 8, [0003, 0019-0021, 0071, 0083, 0093-0097], note determining relationships among nodes; note nodes are associated with the node embeddings of the images; note determining/deriving the edges/relationships is interpreted as a prescribed similarity test; note identifying common objects, which is interpreted to include common image content, common text, and common entities since images may comprise text and entities) (Betthauser, figures 1-2, 5, and 7, [0003, 0019-0021, 0028, 0034, 0053], note combining nodes of various embedding types such as text and image nodes; note generating a graph neural network with different edge types for various data types, such as text, image, and classifications (e.g., graphs, languages, etc. are examples of a classification result).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 5:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
identifying entities associated with the plurality of screenshots based on the features using the third machine-trained model (Wang, figure 8, [0019-0021, 0091-0097], note feature extraction component identifies features to generate a node embeddings for the images, e.g., identifying entities; note identifying common objects) (Dines, [0031-0032], note using multiple AI/ML models analyze data. When combined with the previously cited references this would be for identifying features associated with the screenshots) (Betthauser, figures 1 and 5, [0003, 0019-0020, 0028, 0053], note using different machine learning models for different data types; note the different machine learning models may be for images, text, and classifications (e.g., graphs, languages, etc. are examples of a classification result); identifying common/similar content. When combined with the previously cited references this would be for the images/screenshots); and
linking screenshot nodes associated with screenshots that are associated with common entities (Wang, figure 8, [0019-0021, 0091-0097], note determining relationships amongst nodes and linking the nodes) (Dines, [0031-0032], note using multiple AI/ML models analyze data. When combined with the previously cited references this would be for identifying features associated with the screenshots) (Betthauser, figures 1 and 5, [0003, 0019-0020, 0028, 0053], note using different machine learning models for different data types; note the different machine learning models may be for images, text, and classifications (e.g., graphs, languages, etc. are examples of a classification result); note linking common/similar content. When combined with the previously cited references this would be for the images/screenshots).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 6:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the third machine-trained model is: a classification machine-trained model that classifies a topic expressed by the particular screenshot; or a classification machine-trained model that classifies a named entity expressed by the particular screenshot; or a classification machine-trained model that classifies an activity expressed by the particular screenshot (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image) (Dines, [0031, 0036], note task mining may include taking screenshots) (Betthauser, figures 1 and 5, [0003, 0019-0020, 0028, 0053], note using different machine learning models for different data types; note the different machine learning models may be for images, text, and classifications (e.g., graphs, languages, etc. are examples of a classification result). When combined with the previously cited references this would be for the images/screenshots).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 7:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the third edge type represents: a common occurrence of at least one topic in the two previously captured screenshots; or a common occurrence of at least one named entity in the two previously screenshots; or a common activity associated with the two previously captured screenshots (Wang, figure 8, [0003, 0019-0021, 0071, 0083, 0091-0097], note identifying common objects; note determining relationships amongst nodes and linking the nodes) (Dines, [0031, 0036], note task mining may include taking screenshots) (Betthauser, figures 1 and 5, [0003, 0019-0020, 0028, 0053], note generating a graph neural network with different edge types for various data types, such as text, image, and classifications (e.g., graphs, languages, etc. are examples of a classification result).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 8:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the one or more servers have produced the trained graph neural network by: producing a pretrained graph neural network by performing pretraining based on a pretrained graph that represents a pretraining set of training examples that include pretraining images and pretraining instances of text associated with the pretraining images, wherein the pretraining set of training examples includes images that are not screenshot images (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note models may be trained with various data such as web pages, scanned documents, interfaces, screens, etc., which are interpreted to include images that are not screenshot images) (Betthauser, [0020], note models being trained broad categories such as ImageNet or WordNet. When combined with the previously cited references, this would be for the pretraining dataset used by Dines which includes images that are not screenshot images); and
producing a finetuned graph neural network by performing finetuning on the pretrained graph neural network based on a finetuning graph that represents a finetuning set of training examples that describe example screenshots and instances of text associated with the example screenshots (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note finetuning includes training using text, screenshots, and contextual information, which is interpreted as example screenshots and instances of text associated with the example screenshots),
wherein the trained graph neural network that is provided to the local computing device is the finetuned graph neural network, and wherein the first graph is the finetuning graph (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 18:
Wang as modified shows the computer-readable storage medium as disclosed above;
Wang as modified further teaches:
wherein the parameters of the graph neural network have been trained by the one or more servers by: producing a pretrained graph neural network by performing pretraining based on a pretrained graph that represents a pretraining set of training examples that include pretraining images and pretraining instances of text associated with the pretraining images, wherein the pretraining set of training examples includes images that are not screenshot images (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note models may be trained with various data such as web pages, scanned documents, interfaces, screens, etc., which are interpreted to include images that are not screenshot images) (Betthauser, [0020], note models being trained broad categories such as ImageNet or WordNet. When combined with the previously cited references, this would be for the pretraining dataset used by Dines which includes images that are not screenshot images); and
producing a finetuned graph neural network by performing finetuning on the pretrained graph neural network based on a finetuning graph that represents a finetuning set of training examples that describe example screenshots and instances of text associated with the example screenshots (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note finetuning includes training using text, screenshots, and contextual information, which is interpreted as example screenshots and instances of text associated with the example screenshots),
wherein the finetuned graph neural network is the graph neural network having parameters that are transferred to the local computing device, and wherein the first graph is the finetuning graph (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 19:
Wang teaches:
A computer-readable storage medium for storing computer-readable instructions, a processing system executing the computer-readable instructions to perform operations, the operations comprising:
generating a first graph by assigning nodes in the first graph to represent the plurality of first images and the first instances of text, and assigning edges that connect the nodes in the first graph (Wang, figures 1-2 and 8, [0020, 0045, 0090-0097], note creating a graph that comprises assigning a plurality of image nodes and assigning edges that connect the nodes. Wang may be used to create multiple graphs and this is an example of the model that may be used as the pretrained model as taught by Dines below);
graph neural network to a particular local computing device (Wang, figures 1-2, [0044-0045, 0092], note the computer vision system, which comprises the graph neural network, may be stored on the local computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture, which means the computing device may receive a trained graph neural network);
where the graph neural network is used to produce final target embeddings for screenshots locally captured by the local computing device, the screenshots that are locally captured being represented by a second graph that is generated by the local computing device using different machine-trained models (Wang, figures 1-2 and 8, [0018-0022, 0045, 0048, 0090-0097], note receiving a collection of images; note the images may be at different respective times, e.g., video sequence of images; note the images may be captured and stored; note the computer vision system may be stored on the computing devices, e.g., local computing device; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines; note that each image node has node embeddings, e.g., target embeddings associated with the screenshot nodes; note using the graph to train/update the graph neural network and obtain high-order relationship information and spatial information to perform segmentation functions on image content such as identifying and segmenting target objects or common objects, e.g., target embeddings; note the computer vision system, which comprises the graph neural network, may be stored on the computing devices);
wherein the graph neural network is a neural network that maps initial node embeddings into updated node embeddings (Wang, figures 1-2, [0044-0045, 0092], note the computer vision system, which comprises the graph neural network, may be stored on the local computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture, which means the computing device may receive a trained graph neural network).
While Wang teaches the creation and use of graph neural networks for images, Wang doesn’t specifically teach receiving a first set of training examples that includes a plurality of first images and first instances of text associated with the first images; training a graph neural network based on the first graph; transferring parameters of the graph neural network to a particular local computing device; and text nodes. However, Dines is in the same field of endeavor, data analysis and information retrieval, and Dines teaches:
receiving a first set of training examples that includes a plurality of first images and first instances of text associated with the first images (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note models may be trained with various data such as web pages, scanned documents, interfaces, screens, etc., which are interpreted to include images and text associated with the images);
generating a first graph by assigning nodes in the first graph to represent the plurality of first images and the first instances of text, and assigning edges that connect the nodes in the first graph (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally, which means the trained model is the neural network that maps the initial node embeddings; note using multiple AI/ML models analyze data; note the AI/ML models may be deep learning neural networks (DLNN), or any other suitable neural network, and graph neural networks are a type of DLNNs; note training set uses images. Also, when combined with the previously cited references the neural network would be a graph neural network as taught by Wang);
training a graph neural network based on the first graph (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally),
transferring parameters of the graph neural network to a particular local computing device, where the graph neural network is used to produce final target embeddings for screenshots locally captured by the local computing device, the screenshots that are locally captured being represented by a second graph that is generated by the local computing device using different machine-trained models (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note using multiple AI/ML models analyze data; note task mining may include taking screenshots. When combined with the previously cited references the deployed model is used to produce the final target embeddings in a second graph as taught by Wang);
wherein the graph neural network is a neural network that maps initial node embeddings into updated node embeddings, and the graph neural network is different than the plurality of different machine-trained models that are used to produce the first graph and the second graph (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally, which means the trained model is the neural network that maps the initial node embeddings into updated node embeddings; note since the training is performed separately and remote, it is interepted as different than the plurality of different machine-trained models and second graph; note since the first graph is fine-tuned which is interpreted as a different graph neural network; note using multiple AI/ML models analyze data; note the AI/ML models may be deep learning neural networks (DLNN), or any other suitable neural network, and graph neural networks are a type of DLNNs; note training set uses images. Also, when combined with the previously cited references the neural network would be a graph neural network as taught by Wang).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
While Wang as modified teaches the creation and use of graph neural networks for images, Wang as modified doesn’t specifically teach text nodes. However, Betthauser is in the same field of endeavor, data analysis and information retrieval, and Betthauser teaches:
receiving a first set of training examples that includes a plurality of first images and first instances of text associated with the first images (Betthauser, [0020], note the use of ImageNet and WordNet to train models. When combined with the previously cited references this would be for the pretrained graph as taught by Dines);
using a plurality of different machine-trained models to identify features associated with the plurality of screenshots (Betthauser, figure 5, [0008, 0019], note using different machine learning models for different data types)
where the graph neural network is used to produce final target embeddings for screenshots locally captured by the local computing device, the screenshots that are locally captured being represented by a second graph that is generated by the local computing device using different machine-trained models (Betthauser, [0008, 0019-0020], note generating a graph neural network with different edge types for various data types, such as text and image);
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 20:
Wang as modified shows the computer-readable storage medium as disclosed above;
Wang as modified further teaches:
wherein the plurality of different machine-trained models includes: a first machine-trained model that produces an image embedding based on image content of a particular screenshot represented by the second graph; or a second machine-trained model that produces a text embedding based on text content of the particular screenshot; or a third machine-trained model that identifies a topic expressed by the particular screenshot; or a fourth machine-trained model that identifies a named entity expressed by the particular screenshot; or a fifth machine-trained model that identifies an activity expressed by the particular screenshot; or any combination of the first, second, third, fourth, and first machine-trained models (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image, e.g., image embedding model) (Dines, [0031-0032], note using multiple AI/ML models analyze data. When combined with the previously cited references this would be for identifying features associated with the screenshots) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note using different machine learning models for different data types; note the different machine learning models may be for images, text, and classifications (e.g., graphs, languages, etc. are examples of a classification result). When combined with the previously cited references this would be for the images/screenshots).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 24:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the plurality of different machine-trained models include: a first machine- trained model that produces an image embedding based on at least one image region in a particular screenshot, a second machine-trained model that produces a text embedding based on at least one text region in the particular screenshot, and a third machine-trained model that produces a classification result based on the at least one text region and/or the at least one image region in the particular screenshot (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image, e.g., image embedding model) (Dines, [0031-0032], note using multiple AI/ML models analyze data. When combined with the previously cited references this would be for identifying features associated with the screenshots) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note using different machine learning models for different data types; note the different machine learning models may be for images, text, and classifications (e.g., graphs, languages, etc. are examples of a classification result). When combined with the previously cited references this would be for the images/screenshots), and
wherein the different types of edges include a first edge of a first edge type that represents common image embeddings produced by the first machine-trained model for two previously captured screenshots, a second edge of a second edge type that represents common text embeddings produced by the second machine-trained model for the two previously captured screenshots, and a third edge of a third edge type that represents a common classification result produced by the third machine-trained model for the two previously captured screenshots (Wang, figure 8, [0003, 0019-0021, 0071, 0083, 0093-0097], note determining relationships/edges among nodes; note nodes are associated with the node embeddings of the images; note determining/deriving the edges/relationships; note the edges are for similar image, e.g., image edge type; note identifying common objects; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note generating a graph neural network with different edge types for various data types, such as text, image, and classifications (e.g., graphs, languages, etc. are examples of a classification result).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Claim Rejections - 35 USC § 103
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Dines, Betthauser and Malhotra et al. (US2025/0068910, previously presented in ‘892), hereinafter Malhotra.
Regarding Claim 9:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the pretraining uses supervised learning by: masking nodes in the pretraining graph that describe the pretraining set of training examples, to produce masked nodes (Wang, [0022], note masks that identify semantically similar objects in a collection of images) (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note supervised learning; note models may be trained with various data such as web pages, scanned documents, interfaces, screens, etc., which are interpreted to include images that are not screenshot images) (Betthauser, [0002, 0020, 0035], note models being trained broad categories such as ImageNet or WordNet. When combined with the previously cited references, this would be for the pretraining dataset used by Dines which includes images that are not screenshot images);
predicting identities of the masked nodes using the pretrained graph neural network (Wang, [0056], note prediction masks) (Dines, figures 1 and 6-7, [0030-0032, 0054-0055, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note supervised learning; note models may be trained with various data such as web pages, scanned documents, interfaces, screens, etc., which are interpreted to include images that are not screenshot images);
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
While Wang as modified teaches the creation and use of graph neural networks, Wang as modified doesn’t specifically teach generating loss information based an extent to which the predicted identities of the masked nodes accurately match actual identities of the mask modes; and updating parameters of the pretrained graph neural network based on the loss information. However, Malhotra, is in the same field of endeavor, data analysis, and Malhotra teaches:
wherein the pretraining uses supervised learning by: masking nodes in the pretraining graph that describe the pretraining set of training examples, to produce masked nodes (Malhotra, [0036], note masking features corresponding to nodes of a graph);
predicting identities of the masked nodes using the pretrained graph neural network (Malhotra, [0036], note predicating the identities of the masks nodes);
generating loss information based an extent to which the predicted identities of the masked nodes accurately match actual identities of the mask modes (Malhotra, [0036], note calculating reconstruction loss); and
updating parameters of the pretrained graph neural network based on the loss information (Malhotra, [0036], note fine-tuning the models based on the loss information).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Malhotra because all references are directed to data analysis and because Malhotra would expand upon the teachings of the previously cited references in image and text analysis which would improve the performance of the machine learning models by learning form more information than conventionally possible (Malhotra, [0042-0043]).
Claim Rejections - 35 USC § 103
Claim(s) 12-16, and 25-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Dines, Betthauser, and Choubey et al. (US2025/0225370, previously cited in ‘892), hereinafter Choubey.
Regarding Claim 12:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
retrieving, using the index, a previously captured screenshot that is associated with the target embedding (Wang, figures 2 and 8, [0018-0022, 0048], note using the graph to train/update the graph neural network and obtain high-order relationship information and spatial information to perform segmentation functions on image content such as identifying and segmenting target objects or common objects, e.g., target embeddings; note the segmentation results are stored in the database, e.g., data store; note storing the graph neural networks in storage, which is broadly interepted as being stored as an index; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines) (Betthauser, figure 3, [0022, 0037, 0039, 0041], note indexing the embeddings. When combined with the previously cited references this would be for the target image embeddings for screenshots);
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
While Wang as modified teaches the creation and use of graph neural networks, Wang as modified doesn’t specifically teach adding a query node that describes the query to the second graph to produce a third graph that represents an updated version of the second graph; generating a query embedding using the graph neural network based on the third graph; identifying a target embedding associated with the query node that matches the query embedding; and retrieving, using the index, a previously captured screenshot that is associated with the target embedding provided by the identifying. However, Choubey, is in the same field of endeavor, data analysis, and Choubey teaches:
wherein the method further includes using the second graph to perform a retrieval operation by: adding a query node that describes the query to the second graph to produce a third graph that represents an updated version of the second graph (Choubey, figure 4A-4B, [0019, 0074-0079], note adding a query node to the graph, which would produce a third/updated graph);
generating a query embedding using the graph neural network based on the third graph (Choubey, figures 4A-4B, [0019, 0074-0079], note generating a query representation, e.g., query embedding);
identifying a target embedding associated with the query node that matches the query embedding (Choubey, figures 4A-4B, [0019, 0074-0081], note determining a portion of the graph associated with the query, e.g., identifying a target embedding); and
retrieving, using the index, a previously captured screenshot that is associated with the target embedding provided by the identifying (Choubey, figures 4A-4B, [0019, 0077-0083], note transmitting the results to a computer device, when combined with the previously cited references this would be the images/screenshots in the graph stored in the index as taught by Wang, Dines, and Betthauser).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Choubey because all references are directed to data analysis and because Choubey would expand upon the teachings of the previously cited references in data analysis which would improve the accuracy of the machine learned models by utilizing techniques that expose data that is relevant to a user’s input (Choubey, [0001, 0010-0011]).
Regarding Claim 13:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the adding the query node comprises: using the plurality of different machine-trained models to identify query features of the query (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image) (Dines, [0031-0032], note using multiple AI/ML models analyze data) (Betthauser, figure 5, [0008, 0019], note using different machine learning models for different data types) (Choubey, figure 4A-4B, [0019, 0074-0079], note adding a query node to the graph)and
using the query features to identify one or more links that connect the query node to one or more other nodes in the second graph (Choubey, figures 4A-4B, [0019, 0074-0081], note determining a portion of the graph associated with the query); and
adding one or more edges to the second graph associated with the one or more links (Choubey, figures 4A-4B, [0019, 0074-0081], note adding a query node to the graph, note linking the node by adding edges to the other associated nodes; note determining a portion of the graph associated with the query).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Choubey because all references are directed to data analysis and because Choubey would expand upon the teachings of the previously cited references in data analysis which would improve the accuracy of the machine learned models by utilizing techniques that expose data that is relevant to a user’s input (Choubey, [0001, 0010-0011]).
Regarding Claim 14:
Wang further teaches:
A computing system for accessing screenshot information, comprising: an instruction data store for storing computer-readable instructions (Wang, figures 1-2, [0027-0031], note computing devices, memory, and instructions); and
a processing system for executing the computer-readable instructions in the data store (Wang, figure 1, [0027-0031], processing units), to perform operations including:
receiving parameters of a trained graph neural network from one or more servers (Wang, figures 1-2, [0044-0045, 0092], note the computer vision system, which comprises the graph neural network, may be stored on the local computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture, which means the computing device may receive a trained graph neural network);
generating a second graph having screenshot nodes that are associated with a plurality of screenshots captured by the computing system, the screenshots being images that are different than the first images and the screenshot nodes being graph nodes that are associated with respective target embeddings produced by the trained graph neural network and stored in an index (Wang, figures 1-2 and 8, [0019-0022, 0045, 0048, 0083, 0090-0097], note receiving a collection of images; note the images may be captured and stored; note the images are different than the training images; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines; note feature extraction component identifies features to generate a nodes embeddings for a graph neural network; note creating a graph that comprises assigning a plurality of image/screenshot nodes; note nodes are associated with the node embeddings of the images; note using the graph to train/update the graph neural network and obtain high-order relationship information and spatial information to perform segmentation functions on image content such as identifying and segmenting target objects or common objects, e.g., target embeddings; note the segmentation results are stored in the database, e.g., data store; note storing the graph neural networks in storage, which is broadly interepted as being stored as an index;);
receiving a query (Wang, [0017-0025], note identifying target objects is interpreted as a query for the targeted objects);
wherein the trained graph neural network is a neural network that maps initial node embeddings into updated node embeddings (Wang, figures 1-2, [0044-0045, 0092], note the computer vision system, which comprises the graph neural network, may be stored on the local computing devices; note the graph neural network architecture may be a pre-trained or preexisting neural network architecture, which means the computing device may receive a trained graph neural network)
While Wang teaches the creation and use of graph neural networks for images, Wang doesn’t specifically teach the trained graph neural network having been generated by the one or more servers based on a first graph that represents at least a first set of first images; text nodes; and the trained graph neural network is different than a plurality of different machine-trained models that are used by the computing system to produce the second graph. However, Dines is in the same field of endeavor, data analysis and information retrieval, and Dines teaches:
receiving parameters of a trained graph neural network from one or more servers, the trained graph neural network having been generated by the one or more servers based on a first graph that represents at least a first set of first images (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally; note the AI/ML models may be deep learning neural networks (DLNN), or any other suitable neural network, and graph neural networks are a type of DLNNs; note training set uses images. Also, when combined with the previously cited references the neural network would be a graph neural network as taught by Wang);
generating a second graph having screenshot nodes that are associated with a plurality of screenshots captured by the computing system, the screenshots being images that are different than the first images (Dines, [0031-0032, 0036], note task mining may include taking screenshots, which are different than the images used to train the models);
wherein the trained graph neural network is a neural network that maps initial node embeddings into updated node embeddings, and the trained graph neural network is different than a plurality of different machine-trained models that are used by the computing system to produce the second graph (Dines, figures 1 and 7, [0030-0032, 0099, 0102, 0124-0128], note training and fine-tuning AI/ML models on a remote server and then deploying it locally, which means the trained model is the neural network that maps the initial node embeddings into updated node embeddings; note since the training is performed separately and remote, it is interepted as different than the plurality of different machine-trained models; note using multiple AI/ML models analyze data; note the AI/ML models may be deep learning neural networks (DLNN), or any other suitable neural network, and graph neural networks are a type of DLNNs; note training set uses images. Also, when combined with the previously cited references the neural network would be a graph neural network as taught by Wang).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
While Wang as modified teaches the creation and use of graph neural networks for images, Wang as modified doesn’t specifically teach text nodes. However, Betthauser is in the same field of endeavor, data analysis and information retrieval, and Betthauser teaches:
the trained graph neural network having been generated by the one or more servers based on a first graph that represents at least a first set of first images (Betthauser, [0020], note the use of ImageNet and WordNet to train models. When combined with the previously cited references this would be for the pretrained graph as taught by Dines);
generating a second graph having screenshot nodes that are associated with a plurality of screenshots captured by the computing system and text nodes associated with instances of text that are associated with the screenshots, the screenshots being images that are different than the first images and the screenshot nodes and text nodes being graph nodes that are associated with respective target embeddings produced by the trained graph neural network and stored in an index (Betthauser, figures 1-3, 5, and 7, [0003, 0019-0022, 0028, 0034, 0037-0039, 0041, 0053], note using different machine learning models for different data types; note combining nodes of various embedding types such as text and image nodes, when combined with the previously cited references this would be for the images/screenshots as taught by Wang and Dines; note indexing the embeddings, when combined with the previously cited references this would be for the target image embeddings for screenshots);
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
While Wang as modified teaches the creation and use of graph neural networks, Wang as modified doesn’t specifically teach adding a query node that describes the query to a second graph to produce a third graph that represents an updated version of the second graph, generating a query embedding associated with the query node using the trained graph neural network based on the third graph; identifying a target embedding in the index that matches the query embedding; and retrieving a previously captured screenshot that is associated with the target embedding. However, Choubey, is in the same field of endeavor, data analysis, and Choubey teaches:
receiving a query (Choubey, figure 4A-4B, [0018-0019, 0074-0079], note receiving a query);
adding a query node that describes the query to a second graph to produce a third graph that represents an updated version of the second graph (Choubey, figure 4A-4B, [0019, 0074-0079], note adding a query node to the graph, which would produce a third/updated graph);
generating a query embedding associated with the query node using the trained graph neural network based on the third graph (Choubey, figures 4A-4B, [0019, 0074-0079], note generating a query representation, e.g., query embedding);
identifying a target embedding in the index that matches the query embedding (Choubey, figures 4A-4B, [0019, 0074-0081], note determining a portion of the graph associated with the query, e.g., identifying a target embedding. When combined with the previously cited references this would be the images/screenshots in the graph stored in the index as taught by Wang, Dines, and Betthauser); and
retrieving a previously captured screenshot that is associated with the target embedding (Choubey, figures 4A-4B, [0019, 0077-0083], note transmitting the results to a computer device, when combined with the previously cited references this would be the images/screenshots in the graph stored in the index as taught by Wang, Dines, and Betthauser).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Choubey because all references are directed to data analysis and because Choubey would expand upon the teachings of the previously cited references in data analysis which would improve the accuracy of the machine learned models by utilizing techniques that expose data that is relevant to a user’s input (Choubey, [0001, 0010-0011]).
Regarding Claim 15:
Wang as modified shows the system as disclosed above;
Wang as modified further teaches:
wherein the adding the query node comprises: using the plurality of machine-trained models to identify query features of the query (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image) (Dines, [0031-0032], note using multiple AI/ML models analyze data) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note using different machine learning models for different data types) (Choubey, figure 4A-4B, [0019, 0074-0079], note adding a query node to the graph);
using the query features to identify one or more links that connect the query node to one or more of the screenshot nodes in the second graph (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image) (Dines, [0031-0032], note using multiple AI/ML models analyze data) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note using different machine learning models for different data types) (Choubey, figure 4A, [0019, 0074-0081], note determining a portion of the graph associated with the query); and
adding one or more edges to the second graph associated with the one or more links (Dines, [0031-0032], note using multiple AI/ML models analyze data) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note generating a graph neural network with different edge types for various data types) (Choubey, figure 4A, [0019, 0074-0081], note adding a query node to the graph, note linking the node by adding edges to the other associated nodes; note determining a portion of the graph associated with the query).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Choubey because all references are directed to data analysis and because Choubey would expand upon the teachings of the previously cited references in data analysis which would improve the accuracy of the machine learned models by utilizing techniques that expose data that is relevant to a user’s input (Choubey, [0001, 0010-0011]).
Regarding Claim 16:
Wang as modified shows the system as disclosed above;
Wang as modified further teaches:
wherein the previously captured screenshot is associated with a target node in the third graph, and wherein the operations further include identifying neighbor graph nodes of the target node and retrieving information regarding one or more other screenshots that are associated with the neighbor graph nodes (Wang, figures 1 and 8, [0019-0022, 0045, 0048, 0090-0097], note receiving a collection of images; note the images may be captured and stored; note screenshots are images; note feature extraction component identifies features to generate a nodes embeddings for a graph neural network; note determining relationships among nodes; note nodes are associated with the node embeddings of the images) (Dines, [0031-0032], note using multiple AI/ML models analyze data) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note using different machine learning models for different data types) (Choubey, figures 3 and 4A-4B, [0019, 0067, 0077-0083], note determining a portion of the graph associated with the query, e.g., identifying a target embedding, includes identifying neighbor nodes which is interpreted as information regarding one or more other screenshots associated with the neighbor nodes when combined with the previously cited references; note transmitting the results to a computer device. When combined with the previously cited references this would be the images/screenshots as taught by Wang, Dines, and Betthauser).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Regarding Claim 25:
Wang as modified shows the method as disclosed above;
Wang as modified further teaches:
wherein the instances of text associated with the text nodes in the second graph correspond to text found in the screenshots represented by the second graph, and text provided by queries that were previously used to access particular screenshots (Wang, figure 8, [0003, 0019-0021, 0071, 0083, 0093-0097], note determining relationships/edges among nodes; note nodes are associated with the node embeddings of the images; note identifying common objects; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines) (Dines, [0031, 0036], note task mining may include taking screenshots) (Betthauser, figures 1-2, 5, and 7, [0003, 0019-0021, 0028, 0034, 0053], note combining nodes of various embedding types such as text and image nodes; note the text and image nodes may be shared content, which is interpreted as text nodes in the second graph corresponding to text found in screenshots).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
While Wang as modified teaches the creation and use of graph neural networks, Wang as modified doesn’t specifically teach text provided by queries that were previously used to access particular screenshots. However, Choubey, is in the same field of endeavor, data analysis, and Choubey teaches:
wherein the instances of text associated with the text nodes in the second graph correspond to text found in the screenshots represented by the second graph, and text provided by queries that were previously used to access particular screenshots (Choubey, figure 4A-4B, [0010, 0014 0019, 0052, 0074-0083], note adding a query node to the graph, which would produce a third/updated graph; note generating a query representation, e.g., query embedding; note determining a portion of the graph associated with the query, e.g., identifying a target embedding; note transmitting the results to a computer device, when combined with the previously cited references this would be the images/screenshots in the graph stored in the index as taught by Wang, Dines, and Betthauser; note nodes on the graph may also be text and image nodes).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Choubey because all references are directed to data analysis and because Choubey would expand upon the teachings of the previously cited references in data analysis which would improve the accuracy of the machine learned models by utilizing techniques that expose data that is relevant to a user’s input (Choubey, [0001, 0010-0011]).
Regarding Claim 26:
Wang as modified shows the system as disclosed above;
Wang as modified further teaches:
wherein the second graph includes different types of edges associated with different types of relationships between pairs of the screenshots (Betthauser, [0020], note generating a graph neural network with different edge types for various data types, such as text and image),
wherein the plurality of different machine-trained models include: a first machine- trained model that produces an image embedding based on at least one image region in a particular screenshot, a second machine-trained model that produces a text embedding based on at least one text region in the particular screenshot, and a third machine-trained model that produces a classification result based on the at least one text region and/or the at least one image region in the particular screenshot (Wang, figure 8, [0020, 0092, 0097], note feature extraction component identifies features using pre-trained or preexisting neural networks to generate a node embedding for the image, e.g., image embedding model) (Dines, [0031-0032], note using multiple AI/ML models analyze data. When combined with the previously cited references this would be for identifying features associated with the screenshots) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note using different machine learning models for different data types; note the different machine learning models may be for images, text, and classifications (e.g., graphs, languages, etc. are examples of a classification result). When combined with the previously cited references this would be for the images/screenshots), and
wherein the different types of edges include a first edge of a first edge type that represents common image embeddings produced by the first machine-trained model for two previously captured screenshots, a second edge of a second edge type that represents common text embeddings produced by the second machine-trained model for the two previously captured screenshots, and a third edge of a third edge type that represents a common classification result produced by the third machine-trained model for the two previously captured screenshots (Wang, figure 8, [0003, 0019-0021, 0071, 0083, 0093-0097], note determining relationships/edges among nodes; note nodes are associated with the node embeddings of the images; note determining/deriving the edges/relationships; note the edges are for similar image, e.g., image edge type; note identifying common objects; note screenshots are images and also when combined with the other references the images would be screenshots as taught by the other references cited below, such as Dines) (Betthauser, figures 1 and 5, [0008, 0019-0020, 0028, 0053], note generating a graph neural network with different edge types for various data types, such as text, image, and classifications (e.g., graphs, languages, etc. are examples of a classification result).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Dines because all references are directed to data analysis and because Dines would expand upon the teachings of the previously cited references in data analysis which would improve the analysis and accuracy of the system by utilizing pretraining and finetuning AI/ML models (Dines, [0055]).
It would have been obvious to one of ordinary skill in the art before the effective date of filing to modify the cited references to incorporate the teachings of Betthauser because all references are directed to data analysis and information retrieval and because Betthauser would expand upon the teachings of the previously cited references in data analysis information retrieval which would improve the performance and usability of the system by using machine learning graphs of various data types (Betthauser, abstract, [0037]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Yao et al. (US2020/0394499) teaches training a graph neural network;
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN J MORRIS whose telephone number is (571)272-3314. The examiner can normally be reached M-F 6:00-2:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James Trujillo can be reached at 571-272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN J MORRIS/Examiner, Art Unit 2151 6/18/2026
/James Trujillo/Supervisory Patent Examiner, Art Unit 2151
Prosecution Insights