CTNF 19/281,254 CTNF 82689 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. This Office action is issued in response to application, 19/281,254, filed on 7/25/2025. Claim(s) 1-20 is/are pending. Priority Acknowledgement is made of applicant’s claim for priority to provisional application, 63/676,425, filed on 7/28/2024. Specification 06-31 AIA The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification. Information Disclosure Statement The information disclosure statement(s) (IDS), submitted on 11/26/2025, is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) is/are being considered by the examiner. Claim Objections 07-29-01 AIA Claim (s) 12 is/are objected to because of the following informalities: in line 2, “nodes comprise s one” should be corrected to “nodes comprise one” . Appropriate correction is required. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claim(s) 1-11 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. 07-34-05 AIA Claim 1 recites the limitation " the scene data " in line 2 . There is insufficient antecedent basis for this limitation in the claim. Claim(s) 2-11 inherit(s) the deficiencies of the claim it/they depend(s) from. 07-34-05 AIA Claim 1 recites the limitation " the scene " in line 5 . There is insufficient antecedent basis for this limitation in the claim. Claim(s) 2-11 inherit(s) the deficiencies of the claim it/they depend(s) from. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al., US 2022/0391433 A1 (hereinafter “Mah”) in view of Mehedy et al., :US 2017/0277693 A1 (hereinafter “Meh”) . Claims 1 and 20 Mah discloses one or more processors comprising one or more processing units (Mah, [0061], see graphics processing unit) to: in response to an extraction of the scene data (Mah, [0081], see, as illustrated in FIG. 7 , an image is input to a scene graph generator 700 to produce a scene graph . The scene graph includes at least one node representing an object and at least one edge representing a relationship between two objects . Scene graph generator 700 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . The scene graph is then fed into a word embedding component 705. Word embedding component 705 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . Word embedding component 705 generates a node vector for the node. The node vector represents semantic information of the object . Word embedding component 705 also generates an edge vector for the edge, and the edge vector represents semantic information of the relationship . The node vector and the edge vector are input to GCN 710 to produce a scene graph embedding. GCN 710 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . The scene graph embedding is then input to metadata component 715 to produce metadata [i.e., corresponds to “extracted scene data”]), index the scene data to cause the scene data to be queryable (Mah, [0098], see, at operation 825, the system assigns metadata to the image based on the scene graph embedding. For example, the metadata may include values from the scene graph embedding that can be used for indexing the image in a database . In some cases, metadata information for each of the search objects (i.e., other images) are stored in the database. Metadata of the image (e.g., a query image) is compared to metadata associated with each of the search objects stored in a database, for example. The system then runs a nearest neighbor search over the metadata of the image and the metadata of the search objects to retrieve one or more matching images [i.e., images are able to be retrieved, which discloses “using an index to cause the scene data to be queryable”]); based at least on the extraction of the scene data and the scene data being indexed, detect first information associated with the scene by generating a first query via an Al agent; and in response to the first query being generated, automatically detect second information associated with the scene by generating a second query via the Al agent. Mah does not appear to explicitly disclose based at least on the extraction of the scene data and the scene data being indexed, detect first information associated with the scene by generating a first query via an Al agent; and in response to the first query being generated, automatically detect second information associated with the scene by generating a second query via the Al agent. Meh discloses based at least on the extraction of the scene data and the scene data being indexed, detect first information associated with the scene by generating a first query via an Al agent (Meh, [0005], see one or more of the hardware processors may be further operable to generate one or more search queries based on the ontology graph , by inputting the ontology graph to a query generating machine learning model [i.e., and the process that actually invokes the machine learning model corresponds to the “AI agent”] trained to predict the one or more search queries); and in response to the first query being generated, automatically detect second information associated with the scene by generating a second query via the Al agent (Meh, [0031], see the ontology processor output 402 is generated into search queries for relevant items at 404 . The ontology processor 304 in one embodiment may construct search queries [i.e., more than 1 is generated] and dispatch to a background search scheduler . The search queries may be launched, for example, as a computer executable process, for example, scheduled into a background search scheduler 406 for execution; and Meh, Fig. 4, see “Ontology processor output” 402 inputted into “Search queries for relevant items” 404 and at least 4 background search tasks [i.e., more than 1 is generated] generated in 406). Mah and Meh are analogous art because they are from the same field of endeavor of image/scene searching. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, having the teachings of Mah and Meh before him/her, to modify the scene searching of Mah to include the query generation of Meh because it would improve search results. The suggestion/motivation for doing so would have been to automate analysis and interpretation of project data and storage of that information and automate content searching for tailored content relevant, see Meh, [0019]. Therefore, it would have been obvious to combine Meh with Mah to obtain the invention as specified in the instant claim(s). Claim(s) 20 recite(s) similar limitations to claim 1 and is/are rejected under the same rationale. Claims 2 and 13 With respect to claims 2 and 13, the combination of Mah and Meh discloses wherein the scene data is extracted by at least one of: detecting an object in the scene via object detection, extracting a spatial property of the scene, extracting a visual property of the scene, extracting a natural language semantic label of the object in the scene (Meh, [0027], see generating ontology the result of natural language processing may store identified keywords associated by context relationships represented by a graph network; Meh, [0030], see the search results generated from the search may be categorized and weighted as shown at 308. User feedback, for example, may be received and used to generate weights to associate with the results. The results may be tagged with weights and fed back to the ontology processor 304 as input as shown at 302. The ontology processor 304 may further execute one or more cognitive or machine-learning based algorithms to further learn and correct its process of generating the result sub-graph , for example, based on feedback (e.g., weighted results); and Meh, [0029] above for disclosing a semantic search/information), extracting an embedding that captures a property of the scene (Mah, [0020], see a scene graph embedding of a query image), or extracting one or more object-assigned data attributes. Claim 3 With respect to claim 3, the combination of Mah and Meh discloses wherein the scene data is extracted by extracting one or more object-assigned data attributes including at least one of: a physical property, a technical specification, an origin identifier, a value indicator (Meh, [0029], see a node in the graph contains a context keyword and an edge between the nodes contains an extracted and weighted feature value), a reference to an external system, or dynamic data associated with the object from a real-time data source. Claims 4 and 14 With respect to claims 4 and 14, the combination of Mah and Meh discloses wherein the scene data in indexed by indexing the scene data into at least one of: a graph database storing objects as nodes and their relationships as edges (Meh, [0029], see a node in the graph contains a context keyword and an edge between the nodes contains an extracted and weighted feature value), a spatial database storing geometric properties and spatial relationships of objects for spatial queries, or a dependency structure capturing dependency information between the scene and a second scene. Claims 5 and 15 With respect to claims 5 and 15, the combination of Mah and Meh discloses wherein the first information and the second information are detected based on receiving a response from one or more Application Programming Interfaces (APIs) that retrieve at least a portion of the scene data from at least one of the graph database (Meh, [0022], see a service application programming interface (API) 104 may receive the captured data from the information capturing device 102 and store processed information in an image and metadata store or database 106. The service API 104 may be also communicatively and/or operatively coupled to other data source, for example, collaborative tool or others 108, for instance, being used during project collaboration session, from which the service API 104 may directly receive data or information generated or discussed, e.g., during the collaboration session; and Mah, [0081], see, as illustrated in FIG. 7 , an image is input to a scene graph generator 700 to produce a scene graph . The scene graph includes at least one node representing an object and at least one edge representing a relationship between two objects . Scene graph generator 700 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . The scene graph is then fed into a word embedding component 705. Word embedding component 705 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . Word embedding component 705 generates a node vector for the node. The node vector represents semantic information of the object . Word embedding component 705 also generates an edge vector for the edge, and the edge vector represents semantic information of the relationship . The node vector and the edge vector are input to GCN 710 to produce a scene graph embedding. GCN 710 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . The scene graph embedding is then input to metadata component 715 to produce metadata ), the spatial database, or the dependency structure. Claims 6 and 16 With respect to claims 6 and 16, the combination of Mah and Meh discloses wherein the AI agent generates the first query and the second query based at least on one of prompt engineering or tuning on example query-response pairs (Mah, [0053], see there are two example queries (i.e., query image 500). The first query image 500 shows a man throwing frisbee. The second query image 500 shows a man playing with dog. The first row (having top eight retrieved images 505) associated with the first query image show results retrieved using scene graph embeddings as described in the present disclosure. The second row (having top eight retrieved images 505) associated with the first query image 500 is retrieved using classification features in conventional image processing systems ). Claims 7 and 17 With respect to claims 7 and 17, the combination of Mah and Meh discloses wherein the one or more processing units are further to: detect, based at least on the first information and the second information, a gap in scene understanding of the scene (Mah, [0040], see scene graphs are used to close or reduce the semantic gap between low-level visual features and high-level concepts in image retrieval tasks); and trigger, based at least on the gap being detected, a follow-up query to detect additional information associated with the scene (Mah, [0101], see a single GCN layer updates all the node and edge representations by pooling information from local neighborhoods. When this process is repeated iteratively through [i.e., corresponds to the “follow-up query”] a stack of GCN layers , the resulting state vectors capture information from the entire scene graph). Claims 8 and 18 With respect to claims 8 and 18, the combination of Mah and Meh discloses wherein the second query is generated in response to detecting a change in the scene based at least on monitoring the scene and updating the indexed scene data (Mah, [0033], see the system may also keep a log of changes in the captured images for later reference. ... In one embodiment, found items are visually linked to the extracted items on the original input data and image . In one embodiment, the system may further allow the users to add additional keywords/annotations associated with each of the extracted items to improve the automated search results and associated machine-learning algorithms . In one embodiment, the machine-learning algorithms are used to continuously to improve both the search criteria and to better tailor and associate the domain expertise of the collaborators with that of the most appropriate search results). Claim 9 With respect to claim 9, the combination of Mah and Meh discloses wherein the one or more processing units are further to: generate, via the AI agent , at least a third query (Meh, [0005], see one or more of the hardware processors may be further operable to generate one or more search queries based on the ontology graph , by inputting the ontology graph to a query generating machine learning model [i.e., and the process that actually invokes the machine learning model corresponds to the “AI agent”] trained to predict the one or more search queries) in a loop (Mah, [0101], see a single GCN layer updates all the node and edge representations by pooling information from local neighborhoods. When this process is repeated iteratively through [i.e., corresponds to the “third query”] a stack of GCN layers , the resulting state vectors capture information from the entire scene graph) to detect at least one of spatial information, semantic information (Meh, [0029], see the ontology processor 304 in one embodiment may traverse the ontology graph and based on the ontology graph and the context keywords, the ontology processor 304 may generate a result sub-graph comprising relevant items. For instance, to generate the result sub-graph, the ontology graph may be pruned based on the weighting of the nodes and edges. The relevant items associated with the nodes of the sub-graph may be identified or determined by executing a text analytics semantic query [i.e., where at what similarity is deemed to be relevant corresponds to the “threshold”] on a knowledgebase to identify items associated with the nodes (e.g., keywords represented by the nodes)), or dependency-based information associated with the scene; and based at least on the generation of the third query in loop, update an index with the at least one of spatial information, semantic information (Meh, [0030], see the search results generated from the search may be categorized and weighted as shown at 308. User feedback, for example, may be received and used to generate weights to associate with the results. The results may be tagged with weights and fed back to the ontology processor 304 as input as shown at 302. The ontology processor 304 may further execute one or more cognitive or machine-learning based algorithms to further learn and correct its process of generating the result sub-graph , for example, based on feedback (e.g., weighted results); and Meh, [0029] above for disclosing a semantic search/information), or dependency-based information. Claim 10 With respect to claim 10, the combination of Mah and Meh discloses wherein the one or more processing units are further to: execute a user query based at least on accessing the updated index and matching one or more terms of the user query to one or more terms stored to the updated index (Mah, [0101], see a single GCN layer updates all the node and edge representations by pooling information from local neighborhoods. When this process is repeated iteratively through a stack of GCN layers , the resulting state vectors capture information from the entire scene graph). Claim 11 With respect to claim 11, the combination of Mah and Meh discloses wherein the one or more processors is comprised in at least one of: a control system for an autonomous or semi-autonomous machine (Meh, [0028], see the ontology processor's process is a continuous and autonomous process that receives input, automatically generates relevant items for search queries, automatically learns to improve its process of generating the relevant items for search queries based on feedback received as input); a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system for generating synthetic data using one or more large language models (LLMs); a system for generating synthetic data using one or more vision language models (VLMs); a system for generating synthetic data using one or more multi-modal language models; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Mah, [0027], see the user device 105 communicates with the image processing apparatus 110 via the cloud 115). Claim 12 Mah discloses a data center system comprising a plurality of computing nodes, wherein two or more computing nodes of the plurality of computing nodes (Mah, [0035], see data center distributed over multiple locations from central servers [i.e., corresponds to the “computing nodes”]) comprises one or more graphics processing units (GPUs) (Mah, [0061], see graphics processing unit) to: obtain extracted scene data of a scene (Mah, [0081], see, as illustrated in FIG. 7 , an image is input to a scene graph generator 700 to produce a scene graph . The scene graph includes at least one node representing an object and at least one edge representing a relationship between two objects . Scene graph generator 700 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . The scene graph is then fed into a word embedding component 705. Word embedding component 705 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . Word embedding component 705 generates a node vector for the node. The node vector represents semantic information of the object . Word embedding component 705 also generates an edge vector for the edge, and the edge vector represents semantic information of the relationship . The node vector and the edge vector are input to GCN 710 to produce a scene graph embedding. GCN 710 is an example of, or includes aspects of, the corresponding element described with reference to FIG. 6 . The scene graph embedding is then input to metadata component 715 to produce metadata [i.e., corresponds to “extracted scene data”]); store, in response to the obtaining of the extracted scene data (Mah, [0081], see the scene graph with nodes and edges described above), the extracted scene data using an index to cause the scene data to be queryable (Mah, [0098], see, at operation 825, the system assigns metadata to the image based on the scene graph embedding. For example, the metadata may include values from the scene graph embedding that can be used for indexing the image in a database . In some cases, metadata information for each of the search objects (i.e., other images) are stored in the database. Metadata of the image (e.g., a query image) is compared to metadata associated with each of the search objects stored in a database, for example. The system then runs a nearest neighbor search over the metadata of the image and the metadata of the search objects to retrieve one or more matching images [i.e., images are able to be retrieved, which discloses “using an index to cause the scene data to be queryable”]); based at least on the extracted scene data being obtained and stored using the index, automatically generate, via an Al agent, a plurality of queries until a threshold of at least one of spatial information, semantic information, or dependency-based information associated with the scene is met; and based at least on the generating of the plurality of queries and the threshold being met, update the index with at least one of the spatial information, semantic information, or dependency-based information . Mah does not appear to explicitly disclose based at least on the extracted scene data being obtained and stored using the index, automatically generate, via an Al agent, a plurality of queries until a threshold of at least one of spatial information, semantic information, or dependency-based information associated with the scene is met; and based at least on the generating of the plurality of queries and the threshold being met, update the index with at least one of the spatial information, semantic information, or dependency-based information. Meh discloses based at least on the extracted scene data being obtained and stored using the index, automatically generate, via an Al agent (Meh, [0005], see one or more of the hardware processors may be further operable to generate one or more search queries based on the ontology graph , by inputting the ontology graph to a query generating machine learning model [i.e., and the process that actually invokes the machine learning model corresponds to the “AI agent”] trained to predict the one or more search queries), a plurality of queries (Meh, [0031], see the ontology processor output 402 is generated into search queries for relevant items at 404 . The ontology processor 304 in one embodiment may construct search queries and dispatch to a background search scheduler . The search queries may be launched, for example, as a computer executable process, for example, scheduled into a background search scheduler 406 for execution; and Meh, Fig. 4, see “Ontology processor output” 402 inputted into “Search queries for relevant items” 404 and at least 4 background search tasks generated in 406) until a threshold of at least one of spatial information, semantic information, or dependency-based information associated with the scene is met (Meh, [0029], see the ontology processor 304 in one embodiment may traverse the ontology graph and based on the ontology graph and the context keywords, the ontology processor 304 may generate a result sub-graph comprising relevant items. For instance, to generate the result sub-graph, the ontology graph may be pruned based on the weighting of the nodes and edges. The relevant items associated with the nodes of the sub-graph may be identified or determined by executing a text analytics semantic query [i.e., where at what similarity is deemed to be relevant corresponds to the “threshold”] on a knowledgebase to identify items associated with the nodes (e.g., keywords represented by the nodes)); and based at least on the generating of the plurality of queries and the threshold being met, update the index with at least one of the spatial information, semantic information , or dependency-based information (Meh, [0030], see the search results generated from the search may be categorized and weighted as shown at 308. User feedback, for example, may be received and used to generate weights to associate with the results. The results may be tagged with weights and fed back to the ontology processor 304 as input as shown at 302. The ontology processor 304 may further execute one or more cognitive or machine-learning based algorithms to further learn and correct its process of generating the result sub-graph , for example, based on feedback (e.g., weighted results); and Meh, [0029] above for disclosing a semantic search/information). See claims 1 and 20 above for the motivation to combine. Claim 19 With respect to claim 19, the combination of Mah and Meh discloses wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine (Meh, [0028], see the ontology processor's process is a continuous and autonomous process that receives input, automatically generates relevant items for search queries, automatically learns to improve its process of generating the relevant items for search queries based on feedback received as input); a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system for generating synthetic data using one or more large language models (LLMs); a system for generating synthetic data using one or more vision language models (VLMs); a system for generating synthetic data using one or more multi-modal language models; or a system incorporating one or more virtual machines (VMs) . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. – Riscutia et al., 2021/0064609 for generating data retrieval queries using a knowledge graph; – Wang, 2025/0217407 for multi-modal utility asset searching; – Sharifi et al., 2020/0050610 for mapping images to search queries; – Sun et al., 2022/0414138 for video processing optimization and content searching; – Ma et al., KR 20220052705 for providing video; – Koo et al., WO 202121209 for searching for information inside video; – Watanabe et al., WO 0235390 for dynamic image content search information management; –JP 2463820 for network-based information retrieval; and – Watanabe et al., CN 1312616 for moving image content search. Point of Contact Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUBERT G CHEUNG whose telephone number is (571) 270-1396. The examiner can normally be reached M-R 8:00A-5:00P EST; alt. F 8:00A-4:00P EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Apu Mofiz can be reached at (571) 272-4080. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. HUBERT G. CHEUNG Assistant Examiner Art Unit 2161 Examiner: Hubert Cheung /Hubert Cheung/Assistant Examiner, Art Unit 2161Date: May 26, 2026 /APU M MOFIZ/Supervisory Patent Examiner, Art Unit 2161 Application/Control Number: 19/281,254 Page 2 Art Unit: 2161 Application/Control Number: 19/281,254 Page 3 Art Unit: 2161 Application/Control Number: 19/281,254 Page 4 Art Unit: 2161 Application/Control Number: 19/281,254 Page 5 Art Unit: 2161 Application/Control Number: 19/281,254 Page 6 Art Unit: 2161 Application/Control Number: 19/281,254 Page 7 Art Unit: 2161 Application/Control Number: 19/281,254 Page 8 Art Unit: 2161 Application/Control Number: 19/281,254 Page 9 Art Unit: 2161 Application/Control Number: 19/281,254 Page 10 Art Unit: 2161 Application/Control Number: 19/281,254 Page 11 Art Unit: 2161 Application/Control Number: 19/281,254 Page 12 Art Unit: 2161 Application/Control Number: 19/281,254 Page 13 Art Unit: 2161 Application/Control Number: 19/281,254 Page 14 Art Unit: 2161 Application/Control Number: 19/281,254 Page 15 Art Unit: 2161 Application/Control Number: 19/281,254 Page 16 Art Unit: 2161 Application/Control Number: 19/281,254 Page 17 Art Unit: 2161 Application/Control Number: 19/281,254 Page 18 Art Unit: 2161 Application/Control Number: 19/281,254 Page 19 Art Unit: 2161 Application/Control Number: 19/281,254 Page 20 Art Unit: 2161
Prosecution Insights