DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/09/2026 has been entered.
Response to Arguments
Applicant's arguments, see Remarks pages 1-4, filed 06/09/2026, with respect to the rejections of amended claim(s) 1 & 9 under 35 U.S.C. 103 have been fully considered but they are not persuasive.
On page 3 of Remarks, Applicant argues:
PNG
media_image1.png
593
877
media_image1.png
Greyscale
Examiner respectfully disagrees.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., determination of “intrinsic physical or functional states of an object itself, such as a door being "opened," "closed," "broken," or "locked," as in embodiments corresponding to the present amended claims.”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
On paragraph 0074 of Im discloses “In Fig. 6, the relationship between the player and the ball can have various semantic relationships such as throws, hold, kick, and head, but since the context found by PLSI is basketball, the probability of it being a hold or throw in the input image is high. Also, if you look at the distance between the player and the ball, the probability of it being a hold is very high”. A deep learning-based relationship detection method is disclosed, which based on the objects detected in an image, such as a player and a ball, and based on a determined context of the image, semantic relationships between objects can be determined. Wherein, the semantic relationships between objects, such as a player holding a ball, discloses a physical state, where the ball is being held by the player, and a functional state, where the player is performing the function of holding the ball.
Therefore, Im discloses the acquiring of information about possible physical or functional states of objects.
On page 3 of Remarks, Applicant argues:
PNG
media_image2.png
401
880
media_image2.png
Greyscale
Applicant’s arguments have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 7, 9-12, and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Moon et al. (PDDL Planning with Natural Language-Based Scene Understanding for UAV-UGV Cooperation) hereinafter referenced as Moon, in view of Im et al. (KR102185777B1) hereinafter referenced as Im, and Lecue et al. (US2021264226A1) hereinafter referenced as Lecue.
Regarding claim 1, Moon discloses: A method for organizing and merging a scene of an Artificial Intelligence (Al) agent (Moon: Figure 2; Abstract), comprising: acquiring an image of a first space (Moon: Figure 3; 4.1. Experiment Setting: “Each Smart Cookie has 2D laser sensors and an RGB-D camera. Beyond is equipped with an RGB-D camera. The laser sensor is used for navigation on the execution part while the RGB-D cameras are used for cognition for the natural language-based scene understanding.”);
recognizing objects in the image; structuring information about a relationship between the objects and information about states of the objects in a form of a graph (Moon: Figure 8b; 3.1. Natural Language-Based Cognition: “In this study, we assume that the robots use a graph map (for motion planning) generated using semantic SLAM, which is a widely used environment representation method in robotics [7]. To utilize the graph map G = (V, E) that contains features and positions of the detected object as nodes vi ∈ V and their relationships as edges eij = (vi , vj) ∈ Eij, we closely follow Moon and Lee [41] for generating the language description and graph inference phase of Xu et al. [42] for the scene graph generation.”; Wherein the scene graph generated from the graph map contains the information regarding object relationships and object states.); and
merging the structured information with information received from another AI agent (Moon: 3. Architecture: “our framework entails natural language-based cognition and a knowledge engine for multiple agents…sensor information obtained from environments is continuously passed to cognition. During cognition, scene understanding-based natural language is created by generating language description and scene understanding using deep learning techniques. Then, the generated semantic information is passed on to the knowledge engine while raw sensor data are sent to episodic memory storage.”; 3.2. Knowledge Engine: “The knowledge engine obtains semantic environmental information in XML and stores it in triple store, which uses a resource description framework (RDF) such as ”subject-predicate-object” or ”resource-property type-value” unlike the conventional relational database that saves data in ”key-value.” Triple store uses the SPARQL protocol and RDF query language (SPARQL) to create, read, update, and delete the graph data that contain relations between objects.”; Wherein the multiple agents are continuously providing semantic environmental data, which is stored and shared between all agents.),
wherein merging the structured information comprises: detecting a change in a scene at a specific time point (Moon: Section: 4.2: “Every robot was required to report the current situation to the DICQ.R as well as if an unusual situation occurred. During the mission, we surmised what may happen if a dynamic obstacle, which a robot could not approach, were to suddenly appear at the POI. In this situation, the robot will generate natural language to report the current situation to the DICQ.R.”);
generating a summary sentence describing the detected change in the scene using a language-vision model based on words and visual features corresponding to the detected change (Moon: 3.1. Natural Language-Based Cognition: “Then, an RNN is used to generate a language description over the graph. The RNN takes the encoded graph features concatenated with a word vector and predicts the probabilistic distribution of the target word vector. Given that we also back-propagate the GCN when training the RNN, we can expect that graph features that fit the generated sentence will be extracted. The generated description can be used to understand the surrounding environment when an unexpected situation occurs.”); and
exchanging the generated summary sentence with the other AI agent (Moon: Figure 7: “• Perform ‘find missing child’ mission received from DICQ.R (A2)
1. DICQ.R command Smart cookie1,2,3, and Beyond-pf1 to find a missing child
2. Smart cookie1,2,3, and Beyond-pf1 visit every POI to find the missing child
3. Beyond-pf1 find a human
4. Beyond-pf1 create POI at human position
5. DICQ.R generate new mission for Smart cookie1,2,3 to go to created POI to check the found human is the missing child…
•Smart cookie cannot approach POI due to a dynamic obstacle (A3)
1. Smart cookie generates natural language to describe the current situation to DICQ.R
2. Replanning”;
3. Architecture: “our framework entails natural language-based cognition and a knowledge engine for multiple agents…sensor information obtained from environments is continuously passed to cognition. During cognition, scene understanding-based natural language is created by generating language description and scene understanding using deep learning techniques. Then, the generated semantic information is passed on to the knowledge engine while raw sensor data are sent to episodic memory storage. Using the episodic memory and knowledge collected from multiple robots, the PDDL planning agent builds a sequence of actions for each agent.”;
Wherein the generated semantic information, such as natural language and scheme graphs, from each robot/agent are collected and processed in order to form a collective scene understanding for action coordination.).
Moon does not disclose expressly: wherein merging the structured information comprises: inputting labels of respective objects in the scene to a pretrained language model configured to determine contextually appropriate state-related candidate words combinable with the object labels; and generating a summary sentence describing the detected change in the scene using a language-vision model based on the candidate words and visual features corresponding to the detected change; wherein the inputting labels of respective objects in the scene comprises: referring to a predefined language knowledge graph to acquire, based on the labels, information about possible physical or functional states of the objects; and selecting, by the pretrained language model, the candidate words from the acquired information.
Im discloses: inputting labels of respective objects in the scene to a pretrained language model configured to determine contextually appropriate state-related candidate words combinable with the object labels (Im: 0001: “The present invention relates to a method for recognizing semantic relationships of image objects based on deep learning and PLSI using a computer, and more specifically, to a method for recognizing semantic relationships of image objects based on deep learning and PLSI that introduces PLSI into the recognition of semantic relationships of image objects using deep learning to recognize objects contained in an image and find relationships between the recognized objects.”); wherein the inputting labels of respective objects in the scene comprises: referring to a knowledge base to acquire, based on the labels, information about possible physical or functional states of the objects; and selecting, by the pretrained language model, the candidate words from the acquired information (Im: 0015: “An ontology expresses the concepts, characteristics, and associations of words used within a certain range, explicitly revealing general knowledge about the words and enabling the understanding of sentence meaning through the definition of relationships between words; thus, relationship detection using ontology is a method of identifying relationships between objects by utilizing a knowledge base.”;
0073-0074: “Then, using context, existing deep learning-based relationship detection methods, such as spatial relationship detection utilizing spatial features and ontology methods, are used to find relationships between objects.
In Fig. 6, the relationship between the player and the ball can have various semantic relationships such as throws, hold, kick, and head, but since the context found by PLSI is basketball, the probability of it being a hold or throw in the input image is high. Also, if you look at the distance between the player and the ball, the probability of it being a hold is very high.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the semantic graph generation method for the generation of natural language descriptions and scene graphs disclosed by Moon with the deep learning based scene graph generation method taught by Im. The suggestion/motivation for doing so would have been “existing scene graph generation methods primarily rely on locating objects within an image and then identifying their relationships; consequently, they have disadvantages such as the possibility of incorrectly detecting relationships between objects and the inability to detect relationships more precisely” (Im: 0017). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results.
Moon in view of Im does not disclose expressly: wherein the inputting labels of respective objects in the scene comprises: referring to a predefined language knowledge graph to acquire, based on the labels, information about possible physical or functional states of the objects.
Lecue discloses: A method of semantic object detection within an image by detecting objects within an image, and comparing the detected objects to semantic links, which were extracted based on their relevance to the image dataset (Lecue: Abstract). Wherein the method further comprises: referring to a predefined language knowledge graph to acquire, based on the labels, information about possible physical or functional states of the objects (Lecue: 0009: “spatially relative terms, such as “beneath,” “below,” “lower,” “above,” “upper” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures.”;
0014: “A knowledge graph 114 is a database of labels and semantic links. A semantic link 122 defines the relationships between objects represented by labels. For example, the semantic link ““NOSE” is a part of “FACE”” defines the relationship between a nose and a face.”;
0039-0041: “The 100 initial object predictions 336 are processed by the semantic augmenter 306. Comparing semantic augmentation hyperparameters 338 such as thresholds 340, a select predictions module selects predictions to augment 344. Using the semantic links between labels 322 generated by the context extractor 302, a semantic coherence module 346 compares the predictions to augment 344 with the semantic links between labels 322…The method semantically interprets objects in data, e.g., identifying an object as a car because the object has been identified as a vehicle with four wheels, windows, on a road, with people inside, or the like. A structured database, such as a knowledge graph, is used to correlate objects that compose the scene, and to extract a logical picture of the object interrelations.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the use of predefined knowledge graphs containing semantic links as taught by Lecue for the context and deep learning-based relationship detection methods disclosed by Moon in view of Im. The suggestion/motivation for doing so would have been “The method semantically interprets objects in data, e.g., identifying an object as a car because the object has been identified as a vehicle with four wheels, windows, on a road, with people inside, or the like. A structured database, such as a knowledge graph, is used to correlate objects that compose the scene, and to extract a logical picture of the object interrelations” (Lecue: 0041). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Moon in view of Im with Lecue to obtain the invention as specified in claim 1.
Regarding claim 2, Moon in view of Im and Lecue discloses: The method of claim 1, wherein: the objects are recognized based on a first AI neural network that is pretrained by receiving an image of a specific space as input (Im: 0019-0020: “To achieve the objective of the present invention, a method for recognizing semantic relationships of image objects based on deep learning and PLSI according to the present invention comprises: a step of receiving an image input; A step of detecting objects in the image using a deep learning-based object detection method”), and the information about the states of the objects is acquired by inputting the objects to a second AI neural network that is trained based on training data formed by labeling objects with respective states (Im: 0072-0073: “Then, PLSI is used to detect the context of the image. As shown in Fig. 6, the object named person is replaced by the player object due to the contextual situation, and the object named ball is materialized into the basketball object. In other words, objects within a comprehensive upper scope are limited to objects within a more detailed lower scope. Then, using context, existing deep learning-based relationship detection methods, such as spatial relationship detection utilizing spatial features and ontology methods, are used to find relationships between objects.”).
Regarding claim 3, Moon in view of Im and Lecue discloses: The method of claim 1, wherein merging the structured information comprises merging information corresponding to the first space with information corresponding to a second space of the other AI agent (Moon: 3. Architecture: “our framework entails natural language-based cognition and a knowledge engine for multiple agents…sensor information obtained from environments is continuously passed to cognition. During cognition, scene understanding-based natural language is created by generating language description and scene understanding using deep learning techniques. Then, the generated semantic information is passed on to the knowledge engine while raw sensor data are sent to episodic memory storage.”; Wherein the agents each share information regarding their current location).
Regarding claim 4, Moon in view of Im and Lecue discloses: The method of claim 3, wherein the merged information is configured such that a positional relationship between the first space and the second space is represented in a form of a link (Moon: 3. Architecture: “our framework entails natural language-based cognition and a knowledge engine for multiple agents…sensor information obtained from environments is continuously passed to cognition. During cognition, scene understanding-based natural language is created by generating language description and scene understanding using deep learning techniques. Then, the generated semantic information is passed on to the knowledge engine while raw sensor data are sent to episodic memory storage.”;
3.2. Knowledge Engine: “The knowledge engine obtains semantic environmental information in XML and stores it in triple store…The triple store facilitates the reasoning process by using the relations and attributes between objects to find new relations.”;
4.2. Scenario: “we surmised what may happen if a dynamic obstacle, which a robot could not approach, were to suddenly appear at the POI. In this situation, the robot will generate natural language to report the current situation to the DICQ.R.”; Wherein in the case that a robot/agent’s mission were to fail due to a dynamic object at its location, the knowledge engine, and thus the DICQ.R is updated for the replanning of robot/agents’ actions, thus indicating a link between each robot’s location based on the information in the knowledge engine.).
Regarding claim 7, Moon in view of Im and Lecue discloses: The method of claim 1, wherein merging the structured information comprises exchanging information about an object having a state that is updated over time, with the other AI agent (Moon: Figure 7: “• Perform ‘find missing child’ mission received from DICQ.R (A2)
1. DICQ.R command Smart cookie1,2,3, and Beyond-pf1 to find a missing child
2. Smart cookie1,2,3, and Beyond-pf1 visit every POI to find the missing child
3. Beyond-pf1 find a human
4. Beyond-pf1 create POI at human position
5. DICQ.R generate new mission for Smart cookie1,2,3 to go to created POI to check the found human is the missing child…
•Smart cookie cannot approach POI due to a dynamic obstacle (A3)
1. Smart cookie generates natural language to describe the current situation to DICQ.R
2. Replanning”;
3. Architecture: “our framework entails natural language-based cognition and a knowledge engine for multiple agents…sensor information obtained from environments is continuously passed to cognition. During cognition, scene understanding-based natural language is created by generating language description and scene understanding using deep learning techniques. Then, the generated semantic information is passed on to the knowledge engine while raw sensor data are sent to episodic memory storage. Using the episodic memory and knowledge collected from multiple robots, the PDDL planning agent builds a sequence of actions for each agent.”).
As per claim 9, arguments made in rejecting claim 1 are analogous. In addition, Section 4.1. Experiment Setting of Moon discloses the experiment and its components, which comprises memory in which at least one program is recorded; and a processor for executing the program.
As per claim 10, arguments made in rejecting claim 2 are analogous.
As per claim 11, arguments made in rejecting claim 3 are analogous.
As per claim 12, arguments made in rejecting claim 4 are analogous.
As per claim 15, arguments made in rejecting claim 7 are analogous.
Claim(s) 5-6 and 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Moon in view of Im and Lecue, and further in view of Dorr et al. (Cooperative Longterm SLAM for Navigating Mobile Robots in Industrial Applications) hereinafter referenced as Dorr.
Regarding claim 5, Moon in view of Im and Lecue discloses: The method of claim 1.
Moon in view of Im and Lecue does not disclose expressly: wherein merging the structured information comprises determining whether to merge the structured information with the information received from the other AI agent based on metadata of the other AI agent, and the metadata includes timestamp information corresponding to the information about the states of the objects.
Thus, Moon in view of Im and Lecue does not disclose merging the stored environmental data collected by its agents based on metadata, including timestamp information.
Dorr discloses: the merging of environmental data, using data collected by mobile agents, by dividing the environmental map into a grid and merging the cells based on the cell’s most recent observation (Dorr: III. D. Map Transmission: “the map upstream from the agent to the server is realized by a periodic request by the server. This enables efficient work load management. In order to reduce bandwidth requirements, cells are tagged, that have been observed since the last map upload. As a result, only these cells have to be transmitted and merged, which substantially increases the efficiency.”; III. E. Map Merging: “Map merging is needed when agents receive new map information from the server or when detected changes of the agents have to be merged with the server's global map. In both cases, we use a timestamp-based strategy where each cell keeps the timestamp of its most current observation. Following this approach, maps can easily be merged by only using the latest updated cell and discarding all obsolete one”; Wherein the observation timestamp of a cell is updated based on when a robot observes the cell.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of merging map information based on the most recent observation disclosed by Dorr by merging/processing the robot/agent generated semantic information stored in the Knowledge Engine disclosed by Moon in view of Im and Lecue based on each object’s most recent observation. The suggestion/motivation for doing so would have been “Following this approach, maps can easily be merged by only using the latest updated cell and discarding all obsolete ones. This produces almost no overhead and leverages prior available information” (Dorr: III. E. Map Merging). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Moon in view of Im and Lecue with Dorr to obtain the invention as specified in claim 5.
Regarding claim 6, Moon in view of Im, Lecue, and Dorr discloses: The method of claim 5, wherein merging the structured information comprises merging the information about the states of the objects determined to have been updated within a preset time period back from a current time using the timestamp information (Dorr: III. E. Map Merging: “Map merging is needed when agents receive new map information from the server or when detected changes of the agents have to be merged with the server's global map. In both cases, we use a timestamp-based strategy where each cell keeps the timestamp of its most current observation. Following this approach, maps can easily be merged by only using the latest updated cell and discarding all obsolete one”; Wherein the time between the last updated timestamp and the current time constitutes a preset time period.).
As per claim 13, arguments made in rejecting claim 5 are analogous.
As per claim 14, arguments made in rejecting claim 6 are analogous.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights