DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/19/2026 has been entered.
Status of Claims
Claims 1-20 of U.S. Application No. 19/327240 filed on 06/01/2026 have been examined.
Office Action is in response to the Applicant's amendments and remarks filed06/01/2026. Claims 1, 8-9, & 15-16 are presently amended. Claims 1-20 are presently pending and are presented for examination.
Response to Arguments
In regards to the previous rejection under 35 U.S.C. § 103: Applicant’s remaining arguments with respect to the independent claim(s) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. A new grounds of rejection is made in view of US 2022/0161830A1 (“Devassy”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 8-10, & 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2025/0307668A1 (“Sankaradas”), in view of US 2020/0081445A1 (“Stetson”), in view of US 2020/0311462A1 (“Papon”), in view of US 2022/0161830A1 (“Devassy”).
As per claim 1 Stetson discloses
A system for processing multi-modal data representing an environment to generate scene graphs of the environment (see at least Sankaradas, para. [0014]: While extracting all actors and their relationships within the real-time scene's knowledge graph may not be computationally feasible, key aspects may be prioritized for rapid response. Scene-object-entity relationships are extracted in a contextual manner considering system constrains, operational limitations, and the evolving event context. The knowledge graph provides an efficient representation of the scene and facilitates real-time context-aware information retrieval), the system comprising:
one or more processors configured to (see at least Sankaradas, para. [0056]: The processor 410 may be embodied as any type of processor capable of performing the functions described herein.):
obtain first sensor data associated with a vehicle operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents. Thus the streaming information may include real-time information about the actions being performed within the scene. In some embodiments the streaming information may include multivariate time series data, for example being generated by a plurality of different sensors, such as Internet of Things sensors in a given facility, where information from the sensors can be used to identify anomalous activity to control the behavior of systems in the facility.);
determine a set of features associated with the environment based on the first sensor data, the set of features comprising one or more objects and one or more agents (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents.);
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing the one or more objects and the one or more agents relative to the environment (see at least Sankaradas, para. [0032]: A knowledge graph may be represented using semantic tuples, such as a subject-predicate-object tuple. The subject and object represent an entity pair, such as a person and a location, or an object and its property. The predicate specifies the relationship label that connects the entities. The knowledge graph may be a dynamic and evolving representation of relationships and entities within the streaming information, which may be updated in real-time based on the information derived from incoming frames. The knowledge graph acts as a dynamic memory of the system, capturing the nuances and contextual details needed for intelligent processing and response generation. & para. [0062]: Each example may be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data may include a variety of different datatypes, and may include multiple distinct values. The network can have one input node for each value making up the example's input data, and a separate weight can be applied to each input value.);
in response to generating the scene graph, generate a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the vehicle as represented by the first plurality of nodes of the scene graph (see at least Sankaradas, para. [0034]: A knowledge builder 212 uses the knowledge base and user-directed contexts to construct and continually refine the knowledge graph. This process adapts dynamically to the streaming information, ensuring that the knowledge graph remains current and reflective of real-time events. By interfacing with the knowledge base, the knowledge builder 212 enriches the knowledge graph with insights from both historical and live data sources. Knowledge graph generation from the scene-level spatial understanding is achieved by modeling the probability Pr(G|F)=Pr(B,L,R), where F is an input frame, G is the knowledge graph, b.sub.i ∈ B is a bounding box in the frame, L is a set of object labels, and R isa set of relations among the objects L. The probability distribution Pr(G|F) models the likelihood of generating a knowledge graph G given an input frame F. This probability distribution may be modeled based on the joint probability of bounding boxes B, class labels L, and relationships R between the labels in the input frame. & para. [0062]: Each example may be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data may include a variety of different datatypes, and may include multiple distinct values. The network can have one input node for each value making up the example's input data, and a separate weight can be applied to each input value.);
generate a control signal configured to adjust an operation of the vehicle based on one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph (see at least Sankaradas, para. [0050-0054]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident. In a security context, the cameras 102 may monitor a sensitive location and queries may be directed to the detection of unauthorized personnel. In such a context, the responsive action may include summoning security personnel and performing an automatic action with security devices, such as locking or unlocking doors and setting off visual and auditory alarms.); and
provide the control signal to the vehicle to cause the operation of the vehicle (see at least Sankaradas, para. [0050-0054]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
However Sankaradas does not explicitly disclose
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment;
generate a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment;
determine whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph;
in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph,
revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph;
generate a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph;
provide the control signal to the vehicle to cause the adjusted operation of the vehicle.
Stetson teaches
obtain first sensor data associated with a vehicle operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor (see at least Stetson, para. [0118]: Scenario data can be any data describing different scenarios that an AV can encounter, including, but not limited to, video data, unstructured and/or structured text data, sensor data, and/or any other data that can describe scenarios as appropriate to the requirements of specific applications of embodiments of the invention. Regardless of the types of data obtained, the knowledge graph can be generated using any of a variety of processes including, but not limited to, those described herein.);
determine a set of features associated with the environment based on the first sensor data, the set of features comprising one or more objects and one or more agents (see at least Stetson, para. [0131]: The tracked objects are stored (1230) in a knowledge graph structure. Spatiotemporal features of the tracked objects are calculated (1240). Spatiotemporal features can be any feature that describes the object, such as, but not limited to, trajectories, velocities, accelerations, center of mass, transparency, and/or any other descriptive feature as appropriate to the requirements of specific applications of embodiments of the invention.);
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment (see at least Stetson, para. [0131-0132]: Process 1200 includes obtaining (1210) video data. In many embodiments, the video data is relevant to the task that the AI will be trained to perform. For example, in the context of AVs, video data could include dash cam footage, video footage from embedded sensors, simulated video, and/or any other type of video as appropriate to the requirements of specific applications of embodiments of the invention. Video data can include multiple different “scenes,” each of which can depict a scenario… The video scenes can been coded into the scenario component subgraphs, as a subgraph of the knowledge graph. In the illustrated embodiment, a cluster of nodes (grey cluster in center graph) directly reflects objects in the video associated with a given scene. When projected, these nodes can be seen to visually reflect a frame of video (left graph). In the illustrated embodiment, the positions over each frame of video are encoded and can be plotted in a separate projection (right) showing the path of different objects overtime.);
in response to generating the scene graph, generate a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the vehicle as represented by the first plurality of nodes of the scene graph (see at least Stetson, para. [0131-0132]: The tracked objects are stored (1230) in a knowledge graph structure. Spatiotemporal features of the tracked objects are calculated (1240). Spatiotemporal features can be any feature that describes the object, such as, but not limited to, trajectories, velocities, accelerations, center of mass, transparency, and/or any other descriptive feature as appropriate to the requirements of specific applications of embodiments of the invention. The spatiotemporal features are then stored (1250) in the knowledge graph. The spatiotemporal features are clustered to generate spatiotemporal scenarios(1260) across the multiple video scenes. In numerous embodiments, additional semantic meaning can be added (1270). For example, unidentified or poorly identified objects can be provided additional semantic meaning to make the graph structure more understandable…The video scenes can been coded into the scenario component subgraphs, as a subgraph of the knowledge graph. In the illustrated embodiment, a cluster of nodes (grey cluster in center graph) directly reflects objects in the video associated with a given scene. When projected, these nodes can be seen to visually reflect a frame of video (left graph). In the illustrated embodiment, the positions over each frame of video are encoded and can be plotted in a separate projection (right) showing the path of different objects overtime.);
generating a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph (see at least Stetson, para. [0115-0118]: The scenarios can be incorporated into a graph, with each parameter being represented by nodes. These nodes can then be linked in any of a number of ways (and indeed in all possible ways) to generate hybrid scenarios that are various mixtures of all available scenarios. In the illustrated embodiments, all recorded scenarios are colored black, and it becomes clear that there are large parts of the parameter space in between the recorded data from the two recorded intersections which are not filled in, but can be simulated(hollow dots). One of those in-between scenarios involves both cars and pedestrians running lights, and would be sufficient to train the AV that there were few or no safe ways to complete the unprotected left. Trained on such a system, the AV would learn to take a third, safer action which was different than any combination of actions from the first two—in this case, to continue straight through the intersection…Scenario data can be any data describing different scenarios that an AV can encounter, including, but not limited to, video data, unstructured and/or structured text data, sensor data, and/or any other data that can describe scenarios as appropriate to the requirements of specific applications of embodiments of the invention. Regardless of the types of data obtained, the knowledge graph can be generated using any of a variety of processes including, but not limited to, those described herein. In numerous embodiments, nodes in the knowledge graph represents at least one scenario. In a variety of embodiments, nodes in the knowledge graph reflect parameters of scenarios which can be connected to a scenario node, where the edge weights represent values for the parameters for the scenario. However, different subsets of nodes within a knowledge graph can represent any number of different things as discussed herein.);
providing the control signal to the vehicle to cause the adjusted operation of the vehicle (see at least Stetson, para. [0115]: One of those in-between scenarios involves both cars and pedestrians running lights, and would be sufficient to train the AV that there were few or no safe ways to complete the unprotected left. Trained on such a system, the AV would learn to take a third, safer action which was different than any combination of actions from the first two—in this case, to continue straight through the intersection.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment, generate a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph, provide the control signal to the vehicle to cause the adjusted operation of the vehicle of Stetson, with a reasonable expectation of success, in order for generating more effective training and testing paradigms for AIs by encoding data in a structure easily navigated by both human and AI (see at least Stetson, para. [0002]).
Papon teaches
generate a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment (see at least Papon, para. [0026-0027]: The second scene graph 200b may include the same set or a different set attributes than are included in the first scene graph 200a, and those attributes that are included are determined based on the perspective of the second sensor 120b. For example, the second scene graph 200bdoes not include a focus attribute 260 like the first scene graph 200a does. Additionally, because of the different perspectives of the first sensor 120a and the second sensor 120b, a different set of objects 110 may be identified in the second scene graph 200b from the first scene graph 200a, and the same object 110 may be given different identifiers by the different sensors 120…For example, the second sensor 120b may include more accurate or faster algorithms/models for determining a pose attribute 230, and return results faster than a first sensor 120a, e.g., returning a result at time t.sub.2 based on an image captured at time t.sub.2 rather than based on an image captured at time t.sub.1. & para. [0031]: In one example, each of the sensors 120 provide local scene graphs 200 to the collector 130 that include up-to-date position attributes 220. The collector 130 merges the position attributes 220 from the local scene graphs 200 (and locations of the sensors 120 in the environment 100 in some embodiments) to triangulate or otherwise determine the position of an object 110 in the environment100 for the most recent time.);
revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph (see at least Papon, para. [0029-0030]: FIG. 2C illustrates a third scene graph 200c at time t.sub.2 from the perspective of the collector 130 shown in FIGS. 1A-1C, according to various embodiments of the present disclosure. The third scene graph 200c is a global scene graph 200 that merges the data from the local scene graphs 200at a given time and provides a coherent dataset to downstream applications that hides any delays in algorithms /models that process at different rates. The global scene graph 200 incorporates the data received from the local scene graphs 200, therefore the third scene graph 200c includes an identity attribute 210, a position attribute 220, a pose attribute 230, an association attribute 240, a mood attribute 250, and a focus attribute 260…In various embodiments, the collector 130 may reformat the data from the local scene graphs 200 when creating the global scene graph 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of generate a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment, revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
Devassy teaches
determine whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph (see at least Devassy, para. [0086-0087]: Each identified point in time that involves a change to the relevant agents or non-agents may be designated as a boundary point between two dynamic scenes for the vehicle. Accordingly, the interaction prediction model may divide a given period of operation of a vehicle into a series of dynamic scenes, each of which is defined by a pair of consecutive boundary points the designate changes to the agents and non-agents that were relevant to the vehicle's decision making. Put another way, each dynamic scene represents a time interval during the period of operation of the vehicle when there were no changes to the agents or non-agents that affected the vehicle's decision making. Thus, each dynamic scene may represent a discrete decision unit for the vehicle. & para. [0129-0130]: Further, the function of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may include various aspects. For instance, one aspect of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may involve determining a current state of vehicle 500 itself, such as a current position, a current orientation, a current velocity, and/or a current acceleration, among other possibilities… Another aspect of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may involve detecting objects within the vehicle's surrounding environment, which may result in the determination of class labels, bounding boxes, or the like for each detected object. In this respect, the particular classes of objects that are detected by perception subsystem 502a (which may be referred to as “agents”) may take various forms, including both (i) “dynamic” objects that have the potential to move, such as vehicles, cyclists, pedestrians, and animals, among other examples, and (ii) “static” objects that generally do not have the potential to move, such as streets, curbs, lane markings, traffic lights, stop signs, and buildings, among other examples. );
in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph, revising the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph (see at least Devassy, Figs. 4A-4D & para. [0091-0095]: One possible example of utilizing the scene prediction model 312 in this way is illustrated in FIGS. 4A-4D. Beginning with FIG. 4A, the vehicle 201 shown in FIG. 2D is illustrated at a boundary point between dynamic scene S2, which has just concluded, and a new dynamic scene S3. Similar to FIG. 2D, the vehicle 201 has a past trajectory 202 and a planned future trajectory 203 that sees it continuing to travel straight in traffic lane 208, including stopping temporarily at stop sign 220. Two additional agents are shown, including the pedestrian 216 with a past trajectory 217 and a predicted future trajectory that sees the pedestrian 216 continue straight into crosswalk 219. Further, the vehicle 213 includes the past trajectory 214 and a predicted future trajectory 215 that sees the vehicle 213 continuing to travel straight in traffic lane 221, as discussed with respect to FIG. 2D.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of determine whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph, in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph, of Devassy, with a reasonable expectation of success, in order for searching for scenes within captured sensor data that include an interaction of interest may become more efficient (see at least Devassy, para. [0061]).
As per claim 2 Sankaradas discloses
wherein the one or more processors are further configured to: obtain sensor data associated with the vehicle operating in the environment, the sensor data comprising a portion associated with the first sensor and a portion associated with second sensor, the sensor data generated after the first sensor data is generated (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents. Thus the streaming information may include real-time information about the actions being performed within the scene. In some embodiments the streaming information may include multivariate time series data, for example being generated by a plurality of different sensors, such as Internet of Things sensors in a given facility, where information from the sensors can be used to identify anomalous activity to control the behavior of systems in the facility. In some embodiments the streaming information may include video of a road scene, where information relating to traffic and accidents can be used to control traffic. & para. [0021]: The knowledge extraction 104 receives real-time streaming information from, e.g., camera 102 or other sensors. & para. [0032]: The knowledge graph may be a dynamic and evolving representation of relationships and entities within the streaming information, which may be updated in real-time based on the information derived from incoming frames.); and
update at least one relationship represented by the knowledge graph based on second sensor data (see at least Sankaradas, para. [0051]: Referring now to FIG. 3, a method for processing streaming information is shown. Block 302receives new streaming information, such as new frames in a video stream. Block 304 then uses the new streaming information to update the knowledge graph as described above. This may include processing the new frame using one or more VLMs in accordance with available processing resources to extract information relating to, e.g., actions and objects depicted within the new frame.).
Sankaradas does not explicitly disclose
obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the second sensor data is generated; and
update at least one relationship represented by the knowledge graph based on the third sensor data.
Papon teaches
obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the second sensor data is generated (see at least Papon, para. [0032]: For example, if at time t.sub.x, neither the first scene graph 200a nor the second scene graph 200b include a value for an attribute determined based on time t.sub.x, but the sensors 120 provided values at times t.sub.x−1 and t.sub.x−3, the collector 130 may produce a global value in the third scene graph 200c using the values reported in the local scene graphs at timet.sub.x−1. In another example, when the first scene graph 200a but not the second scene graph 200bfor time t.sub.x includes a value for an attribute determined based on time t.sub.x, the collector 130may produce a global value in the third scene graph 200c for time t.sub.x using the values reported in the first scene graph 200a and not the second scene graph 200b. In a further example, when the first scene graph 200a but not the second scene graph 200b for time t.sub.x includes a value for an attribute determined based on time t.sub.x, the collector 130 may produce a global value for that attribute in the third scene graph 200c for time t.sub.x using the values reported in the first scene graph 200a for t.sub.x and the most recent value for that attribute reported in the second scene graph200b.); and
update at least one relationship represented by the knowledge graph based on the third sensor data (see at least Papon, para. [0032]: In cases in which the collector 130 uses data from prior times, the collector 130 may reduce a confidence score for the earlier determined values and/or report a result in the global scene graph 200 with a lower confidence than if the values were determined from the most recent local scene graphs 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the second sensor data is generated; and update at least one relationship represented by the knowledge graph based on the third sensor data of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
As per claim 3 Sankaradas discloses
wherein the control signal comprises a first control signal, and wherein the one or more processors are further configured to: generate a second control signal configured to adjust the operation of the vehicle in response to updating the knowledge graph (see at least Sankaradas, para. [0052]: Based on the updated knowledge graph, block 306 processes queries. Standing queries maybe evaluated in view of the updated knowledge graph, to determine whether a new response is needed. Any dynamic and interactive queries that have been received may similarly be processed. Based on the responses to these queries, block 308 performs a responsive action.).
However Sankaradas does not explicitly disclose
updating the knowledge graph based on the third sensor data.
Papon teaches
updating the knowledge graph based on the third sensor data (see at least Papon, para. [0032]: In cases in which the collector 130 uses data from prior times, the collector 130 may reduce a confidence score for the earlier determined values and/or report a result in the global scene graph 200 with a lower confidence than if the values were determined from the most recent local scene graphs 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of updating the knowledge graph based on the third sensor data of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
As per claim 8 Sankaradas discloses
A method comprising (see at least Sankaradas, para. [0051]):
obtaining first sensor data associated with a vehicle operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents. Thus the streaming information may include real-time information about the actions being performed within the scene. In some embodiments the streaming information may include multivariate time series data, for example being generated by a plurality of different sensors, such as Internet of Things sensors in a given facility, where information from the sensors can be used to identify anomalous activity to control the behavior of systems in the facility.);
determining a set of features associated with the environment based on the first sensor data, the set of features comprising one or more objects and one or more agents (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents.);
in response to determining the set of features, generating a scene graph comprising a first plurality of nodes and representing the one or more objects and the one or more agents relative to the environment (see at least Sankaradas, para. [0032]: A knowledge graph may be represented using semantic tuples, such as a subject-predicate-object tuple. The subject and object represent an entity pair, such as a person and a location, or an object and its property. The predicate specifies the relationship label that connects the entities. The knowledge graph may be a dynamic and evolving representation of relationships and entities within the streaming information, which may be updated in real-time based on the information derived from incoming frames. The knowledge graph acts as a dynamic memory of the system, capturing the nuances and contextual details needed for intelligent processing and response generation. & para. [0062]: Each example may be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data may include a variety of different datatypes, and may include multiple distinct values. The network can have one input node for each value making up the example's input data, and a separate weight can be applied to each input value.);
in response to generating the scene graph, generating a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the vehicle as represented by the first plurality of nodes of the scene graph (see at least Sankaradas, para. [0034]: A knowledge builder 212 uses the knowledge base and user-directed contexts to construct and continually refine the knowledge graph. This process adapts dynamically to the streaming information, ensuring that the knowledge graph remains current and reflective of real-time events. By interfacing with the knowledge base, the knowledge builder 212 enriches the knowledge graph with insights from both historical and live data sources. Knowledge graph generation from the scene-level spatial understanding is achieved by modeling the probability Pr(G|F)=Pr(B,L,R), where F is an input frame, G is the knowledge graph, b.sub.i ∈ B is a bounding box in the frame, L is a set of object labels, and R isa set of relations among the objects L. The probability distribution Pr(G|F) models the likelihood of generating a knowledge graph G given an input frame F. This probability distribution may be modeled based on the joint probability of bounding boxes B, class labels L, and relationships R between the labels in the input frame. & para. [0062]: Each example may be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data may include a variety of different datatypes, and may include multiple distinct values. The network can have one input node for each value making up the example's input data, and a separate weight can be applied to each input value.);
generating a control signal configured to adjust an operation of the vehicle based on one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph (see at least Sankaradas, para. [0050-0054]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident. In a security context, the cameras 102 may monitor a sensitive location and queries may be directed to the detection of unauthorized personnel. In such a context, the responsive action may include summoning security personnel and performing an automatic action with security devices, such as locking or unlocking doors and setting off visual and auditory alarms.); and
providing the control signal to the vehicle to cause the operation of the vehicle (see at least Sankaradas, para. [0050-0054]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
However Sankaradas does not explicitly disclose
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment;
generating a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment;
determining whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph;
in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph,
revising the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph;
generating a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph;
providing the control signal to the vehicle to cause the adjusted operation of the vehicle.
Stetson teaches
obtain first sensor data associated with a vehicle operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor (see at least Stetson, para. [0118]: Scenario data can be any data describing different scenarios that an AV can encounter, including, but not limited to, video data, unstructured and/or structured text data, sensor data, and/or any other data that can describe scenarios as appropriate to the requirements of specific applications of embodiments of the invention. Regardless of the types of data obtained, the knowledge graph can be generated using any of a variety of processes including, but not limited to, those described herein.);
determine a set of features associated with the environment based on the first sensor data, the set of features comprising one or more objects and one or more agents (see at least Stetson, para. [0131]: The tracked objects are stored (1230) in a knowledge graph structure. Spatiotemporal features of the tracked objects are calculated (1240). Spatiotemporal features can be any feature that describes the object, such as, but not limited to, trajectories, velocities, accelerations, center of mass, transparency, and/or any other descriptive feature as appropriate to the requirements of specific applications of embodiments of the invention.);
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment (see at least Stetson, para. [0131-0132]: Process 1200 includes obtaining (1210) video data. In many embodiments, the video data is relevant to the task that the AI will be trained to perform. For example, in the context of AVs, video data could include dash cam footage, video footage from embedded sensors, simulated video, and/or any other type of video as appropriate to the requirements of specific applications of embodiments of the invention. Video data can include multiple different “scenes,” each of which can depict a scenario… The video scenes can been coded into the scenario component subgraphs, as a subgraph of the knowledge graph. In the illustrated embodiment, a cluster of nodes (grey cluster in center graph) directly reflects objects in the video associated with a given scene. When projected, these nodes can be seen to visually reflect a frame of video (left graph). In the illustrated embodiment, the positions over each frame of video are encoded and can be plotted in a separate projection (right) showing the path of different objects overtime.);
in response to generating the scene graph, generate a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the vehicle as represented by the first plurality of nodes of the scene graph (see at least Stetson, para. [0131-0132]: The tracked objects are stored (1230) in a knowledge graph structure. Spatiotemporal features of the tracked objects are calculated (1240). Spatiotemporal features can be any feature that describes the object, such as, but not limited to, trajectories, velocities, accelerations, center of mass, transparency, and/or any other descriptive feature as appropriate to the requirements of specific applications of embodiments of the invention. The spatiotemporal features are then stored (1250) in the knowledge graph. The spatiotemporal features are clustered to generate spatiotemporal scenarios(1260) across the multiple video scenes. In numerous embodiments, additional semantic meaning can be added (1270). For example, unidentified or poorly identified objects can be provided additional semantic meaning to make the graph structure more understandable…The video scenes can been coded into the scenario component subgraphs, as a subgraph of the knowledge graph. In the illustrated embodiment, a cluster of nodes (grey cluster in center graph) directly reflects objects in the video associated with a given scene. When projected, these nodes can be seen to visually reflect a frame of video (left graph). In the illustrated embodiment, the positions over each frame of video are encoded and can be plotted in a separate projection (right) showing the path of different objects overtime.);
generating a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph (see at least Stetson, para. [0115-0118]: The scenarios can be incorporated into a graph, with each parameter being represented by nodes. These nodes can then be linked in any of a number of ways (and indeed in all possible ways) to generate hybrid scenarios that are various mixtures of all available scenarios. In the illustrated embodiments, all recorded scenarios are colored black, and it becomes clear that there are large parts of the parameter space in between the recorded data from the two recorded intersections which are not filled in, but can be simulated(hollow dots). One of those in-between scenarios involves both cars and pedestrians running lights, and would be sufficient to train the AV that there were few or no safe ways to complete the unprotected left. Trained on such a system, the AV would learn to take a third, safer action which was different than any combination of actions from the first two—in this case, to continue straight through the intersection…Scenario data can be any data describing different scenarios that an AV can encounter, including, but not limited to, video data, unstructured and/or structured text data, sensor data, and/or any other data that can describe scenarios as appropriate to the requirements of specific applications of embodiments of the invention. Regardless of the types of data obtained, the knowledge graph can be generated using any of a variety of processes including, but not limited to, those described herein. In numerous embodiments, nodes in the knowledge graph represents at least one scenario. In a variety of embodiments, nodes in the knowledge graph reflect parameters of scenarios which can be connected to a scenario node, where the edge weights represent values for the parameters for the scenario. However, different subsets of nodes within a knowledge graph can represent any number of different things as discussed herein.);
providing the control signal to the vehicle to cause the adjusted operation of the vehicle (see at least Stetson, para. [0115]: One of those in-between scenarios involves both cars and pedestrians running lights, and would be sufficient to train the AV that there were few or no safe ways to complete the unprotected left. Trained on such a system, the AV would learn to take a third, safer action which was different than any combination of actions from the first two—in this case, to continue straight through the intersection.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment, generate a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph, provide the control signal to the vehicle to cause the adjusted operation of the vehicle of Stetson, with a reasonable expectation of success, in order for generating more effective training and testing paradigms for AIs by encoding data in a structure easily navigated by both human and AI (see at least Stetson, para. [0002]).
Papon teaches
generating a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment (see at least Papon, para. [0026-0027]: The second scene graph 200b may include the same set or a different set attributes than are included in the first scene graph 200a, and those attributes that are included are determined based on the perspective of the second sensor 120b. For example, the second scene graph 200bdoes not include a focus attribute 260 like the first scene graph 200a does. Additionally, because of the different perspectives of the first sensor 120a and the second sensor 120b, a different set of objects 110 may be identified in the second scene graph 200b from the first scene graph 200a, and the same object 110 may be given different identifiers by the different sensors 120…For example, the second sensor 120b may include more accurate or faster algorithms/models for determining a pose attribute 230, and return results faster than a first sensor 120a, e.g., returning a result at time t.sub.2 based on an image captured at time t.sub.2 rather than based on an image captured at time t.sub.1. & para. [0031]: In one example, each of the sensors 120 provide local scene graphs 200 to the collector 130that include up-to-date position attributes 220. The collector 130 merges the position attributes 220from the local scene graphs 200 (and locations of the sensors 120 in the environment 100 in some embodiments) to triangulate or otherwise determine the position of an object 110 in the environment100 for the most recent time.);
revising the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph (see at least Papon, para. [0029-0030]: FIG. 2C illustrates a third scene graph 200c at time t.sub.2 from the perspective of the collector130 shown in FIGS. 1A-1C, according to various embodiments of the present disclosure. The third scene graph 200c is a global scene graph 200 that merges the data from the local scene graphs 200at a given time and provides a coherent dataset to downstream applications that hides any delays in algorithms /models that process at different rates. The global scene graph 200 incorporates the data received from the local scene graphs 200, therefore the third scene graph 200c includes an identity attribute 210, a position attribute 220, a pose attribute 230, an association attribute 240, a mood attribute 250, and a focus attribute 260…In various embodiments, the collector 130 may reformat the data from the local scene graphs 200 when creating the global scene graph 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of generate a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment, revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
Devassy teaches
determining whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph (see at least Devassy, para. [0086-0087]: Each identified point in time that involves a change to the relevant agents or non-agents may be designated as a boundary point between two dynamic scenes for the vehicle. Accordingly, the interaction prediction model may divide a given period of operation of a vehicle into a series of dynamic scenes, each of which is defined by a pair of consecutive boundary points the designate changes to the agents and non-agents that were relevant to the vehicle's decision making. Put another way, each dynamic scene represents a time interval during the period of operation of the vehicle when there were no changes to the agents or non-agents that affected the vehicle's decision making. Thus, each dynamic scene may represent a discrete decision unit for the vehicle. & para. [0129-0130]: Further, the function of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may include various aspects. For instance, one aspect of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may involve determining a current state of vehicle 500 itself, such as a current position, a current orientation, a current velocity, and/or a current acceleration, among other possibilities… Another aspect of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may involve detecting objects within the vehicle's surrounding environment, which may result in the determination of class labels, bounding boxes, or the like for each detected object. In this respect, the particular classes of objects that are detected by perception subsystem 502a (which may be referred to as “agents”) may take various forms, including both (i) “dynamic” objects that have the potential to move, such as vehicles, cyclists, pedestrians, and animals, among other examples, and (ii) “static” objects that generally do not have the potential to move, such as streets, curbs, lane markings, traffic lights, stop signs, and buildings, among other examples. );
in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph, revising the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph (see at least Devassy, Figs. 4A-4D & para. [0091-0095]: One possible example of utilizing the scene prediction model 312 in this way is illustrated in FIGS. 4A-4D. Beginning with FIG. 4A, the vehicle 201 shown in FIG. 2D is illustrated at a boundary point between dynamic scene S2, which has just concluded, and a new dynamic scene S3. Similar to FIG. 2D, the vehicle 201 has a past trajectory 202 and a planned future trajectory 203 that sees it continuing to travel straight in traffic lane 208, including stopping temporarily at stop sign 220. Two additional agents are shown, including the pedestrian 216 with a past trajectory 217 and a predicted future trajectory that sees the pedestrian 216 continue straight into crosswalk 219. Further, the vehicle 213 includes the past trajectory 214 and a predicted future trajectory 215 that sees the vehicle 213 continuing to travel straight in traffic lane 221, as discussed with respect to FIG. 2D.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of determine whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph, in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph, of Devassy, with a reasonable expectation of success, in order for searching for scenes within captured sensor data that include an interaction of interest may become more efficient (see at least Devassy, para. [0061]).
As per claim 9 Sankaradas discloses
obtaining sensor data associated with the vehicle operating in the environment, the sensor data comprising a portion associated with the first sensor and a portion associated with second sensor, the sensor data generated after the first sensor data is generated (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents. Thus the streaming information may include real-time information about the actions being performed within the scene. In some embodiments the streaming information may include multivariate time series data, for example being generated by a plurality of different sensors, such as Internet of Things sensors in a given facility, where information from the sensors can be used to identify anomalous activity to control the behavior of systems in the facility. In some embodiments the streaming information may include video of a road scene, where information relating to traffic and accidents can be used to control traffic. & para. [0021]: The knowledge extraction 104 receives real-time streaming information from, e.g., camera 102 or other sensors. & para. [0032]: The knowledge graph may be a dynamic and evolving representation of relationships and entities within the streaming information, which may be updated in real-time based on the information derived from incoming frames.); and
updating at least one relationship represented by the knowledge graph based on second sensor data (see at least Sankaradas, para. [0051]: Referring now to FIG. 3, a method for processing streaming information is shown. Block 302receives new streaming information, such as new frames in a video stream. Block 304 then uses the new streaming information to update the knowledge graph as described above. This may include processing the new frame using one or more VLMs in accordance with available processing resources to extract information relating to, e.g., actions and objects depicted within the new frame.).
Sankaradas does not explicitly disclose
obtaining third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the second sensor data is generated; and
update at least one relationship represented by the knowledge graph based on the third sensor data.
Papon teaches
obtaining third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the second sensor data is generated (see at least Papon, para. [0032]: For example, if at time t.sub.x, neither the first scene graph 200a nor the second scene graph 200b include a value for an attribute determined based on time t.sub.x, but the sensors 120 provided values at times t.sub.x−1 and t.sub.x−3, the collector 130 may produce a global value in the third scene graph 200c using the values reported in the local scene graphs at timet.sub.x−1. In another example, when the first scene graph 200a but not the second scene graph 200bfor time t.sub.x includes a value for an attribute determined based on time t.sub.x, the collector 130may produce a global value in the third scene graph 200c for time t.sub.x using the values reported in the first scene graph 200a and not the second scene graph 200b. In a further example, when the first scene graph 200a but not the second scene graph 200b for time t.sub.x includes a value for an attribute determined based on time t.sub.x, the collector 130 may produce a global value for that attribute in the third scene graph 200c for time t.sub.x using the values reported in the first scene graph 200a for t.sub.x and the most recent value for that attribute reported in the second scene graph200b.); and
update at least one relationship represented by the knowledge graph based on the third sensor data (see at least Papon, para. [0032]: In cases in which the collector 130 uses data from prior times, the collector 130 may reduce a confidence score for the earlier determined values and/or report a result in the global scene graph 200 with a lower confidence than if the values were determined from the most recent local scene graphs 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the second sensor data is generated; and update at least one relationship represented by the knowledge graph based on the third sensor data of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
As per claim 10 Sankaradas discloses
wherein the control signal comprises a first control signal, the method further comprising: generating a second control signal configured to adjust the operation of the vehicle in response to updating the knowledge graph (see at least Sankaradas, para. [0052]: Based on the updated knowledge graph, block 306 processes queries. Standing queries maybe evaluated in view of the updated knowledge graph, to determine whether a new response is needed. Any dynamic and interactive queries that have been received may similarly be processed. Based on the responses to these queries, block 308 performs a responsive action.).
However Sankaradas does not explicitly disclose
updating the knowledge graph based on the third sensor data.
Papon teaches
updating the knowledge graph based on the third sensor data (see at least Papon, para. [0032]: In cases in which the collector 130 uses data from prior times, the collector 130 may reduce a confidence score for the earlier determined values and/or report a result in the global scene graph 200 with a lower confidence than if the values were determined from the most recent local scene graphs 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of updating the knowledge graph based on the third sensor data of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
As per claim 15 Sankaradas discloses
One or more non-transitory computer-readable mediums storing instructions thereon that, when executed by one or more processors (see at least Sankaradas, para. [0057]: The memory 430 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory 430 may store various data and software used during operation of the computing device 400, such as operating systems, applications, programs, libraries, and drivers. The memory 430 is communicatively coupled to the processor 410 via the I/O subsystem 420, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor 410, the memory 430, and other components of the computing device 400.), cause the one or more processors to:
obtain first sensor data associated with a device operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents. Thus the streaming information may include real-time information about the actions being performed within the scene. In some embodiments the streaming information may include multivariate time series data, for example being generated by a plurality of different sensors, such as Internet of Things sensors in a given facility, where information from the sensors can be used to identify anomalous activity to control the behavior of systems in the facility.);
determine a set of features associated with the environment based on the sensor data, the set of features comprising one or more objects and one or more agents (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents.);
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing the one or more objects and the one or more agents relative to the environment (see at least Sankaradas, para. [0032]: A knowledge graph may be represented using semantic tuples, such as a subject-predicate-object tuple. The subject and object represent an entity pair, such as a person and a location, or an object and its property. The predicate specifies the relationship label that connects the entities. The knowledge graph may be a dynamic and evolving representation of relationships and entities within the streaming information, which may be updated in real-time based on the information derived from incoming frames. The knowledge graph acts as a dynamic memory of the system, capturing the nuances and contextual details needed for intelligent processing and response generation. & para. [0062]: Each example may be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data may include a variety of different datatypes, and may include multiple distinct values. The network can have one input node for each value making up the example's input data, and a separate weight can be applied to each input value.);
in response to generating the scene graph, generate a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the device as represented by the first plurality of nodes of the scene graph (see at least Sankaradas, para. [0034]: A knowledge builder 212 uses the knowledge base and user-directed contexts to construct and continually refine the knowledge graph. This process adapts dynamically to the streaming information, ensuring that the knowledge graph remains current and reflective of real-time events. By interfacing with the knowledge base, the knowledge builder 212 enriches the knowledge graph with insights from both historical and live data sources. Knowledge graph generation from the scene-level spatial understanding is achieved by modeling the probability Pr(G|F)=Pr(B,L,R), where F is an input frame, G is the knowledge graph, b.sub.i ∈ B is a bounding box in the frame, L is a set of object labels, and R isa set of relations among the objects L. The probability distribution Pr(G|F) models the likelihood of generating a knowledge graph G given an input frame F. This probability distribution may be modeled based on the joint probability of bounding boxes B, class labels L, and relationships R between the labels in the input frame. & para. [0062]: Each example may be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data may include a variety of different datatypes, and may include multiple distinct values. The network can have one input node for each value making up the example's input data, and a separate weight can be applied to each input value.);
generate a control signal configured to adjust an operation of the device based on one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph (see at least Sankaradas, para. [0050-54]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident. In a security context, the cameras 102 may monitor a sensitive location and queries may be directed to the detection of unauthorized personnel. In such a context, the responsive action may include summoning security personnel and performing an automatic action with security devices, such as locking or unlocking doors and setting off visual and auditory alarms.); and
provide the control signal to the device to cause the operation of the device (see at least Sankaradas, para. [0050-0054]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
However Sankaradas does not explicitly disclose
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment;
generate a second scene graph based on second sensor data associated with the device operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment;
determining whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph;
in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph, revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph;
generate a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph;
provide the control signal to the vehicle to cause the adjusted operation of the vehicle.
Stetson teaches
obtain first sensor data associated with a vehicle operating in an environment, the first sensor data comprising a first portion associated with a first sensor and a second portion associated with a second sensor (see at least Stetson, para. [0118]: Scenario data can be any data describing different scenarios that an AV can encounter, including, but not limited to, video data, unstructured and/or structured text data, sensor data, and/or any other data that can describe scenarios as appropriate to the requirements of specific applications of embodiments of the invention. Regardless of the types of data obtained, the knowledge graph can be generated using any of a variety of processes including, but not limited to, those described herein.);
determine a set of features associated with the environment based on the first sensor data, the set of features comprising one or more objects and one or more agents (see at least Stetson, para. [0131]: The tracked objects are stored (1230) in a knowledge graph structure. Spatiotemporal features of the tracked objects are calculated (1240). Spatiotemporal features can be any feature that describes the object, such as, but not limited to, trajectories, velocities, accelerations, center of mass, transparency, and/or any other descriptive feature as appropriate to the requirements of specific applications of embodiments of the invention.);
in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment (see at least Stetson, para. [0131-0132]: Process 1200 includes obtaining (1210) video data. In many embodiments, the video data is relevant to the task that the AI will be trained to perform. For example, in the context of AVs, video data could include dash cam footage, video footage from embedded sensors, simulated video, and/or any other type of video as appropriate to the requirements of specific applications of embodiments of the invention. Video data can include multiple different “scenes,” each of which can depict a scenario… The video scenes can been coded into the scenario component subgraphs, as a subgraph of the knowledge graph. In the illustrated embodiment, a cluster of nodes (grey cluster in center graph) directly reflects objects in the video associated with a given scene. When projected, these nodes can be seen to visually reflect a frame of video (left graph). In the illustrated embodiment, the positions over each frame of video are encoded and can be plotted in a separate projection (right) showing the path of different objects overtime.);
in response to generating the scene graph, generate a knowledge graph based on the scene graph and stored contextual information, the knowledge graph representing relationships involving the one or more objects and the one or more agents in the environment for use in strategic planning of the vehicle as represented by the first plurality of nodes of the scene graph (see at least Stetson, para. [0131-0132]: The tracked objects are stored (1230) in a knowledge graph structure. Spatiotemporal features of the tracked objects are calculated (1240). Spatiotemporal features can be any feature that describes the object, such as, but not limited to, trajectories, velocities, accelerations, center of mass, transparency, and/or any other descriptive feature as appropriate to the requirements of specific applications of embodiments of the invention. The spatiotemporal features are then stored (1250) in the knowledge graph. The spatiotemporal features are clustered to generate spatiotemporal scenarios(1260) across the multiple video scenes. In numerous embodiments, additional semantic meaning can be added (1270). For example, unidentified or poorly identified objects can be provided additional semantic meaning to make the graph structure more understandable…The video scenes can been coded into the scenario component subgraphs, as a subgraph of the knowledge graph. In the illustrated embodiment, a cluster of nodes (grey cluster in center graph) directly reflects objects in the video associated with a given scene. When projected, these nodes can be seen to visually reflect a frame of video (left graph). In the illustrated embodiment, the positions over each frame of video are encoded and can be plotted in a separate projection (right) showing the path of different objects overtime.);
generate a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph (see at least Stetson, para. [0115-0118]: The scenarios can be incorporated into a graph, with each parameter being represented by nodes. These nodes can then be linked in any of a number of ways (and indeed in all possible ways) to generate hybrid scenarios that are various mixtures of all available scenarios. In the illustrated embodiments, all recorded scenarios are colored black, and it becomes clear that there are large parts of the parameter space in between the recorded data from the two recorded intersections which are not filled in, but can be simulated(hollow dots). One of those in-between scenarios involves both cars and pedestrians running lights, and would be sufficient to train the AV that there were few or no safe ways to complete the unprotected left. Trained on such a system, the AV would learn to take a third, safer action which was different than any combination of actions from the first two—in this case, to continue straight through the intersection…Scenario data can be any data describing different scenarios that an AV can encounter, including, but not limited to, video data, unstructured and/or structured text data, sensor data, and/or any other data that can describe scenarios as appropriate to the requirements of specific applications of embodiments of the invention. Regardless of the types of data obtained, the knowledge graph can be generated using any of a variety of processes including, but not limited to, those described herein. In numerous embodiments, nodes in the knowledge graph represents at least one scenario. In a variety of embodiments, nodes in the knowledge graph reflect parameters of scenarios which can be connected to a scenario node, where the edge weights represent values for the parameters for the scenario. However, different subsets of nodes within a knowledge graph can represent any number of different things as discussed herein.);
provide the control signal to the vehicle to cause the adjusted operation of the vehicle (see at least Stetson, para. [0115]: One of those in-between scenarios involves both cars and pedestrians running lights, and would be sufficient to train the AV that there were few or no safe ways to complete the unprotected left. Trained on such a system, the AV would learn to take a third, safer action which was different than any combination of actions from the first two—in this case, to continue straight through the intersection.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of in response to determining the set of features, generate a scene graph comprising a first plurality of nodes and representing poses and velocities of the one or more objects and the one or more agents relative to the environment, generate a control signal configured to adjust an operation of the vehicle based on the updated relationships of the updated knowledge graph using one or more first attributes representing first states of the one or more objects or one or more second attributes representing second states of the one or more agents from the knowledge graph, provide the control signal to the vehicle to cause the adjusted operation of the vehicle of Stetson, with a reasonable expectation of success, in order for generating more effective training and testing paradigms for AIs by encoding data in a structure easily navigated by both human and AI (see at least Stetson, para. [0002]).
Papon teaches
generate a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment (see at least Papon, para. [0026-0027]: The second scene graph 200b may include the same set or a different set attributes than are included in the first scene graph 200a, and those attributes that are included are determined based on the perspective of the second sensor 120b. For example, the second scene graph 200bdoes not include a focus attribute 260 like the first scene graph 200a does. Additionally, because of the different perspectives of the first sensor 120a and the second sensor 120b, a different set of objects 110 may be identified in the second scene graph 200b from the first scene graph 200a, and the same object 110 may be given different identifiers by the different sensors 120…For example, the second sensor 120b may include more accurate or faster algorithms/models for determining a pose attribute 230, and return results faster than a first sensor 120a, e.g., returning a result at time t.sub.2 based on an image captured at time t.sub.2 rather than based on an image captured at time t.sub.1. & para. [0031]: In one example, each of the sensors 120 provide local scene graphs 200 to the collector 130that include up-to-date position attributes 220. The collector 130 merges the position attributes 220from the local scene graphs 200 (and locations of the sensors 120 in the environment 100 in some embodiments) to triangulate or otherwise determine the position of an object 110 in the environment100 for the most recent time.);
revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph (see at least Papon, para. [0029-0030]: FIG. 2C illustrates a third scene graph 200c at time t.sub.2 from the perspective of the collector130 shown in FIGS. 1A-1C, according to various embodiments of the present disclosure. The third scene graph 200c is a global scene graph 200 that merges the data from the local scene graphs 200at a given time and provides a coherent dataset to downstream applications that hides any delays in algorithms /models that process at different rates. The global scene graph 200 incorporates the data received from the local scene graphs 200, therefore the third scene graph 200c includes an identity attribute 210, a position attribute 220, a pose attribute 230, an association attribute 240, a mood attribute 250, and a focus attribute 260…In various embodiments, the collector 130 may reformat the data from the local scene graphs 200 when creating the global scene graph 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of generate a second scene graph based on second sensor data associated with the vehicle operating in the environment, the second scene graph comprising a second plurality of nodes and representing updated poses and velocities of the one or more objects and the one or more agents relative to the environment, revise the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
Devassy teaches
determining whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph (see at least Devassy, para. [0086-0087]: Each identified point in time that involves a change to the relevant agents or non-agents may be designated as a boundary point between two dynamic scenes for the vehicle. Accordingly, the interaction prediction model may divide a given period of operation of a vehicle into a series of dynamic scenes, each of which is defined by a pair of consecutive boundary points the designate changes to the agents and non-agents that were relevant to the vehicle's decision making. Put another way, each dynamic scene represents a time interval during the period of operation of the vehicle when there were no changes to the agents or non-agents that affected the vehicle's decision making. Thus, each dynamic scene may represent a discrete decision unit for the vehicle. & para. [0129-0130]: Further, the function of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may include various aspects. For instance, one aspect of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may involve determining a current state of vehicle 500 itself, such as a current position, a current orientation, a current velocity, and/or a current acceleration, among other possibilities… Another aspect of deriving the representation of the surrounding environment perceived by vehicle 500 using the raw data may involve detecting objects within the vehicle's surrounding environment, which may result in the determination of class labels, bounding boxes, or the like for each detected object. In this respect, the particular classes of objects that are detected by perception subsystem 502a (which may be referred to as “agents”) may take various forms, including both (i) “dynamic” objects that have the potential to move, such as vehicles, cyclists, pedestrians, and animals, among other examples, and (ii) “static” objects that generally do not have the potential to move, such as streets, curbs, lane markings, traffic lights, stop signs, and buildings, among other examples. );
in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph, revising the knowledge graph by updating the relationships involving the one or more objects and the one or more agents in the environment based on the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph (see at least Devassy, Figs. 4A-4D & para. [0091-0095]: One possible example of utilizing the scene prediction model 312 in this way is illustrated in FIGS. 4A-4D. Beginning with FIG. 4A, the vehicle 201 shown in FIG. 2D is illustrated at a boundary point between dynamic scene S2, which has just concluded, and a new dynamic scene S3. Similar to FIG. 2D, the vehicle 201 has a past trajectory 202 and a planned future trajectory 203 that sees it continuing to travel straight in traffic lane 208, including stopping temporarily at stop sign 220. Two additional agents are shown, including the pedestrian 216 with a past trajectory 217 and a predicted future trajectory that sees the pedestrian 216 continue straight into crosswalk 219. Further, the vehicle 213 includes the past trajectory 214 and a predicted future trajectory 215 that sees the vehicle 213 continuing to travel straight in traffic lane 221, as discussed with respect to FIG. 2D.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of determine whether the updated poses and velocities of the one or more objects and the one or more agents as represented by the second plurality of nodes of the second scene graph indicate at least one change in relationship indicated in the knowledge graph, in response to determining the updated poses and velocities indicate at least one change in relationship indicated in the knowledge graph, of Devassy, with a reasonable expectation of success, in order for searching for scenes within captured sensor data that include an interaction of interest may become more efficient (see at least Devassy, para. [0061]).
As per claim 16 Sankaradas discloses
wherein the instructions further configured to cause the one or more processors to: obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the second sensor data is generated (see at least Sankaradas, para. [0015]: Streaming information is provided by, e.g., a video camera 102. In this example video information includes a series of image frames that depict a particular scene, which may include various objects and agents. Thus the streaming information may include real-time information about the actions being performed within the scene. In some embodiments the streaming information may include multivariate time series data, for example being generated by a plurality of different sensors, such as Internet of Things sensors in a given facility, where information from the sensors can be used to identify anomalous activity to control the behavior of systems in the facility. In some embodiments the streaming information may include video of a road scene, where information relating to traffic and accidents can be used to control traffic. & para. [0021]: The knowledge extraction 104 receives real-time streaming information from, e.g., camera 102 or other sensors. & para. [0032]: The knowledge graph may be a dynamic and evolving representation of relationships and entities within the streaming information, which may be updated in real-time based on the information derived from incoming frames.); and
update at least one relationship represented by the knowledge graph based on second sensor data (see at least Sankaradas, para. [0051]: Referring now to FIG. 3, a method for processing streaming information is shown. Block 302receives new streaming information, such as new frames in a video stream. Block 304 then uses the new streaming information to update the knowledge graph as described above. This may include processing the new frame using one or more VLMs in accordance with available processing resources to extract information relating to, e.g., actions and objects depicted within the new frame.).
Sankaradas does not explicitly disclose
obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the third sensor, the third sensor data generated after the second sensor data is generated; and
update at least one relationship represented by the knowledge graph based on the third sensor data.
Papon teaches
obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the third sensor, the third sensor data generated after the second sensor data is generated (see at least Papon, para. [0032]: For example, if at time t.sub.x, neither the first scene graph 200a nor the second scene graph 200b include a value for an attribute determined based on time t.sub.x, but the sensors 120 provided values at times t.sub.x−1 and t.sub.x−3, the collector 130 may produce a global value in the third scene graph 200c using the values reported in the local scene graphs at timet.sub.x−1. In another example, when the first scene graph 200a but not the second scene graph 200bfor time t.sub.x includes a value for an attribute determined based on time t.sub.x, the collector 130may produce a global value in the third scene graph 200c for time t.sub.x using the values reported in the first scene graph 200a and not the second scene graph 200b. In a further example, when the first scene graph 200a but not the second scene graph 200b for time t.sub.x includes a value for an attribute determined based on time t.sub.x, the collector 130 may produce a global value for that attribute in the third scene graph 200c for time t.sub.x using the values reported in the first scene graph 200a for t.sub.x and the most recent value for that attribute reported in the second scene graph200b.); and
update at least one relationship represented by the knowledge graph based on the third sensor data (see at least Papon, para. [0032]: In cases in which the collector 130 uses data from prior times, the collector 130 may reduce a confidence score for the earlier determined values and/or report a result in the global scene graph 200 with a lower confidence than if the values were determined from the most recent local scene graphs 200.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of obtain third second sensor data associated with the vehicle operating in the environment, the third sensor data comprising a third portion associated with the first sensor and a fourth portion associated with the second sensor, the third sensor data generated after the third sensor data is generated; and update at least one relationship represented by the knowledge graph based on the third sensor data of Papon, with a reasonable expectation of success, in order for reducing the computational resources needed to translate and reconcile data received from disparate sensors (see at least Papon, para. [0015]).
As per claim 17 Sankaradas discloses
wherein the control signal comprises a first control signal, and wherein the instructions further cause the one or more processors to: generate a second control signal configured to adjust the operation of the device in response to updating the knowledge graph (see at least Sankaradas, para. [0052]: Based on the updated knowledge graph, block 306 processes queries. Standing queries maybe evaluated in view of the updated knowledge graph, to determine whether a new response is needed. Any dynamic and interactive queries that have been received may similarly be processed. Based on the responses to these queries, block 308 performs a responsive action.).
Claim(s) 4, 6-7, 11, 13-14, 18, & 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sankaradas, in view of Stetson, in view of Papon, in view of Devassy, in view of US 2025/0242491A1 (“Halilaj”).
As per claim 4 Sankaradas does not explicitly disclose
wherein the one or more processors are further configured to: determine that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment; and
in response to determining that the relationship violates the operating parameter, determine to generate the control signal to adjust operation of the vehicle.
Halilaj teaches
wherein the one or more processors are further configured to: determine that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment (see at least Halilaj, para. [0073-0074]: The method may further comprise validating the generated behaviour tree validated again stone or more pre-defined constraints in order to be sure that it is consistent and does violate any rule. For example, it is only used for robot control if it passes validation. Otherwise, it is for example modified or re-generated.); and
in response to determining that the relationship violates the operating parameter, determine to generate the control signal to adjust operation of the vehicle (see at least Halilaj, para. [0073-0074]: The method may further comprise validating the generated behaviour tree validated again stone or more pre-defined constraints in order to be sure that it is consistent and does violate any rule. For example, it is only used for robot control if it passes validation. Otherwise, it is for example modified or re-generated…The approach of FIG. 3 can be used to determine a behaviour tree which may then be used compute a control signal for controlling a technical system, like e.g. a computer-controlled machine, like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant or an access control system.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of wherein the one or more processors are further configured to: determine that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment; and in response to determining that the relationship violates the operating parameter, determine to generate the control signal to adjust operation of the vehicle of Halilaj, with a reasonable expectation of success, in order for a deeper understanding of the domain and generating more accurate and contextually relevant behaviour trees and improved contextual understanding (see at least Halilaj, para. [0013-0015]).
As per claim 6 Sankaradas discloses
wherein the vehicle is operating in accordance with a first path (see at least Sankaradas, para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.), and
wherein the one or more processors configured to determine to generate the control signal are configured to: determine to adjust the operation of the vehicle by transitioning operation of the vehicle from a first path to a second path (see at least Sankaradas, para. [0052]: Based on the updated knowledge graph, block 306 processes queries. Standing queries maybe evaluated in view of the updated knowledge graph, to determine whether a new response is needed. Any dynamic and interactive queries that have been received may similarly be processed. Based on the responses to these queries, block 308 performs a responsive action.), and
generate the control signal to cause the vehicle to operate in accordance with the second path (see at least Sankaradas, para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
As per claim 7 Sankaradas discloses
wherein the one or more processors are further configured to: generate the second path based on the operating parameter associated with the relationship (see at least Sankaradas, para. [0044-0045]: The batch layer analyzes batches on historical data, pre-processing and updating the knowledge graph with historical context to provide a foundation for contextual understanding. This result is then infused into the real-time extraction pipeline 200 and knowledge pipeline 210 if needed….The serving layer 224 provides efficient retrieval of relevant information from the knowledge graph, thereby enhancing the retrieval process for interactive queries. The speed layer 226 ensures that data remains adaptive and responsive to evolving contexts. para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
As per claim 11 Sankaradas does not explicitly disclose
further comprising: determining that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment; and
in response to determining that the relationship violates the operating parameter, determining to generate the control signal to adjust operation of the vehicle.
Halilaj teaches
further comprising: determining that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment (see at least Halilaj, para. [0073-0074]: The method may further comprise validating the generated behaviour tree validated again stone or more pre-defined constraints in order to be sure that it is consistent and does violate any rule. For example, it is only used for robot control if it passes validation. Otherwise, it is for example modified or re-generated.); and
in response to determining that the relationship violates the operating parameter, determining to generate the control signal to adjust operation of the vehicle (see at least Halilaj, para. [0073-0074]: The method may further comprise validating the generated behaviour tree validated again stone or more pre-defined constraints in order to be sure that it is consistent and does violate any rule. For example, it is only used for robot control if it passes validation. Otherwise, it is for example modified or re-generated…The approach of FIG. 3 can be used to determine a behaviour tree which may then be used compute a control signal for controlling a technical system, like e.g. a computer-controlled machine, like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant or an access control system.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of further comprising: determining that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment; and in response to determining that the relationship violates the operating parameter, determining to generate the control signal to adjust operation of the vehicleof Halilaj, with a reasonable expectation of success, in order for a deeper understanding of the domain and generating more accurate and contextually relevant behaviour trees and improved contextual understanding (see at least Halilaj, para. [0013-0015]).
As per claim 13 Sankaradas discloses
wherein the vehicle is operating in accordance with a first path (see at least Sankaradas, para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.), and
wherein determining to generate the control signal comprises: determining to adjust the operation of the vehicle by transitioning operation of the vehicle from a first path to a second path (see at least Sankaradas, para. [0052]: Based on the updated knowledge graph, block 306 processes queries. Standing queries maybe evaluated in view of the updated knowledge graph, to determine whether a new response is needed. Any dynamic and interactive queries that have been received may similarly be processed. Based on the responses to these queries, block 308 performs a responsive action.), and
generating the control signal to cause the vehicle to operate in accordance with the second path (see at least Sankaradas, para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
As per claim 14 Sankaradas discloses
further comprising: generating the second path based on the operating parameter associated with the relationship (see at least Sankaradas, para. [0044-0045]: The batch layer analyzes batches on historical data, pre-processing and updating the knowledge graph with historical context to provide a foundation for contextual understanding. This result is then infused into the real-time extraction pipeline 200 and knowledge pipeline 210 if needed….The serving layer 224 provides efficient retrieval of relevant information from the knowledge graph, thereby enhancing the retrieval process for interactive queries. The speed layer 226 ensures that data remains adaptive and responsive to evolving contexts. para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
As per claim 18 Sankaradas does not explicitly disclose
wherein the instructions further cause the one or more processors to:
determine that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment; and
in response to determining that the relationship violates the operating parameter, determine to generate the control signal to adjust operation of the device.
Halilaj teaches
wherein the instructions further cause the one or more processors to: determine that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment (see at least Halilaj, para. [0073-0074]: The method may further comprise validating the generated behaviour tree validated again stone or more pre-defined constraints in order to be sure that it is consistent and does violate any rule. For example, it is only used for robot control if it passes validation. Otherwise, it is for example modified or re-generated.); and
in response to determining that the relationship violates the operating parameter, determine to generate the control signal to adjust operation of the device (see at least Halilaj, para. [0073-0074]: The method may further comprise validating the generated behaviour tree validated again stone or more pre-defined constraints in order to be sure that it is consistent and does violate any rule. For example, it is only used for robot control if it passes validation. Otherwise, it is for example modified or re-generated…The approach of FIG. 3 can be used to determine a behaviour tree which may then be used compute a control signal for controlling a technical system, like e.g. a computer-controlled machine, like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant or an access control system.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of wherein the instructions further cause the one or more processors to determine that the first states of the one or more objects indicates a relationship that violates an operating parameter of the environment; and in response to determining that the relationship violates the operating parameter, determine to generate the control signal to adjust operation of the device of Halilaj, with a reasonable expectation of success, in order for a deeper understanding of the domain and generating more accurate and contextually relevant behaviour trees and improved contextual understanding (see at least Halilaj, para. [0013-0015]).
As per claim 20 Sankaradas discloses
wherein the device is operating in accordance with a first path (see at least Sankaradas, para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.), and
wherein the instructions that cause the one or more processors to determine to generate the control signal cause the one or more processors to: determine to adjust the operation of the device by transitioning operation of the device from a first path to a second path (see at least Sankaradas, para. [0052]: Based on the updated knowledge graph, block 306 processes queries. Standing queries maybe evaluated in view of the updated knowledge graph, to determine whether a new response is needed. Any dynamic and interactive queries that have been received may similarly be processed. Based on the responses to these queries, block 308 performs a responsive action.), and
generate the control signal to cause the device to operate in accordance with the second path (see at least Sankaradas, para. [0053]: For example, if a query identifies that a traffic accident has occurred, block 308 may summon emergency personnel to provide assistance. Block 308 may furthermore send automatic instructions to other vehicles on the road and to traffic control devices to route traffic away from the site of the incident.).
Claim(s) 5, 12, & 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sankaradas, in view of Stetson, in view of Papon, in view of Devassy, in view of Halilaj, in view of US 2019/0329762A1 (“Kwon”).
As per claim 5 Sankaradas does not explicitly disclose
wherein the one or more processors configured to determine to generate the control signal are configured to:
determine to adjust the operation of the vehicle by reducing a speed of the vehicle from a first speed to a second speed, and
generate the control signal to cause the vehicle to operate at the second speed.
Kwon teaches
wherein the one or more processors configured to determine to generate the control signal are configured to (see at least Kwon, para. [0067]: Referring to FIG. 4, the vehicle collision avoidance control device 200…):
determine to adjust the operation of the vehicle by reducing a speed of the vehicle from a first speed to a second speed (see at least Kwon, para. [0067]: It is then determined (S420) whether a collision with the first target vehicle 120 will occur or not. When there is a possibility that the driver vehicle 110 will undergo a primary collision with the first target vehicle 120 because the first target vehicle 120 has cut in the traveling lane 140 of the driver vehicle 110, has abruptly decreased the vehicle velocity, or has stopped, a primary vehicle behavior control signal is first generated (S430).), and
generate the control signal to cause the vehicle to operate at the second speed (see at least Kwon, para. [0068]: The primary vehicle control signal is a signal for preventing a primary collision with the first target vehicle, and may include a velocity control signal for reducing the velocity of the driver vehicle or braking the driver vehicle.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of wherein the one or more processors configured to determine to generate the control signal are configured to determine to adjust the operation of the vehicle by reducing a speed of the vehicle from a first speed to a second speed, and generate the control signal to cause the vehicle to operate at the second speed of Kwon, with a reasonable expectation of success, in order for smooth traffic flow or vehicle traveling is guaranteed, thereby reducing fuel consumption and the amount of exhaust gas (see at least Kwon, para. [0014]).
As per claim 12 Sankaradas does not explicitly disclose
wherein determining to generate the control signal comprises:
determining to adjust the operation of the vehicle by reducing a speed of the vehicle from a first speed to a second speed, and
generating the control signal to cause the vehicle to operate at the second speed.
Kwon teaches
wherein determining to generate the control signal comprises (see at least Kwon, para. [0067]: Referring to FIG. 4, the vehicle collision avoidance control device 200…):
determining to adjust the operation of the vehicle by reducing a speed of the vehicle from a first speed to a second speed (see at least Kwon, para. [0067]: It is then determined (S420) whether a collision with the first target vehicle 120 will occur or not. When there is a possibility that the driver vehicle 110 will undergo a primary collision with the first target vehicle 120 because the first target vehicle 120 has cut in the traveling lane 140 of the driver vehicle 110, has abruptly decreased the vehicle velocity, or has stopped, a primary vehicle behavior control signal is first generated (S430).), and
generating the control signal to cause the vehicle to operate at the second speed (see at least Kwon, para. [0068]: The primary vehicle control signal is a signal for preventing a primary collision with the first target vehicle, and may include a velocity control signal for reducing the velocity of the driver vehicle or braking the driver vehicle.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of wherein determining to generate the control signal comprises determining to adjust the operation of the vehicle by reducing a speed of the vehicle from a first speed to a second speed, and generating the control signal to cause the vehicle to operate at the second speed of Kwon, with a reasonable expectation of success, in order for smooth traffic flow or vehicle traveling is guaranteed, thereby reducing fuel consumption and the amount of exhaust gas (see at least Kwon, para. [0014]).
As per claim 19 Sankaradas does not explicitly disclose
wherein the instructions that cause the one or more processors to determine to generate the control signal cause the one or more processors to:
determine to adjust the operation of the device by reducing a speed of the device from a first speed to a second speed, and
generate the control signal to cause the device to operate at the second speed.
Kwon teaches
wherein the instructions that cause the one or more processors to determine to generate the control signal cause the one or more processors to (see at least Kwon, para. [0067]: Referring to FIG. 4, the vehicle collision avoidance control device 200…):
determine to adjust the operation of the device by reducing a speed of the device from a first speed to a second speed (see at least Kwon, para. [0067]: It is then determined (S420) whether a collision with the first target vehicle 120 will occur or not. When there is a possibility that the driver vehicle 110 will undergo a primary collision with the first target vehicle 120 because the first target vehicle 120 has cut in the traveling lane 140 of the driver vehicle 110, has abruptly decreased the vehicle velocity, or has stopped, a primary vehicle behavior control signal is first generated (S430).), and
generate the control signal to cause the device to operate at the second speed (see at least Kwon, para. [0068]: The primary vehicle control signal is a signal for preventing a primary collision with the first target vehicle, and may include a velocity control signal for reducing the velocity of the driver vehicle or braking the driver vehicle.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sankaradas to incorporate the teaching of wherein the instructions that cause the one or more processors to determine to generate the control signal cause the one or more processors to determine to adjust the operation of the device by reducing a speed of the device from a first speed to a second speed, and generate the control signal to cause the device to operate at the second speed of Kwon, with a reasonable expectation of success, in order for smooth traffic flow or vehicle traveling is guaranteed, thereby reducing fuel consumption and the amount of exhaust gas (see at least Kwon, para. [0014]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABDO ALGEHAIM whose telephone number is (571)272-3628. The examiner can normally be reached Monday-Friday 8-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fadey Jabr can be reached at 571-272-1516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABDO ALGEHAIM/Primary Examiner, Art Unit 3668
Prosecution Insights