DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
Figure(s) 1, 2A-2C are objected to as depicting a block diagram without “readily identifiable” descriptors of each block, as required by 37 CFR 1.84(n). Rule 84(n) requires “labeled representations” of graphical symbols, such as blocks; and any that are “not universally recognized may be used, subject to approval by the Office, if they are not likely to be confused with existing conventional symbols, and if they are readily identifiable.” In the case of figure(s) 1, 2A-2C , the blocks are not readily identifiable per se and therefore require the insertion of text that identifies the function of that block. That is, each vacant block should be provided with a corresponding label identifying its function or purpose.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, and 9-14 are rejected under 35 U.S.C. 103 as being unpatentable over Vignard et al. (US 2021/0334556 A1; hereinafter “Vignard”) in view of Monka et al. (US 2022/0198781 A1; hereinafter “Monka”).
Regarding claim 1:
Vignard discloses: a method for training a base model for object detection, semantic segmentation, trajectory prediction, and/or motion planning of a vehicle (¶ 0071] FIG. 3 shows a schematic flow chart of the steps of a method for determining a semantic grid. Also see ¶¶ [0074] – [0079] and [0095] – [0100]
PNG
media_image1.png
546
1372
media_image1.png
Greyscale
), the method comprising:
providing (S1) a training data set of image data, each piece of training data having information about at least one driving scene from a point of view of the vehicle (Vignard teaches that a digital camera records a scene in front of vehicle 100 and provides RGB image data representing that scene, wherein the camera may be a monocular camera generating a two-dimensional image of the environment (¶[0063]). Vignard further teaches that vehicle 100 scans the scene in front of the vehicle using the digital camera (¶¶[0068]–[0069]), that the method receives 2-D image input (x) from the RGB camera at step S1 (¶[0072]), and that a deep neural network receives the monocular RGB input (x) and is trained on sufficiently large databases, including fine-tuning on target datasets for segmentation (¶[0075]). Vignard further discloses that the method receives the LIDAR input Iin S6 (¶¶ [0072] – [0073));
Vignard does not expressly teach providing (S2) a knowledge graph comprising domain-specific knowledge of the at least one driving scene
However, in the same field of endeavor, Monka teaches: providing (S2) a knowledge graph comprising domain-specific knowledge of the at least one driving scene (Monka teaches that prior knowledge concerning a domain or context may be provided as additional information (Monka, ¶[0013]), and that a knowledge graph serves as a medium for encoding such prior knowledge, which can be transformed into a dense vector representation using embedding methods (¶¶[0014]–[0018]). Monka further teaches a knowledge graph specifically for the road-sign domain, wherein the knowledge graph contains the classes of the training data set within a domain ontology comprising entities and relationships, including “Road Sign,” “Shape,” “Icon,” and “Road Sign Feature” entities and corresponding relationships (FIG. 11 and ¶[0085]));
optionally partitioning (S3) the image data into a plurality of image sections (Vignard ¶¶ [0030], [0075] – [0079]);
generating (S4) information matrices corresponding to the image sections (Vignard teaches projecting the segmented semantic image into a plurality of predetermined semantic planes (¶[0029]), wherein each semantic plane comprises a plurality of semantic plane cells and each plane cell may contain a semantic label (¶¶[0031]–[0032]). Vignard further describes intermediate representation (p={p_i}) as a layered representation comprising a collection of (D) planes, wherein each cell in (p) contains a semantic-class value and is spatially registered with the environment (¶¶[0091]–[0094]). [The spatial semantic planes (p_i), each comprising an organized array of semantic cells, correspond to the claimed information matrices])
by assigning domain-specific knowledge about the at least one driving scene extracted directly from the image data to the plurality of image sections of the image data (Vignard assigns semantic labels to pixels of the RGB image to generate semantic image segments (¶[0030]), and teaches that each pixel of the semantic image may be assigned to a corresponding semantic-plane cell (¶¶[0031]–[0032]). Vignard further teaches that the semantic label of a pixel in the segmented image is assigned to corresponding points in intermediate representation (p), with the cells of (p) taking values from the same semantic-class alphabet as the segmented image (¶¶[0093]–[0095]). [Thus, Vignard teaches assigning domain-specific semantic information obtained directly from the image data to corresponding spatial image/semantic regions]);
or from the knowledge graph (Monka teaches providing a knowledge-graph embedding that represents entities and relationships of the graph as a dense vector representation (¶¶[0081]–[0083]), and specifically provides a road-sign-domain knowledge graph having domain entities and relationships (¶[0085]). Monka further teaches providing the knowledge-graph embedding and updating parameters of the image-classifier embedding portion by contrasting image inputs against labels encoded in the knowledge-graph embedding space (¶¶[0087]–[0091]). [Thus, Monka additionally establishes that it was known to extract and use domain knowledge represented in a knowledge graph during visual-model training])
training (S5) the base model based on the information matrices (Vignard states that the objective is to train a learned mapping integrating intermediate representation (p) with the occupancy-grid representation (¶[0095]). The intermediate representation (p) is fed into a deep neural network (¶[0096]), and the input data expressly include (D) planes corresponding to intermediate representation (p) calculated from the segmented RGB image (¶[0098]). Vignard further teaches that dimensionality reduction and fusion are trained jointly, end-to-end (¶[0099]) and uses a multiclass softmax classifier and cross-entropy loss during training (¶[0100])); and
providing (S6) the trained base model for scene understanding, object detection, trajectory prediction, and/or motion planning of the vehicle (Vignard teaches that the computer-vision system determines a semantic representation of the vehicle environment, including detecting the presence and location of objects and determining their semantic classes, and that the resulting object-detection information may be used to provide warnings or take appropriate vehicle actions (¶[0061]). Vignard further explains that camera information is used to understand the scene around the vehicle and generate semantic information (¶¶[0068]–[0069]). Vingard further explains that the method fuses 3D point cloud input I (step S6) from a LIDAR with 2D image input x from an RGB camera (step S1) in order to obtain a semantic grid y (step S11) from a bird's eye view centered on the system (¶ [0072]).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Vignard to additionally provide the domain-specific knowledge graph taught by Monka and utilize such knowledge-graph information in connection with training the visual model in order to obtain a classifier that is robust and context- or domain-invariant (Monka, ¶[0008]). Monka specifically teaches accomplishing this through prior domain/context knowledge encoded in a knowledge graph (¶¶[0013] - [0018]). The proposed combination therefore does not require changing the fundamental operating principle of either reference, but rather employs Monka's known knowledge-graph training technique to improve the closely related image-based vehicle-perception system of Vignard.
Regarding claim 9:
Vignarad further teaches: wherein the image data is acquired from at least one optical sensor or generated by data augmentation from existing image and/or video data (Vignard ¶¶[0020], [0045]–[0049], [0063], and [0072]).
Regarding claim 9:
Vignarad further teaches: wherein the image data is acquired from at least one optical sensor or generated by data augmentation from existing image and/or video data (Vignard ¶¶[0020], [0045]–[0049], [0063], and [0072]).
Regarding claim 10:
Vignarad further teaches: wherein a computer program comprises program code configured to execute at least portions of the method when the computer program is executed on a computer (Vignard ¶¶[0062] and [0084]).
Regarding claim 11:
Vignarad further teaches: A non-transitory computer-readable data carrier comprising program code of a computer program configured to execute at least portions of the method according to claim 1 when the computer program is executed on a computer (Vignard ¶¶[0062] and [0084]).
Regarding claim 12:
Vignarad further teaches: A method for object detection, semantic segmentation, trajectory prediction, and/or motion planning of a vehicle utilizing a trained base model according to claim 1 (Vignard ¶¶ [0061], [0075] – [0079], [0095] – [0100]).
Regarding claim 13:
Vignarad further teaches: An evaluation and/or control device of an imaging sensor configured to perform a method according to claim 12 (Vignard ¶¶ [0061] – [0063]).
Regarding claim 14:
Vignarad teaches: A system for training a base model for object detection, semantic segmentation, trajectory prediction, and/or motion planning of a vehicle, the system comprising: an evaluation and/or computational device (Vignard (¶¶ [0045] – [0049], [0062], and ¶¶ [0095] – [0100]).
configured to (the functional steps are similar to those of claim 1, and rejected in the same manner as applied above).
Claim(s) 2–4 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Vignard et al. (US 2021/0334556 A1; hereinafter “Vignard”) in view of Monka et al. (US 2022/0198781 A1; hereinafter “Monka”), and and further in view of Al Faruque et al. (US 2023/0230484 A1; ; hereinafter “Al Faruque”).
Regarding claim 2:
Vignard in view of Monka teaches the method of claim 1. Vignard in view of Monka does not expressly teach wherein the base model is trained based on the information matrices to determine spatial-temporal relationships of entities within the at least one driving scene, a context of the entities within the driving scene, and/or a time progression of the driving scene.
In the same field of endeavor, Al Faruque teaches those features in the autonomous-driving scene-understanding field.
determine spatial-temporal relationships of entities within the at least one driving scene (FIGS. 3 and 6 explains scene graphs containing lanes, roads, traffic signs, vehicles, and pedestrians, with pairwise directional/proximity/lane relations and a time-series of on-board-camera images. ¶ [0025] discloses estimating a position of the one or more objects relative to the ego-object based on the one or more bounding boxes of each BEV representation, identifying a plurality of relations between objects for each image, and generating a scene-graph for each image based on the aforementioned calculations. ¶ [0026] teaches condensing the one or more scene-graphs into a spatial graph embedding, generating a spatio-temporal graph embedding from the spatial graph embedding through use of a long short-term memory (LSTM) network, and calculating a confidence value for whether or not a collision will occur. The method may further comprise processing the spatio-temporal graph embedding through a temporal attention layer of the LSTM network to generate a context vector, processing the context vector through an LSTM decoder to generate a final spatial-temporal graph embedding);
a context of the entities within the driving scene (Al Faruque claim 5 and claim 13);
and/or a time progression of the driving scene (Al Faruque claim 1, element iii.D, and claim 9, element o).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Vignard in view of Moka to apply Al Faruque’s known spatio-temporal scene-modeling technique so the trained model would learn not only what semantic entities are present and where they are located, but also their relationships and evolution across successive driving scenes. Al Faruque teaches that structured scene representations can be converted into spatial embeddings, processed as a temporal sequence, and trained for an autonomous-driving safety task. A person of ordinary skill would have expected success because the references operate on the same class of vehicle-scene entities and structured spatial representations, and the temporal model merely adds the known dimension of sequence evolution to successive scene representations.
Regarding claim 3:
Vignard in view of Monka does not specifically teach: wherein: the domain-specific knowledge of the at least one driving scene contained in the knowledge graph comprises structured information about the at least one driving scene obtained from autonomous driving data sets, and the structured information includes relationships, hierarchies, and/or contextual information, about objects occurring in the at least one driving scene.
However, Al Faruque teaches: wherein: the domain-specific knowledge of the at least one driving scene contained in the knowledge graph comprises structured information about the at least one driving scene obtained from autonomous driving data sets (Al Faruque ¶ [0119] discloses FIG. 10 describes how an input dataset is converted into a “SceneGraphDataset” object via the scene-graph extraction framework. A list of roadscene2vec′s user-configurable scene-graph extraction settings is shown in FIG. 10. In the formulation, each actor (object) in the scene-graph is assigned a type from the set {car, motorcycle, bicycle, pedestrian, lane, light, sign}, matching those defined by CARLA. Users can reconfigure the set of object types to support other dataset types, applications, or ontologies.);
and the structured information includes relationships, hierarchies, and/or contextual information, about objects occurring in the at least one driving scene (Al Faruque ¶ [0147] teaches that the extracted scene-graph is denoted for the frame I, by G=0, Each scene-graph G, is a directed, heterogeneous multi-graph, where 0, denotes the nodes and A, is the adjacency matrix of the graph G. As shown in FIG. 3, nodes represent the identified objects such as lanes, roads, traffic signs, vehicles, pedestrians, etc., in a traffic scene. The adjacency matrix An indicates the pairwise relations between each object in Or The extraction pipeline first identifies the objects On by using Mask R-CNN. Then, it generates an inverse perspective mapping (also known as a “birds-eye view” projection) of the image to estimate the locations of objects relative to the ego car, which are used to construct the pairwise relations between objects in An).
Regarding claim 4:
Vignard in view of Monka does not specifically teach: wherein the training data set of image data is generated by test drives with the vehicle and/or by historical travel data with the vehicle.
However, Al Faruque teaches: wherein the training data set of image data is generated by test drives with the vehicle and/or by historical travel data with the vehicle (Al Faruque [0191] teaches this dataset, denoted as 571-honda, was a subset of the Honda@ Driving Dataset (HOD) containing 571 lane-change video clips from real-world driving with a distribution of 7.21:1. The HOD was recorded on the same vehicle during mostly safe driving in the California Bay Area. ¶ [0192] teaches the synthetic datasets and the 571-honda dataset were labeled using human annotators. The final label assigned to a dip was the average of the labels assigned by the human annotators rounded to 0 (no collision) and 1 (collision/near collision). Each frame in a video dip was given a label identical to the entire dip's label to train the model to identify the preconditions of a future collision.).
Regarding claim 8:
Vignard further teaches: wherein: a number of rows and columns of the information matrices corresponds to a number of the image sections (Vignard ¶¶[0031]–[0032] teach semantic planes made of spatial plane cells with segmented-image pixels assigned to corresponding cells; ¶¶[0091]–[0094] teach that representation p is spatially registered and that cells at spatial coordinates correspond to the same environment coordinates),
each cell of the information matrices has domain-specific knowledge including semantic concepts of the entities or events present in spatial dimensions of the image sections (Vignard ¶[0031] teaches each semantic-plane cell comprising a semantic label; ¶[0044] teaches semantic-grid cells containing semantic information/class probabilities; ¶¶[0079] and [0093] teach semantic classes such as road, car, pedestrian, building, and signage and assign those class values to spatial cells.);
and the domain-specific knowledge includes information about road infrastructure facilities and/or pedestrians, and/or traffic signs and/or stop areas and/or construction site markings and/or pedestrian crossings and/or potential vehicle trajectories/paths and/or vehicles (Vignard ¶[0021] and ¶[0079] expressly include road, car/vehicle, pedestrian, and signage semantic classes and see FIG. 3 of Al Faruque)
Al Faruque further teaches: annotated with actions and/or context-relevant information including a path traveled since a previous driving scene and/or a traffic participant’s orientation difference between the driving scene and the previous driving scene and/or a country and/or an intended route and/or direction. (FIGS. 3 and 6 explains scene graphs containing lanes, roads, traffic signs, vehicles, and pedestrians, with pairwise directional/proximity/lane relations and a time-series of on-board-camera images. ¶ [0025] discloses estimating a position of the one or more objects relative to the ego-object based on the one or more bounding boxes of each BEV representation, identifying a plurality of relations between objects for each image, and generating a scene-graph for each image based on the aforementioned calculations. ¶ [0067] teaches identifying a proximity relation between the ego-object and each object of the object dataset for each image by measuring a distance between the ego-object and each object, identifying a directional relation between the ego-object and each object of the object dataset for each image by determining a relative orientation of the ego-object and each object, and identifying a belonging relation between each object and a lane selected from a group consisting of a left lane, a middle lane, and a right lane by measuring a horizontal displacement of each object the object dataset relative to the ego-object for each image. ¶ [0070] teaches extracting a rotation, acceleration, and velocity for each car, motorcycle, and bicycle object. ¶ [0123] teaches that each node is assigned its type label from the set of actor names and its corresponding attributes (e.g., position, angle, velocity, current lane, light status, etc.) for relation extraction).
Claim(s) 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Vignard et al. (US 2021/0334556 A1; hereinafter “Vignard”) in view of Monka et al. (US 2022/0198781 A1; hereinafter “Monka”), and further in view of Bao et al., (“BEiT: BERT Pre-Training of Image Transformers,” 09/2022).
Regarding claim 5:
Vignard in view of Monka teaches the method of claim 1. Vignard in view of Monka does not expressly teach wherein the base model comprises a machine learning model including an autoregression based transformer model or a masking based transformer model.
However, in the same field of endeavor, Bao teaches: wherein the base model comprises a machine learning model including an autoregression based transformer model or a masking based transformer model (Bao Abstract and §2.3 teach masked image modeling for pre-training vision Transformers by randomly masking image patches and predicting the corresponding visual tokens. Bao §2.2 expressly uses a standard Transformer as the backbone network, and §2.5 identifies a 12-layer Transformer implementation. Bao §2.6 further fine-tunes the pretrained Transformer for downstream semantic segmentation).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Vignard in view of Moka to apply masking-based vision-Transformer architecture as the base-model architecture in the Vignard/Monka training framework as taught by Bao in order to obtain contextual visual representations suitable for the same downstream segmentation/scene-understanding task. A reasonable expectation of success is supported by Bao’s demonstrated fine-tuning for semantic segmentation.
Regarding claim 6:
Vignard further teaches: wherein when the base model comprises a masking-based transformer model, one or more information entries of the information matrices are masked and/or hidden randomly or in a predetermined manner (Vignard ¶¶[0031]–[0032] and [0091]–[0094] supply the spatial semantic-plane cell entries of the claimed information matrices. Bao §2.1.1 splits a 2-D image into a grid of spatial patches; §2.3 randomly masks a percentage of those spatial positions, including blockwise masking according to Algorithm 1. Bao therefore teaches the known training operation of hiding spatially indexed input units at random or according to a defined blockwise scheme.)
to train the base model to predict and/or determine the masked and/or hidden information entries (Bao §2.3 teaches that, for each masked position, the Transformer predicts the corresponding visual token and trains by maximizing the likelihood of the correct token at masked positions. The Abstract and Fig. 1 likewise state that the pre-training objective is recovery of original visual tokens from corrupted/masked inputs).
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Vignard et al. (US 2021/0334556 A1; hereinafter “Vignard”) in view of Monka et al. (US 2022/0198781 A1; hereinafter “Monka”), and and further in view of Chen et al., “Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving,” 10/2023).
Regarding claim 7:
Vignard in view of Monka teaches the method of claim 1. Vignard in view of Monka does not expressly teach wherein the base model comprises a pre-trained large language model.
However, in the same field of endeavor, Chen teaches: wherein the base model comprises a pre-trained large language model (Chen Abstract teaches an object-level multimodal LLM architecture that merges vectorized numeric driving modalities with a pre-trained LLM to improve context understanding in driving situations. §1 further states that object-level 2-D scene representations commonly used in autonomous driving are fused into a pre-trained LLM to enable interpretation/reasoning and action prediction. §3.4 (“Training the Driving LLM Agent”) teaches integrating object-level vector modality into pre-trained LLMs and fine-tuning the resulting driving model. §3.4 expressly identifies LLaMA-7B as the pre-trained LLM used in the experiments and applies LoRA for parameter-efficient fine-tuning).
Therefore, It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Vignard in view of Monka to a pre-trained LLM as taught by Chen in order to leverage pretrained language-model reasoning/generalization while adapting the model to the driving domain. Chen demonstrates that structured object-level autonomous-driving scene representations can be grounded into a pre-trained LLM and fine-tuned so the model reasons about the current driving environment and predicts driving actions.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Wu (US 20240420418) teaches: an LLM that may be trained to identify a next token in a graph using an underlying knowledge of how road objects and networks are connected in addition to road graph data that has already been presented to the LLM. One or more LLMs may be trained on languages for robotics environments (e.g., warehouses, factories, facilities, labs, buildings, etc.), languages for aerial vehicle environments. Wu also teaches that the graph can be represented as a knowledge graph that expresses road objects, road object relationships, and road network topology, rather than generic knowledge.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665
Prosecution Insights