DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
2. This communication is in response to the application filled on 10/31/2024.
3. Claims 1-20 are pending.
4. Limitations appearing inside {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Information Disclosure Statement
5. The information disclosure statements (IDS) submitted on 10/30/2024 have been considered by the examiner.
Specification
6. The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required:
Claim 5 recites “…generating the textual features and the image features further comprises generating photorealistic synthetic images that align with the textual features using a generative model”. Specifically, the examiner highlights that “… using a generative model” is not recited within the specifications. Specifically, par. [0038], [0052], and [0081] recite the corresponding generation of synthetic images, but fail to specifically recite wherein this is using a generative model. Specifically, the specification recites that the VLM or MMDC can provide such synthetic images, but do not mention a “generative model”. For the sake of compact prosecution, the examiner has interpreted “…using a generative model” to specifically be analogous to the recitation as provided in claim 5, i.e., “… using a generative model” being a separate model from the VLM or MMDC. The specification should be amended such that either a generative model is recited (e.g., for par. [0081]), or the claim may be amended to recite the VLM or MMDC.
Claim Objections
7. Claim 15 is objected to because of the following informalities:
Claim 15 recites “A non-transitory computer program product comprising a computer-readable storage medium including program code…”. The examiner suggests amending to recite “A computer program product comprising a non-transitory computer readable storage medium including program code…”, or the like. This specifically clarifies the non-transitory nature of the program product is via the storage medium, and thus avoids confusion.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
8. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 1, 8, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over “Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP” to Esmaeilpour et al. (hereinafter Esmaeilpour), in view of “Otter: A Multi-Modal Model with In-Context Instruction Tuning” to Li et al. (hereinafter Li), and further in view of U.S. Publication No. 2024/0203085 to Bangalath et al. (hereinafter Bangalath)
10. Regarding Claim 1, Esmaeilpour discloses a computer-implemented method for training a self-improving data engine {for autonomous vehicles} (SIDE), comprising ([pg. 6568, col. 1, Abstract, par. 1, ln. 1-20] “In an out-of-distribution (OOD) detection problem, samples of known classes (also called in-distribution classes) are used to train a special classifier. In testing, the classifier can (1) classify the test samples of known classes to their respective classes and also (2) detect samples that do not belong to any of the known classes (i.e., they belong to some unknown or OOD classes). This paper studies the problem of zero-shot out-of-distribution (OOD) detection, which still performs the same two tasks in testing but has no training except using the given known class names. This paper proposes a novel and yet simple method (called ZOC) to solve the problem. ZOC builds on top of the recent advances in zero-shot classification through multi-modal representation learning. It first extends the pre-trained language-vision model CLIP by training a text-based image description generator on top of CLIP. In testing, it uses the extended model to generate candidate unknown class names for each test sample and computes a confidence score based on both the known class names and candidate unknown class names for zero-shot OOD detection.”):
detecting unrecognized classes from diversified descriptions for input images generated using a {multi-modality dense} captioning (MMDC) model ([pg. 6568, col. 1, Introduction, par. 1, ln. 1-15] “The primary assumption in conventional supervised learning is that the samples encountered at the test time are from the same classes (called known or seen classes) that the model has observed and learned during training. However, this assumption, called the closed-world assumption (Fei and Liu 2016), is often violated when a machine learning model is deployed in the real world; i.e., in addition to the seen classes, samples from unseen classes may appear at test. The seen class samples are referred to as the in-distribution samples while unseen class samples are called out-of-distribution (OOD) samples. It is crucial for an intelligent ML model to detect OOD samples specially in safety critical applications such as autonomous driving or healthcare since detecting OOD samples as in-distribution ones in such applications can have catastrophic consequences.”, [pg. 6568, col. 2, par. 3, ln. 1-8] “This paper defines the zero-shot OOD detection problem to take advantage of pre-trained models. Given a set of seen class labels/names, Ys, the goal of zero-shot OOD detection is to 1) classify each seen class test sample to one of the seen classes and 2) detect samples that do not belong to any of the seen classes. These are done based on only the names of the seen classes in Ys. There is no given training data of the seen classes and thus no closed-world classifier is built.”, [pg. 6568, col. 1, par. 2, ln. 1-15] “Although using CLIP eliminates the need for training a closed-world classifier, it does not possess the OOD detection functionality in its original form. That is, it will match any given image to one of the given seen class labels. Therefore, to function in an OOD setting, we need to present another set of candidate labels in addition to the seen class labels/names. The proposed method, called ZOC (Zero-shot OOD detection based on CLIP), does not need this set of candidate labels to represent possible OOD labels as ZOC can dynamically generate candidate OOD labels for inference. ZOC works based on comparing the similarity of the semantic meaning of the given image to seen labels vs its similarity to some generated candidate labels. For this to work, we need a text generator to generate candidate labels, which does not exist in CLIP.”, [pg. 6570, col. 1, Training the Image Description Generator, par. 1, ln. 1 to col. 2, par. 4, ln. 11] “Since our image description generator uses the output features from the CLIP image encoder for training, we briefly describe the CLIP image encoder here. CLIP uses ResNet 50 (He et al. 2016) and the recently proposed vision trans former (ViT) (Dosovitskiy et al. 2020) as its image encoder backbone. We found that the ViT backbone is more compatible with the task of sequence generation from a given image since it processes the image as a sequence of tokens similar to the transformer model (Vaswani et al. 2017). The ViT encoder in CLIP is a hybrid ViT architecture which uses a convolutional layer in the beginning to extract image features. Then, N feature maps are used as N embedding vectors to represent the image as a sequence of embeddings. A classification embedding vector is concatenated to the image embeddings similar to the CLS token in BERT model (Devlin et al. 2018). Then, positional embeddings are added and the output is passed to a transformer encoder (Vaswani et al. 2017) with L1 layers. The hidden state
z
o
u
t
in the output is treated as the semantic representation of the whole image. We train the text generator on a large image captioning data which is a set of image and caption pairs. Text generator, which is a decoder, attends to the encoder output feature
z
o
u
t
in every layer of the decoder (see below). Please refer to Figure 1 for architecture details. Text decoder consists of L2 stacked transformer layers. In each layer, the multi-head cross-attention sub-layer takes
z
o
u
t
as key and value for the scaled dot product attention mechanism. The output from the final layer of the decoder is projected through a linear layer to the vocabulary space of the decoder… As we will explain in the next section, the output description from the decoder will eventually be processed to be used at the second step of inference. Therefore, a generated de scription with relevant words to the image is sufficient for our purpose. We refer to the decoder as
D
e
c
o
d
e
r
t
e
x
t
in the rest of the paper.
D
e
c
o
d
e
r
t
e
x
t
outputs a textual description for a given image based on the hidden state of the CLIP image encoder which we refer to as
C
L
I
P
i
m
a
g
e
. In this regard, the image to sequence architecture is a full transformer model which has
C
L
I
P
i
m
a
g
e
on the encoder side and
D
e
c
o
d
e
r
t
e
x
t
on the decoder side (see Figure 1).”, [pg. 6571, Figure 1] see
D
e
c
o
d
e
r
t
e
x
t
);
generating, with a vision-language-model (VLM), textual features from the diversified descriptions and image features from corresponding images to the diversified descriptions ([pg. 6570, col. 1, Method, par. 1, ln. 1 to par. 3, ln. 5] “We propose to solve the zero-shot OOD detection problem by extending zero-shot CLIP (Radford et al. 2021), which is a closed-world zero-shot classification method, to work in the OOD setting. As mentioned in the introduction, the zero-shot CLIP model is not equipped with a specialized technique for OOD detection. Although for any given closed-world classifier, maximum softmax probability (MSP) (Hendrycks and Gimpel 2016) is commonly used as a baseline score for OOD detection, we show in our experiments that our proposed method ZOC can significantly improve the detection performance. ZOC detects an OOD test sample by comparing the encoded image sample to two sets of encoded label names. The first set is the set of seen la bels, and the second set is the set of unseen labels which are unknown. ZOC trains a text description generator to obtain the second set. In the following, we briefly explain CLIP’s matching algorithm for closed-world zero-shot classification and discuss its shortcomings for OOD detection. For zero-shot closed-world classification in CLIP, we are only given a set of textual words as class labels
Y
s
=
{
y
1
,
y
2
,
…
,
y
n
}
. For a test image, the mutli-modal CLIP calculates the cosine similarity of the encoded image to each encoded textual description in the form of “{This is a photo of a <
y
i
>},” e.g., “This is a photo of a dog,”… Taking the softmax over all the n similarity scores gives a categorical probability distribution that determines the label for the image. It is easily seen that any given image can be matched to one of the given (possibly irrelevant) labels based on the maximum softmax score. As we can see, this method does not deal with zero-shot OOD detection. To do so, we propose to present CLIP with another set of possible labels
Y
u
for each test image sample for zero-shot matching. For this, we need a text-based image description generator. We train such a generator and use it to extract
Y
u
from a given test image. The next question is how the second set
Y
u
can assist in detecting an OOD sample. We will show later how the seen (known) labels together with the dynamic set
Y
u
can be used to define a confidence score per test image. Since CLIP does not have a text generator,
Y
u
for a given image, we propose to train one on top of CLIP’s image encoder using a large image captioning dataset. We explain the training of the generator next. We also call the text generator the image description generator.”, [pg. 6570, col. 1, Training the Image Description Generator, par. 1, ln. 1 to col. 2, par. 4, ln. 11] specifically CLIP description, [pg. 6571, Figure 1] see
C
L
I
P
t
e
x
t
and
C
L
I
P
i
m
a
g
e
);
obtaining curated features, including curated textual features and curated image features, by comparing similarity scores between the textual features and top-ranked image features based on their likelihood scores ([pg. 6570, col. 2, Inference in Texting, par. 1, ln. 1 to pg. 6571, col. 2, par. 2, ln. 8] “
D
e
c
o
d
e
r
t
e
x
t
is the central component for inference (testing) in our ZOC. The inference is performed in lines 4-18 in Algorithm 1 which is composed of two steps. In the first step,
D
e
c
o
d
e
r
t
e
x
t
generates the image description for the given test image by attending to the image semantic representation in the output of
C
L
I
P
i
m
a
g
e
. The generation follows the standard procedure of sequence to sequence models (predicting the next word based on the output of the model for the previous words until the maximum length is reached). ZOC needs to retrieve candidate unseen labels
Y
u
from the generated description.1 Since
Y
u
is eventually used to define the confidence score for OOD detection, we would like the retrieved words to be diverse and relevant to the input image. i.e, diversity results in a more reliable confidence score for detection. However, canonical inference methods such as greedy generation, beam search, nucleus sampling (Holtzman et al. 2019) or top-k sampling (Fan, Lewis, and Dauphin 2018) targets to generate the best description rather than diverse descriptions. Since we need a holistic description of the image in general, the best description does not suit our purpose as it is not diverse enough. Thus, we do not limit the set of candidate labels
Y
u
to be the same as the best generated description. Instead, we form
Y
u
with some post-processing as follows: assuming the maximum generation length is T, at each position
p
i
of
{
p
1
,
p
2
,
…
,
p
T
}
, we pick the top k words from the vocabulary with the highest probabilities. The union of all these words is
Y
u
(line 8 in Algorithim 1). We fix k for all of our experiments. Then, we form the union of seen labels
Y
s
and candidate unseen labels
Y
u
(line 9). The second step follows the CLIP zero-shot classification technique based on zero-shot labels
Y
s
∪
Y
u
. Each
y
i
∈
Y
s
∪
Y
u
is put in the template (i.e., “This is a photo of a <
y
i
>”) required by CLIP. The text and the image are encoded through
C
L
I
P
t
e
x
t
and
C
L
I
P
i
m
a
g
e
and the cosine similarity of the encoded image and encoded label (in template) is calculated (lines 11-15). The softmax of all calculated similarities gives a probability distribution over
Y
s
∪
Y
u
(line 16). We define the OOD confidence score (line 17) as follows:
S
x
=
1
-
∑
y
∈
Y
s
P
(
y
|
x
)
(2) where
P
(
y
|
x
)
is the softmax probability for label y. Thus S(x) is the accumulative probability of labels
Y
u
. Even though ZOC inference is done in two steps, the implementation and usage of our technique is straightforward as the second step is done by querying the CLIP encoders. Figure 1 is a graphical illustration of the inference procedure of ZOC. The used example describes how ZOC detects a sample as OOD. The input image is from class ‘boat’ which is not among the seen labels and therefore it is an unseen class or OOD sample. It is interesting to note that the actual unseen label ‘boat’ is not among the set of candidate unseen labels, and yet ZOC uses other candidate unseen labels to come to the correct conclusion.”);
generating annotations, including {bounding boxes and} labels, for the curated features by comparing the similarity scores of labels generated by a zero-shot classifier and the curated textual features ([pg. 6570, col. 1, Method, par. 1, ln. 1 to par. 3, ln. 5], [pg. 6570, col. 2, Inference in Texting, par. 1, ln. 1 to pg. 6571, col. 2, par. 2, ln. 8], [pg. 6571, Figure 1]); and
training the SIDE using the curated features, annotations, and feedback ([pg. 6572, col. 1, Model Architecture and Training Details, par. 1, ln. 1-31] “Recall that ZOC consists of 3 modules. The two encoders
C
L
I
P
i
m
a
g
e
and
C
L
I
P
t
e
x
t
are pre-trained transformer models for image and text (Radford et al. 2021), respectively. We do not change or fine-tune the encoders… For the proposed
D
e
c
o
d
e
r
t
e
x
t
, we choose the BERT large model from huggingface (Wolf et al. 2020) with 24 layers and hidden size of 1024. We train
D
e
c
o
d
e
r
t
e
x
t
using Adam optimizer (Kingma and Ba 2017) with a constant learning rate of
10
-
5
for 25 epochs. Batch size is 128. The training data for fine-tuning is the training split of MS-COCO (2017 release) (Lin et al. 2014)2 which is a commonly used dataset for image captioning. We used MS-COCO validation dataset to choose the k value. We empirically found that the meaningful candidate unseen labels are present at top 35 level of the annotations. We used the basic teacher forcing method to train
D
e
c
o
d
e
r
t
e
x
t
as it is sufficient for our purpose.”).
Esmaeilpour does not specifically disclose wherein the model is specifically for autonomous vehicles, though it is noted in Esmaeilpour that this is one of the proposed uses ([pg. 6568, col. 1, Introduction, par. 1, ln. 1-15] see “…It is crucial for an intelligent ML model to detect OOD samples specially in safety critical applications such as autonomous driving or healthcare since detecting OOD samples as in-distribution ones in such applications can have catastrophic consequences.”). Furthermore, while Esmaeilpour discloses an analogous captioning model, the captioning model of Esmaeilpour is not multi-modal. Likewise, though Esmaielpour does generate annotations including unseen labels, Esmaeilpour does not specifically disclose wherein the annotations include a bounding box.
However, Li specifically discloses a multi-modal dense captioning model ([pg. 2, par. 4, ln. 1-12] “… we purpose our MutlI-Modal In-Context Instruction Tuning (MIMIC-IT) dataset and provide details on its construction in Sec. 3.1. We then introduce Otter, a multi-modal model with in-context instruction tuning based on OpenFlamingo. We illustrate the relationship between Otter and OpenFlamingo in Fig. 1. Finetuned on MIMIC-IT dataset, our Otter model demonstrates improved instruction-following ability compared to OpenFlamingo, as shown in our qualitative analysis in Sec. 4.1. Meanwhile, Otter is capable of learning to execute instructions with provided in-context learning examples, as shown in Sec. 4.2. From the engineering perspective, we optimized OpenFlamingo’s implementation to make it more accessible to researchers. Our optimizations include optimizing the training requirements from at least 1× A100 GPU to only 4× RTX3090 GPUs and integrating it into Hugging Face Transformers [34] to simplify training and inference with a few lines of code.”, [pg. 5, Fig. 2], [pg. 4, 3.2. Training Details, par, 1 ln. 1-6] “Our approach adopts the OpenFlamingo training paradigm to train the Otter model. The pretrained OpenFlamingo model comprises a LLaMA-7B [33] language encoder and a CLIP ViT-L/14 [24] vision encoder. To prevent overfitting and leverage pretrained knowledge, we freeze both the encoders and only finetune the Perceiver resampler module, cross-attention layers inserted into the language encoder and input/output embeddings of the language encoder. This results in approximately 1.3 billion trainable parameters for the Otter model.”, [pg. 6, 3.3 Integration with Hugging Face, pear. 1, ln. 1 to par. 2, ln. 5] “We have integrated Otter into Hugging Face Transformers [34] and trained it using the Hugging Face Accelerator2, which enables automatic mapping of the model weights to different GPU devices and offloading of overflowed weights to CPU or disk. Additionally, we use bf16 mixed precision during training. The total optimizations enable our model to be trained on 4×RTX-3090 GPUs, each with 24GB memory. Meanwhile, since Otter has been integrated into Hugging Face Transformers, it can now be reused with less than five lines of code, making it much easier for researchers to integrate into their respective training and inference pipelines (compared to the original OpenFlamingo implementation). We also provide the support of Fully Sharded Data Parallel (FSDP) and DeepSpeed to enable greater training efficiency and less memory consumption”, [pg. 7, Fig. 3 and 4]). One of ordinary skill in the art, before the effective filing date of the claimed invention, would specifically recognize Esmaeilpour and Li as within the same field of captioning for images, and as analogous to the claimed invention. The motivation to combine is disclosed in Li, wherein it provides improved training efficiency and memory consumption, as well as allowing improved in-context learning ([pg. 4, 3.1 Mutli-Modal In-Context Instruction Tuning, par. 1, ln. 1-8] “The OpenFlamingo framework leverages the interleaved multi-modal MMC4 dataset to emerge in its few-shot, in-context learning capabilities. The MMC4 dataset is composed of image-text pairs derived from individual HTML files, with significant contextual relationships between different pairs, as depicted in Fig 2(a). An MMC4 training data sample contains (i) a queried image-text pair, where the text typically describes the image, and (ii) context, which includes the remaining image-text pairs from the same HTML file. The primary training objective of OpenFlamingo is to generate text for the queried image-text pair, and the paradigm of generating query text conditioned on in-context examples ensures OpenFlamingo’s in-context learning capacity during the inference phase.”, [pg. 2, par. 4, ln. 1-12], [pg. 4, 3.2. Training Details, par, 1 ln. 1-6], [pg. 6, 3.3 Integration with Hugging Face, pear. 1, ln. 1 to par. 2, ln. 5]). One of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of Esmaeilpour with the multi-modal dense captioning model of Li, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have replaced the
D
e
c
o
d
e
r
t
e
x
t
of Esmaeilpour with the multi-modal dense captioning model of Li to generate improved in-context dense captions for the unseen classes analogous to
D
e
c
o
d
e
r
t
e
x
t
of Esmaeilpour.
Li does not specifically disclose generating annotations including bounding boxes. Therefore, a combination of Esmaeilpour and Li does not specifically disclose wherein the annotations include generating a bounding box.
However, Bangalath specifically teaches wherein an analogous SIDE can be used for autonomous vehicles ([Fig. 2], [par. 0119, ln. 1-13] “… a vehicle equipped with ADAS or a fully automated vehicle (self-driving vehicle) is configured with a machine learning module including processing circuitry of FIG. 14 for performing object detection. As in the exemplary vehicle of FIG. 2, the machine learning module receives images from multiple cameras 204 and detects objects (base and novel) in the images. The object detection can be performed in the present OVD detector 400. The machine learning module can execute program instructions to make high-level decisions based on the detected objects, including avoiding collisions, incorporate traffic warnings, alert drivers to possible obstacles, and assist in lane departure and lane centering.”, [par. 0120, ln. 1-15] “Training of an OVD detector that is embedded in a computer system of an autonomous vehicle (self-driving vehicle or SAE level 2 or above ADAS) can be performed using a system, such as that in FIG. 1. MViTs for the OVD detector can be trained as new annotated images are added to the database 120. The OVD detector 400 can be updated as new images are received from cameras 130. The cameras 130 may be in the vehicle itself, may be distributed among multiple vehicles, and/or may be various cameras including cameras for mobile devices. The training of the OVD detector 400 can be performed in an AI workstation 102 or in a cloud service 110, or a combination as needed. In one embodiment, a vehicle may be equipped with a machine learning module that is capable of both learning and inference, such as the NVIDIA Drive processor family.”) and annotations include bounding boxes ([par. 0052, ln. 1-12] “Faster R-CNN uses a convolutional network, referred to as a region proposal network (RPN) 304, in combination with a Fast R-CNN 308. The region proposal network (RPN) 304 starts with the input image 302 being fed into the backbone convolutional neural network. The input image 302 is first resized such that its shortest side is 600 px with the longer side not exceeding 1000 px. As the network moves through each pixel in the output feature map, it has to check whether these k corresponding anchors spanning the input image actually contain objects, and refine these anchors' coordinates to give bounding boxes as “Object proposals” or regions of interest 306.”, [par. 0053, ln. 1-15] “The Fast R-CNN detector 308 also consists of a CNN backbone, an ROI pooling layer 312 and fully connected layers followed by two sibling branches for classification 314 and bounding box regression 316. The ROI pooling layer 312 works by a) Taking the region corresponding to a proposal from the backbone feature map: b) Dividing this region into a fixed number of sub-windows: c) Performing max-pooling over these sub-windows to give a fixed size output. After passing them through two fully connected layers, the features are fed into the sibling classification 314 and regression 316 branches. The classification 314 is trainable with a classification loss L.sub.c(p.sub.i, p.sub.i*) that is the log loss over two classes (object vs not object). The regression loss L.sub.re(t.sub.i, t.sub.i*) is activated only if the anchor actually contains an object.”, [par. 0054, ln. 1-8] “At test time, all the boxes are arranged according to their cls scores. Then, a non-maximum suppression (NMS) is applied with a threshold of 0.7. From the top down, all of the bounding boxes which have an IoU of greater than 0.7 with another bounding box are discarded. Thus, the highest-scoring bounding box is retained for a group of overlapping boxes. The cross-boundary bounding boxes are retained and clipped to image boundary.”). One of ordinary skill in the art, before the effective filing date of the claimed invention, would recognize Esmaeilpour, Li, and Bangalath as within the same field of image captioning, and Esmaielpour and Bangalath as further within the same field of VLM’s for zero-shot classification of novel objects. Specifically, the motivation to combine is disclosed in Bangalath, wherein applying the model to autonomous driving allows for various safety improvement such as alerts for obstacles and avoiding collisions ([par. 0119, ln. 1-13]), and wherein determining the bounding box allows for improved novel class generalization ([par. 0056, ln. 1-23] “conventional OVD methods that use different forms of weak supervision employ image-centric representations, making them less suited for the end object detection task. The present OVD method aims to bridge the gap between image and object-centric visual-language (VL) representations. The present framework 400 has three main elements. 1) A region-based knowledge distillation 410 adapts image-centric language representations to be object-centric. A VL mapping 426 learns to align the local region representations of the detector 440 to the language representations by distilling the detector's region representations with region representations from a VL model (CLIP). 2) Given weak image-level supervision, s pseudo-labeling from pretrained multi-modal ViTs improve generalization of the detector 440 to novel classes. 3) For efficient combination of the above two proposed components, the VL mapping learned during the weak supervision is conditioned on the VL mapping 426 learned with region-based distillation via a novel weight transfer function 452. Specifically, a stage-wise learning strategy is to first align the region and language embeddings using RKD, and then use the distilled VL mapping for object-centric visual and language alignment in the subsequent stage 440.”). One of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of Esmaeilpour with the multi-modal dense captioning model of Li, and further combined the method of the combination of Esmaeilpour and Li with the bounding box annotations and application to autonomous vehicles of Bangalath, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of the combination of Esmaeilpour and Li with a bounding box generation network analogous to Bangalath to allow for localization of class labels within an image for the purpose of autonomous driving (e.g., detecting object is in front of moving vehicle based on image location to avoid collision).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li and the bounding box annotations and application to autonomous vehicles of Bangalath to obtain the invention as specified in claim 1.
11. Regarding Claim 8, the claim language is analogous to claim 1, with the exception of “A system…, comprising: a memory device; one or more processor devices operatively coupled with the memory device to:”, wherein the remainder of the claim is analogous to claim 1. Esmaeilpour and Li do not specifically disclose a system, comprising: a memory device; one or more processor devices operatively coupled with the memory device to perform the method. However, Bangalath discloses disclose a system, comprising: a memory device; one or more processor devices operatively coupled with the memory device to perform the method ([par. 0117, ln. 1-15] “FIG. 14 is a block diagram illustrating an example computer system for implementing the machine learning training and inference methods according to an exemplary aspect of the disclosure. As noted above, machine learning models are trained using a computer system configured with 8 Nvidia A100 GPUs and program instructions stored in a computer readable storage medium… The computer system 1400 may include one or more central processing units (CPU) 1450 having multiple cores. The computer system 1400 may include a graphics board 1412 having multiple GPUs, each GPU having GPU memory. The graphics board 1412, such as the A100 GPUs, may perform many of the mathematical operations of the disclosed machine learning methods. The computer system 1400 includes main memory 1402, typically random access memory RAM, which contains the software being executed by the processing cores 1450 and GPUs 1412, as well as a non-volatile storage device 1404 for storing data and the software programs”). The motivation to combine would have been obvious to one of ordinary skill in the art, in that it allows for training the models and the real-world application of the methods of the combination of Esmaielpour and Li. One of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of Esmaeilpour with the multi-modal dense captioning model of Li, and further combined the method of the combination of Esmaeilpour and Li with the bounding box annotations, application to autonomous vehicles, and system of Bangalath, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li and the bounding box annotations, application to autonomous vehicles, and system of Bangalath to obtain the invention as specified in claim 8.
12. Regarding Claim 15, the claim language is analogous to claim 1, with the exception of “A non-transitory computer program product comprising a computer-readable storage medium including program code… wherein the program code when executed on a computer causes the computer to…”, wherein the remainder of the claim is analogous to claim 1. Bangalath specifically discloses A non-transitory computer program product comprising a computer-readable storage medium including program code wherein the program code when executed on a computer causes the computer to perform the method ([par. 0117, ln. 1-15]). The motivation to combine remains analogous to claim 8. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li and the bounding box annotations, application to autonomous vehicles, and non-transitory computer-readable storage medium of Bangalath to obtain the invention as specified in claim 15.
13. Claims 2, 9, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over “Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP” to Esmaeilpour, in view of “Otter: A Multi-Modal Model with In-Context Instruction Tuning” to Li, in view of U.S. Publication No. 2024/0203085 to Bangalath, and further in view of U.S. Publication No. 2023/0037142 to Nayhouse et al. (hereinafter Nayhouse).
14. Regarding Claim 2, a combination of Esmaeilpour, Li, and Bangalath teaches the method of claim 1. Esmaeilpour, Li, and Bangalath do not specifically disclose generating trajectories within a traffic scene simulation to control an autonomous vehicle using the trained SIDE.
However, Nayhouse specifically teaches generating trajectories within a traffic scene simulation to control an autonomous vehicle using the trained SIDE ([par. 0032, ln. 1-17] “The perception system may include one or more processors, and computer-readable memory with programming instructions and/or trained artificial intelligence models that, during a run of the AV, will process the perception data to identify objects and assign categorical labels and unique identifiers to each object detected in a scene. Categorical labels may include categories such as vehicle, bicyclist, pedestrian, building, and the like. Methods of identifying objects and assigning categorical labels to objects are well known in the art, and any suitable classification process may be used, such as those that make bounding box predictions for detected objects in a scene and use convolutional neural networks or other computer vision models.”, [par. 0034, ln. 1-21] “The vehicle's perception system 202, as well as the vehicle's forecasting system 203, will deliver data and information to the vehicle's motion planning system 204 and motion control system 205 so that the receiving systems may assess such data and initiate any number of reactive motions to such data. The motion planning system 204 and motion control system 205 include and/or share one or more processors and computer-readable programming instructions that are configured to process data received from the other systems, compute a trajectory for the vehicle, and output commands to vehicle hardware to move the vehicle according to the determined trajectory. Example actions that such commands may cause include causing the vehicle's brake control system to actuate, causing the vehicle's acceleration control subsystem to increase speed of the vehicle, or causing the vehicle's steering control subsystem to turn the vehicle.”, [par. 0035, ln. 1-16] “The subsystems described above may be implemented as components of an AV stack, which may be trained on various simulation scenarios… the system 201 on which the subsystems may be installed may be a vehicle's computer processing hardware, or it may be one or more memory devices that are offboard the vehicle. The system 201 may be in communication with a remote server 206 that provides updates and/or commands, or which receives data from the AV stack… 201 on which the AV stack is installed will be in electronic communication with a training system 209. The training system 209 will include a processor 211, a data store 212 containing a variety of stored simulation scenarios, and a memory containing programming instructions 213 for generating, modifying and using simulation scenarios to train the system 201.”, [par. 0051, ln. 1-12] “Once the system generates an augmented simulation scenario, at 311 the system may test the augmented simulation scenario by applying the augmented simulation scenario to the AV stack, optionally over multiple iterations and optionally with varied parameters. To test the augmented simulation scenario, the system will apply a planned trajectory of the vehicle to the scene in the augmented simulation scenario. The vehicle's perception system will detect the augmentation element in the simulation, and the vehicle's motion planning system will compute a continued trajectory response to the detected augmentation element.”, [par. 0066, ln. 1-27] “An obstruent augmentation element is an object having a location that will at least partially block the vehicle's planned path. An example of this is shown in FIG. 7, in which vehicle 701 is moving on road 717 along a planned trajectory 702. Parked vehicle 719 is obstruent because its position is skewed, not parallel to the road, and therefore partially within a threshold distance from the centerline of the vehicle's planned trajectory 702. In the augmented simulation, when the vehicle's perception system detects the parked vehicle 719, the vehicle's motion planning system may alter the trajectory of vehicle 701 to: (a) veer slightly to the left to maintain a threshold distance between vehicle 701 and parked vehicle 719; and/or (b) to move more slowly as it approaches the parked vehicle 719 in case the parked vehicle begins motion and pulls into the lane in front of the vehicle 701… an obstruent augmentation element may include any known object class, such as vehicle, pedestrian, cyclist, animal, vegetation, or even unknown/unidentifiable. The system may include a mapping for each object class with potential behaviors/states, along with probabilities of each behavior or state. By way of example, potential states of a vehicle may include parked, moving forward, turning left, turning right, accelerating, decelerating, among other states. Potential states of a pedestrian may include examples such as walking into lane, walking parallel to lane, standing facing lane, standing facing away from lane, among other states.”). One of ordinary skill in the art, before the effective filing date of the claimed invention, would specifically recognize the method of the combination of Esmaeilpour, Li, and Banagalath and Nayhouse as within the same field of object identification and classification for autonomous vehicles, and as analogous to the claimed invention. The motivation to combine is disclosed in Nayhouse, wherein generating trajectories allows for control of the autonomous vehicle to avoid obstacles ([par. 0066, ln. 1-27]). One of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of the combination of Esmaeilpour, Li, and Banagalath with the traffic scene simulation and trajectories of Nayhouse through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have combined the method of the combination of Esmaeilpour, Li, and Banagalath with the traffic scene simulation and trajectories of Nayhouse such that trajectories were identified for within the traffic scene simulation for objects identified using the trained SIDE to control the vehicle in the traffic scene as disclosed in Nayhouse.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, the bounding box annotations and application to autonomous vehicles of Bangalath, and the traffic scene simulation and trajectories of Nayhouse to obtain the invention as specified in claim 2.
15. Regarding Claim 9, a combination of Esmeailpour, Li, and Bangalath teaches the system of claim 8. Rejections analogous to claim 2 are further applicable to claim 9 in view of the analogous claim language. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, the bounding box annotations, application to autonomous vehicles, and system of Bangalath, and the traffic scene simulation and trajectories of Nayhouse to obtain the invention as specified in claim 9.
16. Regarding Claim 16, a combination of Esmeailpour, Li, and Bangalath teaches the non-transitory computer program product of claim 15. Rejections analogous to claim 2 are further applicable to claim 16 in view of the analogous claim language. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, application to autonomous vehicles, and non-transitory computer-readable storage medium of Bangalath, and the traffic scene simulation and trajectories of Nayhouse to obtain the invention as specified in claim 16.
17. Claims 5, 12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over “Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP” to Esmaeilpour, in view of “Otter: A Multi-Modal Model with In-Context Instruction Tuning” to Li, in view of U.S. Publication No. 2024/0203085 to Bangalath, and further in view of “Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infilings” to Rose et al. (hereinafter Rose).
18. Regarding Claim 5, a combination of Esmaeilpour, Li, and Bangalath teaches the method of claim 1. Esmaeilpour, Li, and Bangalath do not specifically disclose wherein generating the textual features and the image features further comprises generating photorealistic synthetic images that align with the textual features using a generative model.
However, Rose specifically teaches wherein generating the textual features and the image features further comprises generating photorealistic synthetic images that align with the textual features using a generative model ([pg. 3, Fig. 2], [pg. 2, col. 2, 3 Problem Formulation, par. 1, ln. 1 to pg. 3, col. 1, par. 1, ln. 11] “To improve temporal reasoning in language models, we define the multimodal infiling task to bridge the logical gaps in sequential data and explicitly guide the vision-language model reasoning process. We generate an infiling {
(
v
i
,
t
i
)
} given two text-visual sequential pairs {
v
i
-
1
,
t
i
-
1
,
(
v
i
+
1
,
t
i
+
1
)
} through generator g using model M (Equation 1). The best infiling
(
v
i
b
e
s
t
,
t
i
b
e
s
t
)
is judged using the judgement function
j
(
v
i
,
t
i
)
(Equation 2). For a downstream task T measured by performance p (e.g., novelty, consistency, coherence, descriptiveness), we defined the optimal filing
(
v
i
o
p
t
,
t
i
o
p
t
)
to be
(
v
i
b
e
s
t
,
t
i
b
e
s
t
)
only when the infiling improves the downstream task performance (Equation 3); otherwise, the optimal infiling is null, meaning an infiling is not beneficial (Equation 4).”, [pg. 3, col. 2, 3.1. Task Unification, par. 1, ln. 1 to pg. 4, col. 1, par. 2, ln. 17] “To apply VCOT, we reformat sequential data into text-visual pairs. For text-only data, we generate visuals
v
1
'
,
…
,
v
n
'
from input text
{
t
1
,
…
,
t
n
}
with STABLE DIFFUSION, where
v
i
'
contains multiple candidate visuals for
t
i
. We use CLIP embedding to asses similarity between the candidate visuals and the surrounding input texts
t
i
-
1
,
t
i
+
1
and select the visual with the highest similarity score to yield consistent visuals that unify general sequential tasks as text-visual pairs. 4.2 Multipoint Foveation… To preserve consistency of the input sequence when generating multimodal infilings, we use foveation (Mei et al., 2022) to identify the main focus. Since pairwise sequential elements may omit relevant fixation points, we define multipoint foveation (MPF) to identify all of the core fixation points (setting, characters, etc.) of the entire text-visual input sequence (Figure 3). To construct the multipoint foveation, we project the text-visual pairs into a unimodal text space by captioning the visuals using an image-captioning model:
v
1
,
t
1
…
v
n
,
t
n
→
c
1
,
t
1
…
c
n
,
t
n
. The projected output, along with diverse few-shot exemplars, is fed into GPT 3 to generate a maximum likelihood3 summary, from which the multipoint foveation f is extracted (Equation 5). The foveation guides infilings to be consistent and not introduce excessive information…”). One of ordinary skill in the art, before the effective filing date of the claimed invention, would specifically recognize Esmaeilpour, Li, and Bangalath and Rose as within the same field of image captioning for image processing, and as analogous to the claimed invention. Specifically, the motivation to combine is disclosed in Rose, wherein it generating photorealistic synthetic images that align with the textual features using a generative model can improve task performance and reasoning of the model ([pg. 2, col. 2, 3 Problem Formulation, par. 1, ln. 1 to pg. 3, col. 1, par. 1, ln. 11]). One of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of the combination of Esmaeilpour, Li, and Bangalath with the synthetic image generation of Rose through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have used the text features and image features of the models of the method of the combination of Esmaeilpour, Li, and Bangalath as the input to the generative model of Rose to further generate photorealistic synthetic images that can supplement the models existing knowledge.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, the bounding box annotations and application to autonomous vehicles of Bangalath, and the synthetic image generation of Rose to obtain the invention as specified in claim 5.
19. Regarding Claim 12, a combination of Esmeailpour, Li, and Bangalath teaches the system of claim 8. Rejections analogous to claim 5 are further applicable to claim 12 in view of the analogous claim language. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, the bounding box annotations, application to autonomous vehicles, and system of Bangalath, and the synthetic image generation of Rose to obtain the invention as specified in claim 12.
20. Regarding Claim 16, a combination of Esmeailpour, Li, and Bangalath teaches the non-transitory computer program product of claim 15. Rejections analogous to claim 5 are further applicable to claim 16 in view of the analogous claim language. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, application to autonomous vehicles, and non-transitory computer-readable storage medium of Bangalath, and the synthetic image generation of Rose to obtain the invention as specified in claim 16.
21. Claim 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over “Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP” to Esmaeilpour, in view of “Otter: A Multi-Modal Model with In-Context Instruction Tuning” to Li, in view of U.S. Publication No. 2024/0203085 to Bangalath, and further in view of “nuScenes: A multimodal dataset for autonomous driving” to Caesar et al. (hereinafter Caesar).
22. Regarding Claim 6, a combination of Esmaeilpour, Li, and Bangalath teaches the method of claim 1. Esmaeilpour discloses wherein the generating the annotations further comprises combining a base label space including the curated features with existing datasets {to include objects likely present on road scenes} for zero-shot classification ([pg. 6572, col. 2, Datasets, par. 1, ln. 1-35] “We evaluate the performance of our proposed method ZOC on splits of CIFAR10, CIFAR100, CIFAR+10, CIFAR+50, and TinyImagenet… For CIFAR+10 (Krizhevsky, Hinton et al. 2009)4 4 non-animal classes of CIFAR10 are used as in-distribution (or seen) classes. 10 animal classes are chosen from CIFAR100 as the OOD (unseen) classes. The reported score is averaged over 5 splits (Openness = 33.33). For CIFAR+50 (Krizhevsky, Hinton et al. 2009)5 4 non-animal classes from CIFAR10 are in-distribution (or seen). All 50 animal classes from CI FAR100 are used as the OOD classes (Openness = 62.86%). For TinyImagenet. (Le and Yang 2015)6 20 classes are used as the in-distribution (or seen) classes. The remaining 180 classes are used as OOD (unseen) classes…”). Esmaeilpour does not specifically disclose wherein the dataset includes objects likely present on road scenes. Likewise, Li and Bangalath do not specifically disclose wherein the dataset includes objects likely present on road scenes. Specifically, the examiner notes that while Bangalath teaches analogous base label space and/or datasets, Bangalath does not teach a specific dataset includes objects likely to be present on road scenes (i.e., a task specific dataset).
However, Caesar teaches wherein the dataset includes objects likely present on road scenes ([pg. 11621, Fig. 1], [pg. 11624, col. 2, Data annotation, par, 1, ln. 1-14] “Having selected the scenes, we sample keyframes (image, lidar, radar) at 2Hz. We annotate each of the 23 object classes in every keyframe with a semantic category, attributes (visibility, activity, and pose) and a cuboid modeled as x, y, z, width, length, height and yaw angle. We annotate objects continuously throughout each scene if they are covered by at least one lidar or radar point. Using expert annotators and multiple validation steps, we achieve highly accurate annotations. We also release intermediate sensor frames, which are important for tracking, prediction and object detection as shown in Section 4.2. At capture frequencies of 12Hz, 13Hz and 20Hz for camera, radar and lidar, this makes our dataset unique. Only the Waymo Open dataset provides a similarly high capture frequency of 10Hz”). One of ordinary skill in the art, before the effective filing date of the claimed invention, would specifically recognize Esmaeilpour, Li, Bangalath, and Caesar as within the same field of multi-modal machine learning, and as analogous to the claimed invention. The motivation to combine would have been obvious to one of ordinary skill in the art, in that training the SIDE of the method of the combination of Esmaeilpour, Li, and Bangalath on a task specific dataset for autonomous vehicles including objects likely present on road scenes as taught in Caesar would improve the ability of the SIDE to detect and identify objects likely present on road scenes. One of ordinary skill in the art, before the effective filing date of the claimed invention, would have combined the method of the combination of Esmaeilpour, Li, and Bangalath with the dataset including objects likely present on road scenes of Caesar through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have further trained the SIDE of the combination of Esmaeilpour, Li, and Bangalath using the dataset of Caesar to improve the SIDE performance for autonomous vehicles applications.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, the bounding box annotations and application to autonomous vehicles of Bangalath, and the dataset including objects likely present on road scenes of Caesar to obtain the invention as specified in claim 6.
23. Regarding Claim 13, a combination of Esmeailpour, Li, and Bangalath teaches the system of claim 8. Rejections analogous to claim 6 are further applicable to claim 13 in view of the analogous claim language. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, the bounding box annotations, application to autonomous vehicles, and system of Bangalath, and the dataset including objects likely present on road scenes of Caesar to obtain the invention as specified in claim 13.
24. Regarding Claim 19, a combination of Esmeailpour, Li, and Bangalath teaches the non-transitory computer program product of claim 15. Rejections analogous to claim 6 are further applicable to claim 19 in view of the analogous claim language. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the method of Esmaeilpour with the multi-modal dense captioning model of Li, application to autonomous vehicles, and non-transitory computer-readable storage medium of Bangalath, and the synthetic image generation of Rose to obtain the invention as specified in claim 19.
Allowable Subject Matter
25. Claims 3-4, 7, 10-11, 14, 17, and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
26. The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claims 3-4, 10-11, and 17, the examiner specifically notes that while references of record teach generating traffic simulations, said traffic simulations are not taught to be generated by the VLM. Specifically, the examiner notes that the closes teaching to a traffic simulation generation is found in references Rose and Nayhouse. Specifically, Nayhouse teaches to generate traffic simulations, and Rose teaches wherein the SIDE can generate synthetic data for the purpose of verifying its ability, but the examiner specifically notes that it would not have been obvious to combine the references of record to obtain the invention as specified. This is because Nayhouse fails to specifically reference and VLM with regard to the traffic simulation generation, and while the synthetic data of Rose may be recognized as a “traffic simulation” if combined, Rose uses a separate model (i.e., Stable diffusion) to perform the data generation, and thus fails to specifically disclose that the generating diverse traffic simulations is using the VLM. Regarding Claims 7, 14, and 20, the examiner notes that while references of record specifically teach pseudo-labels with relation to unknown classes (see Bangalath or U.S. Publication No. 2025/038450 to Kuo et al.), they do not specifically disclose wherein known class pseudo-labels are retained to continuously train the SIDE. Specifically, references or record only teach using the pseudo-labels for unknown or unrecognized class recognition, and do not specifically disclose retaining them after they have been learned for a specific class instance.
Conclusion
27. The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. See PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAULO ANDRES GARCIA whose telephone number is (703)756-5493. The examiner can normally be reached Mon-Fri, 8-4:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached on (571)272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAULO ANDRES GARCIA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights