DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US Pub. 2026/0141702), hereinafter Chen, in view of Wang et al. (NPL: Learning Combinatorial Prompts for Universal Controllable Image Captioning), hereinafter Wang.
Regarding claim 1, Chen discloses a method performed by one or more data processing apparatus, the method comprising: obtaining a plurality of images for captioning (Paragraph [0048]: generating predictions on-board an autonomous vehicle, more generally, the described techniques can be implemented on any system of one or more computers that receives images of scenes in an environment. That is, once the training system 130 has trained the observation encoding system 120, the observation encoding system 120 can be used by any system of one or more computers); processing, by an image captioning machine learning model and for each image of the plurality of images, a model input comprising the image and an input prompt to generate a caption for the image (Paragraph [0171]: a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on; Paragraph [0178]: system can process the observation embedding with each of the received prompts using a caption system to generate a respective caption for the observation. (step 426). For example, the caption system can be a language model (e.g., a vision language model) configured to process input token sequences that include input prompts to generate output token sequences characterizing captions for the observation generated in accordance with the input prompts. As a further example, the language model can be a Transformer model (e.g., a vision transformer model) configured to generate the output token sequences representing the captions by performing sequences of attention operations to process the input token sequences. The caption system can have any appropriate architecture for conditionally generating captions for the observation as conditioned on the observation embedding. As one example, the caption system can be configured to process an input token sequence that includes the received observation embedding. As another example, the caption system can include one or more cross-attention layers that can perform cross-attention operations using the received observation embedding to generate the captions for the observation. In some implementations, the caption system can be trained (e.g., pre-trained) to generate captions for observations of image data by processing observation embeddings for the observations of image data; Paragraph [0184]: described in more detail above with reference to FIG. 2A and FIG. 2B, each example input query can include example task embeddings for the prediction task characterized by the input query. For example, the example task embeddings can be machine-learned parameters (e.g., machine learned vectors). As another example the example task embeddings can be generated by processing corresponding text prompts using a text embedding neural network (e.g., using a language model); wherein the input prompt comprises an indication of a level of specificity for the caption (Paragraph [0171]: a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on; Paragraph [0177]: system can receive multiple prompts for captioning the observation (step 424). Each of the multiple prompts can include a distinct request to generate a caption (e.g., a text description) of the observation. For example, the multiple prompts can include requests to generate captions for the observation, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on).
Chen does not explicitly disclose wherein the generated captions for the plurality of images have mixed levels of specificity.
However, Wang teaches captioning images based on prompts (Abstract), wherein the generated captions for the plurality of images have mixed levels of specificity (Fig. 2; Section 3.2: the key to the success of prompt-based image captioning lies in the way of generating specific prompts for each image, i.e., translating the visual information to the same textual semantic space of PLM. As showninFig.2c, to achieve universal controllable image captioning, we propose a prompt generation network (PGN) to learn combinatorial prompts, which consists of the information from both input image and all different control signals. In this manner, we hope this combinatorial prompt can guide the generation of PLM to describe the image content while the sentences meet the content or structure constraints; Section 3.2.1: Since we mainly discuss the combined control signals from two types of constraints (i.e., content-/structure-related), we first show the combinatorial prompts here are composed of two kinds of sub-prompts. Specifically, we call the two sub prompts as content prompt pC …and structure prompt pS …where M and N are prompt lengths, and D is the hidden dimension same to word embedding for PLM). Wang teaches that this will allow for high-quality captions with enhanced user control (Section 3). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen with the features of above as taught by Wang so as to allow for high-quality captions with enhanced user control as presented by Wang.
Regarding claim 2, Chen, in view of Wang teaches the method of claim 1, Wang discloses wherein the model input comprises one or more variables (Fig. 3; Fig. 4; Section 3.2.2: shown in Fig. 3, given an input image I,PGN takes the region features R ={r1,...,rE} of the image and two sets of learnable variables, the content variables vC …as inputs. Specifically, R is a set of visual features of the corresponding regions in I extracted by an object detector).
Regarding claim 3, Chen, in view of Wang teaches the method of claim 2, Chen discloses wherein the one or more variables includes the indication of a level of specificity for the caption (Paragraph [0171]: a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on; Paragraph [0177]: system can receive multiple prompts for captioning the observation (step 424). Each of the multiple prompts can include a distinct request to generate a caption (e.g., a text description) of the observation. For example, the multiple prompts can include requests to generate captions for the observation, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on).
Regarding claim 4, Chen, in view of Wang teaches the method of claim 1, Wang discloses wherein the indication of a level of specificity comprises a category selected from a set of categories defining a hierarchy (Fig. 7; Section 4.4.1: illustrated the generated captions from our ComPro on both single and combined control signals in Fig.7. We can observe that ComPro can describe each region set by orders(i.e., content control),or generate captions inside specific length ranges (i.e., structure control). Particularly, when facing complex combined controls, our ComPro can generate more details to meet the length constraint and keep the description orders simultaneously; Section 4.4.2: Signal-specific Settings Specifically, the verb and each entity are grounded with a set of region features. The VSR CIC framework (Chen et al., 2021) further fed the initial region set sequence into the semantic structure planner (SSP) to learn a structured region set sequence, i.e., predicting the order of the verb and roles, and ranking all sub-roles within the same semantic roles. The structured region set sequence is then fed into the captioning model to generate the final caption).
Regarding claim 5, Chen, in view of Wang teaches the method of claim 1, Chen discloses wherein the indication of a level of specificity is based upon one or more captioning dimensions (Paragraph [0171]: described above, a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on).
Regarding claim 6, Chen, in view of Wang teaches the method of claim 5, Chen discloses wherein the one or more captioning dimensions comprises one or more of the following: caption length, attribute specificity, quantification, location, scale, objectivity, in-image text, and image style (Paragraph [0171]: described above, a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on).
Regarding claim 7, Chen, in view of Wang teaches the method of claim 6, Chen discloses wherein the method further comprises: determining a value for one or more of the captioning dimensions (Paragraph [0154]: system can determine whether the pre-training is complete (step 330). The system can use any of a variety of criteria to determine whether the training is complete. For example, the system can determine that pre-training is complete after a pre-determined number of training iterations. As another example, the system can determine that pre-training is complete when a value of the pre-training objective function falls below a pre-determined threshold. As another example, the system can determine that pre-training is complete when a difference between values of the pre-training objective function for the current training iteration and a previous training iteration falls below a pre-determined threshold).
Regarding claim 8, Chen, in view of Wang teaches the method of claim 7, Wang discloses wherein determining a value for one or more of the captioning dimensions comprises: obtaining a probability distribution over the range of possible values for each captioning dimension of the one or more captioning dimensions (Section 4.2.2: For the content-related control, for both two datasets (i.e., COCO/Flickr30K Entities), we followed the same process as Cornia et al. (2019) to extract the region sequence for each image and its associated caption. For the structure-related control, for the COCO Entities, we used the same 4-level setting as Deng et al. (2020), i.e., the length ranges are [1,9], [10,14], [15,19], and [20,25]. For the Flickr30K Entities, according to the distribution of caption length, we used a similar 4-level setting, and the corresponding length ranges are [1,8], [9,11], [12,14], and [15,25]. Following the conventions of prior CIC works); and selecting a value for each captioning dimension based upon sampling from the corresponding probability distribution (Fig. 2; Section 3.1.2: as shown in Fig. 2b, they first use a mapping network (MN) to map the input image into a prompt P, which con sists of a set of continuous vectors, and each vector has the same dimension as the word embeddings for PLM. This prompt acts as a sequence of prepended word embeddings contained with visual information of the image to steer the generation of PLM. Then, the PLM can predict the caption tokens starting from the prompt, and model the following probability sequentially to generate the whole caption).
Regarding claim 9, Chen, in view of Wang teaches the method of claim 7, Wang discloses wherein each captioning dimension is associated with a set of sub-prompts and wherein determining a value for a captioning dimension comprises selecting a sub-prompt from the associated set of sub-prompts (Section 3.2.1: Since we mainly discuss the combined control signals from two types of constraints (i.e., content-/structure-related), we first show the combinatorial prompts here are composed of two kinds of sub-prompts. Specifically, we call the two sub prompts as content prompt pC …and structure prompt pS …where M and N are prompt lengths, and D is the hidden dimension same to word embedding for PLM).
Regarding claim 10, Chen, in view of Wang teaches the method of claim 9, Wang discloses wherein the indication of a level of specificity is based upon a combination of the selected sub-prompts (Fig. 8; Section 4.4.2: Similar to the generation process of the combined control shown in Sect.3.3.3, ComPro can be easily extended to tense control by generating and concatenating more new sub-prompts. As shown in Fig.8, following the same region feature selection and content variable expansion process, we only need to expand more structure variables (i.e., 9 in total, 4 for length level and 5 for tense type) and select the corresponding structure variable for concatenation to generate the combinatorial prompt).
Regarding claim 11, Chen, in view of Wang teaches the method of claim 2, Chen discloses wherein the method further comprises: identifying, by the image captioning machine learning model, the entities depicted in the image and their relationships (Paragraph [0072]: the output prediction 206 can include predicted classifications for the vehicle or for the driving environment of the vehicle. For example, the output prediction 206 can include predicted classifications for a state of the driving environment of the vehicle (e.g., classifications of whether the driving environment is safe, unsafe, obstructed, flooded, etc.), for states of regions of the driving environment of the vehicle (e.g., classifications of whether the regions are safe to enter, unsafe to enter, obstructed, flooded, etc.), for a state of the vehicle (e.g., classifications of whether the vehicle is operating safely, operating unsafely, damaged, operating unexpectedly, etc.), for states of other agents (e.g., vehicles, pedestrians, objects) in the driving environment of the vehicle (e.g., classifications of types of the agents, whether the agents are damaged, whether the agents are moving, whether the agents are merging, etc.), and so on; Paragraph [0131]: the training data 134 can include example region proposals for the example observations 302. As described in more detail above with reference to FIG. 2A and FIG. 2B, the observation processing system 120 can process the example observations 302 and the example region proposals to generate the observation embeddings 304 as region embeddings that representing spatial regions of the example observations 302 specified by the region proposals. For example, the region proposals can specify bounding boxes for detected objects within the example observations 302 (e.g., for vehicles, pedestrians, obstacles, etc. detected by performing object detection for the example observations 302). As another example, the region proposals can specify areas of the example observations 302 (e.g., non-rectangular spatial regions, irregular spatial regions, etc. generated by performing segmentation of the example observations 302) associated with, e.g., roadways, lanes, intersections, entrances, exits, vehicles, objects, pedestrians, and so on within example observations 302); and wherein the one or more variables are based upon the identified entities and their relationships (Paragraphs [0166]-[0168]: training data 134 includes a region proposal specifying a spatial region of the example observation 302-A, the training data 134 can include a corresponding region proposal specifying a spatial region of the example observation 302-B. The reference encoding system 402 can generate the observation embedding 202-B as a region embedding representing the spatial region of the example observation 302-B specified by the corresponding region proposal and the caption system 312-B can generate the example caption 308 to describe the spatial region of the example observation 302-B specified by the corresponding region proposal… the caption system 312-B can generate multiple example captions 308 for each example observation 302-B. For example, the input query 206 can include multiple different prompts for the caption system 312-B and the caption system 312-B can generate an example caption 308 for each of the multiple prompts. Generating multiple example captions 308 for the example observation 302-B is described in more detail below with reference to FIG. 4B).
Regarding claim 12, Chen, in view of Wang teaches the method of claim 11, Chen discloses wherein the one or more variables are further based upon one or more of the following: synonyms for an entity, attributes of an entity, position of an entity, or actions being performed by an entity (Paragraph [0131]: the training data 134 can include example region proposals for the example observations 302. As described in more detail above with reference to FIG. 2A and FIG. 2B, the observation processing system 120 can process the example observations 302 and the example region proposals to generate the observation embeddings 304 as region embeddings that representing spatial regions of the example observations 302 specified by the region proposals. For example, the region proposals can specify bounding boxes for detected objects within the example observations 302 (e.g., for vehicles, pedestrians, obstacles, etc. detected by performing object detection for the example observations 302). As another example, the region proposals can specify areas of the example observations 302 (e.g., non-rectangular spatial regions, irregular spatial regions, etc. generated by performing segmentation of the example observations 302) associated with, e.g., roadways, lanes, intersections, entrances, exits, vehicles, objects, pedestrians, and so on within example observations 302).
Regarding claim 13, Chen, in view of Wang teaches the method of claim 12, Wang discloses wherein the indication of a level of specificity comprises an indication of a mixture of a plurality of variables (Fig. 3; Fig. 4; Fig. 8; Section 3.2.2: shown in Fig. 3, given an input image I,PGN takes the region features R ={r1,...,rE} of the image and two sets of learnable variables, the content variables vC …as inputs. Specifically, R is a set of visual features of the corresponding regions in I extracted by an object detector; Section 4.4.2: Similar to the generation process of the combined control shown in Sect.3.3.3, ComPro can be easily extended to tense control by generating and concatenating more new sub-prompts. As shown in Fig.8, following the same region feature selection and content variable expansion process, we only need to expand more structure variables (i.e., 9 in total, 4 for length level and 5 for tense type) and select the corresponding structure variable for concatenation to generate the combinatorial prompt).
Regarding claim 14, Chen, in view of Wang teaches the method of claim 1, Wang discloses wherein an image of the plurality of images is associated with an existing image caption (Section 3.1.1: Benefiting from the large amounts of text data, Pretrained autoregressive Language Models(PLMs) have demonstrated a remarkable language generation ability to generate rich and diverse texts. As shown in Fig.2a, the PLM predicts one token at each time step conditioned on the previous tokens to generate a whole sentence y); wherein the processing to generate a caption for the image is further based upon the existing image caption (Section 3.1.1: they model the following word probability sequentially; Section 3.1.2: Inspired by the advantages of sentence generation via PLM, some recent image captioning works (Mokady et al., 2021) also try to generate captions in a similar manner. Specifically, as shown in Fig. 2b, they first use a mapping network (MN) to map the input image into a prompt P, which con sists of a set of continuous vectors, and each vector has the same dimension as the word embeddings for PLM. This prompt acts as a sequence of prepended word embeddings contained with visual information of the image to steer the generation of PLM. Then, the PLM can predict the caption tokens starting from the prompt, and model the following probability sequentially to generate the whole caption); and wherein the generated caption is different to the existing image caption (Fig. 3; Section 3.2.1: Since we mainly discuss the combined control signals from two types of constraints (i.e., content-/structure-related), we first show the combinatorial prompts here are composed of two kinds of sub-prompts. Specifically, we call the two sub prompts as content prompt pC …and structure prompt pS …where M and N are prompt lengths, and D is the hidden dimension same to word embedding for PLM).
Regarding claim 15, Chen, in view of Wang teaches the method of claim 1, Chen discloses wherein the image captioning machine learning model is based upon a multi-modal large language model (LLM) type generative machine learning model (Paragraph [0064]: embedding neural network can be a neural network that has been trained (e.g., pre-trained) to perform a different processing task before being trained to generate observation embeddings for the particular sensor modality. For example, the embedding neural network can be a vision encoding neural network for, e.g., a language model, a vision language model, and so on that is further trained (e.g., following the process 320 of FIG. 3B) to generate observation embeddings for the particular sensor modality. As another example, the embedding neural network can be a distillation of a vision encoding neural network for, e.g., a language model, a vision language model, and so on, that is further trained (e.g., following the process 320 of FIG. 3B) to generate observation embeddings for the particular sensor modality).
Regarding claim 16, Chen, in view of Wang teaches the method of claim 1, Wang discloses wherein the input prompt comprises an indication of a plurality of levels of specificity (Fig. 2; Section 3.2: the key to the success of prompt-based image captioning lies in the way of generating specific prompts for each image, i.e., translating the visual information to the same textual semantic space of PLM. As showninFig.2c, to achieve universal controllable image captioning, we propose a prompt generation network (PGN) to learn combinatorial prompts, which consists of the information from both input image and all different control signals. In this manner, we hope this combinatorial prompt can guide the generation of PLM to describe the image content while the sentences meet the content or structure constraints; Section 3.2.1: Since we mainly discuss the combined control signals from two types of constraints (i.e., content-/structure-related), we first show the combinatorial prompts here are composed of two kinds of sub-prompts. Specifically, we call the two sub prompts as content prompt pC …and structure prompt pS …where M and N are prompt lengths, and D is the hidden dimension same to word embedding for PLM); and wherein one or more captions are generated for each of the levels of specificity of the plurality of levels of specificity (Fig. 8; Section 4.4.2: Similar to the generation process of the combined control shown in Sect.3.3.3, ComPro can be easily extended to tense control by generating and concatenating more new sub-prompts. As shown in Fig.8, following the same region feature selection and content variable expansion process, we only need to expand more structure variables (i.e., 9 in total, 4 for length level and 5 for tense type) and select the corresponding structure variable for concatenation to generate the combinatorial prompt).
Regarding claim 17, Chen, in view of Wang teaches the method of claim 1, Chen discloses wherein the method further comprises: training an image generation machine learning model using the generated captions (Paragraph [0160]: as part of training the observation encoding system 120, the system can process the example observation 302-A using a caption system 312-A configured to generate an output caption 310 describing the example observation 302-A. The caption system 312-A can be configured to process an input query 206 that characterizes a request to generate the output caption 310. As an example, the caption system 312-A can be a language model (e.g., a visual language model) configured to process an input token sequence that includes the input query 206 to generate an output token sequence representing the output caption 310. As a further example, the language model can be a Transformer model (e.g., a vision transformer model) configured to generate the output token sequence representing the output caption 310 by performing a sequence of attention operations to process the input token sequence. The caption system 312-A can have any appropriate architecture for conditionally generating the output caption 310 as conditioned on the observation embedding 202-A. As one example, the caption system 312-A can be configured to process an input token sequence that includes the observation embedding 202-A. As another example, the caption system 312-A can include one or more cross-attention layers that can perform cross-attention operations using the observation embedding 202-A to generate the output caption 310. In some implementations, the caption system 312-A can be trained (e.g., pre-trained) to generate captions for observations of image data by processing observation embeddings for the observations of image data).
Regarding claim 18, Chen discloses a system comprising: one or more data processing apparatus; and a memory storing instructions that when executed by the one or more data processing apparatus cause the one or more data processing apparatus to carry out a method (Paragraph [0231]: computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus) comprising: obtaining a plurality of images for captioning (Paragraph [0048]: generating predictions on-board an autonomous vehicle, more generally, the described techniques can be implemented on any system of one or more computers that receives images of scenes in an environment. That is, once the training system 130 has trained the observation encoding system 120, the observation encoding system 120 can be used by any system of one or more computers); generating, by an image captioning machine learning model, one or more captions for each image of the plurality of images (Paragraph [0171]: a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on; Paragraph [0178]: system can process the observation embedding with each of the received prompts using a caption system to generate a respective caption for the observation. (step 426). For example, the caption system can be a language model (e.g., a vision language model) configured to process input token sequences that include input prompts to generate output token sequences characterizing captions for the observation generated in accordance with the input prompts. As a further example, the language model can be a Transformer model (e.g., a vision transformer model) configured to generate the output token sequences representing the captions by performing sequences of attention operations to process the input token sequences. The caption system can have any appropriate architecture for conditionally generating captions for the observation as conditioned on the observation embedding. As one example, the caption system can be configured to process an input token sequence that includes the received observation embedding. As another example, the caption system can include one or more cross-attention layers that can perform cross-attention operations using the received observation embedding to generate the captions for the observation. In some implementations, the caption system can be trained (e.g., pre-trained) to generate captions for observations of image data by processing observation embeddings for the observations of image data; Paragraph [0184]: described in more detail above with reference to FIG. 2A and FIG. 2B, each example input query can include example task embeddings for the prediction task characterized by the input query. For example, the example task embeddings can be machine-learned parameters (e.g., machine learned vectors). As another example the example task embeddings can be generated by processing corresponding text prompts using a text embedding neural network (e.g., using a language model); wherein generating a caption for an image comprises: identifying, by the image captioning machine learning model, one or more entities depicted in the image and one or more properties of the one or more entities (Paragraph [0072]: the output prediction 206 can include predicted classifications for the vehicle or for the driving environment of the vehicle. For example, the output prediction 206 can include predicted classifications for a state of the driving environment of the vehicle (e.g., classifications of whether the driving environment is safe, unsafe, obstructed, flooded, etc.), for states of regions of the driving environment of the vehicle (e.g., classifications of whether the regions are safe to enter, unsafe to enter, obstructed, flooded, etc.), for a state of the vehicle (e.g., classifications of whether the vehicle is operating safely, operating unsafely, damaged, operating unexpectedly, etc.), for states of other agents (e.g., vehicles, pedestrians, objects) in the driving environment of the vehicle (e.g., classifications of types of the agents, whether the agents are damaged, whether the agents are moving, whether the agents are merging, etc.), and so on; Paragraph [0131]: the training data 134 can include example region proposals for the example observations 302. As described in more detail above with reference to FIG. 2A and FIG. 2B, the observation processing system 120 can process the example observations 302 and the example region proposals to generate the observation embeddings 304 as region embeddings that representing spatial regions of the example observations 302 specified by the region proposals. For example, the region proposals can specify bounding boxes for detected objects within the example observations 302 (e.g., for vehicles, pedestrians, obstacles, etc. detected by performing object detection for the example observations 302). As another example, the region proposals can specify areas of the example observations 302 (e.g., non-rectangular spatial regions, irregular spatial regions, etc. generated by performing segmentation of the example observations 302) associated with, e.g., roadways, lanes, intersections, entrances, exits, vehicles, objects, pedestrians, and so on within example observations 302); and processing, by the image captioning machine learning model, a model input based upon the image, the identified one or more entities and properties, and an indication of a level of specificity to generate the caption for the image (Paragraph [0171]: a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on; Paragraph [0177]: system can receive multiple prompts for captioning the observation (step 424). Each of the multiple prompts can include a distinct request to generate a caption (e.g., a text description) of the observation. For example, the multiple prompts can include requests to generate captions for the observation, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on).
Chen does not explicitly disclose wherein the generated captions for the plurality of images have mixed levels of specificities.
However, Wang teaches captioning images based on prompts (Abstract), wherein the generated captions for the plurality of images have mixed levels of specificities (Fig. 2; Section 3.2: the key to the success of prompt-based image captioning lies in the way of generating specific prompts for each image, i.e., translating the visual information to the same textual semantic space of PLM. As showninFig.2c, to achieve universal controllable image captioning, we propose a prompt generation network (PGN) to learn combinatorial prompts, which consists of the information from both input image and all different control signals. In this manner, we hope this combinatorial prompt can guide the generation of PLM to describe the image content while the sentences meet the content or structure constraints; Section 3.2.1: Since we mainly discuss the combined control signals from two types of constraints (i.e., content-/structure-related), we first show the combinatorial prompts here are composed of two kinds of sub-prompts. Specifically, we call the two sub prompts as content prompt pC …and structure prompt pS …where M and N are prompt lengths, and D is the hidden dimension same to word embedding for PLM). Wang teaches that this will allow for high-quality captions with enhanced user control (Section 3). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen with the features of above as taught by Wang so as to allow for high-quality captions with enhanced user control as presented by Wang.
Regarding claim 19, Chen discloses a non-transitory computer-readable storage medium comprising instructions that when executed by one or more data processing apparatus cause the one or more data processing apparatus to carry out a method (Paragraph [0231]: computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus) comprising: obtaining a plurality of images for captioning (Paragraph [0048]: generating predictions on-board an autonomous vehicle, more generally, the described techniques can be implemented on any system of one or more computers that receives images of scenes in an environment. That is, once the training system 130 has trained the observation encoding system 120, the observation encoding system 120 can be used by any system of one or more computers); generating, by an image captioning machine learning model, one or more captions for each image of the plurality of images (Paragraph [0171]: a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on; Paragraph [0178]: system can process the observation embedding with each of the received prompts using a caption system to generate a respective caption for the observation. (step 426). For example, the caption system can be a language model (e.g., a vision language model) configured to process input token sequences that include input prompts to generate output token sequences characterizing captions for the observation generated in accordance with the input prompts. As a further example, the language model can be a Transformer model (e.g., a vision transformer model) configured to generate the output token sequences representing the captions by performing sequences of attention operations to process the input token sequences. The caption system can have any appropriate architecture for conditionally generating captions for the observation as conditioned on the observation embedding. As one example, the caption system can be configured to process an input token sequence that includes the received observation embedding. As another example, the caption system can include one or more cross-attention layers that can perform cross-attention operations using the received observation embedding to generate the captions for the observation. In some implementations, the caption system can be trained (e.g., pre-trained) to generate captions for observations of image data by processing observation embeddings for the observations of image data; Paragraph [0184]: described in more detail above with reference to FIG. 2A and FIG. 2B, each example input query can include example task embeddings for the prediction task characterized by the input query. For example, the example task embeddings can be machine-learned parameters (e.g., machine learned vectors). As another example the example task embeddings can be generated by processing corresponding text prompts using a text embedding neural network (e.g., using a language model); wherein generating a caption for an image comprises: determining a value for one or more captioning dimensions (Paragraph [0154]: system can determine whether the pre-training is complete (step 330). The system can use any of a variety of criteria to determine whether the training is complete. For example, the system can determine that pre-training is complete after a pre-determined number of training iterations. As another example, the system can determine that pre-training is complete when a value of the pre-training objective function falls below a pre-determined threshold. As another example, the system can determine that pre-training is complete when a difference between values of the pre-training objective function for the current training iteration and a previous training iteration falls below a pre-determined threshold); generating an input prompt based upon the determined values to indicate the level of specificity of the caption to be generated (Paragraph [0171]: a caption system 312-B can process an observation embedding 202-B for an observation of sensor data and multiple captioning prompts 412 to generate multiple example captions 308 for the observation embedding 202-B. The multiple captioning prompts 412 can include requests to generate captions for the observation embedding 202-B, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on. For example, the captioning prompts 412 can include requests to describe the observation at different levels of detail, e.g., “Write a short description”, “Briefly describe”, “Write a detailed description”, “Fully describe”, and so on. As another example, the captioning prompts 412 can include requests to describe different aspects of the observation, e.g., “Describe the entire scene”, “Write a description with information about the different objects in the scene”, “Write a description with information about the different objects that are relevant to driving”, “Describe the scene with a focus on any driving hazards in the scene”, and so on; Paragraph [0177]: system can receive multiple prompts for captioning the observation (step 424). Each of the multiple prompts can include a distinct request to generate a caption (e.g., a text description) of the observation. For example, the multiple prompts can include requests to generate captions for the observation, e.g., with different levels of detail, with different captioning styles, focusing on different details of the observation, and so on); and processing, by the image captioning machine learning model, the image and the input prompt to generate the caption for the image (Paragraph [0178]: system can process the observation embedding with each of the received prompts using a caption system to generate a respective caption for the observation. (step 426). For example, the caption system can be a language model (e.g., a vision language model) configured to process input token sequences that include input prompts to generate output token sequences characterizing captions for the observation generated in accordance with the input prompts. As a further example, the language model can be a Transformer model (e.g., a vision transformer model) configured to generate the output token sequences representing the captions by performing sequences of attention operations to process the input token sequences. The caption system can have any appropriate architecture for conditionally generating captions for the observation as conditioned on the observation embedding. As one example, the caption system can be configured to process an input token sequence that includes the received observation embedding. As another example, the caption system can include one or more cross-attention layers that can perform cross-attention operations using the received observation embedding to generate the captions for the observation. In some implementations, the caption system can be trained (e.g., pre-trained) to generate captions for observations of image data by processing observation embeddings for the observations of image data).
Chen does not explicitly disclose wherein the generated captions for the plurality of images have mixed levels of specificities.
However, Wang teaches captioning images based on prompts (Abstract), wherein the generated captions for the plurality of images have mixed levels of specificities (Fig. 2; Section 3.2: the key to the success of prompt-based image captioning lies in the way of generating specific prompts for each image, i.e., translating the visual information to the same textual semantic space of PLM. As showninFig.2c, to achieve universal controllable image captioning, we propose a prompt generation network (PGN) to learn combinatorial prompts, which consists of the information from both input image and all different control signals. In this manner, we hope this combinatorial prompt can guide the generation of PLM to describe the image content while the sentences meet the content or structure constraints; Section 3.2.1: Since we mainly discuss the combined control signals from two types of constraints (i.e., content-/structure-related), we first show the combinatorial prompts here are composed of two kinds of sub-prompts. Specifically, we call the two sub prompts as content prompt pC …and structure prompt pS …where M and N are prompt lengths, and D is the hidden dimension same to word embedding for PLM). Wang teaches that this will allow for high-quality captions with enhanced user control (Section 3). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen with the features of above as taught by Wang so as to allow for high-quality captions with enhanced user control as presented by Wang.
Regarding claim 20, Chen, in view of Wang teaches the non-transitory computer-readable storage medium of claim 19, Wang discloses wherein each captioning dimension is associated with a set of sub-prompts and wherein determining a value for a captioning dimension comprises selecting a sub-prompt from the associated set of sub- prompts (Section 3.2.1: Since we mainly discuss the combined control signals from two types of constraints (i.e., content-/structure-related), we first show the combinatorial prompts here are composed of two kinds of sub-prompts. Specifically, we call the two sub prompts as content prompt pC …and structure prompt pS …where M and N are prompt lengths, and D is the hidden dimension same to word embedding for PLM).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights