Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 14, 18-20, 23, are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Luddecke (Image Segmentation Using Text and Image Prompts).
Regarding claim 18, 1, 23,
Luddecke teaches:
An electronic device comprising: at least one processing unit; and at least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, (Luddecke Appendix “Throughout our experiments we use PyTorch [52] with CLIP ViT-B/16 [8]. We train on PhraseCut [20] for 20,000 iterations on batches of size 64 with an initial learning rate of 0.001 (for VitSeg 0.0001) which decays following a co- sine learning rate schedule to 0.0001 (without warmup).” Note: Luddecke teaches that its model was trained by the research team, specifying a 20,000 training cycles with a batch size of 64. To perform this, and any similar machine learning training, a processor with memory coupled to it is implicitly required, thus a processor and memory are taught. ) the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method of processing a visual task by a generic processing model, (Luddecke Abstract “This approach enables us to create a unified model (trained once) for three common segmentation tasks, which come with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation. We build upon the CLIP model as a backbone which we ex- tend with a transformer-based decoder that enables dense prediction. After training on an extended version of the PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on an additional image expressing the query. We analyze different variants of the latter image-based prompts in detail. This novel hybrid input allows for dynamic adaptation not only to the three segmentation tasks mentioned above, but to any binary segmentation task where a text or image query can be formulated.” Note: A generic processing model is defined in the specifications as ¶32 “According to an example implementation of the present disclosure, specific types of visual tasks are not distinguished, but a generic processing model 220 is developed for various types of visual tasks 230. According to an example implementation of the present disclosure, the visual task 230 can be any visual task described above, including but not limited to: object detection, instance segmentation, multi-object tracking, multi-object tracking and segmentation, video instance segmentation, referring expression compression, referring expression segmentation, referring video object segmentation, single object tracking, video object segmentation, and the like.” Luddecke teaches the described concept of a generic processing model that can perform a number of generic visual tasks such as referring expression segmentation, mentioned specifically by the specs, and other types of segmentation types from multiple types of inputs such as zero-shot segmentation.) the method comprising: receiving visual data and prompt data associated with the visual task, the visual task specifying that a processing result associated with the prompt data is to be determined from the visual data; (Luddecke Abstract “Here we propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image” 3 “Our model receives information about the segmentation target (“what to segment?”) through a conditional vector. This can be provided either by text or an image (through visual prompt engineering). Since CLIP uses a shared embedding space for images and text captions, we can interpolate between both in the embedding space and condition on the interpolated vector.”
PNG
media_image1.png
576
1530
media_image1.png
Greyscale
Note: Luddecke teaches that visual data, the “query image” seen above in Fig. 2, is received by the model. The visual data is an input image that the model is to segment, the prompt data specifies what should be segmented. Luddecke teaches the prompt can be text data or image data. The model creates a CLIP embedding based on the prompt in order to segment the same thing described in the prompt.) obtaining a generic prompt representation of the prompt data, (Luddecke 3 “Our model receives information about the segmentation target (“what to segment?”) through a conditional vector. This can be provided either by text or an image (through visual prompt engineering) …Formally, let si be the embedding of the support image and ti the text embedding of a sample i ” Note: The specifications define the “generic prompt representation” as ¶42 “Here, the prompt encoder can transform the original multi-modality (image-related prompts and language-related prompts) prompts into a unified form. Specifically, if it is determined that the format of the prompt data 214 is a language expression format, the language expression encoder can be used to extract the prompt representation 224. To process language-related prompts, language encoders (e.g., EncL encoders) that are currently known and/or will be developed in the future can be used” The specifications state an example of a generic prompt representation is the text prompt that has been encoded by a language encoder. This is taught by Luddecke which teaches that both text and image prompts are embedded by CLIP, or encoded. This can also be seen in Fig. 2 where the prompt data will be fed to its respective text or visual CLIP transformer for encoding.) the prompt data including either an image format or a language expression format;(Luddecke Abstract, 3, and Fig. 2, cited above all teach that a language expression format, or a text description, and an image are both accepted prompt types.) obtaining a generic visual representation of the visual data,(Luddecke 3 “While the query image ( RW × H × 3) is passed through the CLIP visual transformer, activations at certain layers S are read out and projected to the token embedding size D of our decoder. Then, these extracted activations (including CLS token)”Note: The specifications state ¶39 “As shown in FIG. 4, the visual data 212 and the prompt data 214 may be received. A generic visual representation 222 (e.g., represented by the symbol F,) of the visual data 212 may be determined using an image-based encoder that is currently known and/or to be developed in the future. Further, a generic prompt representation 224 of the prompt data 214 may be determined using a prompt encoder (e.g., represented by the symbol F, ).” The specifications teach that the input image is encoded to make the generic visual representation, this same concept is taught by Luddecke which teaches the CLIP visual transformer will embed, or encode, the visual representation.) the visual data including either an image format or a video format;(Ludeke Abstract, 3, and Fig, 2, cited above, teach the visual data the model will perform segmentation on is an image.) and determining the processing result based on the generic prompt representation and the generic visual representation.(Fig. 2, cited above, teaches that the output processing result, which is a segmentation of the input image, is produced after the input image and prompt are embedded, or encoded, by their respective CLIP transformer.)
Regarding claim 19, 2,
Luddecke teaches:
The device of claim 18, wherein obtaining the generic prompt representation comprises: extracting the generic prompt representation based on a format of the prompt data. (
PNG
media_image1.png
576
1530
media_image1.png
Greyscale
Luddecke 3 “Our model receives information about the segmentation target (“what to segment?”) through a conditional vector. This can be provided either by text or an image (through visual prompt engineering) …Formally, let si be the embedding of the support image and ti the text embedding of a sample i ” Note: Luddecke teaches that the input prompt can be either a text or an image, and once provided will be handled by a visual or text transformer depending on the input type, teaching that the format or type of the prompt data determines how the generic prompt representation will be “extracted”, in this case via embedding.)
Regarding claim 20, 3,
The device of claim 19, wherein extracting the generic prompt representation comprises at least one of: in response to determining that the format of the prompt data is a language expression format, extracting the prompt representation using a language expression encoder;(Luddecke 3 “This conditional vector can be obtained in two ways: (1) Using the CLIP text-transformer embedding of a text query and (2) using the CLIP visual transformer on a feature engineered prompt image. CLIP itself is not trained, but only used as a frozen feature extractor. Due to the compact de- coder, CLIPSeg has only 1,122,305 trainable parameters for D = 64.” Note: If the prompt data is a text query, or language expression, the model will employ a specific encoder for language expression, in this case a CLIP text-transformer which will produce an encoding.) in response to determining that the format of the prompt data is an image format, extracting the generic prompt representation based on an extended image including an annotation image specified by the prompt data. (Luddecke 4 “Instead of applying the mask inside the model, we can also combine mask and image to a new image, which can then processed by the visual transformer … We find that the exact form of how the mask and image are combined matters a lot. Generally, we identify three image operations that improve the alignment between the object text prompts and the images: decreasing the background brightness, blurring the background (using a Gaussian filter) and cropping to the object. The combination of all three performs best (Tab. 2, last row). We will use this variant in the remainder.”
PNG
media_image2.png
834
1050
media_image2.png
Greyscale
PNG
media_image3.png
282
1388
media_image3.png
Greyscale
Note: The specifications define an extended image and annotation image as ¶45 “For example, the extended image 530 with 22 times (or other times) the annotation image 520 may be cropped centered on the annotation image 520 in the reference image.” ¶46 “Specifically, based on a plurality of pixel data in the extended image 530, a first representation associated with the prompt data (e.g., template portion 522 in FIG. 5) can be determined. In order to introduce more precise target information, additional channels (referred to as the target priors) can be added to the extended image to form a 4-channel combination feature 540. Here, the prior value of the pixel data can indicate whether the pixel data belongs to the annotation image. Specifically, the prior values of the pixels in the annotation image 520 are 1 and the prior values of other pixels can be set to 0. Alternatively, and/or additionally, the prior data can be set based on other methods. A second representation (e.g., prior data 524) associated with the prompt data can be determined based on the prior values of the plurality of pixel data in the extended image 530. Furthermore, a concatenation operation can be performed on the target portion 522 and the prior data 524, and the extended image 530 can be expanded from three channels to four channels. At this time, a four-channel combination feature 540 can be obtained.”
PNG
media_image4.png
528
728
media_image4.png
Greyscale
. The specifications clarify the pixels in the annotation image are 0 or 1 denoting whether or not they are on/near the target, thus it can be understood as a prior map/binary mask which provides denotes what the object of interest is. The extended image is clarified to be in some cases cropped and centered on the annotation image, but larger than the annotation image. This can be seen in Fig. 5 where the Annotation image denotes the item to segment, a zebra, and the extended image is cropped to be smaller and centered on the zebra, but is larger than just the object itself. This same idea can be seen in Luddecke which teaches that a mask that specifies the target object is used to make an “extended image”, which can be an image cropped around the target of interest. As seen in Luddecke Table 2 cropping of various sizes is done, some are closer to the target object and some are a “larger context”, comparable to the claims “extended image” as seen in Fig. 5 that shows a larger crop around the object. Examples of extended images, images that highlight the target object by applying an “annotation image” like a blur, outlines around the target, and cropping around the target. Luddecke also teaches that the extended image is what is input to the CLIP transformer which encodes it, making it into the generic prompt representation.)
Regarding claim 14,
The method of claim 1, wherein the visual task comprises at least any of: object detection, instance segmentation, multi-object tracking, multi-object tracking and segmentation, video instance segmentation, referring expression comprehension, referring expression segmentation, referring video object segmentation, single object tracking, video object segmentation. (Luddecke Abstract teaches multiple types of object segmentation including referring expression segmentation.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4, 21, are rejected under 35 U.S.C. 103 as being unpatentable over Luddecke (Image Segmentation Using Text and Image Prompts) in view of Sofiiuk (f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation).
Regarding claim 21, 4,
Luddecke teaches
The device of claim 20, wherein extracting the generic prompt representation based on the extended image comprises: determining a first representation associated with the prompt data based on a plurality of pixel data in the extended image; (Luddecke 3 “This conditional vector can be obtained in two ways: (1) Using the CLIP text-transformer embedding of a text query and (2) using the CLIP visual transformer on a feature engineered prompt image. CLIP itself is not trained, but only used as a frozen feature extractor. Due to the compact de- coder, CLIPSeg has only 1,122,305 trainable parameters for D = 64.” Note: Luddecke teaches that the “feature engineered prompt image” is encoded via the CLIP visual transformer. The feature engineered prompt image is the claims extended image, as shown previously in claim 3, and as seen in Fig. 3 cited previously. Thus, Luddecke teaches the determining a “first representation” based on the pixel data of an extended image associated.)
While Ludecke teaches that an extended image associated with the prompt data can be used to determine a first representation it is not taught that a second representation is determined or that a representation can be determined using prior values of the pixel data in the extended image. This is taught by Sofiiuk which teaches determining a first representation associated with the prompt data based on a plurality of pixel data in the extended image;(
PNG
media_image5.png
484
608
media_image5.png
Greyscale
Sofiiuk 4 “Previous works on interactive segmentation often used inference on image crops to achieve speed-up and preserve fine details in the segmentation mask. Cropping helps to infer the masks of small objects, but it also may degrade results in cases when an object of interest is too large to fit into one crop. In this work, we use an alternative technique (we call it Zoom-In), which is quite simple but improves both quality and speed of the interactive segmentation.” Note: Sofiiuk teaches a zoom in cropping technique for its original input image, or prompt image, where after a rough predicted mask is obtained a bounding box for the mask is made. This initial region can be seen in Fig. 3 above in teal. This box is then extended by a ratio of 1.4 to create the cropped extended image, analogous to the claims “extended image”.) determining a second representation associated with the prompt data based on prior values of the plurality of pixel data in the extended image, the prior value of the pixel data in the plurality of pixel data indicating whether the pixel data belongs to the annotation image;(Sofiiuk 1 “The main scenario considered in the papers is click-based segmentation when the user provides input in a form of positive and negative clicks.” 2 “They calculate distance maps from positive and negative clicks, stack them together with an input image and pass into a network that predicts an object mask. This approach was later used in most of the following works. Liew et al. [19] propose to combine local predictions on patches containing user clicks and thus refine network output”
PNG
media_image6.png
714
1526
media_image6.png
Greyscale
) and combining the first representation and the second representation to form the generic prompt representation.(Sofiiuk 5 “The model contains Distance Maps Fusion (DMF) block for adaptive fusion of RGB image and distance maps. It takes a concatenation of RGB image and 2 distance maps (one for positive clicks and one for negative clicks) as an input.”
PNG
media_image7.png
592
1246
media_image7.png
Greyscale
Note: Sofiiuk teaches that the input prompt image and the distance map image, analogous to the first and second representation, are fused or combined together and then provided to a Res-Net-50 model. The multiple Res-Net blocks perform feature extraction, in this case by encoding, on the fused first and second representation, where they are then decoded after by the DeepLab decoder.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Luddecke with Sofiiuk where a first representation and second representation are combined to form the generic prompt representation.
There are several reasons that would motivate one to do so, a prior map provides relevant info known about the image prior to it being processed, if such data is available for use the accuracy of the model could be increased by leveraging the prior map by combining a first representation with a second that contains the prior information.
Claims 5, 6, 7, 22, are rejected under 35 U.S.C. 103 as being unpatentable over Luddecke (Image Segmentation Using Text and Image Prompts) in view of Tan (LXMERT: Learning Cross-Modality Encoder Representations from Transformers).
Regarding claim 22, 5,
Luddecke teaches:
The device of claim 18, wherein determining the processing result comprises:
While Luddecke teaches obtaining a generic prompt representation and generic visual representation it is not taught that attention operations are performed on representations to update and leverage them to obtain the processing result. This is taught by Tan which teaches performing an attention operation on the generic prompt representation and the generic visual representation (Tan 1 “Each image is represented as a sequence of objects, and each sentence is represented as a sequence of words. Via careful design and combination of these self-attention and cross-attention layers, our model is able to generate language representations, image representations, and cross-modality representations from the inputs.”
PNG
media_image8.png
408
1050
media_image8.png
Greyscale
Note: Tan teaches that for a given text input, analogous to a text prompt representation/input text info, and a given visual representation both visual and prompt representations have an individual attention operation performed on them via a self-attention layer.) to update the generic prompt representation and the generic visual representation, respectively; and obtaining the processing result using the updated generic prompt representation and the updated generic visual representation. (Tan 2.2 “After the embedding layers, we first apply two transformer encoders (Vaswani et al., 2017), i.e., a language en coder and an object-relationship encoder, and each of them only focuses on a single modal ity (i.e., language or vision) … Each layer (left dashed blocks in Fig. 1) in a single modality encoder contains a self-attention (‘Self’) sub-layer and a feed-forward (‘FF’) sub-layer, where the feed-forward sub-layer is further com posed of two fully-connected sub-layers. We take NL and NR layers in the language encoder and the object-relationship encoder, respectively. We add a residual connection and layer normalization (annotated by the ‘+’ sign in Fig. 1) after each sub layer” Note: Tan teaches that once the text prompt and visual info are provided to the self attention layer the result is added back to the encoded representations, teaching that the visual and prompt representations are individually updated as a result of attention operations performed on them. This can be seen in Fig. 1, where it is also seen that the updated representations are then passed to another encoder for use in generating the final output processing result.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Luddecke with Tan where a processing result is created by a prompt representation and visual representation that have been updated after respective attention operations to the representations.
There are several reasons that would motivate one to do so, leveraging an attention operation to update the representations, referring to encodings of our input data, allows for the relevant parts of data to be brought out more and improve context understanding, qualities which could improve the model’s accuracy.
Regarding claim 6,
Luddecke teaches:
The method of claim 5, wherein updating the generic prompt representation comprises:
Luddecke does not teach an attention operation in a first direction that determines attention between a prompt and visual representation, this is described in Tan which teaches performing an attention operation in a first direction on the generic prompt representation and the generic visual representation to determine a first attention representation between the generic prompt representation and the generic visual representation; (
PNG
media_image8.png
408
1050
media_image8.png
Greyscale
Tan 2.2 “Each cross-modality layer (the right dashed block in Fig. 1) in the cross modality encoder consists of two self-attention sub-layers, one bi-directional cross-attention sub layer, and two feed-forward sub-layers. We stack (i.e., using the output of k-th layer as the input of (k+1)-th layer) NX these cross-modality layers in our encoder implementation. Inside the k-th layer, the bi-directional cross-attention sub-layer (‘Cross’) is first applied, which contains two uni-directional cross-attention sub-layers: one from language to vision and one from vision to language.” Note: Tan teaches a cross-modality layer which includes a layer that performs two attention operations in two directions, making the layer bi-directional. The bi-directional layer contains two single direction attention layers that each perform their attention operation in a different direction, one from text prompt to visual representation and vice versa. Thus, a “first” direction with a second direction of an attention operation between the visual and prompt representation is taught.) and updating the generic prompt representation with the first attention representation. (Tan 2.2 “The cross-attention sub-layer is used to exchange the information and align the entities between the two modalities in order to learn joint cross modality representations.” Note: As seen in Fig. 1 on the bottom half of the diagram the prompt representation, in this case a text prompt, has the output of one direction of the cross-attention layer)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Luddecke with Tan where an attention operation between a generic prompt and visual representation is performed in a first direction to update the generic prompt representation.
There are several reasons that would motivate one to do so, as the prompt and visual representations describe the same object(s) performing attention between the two to update the prompt allows the information present in the visual data allows the model to extract more relevant data from the prompt.
Regarding claim 7,
Luddecke teaches:
The method of claim 5, wherein updating the generic visual representation comprises:
Luddecke does not however teach performing an attention operation to update its representations. This is found in Tan which teaches performing an attention operation in a second direction on the generic prompt representation and the generic visual representation to determine a second attention representation between the generic prompt representation and the generic visual representation;(Tan Fig. 1, 2.2, cited previously in claim 6, teach a cross-attention sub-layer consisting of “two uni-directional cross-attention sub-layers: one from language to vision and one from vision to language.” Thus teaching a first and second direction of an attention operation, one from the visual representation to the prompt and one from the prompt to the visual representation, in this case the prompt to visual representation is designated as the “second” ) and updating the generic visual representation with the second attention representation(Tan Fig. 1 teaches that each half of its flow chart, one for the text prompt one for the visual representation, are updated from the other’s attention operation in the cross attention layer as seen from the concatenation operation in Fig. 1 that adds the “second” attention direction from the prompt to the visual representation.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Luddecke with Tan where an attention operation between a generic prompt and visual representation is performed in a first direction to update the generic visual representation.
There are several reasons that would motivate one to do so, as the prompt and visual representations describe the same object(s) performing attention between the two to update the visual representation allows the information present in the visual data allows the model to extract more relevant data from the prompt.
Claims 8, 10 are rejected under 35 U.S.C. 103 as being unpatentable over Luddecke (Image Segmentation Using Text and Image Prompts) in view of Tan (LXMERT:Learning Cross-Modality Encoder Representations from Transformers) in view of Kamath (MDETR-Modulated Detection for End-to-End Multi-Modal Understanding).
Regarding claim 8,
Luddecke teaches:
The method of claim 5, wherein obtaining the processing result comprises: the updated generic prompt representation and the updated generic visual representation
While an updated generic prompt representation and visual representation have been shown to be taught, querying a plurality of candidate query results to determine scores for weights based on prompt data is found in Kamath which teaches querying a plurality of candidate query results corresponding to the prompt data in the image data based on the generic prompt representation and the generic visual representation;(Kamath 1 “We encode the text using a pre-trained transformer language model to produce a sequence of hidden vectors of same size as the input. We then apply a modality dependent linear projection to both the image and text features to project them into a shared embedding space. These feature vectors are then concatenated on the sequence dimension to yield a single sequence of image and text features. This sequence is fed to a joint transformer encoder termed as the cross encoder. Following DETR, we apply a transformer decoder on the object queries while cross attending to the final hidden state of the cross encoder. The decoder’s output is used for predicting the actual boxes.” 2.1 “The DETR encoder operates on 2D flattened image features from the backbone and applies a series of transformer layers. The decoder takes as input a set of N learned embeddings called object queries, that can be viewed as slots that the model needs to fill with detected objects. All the object queries are fed in parallel to the decoder, which uses cross-attention layers to look at the encoded image and predicts the output embeddings for each of the queries. The final representation of each object query is independently decoded into box coordinates and class labels using a shared feed-forward layer. The number of object queries acts as a de facto upper-bound on the number of objects the model can detect simultaneously.” Note: Kamath teaches that a DETR encoder accepts multiple “object queries”, DETR will leverage the text prompt and the input image to fill these object queries with actual predictions on objects present from both the text and image content. These are the claims “plurality of candidate queries” which are themselves queried. The querying of the candidate queries is seen in Kamath 2.1 which states that a cross attention operation on the object queries, which as an attention operation requires querying, is performed to produce the final predictions on objects present, or final processing result.) determining scores of the plurality of candidate query results respectively based on the plurality of candidate query results and a weight associated with the prompt data;(
PNG
media_image9.png
410
1258
media_image9.png
Greyscale
Kamath 2.2.1 “We encode the text using a pre-trained transformer language model to produce a sequence of hidden vectors of same size as the input. We then apply a modality dependent linear projection to both the image and text features to project them into a shared embedding space. These feature vectors are then concatenated on the sequence dimension to yield a single sequence of image and text features” 2.2.2 “For modulated detection, unlike in the standard detection setting, we are not interested in predicting a categorical class for each detected object. Instead, we predict the span of tokens from the original text that refers to each matched object. Concretely, we first set the maximum number of tokens for any given sentence to be L=256. For each predicted box that is matched to a ground truth box using the bi-partite matching, the model is trained to predict a uniform distribution over all token positions that correspond to the object” Kamath A “In the example depicted in Fig 6: to rank the boxes for the phrase “a cat”, we use the probability mass that each query assigns to the positions that correspond to “a cat” in the sentence “a cat with white paws jumps over a fence in front of a yellow tree” (in this example, the first few tokens). Through this approach, the red box is found to be the highest-ranked box.” Note: Fig. 6 shows multiple candidate queries in the form of colored boxes denoting identified objects, these candidate queries have a score assigned to them detailing how closely they define a respective “weight associated with the prompt data”. The specifications define the weight as ¶63 “Specifically, given by prompt embeddings Fp, for tasks that take categories as prompts, the embedding of each category name can be used as a weight matrix”. Thus, the different embeddings Kamath makes for different objects in the text and visual data are the “weights”. The weights are the embedding describing text and image features the model obtains; it will then obtain a score describing how accurately each candidate query corresponds to the given weight, as seen in Fig. 6.) and determining the processing result based on the scores of the plurality of candidate query results. (Kamath A, cited above, teaches that depending on what the object of interest is the candidate queries can be ranked by how high their score is respective to the object described by the prompt, thus the highest scoring box is the processing result indicating where the desired object is in the image.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Luddecke where candidate query results based on updated generic prompt and visual representation are obtained and have a score assigned to weights based on the prompt data.
There are several reasons that would motivate one to do so, in a task such as object segmentation or object detection where multiple objects to handle may be present in both the visual and prompt data having the model produce multiple candidate query results and scoring them based on weights representing different objects in the image provides an efficient means to determine which candidate query aligns best with a desired object.
Regarding claim 10,
Luddecke teaches:
The method of claim 5, further comprising performing multi-stage training on the generic processing model, the multi-stage training comprising:
Luddecke does not however teach a first-stage of training based on an object detection dataset which performs object detection, this is found in Kamath which teaches performing first-stage training based on an object detection dataset such that the generic processing model describes an association between image data in the object detection dataset and a bounding box of an object in the image data. (Kamath 4 “We conduct our experiments on the LVIS dataset [11], a detection dataset with a large vocabulary of 1.2k categories, with a long-tail that contains very few training samples, making it a challenging dataset for current approaches.” 1 “The DETR encoder operates on 2D flattened image features from the backbone and applies a series of transformer layers. The decoder takes as input a set of N learned embeddings called object queries, that can be viewed as slots that the model needs to fill with detected objects. All the object queries are fed in parallel to the decoder, which uses cross-attention layers to look at the encoded image and predicts the output embeddings for each of the queries. The final representation of each object query is independently decoded into box coordinates and class labels using a shared feed-forward layer. The number of object queries acts as a de facto upper-bound on the number of objects the model can detect simultaneously.”
PNG
media_image10.png
554
1284
media_image10.png
Greyscale
PNG
media_image9.png
410
1258
media_image9.png
Greyscale
Note: While the scores, or soft-token classification loss, seen in Fig. 6 are determined by the second stage, the first stage of the model employs a DETR encoder, a known object detection model that will produce the initial candidate query results, or the bounding boxes which denote detected objects. The object detection dataset DETR is trained on allows it to learn to identify objects in an image and reuse, or associate, what it learned when it receives a test image it has not yet seen, thus the claims language the model describes an association between image data and the detection dataset to produce bounding box of the object is taught.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Luddecke with Kamath where a generic processing model that outputs results like object segmentations includes a first stage training where an object detection is performed to produce bounding boxes on objects in the image.
There are several reasons that would motivate one to do so, in machine learning object segmentation tasks a known method to improve accuracy in segmentation is to first have the model, or part of a model, identify the objects first.
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Luddecke (Image Segmentation Using Text and Image Prompts) in view of Tan (LXMERT:Learning Cross-Modality Encoder Representations from Transformers) in view of Kamath (MDETR-Modulated Detection for End-to-End Multi-Modal Understanding) and further in view of Gu (OPEN-VOCABULARY OBJECT DETECTION VIA VISION AND LANGUAGE KNOWLEDGE DISTILLATION)
Regarding claim 9,
Luddecke teaches:
The method of claim 8, further comprising at least any of: in response to determining that the prompt data is a category name defined according to the language expression format, determining the weight based on the generic prompt representation;(It has previously been shown how Luddecke Luddecke 3 and Fig. 2, teach that the generic prompt representation is leveraged if the input is a language expression format, which can include things such as category names.) in response to determining that the prompt data is either of a description defined according to the language expression format or the image format, determining the weight based on the prompt representation. (Luddecke Fig. 2, cited above, teaches that an image input prompt will have a different response, and will be handled by an image encoder rather than a text encoder, teaching that different input types are handled differently. The weights, as shown in the specifications previously cited ¶63, refer to embeddings describing object categories, are also shown to be taught by Luddecke as embeddings for the text and image data are obtained.)
Luddecke does not however teach an updated generic prompt representation and visual representation, this is taught by Tan determining the weight based on the updated generic prompt representation; (Tan 2.1 and Fig. 1 teach performing an attention operation to obtain)
Neither Tan nor Luddecke teach that global average pooling of the updated prompt representation is performed, as Luddecke specifically relies on CLIP rather than convolution layers where pooling would be performed. This is found in Gu which teaches in response to determining that the prompt data is either of a description defined according to the language expression format or the image format, determining the weight based on global average pooling of the updated prompt representation. (Gu D “in response to determining that the prompt data is either of a description defined according to the language expression format or the image format, determining the weight based on global average pooling of the updated prompt representation.” Note: Gu teaches that if the input data is in image format, then its weight is determined by the global average pooling of the updated prompt representation.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Luddecke with Gu where when an image prompt and text prompt are handled differently to obtain their weights, where an image prompt is based on global average pooling.
There are several benefits of doing so, data in image format benefits from global average pooling, a common step in machine learning approaches like convolution, which allows for data to be smoothed and improves computational efficiency.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Luddecke (Image Segmentation Using Text and Image Prompts) in view of Tan (LXMERT:Learning Cross-Modality Encoder Representations from Transformers) in view of Kamath (MDETR-Modulated Detection for End-to-End Multi-Modal Understanding) and further in view of He (Mask R-CNN).
Regarding claim 11,
Luddecke teaches:
The method of claim 10, further comprising:
Luddecke does not however teach performing a second-stage training based on a mixed object detection dataset that allows the model to perform both segmentation, or creating a mask, and object detection via a bounding box, this can be found in He which teaches after the first-stage training, performing a second-stage training based on a mixed object detection dataset such that the generic processing model describes an association between image data in the mixed object detection dataset, a bounding box of an object in the image data, and a mask of the object.(He 1 “Instance segmentation is challenging because it requires the correct detection of all objects in an image while also precisely segmenting each instance. It therefore combines elements from the classical computer vision tasks of object detection, where the goal is to classify individual objects and localize each using a bounding box, and semantic segmentation, where the goal is to classify each pixel into a fixed set of categories without differentiating object in stances.” 3 “Mask R-CNN is conceptually simple: Faster R-CNN has two outputs for each candidate object, a class label and a bounding-box offset; to this we add a third branch that out puts the object mask. Mask R-CNN is thus a natural and intuitive idea. But the additional mask output is distinct from the class and box outputs, requiring extraction of much finer spatial layout of an object.” Fig. 4
PNG
media_image11.png
202
650
media_image11.png
Greyscale
3.1 As in Fast R-CNN, an RoIis considered positive if it has IoU with a ground-truth box of at least 0.5 and negative otherwise. The mask loss Lmask is defined only on positive RoIs. The mask target is the intersection between an RoI and its associated ground-truth mask. Note: He teaches a two stage model that starts with Fast R-cnn, an object detection model which produces bounding boxes on objects it detects, and then leverages what has been learned from object detection to then perform a second stage training where a mask over the objects is produced. This two-stage approach is visualized above in Fig. 4, showing that a mask is produced as a second stage after a first stage of object detection. Furthermore, as seen in He 3.1, a mixed object detection dataset is leveraged including both ground-truth boxes for the object detection, and ground-truth masks for object segmentation.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Luddecke with He where the two-stage training of a model leverages a mixed object detection dataset to make associates between the image data in the dataset, a bounding box, and a mask.
There are several reasons that would motivate one to do so, leveraging a multi-step model that first learns from detecting objects to enhance its effectiveness in segmentation via a mixed dataset enhances the accuracy of the final segmentation result.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Luddecke (Image Segmentation Using Text and Image Prompts) in view of Tan (LXMERT:Learning Cross-Modality Encoder Representations from Transformers) in view of Kamath (MDETR-Modulated Detection for End-to-End Multi-Modal Understanding) in view of He (Mask R-CNN) and further in view of Yang (Video Instance Segmentation).
Regarding claim 12,
Luddecke teaches
The method of claim 11, further comprising:
Luddecke does not however teach a two-stage model in which a bounding box and mask of the object can be produced. This is found in He which teaches the generic processing model describes an association between image data, a bounding box of an object in the image data, and a mask of the object. (He Abstract, Fig. 4, 3.1, cited above, clearly teach a multi part model capable of performing both object detection and image segmentation.)
He’s Mask R-CNN model operates on single images, and does not teach processing on video data. This is found in Yang which teaches after the second-stage training, performing third-stage training based on a video dataset such that the generic processing model describes an association between image data in a video in the video data set, a bounding box of an object in the image data, and a mask of the object. (Yang Abstract “In this paper we present a new computer vision task, named video instance segmentation … Our new method introduces a new tracking branch to Mask R-CNN to jointly perform the detection, segmentation and tracking tasks simultaneously. Finally, we evaluate the proposed method and several strong baselines on our new dataset”
PNG
media_image12.png
594
602
media_image12.png
Greyscale
7 “We present a new task named video instance segmentation and an accompany dataset named YouTubeVIS in this work. The new tasks is a combination of object detection, segmentation, and tracking, which poses specific challenges given the rich and complex scenes. We also propose a new method combining single-frame instance segmentation and object tracking, which aims to provide some early explorations towards this task.”
PNG
media_image13.png
578
1212
media_image13.png
Greyscale
Note: He teaches that in order to perform object detection and segmentation on a video Mask R-CNN is employed to handle the first two stages of object detection then segmentation. This same Mask R-CNN model was introduced in the prior art He, thus Yang teaches the association between image data with a bounding box, and mask of the object, and the first two stages of training. Yang teaches that a third branch/stage is introduced, as seen above in Fig. 3 memory is leveraged to store objects previously detected so the work from a previous frame can be leveraged in the next, teaching a third stage in training where the model is extended to work with video data. As seen in Yang 7 a video dataset is leveraged to train the model to associate the video data with the bounding boxes and masks it will create.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Luddecke with Yang where the previously described two-stage training of a model has a third stage for processing video data.
There are several reasons that would motivate one to do so, if a user wishes to perform detection and segmentation on a video rather than a single image instead of using an entirely new model the effectiveness of existing image segmentation and detection models can be reused to make a high accuracy video processor.
Claims 13, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Luddecke (Image Segmentation Using Text and Image Prompts) in view of Tan (LXMERT:Learning Cross-Modality Encoder Representations from Transformers) in view of Kamath (MDETR-Modulated Detection for End-to-End Multi-Modal Understanding) in view of He (Mask R-CNN) in view of Yang (Video Instance Segmentation) and further in view of Wojcke (SIMPLE ONLINE AND REAL TIME TRACKING WITH A DEEP ASSOCIATION METRIC).
Regarding claim 13,
Luddecke teaches:
The method of claim 12,
Luddecke does not however teach that a distance between two images or frames in the video data can be made closer if they contain the same object, and increasing distance if they have different objects. This is taught in Wojcke further comprising: performing third-stage training based on the video dataset such that the generic processing model pulls closer a distance between generic visual representations of two image data in the video including the same object and pushes farther a distance between generic visual representations of two image data in the video including different objects. (Wojke Abstract “During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space.” 2.1 “Tracks that are not successfully associated to a measurement within their first three frames are deleted” 2.2 “In combination, both metrics complement each other by serving different aspects of the assignment problem. On the one hand, the Mahalanobis distance provides information about possible object locations based on motion that are particularly useful for short-term predictions. On the other hand, the cosine distance considers appearance information that are particularly useful to recover identities after long term occlusions, when motion is less discriminative. To build the association problem we combine both metrics using a weighted sum” Note: Wojke teaches that tracks, a series of video frames with related objects being tracked, have the distance between them calculated using Mahalanobis distance and cosine distance where both leverage object information, the cosine distance being more relevant to object identities, or identifying the same object that has been previously identified. Thus, Wojcke teaches that a distance measurement is performed on two or more frames in the video where if a similar object is identified the two frames will have a closer distance, and if they do not they will have a farther distance.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Luddecke with Wojcke where the video processing stage compares two images and sets a distance between them where images with the same object are closer.
There are several reasons that would motivate one to do so, a unique quality of video data processing as opposed to single image is that a current image, or frame, is likely to be very similar in image content to the previous, by comparing frames and checking for a shared object the valuable knowledge learned about the first frame can be reused when evaluating the second.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN GREGORY HAKALA whose telephone number is (571)272-7863. The examiner can normally be reached 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN GREGORY HAKALA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights