DETAILED ACTION
This examination is in response to the communication filed on 06/23/2026. Claims 1-20 are currently pending, where claims 1, 5-9, 12-15 and 17 have been amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 2, 7-9 and 14-16 are rejected under 35 U.S.C. §103 as being unpatentable over Kharbanda et al. (US 11,978,271 B1; herein “Kharbanda”) in view of Xu et al. "Multimodal Learning With Transformers: A Survey," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12113-12132, Oct. 2023 (herein “Xu”).
Regarding claims 1, 8 and 15, Kharbanda teaches a system (Fig. 9A), comprising: a processor (Fig. 9A, processors 112 and 132); and a memory including instructions executable by the processor (FIG. 9A memory 114 and 134 and instructions 118 and 134), a method, and a computer-readable medium (Col. 2, lines 42-46 teaches “The system can includes…one or more non-transitory computer-readable media…”) storing instructions that are operative upon execution by a processor to:
receive a multimodal prompt including a media file and information related to a region of interest (ROI) of the media file (Under a broadest reasonable interpretation, “information related to a region of interest” is interpreted as being any information utilized to determine/selected objects within the media file; Fig. 2, image data 212 and text data 232; Fig. 4A, elements 402 and 404; Fig. 5, elements 502 and 504; Col. 9, lines 57-64 teaches “…the generative model leveraged search system 200 can obtain input data, which can include image data 212 descriptive of one or more input image …and text data 232 descriptive of a request for particular information…” The one or more input images can be descriptive of an environment and one or more input images” and col. 11, lines 21-26 teaches “In some implementations, the particular object selected for processing may be based on the text data 232 (e.g., ‘what are recipes for this item?’ causes food items to be processed, while ‘what is that on the right?’ causes objects on the right of the input images to be processed” Thus, the input text data 232 is utilized to determine which objects in the input image are of interest);
determine the ROI of the media file based on the information related to the ROI of the media file (Fig. 2, object recognition 214; col. 11, lines 15-26 teaches “…Alternatively and/or additionally, the object recognition block 14 can determine a focal object and/or an object of interest based on object location, object size, image semantics, image focus, occurrence in a sequence of input images, and/or other contextual attributes. In some implementations, the particular object selected for processing may be based on the text data 232 (e.g., ‘what are recipes for this item?’ causes food items to be processed, while ‘what is that on the right?’ causes objects on the right of the input images to be processed” The focal object and/or object of interest is interpreted as a ROI), wherein the ROI of the media file is smaller than a global version of the media file (In some implementations, the particular object selected for processing may be based on the text data 232 (e.g., ‘what are recipes for this item?’ causes food items to be processed, while ‘what is that on the right?’ causes objects on the right of the input images to be processed” because the object of interest or focal object is selected from a portion of the input image it inherently smaller than a global version of the media file. In addition, col 10, lines 42-48 teaches “The object recognition model [of the object recognition block 214] may be trained and/or configured to process an image, detect an object, segment a portion of the image that includes the object…” Thus, the detected object is a sub-part of the image, making it smaller than the global image);
generate a plurality of media tiles of interest (MTIs) associated with the ROI of the media file (Under a broadest reasonable interpretation MTI is interpreted as segments or portions of the image corresponding to the object(s) of interest; Col. 10, lines 48-53 teaches “The object recognition model may include a detection model that processes the input images to generate bounding boxes indicating a position of the detected objects. A segmentation model of the object recognition model may then segment the detected objects based on the bounding boxes to generate image segments for the detected objects” );
encode the plurality of MTIs together with a natural-language input received with the multimodal prompt to generate a modified prompt, (Fig. 2, Augmented language output 222; col. 1, lines 4-7 teaches “The object recognition output 216, the language output 220, and/or the text data 232 can then be processed with an augmentation model 224 to generate the augmented language output 222”);
send the modified prompt to a large multimodal model (LMM) to process the modified prompt (Fig. 2, input to generative model 228; col. 12, lines 33-40 teaches “…the augmented language output 222 and/or the text data 232 can be processed with a generative model 228 (e.g., a large language model…) to generate one or more model-generated responses 230. The augmented language output 222… maybe formatted as a prompt…”); and
receive a response to the modified prompt from the LMM (Fig. 2, model-generated response 230 ).
Although Kharbanda discloses that the vision language model can include one or more transformer models (col. 13, lines 65-66), Kharbanda fails to disclose that the generative model 228 to which the modified prompt is input is a multimodal transformer. Thus, Kharbanda fails to explicitly disclose the modified prompt comprising a plurality of media tokens, the plurality of media tokens tokenized from the plurality of MTIs.
Xu teaches that multimodal transformers which tokenize and embedding both image and text data were known in the art prior to the effective filing date of the invention. More specifically, page 12117, section C Multimodal Transformers teaches “Given an input from arbitrary modality, users only need to perform to main steps (1) tokenize the input, and (2) select and embedding space to represent the tokens, before inputting the data into Transformers”. In addition, page 12118, first column, teaches that “Users can choose or design tokenization at multiple granularity levels – coarse-grained versus fine-grained. e.g., user ROIs (obtained by an object detector) and CNN features as tokens and token embeddings” Thus, Xu teaches tokenizing ROIs which are interpreted as corresponding to media tokens representing media tiles of interest.
Kharbanda differs from the claimed invention, as defined by claims 1, 8 and 15, in that Kharbanda fails to explicitly disclose utilizing a multimodal prompt which comprises tokenized media tiles of interest. Multimodal transformers which tokenize ROIs were known in the art as evidenced by Xu. Therefore, it would have been obvious to one having ordinary skill in the art before the effecting filing date of the invention to have modified the generative model taught by Kharbanda to utilize a multimodal transformer as taught by Xu as it merely constitutes the substitution of known models to achieve the predictable result of projecting different input modalities using a unified embedding space.
Regarding claims 2, 9 and 16, the combination of Kharbanda and Xu teaches all of the elements of claims 1, 8, and 15 (see detailed element mapping above). In addition, Kharbanda further teaches the information related to the ROI comprises one of: defined ROI parameters (the “comprises one of” language renders this element optional ); and instructions for automatically determining the ROI of the media file using one or more ROI policies (Under a broadest reasonable interpretation instructions for automatically determining, it interpreted as “instructions associated with the media file”, this is supported by ¶[0019] of the specification; Kharbanda Fig. 4A, element 402 “How Much Water Does This Need?”, Fig. 4B element 432 “Write A Haiku About This Place”, and Fig. 5 element 502 “Create a listing to sell this”.)
Regarding claims 7 and 14, the combination of Kharbanda and Xu teaches all of the elements of claims 1 and 9 (see detailed element mapping above). In addition, Kharbanda further teaches the media file is an image file (Fig. 4A, element 404, Fig. 4B element 434, and Fig. 5 element 504); and
the memory further comprises instructions executable by the processor to present the response via a user interface, the response being a natural-language description of the image depicted in the image file (Fig. 9A, user interface 124 and 144; Fig. 5, model-generated response 506; and col. 17, lines14-22 teaches “generative model response system 500 can obtain an input image 504 and input text 502 to generate a model-generated response 506. For example, the input text 502 can include "create a listing to sell this," the input image 504 can depict a white chair in a room, and the model-generated response 506 can include a model-generated listing.”).
Claims 3, 4, 10, 11, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kharbanda and Xu as applied to claims 1, 8 and 15 above, and further in view of M. Cai et al., "ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 12914-12923. (herein “Cai”).
Regarding claims 3, 10 and 17, the combination of Kharbanda and Xu teaches all of the elements of claims 2, 9, and 16 (see detailed element mapping above). In addition, Kharbanda further teaches the defined ROI parameters includes one of: mask information defining the ROI of the media file (the one of language renders this element optional); and coordinate information defining the ROI of the media file (col. 11, lines 17-21 teaches “The object recognition block 14 can determine a focal object and/or an object of interest based on object location…” object location is interpreted as coordinate information).
However, the combination of Kharbanda and Xu fails to disclose that the defined ROI parameters are included in the information provided in the received multimodal prompt as defined in claims 2, 9, and 16, from which claims 3, 10, and 17 depend, respectively.
Cai teaches providing spatial references in multimodal prompts, such as using textual representations of coordinates, learned positional embeddings, or ROI features. (See page 12914, second column).
The combination of Kharbanda and Xu differs from the claimed invention, as defined in claims 3, 10 and 17, in that the combination fails to disclose providing ROI parameters as part of the multimodal prompt. Multimodal prompts which include spatial references such as ROI features is known in the art as evidenced by Cai. Therefore, it would have been obvious to one having ordinary skill in the art, before the effective filing date of the invention, to have modified the system taught by the combination of Kharbanda and Xu to include the ROI parameters as part of the multimodal prompt as taught by Cai in order to allow the LLM to process region-specific information in complex scenes. (Cai, page 12914, section 1, second paragraph).
Regarding claims 4, 11 and 18, the combination of Kharbanda and Xu teaches all of the elements of claims 2, 9, and 16 (see detailed element mapping above). In addition, Kharbanda further teaches instructions executable by the processor to:
apply the defined ROI parameters to a global tile associated with the media file to determine the ROI of the media file (col. 11, lines 17-21 teaches “The object recognition block 14 can determine a focal object and/or an object of interest based on object location, object size, image semantics…” Col. 10, lines 48-53 teaches “The object recognition model may include a detection model that processes the input images to generate bounding boxes indicating a position of the detected objects. A segmentation model of the object recognition model may then segment the detected objects based on the bounding boxes to generate image segments for the detected objects” ); and
generate the plurality of MTIs based on the determined ROI (Under a broadest reasonable interpretation MTI is interpreted as segments or portions of the image corresponding to the object(s) of interest; Col. 10, lines 48-53 teaches “The object recognition model may include a detection model that processes the input images to generate bounding boxes indicating a position of the detected objects. A segmentation model of the object recognition model may then segment the detected objects based on the bounding boxes to generate image segments for the detected objects”).
However, the combination of Kharbanda and Xu fails to disclose that the defined ROI parameters are included in the information provided in the received multimodal prompt as defined in claims 2, 9, and 16, from which claims 3, 10, and 17 depend, respectively.
Cai teaches providing spatial references in multimodal prompts, such as using textual representations of coordinates, learned positional embeddings, or ROI features. (See page 12914, second column).
The combination of Kharbanda and Xu differs from the claimed invention, as defined in claims 3, 10 and 17, in that Kharbanda fails to disclose providing ROI parameters as part of the multimodal prompt. Multimodal prompts which include spatial references such as ROI features is known in the art as evidenced by Cai. Therefore, it would have been obvious to one having ordinary skill in the art, before the effective filing date of the invention, to have modified the system taught by the combination of Kharbanda and Xu to include the ROI parameters as part of the multimodal prompt as taught by Cai in order to allow the LLM to process region-specific information in complex scenes. (Cai, page 12914, section 1, second paragraph).
Claims 5, 6, 12, 13, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kharbanda and Xu as applied to claims 1, 8 and 15 above, and further in view of TAL et al. (US 2010/0322521 A1; herein “TAL”).
Regarding claims 5, 12 and 19, Kharbanda teaches all of the elements of claims 2, 9, and 16 (see detailed element mapping above). In addition, Kharbanda further teaches instructions executable by the processor to:
apply the plurality of rules to a global tile associated with the media file to determine the ROI of the media file (Under a broadest reasonable interpretation “apply the plurality of rules” is interpreted as applying algorithms to determine objects of interest; Kharbanda, col. 9 lines 3-7 teaches “…the object recognition block 14 can determine a focal object and/or an object of interest based on object location, object size, image semantics, image focus, occurrence in a sequence of input images, and/or other contextual attributes” Thus, Kharbanda teaches applying a plurality of algorithm/rules to determine an object of interest); and
generate the plurality of MTIs based on the determined ROI (Under a broadest reasonable interpretation MTI is interpreted as segments or portions of the image corresponding to the object(s) of interest; Col. 10, lines 48-53 teaches “The object recognition model may include a detection model that processes the input images to generate bounding boxes indicating a position of the detected objects. A segmentation model of the object recognition model may then segment the detected objects based on the bounding boxes to generate image segments for the detected objects”)
Although Kharbanda teaches the instructions are executable by the processor to: utilize a plurality of algorithms to detect objects, Kharbanda fails to explicitly disclose instructions executable by the processor to: access a view composer policy storing a plurality of rules wherein at least some of the plurality of rules instruct a view composer to exclude low-value regions of the media file in the ROI.
TAL teaches a region of interest (ROI) extraction method which incorporates segmentations methods with context-aware saliency which follows four basic principles (1) Local low-level considerations, including factors such as contrast and color, (2) Global consideration, which suppress frequently occurring features, while maintaining features that deviate from the norm, (3) Visual organization rules, and (4) High-level factors (See ¶¶[0129]-[0135]). In addition, TAL teaches in ¶[0138] that “homogeneous or blurred areas should obtain low saliency values” and ¶[0146] teaches “we consider multiple scales, so that the saliency of background pixels is further decreased, improving the contrast between salient and non-salient regions” and ¶[0104] teaches “the non-salient background is eliminated in the images…” Accordingly, TAL teaches the excluding low-value, i.e., non-salient regions such as the background. Thus, TAL teaches a plurality of rules wherein at least some of the plurality of rules instruct the view composer to exclude low-value regions of the media file in the ROI.
The combination of Kharbanda and Xu differs from the invention, as defined in claims 5, 12 and 19, in that the combination fails to explicitly disclose that the plurality of algorithms/rules for detecting objects within an image includes rules/policies which exclude low-value regions. Policies excluding low-value, e.g., non-salient, regions, such as backgrounds in known in the art as evidenced by TAL. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to have modified the ROI extraction/segmenting method taught by the combination of Kharbanda and Xu to include rules/policies which exclude low-value regions as taught by TAL in order to keep some of the background in the region of interest when the background is required for conveying the context (TAL, ¶[0097]).
Regarding claims 6, 13 and 20, the combination of Kharbanda and TAL teaches all of the elements of claims 5, 12 and 19 (see detailed element mapping above). In addition, Kharbanda further teaches the media file is an image file (Fig. 4A, element 404, Fig. 4B element 434, and Fig. 5 element 504).
In addition, TAL further teaches that at least some of the low-value regions are defined as a region of the global media tile containing little-to-no contrast in color or texture (¶[0090] teaches “Manny approaches have been proposed for detecting regions with maximum local saliency of low-level factors. These factors usually consist of intensity, color, orientation, texture, size, and shape” and ¶[0124] teaches “…Local low-level considerations including factors such as contrast, color, orientation, etc. …” ).
The combination of Kharbanda and Xu differs from the invention, as defined in claims 6, 13 and 20, in that the combination fails to explicitly disclose that at least some of the low-value regions are defined as regions containing little to no contrast in color or texture. Excluding/defining backgrounds regions based on no contrast in color or texture in known in the art as evidenced by TAL. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the invention to have modified the ROI extraction/segmenting method taught by the combination of Kharbanda and Xu to include rules/policies which exclude low-value regions as taught by TAL in order to keep some of the background in the region of interest when the background is required for conveying the context (TAL, ¶[0097])
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PENNY L CAUDLE whose telephone number is (703)756-1432. The examiner can normally be reached M-Th 8:00 am to 5:00 pm eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at 571-272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PENNY L CAUDLE/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights