PERFORMING VISUAL RELATIONAL REASONING

DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Continued Examination Under 37 CFR 1.114 A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/25/2026 has been entered. Response to Amendment The amendment filed on 09/25/2026 has been entered. Claims 1, 6, 11, 16, 20 were amended, claims 2, 3, 8-10, 12, 13, 18, 19, 21 was/were cancelled and claim 23 was newly added. Claims 1, 4-7, 11, 14-17, 20, 22, 23 remain pending in the application. Response to Arguments Applicant’s arguments (Remarks filed 09/25/2026) have been considered but are moot in view of the new ground(s) of rejection in view of Li (“Improving Human-Object Interaction Detection via Phrase Learning and Label Composition”) regarding the amended limitations. Specification The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 1, 4-7, 11, 14-17, 20, 22, 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Huang (”Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning”, 2021), in view of Li (“Improving human-object interaction detection via phrase learning and label composition”, 28 June 2022). Regarding claims 1, 11, and 20, Huang teaches A method comprising (Huang, abstract: “In this paper, we propose SOHO to “See Out of tHe bOx” that takes a whole image as input, and learns vision-language representation in an end-to-end manner”), at a device: processing an image (Huang, pg 3, column 1, Section 3, ¶1, “The visual encoder takes animage as input”, which is being interpreted as “processing an image”), by a vision transformer (Huang, see image below: “Transformer” and “visual” is being interpreted as including a “vision transformer”) pretrained (Huang, see image below: “we propose a novel Masked Visual Modeling pre-training” is being interpreted as involving “pretrained”) on a predefined (Huang, see image below, “based on the virtual visual semantic labels produced by the visual dictionary” is being interpreted as involving “predefined”, which is being interpreted as existing before; just as classification labels are predefined in classical machine learning algorithms) concept-feature dictionary (Huang, pg 4, column 2, Section 3.3, ¶1, reproduced below: PNG media_image1.png 340 550 media_image1.png Greyscale . “Visual dictionary” is being interpreted as involving a concept-feature dictionary as seen from the “image-text matching pre-training tasks”) that correlates image features with image concepts (Huang, pg 8, Section 4.4, reproduced below: PNG media_image2.png 338 552 media_image2.png Greyscale . “VD index” is being interpreted as image concept [as can be seen in Figure 3 where index 191 is the image concept “head” and index 1074 is the image concept “building”], “visual feature” is being interpreted as “image features”) to infer an associated concept for the image (Huang, Figure 1 and Figure 1 text shows “Ours: A couple sit on the shore next to a boat on the sea”, which is being interpreted as involving inferring “an associated concept for the image”) indicates an interaction (Huang, Figure 1, “Ours: A couple sit on the shore next to a boat on the sea”. “sit on the shore”, “next to a boat” are being interpreted as indicating “an interaction”) between two or more objects as depicted in the image (Huang, Figure 1, “Ours: A couple sit on the shore next to a boat on the sea”. “Couple, shore, boat, sea” are being interpreted as “two or more objects depicted in the image”), wherein the vision transformer (Huang, pg 3, Section 3, ¶1: “a visual dictionary (VD) embedding module, and a multi-layer Transformer”. Which is being interpreted as involving a vision transformer) is comprised of a tokenizer (Huang, pg 3, Section 3, ¶1: “VD embedding module is designed to aggregate diverse visual semantic information into visual tokens with a proposed visual dictionary”. “Tokens” is being interpreted as involving a “tokenizer”), at least one layer for generating patch embeddings (Huang, see Section 4.4 image above, “image patch”; when combined with pg 3, Figure 2 of “Visual Dictionary-based embedding features”, shows “at least one layer for generating patch embeddings”), and at least one multi-head self-attention layer ((Huang, pg 9, Section A.4. Discussion, ¶1, reproduced below: PNG media_image3.png 572 558 media_image3.png Greyscale “Multi-layer Transformer” is being interpreted as involving at least one multi-head. “Self-attention mechanism” is being interpreted as being part of one of the layer, resulting in a “multi-head self-attention layer”. Further, a ResNet-101 backbone and 12-layer Transformer is used in this prior art, seen in pg 5, Section 4.1. As with ordinary skill in the art would know, this contains at least one multi-head self attention layer”; for illustrative reference: https://arxiv.org/abs/1810.04805 ; see pg 3, column 2 that mentions “self-attention heads”), and wherein the predefined concept-feature dictionary (Huang, see Section 4.4 image above, “Visual Dictionary” and “a VD index is correlated with many visual features”) is comprised of entries that each link a plurality of image features (Huang, see Section 4.4 image above, ““visual feature” is being interpreted as “image features”) with a key representative of a concept (Huang, see Section 4.4 image above, “Visual Dictionary” and “a VD index is correlated with many visual features”. “VD index” is being interpreted as “a key representative of a concept”. See figure 3 for non-limiting examples of concepts such as “head” and “building” for the non-limiting examples of keys such as “191” and “1074”) that identifies at least two objects (Huang, Figure 1, Ours: “a couple sit on the shore next to a boat on the sea”, which shows non-limiting examples of “at least two objects” such as “a couple”, “shore”. Also see Figure 2(f), where the id 2 and id 5 are non-limiting examples of at least two objects) depicted by the plurality of image features (Huang, see Section 4.4 image above, ““visual feature” is being interpreted as “image features”) and outputting the concept inferred for the image (Huang, Figure 1 and Figure 1 text shows “Ours: A couple sit on the shore next to a boat on the sea”. Which is being interpreted as outputting the concept inferred for the image). However, Huang does not appear to explicitly teach predefined concept-feature dictionary comprising an interaction between the at least two objects. Although Huang does teach an interaction between at least two objects, as seen in Figure 1 (non-limiting example of “A couple sit on the shore next to a boat on the sea”). Pertaining to the same field of endeavor, Li teaches an interaction between the at least two objects (Li, pg 1513, column 1, 2nd to last paragraph, reproduced below: PNG media_image4.png 276 630 media_image4.png Greyscale . “person sit at dining table” is being interpreted as a non-limiting example involving an interaction between the at least two objects. For this non-limiting example, the two objects are “person” and “dining table”, with the interaction being “sit”) that is depicted by the plurality of image features (Li, pg 1511, Figure 2, “feature extractor” from the image on the left is being interpreted as involving “image features”); and Huang and Li are considered to be analogous art because they are directed to object interaction and detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for object interaction detection and captioning (as taught by Huang) to include predefined concept-feature dictionary comprising an interaction between the at least two objects (as taught Li) because the combination provides an improvement to human-object interaction detection (Li, Title and Abstract). Further, Huang already teaches interaction between at least two objects, as seen in Figure 1 (non-limiting example of “A couple sit on the shore next to a boat on the sea”). Additionally, Huang teaches using Flickr30k dataset, reference 31, which includes “region-to-phrase correspondences”. Regarding claim 4, Huang teaches The method of claim 1, wherein the image is not labeled with the concept when received as input (Huang, pg 5, column 1, last paragraph: “Detailed comparisons of pre-training dataset usage of most VLPT works, including our train/test image and text numbers, are included in our supplementary material”. “Test image” is being interpreted as splitting the dataset into training and testing sets, as one with ordinary skill in the art would know. The testing sets are being interpreted as not labeled with the concept when received as input). Regarding claim 5, Huang teaches The method of claim 1, wherein the vision transformer performs action prediction and object prediction utilizing the image (Huang, Figure 1 and Figure 1 text shows “Ours: A couple sit on the shore next to a boat on the sea”. “Sit” is being interpreted as “action prediction”. “Couple”, “shore”, “boat”, “sea” are being interpreted as object prediction. Figure 1 shows an image which these predictions are done. SOHO, the method, is being interpreted as involving a “vision transformer”). Regarding claim 6, Huang teaches The method of claim 1, wherein the concept (Huang, see Figure 1 citation below, the caption is being interpreted as involving “the concept”) indicates the interaction (Huang, see Figure 1 citation below, “A couple sit on the shore next to” is being interpreted as involving an interaction. The non-limiting example of interaction is “sit on”) between the two or more objects (Huang, see Figure 1 citation below, “couple” and “boat” are being interpreted as “two or more objects”) as a tuple (Huang, see Figure 1 citation below. A definition of a tuple, as one with ordinary skill in the art would know, is an ordered, finite sequence of elements. The order of the caption matters and it is a finite sequence of elements) of two objects and an associated action (Huang, Figure 1 and Figure 1 text shows “Ours: A couple sit on the shore next to a boat on the sea”. “Couple” and “boat” are being interpreted as examples of two objects with the association action of “sit”). Regarding claim 7, Huang teaches The method of claim 1, further comprising, at the device: performing one or more classification operations utilizing the concept (Huang, pg 7, column 2, Section 4.2.4, ¶1, reproduced below: PNG media_image5.png 456 548 media_image5.png Greyscale . “Three-classification” is being interpreted as one or more classification operations. “Output of the transformer” is being interpreted as “utilizing the concept”). Regarding claim 22, Huang teaches The method of claim 1, wherein the vision transformer is configured to include: a global task that, during training, clusters images with the same concept together to produce semantically consistent relational representations (Huang, Figure 1, reproduced below: PNG media_image6.png 744 1102 media_image6.png Greyscale . “Global context” is being interpreted as “global task”. For example, “chatting” or “next to a boat” are examples of relational representations”), and a local task that, during training, guides the vision transformer to discover object-centric semantic correspondence across images (Huang, pg 8, Section 4.4, reproduced below: PNG media_image2.png 338 552 media_image2.png Greyscale . “VD index” is being interpreted as object-centric semantic correspondence [as can be seen in Figure 3 where index 191 is the object-centric semantic correspondence “head” and index 1074 is the object-centric semantic correspondence “building” across images]. This object-centric semantic correspondence is being interpreted as “a local task”; pg 2, column 1, second to last paragraph: “VD can be dynamically updated through our trainable CNN backbone directly from visual-language data during pretraining” The Visual Dictionary is being interpreted as during training guides the vision transformer). Regarding claim 23, Huang teaches The method of claim 1, wherein the method further comprises: training the vision transformer on the predefined concept-feature dictionary (Huang, pg 4, column 2, Section Masked Visual Modeling, ¶1, reproduced below: PNG media_image7.png 272 1100 media_image7.png Greyscale . “Transformer” and “image features” are being interpreted as involving “vision transformer”. “Learning” is being interpreted as involving “training”. “Visual dictionary” is being interpreted as involving “predefined concept-feature dictionary”) to infer concepts for given images (Huang, pg 5, column 1, lines 2-5: “MVM can encourage the model to infer visual knowledge from the contextual visual information as well as language”. “Infer visual knowledge” is being interpreted as a non-limiting example involving “infer concepts for given images”). Claim 14 is rejected using the same rationale as applied to claim 4 discussed above. Claim 15 is rejected using the same rationale as applied to claim 5 discussed above. Claim 16 is rejected using the same rationale as applied to claim 6 discussed above. Claim 17 is rejected using the same rationale as applied to claim 7 discussed above. Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Hu et al (“Recognising Human-Object Interaction via Exemplar based Modelling”, 2013) discloses human-object interaction with concept-feature dictionary. Kim et al (“Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)”, 2018) discloses a dictionary of concepts (Appendix A) and trained using concept vectors. Liu et al (“SIFT Flow: Dense Correspondence across Scenes and its Applications”, 2010, as cited in IDS filed 09/16/2022) discloses a concept-feature dictionary (interpreted from visual words and SIFT descriptors, pg 6, column 1, first full paragraph). Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY B DUONG whose telephone number is (571)272-1358. The examiner can normally be reached Monday - Thursday 10a-9p (ET). Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /J.B.D./Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667