CTNF 18/562,771 CTNF 97880 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement The information disclosure statement(s) (IDS) submitted on 11/20/2023 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) has/have been considered by the examiner. Specification 06-11 AIA The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. 06-11-01 AIA The following title is suggested: Method utilizing image captions to locate objects in images . Claim Objections 07-29-01 AIA Claim 4 objected to because of the following informalities: Line 8 says “generating generates fusion information” it should read “generating fusion information. Line 10 says “estimating estimated position information” should read “estimating position information.” Appropriate correction is required. Claim Rejections - 35 USC § 102 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15-aia AIA Claim(s) 1, 5, 10-11,13, 21, and 23 is/are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Gonzalez-Garcia (Gonzalez-Garcia A, Modolo D, Ferrari V. Objects as context for detecting their semantic parts. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 6907-6916).) . Regarding claim 1, Gonzalez-Garcia discloses A position estimation device comprising a processor configured to execute operations comprising: (Gonzalez-Garcia Section 5.1 Validation of our model – Runtime – found on righthand column of p.6913; runtime of method on a Titan X GPU is reported.) generating fusion information, wherein the fusion information includes, according to fusing: position information of a subject object that is an object corresponding to a subject, visual information of the subject object, and relationship information indicating a relationship with a target object paired with the subject object; (Gonzalez-Garcia Section 3.3 Adding relative location – Overview – found p. 6910; an object in its bounding box (visual and position information – see Fig. 5) and its class (class indicates what parts to look for and where in the bounding box of the subject that information would be i.e. relational information) are input and part suggestions are output. In Fig. 2 it can be seen that the information is sent through 2 fully connected layers (fused).) and estimating a position of the target object using an object position estimation model learned in advance on the basis of the fusion information. (Gonzalez-Garcia Section 3.3 Adding relative location – Training OffsetNet and Generating suggested windows – found on left hand column of p. 6911 and Fig. 5 and 6; OffsetNet is trained using objects and their parts to take in an object proposal (including the category – see Section 3.3 Adding relative location – Overview – found on p. 6910 lefthand column; where the location model is specific to each object class) and output bounding boxes of parts (position of target object). Examples of objects (subject) are seen in red and parts (target object) are seen in green. ) Regarding claim 10, Gonzalez-Garcia discloses the claim limitations of claim 1, as described above. Gonzalez-Garcia further discloses wherein the visual information of the subject object includes a tensor output of an image input of a region of the subject object. (Gonzalez-Garcia Fig. 2; a bounding box of the object (subject) is created from an input image (on left) and passed into the OffsetNet. Tensors are represented in convolutional layers and rectangles in teal. See also Fig. 5.) Regarding claim 11, Gonzalez-Garcia discloses the claim limitations of claim 1, as described above. Gonzalez-Garcia further discloses further comprising: displaying, based on the estimated position of the target object, position information of the target object in an image input, wherein the image input indicates at least a part of the subject object and at least a part of the target object. (Gonzalez-Garcia Fig. 6; outputs of bounding boxes of the object (subject) in red and the part (target object) in green can be seen.) Regarding claim 13, Gonzalez-Garcia discloses the claim limitations of claim 1, as described above. Gonzalez-Garcia further discloses wherein the object position estimation model uses a neural network, and the neural network outputs the position of the target object based on the fusion information. (Gonzalez-Garcia Section 3.3 Adding relative location – Training OffsetNet and Generating suggested windows – found on left hand column of p. 6911 and Fig. 5 and 6; OffsetNet (a neural network) is trained using objects and their parts to take in an object proposal (including the category – see Section 3.3 Adding relative location – Overview – found on p. 6910 lefthand column; where the location model is specific to each object class) and output bounding boxes of parts (position of target object). Examples of objects (subject) are seen in red and parts (target object) are seen in green. ) Regarding claims 5, 21, and 23 they are the corresponding method claims to claims 1,10, and 13 respectively and are rejected for similar reasons . Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 3-4 and 16-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez-Garcia (Gonzalez-Garcia A, Modolo D, Ferrari V. Objects as context for detecting their semantic parts. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 6907-6916).) in view of Girshick (Girshick R. Fast r-cnn. InProceedings of the IEEE international conference on computer vision 2015 (pp. 1440-1448).) . Regarding claim 3, Gonzalez-Garcia discloses the claim limitations of claim 1, as described above. Gonzalez-Garcia further discloses wherein the object position estimation model is learned to optimize the position information of the target object and estimated position information to be calculated by; calculating relative position information and the position information of the target object for learning, (Gonzalez-Garcia Section 3.3 Adding relative location – Training OffsetNet and Generating suggested windows – found on left hand column of p. 6911 and Fig. 5 and 6; offset vectors to the object (subject) representing the position of the part (target object) are a learned output.) wherein the relative position information represents a correct answer of the position information of the subject object for learning, (Gonzalez-Garcia Section 3.3 Adding relative location – Training OffsetNet and Generating suggested windows – found on left hand column of p. 6911 and Fig. 5 and 6; offset vectors to the object (subject) representing the position of the part (target object) are a learned output. Ground truth labels are used to train the model. See also Section 4. Implementation details – Training OffsetNet – found on p. 6912 lefthand column to righthand column.) Gonzalez-Garcia does not explicitly disclose updating a parameter so as to reduce a distance between the estimated position information and the relative position information. Girshick, however, discloses updating a parameter so as to reduce a distance between the estimated position information and the relative position information. (Girshick Section Multi-task loss ¶2-3 – found on p. 1442; the distance between a ground truth bounding box (represented by v ) and the predicted box (i.e. estimated position. Represented by t ) through L 1 loss.) It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the position estimation device of Gonzalez-Garcia with the teaches of Girshick by including reduction of the distance between the ground-truth bounding box and the predicted bounding box to provide a fast way to train a region-based CNN (Girshick abstract). Regarding claim 4, Gonzalez-Garcia discloses A position estimation learning device comprising a processor configured to execute operations comprising: (Gonzalez-Garcia Section 5.1 Validation of our model – Runtime – found on righthand column of p.6913; runtime of method on a Titan X GPU is reported.) receiving, as learning data, position information of a subject object that is an object corresponding to a subject, visual information of the subject object, position information of a target object paired with the subject object, and relationship information indicating a relationship with the target object paired with the subject object; (Gonzalez-Gracia Section 3.3 Adding relative location- Training OffsetNet – found on p. 6911 lefthand column and Section 4. Implementation details – Training OffsetNet – found p. 6912 lefthand column to right hand column; training samples include object proposals and bounding boxes (position and visual information of subject), part ground-truth bounding boxes (position information of target) are also used. The object and its associated part class are the relationship information.) generating generates fusion information, wherein the fusion information includes, according to fusing, the position information of the subject object, the visual information of the subject object, and the relationship information (Gonzalez- Garcia Section 3.3 Adding relative location – Overview – found p. 6910; an object in its bounding box (visual and position information – see Fig. 5) and its class (class indicates what parts to look for and where in the bounding box of the subject that information would be i.e. relational information) are input and part suggestions are output. In Fig. 2 it can be seen that the information is sent through 2 fully connected layers (fused).) estimating estimated position information by using an object position estimation model on the basis of the fusion information; and (Gonzalez-Garcia Section 3.3 Adding relative location – Training OffsetNet and Generating suggested windows – found on left hand column of p. 6911 and Fig. 5 and 6; OffsetNet is trained using objects and their parts to take in an object proposal (including the category – see Section 3.3 Adding relative location – Overview – found on p. 6910 lefthand column; where the location model is specific to each object class) and output bounding boxes of parts (position of target object). Examples of objects (subject) are seen in red and parts (target object) are seen in green.) calculating relative position information and the position information of the target object, wherein the relative position information represents a correct answer of the position information of the subject object, and (Gonzalez-Garcia Section 3.3 Adding relative location – Training OffsetNet and Generating suggested windows – found on left hand column of p. 6911 and Fig. 5 and 6; offset vectors to the object (subject) representing the position of the part (target object) are a learned output. Ground truth labels are used to train the model. See also Section 4. Implementation details – Training OffsetNet – found on p. 6912 lefthand column to righthand column.) Gonzalez-Garcia does not explicitly disclose updating a parameter of the object position estimation model to reduce a distance between the estimated position information and the relative position information so as to optimize the position information of the target object and the calculated estimated position information. Girshick, however, discloses updating a parameter of the object position estimation model to reduce a distance between the estimated position information and the relative position information so as to optimize the position information of the target object and the calculated estimated position information. (Girshick Section Multi-task loss ¶2-3 – found on p. 1442; the distance between a ground truth bounding box (represented by v ) and the predicted box (i.e. estimated position. Represented by t ) through L 1 loss.) It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the position estimation learning device of Gonzalez-Garcia with the teaches of Girshick by including reduction of the distance between the ground-truth bounding box and the predicted bounding box to provide a fast way to train a region-based CNN (Girshick abstract). Regarding claims 16-18, , the combination of Gonzalez-Garcia and Girshick disclose the claim limitations with respect to claim 4 as described above. Additional limitations are the corresponding limitations to claims 10, 11, and 13 and are rejected for similar reasons. Regarding claim 19, it is the corresponding method claim to claim 3 and is rejected for similar reasons . 07-21-aia AIA Claim (s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez-Garcia (Gonzalez-Garcia A, Modolo D, Ferrari V. Objects as context for detecting their semantic parts. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 6907-6916).) in view of Sun(Sun X, Zi Y, Ren T, Tang J, Wu G. Hierarchical visual relationship detection. InProceedings of the 27th ACM International Conference on Multimedia 2019 Oct 15 (pp. 94-102).) . Regarding claim 2, Gonzalez-Garcia disclose the claim limitations with respect to claim 1, as described above. Gonzalez-Garcia further discloses the object class informs what parts of the object maybe in the image. (class is utilized in OffsetNet (Gonzalez-Garcia Section 3.2.2 Object class – found on righthand column of p. 6909 to lefthand column of p. 6910 and Section 3.3. Adding relative location – Overview and OffsetNet Model ¶1 – found on p. 6910 lefthand to righthand column) But not explicitly wherein the relationship information uses a vector represented by using a word s, a word o, and a word w, wherein the words indicates a name of the subject object, the word o indicates a name of the target object, and the word w indicates a relationship between the word s and the word o. Sun, however, discloses wherein the relationship information uses a vector represented by using a word s, a word o, and a word w, wherein the words indicates a name of the subject object, the word o indicates a name of the target object, and the word w indicates a relationship between the word s and the word o. (Sun Section 1. Introduction ¶1 and 4 found on p. 94; relationship triplets are used to represent how objects in an image relate to each other. They are set up as word vectors with subject, predicate, object. Examples are seen in Fig. 3.) It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the position estimation device of Gonzalez-Garcia with the teachings of Sun by utilizing the labeling format of Sun because it is a common format for object relationship information and is already found in multiple datasets (Sun Section 2.1 Visual relationship detection ¶1-2) . 07-21-aia AIA Claim (s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez-Garcia (Gonzalez-Garcia A, Modolo D, Ferrari V. Objects as context for detecting their semantic parts. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 6907-6916).) in view of Girshick (Girshick R. Fast r-cnn. InProceedings of the IEEE international conference on computer vision 2015 (pp. 1440-1448).) and Sun(Sun X, Zi Y, Ren T, Tang J, Wu G. Hierarchical visual relationship detection. InProceedings of the 27th ACM International Conference on Multimedia 2019 Oct 15 (pp. 94-102).) . Regarding claim 14, the combination of Gonzalez-Garcia and Girshick disclose the claim limitations with respect to claim 4 as described above. Additional limitations are the corresponding limitations to claim 2 and are rejected for similar reasons . 07-21-aia AIA Claim (s) 9, 12, 20 and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez-Garcia (Gonzalez-Garcia A, Modolo D, Ferrari V. Objects as context for detecting their semantic parts. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 6907-6916).) in view of Lu (Lu C, Krishna R, Bernstein M, Fei-Fei L. Visual Relationship Detection with Language Priors. arXiv preprint arXiv:1608.00187. 2016 Aug.) . Regarding claim 9, Gonzalez-Garcia disclose the claim limitations with respect to claim 1, as described above. Gonzalez-Garcia does not explicitly disclose wherein the subject object includes a person, and the target object includes a smartphone held by the person, and the smartphone is partially hidden in the visual information of the subject object. Lu, however, discloses wherein the subject object includes a person, and the target object includes a smartphone held by the person, and the smartphone is partially hidden in the visual information of the subject object. (Lu Fig. 6; an example of a person holding a phone is seen in (e). It can be seen that the persons hand is partially covering the phone.) It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the position estimation device of Gonzalez-Garcia with the teachings of Lu by including a person with holding a phone as a part category to detect because it is a common occurrence in daily life. It would be particularly useful to detect in instances where it is against the law to use a phone like while driving. Regarding claim 12, Gonzalez-Garcia disclose the claim limitations with respect to claim 1, as described above. Gonzalez-Garcia does not explicitly disclose wherein the relationship information is based on a vector output of a word2vec model, and the word2vec model outputs the vector output based on a word input. Lu, however, discloses wherein the relationship information is based on a vector output of a word2vec model, and the word2vec model outputs the vector output based on a word input. (Lu Algorithm 1- found on p. 15; word2vec is used for text representation of objects and their relation.) It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the position estimation device of Gonzalez-Garcia with the teachings of Lu by including the labeling format disclosed, including using word2vec because word2vec has embedded information that place similar relationships close to each other so that zero shot detections can be made for example of an unseen object (Lu Section 1 Introduction ¶4 – found on p. 2). Regarding claim 20, it is the corresponding method claim to claim 9 and is rejected for similar reasons. Regarding claim 22, Gonzalez-Garcia disclose the claim limitations with respect to claim 1, as described above. Gonzalez-Garcia additionally discloses wherein the visual information of the subject object includes a tensor output of an image input of a region of the subject object, (Gonzalez-Garcia Fig. 2; a bounding box of the object (subject) is created from an input image (on left) and passed into the OffsetNet. Tensors are represented in convolutional layers and rectangles in teal. See also Fig. 5.) Gonzalez-Garcia does not explicitly disclose and wherein the relationship information is based on a vector output of a word2vec model, and the word2vec model outputs the vector output based on a word input. Lu, however, discloses and wherein the relationship information is based on a vector output of a word2vec model, and the word2vec model outputs the vector output based on a word input. (Lu Algorithm 1- found on p. 15; word2vec is used for text representation of objects and their relation.) It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the method of Gonzalez-Garcia with the teachings of Lu by including the labeling format disclosed, including using word2vec because word2vec has embedded information that place similar relationships close to each other so that zero shot detections can be made for example of an unseen object (Lu Section 1 Introduction ¶4 – found on p. 2) . 07-21-aia AIA Claim (s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gonzalez-Garcia (Gonzalez-Garcia A, Modolo D, Ferrari V. Objects as context for detecting their semantic parts. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 6907-6916).) in view of Girshick (Girshick R. Fast r-cnn. InProceedings of the IEEE international conference on computer vision 2015 (pp. 1440-1448).) and Lu (Lu C, Krishna R, Bernstein M, Fei-Fei L. Visual Relationship Detection with Language Priors. arXiv preprint arXiv:1608.00187. 2016 Aug.) . Regarding claim 15, the combination of Gonzalez-Garcia and Girshick disclose the claim limitations with respect to claim 4 as described above. Additional limitations are the corresponding method limitations to claim 9 and are rejected for similar reasons . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kilickaya (Kilickaya M, Ikizler-Cinbis N, Erdem E, Erdem A. Leveraging captions in the wild to improve object detection. InProceedings of the 5th Workshop on Vision and Language 2016 Aug (pp. 29-38).) Captions are used to help in object detection. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEREDITH TAYLOR whose telephone number is (571)270-5805. The examiner can normally be reached M-Th 7:30-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at (571)272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MEREDITH TAYLOR/Examiner, Art Unit 2671 /VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671 Application/Control Number: 18/562,771 Page 2 Art Unit: 2671 Application/Control Number: 18/562,771 Page 3 Art Unit: 2671 Application/Control Number: 18/562,771 Page 4 Art Unit: 2671 Application/Control Number: 18/562,771 Page 5 Art Unit: 2671 Application/Control Number: 18/562,771 Page 6 Art Unit: 2671 Application/Control Number: 18/562,771 Page 7 Art Unit: 2671 Application/Control Number: 18/562,771 Page 8 Art Unit: 2671 Application/Control Number: 18/562,771 Page 9 Art Unit: 2671 Application/Control Number: 18/562,771 Page 10 Art Unit: 2671 Application/Control Number: 18/562,771 Page 11 Art Unit: 2671 Application/Control Number: 18/562,771 Page 12 Art Unit: 2671 Application/Control Number: 18/562,771 Page 13 Art Unit: 2671 Application/Control Number: 18/562,771 Page 14 Art Unit: 2671 Application/Control Number: 18/562,771 Page 15 Art Unit: 2671
Prosecution Insights