DETAILED ACTION
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sun ( CN112733533A).
Regarding claim 1, Sun teaches a computer-implemented method, comprising:
obtaining a first set of embeddings, generated based on a first input( page 4, T in equation (1) ), and a second set of embeddings generated based on a second input( page 4, v in in equation (1) );
generating, based at least in part on the first set of embeddings and the second set of embeddings, a third set of embeddings including one or more placeholder values associated with one or more values removed from at least the first set of embeddings or the second set of embeddings( Page 4, step 1.2, generating a word mark sequence by a BERT marker, and decomposing an unknown word into a plurality of word segment marks); and
reconstructing at least a portion of the first input or the second input based on the third set of embeddings( page 5, step 1.4 … finally, inputting the output of the label T fused with the visual clue into a named entity recognition model).
Regarding claim 2, Sun teaches the computer-implemented method of claim 1, wherein generating the third set of embeddings comprises: removing at least a subset of the one or more values of the first set of embeddings and the second set of embeddings; and replacing the removed subset of the one or more values with the one or more placeholder values ( Page 5, equation (3); a text-to-image relationship propagates through R … when its parameter is true, otherwise takes a value of O; the visual function is discarded when the probability gate G is 0, or the visual function is selected when the probability gate G is 1).
Regarding claim 3, Sun teaches the computer-implemented method of claim 1, wherein generating the third set of embeddings comprises: assigning modality-specific positional embeddings to the first set of embeddings and the second set of embeddings, wherein positional embeddings for the first set of embeddings are determined based on the sequential position of words in the text input ( page 4, = w1, …, Wn Represents a language feature sequence (T = word embedding+ segment embedding+ position embedding) and the positional embeddings for the second set of embeddings are based on spatial positions of image regions within the two-dimensional image input ( v = V1, …,Vm Denotes a visual feature sequence, (V = word embedding + segment embedding + position embedding).
Regarding claim 4, Sun teaches the computer-implemented method of claim 1, further comprising: predicting one or more values corresponding to known values associated with the first set of embeddings and the second set of embeddings, wherein the reconstructing is based on replacing the one or more placeholder values with the one or more predicted values( page 4, step 1.2, generating a word mark sequence by a BERT marker, and decomposing an unknown word into a plurality of word segment marks).
Regarding claim 5, Sun teaches the computer-implemented method of claim 1, further comprising: extracting context information from at least one of the first set of embeddings and the second set of embeddings( Page 4, step 1.3, representing the visual features as block regions, extracting the visual features from the image by ResNet) ; predicting, based at least in part on the context information, one or more values to replace the one or more placeholder values of the third set of embeddings ( Page 5, step 1.4 … using the probability gate G shown in FIG. 1 to generate the probability [ rr0, rr1 ]Then the probability that the text image correlation score is defined as positive is:
r=rr1 (2)
the visual mask matrix R is constructed using the correlation scores:… ).
Regarding claim 6, Sun teaches the computer-implemented method of claim 1, further comprising: determining, based at least in part upon a first loss function ( page 6, equation (7)) and a second loss function( page 7, equation (9)), one or more values corresponding to the one or more placeholder values to replace the one or more placeholder values, wherein the first loss function is utilized to determine one or more words to replace the one or more placeholder values of the third set of embeddings ( page6, step 3.1 … classifying text-image relation; performing text-image relationship classification using image task segmentation of the penbo dataset, the classification attempting to determine whether the content of the image provides additional information beyond text), and wherein the second loss function is utilized to determine pixel values for one or more image regions to replace the one or more placeholder values of the third set of embeddings(Page 7, step 3.2.3 … Y is the tag sequence, wherein Y is all possible tag sequences of the sentence, and s (H; Y) is the feature function modeling conversion).
Regarding claim 7, Sun teaches the computer-implemented method of claim 1, wherein the second set of embeddings are generated based at least in part on converting the image input to a two-dimensional representation ( page 4, v = V1,…,Vm Denotes a visual feature sequence), and assigning one or more sequential numbers corresponding to one or more positional values in the two-dimensional representation( page 5, Arranged as an image block embedding sequence b1 =f1,1 wv, ... , b49=f7,7wv Therein of… ).
Claims 8-20 recite the system and medium for the method in claims 1-7. Since the method in claims 1-7 is inherently running in a system with processor(s) and medium, those claims are also rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANGENG SUN whose telephone number is (571)272-3712. The examiner can normally be reached 8am to 5pm, EST, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Randolph Vincent can be reached at 571 272 8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIANGENG SUN
Examiner
Art Unit 2661
/Jiangeng Sun/Examiner, Art Unit 2671
Prosecution Insights