DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 5, 10, 11, 15, and 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chen et al. (US Pub 2017/0124432).
Regarding claim 1, Chen discloses a computer-implemented method, the computer-implemented method comprising:
generating, by one or more processors and using one or more first layers of a multi-modal machine learning model, one or more text-based intermediate representations for an entity based on textual input data (para 0053 – dense questions being intermediate representation);
generating, by the one or more processors and using one or more second layers of the multi-modal machine learning model, one or more image-based intermediate representations for the entity based on the one or more text-based intermediate representations and one or more input images for the entity (para 0027, 0043-0046 – using the intermediate representation to condition how images features are processed);
generating, by the one or more processors and using one or more third layers of the multi-modal machine learning model, an entity representation summary based on the one or more image-based intermediate representations and an image narrative summary for the one or more input images (para 0059-0062 – “h denotes the final projected feature” which is considered the entity representation summary); and
initiating, by the one or more processors, the performance of a prediction-based action based on the entity representation summary (para 0062 – “answer generated by ABC-CNN is the word with the maximum probability”).
Regarding claim 5, Chen discloses wherein the one or more first layers, the one or more second layers, and the one or more third layers of the multi-modal machine learning model are trained end-to-end using a labeled training dataset (para 0027, 0068).
Regarding claim 10, Chen discloses further comprising: augmenting the textual input data with the entity representation summary (para 0059-0062).
Regarding claims 11 and 17, see rejection of claim 1.
Regarding claim 15, see rejection of claim 5.
Allowable Subject Matter
Claims 2-4, 6-9, 12-14, 16, and 18-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAFIZ E HOQUE whose telephone number is (571)270-1811. The examiner can normally be reached M-F 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ahmad Matar can be reached at (571)272-7488. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NAFIZ E HOQUE/ Primary Examiner, Art Unit 2693