DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner Note: Spoke to the office of the attorney of record. The attorney handling this application is on vacation until June 26th.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a first learned tokenization neural network block” in claim 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10 - 12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 10 line 2 : “encoding neural network” should be “text-video fusion neural network”. It is not clear if this is a mistype or second neural network.
Claim 10 line 8-9: recites a plurality of elements each with an “and” between them. It is not clear if this is a list of4 elements or a list of 2 elements. It is not clear if this is a grammatical error or intentional.
Claim 10 last 2 limitations recites “generating… the attention maps” and “the attention map” claim 7 also generates attention map. It is not clear if this is two different attention maps or the same attention map
Claim 12 recites a plurality of elements each with an “and” between them. It is not clear if this is a list of4 elements or a list of 2 elements. It is not clear if this is a grammatical error or intentional.
Claim 11 is rejected as dependent upon a rejected claim.
Claim Objections
Claims 1-21 are objected to because of the following informalities:
Claim 1 and 15 recites “(i) video sequence”. This should be “(i) a video sequence”.
Claims 2-12,14, 16-21 are objected to as dependent upon an objected claims.
Appropriate correction is required.
Allowable Subject Matter
Claims 1-12 and 14-21 are allowed. (Note: Claims 10-12 are rejected under 35 USC 112 and would be allowable if they overcome the respective rejections above.)
The following is an examiner’s statement of reasons for allowance:
No Prior Art reads on the claims as currently written, in particular the combination of muliti-scale, video and text and fixed number of tokes attention map for each spatial scale.
The closes Prior Art of Record is as follows:
Jang (“TGIF-QA: Towards Spatio-Temporal Reasoning in Visual Question Answering”) teaches 1. (Original) A method performed by one or more computers, the method comprising:
obtaining a network input comprising:
(i) video sequence comprising a plurality of video frames, and (ii) an input text sequence comprising a plurality of text tokens; (Jang, Section 3, QA pairs and animated GIFS)
processing the input text sequence using a language encoder neural network to generate language features that comprise a sequence of tokens representing the input text sequence; (Jang, Section 4.2, Text encoder)
generating a set of one or more video streams from the video sequence, each video stream representing the video sequence at a corresponding space-and-time scale; (Jang, Section 4.2, Video encoder)
for each video stream in the set, processing the video stream using a video encoder neural network to generate respective video stream features of the video stream that include a respective video stream feature at each spatial scale in a set of one or more spatial scales; (Jang only disclose a single scale)
processing the language features and the respective video stream features of each video stream using a text-video fusion neural network to generate a feature representation of the network input, wherein the feature representation comprises a plurality of tokens, the processing comprising:
for each video stream, processing the respective video stream features of the video stream and the language features using a first learned tokenization neural network block to map the respective video stream features of the video stream to a respective fixed number of tokens for each spatial scale; and (Jang does not disclose this limitation)
processing the feature representation using a decoder neural network to generate a network output for the network input. (Jang, Section 4.3,”Answer Decoder”)
Roo (“TokenLearner:Adaptive Space time tokenization for Videos”) teaches only the video processing portion of “processing the language features and the respective video stream features of each video stream using a text-video fusion neural network to generate a feature representation of the network input, wherein the feature representation comprises a plurality of tokens, the processing comprising:
for each video stream, processing the respective video stream features of the video stream and the language features using a first learned tokenization neural network block to map the respective video stream features of the video stream to a respective fixed number of tokens for each spatial scale; and “ (Roo, Fig. 1,
PNG
media_image1.png
496
960
media_image1.png
Greyscale
) Roo in view of Jang teaches the claimed invention except for the multiple scales
Yan(“Multi-view Transformers for Video Recognition”) discloses a transformer using multiple scales. (See Abstract)
Arnab (“ViViT : A video vision transformer”) discloses a transformer based video classification using spatio-temporal tokens. Arnab is similar to Yan
Piergiovanni(“Video Question Answering with Iterative Video-Text Co-tokenization”) discloses the whole claimed invention, but is not prior art based on its publication date.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GANDHI THIRUGNANAM whose telephone number is (571)270-3261. The examiner can normally be reached M-F 8:30-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GANDHI THIRUGNANAM/Primary Examiner, Art Unit 2672
Prosecution Insights