DETAILED ACTION
[1] Remarks
I. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
II. Claims 1-20 are pending and have been examined, where claims 1-6, 8-16 and 18-20 is/are rejected, claim 7 and 17 is/are objected to. Explanations will be provided below.
III. Inventor and/or assignee search were performed and determined no double patenting rejection(s) is/are necessary.
IV. Patent eligibility (updated in 2019) shown by the following: Claims 1-20 pass patent eligibility test because there is/are no limitation or a combination of limitations amounting to an abstract idea. Also, the following limitation or the combinations of the limitations: “inputting, by the at least one processor, the at least one image and the at least one textual description into at least one machine learning model to produce a representation of a degree of significance of at least one portion of the at least one image to the at least one identification of the at least one item in the at least one textual description; inputting, by the at least one processor, the at least one image into an expert system to output at least one label representative of the at least one item; and modifying, by the at least one processor, the at least one textual description of the caption to comprise the at least one label so as to produce a modified caption associated with the at least one item” effects a transformation or a reduction of a particular article to a different state or thing / adds a specific limitation(s) other than what is well-understood, routine and conventional in the field, or adding unconventional steps that confine the claim to a particular useful application and providing improvements to the technical field of deep learning, which recite additional elements that integrate the judicial exception into a practical application and amounting significant more.
[2] Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Use of the word “means” (or “step for”) in a claim with functional language creates a rebuttable presumption that the claim element is to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is invoked is rebutted when the function is recited with sufficient structure, material, or acts within the claim itself to entirely perform the recited function. Absence of the word “means” (or “step for”) in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim elements in this application that use the word “means” (or “step for”) are presumed to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Similarly, claim elements that do not use the word “means” (or “step for”) are presumed not to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
Claim(s) 11-20 are not interpreted under 35 U.S.C. 112(f) or pre-AIA U.S.C. 112 6th paragraph because of the following reason(s): limitations are modified by sufficient structure or material for performing the claimed function.
Claim(s) 1-10 do not require 35 U.S.C. 112(f) or pre-AIA U.S.C. 112 6th paragraph interpretation because they are method claims and / or they are CRM claims.
Upon examination of the specification and claims, the examiner has determined, under the best understanding of the scope of the claim(s), rejection(s) under 35 U.S.C. 112(a)/(b) is not necessitated because of the following reasons: sufficient support are provided in the written description / drawings of the invention.
[3] Grounds of Rejection
Claim Rejections - 35 USC § 102
U.S.C. 102 Conditions for patentability; novelty.
[Editor Note: Applicable to any patent application subject to the first inventor to file provisions of the AIA (see 35 U.S.C. 100 (note) ). See 35 U.S.C. 102 (pre-AIA ) for the law otherwise applicable.]
(a) NOVELTY; PRIOR ART.—A person shall be entitled to a patent unless—
(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention; or
(2) the claimed invention was described in a patent issued under section 151 , or in an application for patent published or deemed published under section 122(b) , in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
(b) EXCEPTIONS.—
(1) DISCLOSURES MADE 1 YEAR OR LESS BEFORE THE EFFECTIVE FILING DATE OF THE CLAIMED INVENTION.—A disclosure made 1 year or less before the effective filing date of a claimed invention shall not be prior art to the claimed invention under subsection (a)(1) if—
(A) the disclosure was made by the inventor or joint inventor or by another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor; or
(B) the subject matter disclosed had, before such disclosure, been publicly disclosed by the inventor or a joint inventor or another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor.
(2) DISCLOSURES APPEARING IN APPLICATIONS AND PATENTS.—A disclosure shall not be prior art to a claimed invention under subsection (a)(2) if—
(A) the subject matter disclosed was obtained directly or indirectly from the inventor or a joint inventor;
(B) the subject matter disclosed had, before such subject matter was effectively filed under subsection (a)(2), been publicly disclosed by the inventor or a joint inventor or another who obtained the subject matter disclosed directly or indirectly from the inventor or a joint inventor; or
(C) the subject matter disclosed and the claimed invention, not later than the effective filing date of the claimed invention, were owned by the same person or subject to an obligation of assignment to the same person.
Claim 1-6, 8-9, 11-16, and 18-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Munusamy (Munusamy, H., C, C.S. Multimodal attention-based transformer for video captioning. Appl Intell 53, 23349–23368, 2023).
Regarding claim 1, Munusamy discloses a method comprising:
obtaining, by at least one processor, at least one image (see Implementation details, the deep learning technique is implement using PyTorch using a processor or a GPU);
PNG
media_image1.png
58
362
media_image1.png
Greyscale
obtaining, by the at least one processor, a caption comprising at least one textual description of the at least one image (see figure 1, the semantic keywords are read as the captions associated with the images);
wherein the at least one textual description comprises at least one identification of at least one item in the at least one image (see figure 1, the keyframes are read as the images);
inputting, by the at least one processor, the at least one image and the at least one textual description into at least one machine learning model to produce a representation of a degree of significance of at least one portion of the at least one image to the at least one identification of the at least one item in the at least one textual description (see figure 1 illustration below):
PNG
media_image2.png
393
951
media_image2.png
Greyscale
;
inputting, by the at least one processor, the at least one image into an expert system to output at least one label representative of the at least one item (see figure 2, the encoder and decoder are read as the expect system); and
modifying, by the at least one processor, the at least one textual description of the caption to comprise the at least one label so as to produce a modified caption associated with the at least one item (see figure 10 illustration below):
PNG
media_image3.png
259
607
media_image3.png
Greyscale
.
Regarding claim 2, Munusamy discloses the method of claim 1, wherein the at least one machine learning model comprises at least one encoder and at decoder (see figure 2, encoder and decoder); and wherein the at least one image is input into the decoder and output by the encoder to as to produce least one heat map (see figure 8):
PNG
media_image4.png
376
836
media_image4.png
Greyscale
.
Regarding claim 3, Munusamy discloses the method of claim 1, wherein the expert system comprises at least one face system configured to output at least name associated with at least one face detected in the at least one image (see figure 2 input illustration below):
PNG
media_image5.png
149
545
media_image5.png
Greyscale
.
Regarding claim 4, Munusamy discloses the method of claim 3, further comprising:
determining, by the at least one processor, at least one person in the at least one image based at least in part on at least one word of the at least one textual description being representative of the at least one person (see figure 11 illustration below, the face of the woman is detected, where the keyword “woman” is read as at least one textual description);
determining, by the at least one processor, that the at least one person and the at least one face based at least in part on a spatial alignment (see figure 11 illustration below); and
PNG
media_image6.png
188
681
media_image6.png
Greyscale
modifying, by the at least one processor, the at least one textual description by replacing the at least one word associated with the at least one person with the at least one name associated with the at least one face to produce modified caption (see figure 11 illustration below, “a woman is showing how to make her hair” is read as the modified caption).
Regarding claim 5, Munusamy discloses the method of claim 1, further comprising:
determining, by the at least one processor, based on at least one rule, that the at least one identification of the at least one item is associated with the expert system (see equation 3, where this equation is read as the rule); and
PNG
media_image7.png
78
647
media_image7.png
Greyscale
inputting, by the at least one processor, the at least one image into the expert system in response to the at least one identification of the at least one item being associated with the expert system (a processor or GPU is employed, see figure 11, the objects identified is read as the items associated with an expert system).
Regarding claim 6, Munusamy discloses the method of claim 1, further comprising utilizing, by the at least one processor, at least one AI captioning model to generate the at least one textual description based at least in part on the at least one image (see figure 2 is read as the AI captioning model, which is based on the input video).
Regarding claim 8, Munusamy discloses the method of claim 1, further comprising: modifying, by the at least one processor, an order of words in the at least one textual description based at least in part on at least one heat map and at least one ordering rule (see figure 8 illustration below):
PNG
media_image8.png
257
971
media_image8.png
Greyscale
Regarding claim 9, Munusamy discloses the method of claim 1, wherein the at least one image comprises at least one frame of a video (see figure 2, video frame).
Regarding claim 11, see the rationale and rejection for claim 1.
Regarding claim 12, see the rationale and rejection for claim 2.
Regarding claim 13, see the rationale and rejection for claim 3.
Regarding claim 14, see the rationale and rejection for claim 4.
Regarding claim 15, see the rationale and rejection for claim 5.
Regarding claim 16, see the rationale and rejection for claim 6.
Regarding claim 18, see the rationale and rejection for claim 8.
Regarding claim 19, see the rationale and rejection for claim 9.
Claim Rejections - 35 USC § 103
1. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claims 10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Munusamy (Munusamy, H., C, C.S. Multimodal attention-based transformer for video captioning. Appl Intell 53, 23349–23368, 2023) in view of Tsai (US 20180284802).
Regarding claim 10, Munusamy discloses all the limitations of claim 1, but is silent in disclosing the method of claim 9, wherein the video comprises a live-stream. Tsai discloses the method of claim 9, wherein the video comprises a live-stream (see paragraph 128, successive camera images are analyzed at the pixel level to extract object movements and velocities. In some implementations, presentation interface 778 includes a video feed integrator provides integration of live video feed from the cameras 508).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to include the method of claim 9, wherein the video comprises a live-stream because to provide real-time, dynamic broadcasts alongside standard Video-on-Demand captioning.
Regarding claim 20, see the rationale and rejection for claim 10.
[4] Claim Objections
Claim(s) 7 and 17 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
With regards to claim 7, the examiner cannot find any applicable prior art providing teachings for the following limitation(s): “the method of claim 1, further comprising: receiving, by the at least one processor, at least one search query comprising at least one search term, determining, by the at least one processor, that the at least one search term of the at least one search query matches to the modified caption; and returning, by the at least one processor, the at least one image in response to the at least one search query based at least in part on the at least one search term matching to the at modified caption”; in combination with the rest of the limitations of claim 1.
Regarding claim 7, Munusamy discloses the method of claim 1, further comprising: receiving, by the at least one processor, at least one search query comprising at least one search term (3.4. Bimodal attention block in encoder illustration below):
PNG
media_image9.png
65
646
media_image9.png
Greyscale
Munusamy is silent in disclosing determining, by the at least one processor, that the at least one search term of the at least one search query matches to the modified caption; and returning, by the at least one processor, the at least one image in response to the at least one search query based at least in part on the at least one search term matching to the at modified caption.
Regarding claim 17 see the rationale for claim 7.
CONTACT INFORMATION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEX LIEW (duty station is located in New York City) whose telephone number is (571)272-8623 (FAX 571-273-8623), cell (917)763-1192 or email alexa.liew@uspto.gov. Please note the examiner cannot reply through email unless an internet communication authorization is provided by the applicant. The examiner can be reached anytime.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MISTRY ONEAL R, can be reached on (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEX KOK S LIEW/Primary Examiner, Art Unit 2674 Telephone: 571-272-8623
Date: 6/22/26
Prosecution Insights