DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed March 17, 2026 have been fully considered and are persuasive regarding the rejection under 35 U.S.C. 112(b), which has been withdrawn, but they are not persuasive regarding the rejections of the claims under 35 U.S.C. 103.
With regard to the rejection of claim 1 under 35 U.S.C. 103 over Kholsa in view of Li, Applicant argues that “Li's text transformer receives text that already exists alongside the image rather than generating text describing features of the visual digital content using a perception tool. As such, Applicant submits that Li does not disclose, teach, or suggest ‘generating, using a verbalization model including at least one perception tool, verbalization tokens by generating text describing at least one feature associated with the item of visual digital content,’ as recited in amended claim 1.”
The examiner disagrees. During the interview, the examiner indicated that the proposed amendment to claim 1, which was identical to the present amendment, would likely overcome the rejection, but that the examiner needed to take a closer look at Li to determine if Applicant’s characterization of Li set forth in the above-quoted language was accurate. The examiner also indicated that if Applicant’s characterization of Li was correct, a further search would need to be conducted.
Upon a further review of Li, the examiner finds that Applicant’s characterization of Li in this regard is inaccurate for two reasons. First, even if Li transforms text that already exists alongside the image, that text is nonetheless part of the “visual digital content”, and therefore Li does disclose "generating, using a verbalization model including at least one perception tool, verbalization tokens by generating text describing at least one feature associated with the item of visual digital content," as recited in amended claim 1. (Emphasis Added). It should be noted that claim 1 does not limit the “visual digital content” to exclusively image data, and therefore text accompanying image data is “at least one feature associated with the item of visual digital content”.
Secondly, para. [0032] of Li states that “[t]he CapFilt method, which is described in co-pending and commonly owned U.S. nonprovisional application Ser. No. 17/745,540, filed May 16, 2022, may be applied to create synthetic captions for the web images.” The CapFilt method described in U.S. nonprovisional application Ser. No. 17/745,540, which corresponds to U.S. Publ. Appl. No. 2023/0237772 A1, also to Li et al. (hereinafter referred to as “Li 2”), discloses a system that uses a captioner model and a filter model to generate a synthetic textual caption of an image from a web image. The caption model is an image-grounded text decoder 230 that generates a synthetic caption from the image (paras. [0022]-[0023] and para. [0037]: “[t]he finetuned captioner 230 may generate, given the web images 403a Iw, synthetic captions Ts with one caption per image, e.g., {(Iw,Ts)} 406).”). It should be noted that the image-grounded text decoder 230 generates a synthetic caption from the image, not from the caption accompanying the image. Therefore, for this additional reason, Li does disclose generating, using a verbalization model (i.e., the caption model of Li 2), verbalization tokens by generating text describing at least one feature associated with the item of visual digital content”, as recited in claim 1.
Regarding the “perception tool” recited in claim 1, the BRI for this element, based on para. [0038] of the present specification, is that the tool detects a feature and provides a textual description: “[i]n general, detection perception tools include tools configured to detect a feature (e.g., an object, an emotion, etc.) and to provide a textual description of the feature.” Since the caption model of Li and Li 2 provides a textual description of detected features of an image, the caption model includes “at least one perception tool”.
Applicant also argues that Kholsa does not disclose this limitation. The examiner did not rely on Kholsa as teaching this limitation, but rather relied on Li as teaching this limitation.
Regarding amended claim 9, Applicant argues that Kholsa does not teach "a long-term memorability training set including real-world content memorability study results analyzing long-term viewer recall of the digital visual content over a duration of at least 12 hours," as recited in amended claim 9. The examiner agrees. The rejection of claim 9 is withdrawn. However, a new rejection is set forth below.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
In the following, some of the terms in the claims have been given BRIs in light of the specification. These BRIs are used for purposes of searching for prior art and examining the claims, but cannot be incorporated into the claims. Should Applicant believe that different interpretations are appropriate, Applicant should point to the portions of the specification that clearly support a different interpretation.
Language token: para. [0038], text to be used as input to a trained machine learning model that generates an output;
Natural language processing (NLP) model: para. [0003], a large language model;
Visual or vision encoder: para. [0039], a trained AI model that generates visual embeddings that represent content of an image that is input to the model.
Verbalization model: para. [0040], a model that generates textual output representing a description of video or image data that is input to the model;
Verbalization token: para. [0040], the textual output of the verbalization model;
Visual encoding token: para. [0038], a visual embedding output from a visual encoding model and that represents the content of the input image inputted to the visual encoding model;
Memorability dataset: para. [0084], a dataset generated based on studies of the ability of individuals to remember information relating to advertising content at a point in time after consuming the content, such as one to five days;
Memorability factor: paras. [0074]-[0075], a factor that contributed to the memorability prediction (score); e.g., the logo may be output along with the score as a factor that materially affected the memorability score.
The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f), is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f), because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“memorability prediction module” in claims 1 and 15.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. The BRI for memorability prediction module, based on para. [0006] of the present disclosure, is a processor that executes computer instructions stored in a memory device to perform the memorability prediction model operations recited in claims 1 and 15, and equivalents.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 6-8, 15-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over an article entitled “Understanding and Predicting Image Memorability at a Large Scale”, by Kholsa et al., published in 2015 in 2015 IEEE International Conference On Computer Vision (hereinafter referred to as “Kholsa”) in view of U.S. Publ. Appl. No. 2024/0161520 A1 to Li et al. (hereinafter referred to as “Li”) and further in view of U.S. Publ. Appl. No. 2023/0237772 A1, also to Li et al. (hereinafter referred to as “Li 2”).
Regarding claim 1, Kholsa discloses a computer-implemented method that determines, using a memorability prediction module (section 1. Introduction” “[b]y fine-tuning Hybrid-CNN [37], a convolutional neural network (CNN) [23, 21] trained to classify more than a thousand categories of objects and scenes, we show that our model, MemNet, achieves a rank correlation of 0.64 on novel images, reaching near human consistency rank correlation (0.68) for memorability. By visualizing the learned representation of the layers of MemNet, we discover the emergent representations, or diagnostic objects, that explain what makes an image memorable or forgettable”), a memorability prediction for the item of digital content via providing the representations of the input images to a prediction model, the prediction module including a model (section 4.1, MemNet hybrid CNN) trained using at least one memorability dataset (sections 2 and 4.3, LaMEM dataset) to simulate memorability of digital visual content (section 2. the MemNet hybrid CNN is trained using a Large-scale Memorability Dataset (LaMEM)) to generate a memorability prediction of the digital visual content being memorable to viewers (section 4, “Predicting Memorability”; section 5, “Applications”: “[p]redicting the memorability of image regions could allow us to build tools for automatically modifying the memorability of images [17], which could have far-reaching applications in various domains ranging from advertising and gaming to education and social networking.”).
Khosla does not explicitly disclose generating and inputting language tokens to the MemNet hybrid CNN model, but it is well known in the art that images to be processed in CNNs are first encoded into embeddings, or vectors, by some type of visual encoder that is either part of the CNN or is connected to the input of the CNN.
Kholsa also does not explicitly disclose that the MemNet model is an NLP model. As indicated above, the BRI for NLP model is a large language model (LLM).
Li, in the same field of endeavor of processing images using machine learning models, discloses a computer-implemented method (para. [0058]-[0066], Fig. 6), comprising:
generating, using a visual encoding model, visual encoding tokens computed by a visual encoding model, the visual encoding tokens representing an item of visual digital content in a language space (first pre-training stage 101, Fig. 1, constitutes a visual encoding model; the output of the image encoder 110 of stage 101 comprises “image representations” that are processed by an image transformer 210 of Fig. 2 of a querying transformer 120 of Fig. 1 of the stage 101 to produce transformed image representation embeddings Z, which constitute visual encoding tokens, paras. [0034]-[0037]),
generating, using a verbalization model including at least one perception tool, verbalization tokens by generating text describing at least one feature associated with the item of visual content (paras. [0031]-[0032] of Li disclose that the image-grounded text decoder 230 of Li 2 is a caption model that generates a synthetic caption 105b by performing the CapFilt method of Li 2; Li discloses that the caption model 230 of Li 2 can be used in the system of Li on the input image 105a; the combination of the caption model 230 of Li 2 and the transformer 120 of Li constitutes a verbalization model; in this case, the synthetic caption 105b generated by the verbalization model is fed into the transformer submodule 220 of the transformer 120; the transformer submodule 220 processes the caption 105b in the self-attention layers 221 and generates the verbalization tokens);
generating language tokens representing the item of visual content by combining the visual encoding tokens and the verbalization tokens (Fig. 2 of Li, the Image-Text Contrastive Learning (ITC) module 232 combines the visual encoding tokens output by the transformer submodule 210 and the verbalization tokens output by transformer submodule 220); and
determining, using a prediction module, a prediction for the item of digital content via providing the language tokens as input to a prediction model (the large language model (LLM) 130 of Li receives the language tokens comprising the visual encoding tokens and the verbalization tokens output from the transformer submodules 210 and 220 of the querying transformer 120, respectively, and determines a prediction of output text describing the digital content, para. [0089], Fig. 8, step 811), the prediction module including a natural language processing (NLP) model (LLM 130 is an NLP model, para. [0004]: “[t]he embodiments relate generally to natural language processing and machine learning systems….”) trained using at least one dataset (para. [0032]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to use the visual encoder model, the verbalization model and the LLM shown in Figs. 1 and 2 of Li and in Fig. 2 of Li 2 in the system of Kholsa, with the LLM 130 of Li being trained on the LaMem dataset of Khosla to perform memorability prediction based on the language tokens output from the querying transformer 120 of Li. One of ordinary skill in art would have been motivated to use these components of Li/Li 2 in the system of Kholsa in this manner to enable the system of Khosla to generate captions for images that accurately convey what is shown in the images. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (combining known components 110, 120 and 130 according to known techniques and training the LLM 130 using the LaMem dataset to predict memorability based on the language tokens output from the querying transformer 120).
Regarding claim 2, Kholsa does not explicitly disclose a visual encoding model that comprises a vision transformer (ViT) encoder and a querying transformer model.
Li discloses that the visual encoding model 101 comprises:
a vision transformer (ViT) encoder 110 trained to generate visual embeddings from the item of digital visual content (para. [0033]: “the image encoder 119 [sic] may be pre-trained vision transformer models, such as ViT-L/14 from CLIP (Radford et al., Learning transferable visual models from natural language supervision, arXiv preprint arXiv:2103.00020, 2021)”), and
a querying transformer model (Fig. 1, querying transformer 120, para. [0026]: “[t]he multi-modal vision-language model that comprises an image encoder 110, a query Transformer 120 and a (large) language model (LLM) 130 ….”), trained to receive the visual embeddings as input and to generate the visual encoding tokens as output, the visual encoding tokens including a description of the item of digital visual content in the language space for use by the NLP model (Figs. 1 and 2, paras. [0026]-[0047], the querying transformer 120 is trained to receive visual embeddings output from visual encoder 110 and to generate the visual encoding tokens in transformer submodule 210; the visual encoding tokens are output to the LLM 130 along with the text description output from transformer submodule 220).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to use the ViT transformer encoder 110 and the querying transformer model 120 combination of Li as the visual encoding model of the system of Kholsa to generate the visual encoding tokens to be processed by the LLM 130 of Li trained on the LaMem dataset of Kholsa to perform memorability prediction. As indicated above in the rejection of claim 1, one of ordinary skill in art would have been motivated to configure the system of Kholsa in this manner to enable the system of Khosla to generate accurate captions for input images. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (combining known components 110, 120 and 130 according to known techniques and training the LLM 130 using the LaMem dataset to predict memorability based on the language tokens output from the querying transformer 120).
Regarding claim 6, Kholsa discloses that the memorability prediction includes a memorability score (section 2.2 discusses memorability prediction scores associated with the images of the dataset), and that the model is trained to determine at least one memorability factor contributing to the memorability score (section 3.2 discusses memorability factors contributing to memorability score, referred to in Kholsa as image attributes such as popularity, saliency, emotions and aesthetics).
Regarding claim 7, Kholsa discloses sending the memorability score and the at least one memorability factor to an electronic display of a client device (section 5; Fig. 7 shows images that are displayed along with the corresponding normalized memorability scores and the memorability factor, such as indications of whether portions of the image have been emphasized or de-emphasized and the resulting memorability prediction score).
Kholsa does not explicitly disclose that the memorability scores and associated factors are sent to a network interface for presentation on a GUI of an electronic display of a client device. Li discloses sending the image along with the decoded textual description produced by the LLM 130 to a client device via a network interface to be displayed on a GUI of the client device (para. [0071], Fig. 7, client device 710 having GUI 712 receives the output of LLM 130 sent over network interface 760, 717, 733 and displays it to the user).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to include the user device 710 and network interface 717/733/760 of Li in the system of Kholsa to allow the memorability scores and the corresponding images and memorability factors to be sent and presented to the user. One of ordinary skill in art would have been motivated to include the user device 710 and network interface 717/733/760 of Li in the system of Kholsa to allow a user when selecting images to be used for advertising to determine what factors contributed to the scores and to allow the user to select images for an advertising campaign that have the memorability factors. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (providing a network interface to the system Kholsa to allow memorability scores and factors to be sent to client devices such as personal computers a smartphones for display on the GUIs of client devices).
Regarding claim 8, the MemNet model of Kholsa is a pre-trained model. Section 4.1 discusses the model being pre-trained on multiple datasets. As indicated above, Kholsa does not explicitly disclose the model being an LLM or using language tokens. Li discloses the LLM 130 being pre-trained (para. [0024]) and using language tokens, as discussed above in the rejection of claim 1. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to use the pre-trained LLM 130 of Li in the system of Kholsa with the LLM 130 being pre-trained to perform memorability prediction for the same reasons described above in the rejection of claim 1.
Regarding claims 15, 16, 19 and 20, the rejections of claims 1, 2, 6 and 7 apply mutatis mutandis to claims 15, 16, 19 and 20, respectively. The system of Kholsa necessarily has some type of computer-readable medium for storing the LaMem dataset and software for implementing the hybrid MemNet CNN.
Regarding claim 17, under MPEP 2111.04, claim scope is not limited by claim language that suggests or makes optional but does not require steps to be performed. In addition, when a claim requires selection of an element from a list of alternatives, the prior art teaches the element if one of the alternatives is taught by the prior art. Fresenius USA, Inc. v. Baxter Int’l, Inc., 582 F.3d 1288, 1298, 92 USPQ2d 1163, 1171 (Fed. Cir. 2009). The BRI for the claim limitation “the at least one perception tool includes one or more of optical character recognition (OCR), audio and speech recognition (ASR), text-to-speech, object detection, emotion detection, color detection, or aesthetics detection” is that the claim requires any one of the listed items.
Kholsa does not explicitly disclose any of the listed perception tools. The system of Li performs object detection because it generates a textual caption based on objects that are detected in the input images 105a. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to perform object detection as taught by Li in the system of Kholsa to perform memorability predictions in Kholsa. One of ordinary skill in art would have been motivated to make the modification to enable the system of Khosla to generate captions for objects contained in images as taught by Li/Li2, which would improve memorability of digital visual content accompanied by accurate captions. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (combining known components 110, 120 and 130 according to known techniques and training the LLM 130 using the LaMem dataset to predict memorability based on the language tokens output from the querying transformer 120).
Claims 4, 9-14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kholsa in view of Li and Li 2 as applied to claims 1, 2, 6-8, 15-20 and 19-20 and further in view of an article entitled “Recognition of Advertisement Emotions with Application to Computational Advertising”, by Shukla et al., published in April 2019 in ARXIV ID: 1904.01778 (hereinafter referred to as “Shukla”).
Regarding claim 4, Kholsa does not explicitly disclose that the memorability dataset includes a long-term memory dataset generated based on a long-term memorability study of viewer memorability of digital visual content over a long-term duration of at least 1 day to about 5 days. Shukla, in the same field of endeavor, discloses using a memorability dataset that is generated based on memorability studies of viewer intermediate and long-term memorability of digital visual content, where the long-term studies corresponded to a long-term duration of at least 1 day to about 5 days (section 6.4: “[b]ased on viewers’ questionnaire responses, we computed the mean proportions for correct recall, ad forgottenness, incorrect recall and good insertions immediately and a day after the experiment”; Fig. 4 shows a day-after ad recall bar chart; the Introduction discusses the dataset that was generated based on the immediate and “long-term” memorability studies and section 6.3 discusses the “long-term (day-after)” memorability studies).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to generate the memorability dataset of Kholsa based on the long-term memorability studies performed in Shukla. One of ordinary skill in art would have been motivated to make the modification to enable the system of Khosla to be trained and used to measure long-term memorability of ads. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (generating a dataset that also trains the prediction model to predict long-term memorability).
Regarding claim 9, Kholsa discloses training the MemNet model on first ILSVRC and Places training datasets (section 4.1) and then further training the MemNet model using a second LaMem training dataset (section 4.3), which, as indicated above, is a memorability training set comprising real-world content memorability study results analyzing viewer recall of the digital visual content, where the training using the first and second datasets configures the MemNet model to simulate viewer memorability of digital visual content (section 2.1 discusses the real-world images that are used in the database and section 2.2 discusses a visual memory game experiment that was used to generate memorability prediction scores by showing the images to viewers; some type of processor and memory are used in Khosla for executing and storing the MemNet model instructions and the LaMem dataset).
Kholsa does not explicitly disclose that the training configures the MemNet model to simulate viewer memorability of digital visual content transformed into a language space. Kholsa also does not explicitly disclose that the first training is training of an NLP model using a first training data set comprising a visual encoder embeddings training set comprising visual encoding tokens computed by a visual encoding model based on a training set of digital visual content. However, as indicated above, some type of visual encoder is necessarily used in Kholsa to transform the image data into visual encoding embeddings suitable for being input a CNN since the MemNet model is a CNN.
In Li, the NLP model 130 is pre-trained using a process that involves inputting images into the visual encoder model 110, which produces visual encoder embeddings that are then input to the querying transformer 120. As discussed above, the transformer submodules 210 and 220 then generate visual encoder tokens and verbalization tokens, respectively, that together comprise the language tokens that are input to the NLP model 130. During pre-training of the NLP model 130, the model 130 updates its parameters based on the language tokens received from the querying transformer 120 to train it to generate text associated with the images in the language space. After pre-training the NLP model 130, the two-stage pretraining process of Li is performed during which the parameters of the visual encoder model 110 and the NLP model 130 are frozen as the parameters of the querying transformer 120 are updated (paras. [0024]-[0025]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to use the NLP pre-training process of Li as the first pre-training process of Khosla to train the NLP model 130 of Li using the ILSVRC and Places datasets of Kholsa based on the language tokens output from querying transformer 120 of Li. One of ordinary skill in art would have been motivated to make the modification to enable the system of Khosla to generate accurate captions for input images, which, as indicated above, would improve memorability of digital content by captioning it with accurate captions. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (combining known components 110, 120 and 130 according to known techniques and training the LLM 130 using the first ILSVRC and Places datasets in first pre-training and using the second LaMem dataset in second training to predict memorability based on the language tokens output from the querying transformer 120).
Kholsa also does not explicitly disclose that the second dataset includes a long-term memorability dataset corresponding to long-term memory of a duration of at least 12 hours. Shukla discloses using a memorability dataset that is generated based on memorability studies of viewer intermediate and long-term memorability of digital visual content, where the long-term studies corresponded to memorability of at least 12 hours (section 6.4: “[b]ased on viewers’ questionnaire responses, we computed the mean proportions for correct recall, ad forgottenness, incorrect recall and good insertions immediately and a day after the experiment”; Fig. 4 shows a day-after ad recall bar chart; the Introduction discusses the dataset that was generated based on the immediate and “long-term” memorability studies and section 6.3 discusses the “long-term (day-after)” memorability studies).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to generate the second training dataset of Kholsa based on the long-term memorability studies performed in Shukla. One of ordinary skill in art would have been motivated to make the modification to enable the system of Khosla to be trained and used to measure long-term memorability of ads. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (generating a dataset that also trains the prediction model to predict long-term memorability).
Regarding claim 10, as indicated above, Kholsa does not explicitly disclose using a long-term memory dataset or training the memorability model to predict long-term memorability of at least twelve hours. The memorability dataset of Shukla includes experimental information indicating properties of a real-world content memorability study (section 6.4, the intermediate and long-term memorability dataset and corresponding trained model are based on real-world experimental studies of users’ ability to recall ads immediately after they are shown the ads as we as their ability to recall the ads a day later, as indicated in the bar charts of Fig. 4 that are generated based on the experimental results).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to generate the second training dataset of Kholsa based on the long-term memorability studies performed in Shukla. One of ordinary skill in art would have been motivated to make the modification to enable the system of Khosla to be trained and used to measure long-term memorability of ads. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (generating a dataset that also trains the prediction model to predict long-term memorability).
Regarding claim 11, the rejection of claim 1 applies mutatis mutandis to claim 11. As indicated above, Kholsa discloses that when the processor executes the MemNet hybrid CNN model, the model simulates the viewer memorability by determining a memorability prediction for an item of digital content input to the model (section 4, Predicting Memorability).
Regarding claim 12, the rejection of claim 1 applies mutatis mutandis to claim 12.
Regarding claim 13, the rejection of claim 2 applies mutatis mutandis to claim 13.
Regarding claim 14, the rejection of claim 17 applies mutatis mutandis to claim 14.
Regarding claim 18, the rejection of claim 4 applies mutatis mutandis to claim 18.
Claims 21 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Kholsa in view of Li and Li 2 as applied to claims 1, 2, 6-8, 15-17 and 19-20 and further in view of U.S. Pat. No. 11,922,675 B1 to Saraee et al. (hereinafter referred to as “Saraee”).
Regarding claim 21, Kholsa does not explicitly disclose that the memorability prediction further includes content recommendations for modifying the item of visual digital content to increase the memorability score. Saraee, in the same field of endeavor, discloses a system 1000 that uses a neural network model trained and configured to process images to determine which images are more likely to “produce greater brand recall, memorability, or sales performance for the brand product” and then “recommend to the user to instead use an alternative image, e.g. a family photo.” (Col. 85, lines 32-50; Col. 119, lines 44-58). In Saraee, the neural network processes images of visual digital content that can be considered ads and generates a performance score for the ads and selects ads to be used based on which images produce the highest performance scores (Col. 2, lines 1-29). Since the performance scores are based on “recall, memorability or sales performance”, they constitute memorability scores.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to train the prediction model of Kholsa based on the teachings of Saraee to include content recommendations with the memorability predictions for modifying the item of visual digital content in order to increase the memorability score as taught by Saraee. One of ordinary skill in art would have been motivated to make the modification to enable the system of Khosla to not only predict content memorability, but to also recommend other content that may have better memorability. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (generating a dataset that also trains the prediction model to recommend content based on predicted memorability).
Regarding claim 22, Kholsa does not explicitly disclose providing context information to the memorability model that includes one or more of an intended audience for the item of visual digital content or a publication platform for the item of visual digital content. In Saraee, a content evaluation system 1005 trains the neural network “to simulate the target audience when generating performance scores for images” (Col. 119, line 44-Col. 120, line 38). Therefore, the system of Saraee provides context information to the neural network in the form of intended audiences when training the system to generate performance scores for visual content.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to train the prediction model of Kholsa based on the teachings of Saraee to provide context information including intended audiences to the model for items of visual digital content during training as taught by Saraee. One of ordinary skill in art would have been motivated to make the modification to enable the system of Khosla to predict content memorability tailored to intended audiences as taught by Saraee. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (generating a dataset that also trains the prediction model to predict memorability of intended audience members for visual digital content).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights