DETAILED ACTION
Response to Arguments
Applicant's arguments filed with respect to claims 1-16, and 21-24 have been fully considered but are moot in view of the new ground(s) of rejection. The rejections are necessitated due to claim amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 10, 16 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755)
Regarding claim 1, Maheshwari teaches a method comprising: providing input text (textual query) and a collection of images (set of candidate images) into a cross-modal matching service (cross-modal network) [Para. 43, 53, 57 and 58], the cross-modal matching service including a fine-tuned model produced by refining a pre-trained model through additional training on a training dataset [Para. 43 states “The architecture 600 improves a ranking quality of an image retrieval process with a cross-modal network that is trained to learn ranking of images for a given textual query”; merely calling a modal “fine-tuned” doesn’t have patentable weight because any training could be considered “tine-tuned”. The claim is silent on the steps of fine-tuning the modal. Merely calling a model “.
However, Maheshwari doesn’t explicitly teach about training a pre-trained model through additional training on a training dataset.
Li teaches the cross-modal matching service (vision-and-language learning (V+L) model) including a fine-tuned model (V+L model fine-tuned on COCO) produced by refining a pre-trained model (pretrained V+L model) through additional training (fine-tuning) on a training dataset (training dataset) [Para. 46, 54, 62, and 66-68].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari’s cross-modal matching service (cross-modal network) by implementing Li’s pre-trained model (pretrained V+L model) and applying additional training (fine-tuning) on a training dataset (training dataset) to produce a “fine-tuned” model for the query-image retrieval task. This modification improves Maheshwari by transferring previously learned vision-language representations to its task-specific image ranking process, thereby improving task specific image text matching and ranking.
Maheshwari teaches the training dataset (dataset that includes the query long of the search engine) having been derived from content items each including text (features of textual queries) and an image (image search results) presented together [Para. 15, 16, and 53-56].
Maheshwari having training dataset including negative examples (Boolean click versus not-clicked label) [Para. 43, 47 and 53].
However, Maheshwari doesn’t explicitly teach examples formed by recombining text and images across different content items so that a negative example pairs text and an image that were not presented together in a content item
Li teaches the training dataset including negative examples (hard negatives) formed by recombining (sampled from the same batch) text (negative text) and images (hard negative images) across different content items (image-text pairs) so that a negative example (negative image-text pair) pairs text and an image (pair of image and text) that were not presented together in a content item [Para. 32, 38, 46, 61 and 64].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari’s training dataset by incorporating Li’s negative examples including sampling text and an image from a batch of content items to from a negative example. This modification improves Maheshwari byh supplying more informative mismatched training samples, thereby improving the model’s discrimination between relevant and irrelevant query image relationships and improving retrieval performance.
Maheshwari teaches receiving a relevance score (relevance measure (e.g. relevance score)) from the cross-modal matching service (cross—modal network) for at least one image (one or more images) in the collection of images (set of candidate images) [Para. 43, and 58-60] wherein the relevance score indicates a relevance (reflects the relative relevancy) of the at least one image with respect to the input text (textual query) [Para. 43, 57, 59 and 60]; and Selecting at least one image to associate with the input text (textual query /returned in response to the query) based on the relevance score (relevance measure (eg relevance score) [Para. 57-60].
Claims 10 and 16 are rejected for the same reason as claim 1 above. Furthermore, Maheshwari teaches a system, a processor, and a memory storing instructions [fig. 3, 8, 9 and related description].
Regarding claim 23, Maheshwari teaches wherein the training dataset includes positive examples each comprising a text and an image derived from a content item for which a desired interaction was observed when the content item was presented [Para. 27, 43, 44, and 53].
Claims 2 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Li et al. (Pub. No. US 20200250537 hereinafter Li2).
Regarding claims 2, and 11 Maheshwari in view of Li doesn’t explicitly teach the claim limitation.
However, Li2 teaches wherein the fine-tuned trained model was trained on a first dataset including pairings of images and text (query image training example), and the first dataset includes a positive example (positive label) of relevance of a first training dataset image to first paired text when a desired interaction (selected) with respect to the first training dataset image and the first paired text has been observed in the specified context (search result) in which the first training dataset image and the first paired text were presented [Para. 51, 52, and 70], wherein relevance between the at least one image and the input text is based on expectations of achieving the desired interaction in the context (search result) in which the first training dataset image and the first paired text are presented [Para. 86 and 52].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li to train the model with text, image and positive example, feature as taught by Li; because the modification enables the system to improve determination between an image and input text by training the model using observed user interaction signals as the ground truth of relevance.
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Li et al. (Pub. No. US 20200250537 hereinafter Li2) further in view of Kulkarni et al. (Pub. No. US 2018/0004847).
Regarding claims 3 and 12, Maheshwari in view of Li further in view of Li2 doesn’t explicitly teach the claim limitation.
However, Kulkarni teaches wherein the first training dataset image and the first paired text occur with respect to an item of invitational content (sponsored content) that was served to a user device by a content delivery system, and the desired interaction is a click or conversion of the item of invitational content on the user device [Para. 4, 27, and 28].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li further in view of Li2 to train the model with negative example, feature as taught by Kulkarni; because the modification enables the system to dynamically optimize content for target user.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Li et al. (Pub. No. US 20200250537 hereinafter Li2) further in view of KLOTZ, JR. (Pub. No. US 2017/0329857).
Regarding claims 4, and 13 Maheshwari in view of Li further in view of Li2 doesn’t explicitly teach the claim limitation.
However, KLOTZ teaches wherein the first training dataset image was presented in an item of invitational content and the first paired text was received in a search input of an App store, wherein the App store is the context in which they were presented [Para. 93, 94, 98, 105; fig. 4, 5 and related description; Para. 23 “In some examples, native applications may be installed on a user device 200 prior to the purchase of the user device 200. In other examples, a user device 200 may download a native application from a digital distribution platform such as the APP STORE® digital distribution platform developed by Apple Inc. or the GOOGLE PLAY® digital distribution platform developed by Google Inc. In these examples, the user device 200 downloads and installs the application at the request of a user”; Para. 7 “the application features may also include digital media related to the application 110, such as images (e.g., icons associated with the application and/or screenshots of the application) or videos (e.g., a sample video of the application)”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li further in view of Li2 to teach the claim limitation, feature as taught by KLOTZ; because the modification enables the system to personalize search results based on a device profile of user device.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Li et al. (Pub. No. US 20200250537 hereinafter Li2) further in view of Baker et al. (Patent No. US 8,503,769).
Regarding claim 5, Maheshwari in view of Li further in view of Li2 doesn’t explicitly teach the claim limitation.
However, Baker teaches wherein the first dataset includes a negative example of relevance of the first training dataset image to paired text, wherein the negative example was derived from remixing a first image with text that was not associated with the first image [Col. 3 lines 55-60; Col. 5 lines 44-65].
It would have been obvious to one of ordinary skill in the art before the effective filing date to Maheshwari in view of Li further in view of Li2 to train the model with negative example, feature as taught by Baker; because the modification enables the system to determine information about the images from the web page.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Li et al. (Pub. No. US 20200250537 hereinafter Li2) further in view of Wright et al. (Pub. No. US 2007/0156887).
Regarding claim 6, Maheshwari in view of Li further in view of Li2 doesn’t explicitly teach the claim limitation.
Wright teaches wherein the fine-tuned model was trained on a second dataset (rating data) including second pairings of images and text, and the second dataset includes human labeled (rating) positive examples of relevance of a second training dataset image to second paired text, and human labeled negative examples of relevance of a third training dataset image to third paired text [Para. 27, 31, 47 and 48].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li further in view of Li2 to train the model with negative example, feature as taught by Wright; because the modification enables the system to determine information about the images from the web page.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Kaftan et al. (Pub. No. US 2015/0324868).
Regarding claims 7, and 14 Maheshwari in view of Li doesn’t explicitly teach the claim limitation.
However, Kaftan teaches wherein the selecting the at least one image to associate with the input text further comprises: receiving, the input text as a search query entered into an App store: generating invitational content (an ad) including the at least one image and the input text, wherein the at least one image is a most relevant image to the input text in the collection of images, wherein the input text is a search term expected to be used in an App store, and the collection of images are images relevant to an App to be presented by the invitational content, presenting the invitational content in response to the search query [Abstract, Para. 3, 4, 7, 26 and 51 “The application features may also include digital media related to the application, such as images (e.g., icons associated with the application and/or screenshots of the application)”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li to teach the claimed limitation, feature as taught by Kaftan; because the modification enables the system to determine a query categorization of a search query.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Kaftan et al. (Pub. No. US 2015/0324868) and further in view of Kulkarni et al. (Pub. No. US 2018/0004847).
Regarding claim 8, Maheshwari in view of Li further in view of Kaftan doesn’t explicitly teach the claim limitation.
However, Kulkarni teaches wherein the at least one image is the image from the collection of images that is most likely to result in a desired interaction (click) when the at least one image is included in the invitational content when the invitational content is served to a user terminal in response to the App store receiving the input text as the search term [Para. 13, claim 7, Para. 16 “The online system 130 selects the creative with the highest prediction score from each category of creatives and combines the selected creatives of each type to create the DCO content item for the target user”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li further in view of Kaftan to teach the claim limitation, feature as taught by Kulkarni; because the modification enables the system to dynamically optimize content for target user.
Claims 9 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of further in view of LIU et al. (Pub. No. US 2012/0117051).
Regarding claims 9 and 15 Maheshwari in view of Li doesn’t explicitly teach the claim limitation.
However, LIU teaches wherein the providing input text and the collection of images into the cross-modal matching service includes receiving the input text from search keywords received into a search input [Para. 2-4]; the collection of images is included in respective items of invitational content [Para. 60, 61, fig. 4 and related description]; and the selecting the at least one image to associate with the input text includes selecting at least one of the respective items of invitational content to be displayed in an App store along with search results that are relevant to the input text [para. 26, 75-76].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li to teach the claimed limitation, feature as taught by LIU; because the modification enables the system to determine result based on multiple modes queries.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of further in view of Khemka et al. (Pub. No. US 2023/0409615).
Regarding claim 21, Maheshwari in view of Li doesn’t explicitly teach the claim limitation.
Khemka teaches wherein an App store is the specified context (application setting) for which the cross-modal matching service is fine-tuned (fine-tuned) from the pre-trained model (pre-trained model) [Para. 300, 290 and 294].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li to teach the claimed limitation, feature as taught by Khemka; because the modification enables the system to determine result based on multiple modes queries.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Li et al. (Pub. No. US 20200250537 hereinafter Li2) further in view of Tan and Basnal (LXMERT: Learning Cross-Modality Encoder representations from Transformers).
Regarding claim 22, Maheshwari in view of Li further in view of Li2 do not explicitly teach the claim limitation.
However, Tan teaches wherein the cross-modal matching service applies a transformer-based language encoder (language encoder) to the input text and a transformer-based object relationship encoder (object relationship encoder) to an image of the collection of images, and applies a cross-modality encoder that learns joint cross-modality representations from outputs of the language encoder (cross-modality encoder) and the object relationship encoder [Abstract and Section 1, 2.2].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li to teach the claimed limitation, feature as taught by Tan, to improve Maheshwari by expressly modeling intra-language and inter-object relationship before cross-modal fusion, thereby producing more contextually informative query image representations for relevance scoring.
Claims 24 are rejected under 35 U.S.C. 103 as being unpatentable over Maheshwari et al. (Pub. No. US 2021/0342389) in view of Li et al. (Pub. No. US 2022/0391755) further in view of Ganapavarapu et al. (Pub. No. US 2023/0376825).
Regarding claim 24. Maheshwari in view of Li the claim limitations.
However, Ganapavarapu teaches wherein the first dataset is repopulated with more recent data and the fine-tuned model is retrained based on the more recent data to update the fine-tuned model for changing definitions (new label definitions) of relevance over time [Para. 28].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Maheshwari in view of Li to teach the claimed limitation, feature as taught by Tan, to improve Maheshwari by allowing its query-image relevance model to respond to changes in user behavior and target label meaning over time, thereby maintaining relevance scoring accuracy as the underlying relevance concept changes.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-8 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 888-786-0101 (IN USA OR CANADA) or 571-272-4000.
/SOLOMON G BEZUAYEHU/
Primary Examiner, Art Unit 2666
