DETAILED ACTION
Claims 1-20 are pending in the Instant Application.
Claims 1-20 are rejected (Final Rejection).
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 12 and 20 are rejected on the grounds of nonstatutory double patenting as being unpatentable over claims 1, 9 and 14 of U.S. Patent No. 12,130,809. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims in the Instant Application describe the elements of U.S. Patent No. 12,130,809 in a broader sense and would encompass the claims. The claims are related as noted below.
Instant Application
U.S. Patent No. 12,130,809
1. A computer-implemented method comprising:
providing a first set of search results based on a search query; receiving a seed search selection selected from the first set of search results provided;
receiving a modifier comprising one or more negative modifiers or one or more positive modifiers;
determining a modifier embedding for the modifier, wherein the modifier embedding is generated via a multimodal model trained to generate image and text embeddings and used to modify a seed search selection embedding of the seed search selection in a vector space direction away from a negative modifier of the one or more negative modifiers and toward a positive modifier of the one or more positive modifiers, wherein modifying the seed search embedding comprises adjusting the seed search embedding using vector-arithmetic operations based on the negative modifier embedding and the positive modifier embedding;
generating a modified seed search selection embedding by modifying the seed search selection embedding of the seed search selection using the modifier embedding; and providing a second set of search results based on the modified seed search selection embedding.
1. A computer-implemented method comprising:
providing a first set of search results based on a search query; receiving a seed search selection selected from the first set of search results provided; receiving a modifier;
determining a modifier embedding for the modifier, wherein the modifier embedding is generated for the seed search selection via a multimodal model trained to generate image and text embeddings, the multimodal model comprising bidirectional encoder representations from transformers (BERT);
generating a modified seed search selection embedding by modifying a seed search selection embedding of the seed search selection using the modifier embedding; and providing a second set of search results based on the modified seed search selection embedding.
12. A computer system comprising: a processor; and a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
providing a first set of search results based on a search query; receiving a seed search selection selected from the first set of search results provided;
receiving a modifier comprising one or more negative modifiers or one or more positive modifiers;
determining a modifier embedding for the modifier, wherein the modifier embedding is generated via a multimodal model trained to generate image and text embeddings and used to modify a seed search selection embedding of the seed search selection in a vector space direction away from a negative modifier of the one or more negative modifiers and toward a positive modifier of the one or more positive modifiers, wherein modifying the seed search embedding comprises adjusting the seed search embedding using vector-arithmetic operations based on the negative modifier embedding and the positive modifier embedding;
generating a modified seed search selection embedding by modifying the seed search selection embedding of the seed search selection using the modifier embedding; and providing a second set of search results based on the modified seed search selection embedding.
9. A computer system comprising: a processor; and a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
receive a first set of search results based on a search query; select a seed from the first set of search results for the generation of a seed embedding;
provide a modifier; and
receive a second set of search results based on a modified seed embedding of the selected seed, wherein the seed embedding of the selected seed is modified via a modifier embedding corresponding to the modifier provided,
wherein the modifier embedding is generated for the seed selected via a multimodal model trained to generate image and text embeddings, the multimodal model comprising bidirectional encoder representations from transformers (BERT).
20. One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising: providing a first set of search results based on a search query; receiving a seed search selection selected from the first set of search results provided;
receiving a modifier comprising one or more negative modifiers or one or more positive modifiers; determining a modifier embedding for the modifier, wherein the modifier embedding is generated via a multimodal model trained to simultaneously generate image and text embeddings and used to modify a seed search selection embedding of the seed search selection in a vector space direction away from a negative modifier of the one or more negative modifiers and toward a positive modifier of the one or more positive modifiers, wherein modifying the seed search embedding comprises adjusting the seed search embedding using vector-arithmetic operations based on the negative modifier embedding and the positive modifier embedding;
generating a modified seed search selection embedding by modifying the seed search selection embedding of the seed search selection using the modifier embedding; and providing a second set of search results based on the modified seed search selection embedding.
14. One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising: receiving a seed search selection selected from a first set of search results provided via a user interface, the first set of search results based on a search query;
receiving a modifier for modifying search results for the search query based on the seed search selection; generating, via a multimodal model trained to generate image and text embeddings, a modifier embedding for the modifier, wherein the multimodal model comprises bidirectional encoder representations from transformers;
generating a modified seed search selection embedding for the seed search selection by applying the modifier embedding to an embedding of the seed search selection; and providing a second set of search results based on the modified seed search selection embedding
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 4, 7, 8, 10-12, 14, 15, 17, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (“Guo”), United States Patent Application Publication No. 2021/0073252, in view of Green, United States Patent Application Publication No. 2018/0121827.
As per claim 1, Guo discloses a computer-implemented method comprising:
providing a first set of search results based on a search query ([0067] wherein the user provides a query and seed image, and an initial result set is including three images is provided to the user);
receiving a seed search selection selected from the first set of search results provided ([0068] wherein a seed selection is received from the first set of results as user states “More like the third one” indicating the user’s selection of image “437”);
receiving a modifier comprising one or more negative modifiers or one or more positive modifiers ([0068] wherein a modifier is received in the form of “but with a belt” (recognized as natural language feedback in the prior art)), wherein the modifier embedding is generated via a multimodal model trained to generate image and text embeddings and used to modify a seed search selection embedding of the seed search selection in a vector space direction away from a negative modifier of the one or more negative modifiers and toward a positive modifier of the one or more positive modifiers ([0082] wherein the modifier (recognized as natural language feedback in the prior art) is transformed to generate a vector, which is used to modify a seed search selection embedding as described in [0085]-[0086]);
generating a modified seed search selection embedding by modifying the seed search selection embedding of the seed search selection using the modifier embedding ([0085]-[0086] wherein the aggregate vector is formed which is a modified seed selection); and providing a second set of search results based on the modified seed search selection embedding ([0089] wherein the candidate images form a second set of search results are provided to the user), but does not disclose wherein modifying the seed search embedding comprises adjusting the seed search embedding using vector-arithmetic operations based on the negative modifier embedding and the positive modifier embedding. However, Green teaches wherein modifying the seed search embedding comprises adjusting the seed search embedding using vector-arithmetic operations based on the negative modifier embedding and the positive modifier embedding ([0054] wherein the “seed search embedding” is recognized as “pages” used to identify recommended pages in the prior art, wherein based on context change, vector arithmetic can be applied to the vector (addition being positive, and subtraction being negative)).
Both Guo and Green describe searching using vectors that have been modified. One could use the vector arithmetic In Green instead of the concatenation described in Guo to teach the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the method of using positive and negative modifiers to search based on a seed embedding in Guo with the use of the vector arithmetic to alter the seed embedding as in Green in order to be able to easily update changes in the user profile.
As per claim 3, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1. Guo further discloses wherein the modifier is associated with an image of item listings or a particular portion of a textual description of each item within the item listings ([0069] wherein the modifier is that a “belt” be included, and “belt” is a portion of the textual description).
As per claim 4, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1. Guo further discloses wherein the modifier includes one or more characters or words entered by a user or an audio input provided by a user (EXAMINER NOTES the use of “or,” requiring either a text input or an audio input, but [0053] includes both an audible input or one entered by the user).
As per claim 7, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1. Guo further discloses wherein the modifier identifies an attribute that is more or less desirable based on the seed search selection ([0069] wherein a belt is more desirable based on the seed selection, since “but with a belt” is in comparison to the seed selection).
As per claim 8, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1. Guo further discloses wherein the multimodal model simultaneously produces image and text embeddings ([0086] wherein the query vector is produced at one time, which is a combination of the image and text embeddings).
As per claim 10, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1. Guo further discloses wherein the multimodal mode is trained to minimize matching loss between text and an image corresponding to an item ([0100] wherein the model is trained by comparing the visual image to the captured words (between text and an image)).
As per claim 11, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1. Guo further discloses wherein the multimodal mode is trained to minimize matching loss between or among text and an image, audio, or video (EXAMINER NOTES the use of “or” only then requiring the matching loss between two of text, image, audio or video and [0100] wherein the model is trained by comparing the visual image to the captured words (between text and an image)).
As per claim 12, Guo discloses a computer system comprising:
a processor ([0012]); and a computer storage medium storing computer-useable instructions ([0012]) that, when used by the processor, causes the computer system to perform the method of claim 1. Thus, claim 12 is rejected for the same rationale and reasoning as claim 1, the combination of Guo and Green as noted above.
As per claim 14, claim 14 is a system that performs the method of claim 3 and is rejected for the same rationale and reasoning.
As per claim 15, claim 15 is a system that performs the method of claim 4 and is rejected for the same rationale and reasoning.
As per claim 17, claim 17 is a system that performs the method of claim 7 and is rejected for the same rationale and reasoning.
As per claim 18, claim 18 is a system that performs the method of claim 8 and is rejected for the same rationale and reasoning.
As per claim 20, Guo discloses one or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
providing a first set of search results based on a search query [0067] wherein the user provides a query and seed image, and an initial result set is including three images is provided to the user);
receiving a seed search selection selected from the first set of search results provided ([0068] wherein a seed selection is received from the first set of results as user states “More like the third one” indicating the user’s selection of image “437”); receiving a modifier comprising one or more negative modifiers or one or more positive modifiers ([0068] wherein a modifier is received in the form of “but with a belt” (recognized as natural language feedback in the prior art)); determining a modifier embedding for the modifier ([0082] wherein the modifier (recognized as natural language feedback in the prior art) is transformed to generate a vector), wherein the modifier embedding is generated via a multimodal model trained to simultaneously generate image and text embeddings ([0086] wherein the query vector is produced at one time, which is a combination of the image and text embeddings) and used to modify a seed search selection embedding of the seed search selection in a vector space direction away from a negative modifier of the one or more negative modifiers and toward a positive modifier of the one or more positive modifiers ([0085]-[0086] wherein the seed search selection embedding is modified to include the modifier); generating a modified seed search selection embedding by modifying the seed search selection embedding of the seed search selection using the modifier embedding ([0085]-[0086] wherein the aggregate vector is formed which is a modified seed selection); and providing a second set of search results based on the modified seed search selection embedding ([0089] wherein the candidate images form a second set of search results are provided to the user), but does not disclose wherein modifying the seed search embedding comprises adjusting the seed search embedding using vector-arithmetic operations based on the negative modifier embedding and the positive modifier embedding. However, Green teaches wherein modifying the seed search embedding comprises adjusting the seed search embedding using vector-arithmetic operations based on the negative modifier embedding and the positive modifier embedding ([0054] wherein the “seed search embedding” is recognized as “pages” used to identify recommended pages in the prior art, wherein based on context change, vector arithmetic can be applied to the vector (addition being positive, and subtraction being negative)).
Both Guo and Green describe searching using vectors that have been modified. One could use the vector arithmetic In Green instead of the concatenation described in Guo to teach the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the method of using positive and negative modifiers to search based on a seed embedding in Guo with the use of the vector arithmetic to alter the seed embedding as in Green in order to be able to easily update changes in the user profile.
Claims 2 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Guo, in view of Green, in further view of Newson et al. (“Newson”), United States Patent Application Publication No. 2011/0202506.
As per claim 2, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1, but does not disclose wherein the positive modifier is a keyword to be included within a title of item listings and the negative modifier is a keyword to be excluded from the title. However, Newson teaches wherein the positive modifier is a keyword to be included within a title of item listings and the negative modifier is a keyword to be excluded from the title.([0031]-[0032] wherein keywords are shown to be included or excluded from the title based on the modifier).
Both Guo and Newson describe using a query and a modifier to filter. One could use the title inclusion and exclusion from Newson with the product searching in Guo to teach the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the method of generating vector expressions for multimodal model based searches in Guo with the title searches in Newson in order to provide more control of the results of the search by specifying fields to search.
As per claim 13, claim 13 is a system that performs the method of claim 2 and is rejected for the same rationale and reasoning.
Claims 5, 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Guo, in view of Green, in further view of Gulati et al. (“Gulati”), United State Patent Application Publication No. 2019/0163768.
As per claim 5, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 4, but does not disclose wherein the modifier is weighted more heavily than other modifiers based on user-provided weights or weights determined by a search engine corresponding to the multimodal machine learning model. However, Gulati teaches wherein the modifier is weighted more heavily than other modifiers based on user-provided weights or weights determined by a search engine corresponding to the multimodal machine learning model now ([0065] wherein the seed image is shown and can be weighted, and [0067] wherein the weight of a modifier (recognized by a search constraint in the prior art) can be based on user-provided values).
Both Guo and Gulati describe a multimodal machine learning model to modify searches. One could use the weights in Gulati with the interface and vector systems in Guo to teach the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the method of modifying a vector for a multimodal model search in Guo with the changing of weights in Gulati in order for the user to determine priorities for the user’s modifiers.
As per claim 6, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 4, but does not disclose wherein the modifier is weighted more heavily than other modifiers based on weights determined by a search engine corresponding to the multimodal machine learning model. However, Gulati teaches wherein the modifier is weighted more heavily than other modifiers based on weights determined by a search engine corresponding to the multimodal machine learning model ([0040] wherein a search engine determines a weight provided by a user for a new modifier (recognized by a search constraint in the prior art), which can be weighted more heavily).Both Guo and Gulati describe a multimodal machine learning model to modify searches. One could use the weights in Gulati with the interface and vector systems in Guo to teach the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the method of modifying a vector for a multimodal model search in Guo with the weights in Gulati in order for the user to determine priorities for the user’s modifiers.
As per claim 16, claim 16 is a system that performs the method of claim 5 and is rejected for the same rationale and reasoning.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Guo, in view of Green, in further view of Jing et al. (“Jing”), Chinese Patent Application Publication No. CN 202110170942 A (Published on 28 May 2021).
As per claim 9, note the rejection of claim 1 where Guo and Green are combined. The combination teaches the computer-implemented method of claim 1, but does not disclose wherein the multimodal model is trained using a plurality of titles from a plurality of item listings. However, Jing teaches wherein the multimodal model is trained using a plurality of titles from a plurality of item listings ([Page 6] wherein the model is trained using titles for clothing listings),
Guo trains a multimodal model using data from a title, but not from the title itself. Jing uses the title itself to train the model. The Supreme Court in KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007) identified a number of rationales to support a conclusion of obviousness which are consistent with the proper “functional approach” to the determination of obviousness as laid down in Graham. One such rationale is (B) Simple substitution of one known element for another to obtain predictable results. In this case the title data can be replaced by the title itself to train the model and the results would be predictable since the different training data would simply just alter the model, but does not alter the how the model is trained or how the computer functions.
As per claim 19, note the rejection of claim 12 where Guo and Green are combined. The combination teaches the system of claim 12, Guo further discloses wherein the multimodal mode is trained to minimize matching loss between text and an image corresponding to an item; or minimize matching loss between or among text and an image, audio, or video (EXAMINER NOTES the use of “or” only then requiring the matching loss between two of text, image, audio or video and [0100] wherein the model is trained by comparing the visual image to the captured words (between text and an image)), but does not disclose wherein the multimodal mode is trained using a plurality of titles from a plurality of item listings. However, Jing teaches wherein the multimodal mode is trained using a plurality of titles from a plurality of item listings ([Page 6] wherein the model is trained using titles for clothing listings),
Guo trains a multimodal model using data from a title, but not from the title itself. Jing uses the title itself to train the model. The Supreme Court in KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007) identified a number of rationales to support a conclusion of obviousness which are consistent with the proper “functional approach” to the determination of obviousness as laid down in Graham. One such rationale is (B) Simple substitution of one known element for another to obtain predictable results. In this case the title data can be replaced by the title itself to train the model and the results would be predictable since the different training data would simply just alter the model, but does not alter the how the model is trained or how the computer functions.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KANNAN SHANMUGASUNDARAM whose telephone number is (571)270-7763. The examiner can normally be reached M-F 9:00 AM -6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Charles Rones can be reached at (571) 272-4085. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KANNAN SHANMUGASUNDARAM/Primary Examiner, Art Unit 2168