Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant's arguments with respect to claims 1, 10, and 17 have been considered but are moot in view of the new ground(s) of rejection. Applicant’s arguments are directed to the amended subject matter; new prior art is provided below.
Reference Liu has been withdrawn and replaced with Howard (see below), to address the harvesting of data to render the harvest searchable for another platform or purpose where a user can enter inquiries relating to media or by including media + a text inquiry.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 2, 10, 11, 17, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20240220530 A1 WILKINS; Julia Lepley et al. (hereinafter WILKINS) in view of US 20200075020 A1 Howard; Newton et al. (hereinafter Howard) and further in view of US 20240249182 A1 KIRSHENBOIM; Gilad et al. (hereinafter KIRSHENBOIM).
Re claim 1, WILKINS teaches
1. A computing system, comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: (fig. 3 and 4 show implementation of fig. 1 device)
obtaining a plurality of audio samples and an associated corpus of descriptive textual data from an audiovisual data hosting entity wherein the corpus of descriptive textual data comprises, for each of the plurality of audio samples… (based on input query, video as well as audio paired with text, and audio extracted thereof as in fig. 3, utilized through a user audio-video interface to produce editing multimedia outputs 0023 with fig. 4 using model training 0028-0030)
for each audio sample of the plurality of audio samples: (samples and pairs as in fig. 4)
processing the audio sample with an audio embedding portion of a machine-learned audio classification model to obtain an audio embedding; (audio, video, text, or image embedding are interchangeable, contrastive loss is calculated based on input query, after embedding at the model using similarities, distances, and analogously differences thereof 0019 and 0047, some of which are weighted 0002-0003…video as well as audio paired with text, and audio extracted thereof as in fig. 3, utilized through a user audio-video interface to produce editing multimedia outputs 0023 with fig. 4 using model training 0028-0030)
…with a text embedding portion of the machine-learned audio classification model to obtain a text embedding; and (processing the text that is embedded paired with audio, then contrastive loss is calculated based on input query, after embedding at the model using similarities, distances, and analogously differences thereof 0019 and 0047, some of which are weighted 0002-0003…video as well as audio paired with text, and audio extracted thereof as in fig. 3, utilized through a user audio-video interface to produce editing multimedia outputs 0023 with fig. 4 using model training 0028-0030)
training the machine-learned audio classification model based on a contrastive loss function that evaluates a difference between the audio embedding and the text embedding. (contrastive loss is calculated based on input query, after embedding at the model using similarities, distances, and analogously differences thereof 0019 and 0047, some of which are weighted 0002-0003…video as well as audio paired with text, and audio extracted thereof as in fig. 3, utilized through a user audio-video interface to produce editing multimedia outputs 0023 with fig. 4 using model training 0028-0030)
However, while WILKINS teaches text that is paired with audio for embedding, it fails to teach text content that necessarily describes the audio, or such as “Is XYZ a good song”, “is ABC a decent movie”, or a simply inquiry “an image of a dog running” or “a movie about action and war”, and thus fails to teach:
processing the one or more portions of textual content that describe… [content such as images, video, songs, media, etc.]… (Howard a video with comments, where the comments are harvested and rendered searchable by another platform, such comments are present in a video viewing platform e.g. YouTube or similar, in which comments about the media are used to describe the context or views of media e.g. a song fig. 6 with 0043 and 0047)
one or more portions of textual content that are provided by users of the audiovisual data hosting entity responsive to the audio sample[[ ]], wherein the one or more portions of textual content comprise comments (Howard a video with comments, where the comments are harvested and rendered searchable by another platform, such comments are present in a video viewing platform e.g. YouTube or similar, in which comments about the media are used to describe the context or views of media e.g. a song fig. 6 with 0043 and 0047)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of WILKINS to incorporate the above claim limitations as taught by Howard to allow for use of a known technique of text to image generation to improve similar devices in the same way such as in the scope of harvesting data for a corpus from multiple users in a learning environment, such that the system of WILKINS is improved to handle multiple user data as a general context and feeling used, for instance data harvested for a media item, tied to thereof, with comments e.g. a song or video, such that the harvested data from comments is used as a searchable corpus or collection in another platform e.g. a generative system or generative search engine, where sentiment is extracted to quantify sentiment of media e.g. the collective impression of a media item, such when a user enters a song or other media item including an image or video per se, and asks about it or generally inquires about a media item without inputting the media itself (i.e. just a title), the system will have a reference based on comments by multiple users for publicly available videos.
However, while the combination teaches text + image with contrastive loss and diffusion to produce a new image, it fails to teach the context of audio paired with a descriptive text input such as “a classical song that is in B minor”, and thus fails to teach:
…portions of textual content that describe the audio sample… (KIRSHENBOIM stable diffusion requiring text + image and contemplating substitution of using audio + text to produce audio instead of images alone 0119)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of WILKINS in view of Howard to incorporate the above claim limitations as taught by KIRSHENBOIM to allow for a finding that one of ordinary skill in the art would consider the use of text decryption to audio as obvious to try in place of text description to image as there was a recognized problem of generative models in a search or generative AI context by a user, such as to utilize Howard’s harvesting user behavior as a corpus for a song or media item to search for or generate, to produce audio/music from a text description to due a variety of market-related issues e.g. copy right and the database of music or voice content to pair with, wherein such a combination would solve such a problem and also improve user satisfaction to get the results for both image and audio generation.
Re claim 10, this claim has been rejected for teaching a broader, or narrower claim based on general inclusion of hardware alone (e.g. processor, memory, instructions), representation of claim 1 omitting/including hardware for instance, otherwise amounting to a virtually identical scope.
Fig. 8 of Wilkins teaches the necessary hardware.
Re claim 17, this claim has been rejected for teaching a broader, or narrower claim based on general inclusion of hardware alone (e.g. processor, memory, instructions), representation of claim 1 omitting/including hardware for instance, otherwise amounting to a virtually identical scope
Fig. 8 of Wilkins teaches the necessary hardware.
Re claims 2, 11, and 18, while the combination teaches text + image with contrastive loss and diffusion to produce a new image, it fails to teach the context of audio paired with a descriptive text input such as “a classical song that is in B minor”, wherein an embedding can only be the input prior or after the first contrastive model, and thus fails to teach:
2. The computing system of claim 1, wherein the operations further comprise: processing a query comprising textual content that indicates a desired type of audio content with a machine-learned generator model to generate an intermediate representation of the textual content of the query. (KIRSHENBOIM stable diffusion requiring text + image and contemplating substitution of using audio + text to produce audio instead of images alone 0119)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of WILKINS in view of Howard to incorporate the above claim limitations as taught by KIRSHENBOIM to allow for a finding that one of ordinary skill in the art would consider the use of text decryption to audio as obvious to try in place of text description to image as there was a recognized problem of generative models to produce audio/music from a text description (including a corpus of user behavior e.g. from a forum in-context in relation to a media of interest) due to a variety of market-related issues e.g. copy right and the database of music or voice content to pair with, wherein such a combination would solve such a problem and also improve user satisfaction to get the results for both image and audio generation.
Allowable Subject Matter
Claims 3-9, 12-16, 19, and 20
Are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
After searching through patent and non-patent literature, there was no evidence that there exists a limitation in direct relation or an obvious variant to such limitations as a whole as precisely limited. When searching for a secondary prior art for the limitation as recited in the above claims, the most relevant topics pertained to material from the same Inventor(s) and Assignee but did not teach or suggest the aforementioned complex limitations as a whole as precisely limited. Such as including a third diffusion model as claimed with further transformation, wherein such a chain of models renders the combination insufficient, in which one of ordinary skill in the art, under BRI, would not seek to mix and match model inputs/outputs and alter the premise of the combination of references to reasonably derive the objected claims and keep the inventive concept analogous.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20210350786 A1 Chen; Zhehuai et al.
Iterative synthetic with transcript unspoken inputs, contrastive
US 10331402 B1 Spector; Daniel Lewis et al.
Learning models for query text and response audio reply
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL COLUCCI whose telephone number is (571)270-1847. The examiner can normally be reached on M-F 9 AM - 7 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL COLUCCI/Primary Examiner, Art Unit 2655 (571)-270-1847
Examiner FAX: (571)-270-2847
Michael.Colucci@uspto.gov