DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-9, 11, 13-22 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5, 6, 11, 13, 16, 17, 21 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Dukellis et al. (U.S. Pub. No. 2010/0223128) in view of Steffensen et al. (U.S. Pub. No. 2022/0147558).
Regarding claim 1, Dukellis et al. discloses a method for generating a video template (see paragraphs 0021-0023; template creation, as distinguished from subsequently producing a video using that template), comprising:
obtaining a first-type material element (see paragraph 0027; inserting images during template creation. The image corresponds to the first-type material element);
determining a target edit element based on the first-type material element and the second-type material element (see paragraph 0027; selecting intra-bin transitions to match the feeding of the song and further adjusting those transitions to the particular image, including targeting a zoom at a person’s eyes. The resulting transition therefore depends on both the music and the image);
generating a target video template based on the first-type material element, the second-type material element and the target edit element (see paragraphs 0027, 0029-0030; template objects incorporating music/audio, images, and transitions linked through a time mesh. These correspond to the second-type material element, first-type material element, and target edit element, respectively).
However, Dukellis et al. fails to disclose complementing a second-type material element based on the first-type material element; wherein complementing the second-type material element comprises: selecting, from a predetermined material library, a material matching the first-type material element as the second-type material element.
Steffensen et al. discloses complementing a second-type material element based on the first-type material element (see paragraphs 0179-0183; extracting image labels, deriving an input feature vector, and selecting music whose semantic features match the image-derived features. The selected music supplies the second-type material element based on the obtained image);
wherein complementing the second-type material element comprises:
selecting, from a predetermined material library, a material matching the first-type material element as the second-type material element (see paragraphs 0163; a storage device containing music tracks and associated feature vectors. See paragraph 0182; selects tracks from that stored collection by matching their semantic features to the image-derived input features. See paragraph 0183; expressly permits a one-track result. The preexisting stored collection corresponds to the predetermined material library, and the selected matching track corresponds to the second-type material element.
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al. with the teachings of Steffensen et al., the motivation being to reduce manual searching for suitable music and provide accompaniment responsive to the visual material.
Regarding claim 11, claim 11 is rejected for the same reason set forth in the rejection of claim 1.
Regarding claim 21, claim 21 is rejected for the same reason set forth in the rejection of claim 1.
Regarding claims 2, 13 and 22, Dukellis et al. and Steffensen et al. discloses everything claimed as applied above (see claims 1, 11 and 21). Dukellis et al. discloses wherein the first-type material element and the second-type material element comprise at least one of: an image material, an audio material, or a reference video template (see paragraphs 0023, 0027); the target edit element comprises at least one of: a transition (see paragraphs 0027, 0030), an animation, an effect, a sticker, a font, or a filter.
Regarding claims 5 and 16, Dukellis et al. and Steffensen et al. discloses everything claimed as applied above (see claims 2 and 13). Dukellis et al. discloses the generating a target video template based on the first-type material element, the second-type material element and the target edit element comprises (see paragraphs 0049): replacing a reference material element in the reference video template with a further material element, other than the reference video template, from the first-type material element and the second-type material element (see paragraphs 0049 and 0070), and replacing a reference edit element in the reference video template with the target edit element to obtain the target video template (see paragraphs 0027, 0033-0034).
Regarding claims 6 and 17, Dukellis et al. and Steffensen et al. discloses everything claimed as applied above (see claims 1 and 11). Dukellis et al. discloses the generating a target video template based on the first-type material element, the second-type material element and the target edit element comprises (see paragraph 0027): determining target spatio-temporal layout information of the target edit element in the first- type material element and/or the second-type material element (see paragraphs 0029-0031, 0045), wherein the target spatio-temporal layout information comprises target temporal layout information and/or target spatial layout information (see paragraphs 0029-0031, 0041); and
associating presentation timing and/or a presentation position of the target edit element with the first-type material element and/or the second-type material element based on the spatio- temporal layout information, to obtain the target video template (see paragraphs 0029-0031, 0045).
Claims 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Dukellis et al. and Steffensen et al. as applied to claims 2 and 13 above, and further in view of Harwath et al. (U.S. Pub. No. 2018/0039859) and Logan et al. (U.S. Pub. No. 2005/0281541).
Regarding claims 3 and 14, Dukellis et al. and Steffensen et al. discloses everything claimed as applied above (see claims 2 and 13). However, Dukellis et al. and Steffensen et al. are silent as to in accordance with a determination that the complementing a second-type material element based on the first-type material element comprises complementing an image material based on an audio material, inputting the audio material and a number of image materials into an audio-image matching model, to output an image set, in an image material library, matching the number of image materials.
Harwath et al. discloses in accordance with a determination that the complementing a second-type material element based on the first-type material element comprises complementing an image material based on an audio material, inputting the audio material and a number of image materials into an audio-image matching model, to output an image set, in an image material library, matching the number of image materials (see paragraphs 0020-0022) and to output an image set, in an image material library, matching the number of image materials (see paragraphs 0043-0045).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al. and Steffensen et al. with the teachings of Harwath et al., the motivation being to invoke audio-to-image retrieval.
Harwath discloses an audio-image similarity processor having learned audio and image representations, together with a search component that retrieves matching images from an image database in response to audio input in paragraphs 0043-0045 and figs. 1-2. However, Harwath does not disclose supplying a required image count as an input to that model.
Logan et al. discloses specifying the number of images to retrieve and selecting that number according to strongest audio similarity (see paragraphs 0026, 0029-0030 and claim 6).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al., Steffensen et al. and Harwath et al. with the teachings of Logan et al., the motivation being to fit the intended number of image positions in the template.
Claims 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Dukellis et al. and Steffensen et al. as applied to claims 2 and 13 above, and further in view of Eronen et al. (U.S. Pub. No. 2015/0194185) and Li et al. (U.S. Pub. No. 2022/0084313).
Regarding claims 4 and 15, Dukellis et al. and Steffensen et al. discloses everything claimed as applied above (see claims 2 and 13). However, Dukellis et al. and Steffensen et al. are silent as to discloses the determining a target edit element based on the first-type material element and the second-type material element comprises: obtaining a reference sequence based on an edit element and a clipping structure of the reference video template; and inputting the image material, the audio material, and the reference sequence into a multimodal model to obtain a target edit element sequence.
Eronen et al. discloses obtaining a reference sequence based on an edit element and a clipping structure of the reference video template (see paragraphs 0142-0156).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al. and Steffensen et al. with the teachings of Eronen et al., the motivation being to provide a plausible correspondence to a sequence reflecting editing choices.
However, Dukellis et al., Steffensen et al. and Eronen et al. are silent as to inputting the image material, the audio material, and the reference sequence into a multimodal model to obtain a target edit element sequence.
Li et al. discloses inputting the image material, the audio material, and the reference sequence into a multimodal model to obtain a target edit element sequence (see paragraphs 0056, 0066-0068).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al., Steffensen et al. and Eronen et al. with the teachings of Li et al., the motivation being to provide a neural-network editing.
Claims 7 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Dukellis et al. and Steffensen et al. as applied to claims 2 and 13 above, and further in view of Miyajima (U.S. Pub. No. 2014/0000441).
Regarding claims 7 and 18, Dukellis et al. and Steffensen et al. discloses everything claimed as applied above (see claims 1 and 11). Dukellis et al. discloses performing beat-matching between the fragment segmented and a sequence of the target image material (see paragraphs 0027-0030), wherein the beat-matching associates the fragment node with a predetermined position in the sequence of the target image material in terms of a playback time (see paragraphs 0029-0030).
However, Dukellis et al. and Steffensen et al. are silent as to the generating a target video template based on the first-type material element, the second-type material element and the target edit element comprises: detecting a climactic fragment of an audio material; segmenting the climactic fragment based on lyric information and/or beat information to obtain a fragment node.
Miyajima discloses the generating a target video template based on the first-type material element, the second-type material element and the target edit element comprises:
detecting a climactic fragment of an audio material (see paragraphs 0005, 0039-0041);
segmenting the climactic fragment based on lyric information and/or beat information to obtain a fragment node (see paragraph 0047, fig. 3).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al. and Steffensen et al. with the teachings of Miyajima, the motivation being to produce a shortened, synchronized presentation that includes a characteristic musical climax instead of an arbitrary musical excerpt.
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Dukellis et al. and Steffensen et al. as applied to claims 2 and 13 above, and further in view of Harwath et al. (U.S. Pub. No. 2018/0039859), Logan et al. (U.S. Pub. No. 2005/0281541) and Xu et al. (U.S. Pub. No. 2022/0165171).
Regarding claims 8 and 19, Dukellis et al., Steffensen et al., Harwath et al. and Logan et al. discloses everything claimed as applied above (see claims 3 and 14). Harwath et al. discloses a step of establishing the audio- image matching model, wherein the establishing the audio-image matching model comprises:
extracting an audio feature of the audio sample and extracting an image feature of the image sample with an initial model (see paragraphs 0046-0055), wherein a matching degree label between the audio sample and the image sample indicates a matching degree between the audio sample and the image sample (see paragraphs 0059);
determining a matching degree between the audio feature and the image feature (see paragraph 0056), and comparing the matching degree with a corresponding matching degree label, and modifying, as feedback, an extraction parameter used in the initial model for extracting the audio feature and the image feature, to obtain the audio-image matching model (see paragraphs 0057-0059).
However, Dukellis et al., Steffensen et al., Harwath et al. and Logan et al. are silent as to obtaining an existing video template from an existing video template library, and determining an audio in the existing video template as an audio sample, and determining an image in the existing video template as an image sample, wherein the audio sample and the image sample belonging to a same existing video template are matched with each other.
Xu et al. discloses obtaining an existing video template from an existing video template library, and determining an audio in the existing video template as an audio sample, and determining an image in the existing video template as an image sample, wherein the audio sample and the image sample belonging to a same existing video template are matched with each other (see paragraph 0007, 0013-0014).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al., Steffensen et al., Harwath et al. and Logan et al. with the teachings of Xu et al., the motivation being to produce positive audio/visual pairs sampled from the same video.
Claims 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Dukellis et al. and Steffensen et al. as applied to claims 2 and 13 above, and further in view of Kalish et al. (U.S. Pub. No. 2015/0340067).
Regarding claims 9 and 20, Dukellis et al. and Steffensen et al. discloses everything claimed as applied above (see claims 1 and 11). Dukellis et al. discloses obtaining an original material (see paragraphs 0057-0058 and fig. D13), reading the target video template (see paragraph 0054) and generating a video based on the target video template (see paragraph 0057).
However, Dukellis et al. and Steffensen et al. are silent as to in response to a video generation instruction comprising a target video template identification.
Kalish et al. discloses in response to a video generation instruction comprising a target video template identification (see paragraph 0035).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Dukellis et al. and Steffensen et al. with the teachings of Kalish et al., the motivation being to communicate unambiguously which available template the generation operation should use and to initiate personalized video generation on demand.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NNENNA NGOZI EKPO whose telephone number is (571)270-1663. The examiner can normally be reached M-W 10:00am - 6:30pm, TH-F 8:00am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Pendleton can be reached at 571-272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NNENNA EKPO
Primary Examiner
Art Unit 2425
/NNENNA N EKPO/Primary Examiner, Art Unit 2425 October 14, 2026.
Prosecution Insights