DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Arguments
Applicant's arguments filed 04/29/2026 have been fully considered.
Applicant argues that the prior art of record does not teach on the amendments to the claims, that Holmberg does not teach on merging at a device, separate from the terminal device.
In response to the argument, Examiner respectfully disagrees. The amendments to the claims clarify that the device (separate from the terminal device) does the stream merging. Holmberg still teaches on this – the host device receives mixed media and then further mixes in more media received/generated. Further, each of the mobile devices in Holmberg have a capability for mixing media. The host device continues to receive images, vocals, and adds this received media to the broadcast mix. These items are received after the first received mixed media.
Holmberg teaches on the claims as recited:
merging, by the device (ie. host device) for stream merging, the first speech stream, the second speech stream, and the first image (ie. video of first and second performers) to obtain first merged streaming data; ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals, the method further includes capturing, at the host device, video that is performance synchronized with the captured second performer vocals, and the broadcast mix is an audiovisual mix of captured audio and video of at least the first and second performers.)
obtaining, by the device (ie. host device) for stream merging, after obtaining the first merged streaming data (ie. broadcast mix), a second image indicating image information of the first user (ie. video feature of the first performer and/or second performer), wherein obtaining the second image comprises retrieving the second image from a server or generating the second image by the device for stream merging; ([0026] The method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. The method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.)
Please see updated rejection below: Claim(s) 1-7, 9-10, 13-23 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US 2018/0288467 Al (Holmberg).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless – (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-7, 9-10, 13-23 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by US 2018/0288467 Al (Holmberg).
Regarding Claim 1:
Holmberg teaches A method of stream merging, comprising:
obtaining, by a device (ie. host device) for stream merging and from a first terminal (ie. guest device, remote peer) associated with a first user (ie. one of the performers), a first speech stream comprising speech information (ie. first or second performer vocals) of the first user associated with a live streaming interaction event (ie. livestream vocal duet), ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals. [0029] The broadcast mix is presented as a vocal duet. Audio of the live stream includes both conversational-type audio portions captured in correspondence with interactive conversation between the first and second performers. [0045] Techniques have been developed to facilitate the livestreaming of group audiovisual performances. Audiovisual performances including vocal music are captured and coordinated with performances of other users in ways that can create compelling user and listener experiences.)
wherein the first terminal (ie. guest device, remote peer) is different from the device (ie. host device) for stream merging; ([0015] The method includes (1 ) receiving at the host device, operating as a local peer, a media encoding of a mixed audio performance (i) including vocal audio captured at the guest device, communicatively coupled as a remote peer, from a first one of the performers.)
obtaining, by the device (ie. host device) for stream merging, a second speech stream (ie. second performer vocals) and a first image, the second speech stream comprising speech information of a second user (ie. one of the performers) associated with the live streaming interaction event, the first image comprising image information (ie. video of the first and second performers) of the second user; ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals, the method further includes capturing, at the host device, video that is performance synchronized with the captured second performer vocals, and the broadcast mix is an audiovisual mix of captured audio and video of at least the first and second performers.)
merging, by the device (ie. host device) for stream merging, the first speech stream, the second speech stream, and the first image (ie. video of first and second performers) to obtain first merged streaming data; ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals, the method further includes capturing, at the host device, video that is performance synchronized with the captured second performer vocals, and the broadcast mix is an audiovisual mix of captured audio and video of at least the first and second performers.)
obtaining, by the device (ie. host device) for stream merging, after obtaining the first merged streaming data (ie. broadcast mix), a second image indicating image information of the first user (ie. video feature of the first performer and/or second performer), wherein obtaining the second image comprises retrieving the second image from a server or generating the second image by the device for stream merging; ([0026] The method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. The method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.)
and encoding the second image and the first merged streaming data to obtain second merged streaming data. ([0026] In some embodiments, the method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. In some embodiments, the method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.) The first merged streaming data is the broadcast mix and it’s encoded with video feature of either or both first and second performer video to provide the second merged streaming data.
Regarding Claim 9:
Holmberg teaches An electronic device comprising a processor and a memory; the memory storing computer execution instructions; the processor executing the computer execution instructions stored in the memory to cause the processor to perform the acts ([0004] Computationally, these computing (mobile) devices offer speed and storage capabilities comparable to engineering workstation or workgroup computers from less than ten years ago, and typically include powerful media processors, rendering them suitable for real-time sound synthesis and other musical applications. [0041] FIG. 5 is a functional block diagram of hardware and software components executable at an illustrative mobile phone-type portable computing device to facilitate processing and communication of a captured audiovisual performance for use in a multi-vocalist livestreaming configuration of network-connected devices in accordance with some embodiments of the present invention(s).) comprising:
obtaining, from a first terminal (ie. guest device, remote peer) associated with a first user (ie. one of the performers), a first speech stream comprising speech information (ie. first or second performer vocals) of the first user associated with a live streaming interaction event (ie. livestream vocal duet), ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals. [0029] The broadcast mix is presented as a vocal duet. Audio of the live stream includes both conversational-type audio portions captured in correspondence with interactive conversation between the first and second performers. [0045] Techniques have been developed to facilitate the livestreaming of group audiovisual performances. Audiovisual performances including vocal music are captured and coordinated with performances of other users in ways that can create compelling user and listener experiences.)
wherein the first terminal (ie. guest device, remote peer) is different from the device (ie. host device) for stream merging; ([0015] The method includes (1 ) receiving at the host device, operating as a local peer, a media encoding of a mixed audio performance (i) including vocal audio captured at the guest device, communicatively coupled as a remote peer, from a first one of the performers.)
obtaining a second speech stream (ie. second performer vocals) and a first image, the second speech stream comprising speech information of a second user (ie. one of the performers) associated with the live streaming interaction event, the first image comprising image information of the second user (ie. video of first and second performers); ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals, the method further includes capturing, at the host device, video that is performance synchronized with the captured second performer vocals, and the broadcast mix is an audiovisual mix of captured audio and video of at least the first and second performers.)
merging the first speech stream, the second speech stream, and the first image (ie. video of first and second performers) to obtain first merged streaming data; ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals, the method further includes capturing, at the host device, video that is performance synchronized with the captured second performer vocals, and the broadcast mix is an audiovisual mix of captured audio and video of at least the first and second performers.)
obtaining, after obtaining the first merged streaming data (ie. broadcast mix), a second image indicating image information (ie. video feature of the first performer and/or second performer) of the first user, wherein obtaining the second image comprises retrieving the second image from a server or generating the second image by the electronic device; ([0026] The method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. The method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.)
and encoding the second image and the first merged streaming data to obtain second merged streaming data. ([0026] In some embodiments, the method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. In some embodiments, the method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.) The first merged streaming data is the broadcast mix and it’s encoded with video feature of either or both first and second performer video to provide the second merged streaming data.
Regarding Claim 10:
Holmberg teaches A non-transitory computer-readable storage medium storing program code for computer execution, the program code comprising instructions for performing the acts ([0086] Embodiments in accordance with the present invention may take the form of, and/or be provided as, a computer program product encoded in a machine-readable medium as instruction sequences and other functional constructs of software.) comprising:
obtaining, by a device (ie. host device) for stream merging and from a first terminal (ie. guest device, remote peer) associated with a first user, a first speech stream comprising speech information of a first user (ie. one of the performers) associated with a live streaming interaction event (ie. livestream vocal duet), ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals. [0029] The broadcast mix is presented as a vocal duet. Audio of the live stream includes both conversational-type audio portions captured in correspondence with interactive conversation between the first and second performers. [0045] Techniques have been developed to facilitate the livestreaming of group audiovisual performances. Audiovisual performances including vocal music are captured and coordinated with performances of other users in ways that can create compelling user and listener experiences.)
wherein the first terminal (ie. guest device, remote peer) is different from the device (ie. host device) for stream merging; ([0015] The method includes (1 ) receiving at the host device, operating as a local peer, a media encoding of a mixed audio performance (i) including vocal audio captured at the guest device, communicatively coupled as a remote peer, from a first one of the performers.)
obtaining, by the device (ie. host device) for stream merging, a second speech stream (ie. second performer vocals) and a first image, the second speech stream comprising speech information of a second user (ie. one of the performers) associated with the live streaming interaction event, the first image comprising image information of the second user (ie. video of first and second performers); ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals, the method further includes capturing, at the host device, video that is performance synchronized with the captured second performer vocals, and the broadcast mix is an audiovisual mix of captured audio and video of at least the first and second performers.)
merging, by the device (ie. host device) for stream merging, the first speech stream, the second speech stream, and the first image (ie. video of first and second performers) to obtain first merged streaming data; ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals, the method further includes capturing, at the host device, video that is performance synchronized with the captured second performer vocals, and the broadcast mix is an audiovisual mix of captured audio and video of at least the first and second performers.)
obtaining, by the device for stream merging, after obtaining the first merged streaming data (ie. broadcast mix),, a second image indicating image information of the first user (ie. video feature of the first performer and/or second performer), wherein obtaining the second image comprises retrieving the second image from a server or generating the second image by the device for stream merging; ([0026] The method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. The method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.)
and encoding the second image and the first merged streaming data to obtain second merged streaming data. ([0026] In some embodiments, the method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. In some embodiments, the method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.) The first merged streaming data is the broadcast mix and it’s encoded with video feature of either or both first and second performer video to provide the second merged streaming data.
Regarding Claims 2, 15, 22:
Holmberg teaches on the inventions of claims 1, 9, 10 as described.
Holmberg teaches wherein the obtaining a first speech stream comprises: obtaining the first speech stream from a forward server, the forward server (ie. content server) configured to forward speech information of the first user. ([0052] FIG. 1, iPhone™ handhelds available from Apple Inc. (or more generally, handhelds 101A, 101B operating as guest and host devices, respectively) execute software that operates in coordination with a content server 110 to provide vocal capture. [0053] A current guest user of current guest device 101A contributes to the group audiovisual performance mix 111 that is supplied ( eventually via content server 110) by current host device 101B as live stream 122. [0058] User vocals 103A and 103B are captured at respective handhelds 101A, 101B, and may be optionally pitch-corrected continuously and in real-time and audibly rendered mixed with the locally-appropriate backing track ( e.g., backing track 107A at current guest device 101A and guest mix 106 at current host device 101B).) The content server obtains/receives and forwards/provides audio and image data between user devices.
Regarding Claims 3, 16, 23:
Holmberg teaches on the inventions of claims 1, 9, 10 as described.
Holmberg teaches wherein the device for stream merging is comprised in a second terminal associated with the second user. ([0041] FIG. 5 is a functional block diagram of hardware and software components executable at an illustrative mobile phone-type portable computing device to facilitate processing and communication of a captured audiovisual performance for use in a multi-vocalist livestreaming configuration of network-connected devices.) Components shown in Figs 4 & 5 capture livestreams and then send to encoder (from both the camera and the mic). The encoder is the device within the mobile device (live streamer terminal).
Regarding Claim 4:
Holmberg teaches on the invention of claim 1 as described.
Holmberg teaches wherein the device for stream merging is comprised in a merge server. ([0054] Content that is mixed to form group audiovisual performance mix 111 is captured, in the illustrated configuration, in the context of karaoke-style performance capture wherein lyrics 102, optional pitch cues 105 and, typically, a backing track 107 are supplied from content server 110 to either or both of current guest device 1 0lA and current host device 101B. Claim 47: receiving at the second device, a media encoding of a mixed audio performance … mixing the captured second performer vocal audio with the received mixed audio performance to provide a broadcast mix that includes the captured vocal audio of the first and second performers and the backing audio track without apparent temporal lag therebetween; and supplying the broadcast mix to a service platform configured to livestream the broadcast mix to plural recipient devices constituting an audience.) The content server (second device as claimed in Claim 47) is a merge server and comprises components for stream merging.
Regarding Claim 17:
Holmberg teaches on the invention of claim 9 as described.
Holmberg teaches wherein the electronic device is comprised in a merge server. ([0054] Content that is mixed to form group audiovisual performance mix 111 is captured, in the illustrated configuration, in the context of karaoke-style performance capture wherein lyrics 102, optional pitch cues 105 and, typically, a backing track 107 are supplied from content server 110 to either or both of current guest device 1 0lA and current host device 101B. Claim 47: receiving at the second device, a media encoding of a mixed audio performance … mixing the captured second performer vocal audio with the received mixed audio performance to provide a broadcast mix that includes the captured vocal audio of the first and second performers and the backing audio track without apparent temporal lag therebetween; and supplying the broadcast mix to a service platform configured to livestream the broadcast mix to plural recipient devices constituting an audience.) The content server (second device as claimed in Claim 47) is a merge server and comprises components for stream merging.
Regarding Claims 5, 18:
Holmberg teaches on the inventions of claims 2, 15 as described.
Holmberg teaches wherein obtaining a second speech stream and a first image comprises: obtaining the second speech stream and the first image from the forward server, the forward server (ie. content server) further configured to forward image information and speech information of the second user. ([0052] FIG. 1, iPhone™ handhelds available from Apple Inc. (or more generally, handhelds 101A, 101B operating as guest and host devices, respectively) execute software that operates in coordination with a content server 110 to provide vocal capture. [0053] A current guest user of current guest device 101A contributes to the group audiovisual performance mix 111 that is supplied ( eventually via content server 110) by current host device 101B as live stream 122. [0058] User vocals 103A and 103B are captured at respective handhelds 101A, 101B, and may be optionally pitch-corrected continuously and in real-time and audibly rendered mixed with the locally-appropriate backing track ( e.g., backing track 107A at current guest device 101A and guest mix 106 at current host device 101B).) The content server obtains/receives and forwards/provides audio and image data between user devices.
Regarding Claims 6, 19:
Holmberg teaches on the inventions of claims 1, 9 as described.
Holmberg teaches wherein the second image comprises an image and visual effect associated with the first speech stream, the image configured to indicate the first user. ([0026] In some embodiments, the method further includes dynamically varying in the broadcast mix at least visual prominence of one or the other of the first and second performers based on evaluation of a computationally audio defined feature of either or both of the first and second performer vocals. In some embodiments, the method further includes applying one or more video effects to the broadcast mix based, at least in part, on a computationally defined audio or video feature of either or both of the first and second performer audio or video.) The first merged streaming data is the broadcast mix and it’s encoded with video feature of either or both first and second performer video to provide the second merged streaming data.
Regarding Claims 7, 20:
Holmberg teaches on the inventions of claims 1, 9 as described.
Holmberg teaches wherein after obtaining the second merged streaming data, the method further comprises: sending the second merged streaming data to a streaming media server (ie. service platform). (Claim 47: receiving at the second device, a media encoding of a mixed audio performance … mixing the captured second performer vocal audio with the received mixed audio performance to provide a broadcast mix that includes the captured vocal audio of the first and second performers and the backing audio track without apparent temporal lag therebetween; and supplying the broadcast mix to a service platform configured to livestream the broadcast mix to plural recipient devices constituting an audience.)
Regarding Claims 13, 14, 21:
Holmberg teaches on the inventions of claims 1, 9, 10 as described.
Holmberg teaches wherein: both the first user and the second user are live streamer users; or the first user is an audience user and the second user is a live streamer user. ([0025] The received media encoding includes video that is performance synchronized with the captured first performer vocals. [0029] The broadcast mix is presented as a vocal duet. Audio of the live stream includes both conversational-type audio portions captured in correspondence with interactive conversation between the first and second performers. [0045] Techniques have been developed to facilitate the livestreaming of group audiovisual performances. Audiovisual performances including vocal music are captured and coordinated with performances of other users in ways that can create compelling user and listener experiences.)
Conclusion & Contact Information
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL J HACKENBERG whose telephone number is (571)272-5417. The examiner can normally be reached 9am-5pm M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Glenton B Burgess can be reached at (571)272-3949. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL J HACKENBERG/Primary Examiner, Art Unit 2454
Prosecution Insights