DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14 are rejected under U.S.C. 103 as being unpatentable over BUSTAMANTE et al. (US 20190246146), in view of Fisher et al. (US 20220303502).
Regarding claim 1, BUSTAMANTE discloses, A method for determining a live streaming delay (Par. 0075, determining consumption timestamp determined to be in future or past time relative to current time), comprising:
sending a first clock synchronization request to a first server end node to obtain a first calibrated time difference, wherein the first server end node is used to process live streaming data (Par. 0069, clock synchronization between a client device (C) and a time server (S) to render live streams from different sources according to some embodiments of the present disclosure. Specifically, the process for live streaming a live event may include a clock synchronization phase for multiple viewpoints to ensure a harmonized experience when interacting among participants and changing viewpoints. During a clock synchronization phase, a client contacts a “time server” and requests that server's time. After receiving the request, the server prepares a response that appends a server time (T), which may be set based on an external source such as a UTC time source. After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2, i.e. sending clock synchronization request to server which render live streams and receives server calibrated time);
calibrating local time based on the first calibrated time difference to obtain calibrated time (Par. 0069, After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2, receives server calibrated time which includes round trip delay between sending time and server response time);
obtaining a live streaming bitstream generated by a peer end from the first server end node to obtain a timestamp of the live streaming bitstream and a local current timestamp (Par. 0054, The source devices 704 represent any electronic device from which a live stream of a live event originates. That is, the source devices 704 are sources of live streams from different viewpoints of the live event. The source devices 704 and at least some client devices 706 that can operate as source devices capture imagery and/or sounds of a live event as media that is streamed to the server 702, which synchronizes the live streams that are sharable among the client devices 706. Par. 0075, When a client device receives the data chunk, it can convert the server's consumption timestamp into a consumption timestamp for the client device. If the client's consumption timestamp is in a future time relative to a current time of the client's clock, the client will wait to render (consume) the associated data chunk until its local time reaches a time that is equal to or greater than the consumption timestamp, i.e. obtaining live stream produced by source device (i.e. peer device of client device 706 as shown in fig. 7) one or more platform servers 702 to obtain a timestamp of the live stream and to obtain local timestamp of consuming client device), wherein the timestamp is determined after clock synchronization is performed between the peer end and server end node (Par. 0006, At least some of the client devices are located at a live event and at least some of the client devices are remotely located from the live event. Further, each client device that is located at the live event is a source of a live stream, client devices can be peer devices that can be source of the content capture and some client devices can be consumer of the captured live streams, par. 0067, All the sources of the synchronized group and corresponding live streams have their clocks synchronized, par. 0069 discloses, client clock synchronization with server), the second server end node and the first server end node are clock-synchronized, and the live streaming bitstream is transmitted from the second server end node to the first server end node; and
determining a live streaming delay based on the local current timestamp and the timestamp of the live streaming bitstream (Par. 0075, When a client device receives the data chunk, it can convert the server's consumption timestamp into a consumption timestamp for the client device. If the client's consumption timestamp is in a future time relative to a current time of the client's clock, the client will wait to render (consume) the associated data chunk until its local time reaches a time that is equal to or greater than the consumption timestamp. If the data chunk arrives at the client device at a point in time different from or past the consumption timestamp, the client device may adapt the live stream to maintain synchronization or discard the data chunk, i.e. determining the difference or delay between timestamp of content stream and current time of local client’s clock).
BUSTAMANTE does not disclose, clock synchronization is performed between the peer end and a second server end node, the second server end node and the first server end node are clock-synchronized, and the streaming bitstream is transmitted from the second server end node to the first server end node.
Fisher discloses, clock synchronization is performed between the peer end and a second server end node (Par. 0099, a real-time media-pipeline on the server-side (hosting platform) applies the relative clock offsets between each Elected Speaker client and the other clients from that Elected Speaker client's Speaker Group to synchronize the devices within a Speaker Group to the Elected Speaker client (821). Then, each Elected Speaker client is further synchronized (using its RTT and local-clock-time offset) to the Bridge/Conference Server 610, i.e. clock synchronization is performed between elected speaker (works as server to other clients) and other client devices, as shown in pic 10), the second server end node and the first server end node are clock-synchronized (par. 0085, conference bridge can also use clock synchronization (described below) with the Elected Speaker client in each Speaker Group and accurate clock synchronization via peer-to-peer messages among clients in each Speaker Group to improve the RTT estimation, i.e. first node and second node are clock synchronized), and the live streaming bitstream is transmitted from the second server end node to the first server end node (par. 0114, The hosting platform 610 routes the audio signal to the Elected Speakers in the other Speaker Groups, including client 620z in Speaker Group 2, i.e. stream is transmitted from selected speaker in one group to (i.e. first server ) elected speaker in second group (i.e. second server).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of BUSTAMANTE by the teachings of clock synchronization is performed between the peer end and a second server end node, the second server end node and the first server end node are clock-synchronized, and the streaming bitstream is transmitted from the second server end node to the first server end node, as taught by Fisher, to determine latency associated with transmitting data to and receiving data from each client and to synchronizes transmissions to the clients based on the latencies, as disclosed in Fisher, 0023.
Regarding claim 2, The method according to claim 1,
BUSTAMANTE further discloses, wherein sending a first clock synchronization request to a first server end node to obtain a first calibrated time difference comprises:
sending a first clock synchronization request to the first server end node, wherein the first clock synchronization request comprises a first timestamp for sending the first clock synchronization request (Par. 0069, During a clock synchronization phase, a client contacts a “time server” and requests that server's time. After receiving the request, the server prepares a response that appends a server time (T), which may be set based on an external source such as a UTC time source. After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2, i.e. sending clock synchronization request to server which render live streams and receives server calibrated time);
receiving a response message fed back by the first server end node, and determining a fourth timestamp for receiving the response message, wherein the response message carries a second timestamp when the first clock synchronization request arrives the first server end node and a third timestamp when the first server end node sends the response message (Par. 0069During a clock synchronization phase, a client contacts a “time server” and requests that server's time. After receiving the request, the server prepares a response that appends a server time (T), which may be set based on an external source such as a UTC time source. After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2); and
determining the first calibrated time difference based on the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp (Par. 0069During a clock synchronization phase, a client contacts a “time server” and requests that server's time. After receiving the request, the server prepares a response that appends a server time (T), which may be set based on an external source such as a UTC time source. After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2, RTT = roundtrip time measurement which includes time of sending, time of destination receiving, time destination reply and time of receiving rely).
Regarding claim 3, The method according to claim 2,
BUSTAMANTE further discloses, wherein sending a first clock synchronization request to the first server end node comprises:
obtaining a start instruction for a live streaming application to start the live streaming application (Par. 0033, a viewpoint 100 of a live event displayed by a client device (“first client device”) on a user interface (UI) administered by the platform to simulate local interactions with other participants sharing an experience of a live event 102. In the illustrated embodiment, the live event 102 is a sports game at a stadium. The UI renders a live stream to display the viewpoint 100 of a source device located at the live event. The viewpoint 100 includes graphical icons that indicate the locations of the source devices 106 at the live event and relative to each other, i.e. server obtains information about live streaming application running on client device in order to stream the source content); and triggering local sending of the first clock synchronization request to the first server end node after the live streaming application is started (Par. 0069, fig. 9, clock synchronization between a client device (C) and a time server (S) to render live streams from different sources according to some embodiments of the present disclosure. Specifically, the process for live streaming a live event may include a clock synchronization phase for multiple viewpoints to ensure a harmonized experience when interacting among participants and changing viewpoints, i.e. clock synchronization after the live streaming process stated or initiated).
Regarding claim 4, The method according to claim 1,
BUSTAMANTE further discloses, wherein obtaining a live streaming bitstream generated by a peer end from the first server end node to obtain a timestamp of the live streaming bitstream and a local current timestamp comprises:
obtaining a live streaming bitstream generated by a peer end from the first server end node (Par. 0075, When a client device receives the data chunk, it can convert the server's consumption timestamp into a consumption timestamp for the client device. If the client's consumption timestamp is in a future time relative to a current time of the client's clock, the client will wait to render (consume) the associated data chunk until its local time reaches a time that is equal to or greater than the consumption timestamp. If the data chunk arrives at the client device at a point in time different from or past the consumption timestamp, the client device may adapt the live stream to maintain synchronization or discard the data chunk, i.e. determining the difference or delay between timestamp of content stream and current time of local client’s clock, and determining the local current timestamp;
parsing the live streaming bitstream to determine a supplemental enhancement information frame in the live streaming bitstream; and extracting a timestamp of the live streaming bitstream from the supplemental enhancement information frame (Par. 0074, each scene is implemented as a DataChunk defined by a sequence of frames associated with a consumption timestamp (ConsumptionTS), where each frame includes a frame ID, a data timestamp (DataTS), and the data. The source of a live stream can send to a subscribed participant, along with a data chunk, metadata including a consumption timestamp of the server-side component that indicates when the data chunk should be rendered on the participant's viewer, Par. 0075, When a client device receives the data chunk, it can convert the server's consumption timestamp into a consumption timestamp for the client device. If the client's consumption timestamp is in a future time relative to a current time of the client's clock, the client will wait to render (consume) the associated data chunk until its local time reaches a time that is equal to or greater than the consumption timestamp. If the data chunk arrives at the client device at a point in time different from or past the consumption timestamp, the client device may adapt the live stream to maintain synchronization or discard the data chunk, i.e. determining the difference or delay between timestamp of content stream and current time of local client’s clock, i.e. frames are enhanced with supplemental information such as metadata that includes consumption timestamp, client device parses the frame to obtain this data to extract timestamp included in stream).
Regarding claim 5, The method according to claim 1,
BUSTAMANTE further discloses, The method according to claim 1, wherein calibrating local time based on the first calibrated time difference to obtain calibrated time comprises:
determining a sum of the first calibrated time difference and the local time as the calibrated time (Par. 0069, During a clock synchronization phase, a client contacts a “time server” and requests that server's time. After receiving the request, the server prepares a response that appends a server time (T), which may be set based on an external source such as a UTC time source. After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2).
Regarding claim 6, The method according to claim 1,
BUSTAMANTE further discloses, wherein determining a live streaming delay based on the local current timestamp and the timestamp of the live streaming bitstream comprises:
determining the live streaming delay based on a difference between the local current timestamp and the timestamp of the live streaming bitstream (Par. 0075, When a client device receives the data chunk, it can convert the server's consumption timestamp into a consumption timestamp for the client device. If the client's consumption timestamp is in a future time relative to a current time of the client's clock, the client will wait to render (consume) the associated data chunk until its local time reaches a time that is equal to or greater than the consumption timestamp. If the data chunk arrives at the client device at a point in time different from or past the consumption timestamp, the client device may adapt the live stream to maintain synchronization or discard the data chunk, i.e. determining the difference or delay between timestamp of content stream and current time of local client’s clock).
Regarding claim 7, A method for determining a live streaming delay (Par. 0075, determining consumption timestamp determined to be in future or past time relative to current time)), comprising:
sending a second clock synchronization request to a second server end node to obtain a second calibrated time difference, wherein the second server end node is used to process live streaming data (par. 0049, The system 700 includes one or more server computers 702, i.e. there are plurality of servers, first and second, therefore request can be received by any server first or second. Par. 0067, fig. 8, For a source to create a new viewpoint or add to an existing viewpoint, the source must execute a clock synchronization algorithm to derive an offset between the group clock and its own local clock, Par. 0069, clock synchronization between a client device (C) and a time server (S) to render live streams from different sources according to some embodiments of the present disclosure. Specifically, the process for live streaming a live event may include a clock synchronization phase for multiple viewpoints to ensure a harmonized experience when interacting among participants and changing viewpoints. During a clock synchronization phase, a client contacts a “time server” and requests that server's time. After receiving the request, the server prepares a response that appends a server time (T), which may be set based on an external source such as a UTC time source. After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2, i.e. source device also sending clock synchronization request to server which render live streams and receives server calibrated time, since client device consuming the produced content by source device also sends request for clock synchronization, there is first and second request to obtain time difference);
calibrating local time based on the second calibrated time difference to obtain calibrated time (Par. 0069, After receiving the reply including the time response, the client sets its local time equal to T+RTTe/2, receives server calibrated time which includes round trip delay between sending time and server response time);
obtaining live streaming data to be sent (Par. 0005, The live streams provide a live broadcast of a live event from viewpoints captured by source devices located at the live event including a second client device), and a timestamp corresponding to the live streaming data to be sent (Par. 0074, each scene is implemented as a DataChunk defined by a sequence of frames associated with a consumption timestamp (ConsumptionTS), where each frame includes a frame ID, a data timestamp (DataTS), and the data. The source of a live stream can send to a subscribed participant, along with a data chunk, metadata including a consumption timestamp of the server-side component that indicates when the data chunk should be rendered on the participant's viewer);
generating a live streaming bitstream based on the timestamp corresponding to the live streaming data to be sent and the live streaming data to be sent (Par. 0074, each scene is implemented as a DataChunk defined by a sequence of frames associated with a consumption timestamp (ConsumptionTS), where each frame includes a frame ID, a data timestamp (DataTS), and the data. The source of a live stream can send to a subscribed participant, along with a data chunk, metadata including a consumption timestamp of the server-side component that indicates when the data chunk should be rendered on the participant's viewer.); and
sending the live streaming bitstream to the second server end node to arrive a peer end (Par. 0054, The source devices 704 represent any electronic device from which a live stream of a live event originates. That is, the source devices 704 are sources of live streams from different viewpoints of the live event. The source devices 704 and at least some client devices 706 that can operate as source devices capture imagery and/or sounds of a live event as media that is streamed to the server 702, which synchronizes the live streams that are sharable among the client devices 706 (i.e. pear device of the source device that would consume the captured content by source device), to determine the live streaming delay at the peer end (Par. 0075, When a client device receives the data chunk, it can convert the server's consumption timestamp into a consumption timestamp for the client device. If the client's consumption timestamp is in a future time relative to a current time of the client's clock, the client will wait to render (consume) the associated data chunk until its local time reaches a time that is equal to or greater than the consumption timestamp. If the data chunk arrives at the client device at a point in time different from or past the consumption timestamp, the client device may adapt the live stream to maintain synchronization or discard the data chunk, i.e. determining the difference or delay between timestamp of content stream and current time of local client’s clock), and the peer end is used to determine the live streaming delay based on time synchronized with the first server end node and the timestamp in the live streaming bitstream (Par. 0075, When a client device receives the data chunk, it can convert the server's consumption timestamp into a consumption timestamp for the client device. If the client's consumption timestamp is in a future time relative to a current time of the client's clock, the client will wait to render (consume) the associated data chunk until its local time reaches a time that is equal to or greater than the consumption timestamp. If the data chunk arrives at the client device at a point in time different from or past the consumption timestamp, the client device may adapt the live stream to maintain synchronization or discard the data chunk, i.e. determining the difference or delay between timestamp of content stream and current time of local client’s clock (client device receiving content has its local clock is synchronized as disclosed in par. 0067 and 0069)).
BUSTAMANTE does not disclose, wherein the second server end node is used to transmit the live streaming bitstream to a first server end node, the first server end node is used to transmit the live streaming bitstream to the peer end, the second server end node and the first server end node are clock-synchronized.
Fisher discloses, clock synchronization is performed between the peer end and a second server end node (Par. 0099, a real-time media-pipeline on the server-side (hosting platform) applies the relative clock offsets between each Elected Speaker client and the other clients from that Elected Speaker client's Speaker Group to synchronize the devices within a Speaker Group to the Elected Speaker client (821). Then, each Elected Speaker client is further synchronized (using its RTT and local-clock-time offset) to the Bridge/Conference Server 610, i.e. clock synchronization is performed between elected speaker (works as server to other clients) and other client devices, as shown in pic 10), the second server end node and the first server end node are clock-synchronized (par. 0085, conference bridge can also use clock synchronization (described below) with the Elected Speaker client in each Speaker Group and accurate clock synchronization via peer-to-peer messages among clients in each Speaker Group to improve the RTT estimation, i.e. first node and second node are clock synchronized), and the streaming bitstream is transmitted from the second server end node to the first server end node (par. 0114, The hosting platform 610 routes the audio signal to the Elected Speakers in the other Speaker Groups, including client 620z in Speaker Group 2, i.e. stream is transmitted from selected speaker in one group to (i.e. first server ) elected speaker in second group (i.e. second server).
Fisher discloses, clock synchronization is performed between the peer end and a second server end node (Par. 0099, a real-time media-pipeline on the server-side (hosting platform) applies the relative clock offsets between each Elected Speaker client and the other clients from that Elected Speaker client's Speaker Group to synchronize the devices within a Speaker Group to the Elected Speaker client (821). Then, each Elected Speaker client is further synchronized (using its RTT and local-clock-time offset) to the Bridge/Conference Server 610, i.e. clock synchronization is performed between elected speaker (works as server to other clients) and other client devices, as shown in pic 10), the second server end node and the first server end node are clock-synchronized (par. 0085, conference bridge can also use clock synchronization (described below) with the Elected Speaker client in each Speaker Group and accurate clock synchronization via peer-to-peer messages among clients in each Speaker Group to improve the RTT estimation, i.e. first node and second node are clock synchronized), wherein the second server end node is used to transmit the live streaming bitstream to a first server end node (par. 0114, The hosting platform 610 routes the audio signal to the Elected Speakers in the other Speaker Groups, including client 620z in Speaker Group 2, i.e. stream is transmitted from selected speaker in one group to (i.e. first server ) elected speaker in second group (i.e. second server).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of BUSTAMANTE by the teachings of the second server end node is used to transmit the live streaming bitstream to a first server end node, the first server end node is used to transmit the live streaming bitstream to the peer end, the second server end node and the first server end node are clock-synchronized, as taught by Fisher, to determine latency associated with transmitting data to and receiving data from each client and to synchronizes transmissions to the clients based on the latencies, as disclosed in Fisher, 0023.
Regarding claim 8, The method according to claim 7,
BUSTAMANTE further discloses, wherein generating a live streaming bitstream based on the timestamp corresponding to the live streaming data to be sent and the live streaming data to be sent comprises:
determining a supplemental enhancement information frame in the live streaming data to be sent; and writing a timestamp corresponding to the live streaming data to be sent into the supplemental enhancement information frame to generate the live streaming bitstream (Par. 0074, each scene is implemented as a DataChunk defined by a sequence of frames associated with a consumption timestamp (ConsumptionTS), where each frame includes a frame ID, a data timestamp (DataTS), and the data. The source of a live stream can send to a subscribed participant, along with a data chunk, metadata including a consumption timestamp of the server-side component that indicates when the data chunk should be rendered on the participant's viewer).
Regarding claim 9, BUSTAMANTE in view of Fisher meets claim limitation as set forth in claim 1.
Regarding claim 10-14, BUSTAMANTE meets claim limitation as set forth in claim 2-6
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AKSHAY DOSHI whose telephone number is (571)272-2736. The examiner can normally be reached M-F 9:30 AM to 6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN W MILLER can be reached at (571)272-7353. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.D./Examiner, Art Unit 2422
/JOHN W MILLER/Supervisory Patent Examiner, Art Unit 2422