DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to the 35 U.S.C. 103 rejections of claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claims 1, 18, and 20 are objected to because of the following informalities: the amendments made to claims 1, 18, and 20 in the claim set filed 12/23/25 are not included in the claim set filed 7/29/26. For purposes of examination, the examiner assumes that this language was inadvertently omitted. The previous language included in those claims will be addressed as part of the rejections below.
Claim 5 is objected to because of the following informalities: Claim 5 was cancelled in the claim set filed 12/23/25, but it is included in the claim set filed 7/29/26. It is marked as “previously presented”. Claim 5 should remain cancelled, as cancelled claims can’t be reintroduced into a pending claim set. Note: a rejection of claim 5 has been added back into the 103 rejections below, in response to the reintroduced claim.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 5 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Claim 5 describes, “the method of claim 3, wherein the size of the mediator buffer is dynamically adjusted based on a change of the bitrate of the inbound audio stream in a streaming process.”
However, claim 1 already includes the limitation, “a size and a number of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream.”
Thus, claim 5 fails to further limit claim 1.
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 10, 11, 13-16, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (US 11,538,481), hereinafter “Li”, in view of Vaidya et al. (US 10,896,021), hereinafter “Vaidya”, in view of Vafin et al. (US 8,855,145), hereinafter “Vafin”, in view of Hauser et al. (US 5,850,395), hereinafter “Hauser”, and further in view of Garg et al. (US 11,295,746), hereinafter “Garg”.
Regarding Claim 1, Li teaches:
1. A computer-implemented method of processing an audio stream, comprising:
receiving an inbound audio stream by a resource-limited node, ; [Li: col. 51, line 27-30— “The speech audio that is so represented within each speech data set 3100 may include any of a variety of types of speech made up of words that spoken by one or more speakers,” which implies receiving inbound audio streams.
Also see col. 50 lns. 9-13: “FIG. 14B illustrates a block diagram of an alternate example embodiment of a non-distributed processing system 2000 in which the processing functionality of the one or more node devices 2300 is incorporated into the at least one control device 2500.”]
splitting the inbound audio stream into segments by the producer thread using a producer-consumer algorithm in the memory of the node, the inbound audio stream being split into the segments based on a detected silence ; [Li: col. 88, line 6-12, line 34-42 — “the speech audio represented by the specified speech data set may begin with either a processor of the control device or processor(s) of one or more node devices of the processing system (e.g., one or more of the node devices 2300) dividing the speech data set into data chunks that each represent a chunk of the speech audio, [line 6-12]” and “ More precisely, where each pause detection technique is assigned to a different node device or to a different thread of execution, it may be that the division of the speech audio into chunks is among the operations that are also so assigned such that separate node devices or separate cores are used to separately generate chunks of speech audio that are of appropriate length for their corresponding one of the pause detection techniques, [line 34-42]” indicating segmentation occurs based on silence detection]
transcribing voice included in a segment into text using a voice-to-text conversion engine; and [ Li: col. 3, lines 6-19— “divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments … generate a transcript of the speech data set”
performing natural language processing on the text to identify situational insights from the segment. [Li: col. 19, line 49-56— “the resulting one or more transcripts of the speech audio may be provided to one or more text analyzers to derive, based on such factors as the frequency with which each word was spoken, such insights as topic(s) spoken about, relative importance of topics, sentiments expressed concerning each topic, etc. It may be that each such stored transcript(s) may be accompanied in storage with metadata indicative of such insights.”
Li doesn’t describe receiving an inbound audio stream by a resource-limited node, the inbound audio stream being a live stream having a specific bitrate; and creating at least one mediator buffer, a producer thread, and a consumer thread in a memory of the node for processing the inbound audio stream, a size and a number of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream.
However, Vaidya describes receiving an inbound audio stream by a resource-limited node, the inbound audio stream being a live stream having a specific bitrate (FIG. 1 and col. 7 lns. 24-38: “A typical audio pipeline is demonstrated by audio playback pipeline 110. A request to produce an audio portion is received by an audio producer thread 112. The request to produce an audio portion can be a request to generate an audio portion, for example, from within an application, or to receive an audio portion, for example, from a streaming service.”); and
creating at least one mediator buffer, a producer thread, and a consumer thread in a memory of the node for processing the inbound audio stream (FIG. 1 and col. 7 lns. 24-46: “Turning now to the Figures, FIG. 1 is an illustration of a diagram of an example audio playback pipeline 100. Audio playback pipeline 100 is shown with three variants moving from a producer execution thread to a consumer execution thread. A typical audio pipeline is demonstrated by audio playback pipeline 110. A request to produce an audio portion is received by an audio producer thread 112. The request to produce an audio portion can be a request to generate an audio portion, for example, from within an application, or to receive an audio portion, for example, from a streaming service. Audio producer thread 112 can be one or more execution threads, or one or more system processes. Audio producer thread 112 processes the request and sends the audio portion to audio buffer 114.
Audio buffer 114 can be a conventional audio buffer. Audio consumer thread 116 can retrieve the audio portion from the audio buffer 114. Audio consumer thread 116 can be one or more execution threads, or one or more system processes. The output of audio consumer thread 116 is typically directed to one or more speakers, headphones, earpieces, amplifiers, and other auditory devices. The output can also be directed to other processes as well.”
Also see FIG. 5 and col. 9 ln. 66 – col. 10 ln. 11: “FIG. 5 is an illustration of a block diagram of an example audio correction system 500. Audio correction system 500 includes an audio system 510 and an audio consumer system 530. The audio system 510 and the audio consumer system 530 can be part of an electronic device, be separate components, or a combination thereof. For example, the audio system 510 can be part of an electronic device, which itself can be part of a larger system. Examples of larger systems include a tablet, a smartphone, a laptop, or an in-vehicle computing system. As an additional example, the audio consumer system 530 can be speakers included with the electronic device, or it can be separate speakers or headphones.”),
a size of the at least one mediator buffer being determined based on [detected system environment conditions] 600 to predict an audio glitch. Method 600 starts at a step 601 and proceeds to a step 605. In a step 605, the environment parameters that have been collected can be analyzed. The environment parameters, e.g., operating state information, can be a collection of process and threads executing on the system, the available resources on the system, the applications running on the system, the size and type of audio portion that is to be played, previous audio underrun conditions occurring on the system, and other factors that can affect the processing of the audio portion, such as the operating state of one or more system components. As the learning neural network improves, additional environment parameters can be added to the collection of environment parameters gathered and analyzed.”
Also see FIG. 7 and col. 11 lns. 35-50: “FIG. 7 is an illustration of a flow diagram of an example method 700, building on FIG. 6, to generate audio correction parameters for dynamically preventing audio underrun. Method 700 starts at a step 701 and proceeds through the previous described steps 605, 610, 612, 615 and 620. Step 620 is further described by the steps 720, 725, and 730. In the step 720, the processing system frequency is increased, i.e., boosted. For example, the increase to operating frequency can be applied to a part or all of the processing system where the processing system can include a CPU, a GPU, an audio system processor unit, other processing units, a bus or other communication channels, communication ports, and other aspects of the processing system. In the step 725, the memory frequency is increased. In the step 730, the audio buffer size is increased. Additional processing system adjustments can be made in addition to those listed here.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Li a system and method comprising receiving an inbound audio stream by a resource-limited node, the inbound audio stream being a live stream having a specific bitrate; and creating at least one mediator buffer, a producer thread, and a consumer thread in a memory of the node for processing the inbound audio stream, a size of the at least one mediator buffer being determined based on [detected system environment conditions], as taught by Vaidya, in order to expand the functionality of the system of Li by enabling it to receive and process streaming audio data, where the system is able to adapt in order to avoid buffer underrun conditions, which enables it to carry out transcription and identify situational insights on stored as well as streaming audio data.
Li in view of Vaidya doesn’t describe a system and method wherein a size of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream, wherein the size of the mediator buffer is further dynamically adjusted in real-time based on detecting changes in the specific bitrate of the inbound audio stream during streaming of the inbound audio stream.
However, Vafin describes a size of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream, wherein the size of the mediator buffer is further dynamically adjusted in real-time based on detecting changes in the specific bitrate of the inbound audio stream during streaming of the inbound audio stream (FIG. 8 and col. 14 lns. 16-37: “In step S802 processing parameters which would be beneficial for the transmitter 102 to use when processing data for transmission to the receiver 108 are determined. These processing parameters may be adjusted versions of the processing parameters which are determined based on the state of the jitter buffer 114 as described above. Alternatively, the processing parameters may be determined in step S802 without considering the state of the jitter buffer 114. As described above, the processing parameters may include one or more of the encoding bit rate, the FEC depth, a packetization delay and an interleaving delay.
In step S804 the processing parameters determined in step S802 are used to determine a state of the jitter buffer 114 which would be suited for receiving data which has been processed in accordance with the processing parameters determined in step S802. For example, if the encoding bit rate is increased then the available space in the jitter buffer 114 may need to be increased to accommodate the extra data that is to be transmitted due to the increase in the encoding bit rate. The jitter buffer size should be adapted in dependence on a change in the encoding bit rate of the data and in dependence on a bottleneck of the transmission path.” (emphasis added)
Also see col. 15 lns. 6-16: “In step S808 the state of the jitter buffer 114 at the receiver 108 is adjusted based on the indication of the jitter buffer state which has been received from the transmitter 102. In this way the jitter buffer 114 is placed in a state which is suited (i.e. optimized) for receiving the data which is transmitted from the transmitter 102 to the jitter buffer 114 of the receiver 108. Therefore when the data is processed at the transmitter 102 (in accordance with the processing parameters, as described above) and transmitted to the jitter buffer 114 of the receiver 108 then the jitter buffer 114 is in a state suited to receiving that data.” (emphasis added)
Further, see col. 6 lns. 41-50: “The data may be encoded using any suitable known encoding technique. Particular encoding techniques may be appropriate for different types of data. For example, where the data includes speech from a user of the transmitter 102 then the encoding technique may include a specific speech encoder for encoding the speech portions of the data. Other encoding techniques may also be used, e.g. to compress the data for transmission to the receiver 108. The data is processed using an encoding bit rate such that the encoded data has the particular encoding bit rate.” (emphasis added)
Finally, see col 6 ln. 57 – col. 7 ln. 9: “For example, the input signal may be a video signal and the encoder may produce a sudden increase (peak) in the instantaneous bit rate, as shown in FIG. 3. FIG. 3 shows a graph representing the number of bytes of each frame of a video signal once it has been encoded at the transmitter 102. It can be seen that the encoding bit rate is not the same for each video frame, and in particular some of the video frames (e.g. frame number 0 and frame number 100) are encoded at a much higher bit rate than the other video frames. The variation in the bit rate of the frames of the video signal may occur because different encoding techniques are used to encode the different video frames. A similar situation may occur when the input signal is an audio signal. For example, when a portion of an audio or a video signal is encoded independently of previously encoded portions then the bit rate of the encoded video or audio frame is typically relatively high. This is in contrast to other ones of the video or audio frames which may be encoded using differential-coding techniques, and therefore the encoded bit rate of those other video or audio frames is typically relatively low.” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Li in view of Vaidya a system and method wherein a size of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream, wherein the size of the mediator buffer is further dynamically adjusted in real-time based on detecting changes in the specific bitrate of the inbound audio stream during streaming of the inbound audio stream, as taught by Vafin, in order to enable the system to be compatible with a wide range of streaming bitrates, without reducing audio quality, which improves system performance, as higher quality audio data leads to higher quality transcription.
Li in view of Vaidya in view of Vafin doesn’t describe a system or method including a number of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream.
However, Hauser describes a system and method including a number of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream (See col. 2 ln. 64 – col. 3 ln. 9: “Through input/output modules 22 and switch control modules 32, ATM switch 20 provides the mechanisms to manage access to four types of resources (allocated bandwidth, dynamic bandwidth, allocated buffers, and dynamic buffers) in order to support four different service traffic types (constant bit rate, variable bit rate, available bit rate, and unspecified bit rate). Application services, such as video, voice, email, bulk data transfer, data transaction processing, etc., require different service traffic types with individual quality of service requirements. ATM switch 20, through input/output module 22 and switch control module 32, supports four service traffic types--constant bit rate, variable bit rate, available bit rate, and unspecified bit rate.” (emphasis added).
Also see col. 3 lns. 31-54: “ATM switch 20 provides the mechanisms to manage four types of resources in order to support the different service traffic types. The resources managed by ATM switch 20 are allocated bandwidth, dynamic bandwidth, allocated buffers, and dynamic buffers. ATM switch 20 provides quality of service guarantees per traffic type and per connection for all network topologies including point-to-point connections, point-to-multipoint connections, multipoint-to-point connections, and multipoint-to-multipoint connections. Allocated bandwidth is used for cell transfer opportunities occurring at regularly scheduled intervals, such as with constant bit rate traffic types. Dynamic bandwidth is not only unallocated bandwidth but also unused allocated bandwidth. This is a shared resource that is given to connections based on service class and priority. Allocated buffers are buffers that are reserved for each connection sustained rate buffering requirements on a connection by connection basis. Dynamic buffers are a pool of buffers that are reserved to perform rate matching between a set of connections incoming peak bandwidth and the outgoing dynamic bandwidth both of which are instantaneously changing. This pool of buffers are shared between connections. Variable bit rate, available bit rate, and unspecified bit rate serve as traffic types which utilize dynamic bandwidth to achieve high line utilization.” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Li in view of Vaidya in view of Vafin a system and method including a number of the at least one mediator buffer being determined based on the specific bitrate of the inbound audio stream, as taught by Hauser, in order to efficiently add buffers when the bitrate of the inbound audio stream is high, while at the same time keeping costs low by not permanently allocating the additional buffers to the stream (see Hauser at col. 3 lns. 10-30).
Li in view of Vaidya in view of Vafin in view of Hauser doesn’t describe the inbound audio stream being split into the segments based on comparing a detected silence to a threshold that is dynamically updated according to speaking habits of each user identified from the inbound audio stream.
However, Garg describes a system and method wherein the inbound audio stream being split into the segments based on comparing a detected silence to a threshold that is dynamically updated according to speaking habits of each user identified from the inbound audio stream (see col. 8 lns. 18-48: “In some situations, breaks or pauses in sentences can occur where a comma, period, semicolon, colon, question mark, or exclamation point might typically be placed when writing such sentences. As another example, the audio processing module 104 can be configured to provide a different label for pauses between two different speakers and/or between speech and music. Pauses such as these can enable the podcast summarization system 100 to divide the audio and text of the podcast content and stitch the divided portions together to form the audio summary.
In some examples, the audio processing module 104 can be configured to determine that a pause has occurred when the audio processing module 104 detects at least a threshold time gap (e.g., three seconds) exists between continuous speech. In other examples, the audio processing module 104 can use machine learning that can, over time and across training data sets of various podcast content, determine and update the threshold time gap and thus enable the podcast summarization system 100 to more efficiently recognize when a pause has occurred. In still other examples, the audio processing module 104 can be configured to map certain pause detection thresholds with certain speakers as the podcast summarization system 100 learns different speaking styles. For example, different speakers (even within the same episode) may have a different style and rate of speaking, and some may pause longer than others. Thus, the audio processing module 104, upon recognizing that a certain speaker is speaking, can select a threshold for that speaker and determine that, during a segment in which that speaker is speaking, a pause has occurred when that threshold time has elapsed with no speaking or music.” (emphasis added)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in Li in view of Vaidya in view of Vafin in view of Hauser a system and method wherein the inbound audio stream being split into the segments based on comparing a detected silence to a threshold that is dynamically updated according to speaking habits of each user identified from the inbound audio stream, as taught by Garg, in order to more accurately identify pauses in the audio stream for different speakers, which reduces the frequency of splitting the audio stream into incomplete segments that contain sentence fragments from a single speaker.
Regarding Claim 10, Li teaches:
10. The computer-implemented method of claim 1, wherein, before transcribing the voice included in a segment, the method further comprises:
performing a padding process on the split segments [Li: col. 10, line 23-27, col. 15, line 49-54— The speech audio is then divided into speech segments at likely sentence pauses and/or at likely speaker changes so that the resulting speech segments are more likely to contain the pronunciations of complete sentences by individual speakers. “In this way, each speech segment is more likely to contain a more complete set of the acoustic information needed to identify graphemes, phonemes, text characters, words, phrases, sentences etc. in the speech-to-text processing operations, thereby enabling greater accuracy in doing so,” which shows a padding process by ensuring a high accuracy in the information extracted from audio streams.]
Regarding Claim 11, Li teaches:
11. The computer-implemented method of claim 1, wherein, before transcribing the voice included in a segment, the method further comprises:
classifying a segment to a target user. [Li: col. 2 lines 65-67, col. 3 line 1-3— “a first speaker diarization technique including: divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio; for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers.]
Regarding Claim 13, Li teaches:
13. The computer-implemented method of claim 1, wherein performing natural language processing on the text further comprises:
filtering out irrelevant information from the text. [Li: col 10, line 35-37— During text analytics post-processing, the corresponding transcript is analyzed to select words that are pertinent to identifying topics or sentiments about topics,” indicating that irrelevant information can be filtered out during analysis.]
Regarding Claim 14, Li teaches:
14. The computer-implemented method of claim 1, wherein performing natural language processing on the text comprises:
performing a sentiment analysis of the text. [Li: col. 59, line 4-8— “various post-processing analyses may be performed of the text within the transcript to identify such features as the one or more topics that were spoken about, the relative importance of each topic, indications of sentiments, etc.,” which confirms that sentiment analysis is performed on the text.]
Regarding Claim 15, Li teaches:
15. The computer-implemented method of claim 1, wherein performing natural language processing on the text comprises:
performing an intent classification of a target user based on the text. [Li: col. 43, line 49-51— “The trained machine-learning model can analyze the new data and provide a result that includes a classification of the new data into a particular class,” which implies intent classification based on the text.”
Regarding Claim 16, Li teaches:
16. The computer-implemented method of claim 1, further comprises:
publishing insights obtained from the segments using a pub/sub process. [Li: col. 38, line 62-67— a publish/subscribe (pub/sub) capability is initialized for ESPE 800. In an illustrative embodiment, a pub/sub capability is initialized for each project of the one or more projects 802. To initialize and enable pub/sub capability for ESPE 800, a port number may be provided. Pub/sub clients can use a host name of an ESP.”]
Regarding Claim 20, Claim 20 is a system claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally, a memory, coupled to the processor of Claim 20 is taught by [Li: col. 52, lines 44-46—The system architecture includes processors with coupled storage/memory. “In various embodiments, each of the multiple node devices 2300 may incorporate one or more processors 2350, one or more neuromorphic devices 2355, a storage 2360.”]
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Vaidya in view Vafin in view of Hauser in view of Garg and further in view of Hodapp [U.S. 20130304916].
Regarding Claim 2, Li, as modified above, does not teach the computer-implemented method of claim 1, wherein the inbound audio stream is an HTTP live stream.
Hodapp discloses
the computer-implemented method of claim 1, wherein the inbound audio stream is an HTTP live stream. [Hodapp: Section 0006 “Apple introduced a system called HTTP Live Streaming, which streams audio and video in segments. The server breaks a media stream into segments, and sends out each segment individually. It has advantages over the technique of streaming from the leading edge. However, Apple's technique requires the clients to maintain a playlist of the segments and render them as a continuous stream. Such a requirement makes the Apple's live streaming client a special client which is incompatible with the existing regular HTTP clients.” [Hodapp: FIG. 1 and FIG. 3
PNG
media_image1.png
349
427
media_image1.png
Greyscale
PNG
media_image2.png
294
393
media_image2.png
Greyscale
]
Li, as modified above, and Hodapp are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Hodapp, which streams audio and video in segments and provides an improved mechanism for efficiently transmitting a live stream through HTTP protocols [Hodapp: Section 0005].
Claim 3-5, 8-9, 12 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Vaidya in view Vafin in view of Hauser in view of Garg and further in view of Pathak [U.S. 20240087572].
Regarding Claim 3, Li, as modified above, does not teach, but Pathak teaches the computer-implemented method of claim 1, wherein splitting the inbound audio stream into segments using a producer-consumer algorithm comprises:
enabling the producer thread to process one or more earliest portions of the inbound audio stream as audio blocks and store the audio blocks in the mediator buffer; and [Pathak: Section 0054— the system stores the initial segment of decoded streaming audio data in a cache. Then, after outputting the first portion of the initial segment, the system clears the cache of the first portion of the initial segment of the decoded streaming audio data.” This indicates a system that allows for processing audio segments (referred to as audio blocks) and storing these blocks in an intermediate storage area (the cache). Additionally, “while clearing the cache of the first portion of the initial segment of decoded streaming audio data, the system retains the second portion of the segment of decoded streaming audio data in the cache,” which suggests a capacity for handling multiple segments as the producer processes the audio data.]
responsive to the mediator buffer being full, enabling the consumer thread to consume the audio blocks in the mediator buffer, [Pathak: Section 0054— the system stores the initial segment of decoded streaming audio data in a cache. Then, after outputting the first portion of the initial segment, the system clears the cache of the first portion of the initial segment of the decoded streaming audio data.” This suggests that when the cache is full (analogous to the mediator buffer), it allows for the consumption of audio blocks. Additionally, it retains parts of the segment while clearing the cache, which implies that the consumer mechanism operates in response to the cache's state, enabling the processing of audio blocks once segments are made available for consumption.]
wherein consuming the audio blocks comprises splitting each of one or more of the audio blocks into two or more segments, and [Pathak: Section 0009— Audio data management is discussed here. “The systems apply a punctuation at the linguistic boundary and output a first portion of the initial segment of the streaming audio data.” This suggests a segmentation process based on linguistic boundaries, which aligns with the concept of dividing audio blocks into multiple segments.]
wherein consuming each audio block in the mediator buffer causes the mediator buffer to have more free space. [Pathak: Section 0054— the system stores the initial segment of decoded streaming audio data in a cache. Then, after outputting the first portion of the initial segment, the system clears the cache of the first portion of the initial segment of the decoded streaming audio data.” This indicates that when portions of audio are consumed from the cache (intermediary storage area), it results in freeing up space within that cache. Effective management of this process will create more free space.]
Li, as modified above, and Pathak are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Pathak, which focuses on the effective management of streaming audio data through improved segmentation and punctuation, enhancing transcription quality and readability for various application [Pathak: Section 0094].
Regarding Claim 4, Li, as modified above, does not teach the computer-implemented method of claim 3, further comprising: responsive to the mediator buffer being emptied by the consumer thread, enabling the producer thread to process one or more updated earliest portions of the inbound audio stream.
Pathak discloses
the computer-implemented method of claim 3, further comprising: responsive to the mediator buffer being emptied by the consumer thread, enabling the producer thread to process one or more updated earliest portions of the inbound audio stream. [Pathak: Section 0054— In some embodiments, the computing system utilizes a cache which facilitates the improved timing of output of the different speech segments. For example, the system stores the initial segment of decoded streaming audio data in a cache. Then, after outputting the first portion of the initial segment, the system clears the cache of the first portion of the initial segment of the decoded streaming audio data. In further embodiments, while clearing the cache of the first portion of the initial segment of decoded streaming audio data, the system retains the second portion of the segment of decoded streaming audio data in the cache. Embodiments that utilize a cache in this manner improve the functioning of the computing system by efficiently managing the storage space of the cache by deleting data that has already been output and retaining data that will be needed in order to continue to generate accurately punctuated outputs. Additionally, it describes how segments are handled indicating that during this process, the system efficiently manages data flow, suggesting a responsive mechanism where the consumption of data by a thread allows for the production of updated audio stream portion. “Automatic speech recognition systems and other speech processing systems are used to process and decode audio data to detect speech utterances (e.g., words, phrases, and/or sentences). The processed audio data is then used in various downstream tasks such as search-based queries, speech to text transcription, language translation, closed captioning, etc. Oftentimes, the processed audio data needs to be segmented into a plurality of audio segments before being transmitted to downstream applications, or to other processes in streaming mode.”]
Li, as modified above, and Pathak are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Pathak, which focuses on the effective management of streaming audio data through improved segmentation and punctuation, enhancing transcription quality and readability for various application [Pathak: Section 0094].
Regarding Claim 5, Li in view of Vaidya doesn’t describe a system and method of claim 3, wherein the size of the mediator buffer is dynamically adjusted based on a change of the bitrate of the inbound audio stream in a streaming process.
However, Vafin describes wherein the size of the mediator buffer is dynamically adjusted based on a change of the bitrate of the inbound audio stream in a streaming process (FIG. 8 and col. 14 lns. 16-37: “In step S802 processing parameters which would be beneficial for the transmitter 102 to use when processing data for transmission to the receiver 108 are determined. These processing parameters may be adjusted versions of the processing parameters which are determined based on the state of the jitter buffer 114 as described above. Alternatively, the processing parameters may be determined in step S802 without considering the state of the jitter buffer 114. As described above, the processing parameters may include one or more of the encoding bit rate, the FEC depth, a packetization delay and an interleaving delay.
In step S804 the processing parameters determined in step S802 are used to determine a state of the jitter buffer 114 which would be suited for receiving data which has been processed in accordance with the processing parameters determined in step S802. For example, if the encoding bit rate is increased then the available space in the jitter buffer 114 may need to be increased to accommodate the extra data that is to be transmitted due to the increase in the encoding bit rate. The jitter buffer size should be adapted in dependence on a change in the encoding bit rate of the data and in dependence on a bottleneck of the transmission path.” (emphasis added)
Also see col. 15 lns. 6-16: “In step S808 the state of the jitter buffer 114 at the receiver 108 is adjusted based on the indication of the jitter buffer state which has been received from the transmitter 102. In this way the jitter buffer 114 is placed in a state which is suited (i.e. optimized) for receiving the data which is transmitted from the transmitter 102 to the jitter buffer 114 of the receiver 108. Therefore when the data is processed at the transmitter 102 (in accordance with the processing parameters, as described above) and transmitted to the jitter buffer 114 of the receiver 108 then the jitter buffer 114 is in a state suited to receiving that data.” (emphasis added)
Further, see col. 6 lns. 41-50: “The data may be encoded using any suitable known encoding technique. Particular encoding techniques may be appropriate for different types of data. For example, where the data includes speech from a user of the transmitter 102 then the encoding technique may include a specific speech encoder for encoding the speech portions of the data. Other encoding techniques may also be used, e.g. to compress the data for transmission to the receiver 108. The data is processed using an encoding bit rate such that the encoded data has the particular encoding bit rate.” (emphasis added)
Finally, see col 6 ln. 57 – col. 7 ln. 9: “For example, the input signal may be a video signal and the encoder may produce a sudden increase (peak) in the instantaneous bit rate, as shown in FIG. 3. FIG. 3 shows a graph representing the number of bytes of each frame of a video signal once it has been encoded at the transmitter 102. It can be seen that the encoding bit rate is not the same for each video frame, and in particular some of the video frames (e.g. frame number 0 and frame number 100) are encoded at a much higher bit rate than the other video frames. The variation in the bit rate of the frames of the video signal may occur because different encoding techniques are used to encode the different video frames. A similar situation may occur when the input signal is an audio signal. For example, when a portion of an audio or a video signal is encoded independently of previously encoded portions then the bit rate of the encoded video or audio frame is typically relatively high. This is in contrast to other ones of the video or audio frames which may be encoded using differential-coding techniques, and therefore the encoded bit rate of those other video or audio frames is typically relatively low.” (emphasis added)).
See the rejection of claim 1 for rationale to modify Li in view of Vaidya based on the teachings of Vafin, as it is equally applicable here.
Regarding Claim 8, Li teaches:
8. The computer-implemented method of claim 3, wherein splitting the inbound audio stream into segments based on silence detection comprises:
detecting a period of silence included in an audio block based on audio waves detected on the audio block; [Li: col. 5, line 21-26— “analyze the speech audio to identify pauses between speech sounds by providing each data chunk of the multiple data chunks to the instance of the acoustic model neural network as an input and monitor the CTC output for at least one corresponding string of blank symbols indicative of a pause.”
comparing a time length of the period of silence to the threshold; and [Li: col. 5, line 26-30— “analyze the lengths of the pauses to identify a first set of likely sentence pauses by comparing a length of each string of blank symbols from the CTC output to a predetermined blank threshold length.”
responsive to the time length of the period of silence being larger than the threshold, determining that the period of silence is a place to split the audio block. [Li: col. 5, line 6-14— “Dividing the speech data set into multiple data segments based on at least the first set of likely sentence pauses may include: using the second relative weighting to combine the first set of likely sentence pauses and the second set of likely sentence pauses to generate a single set of indications of likely sentence pauses; and dividing the speech data set into multiple data segments based on at least the single set of indications of likely speaker changes and the single set of indications of likely sentence pauses,” when thresholds are met.]
Regarding Claim 9, Li teaches:
9. The computer-implemented method of claim 8,
wherein detecting a period of silence included in an audio block comprises detecting multiple periods of silence in the audio block. [Li: col. 11 line 31-39— “In preparation for the performance of the multiple pause detection techniques, the speech audio may be initially divided into equal-length chunks. The full set of chunks of the speech audio may then be provided as an input to each of multiple pause detection techniques, which may be performed, at least partially in parallel, to each independently generate its corresponding data structure specifying its corresponding set of likely sentence pauses present within the speech audio,” which indicates that silence detection can be performed across these division. “The separate sets of indications of likely sentence pauses derived by each of the pause detection techniques may then be used to identify, within each chunk, any fragments that likely include a sentence pause such that there is at least a portion of the speech audio within such fragments that likely does not include speech sounds. Such “non-speech” fragments may then be removed from each chunk.” This support the fact that identifying a silence period involves multiple detection of silence within an audio block.]
Regarding Claim 12, Li teaches the computer-implemented method of claim 1, wherein, before transcribing the voice included in a segment, the method further comprises:
extracting audio features included in the segment, the segment belonging to a target user; and [Li: col. 2 lines 65-67, col. 3 line 1-3— “a first speaker diarization technique including: divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio; for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers.]
Li, as modified above, does not teach detecting emotion of the user based on the extracted audio features.
Pathak discloses
detecting emotion of the user based on the extracted audio features.
[Pathak: Section 0065—The system implemented sentiment identification based on spoken attributes. “Sentiments correspond to an emotion that the user is likely to be experiencing based on attributes of their spoken language utterances.”]
Li, as modified above, and Pathak are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Pathak, which focuses on the effective management of streaming audio data through improved segmentation and punctuation, enhancing transcription quality and readability for various application [Pathak: Section 0094].
Claim 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Vaidya in view Vafin in view of Hauser in view of Garg in view of Pathak and further in view of Koval [U.S. 5339413 A].
Regarding Claim 6, Li, as modified above, does not teach the computer-implemented method of claim 3, wherein the consumer thread is disabled when the producer thread is enabled to process the one or more earliest portions of the inbound audio stream and store the audio blocks in the mediator buffer.
Koval discloses
the computer-implemented method of claim 3, wherein the consumer thread is disabled when the producer thread is enabled to process the one or more earliest portions of the inbound audio stream and store the audio blocks in the mediator buffer.
[Koval: col. 9 lns. 5-29 — “In response to a start stream call being made in the application program, manager 114 sends (via steps 436 and 438--FIG. 4) SHC.START commands first to the source thread handler and then to the target stream handler. The source stream handler needs to be started first to fill stream buffers before the target handler can use the data being transferred thereto. In response to receiving such command in step 206 (FIG. 6), the source stream handler in step 208 unblocks the source thread. In response to being unblocked or awakened in step 210, source thread then requests, in step 216, an empty buffer from manager 114. If an empty buffer is not available, as determined in step 218, step 220 then blocks the thread again. If an empty buffer is available, then step 222 reads data from the source device and fills the buffer. Step 224 then returns the filled buffer to manager 114. Step 226 decides if any more buffers need filling. If so, a branch is made back to step 216 and a loop is formed from steps 216-226 which loop is broken when step 226 decides no more buffers need filling. Then, the thread is blocked. Once the streaming operation has been started, the buffer filling process repeats until the end of the source file is reached at which point the source thread quiesces.” The source thread is enabled to fill buffers with audio data, while the target handler waits to use that data, also the mechanism of one thread being blocked while the other is active is conveyed.]
Li, as modified above, and further in view of Koval are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Koval, which utilizing a multimedia data processing system that effectively manages audio data streaming through synchronized buffer allocation and negotiation protocol [Koval: col. 5, line 18-28].
Regarding Claim 7, Li, as modified above, does not teach the computer-implemented method of claim 3, wherein the producer thread is disabled when the consumer thread is enabled to consume the audio blocks in the mediator buffer.
Koval discloses
the computer-implemented method of claim 3, wherein the producer thread is disabled when the consumer thread is enabled to consume the audio blocks in the mediator buffer. [Koval: col. 9 lns. 5-29 — “In response to a start stream call being made in the application program, manager 114 sends (via steps 436 and 438--FIG. 4) SHC.START commands first to the source thread handler and then to the target stream handler. The source stream handler needs to be started first to fill stream buffers before the target handler can use the data being transferred thereto. In response to receiving such command in step 206 (FIG. 6), the source stream handler in step 208 unblocks the source thread. In response to being unblocked or awakened in step 210, source thread then requests, in step 216, an empty buffer from manager 114. If an empty buffer is not available, as determined in step 218, step 220 then blocks the thread again. If an empty buffer is available, then step 222 reads data from the source device and fills the buffer. Step 224 then returns the filled buffer to manager 114. Step 226 decides if any more buffers need filling. If so, a branch is made back to step 216 and a loop is formed from steps 216-226 which loop is broken when step 226 decides no more buffers need filling. Then, the thread is blocked. Once the streaming operation has been started, the buffer filling process repeats until the end of the source file is reached at which point the source thread quiesces.” The source thread is enabled to fill buffers with audio data, while the target handler waits to use that data, also the mechanism of one thread being blocked while the other is active is conveyed.]
Li, as modified above, and further in view of Koval are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Koval, which utilizing a multimedia data processing system that effectively manages audio data streaming through synchronized buffer allocation and negotiation protocol [Koval: col. 5, line 18-28].
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Vaidya in view of Vafin in view of Hauser in view of Garg and further in view of Hundemer [U.S. 20160294494]
Regarding Claim 17, Li does not teach the computer-implemented method of claim 1, wherein the inbound audio stream comprises a plurality of channels of audio stream.
Hundemer discloses:
the computer-implemented method of claim 1, wherein the inbound audio stream comprises a plurality of channels of audio stream. [Hundemer: Section 0044— “Depending on the manner in which the first signature is generated, the first signature may represent a time of and/or a type of one or more single-channel/multi-channel transitions. Notably, in the case where the first audio-stream does not include any single-channel/multi-channel transitions, the first signature may indicate that no such single-channel/multi-channel transitions are present.”]
Li, as modified above, and Hundemer are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Hundemer, which monitors backup audio broadcast functionality by comparing signatures based on single-channel/multi-channel transition between primary and backup audio streams, outputting alerts when significant differences are detected [Hundemer: Abstract].
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Vaidya in view of Vafin in view of Hauser in view of Garg in view of Hundemer and further in view of Sayadi [US 20200163627].
Regarding Claim 18, Li, as modified above, does not teach the computer-implemented method of claim 17, wherein splitting the inbound audio stream into segments using a producer-consumer algorithm further comprises: creating a worker architecture comprising a plurality of audio processing channel workers, wherein each audio processing channel worker implements a producer-consumer algorithm-based process to split one channel of the inbound audio stream.
Sayadi discloses:
the computer-implemented method of claim 17, wherein splitting the inbound audio stream into segments using a producer-consumer algorithm further comprises: creating a worker architecture comprising a plurality of audio processing channel workers, wherein each audio processing channel worker implements a producer-consumer algorithm-based process to split one channel of the inbound audio stream. [Sayadi: Section 0035— Design of real-time data stream processing has been developed in an event-based form using an actor model of programming. This enables a producer/consumer model for algorithm components that provides a number of advantages over more traditional architectures. For example, it enables reuse and rapid prototyping of processing and algorithm modules. As another example, data streams can be enabled/disabled dynamically and routed to or from modules at any point within a group of modules comprising an algorithmic system, enabling computation to be location-independent (i.e., on a single device, combined with one or more additional devices or servers, on a server only, etc.).”]
Li, as modified above, and Sayadi are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Sayadi, which provides non-contact biosignal monitoring ballistocardiogram sensors to generate synthetic cardiorespiratory data and audio streams for long-term health tracking without physical attachment [Sayadi: Abstract].
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Vaidya in view of Vafin in view of Hauser in view of Garg in view of Hundemer in view of Sayadi and further in view of Woodward [U.S 8983836].
Regarding Claim 19, Li, as modified above, does not teach the computer-implemented method of claim 18, wherein the worker architecture further comprises a plurality of content filtering workers, each content filtering worker implementing a content filtering process to remove noise or irrelevant information from the segments or text transcribed from the segments.
Woodward discloses the computer-implemented method of claim 18, wherein the worker architecture further comprises a plurality of content filtering workers, each content filtering worker implementing a content filtering process to remove noise or irrelevant information from the segments or text transcribed from the segments.
[Woodward: col. 13, line 57-66— “The identification of the background sounds/noises by the segmentation engine 340 may be used to associate a generic audio waveform pattern with the segment for filtering out the background sounds and noises during ASR engine 330 analysis. That is, the background sound/ noise profile may contain background sound/noise waveform patterns such as electric fans or hums etc. that may be applied to the audio portion to reduce or filter the corresponding background sounds/ noise in the audio portion of the multimedia content.”]
Li, as modified above, and Woodward are considered analogous art because they were in the similar field related to Audio Processing Technology, specifically focusing on real-time audio stream management. It utilizes methods such as a producer-consumer algorithm and natural language processing (NLP) for effective data extraction and analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Li, as modified above, to combine the teaching of Woodward, which provide mechanisms for performing noise-free captioning of audio and/or multimedia content using acoustic profiles derived from social network sources [Woodward: col. 2, line 63-65], which results in more accurate captioning/transcribing.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure – see additional references cited on PTO-892.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daniel C Washburn whose telephone number is (571)272-5551. The examiner can normally be reached Monday-Friday 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights