DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Introduction
This office action is in response to communications filed 03/03/2025. Claims 1-20 are pending and likewise have been examined.
Information Disclosure Statement
The information disclosure statement filed 03/08/2025 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered. Non-patent literature document #23 Jensen et al. was not provided by Applicant. A different non-patent literature by Jensen et al. was filed on 03/09/2025, titled “Speech intelligibility prediction based on mutual information”, however, this is not the same reference, the title is significantly different. This filed Jensen NPL was not cited in any other IDS. Examiner also notes that date for NPL Izmailov is incorrect. Examiner has added the correct date to the IDS.
The information disclosure statement (IDS) submitted on 03/04/2025, 03/09/2025 and 08/03/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
In the specification, the number “530” is used for “active speaker 530” in Para 538, as well as “processing layer 530” directly before Para 558. The same number should not be used for two different things.
Appropriate correction is required.
Claim Objections
Claim 1 is objected to because of the following informalities:
The preamble for Claim 1 contains functional limitations which are relied on in the body of the claim for antecedent basis, specifically participants, audio streams and participant interaction data.
Appropriate correction is required.
Claim 17 is objected to because of the following informalities: The “to determines” on line 3 is plural. This is grammatically incorrect given the context of the sentence and causes confusion.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “feature extractor”, in claim 1, “attention neural network”, in claim 1, “an adaptive audio gain controller”, in claim 1, “said adaptive audio gain controller in Claim 4. Claims 2-3 and 5-7 also invoke 112(f) as they contain the limitations of Claim 1. “a wearable device controller” in Claims 6 and 7(addressed in 112(b) rejections below). “distributed receivers” in Claim 14, “attention neural networks” in Claim 14, “distributed adaptive audio gain controllers” in Claim 14. Claims 15 and 16 also invoke 112(f) as they contain the limitations of Claim 14. Structure for above limitations can be found in Para 404 of the specification.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 1 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention. On line 2-5 of Claim 1, “receives input audio streams from participants and transmits an output audio stream to participants, each input audio stream being generated from the output audio streams”. This Claims language causes confusion as to how the inputs audio streams are created from the output audio streams, when the input audio streams are already defined as being from the participants. It is not clear if the output audio streams are from other participants and being received as audio input streams at the current participant. The contradictions of the above limitations are not made clear by the specification.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-13 and 15-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being ambiguous. See MPEP § 2173.02. On line 2-5 of Claim 1, “receives input audio streams from participants and transmits an output audio stream to participants, each input audio stream being generated from the output audio streams”. This Claim omits matter that appears to be necessary to adequately describe the invention. This claim language causes confusion as to how the inputs audio streams are created from the output audio streams, when the input audio streams are already defined as being from the participants. It is not clear if the output audio streams are from other participants and being received as audio input streams at the current participant. The contradictions of the above limitations are not made clear by the specification.
Claim 1 recites the limitation "the output audio streams" in line 5. There is insufficient antecedent basis for this limitation in the claim. While the claim does establish “an output audio stream” on line 4, a plurality is not established, so the antecedent basis in improper.
Claim 1 recites the limitation "the input audio gain" in line 13-14. There is insufficient antecedent basis for this limitation in the claim. “an input audio gain” has not been established prior to this point in the claim.
Claim 2 recites the limitation "the participants" in line 1. There is insufficient antecedent basis for this limitation in the claim. “participants” has been established multiple times in claim 1, on lines 3 and 4, which do not reference each other through antecedent basis. It is not clear which of the instances in Claim 1 is being referenced in Claim 2.
Claim 8 recites the limitation "the input audio gain" in line 11. There is insufficient antecedent basis for this limitation in the claim. “an input audio gain” has not been established prior to this point in the claim.
Claim 10 recites the limitation "the pair consisting of the first and the second participant" in line 4-5. There is insufficient antecedent basis for this limitation in the claim. “the first” should include the word “participant” in order to have proper antecedent basis. The “participant” after the word “second” is not plural, so it is not clear that instant of “participant” is also being applied to “the first”.
Claim 10 recites the limitation "the attention probability for the pair consisting of the first and the second participant" in line 4-5. There is insufficient antecedent basis for this limitation in the claim. No “pair consisting of the first and the second participant” has been established prior to this point in the claim.
Claim 10 recites the limitation "the attention probability for the pair consisting of the first and the second participant" in line 4-5. There is insufficient antecedent basis for this limitation in the claim. No “attention probability for pair consisting of the first and the second participant” has been established prior to this point in the claim.
Claim 10 recites the limitation "the audio output stream" in line 6. There is insufficient antecedent basis for this limitation in the claim. The limitation should read “the output audio stream” in order to have current antecedent basis.
The term “low” in claim line 6 of claim 10, is a relative term which renders the claim indefinite. The term “low” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “low” makes the term “probability” indefinite.
The term “high” in claim line 7 of claim 10, is a relative term which renders the claim indefinite. The term “high” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “high” makes the term “probability” indefinite.
Claim 12 recites the limitation "the participants’ states of attention" in line 2. There is insufficient antecedent basis for this limitation in the claim. “participants’ states of attention” has not been established prior to this point in the claim. Claim 12 was most likely supposed to be dependent on claim 11, rather than claim 8.
Claim 15 recites the limitation "the audio outputs" in line 3. There is insufficient antecedent basis for this limitation in the claim. “audio outputs” has not been established prior to this point in the claim. “an audio output stream” was established in Claim 1 but does not provide proper antecedent basis.
Claim 16 recites the limitation "the multimodal participant input data" in 2. There is insufficient antecedent basis for this limitation in the claim. The word “input” should be “interaction” in order for the limitation to have proper antecedent basis to “multimodal participant interaction data” in claim 14.
Claim 17 recites the limitation "the multimodal participant input data" in line 2. There is insufficient antecedent basis for this limitation in the claim. "multimodal participant input data" has not been established prior to this point in the claim.
Claim 17 recites the limitation “said attention neural networks" in line 1. There is insufficient antecedent basis for this limitation in the claim. " said attention neural networks" has not been established prior to this point in the claim.
Claim 19 recites the limitation “the participants" in line 4. There is insufficient antecedent basis for this limitation in the claim. "the participants" has not been established prior to this point in the claim.
Claims 3-7, 9, 11, 13, 18 and 20 are also rejected under 35 U.S.C. 112(b), as they depend on a claim that has been rejected above.
Claims 6 and 7 recite the limitation “a wearable device controller” which invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. See Claim interpretation above for form paragraph regarding claim interpretation under 112(f). However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The disclosure is devoid of any structure that performs the function in the claim. Therefore, the claims are indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 17 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 17 is a system claim, which is claiming dependency on “the system of claim 13”. Claim 13 is a method claim, not a system claim. Rather than Claim 17 being a method claim, it was most likely intended for Claim 17 to be dependent on a different claim. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claims 18-20 are also rejected under 35 U.S.C. 112(d) as they depend on Claim 17.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 8, 10-11, 14 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Haro (US 20230146178 A1), and further in view of Horton et al (US 20230386469 A1).
Regarding Claim 1:
Haro teaches apparatus for selective auditory attention in multi-participant environments, wherein each of a plurality of wearable devices or a combination thereof receives input audio streams from participants and transmits an output audio stream to participants, each input audio stream being generated from the output audio streams, and wherein one or more sensors capture participant interaction data, the apparatus comprising(Para [0005], Ln 1-29, identifying, at a processor and at a first time, a first estimated gaze direction of a first participant from a plurality of participants within a virtual environment. The method also includes receiving, at the processor and from a compute device of the first participant, first audio data including a first set of at least one audio parameter…… automatically generating, via the processor, second audio data including a second set of at least one audio parameter different from the first set of at least one audio parameter, based on the first audio data and the second estimated gaze direction. The second audio data can include a modification relative to the first audio data, and can be associated with one of (1) a virtual representation of a second participant from the plurality of participants…. The method also includes sending a signal representing the second audio data from the processor to the compute device of the first participant. Para [0041], Ln 1-9, generate immersive spaces with spatial audio for environmental sounds through VR headsets. Para [0059], Ln 1-10, User 1 is wearing headphones):
and an adaptive audio gain controller dynamically controlling the input audio gain received by each wearable device or combination of wearable devices based on the computed attention probabilities(Para [0059], Ln 13-30, predicts that the attention of user 1 is directed to user 2 (e.g., based on the received video of user 4). Based on this prediction, the spatial audio engine applies an adjustment to a parameter of the audio data associated with user 2, and optionally transmits the adjusted audio data to a compute device of user 1 for implementation thereon (e.g., such that user 1 hears the audio output of user 2 more prominently than other users and/or sounds). The spatial audio engine tunes the virtual environment experience in real-time in this manner, such that the relative volumes/audio properties of audio sources. Para [0082], Ln 8-23, An ML-physics kernel can refer to a probabilistic mathematical representation of a physics kernel. Para [0096], Ln 1-10, In some embodiments, a method for attention determination and associated audio adjustment includes generating a 3D auditory display using a HRTF, which may be modeled using a deep neural network (DNN) based on spatial principal component analysis).
Haro does not specifically teach a feature extractor extracting multimodal features from the participant interaction data; an attention neural network determining attention relationships between participants based on the extracted features, and computing attention probabilities for each pair of participants, based on the attention relationships.
In the same field of user attention modeling, Horton teaches a feature extractor extracting multimodal features from the participant interaction data(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input);
an attention neural network determining attention relationships between participants based on the extracted features, and computing attention probabilities for each pair of participants, based on the attention relationships(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input. Para [0264], Ln 1-3, a user's visual attention is directed to an electronic device while the user is speaking (looking and speaking). Haro teaches the bidirectional attention determination, Horton is only needed to teach the multimodal neural network for attention determination).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify Haro with the Multimodal model of Horton as it help improve accuracy of responses(Para [0018], Ln 1-7).
Regarding Claim 2:
The combination of Haro and Horton teaches the apparatus of claim 1, but does not teach wherein the participants include humans and artificial intelligence (AI) assistants.
In the same field of user attention modeling, Horton teaches wherein the participants include humans and artificial intelligence (AI) assistants(Para [0004], Ln 1-17, identifying a second portion of the audio stream to include user speech intended for the electronic device; initiating, by a digital assistant operating on the electronic device, a task based the second portion of the audio stream; and providing an output indicative of the initiated task).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton with the Attention system of Horton as it help improve accuracy of responses(Para [0018], Ln 1-7).
Regarding Claim 3:
The combination of Haro and Horton teaches the apparatus of claim 1, and Haro teaches wherein the participant interaction data comprises at least one of spatial positioning data, head orientation data, body movement data, gaze direction data, audio activity data, EEG data and biometric data(Abstract, Ln 1-7, An attention-based audio adjustment method includes identifying, at a processor and at a first time, a first estimated gaze direction of a first participant within a virtual environment. Also previous combination with Horton teaches this limitation as well, Horton: Para [0263], Ln 1-14, neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input).
Regarding Claim 4:
The combination of Haro and Horton teaches the apparatus of claim 1, and Haro teaches wherein said adaptive audio gain controller dynamically adjusts relative audio levels of different output audio streams based on the attention probabilities(Para [0044], Ln 1-23, one or more audio settings or parameters of a compute device of the first videoconference user and related to the second videoconference user may be adjusted (e.g., automatically via a processor executing processor-readable instructions), for example such that a volume of audio associated with the second user is increased/made louder relative to other videoconference users. In some such implementations, audio settings or parameters for other audio sources associated with the videoconference session (e.g., background noise) may not be adjusted at all, or may be adjusted (e.g., an associated volume thereof may be reduced)).
Regarding Claim 5:
The combination of Haro and Horton teaches the apparatus of claim 1, but does not teach wherein said attention neural network classifies each participant according to his state of attention, based on the participant interaction data.
In the same field of user attention modeling, Horton teaches wherein said attention neural network classifies each participant according to his state of attention, based on the participant interaction data(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input. Para [0264], Ln 1-3, a user's visual attention is directed to an electronic device while the user is speaking (looking and speaking). Attention determination for additional participants is taught by Haro, as shown in rejection to Claim 1).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton with the Attention system of Horton as it help improve accuracy of responses(Para [0018], Ln 1-7).
Regarding Claim 8:
Haro teaches a method for selective audio attention in multi-participant environments, each participant wearing a device or a combination of devices that receives input audio streams from other participants and that transmits an output audio stream to other participants, the method comprising(Para [0005], Ln 1-29, identifying, at a processor and at a first time, a first estimated gaze direction of a first participant from a plurality of participants within a virtual environment. The method also includes receiving, at the processor and from a compute device of the first participant, first audio data including a first set of at least one audio parameter…… automatically generating, via the processor, second audio data including a second set of at least one audio parameter different from the first set of at least one audio parameter, based on the first audio data and the second estimated gaze direction. The second audio data can include a modification relative to the first audio data, and can be associated with one of (1) a virtual representation of a second participant from the plurality of participants…. The method also includes sending a signal representing the second audio data from the processor to the compute device of the first participant. Para [0041], Ln 1-9, generate immersive spaces with spatial audio for environmental sounds through VR headsets. Para [0059], Ln 1-10, User 1 is wearing headphones):
and dynamically controlling the input audio gain received by each wearable device based on the computed attention probabilities(Para [0059], Ln 13-30, predicts that the attention of user 1 is directed to user 2 (e.g., based on the received video of user 4). Based on this prediction, the spatial audio engine applies an adjustment to a parameter of the audio data associated with user 2, and optionally transmits the adjusted audio data to a compute device of user 1 for implementation thereon (e.g., such that user 1 hears the audio output of user 2 more prominently than other users and/or sounds). The spatial audio engine tunes the virtual environment experience in real-time in this manner, such that the relative volumes/audio properties of audio sources. Para [0082], Ln 8-23, An ML-physics kernel can refer to a probabilistic mathematical representation of a physics kernel. Para [0096], Ln 1-10, In some embodiments, a method for attention determination and associated audio adjustment includes generating a 3D auditory display using a HRTF, which may be modeled using a deep neural network (DNN) based on spatial principal component analysis).
Haro does not specifically teach receiving participant interaction data from one or more sensors; determining attention relationships between participants based on the participant interaction data; computing attention probabilities for each pair of participants, based on the attention relationships.
In the same field of user attention modeling, Horton teaches receiving participant interaction data from one or more sensors(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input);
determining attention relationships between participants based on the participant interaction data(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input. Para [0264], Ln 1-3, a user's visual attention is directed to an electronic device while the user is speaking (looking and speaking). Haro teaches the bidirectional attention determination, Horton is only needed to teach the multimodal probability for attention determination);
computing attention probabilities for each pair of participants, based on the attention relationships(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input. Para [0264], Ln 1-3, a user's visual attention is directed to an electronic device while the user is speaking (looking and speaking). Haro teaches the bidirectional attention determination for participants, Horton is only needed to teach the multimodal probability for attention determination).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify Haro with the Multimodal model of Horton as it help improve accuracy of responses(Para [0018], Ln 1-7).
Regarding Claim 10:
The combination of Haro and Horton teaches the method of claim 8, and Haro teaches wherein said dynamically controlling comprises applying gain levels to received audio streams commensurate with the attention probabilities, to attenuate an audio stream arriving from a first participant received by a second participant when the attention probability for the pair consisting of the first and the second participant is a low probability, and to amplify the audio output stream when the attention probability of the pair is a high probability(Para [0059], Ln 13-30, predicts that the attention of user 1 is directed to user 2 (e.g., based on the received video of user 4). Based on this prediction, the spatial audio engine applies an adjustment to a parameter of the audio data associated with user 2, and optionally transmits the adjusted audio data to a compute device of user 1 for implementation thereon (e.g., such that user 1 hears the audio output of user 2 more prominently than other users and/or sounds). The spatial audio engine tunes the virtual environment experience in real-time in this manner, such that the relative volumes/audio properties of audio sources. Para [0082], Ln 8-23, An ML-physics kernel can refer to a probabilistic mathematical representation of a physics kernel. Para [0096], Ln 1-10, In some embodiments, a method for attention determination and associated audio adjustment includes generating a 3D auditory display using a HRTF, which may be modeled using a deep neural network (DNN) based on spatial principal component analysis. Para [0044], Ln 13-23, audio settings or parameters for other audio sources associated with the videoconference session (e.g., background noise) may not be adjusted at all, or may be adjusted (e.g., an associated volume thereof may be reduced), concurrently or overlapping in time with the adjustment to the one or more audio settings or parameters related to the second videoconference user).
Regarding Claim 11:
The combination of Haro and Horton teaches the method of claim 8, but does not teach further comprising classifying each participant according to his state of attention, based on the participant interaction data.
In the same field of user attention modeling, Horton teaches further comprising classifying each participant according to his state of attention, based on the participant interaction data(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input. Para [0264], Ln 1-3, a user's visual attention is directed to an electronic device while the user is speaking (looking and speaking). Attention determination for additional participants is taught by Haro, as shown in rejection to Claim 1).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton with the Attention system of Horton as it help improve accuracy of responses(Para [0018], Ln 1-7).
Regarding Claim 14:
Haro teaches a system for attention-based audio control for a plurality of participants, each participant wearing a device or a combination of devices that receives input audio streams from other participants and that transmits an output audio stream to other participants, the system comprising(Para [0005], Ln 1-29, identifying, at a processor and at a first time, a first estimated gaze direction of a first participant from a plurality of participants within a virtual environment. The method also includes receiving, at the processor and from a compute device of the first participant, first audio data including a first set of at least one audio parameter…… automatically generating, via the processor, second audio data including a second set of at least one audio parameter different from the first set of at least one audio parameter, based on the first audio data and the second estimated gaze direction. The second audio data can include a modification relative to the first audio data, and can be associated with one of (1) a virtual representation of a second participant from the plurality of participants…. The method also includes sending a signal representing the second audio data from the processor to the compute device of the first participant. Para [0041], Ln 1-9, generate immersive spaces with spatial audio for environmental sounds through VR headsets. Para [0059], Ln 1-10, User 1 is wearing headphones):
and distributed adaptive audio gain controllers for modifying audio signal gain from each device or combination of devices, based on the attention patterns(Para [0059], Ln 13-30, predicts that the attention of user 1 is directed to user 2 (e.g., based on the received video of user 4). Based on this prediction, the spatial audio engine applies an adjustment to a parameter of the audio data associated with user 2, and optionally transmits the adjusted audio data to a compute device of user 1 for implementation thereon (e.g., such that user 1 hears the audio output of user 2 more prominently than other users and/or sounds). The spatial audio engine tunes the virtual environment experience in real-time in this manner, such that the relative volumes/audio properties of audio sources. Para [0082], Ln 8-23, An ML-physics kernel can refer to a probabilistic mathematical representation of a physics kernel. Para [0096], Ln 1-10, In some embodiments, a method for attention determination and associated audio adjustment includes generating a 3D auditory display using a HRTF, which may be modeled using a deep neural network (DNN) based on spatial principal component analysis).
Haro does not specifically teach distributed receivers for receiving multimodal participant interaction data from one or more sensors; attention neural networks for determining attention patterns based on the multimodal participant interaction data.
In the same field of user attention modeling, Horton teaches distributed receivers for receiving multimodal participant interaction data from one or more sensors(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input);
attention neural networks for determining attention patterns based on the multimodal participant interaction data(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input. Para [0264], Ln 1-3, a user's visual attention is directed to an electronic device while the user is speaking (looking and speaking). Haro teaches the bidirectional attention determination for each participant, Horton is only needed to teach the multimodal neural network for attention determination).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify Haro with the Multimodal model of Horton as it help improve accuracy of responses(Para [0018], Ln 1-7).
Regarding Claim 16:
The combination of Haro and Horton teaches the system of claim 14, but does not teach wherein said attention neural networks employ machine learning to process the multimodal participant input data, and to determine attention relationships and predict changes in attention patterns.
In the same field of user attention modeling, Horton teaches wherein said attention neural networks employ machine learning to process the multimodal participant input data, and to determine attention relationships and predict changes in attention patterns(Para [0263], Ln 1-14, some examples, audiovisual invocation module 806 implements a machine learned model trained to determine whether a user's visual attention is directed to an electronic device while the user is speaking. The machine learned model is implemented, for instance, as a neural network (e.g., a recurrent neural network, a convolutional neural network, a feed forward neural network, or the like) and is configured to accept the contents of memory buffer 804 (e.g., portions of the audio stream and the video stream) as input. Para [0264], Ln 1-3, a user's visual attention is directed to an electronic device while the user is speaking (looking and speaking)).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton with the Multimodal model of Horton as it help improve accuracy of responses(Para [0018], Ln 1-7).
Claim(s) 6 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Haro and Horton as applied to claim 5 and 8 above, and further in view of Zhang (US 12198427 B2).
Regarding Claim 6:
The combination of Haro and Horton teaches the apparatus of claim 5, but does not teach further comprising a wearable device controller adjusting visual elements on wearable devices based on the participants' states of attention.
In the same field of Attention determination, Zhang teaches further comprising a wearable device controller adjusting visual elements on wearable devices based on the participants' states of attention(Abstract, Ln 1-17, adaptively visualizing content in an artificial environment based on the attention of a user. In one particular aspect, a computer-implemented method is provided that includes obtaining input data from a user, inferring content that is of interest to the user based on features gathered from the user's attention in the input data, identifying virtual content data based on the content that is of interest to the user, determining modifications to be applied to the virtual content data based on relevancy, applying the modifications to the virtual content data to generate a final format for the virtual content data, and rendering virtual content in the extended reality environment displayed to the user. See Fig 2A).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton with the VR system of Zhang, as it improves the user’s ability to understand content(Col 2, Ln 1-4. Col 2, Ln 8-20).
Regarding Claim 12:
The combination of Haro and Horton teaches the method of claim 8, but does not teach further comprising adjusting visual elements on wearable devices based on the participants' states of attention.
In the same field of Attention determination, Zhang teaches further comprising adjusting visual elements on wearable devices based on the participants' states of attention(Abstract, Ln 1-17, adaptively visualizing content in an artificial environment based on the attention of a user. In one particular aspect, a computer-implemented method is provided that includes obtaining input data from a user, inferring content that is of interest to the user based on features gathered from the user's attention in the input data, identifying virtual content data based on the content that is of interest to the user, determining modifications to be applied to the virtual content data based on relevancy, applying the modifications to the virtual content data to generate a final format for the virtual content data, and rendering virtual content in the extended reality environment displayed to the user. See Fig 2A).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton with the VR system of Zhang, as it improves the user’s ability to understand content(Col 2, Ln 1-4. Col 2, Ln 8-20).
Claim(s) 7 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Haro and Horton as applied to claim 5 and 8 above, and further in view of Evans (US 12119021 B1).
Regarding Claim 7:
The combination of Haro and Horton teaches the apparatus of claim 5, but does not teach further comprising a wearable device controller providing haptic feedback via wearable devices in response to changes in one or more participants' states of attention.
In the same field of Attention determination, Evans teaches further comprising a wearable device controller providing haptic feedback via wearable devices in response to changes in one or more participants' states of attention(Abstract, Ln 1-12, Audio data or image data is obtained from a sensor at the physical environment during the CGR experience and an emotional state or direction of attention of a second person other than the first user is detected based on the audio data or image data. Col 8, Ln 20-27, The CGR feedback unit 244 is configured to detect annoyance in people surrounding a CGR experience user or determine undue attention being given to the user from nearby people, and then to provide an audio, haptic, or visual warning thereof).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton, with the attention detection system of Evans, as it can help improve user convenience(Col 1, Ln 30-41).
Regarding Claim 13:
The combination of Haro and Horton teaches the method of claim 8, but does not teach further comprising providing haptic feedback via wearable devices in response to changes in one or more participants' states of attention.
In the same field of Attention determination, Evans teaches further comprising providing haptic feedback via wearable devices in response to changes in one or more participants' states of attention(Abstract, Ln 1-12, Audio data or image data is obtained from a sensor at the physical environment during the CGR experience and an emotional state or direction of attention of a second person other than the first user is detected based on the audio data or image data. Col 8, Ln 20-27, The CGR feedback unit 244 is configured to detect annoyance in people surrounding a CGR experience user or determine undue attention being given to the user from nearby people, and then to provide an audio, haptic, or visual warning thereof).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton, with the attention detection system of Evans, as it can help improve user convenience(Col 1, Ln 30-41).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Haro and Horton as applied to claim 8 and 14 above, and further in view of Duan et al. (US 20260178161 A1).
Regarding Claim 9:
The combination of Haro and Horton teaches the method of claim 8, but does not teach further comprising: detecting sustained interaction periods based on the attention probabilities; and identifying conversation groupings of participants based on the sustained interaction periods.
In the same field of Attention modeling, Duan teaches detecting sustained interaction periods based on the attention probabilities(Para [0020], Ln 1-17, metaverse platform may determine, based on the attention direction, an interest prediction. The interest prediction may be a predicted interest, of the avatar or the user thereof, in a group of other avatars (referred to herein as a “predicted group”), a conversation associated with a group of other avatars (referred to herein as a “predicted conversation”), and/or an avatar of a group of other avatars (referred to herein as a “predicted avatar”). In some implementations, if an attention direction of the avatar is directed to a particular group for a predefined period of time, or if the avatar is moving towards the group, the metaverse platform may determine the group to be a predicted group. Para [0063], Ln 1-12, The interest predictor 222 may generate an interest score for each group or conversation based on various factors such as the avatar's attention direction);
and identifying conversation groupings of participants based on the sustained interaction periods(Para [0020], Ln 1-17, metaverse platform may determine, based on the attention direction, an interest prediction. The interest prediction may be a predicted interest, of the avatar or the user thereof, in a group of other avatars (referred to herein as a “predicted group”), a conversation associated with a group of other avatars (referred to herein as a “predicted conversation”), and/or an avatar of a group of other avatars (referred to herein as a “predicted avatar”). In some implementations, if an attention direction of the avatar is directed to a particular group for a predefined period of time, or if the avatar is moving towards the group, the metaverse platform may determine the group to be a predicted group).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton, with the Attention system of Duan, as it improves the user experience(Para [0003], Ln 8-20).
Regarding Claim 15:
The combination of Haro and Horton teaches the system of claim 14, but does not teach further comprising memory storing historical attention pattern data, and wherein said adaptive audio gain controllers adapt the audio outputs based on the historical attention pattern data.
In the same field of Attention modeling, Duan teaches further comprising memory storing historical attention pattern data, and wherein said adaptive audio gain controllers adapt the audio outputs based on the historical attention pattern data(Para [0068], Ln 1-17, The metaverse server 202 may also provide spatial audio cues, subtly adjusting the perceived direction or volume of conversations based on their relevance to the user. For instance, a highly relevant conversation may sound slightly louder or more distinct, even if it's not the closest group to the user's avatar. Para [0020], Ln 1-17, metaverse platform may determine, based on the attention direction, an interest prediction. The interest prediction may be a predicted interest, of the avatar or the user thereof, in a group of other avatars (referred to herein as a “predicted group”), a conversation associated with a group of other avatars (referred to herein as a “predicted conversation”), and/or an avatar of a group of other avatars (referred to herein as a “predicted avatar”). In some implementations, if an attention direction of the avatar is directed to a particular group for a predefined period of time, or if the avatar is moving towards the group, the metaverse platform may determine the group to be a predicted group. Para [0076], Ln 1-20, memory).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haro and Horton, with the Attention system of Duan, as it improves the user experience(Para [0003], Ln 8-20).
Examiner Note:
Claims 17-20 are not rejected under prior art, because the issues discussed above, regarding the 112(b) and 112(d) rejections of Claims 17-20, make it unclear how to proceed with prior art rejections. Any applicable prior art rejections will be made once applicant has sufficiently corrected the issues related to 112(b) and 112(d).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Caplow-Munro et al. (US 20250191582 A1)
Device directed speech and attention determination
Du et al (US 12061738 B2)
Attention determination using wearable device and feedback to user based on attention determination.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER G MARLOW whose telephone number is (571)272-4536. The examiner can normally be reached Monday - Thursday 10:00 am - 8:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richmond Dorvil can be reached at (571)272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXANDER G MARLOW/ Assistant Examiner, Art Unit 2658
/RICHEMOND DORVIL/ Supervisory Patent Examiner, Art Unit 2658
