Detailed Action
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . See 35 U.S.C. § 100 (note).
Art Rejections
Obviousness
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–7, 9, 11–17, 19 are rejected under 35 U.S.C. § 103 as being unpatentable over the combination of US Patent 11,184,362 (patented 23 November 2021) (“Krol”); US Patent Application Publication 2022/0026986 (published 27 January 2022) (“Ballagas”); US Patent Application Publication 2021/0029479 (published 28 January 2021) (“Donley”) and US Patent Application Publication 2024/0233914 (effectively filed 29 April 2021) (“Wei”).
Claims 8 and 18 are rejected under 35 U.S.C. § 103 as being unpatentable over the combination of Krol, Ballagas, Donley, Wei and US Patent Application Publication 2024/0104864 (effectively filed 22 September 2022) (“Mulliken”).
Claims 10 and 20 are rejected under 35 U.S.C. § 103 as being unpatentable over the combination of Krol; Ballagas; Donley, Wei and US Patent Application Publication 2023/0045237 (published 09 February 2023) (“Wexler”).
Claim 1 is drawn to “a method.” The following table illustrates the correspondence between the claimed method and the Krol reference.
Claim 1
The Krol Reference
“1. A method comprising:
The Krol reference similarly describes a method to modify audio in artificial reality environments, including an augmented reality (AR) or a virtual reality (VR) environment. Krol at Abs., col. 1 l. 10 to col. 3 l. 5.
“receiving audio data comprising speech at a first client device executing an application in an artificial reality environment, wherein the first client device is associated with a speaking user;
Krol’s method includes operating a system 300 that includes a server 302, network 304 and multiple devices 306A and 306B operating together in a virtual video conference. Id. at col. 7 l. 8 to col. 9 l. 4, FIG.3. Each device 306A, 306B records audio and transmits the audio to server 302, which transmits the audio to the other devices in the virtual conference. Id. at col. 9 l. 5 to col. 13 l. 35, FIGs.4A-4D. The audio from a particular device includes the speech produced by the user of the particular device. Id.
“determining whether an object associated with the speaking user is within a field of view of a listening user of a second client device executing the application;
The Krol reference describes a virtual video conference, where each user associated with one of devices 306A, 306B is further associated with a virtual avatar 102 that is displayed visually by devices 306A, 306B. Id. at col. 4 ll. 20–44, FIG.1. The user’s move their avatars through a shared virtual space that has different zones, designated by virtual constructs, like walls. Id. at col. 15 ll. 36–51, FIG.8B.
The system determines if a first user’s avatar is visible to a second user based on the second user’s field of view. Id. at col. 5 ll. 20–33, col. 6 ll. 50–61, FIGs.1, 2. A first user’s avatar may become invisible to a second user because the relative position of the first user’s avatar and the second user’s field of view are not coincident. See id. Additionally, a first user’s avatar may be invisible because the first user’s avatar is behind a wall that obscures the second user’s field of view. See id. at col. 15 ll. 36–51, FIG.8B.
“modifying the audio data based at least in part on a set of rules, the set of rules comprising a rule for determining a listening effort of the listening user by applying a trained machine learning model to predict the listening effort of the listening user based on receiving a set of attributes associated with a plurality of actions of the listening user, and whether the object associated with the speaking user is within the field of view of the listening user, by changing a spatialization of the audio data corresponding to a percentage that is proportional to the listening effort of the listening user received from an output of the model predicting the listening effort of the listening user in a manner that causes the speech to appear to originate from a region closer to the listening user; and
Krol’s system similarly applies multiple rules to determines how sound should be modified and reproduced for each user. Server 302 tracks where each user’s virtual avatar is located in relation to each other in the virtual conference and in relation to the virtual conference environment. Id. at col. 7 ll. 14–28, col. 9 l. 46 to col. 10 l. 23, FIGs.3, 4B, 4C. Krol’s system then modifies the audio produced by each user to simulate distance and location effects on the speech audio produced by a user. Id. at col. 10 l. 65 to col. 11 l. 15. For example, if a first user’s avatar and a second user’s avatar are located in different rooms, designated by walls, audio from the first user will be modified based on a wall transmission factor. Id. at col. 15 l. 36 to col. 16 l. 3, FIG.8B. The wall transmission factor may attenuate the audio or completely prevent its transmission. Id. This is one example of modifying audio based on whether a speaking avatar is visible to a listening avatar.
Additional rules may apply, such as distance, position, whitelists and other rules. For instance, audio is modified to simulate left-right positions so that when a first speaking avatar is not visible because the avatar is to the left of the second user’s field of view, the first avatar’s audio is adjusted to come from the left. Id. at col. 20 l. 26 to col. 21 l. 34, FIG.11.
Krol, however, does not describe the claimed detection of a listening user’s listening effort using a trained machine learning model.
“communicating the modified audio data to the listening user of the second client device.”
Server 302 transmits modified audio to other devices through network 304. Id. at col. 9 ll. 46–61, col. 25 l. 56 to col. 26 l. 27, FIGs.4C, 13.
Table 1
The table above shows that the Krol reference describes a method that corresponds closely to the claimed method. The Krol reference does not anticipate the claimed method because Krol does not describe the claimed modification of audio based on a rule for determining a listening effort of the listening user by applying a trained machine learning model to predict the listening effort of the listening user based on receiving a set of attributes associated with a plurality of actions of the listening user. Krol also does not describe changing spatialization of the audio data (i.e., to cause speech to appear to originate from a region closer to the listening user) corresponding to a percentage that is proportional to the listening effort of the listening user received from an output of the model predicting the listening effort of the listening user.
The differences between the claimed invention and the Krol reference are such that the invention as a whole would have been obvious to one of ordinary skill in the art at the time this Application was effectively filed. The Krol reference describes a system and method for modifying audio in a virtual conference according to a set of rules to improve the realism and privacy of communicating in a virtual conference. Krol at col. 7 ll. 14–28, col. 9 l. 46 to col. 10 l. 23, col. 10 l. 65 to col. 11 l. 15. col. 15 l. 36 to col. 16 l. 3, FIGs.3, 4B, 4C, 8B. In modifying audio, Krol tracks whether a listening user can see an audio source object. Id. But Krol does not use a trained machine learning model to determine (e.g., as a percentage) the listening user’s listening effort and making the audio object sound closer based on the listening effort and whether the user is looking at the audio object.
The Ballagas reference describes an extended reality system, similar to the way Krol describes a virtual conference. Ballagas at ¶ 10. Ballagas teaches and suggests that users may experience a high cognitive load in a virtual environment and struggle to hear sounds in a spatial area of attention. Id. at ¶ 13. The spatial area of attention corresponds to an area within a user’s gaze—namely, the area and associated audio objects in the user’s view. Id. at ¶¶ 10–11, 24. Ballagas teaches tracking physiological signals (i.e., biometric signals) of a user in order to estimate the user’s cognitive load. Id. at 27–28, 51. If the cognitive load is too high, audio in the spatial area of attention is amplified. Id. One of ordinary skill would have understood that amplifying audio will cause it to appear closer to a user in the same way a sound emitted close to a person sound louder than the same sound emitted far from a person. See Krol at col. 14 ll. 16–45, FIG.6.
Further, the Donley reference, like Krol and Ballagas, is related to extended reality systems. Donley at Abs., ¶ 14. The Donley reference teaches and suggests tracking a user’s cognitive load in terms of listening effort and to adjust audio performance (e.g., sensor selection) in an extended reality system to maintain listening effort at a desired level. Id. at ¶¶ 60–62.
The Wei reference describes a machine learning model that is trained on participant tasks/actions labeled with subjective cognitive load (i.e., listening effort) values for physiological sensor signals. Wei at ¶¶ 23–27. The model is trained from (1) sensor signals collected while people perform a set of tasks and (2) labeled data corresponding to a user-perceived amount of cognitive load. Id. During an operational (i.e., inference) phase, the trained model receives sensor signals to output a continuous value (e.g., from 0 to 1), or a percentage, that is proportional to the amount of a user’s experienced cognitive load. Id. at ¶ 28. Thus, Wei teaches the idea of determining a user’s cognitive load based on sensor signals, or attributes, associated with participant tasks/actions. And the cognitive load will correspond to a continuous value, or percentage, that is proportional to cognitive load.
Taken collectively, Krol, Ballagas, Donley and Wei teach and suggest the whole of the claimed invention. Krol describes a base augmented-reality device capable of detecting a user’s position, gaze direction and the relative position of other sources and obstacles in a virtual environment. Krol also describes modifying audio based on these parameters to simulate spatial audio characteristics, such as the distance between a listening user and a speaking user. Ballagas teaches and suggests further modifying desired audio corresponding to a desired speaking user that the listening user is looking at to reflect the listener’s cognitive load. In particular, Ballagas teaches amplifying the desired speaking user audio, making the desired speaking user audio appear closer than it is. Donley teaches the link between estimating cognitive load and estimating listening effort as measures of the ease in which a user may hear a sound. And Wei teaches a specific mechanism for training and operating a machine learning model to predict a user’s cognitive load, or listening effort as suggested by Donley.
Accordingly, it would have been obvious to modify Krol to estimate a user’s listening effort by collecting sensor data, or listening user attributes, and inputting the data into Wei’s trained machine learning model that is trained to predict listening effort, and to use the listening effort in combination with a user’s looking direction to spatially modify audio to make it appear closer than it is. As suggested by Ballagas, Krol’s system would be modified to estimate the user’s cognitive load/listening effort. One of ordinary skill would have reasonably chosen to make the cognitive load/listening effort estimation based on the teachings of Donley and Wei, where a trained machine learning model would be trained by a set of training users that perform a set of tasks while also trying to listen to sounds in order to estimate listening effort. As taught by Wei, the users would provide a set of labeled data corresponding to their perceived amount of listening effort during a set of tasks. Simultaneously, the machine learning system will collect sensor data as the user’s perform the tasks. The machine learning system will then be trained to recognize the relationship between sensor data, or user attributes, and listening effort. During operation, Krol’s system will receive sensor signals from a listening user that will cause the machine learning system to predict an amount of the user’s listening effort by recognizing the correspondence between the sensor signals and the learned amount of listening effort experienced by the training users. The listening effort will be reported as a continuous value between 0 and 1, or a percentage between 0% and 100%, that is proportional to the amount of listening effort predicted from the received sensor signals, or listening user attributes. As suggested by Ballagas, this listening effort will be analyzed alongside a user’s look direction, to determine whether to amplify audio from a desired audio source. When the listening user is looking at a speaking user and experiencing a high-degree of listening effort, audio from the speaking user will be amplified to simulate the speaking user being located closer to the listening user than it is in reality.
One of ordinary skill would have reasonably expected that the resulting system would exhibit an improved user experience since the audibility of a desired speaking user’s audio will be enhanced when the user is experiencing a high-degree of listening effort. For the foregoing reasons, the combination of the Krol, the Ballagas, the Donley and the Wei references makes obvious all limitations of the claim.
Claim 2 depends on claim 1 and further requires the following:
“wherein modifying the audio data comprises: responsive to determining the object associated with the speaking user is not within the field of view of the listening user, diminishing the audio data.”
In Krol, audio from users that cannot be seen because they are in a different room (i.e., behind a wall) is attenuated by wall transmission factors. Krol at col. 15 l. 36 to col. 16 l. 3, FIG.8B. Audio from users that cannot be seen because they are to the left or right of another user’s field of view is attenuated in one of the left and right channels to simulate positional audio. Id. at col. 20 l. 26 to col. 21 l. 34, FIG.11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 3 depends on claim 1 and further requires the following:
“wherein modifying the audio data comprises: responsive to determining the object associated with the speaking user is within the field of view of the listening user, enhancing the audio data.”
When a user’s field of view is centered on a speaking avatar, the avatar’s audio is enhanced by reproducing it equally in two binaural channels. Id. at col. 20 l. 26 to col. 21 l. 34, FIG.11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 4 depends on claim 1 and further requires the following:
“wherein modifying the audio data comprises: identifying one or more noises included in the audio data; and executing, on the one or more noises included in the audio data, one or more selected from the group consisting of: noise blocking, noise cancelling, noise masking, and noise filtering, noise amplification
Claim 5 depends on claim 4 and further requires the following:
“wherein the set of rules comprises:
“enhancing the speech included in the audio data if one or more of:
“the speaking user is within the field of view of the listening user of the second client device,
“the speech is associated with a volume that is at least a threshold volume,
“one or more words associated with safety are included in the speech,
“a whitelist associated with the speaking user identifies the listening user, an additional object associated with the listening user is within a boundary around the object associated with the speaking user, and
“a gaze point of the speaking user matches a location of the additional object associated with the listening user.”1
Claims 4 and 5 are treated together. The claimed set of rules includes are presented as alternative rules for enhancing speech. In Krol, when a user’s field of view is centered on a speaking avatar, the avatar’s audio is enhanced by reproducing it equally in two binaural channels. Krol at col. 20 l. 26 to col. 21 l. 34, FIG.11. Krol also describes amplifying audio that meets a volume threshold, a listening avatar is whitelisted (i.e., allowed) to participate in a private conversation with the speaking avatar, the speaking and listening avatars are in a boundary space, the speaking avatar is looking at the listening avatar so that the speech audio is binaurally enhanced relative to the listening avatar. Id. at col. 10 l. 65 to col. 11 l. 27, col. 15 l. 36 to col. 16 l. 3, col. 17 ll. 28–41, col. 20 l. 65 to col. 21 l. 21, col. 25 ll. 11–30, FIGs.8B, 11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claims.
Claim 6 depends on claim 5 and further requires the following:
“wherein the gaze point of the speaking user is determined by the first client device.”
A first user determines the gaze point (i.e., field of view) of his avatar through inputs 106/1312 on his device. Krol at col. 24 ll. 8–15, FIG.1. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 7 depends on claim 4 and further requires the following:
“wherein the set of rules comprises: diminishing the speech included in the audio data if one or more of: the speaking user is not within the field of view of the listening user of the second client device, the speech is associated with a volume that is less than a threshold volume, a whitelist associated with the speaking user does not identify the listening user, an additional object associated with the listening user is outside a boundary around the object associated with the speaking user, and a gaze point of the speaking user does not match a location of the additional object associated with the listening user.”
The claimed rules are presented as alternative rules for diminishing speech. In Krol, when a user’s field of view does not include a speaking avatar, the avatar’s audio is diminished by reproducing it only in one ear. Krol at col. 20 l. 26 to col. 21 l. 34, FIG.11. Krol also describes attenuating audio that is below a volume threshold, a listening avatar is not whitelisted (i.e., not included in an allowed list) to participate in a private conversation with the speaking avatar, the speaking and listening avatars are not in a boundary space, the speaking avatar is not looking at the listening avatar so that the speech audio is produced in one ear of the listening avatar. Id. at col. 10 l. 65 to col. 11 l. 27, col. 15 l. 36 to col. 16 l. 3, col. 17 ll. 28–41, col. 20 l. 65 to col. 21 l. 21, col. 25 ll. 11–30, FIGs.8B, 11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 8 depends on claim 1 and further requires the following:
“wherein modifying the audio data comprises: identifying one or more noises included in the audio data; sending a prompt to the second client device to select one or more options from a set of options for modifying each of the one or more noises included in the audio data; receiving, from the second client device, the one or more options for modifying each of the one or more noises included in the audio data; and modifying the audio data based at least in part on the one or more options received from the second client device.”
The Krol reference does not describe any mechanism for handling noise in audio data. The Mulliken reference, like Krol, is drawn to extended reality systems. Mulliken at Abs., ¶¶ 3, 4, 155–158, 178, FIG.10. Mulliken teaches and suggests adding the ability to remove distractions by identifying sounds in the environment and modifying them. Id. For example, the system will identify noises and alert the user through a GUI. Id. The user inputs a noise removing preference through the GUI. Id. The system then removes the identified noise based on the user’s input preference. Id. These teachings taken in conjunction with the Krol reference, would have reasonably suggested modifying Krol’s system and method of operation to similarly include a noise identification and modification feature. One of ordinary skill would have reasonably recognized that the noise identification and modification feature would allow for distracting sounds to be removed. See id. For the foregoing reasons, the combination of the Krol, the Ballagas and the Mulliken references makes obvious all limitations of the claims.
Claim 9 depends on claim 1 and further requires the following:
“wherein modifying the audio data is further based at least in part on one or more selected from the group consisting of: a set of preferences associated with the listening user, a setting associated with the application, and a predicted listening effort of the listening user.”
Krol describes modifying audio based on a application settings, such as wall positions. Krol at col. 15 l. 36 to col. 16 l. 3, FIG.8B. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 10 depends on claim 9 and further requires the following:
“wherein predicting the listening effort of the listening user further
“
“receiving the set of attributes associated with the plurality of actions of the listening user, the set of attributes selected from the group consisting of a gaze point of the listening user, a position of a head or at least one hand of the listening user, an orientation of the head or the at least one hand of the listening user, a direction of motion of the head or the at least one hand of the listening user, or a speed at which the head or the at least one hand of the listening user is moving,
“receiving, for each action of the plurality of actions, a label indicating a listening effort of the listening user, the plurality of actions selected from the group consisting of moving towards an object associated with audio data, leaning towards the object, turning the head of the listening user, wherein at least one ear of the head of the listening user is directed towards the object, cupping the at least one hand of the listening user around an outer portion of the at least one ear of the listening user, and
“training the machine learning model based at least in part on the set of attributes and the label for each action of the plurality of actions; and
“applying the machine learning model to a set of attributes associated with an action of the listening user to predict the listening effort of the listening user.”
The obviousness rejection of claim 1, incorporated herein, shows the obviousness of modifying the Krol reference’s method and system to use a trained machine learning model to predict a listening user’s listening effort based on user attributes derived from a set of sensors. During a training phase, the Wei reference teaches training the model with attribute data from a set of sensors and a set of labels indicating cognitive load/listening effort for a set of user actions. Further, Wei teaches that during an inference phase, the trained model is fed sensor data to predict a listening user’s listening effort. However, Wei does not describe sensing the claimed attributes and labeling the claimed actions.
The Wexler reference teaches the use of a learning model that is trained to analyze audio and images corresponding to actions taken by a user when having a hard-time hearing. Wexler at ¶ 500. For example, the model is trained to recognize when a user is leaning towards a speaker, putting his hand up to his ear or asking the speaker to repeat themselves. Id. Read in light of Wei, Wexler reasonably suggests training a model with sensor data, such as images and audio. The images and audio would represent actions typical of high-listening effort, such as when a listening user leans towards a speaking user, puts his hand up to his ear or asking the speaking user to repeat himself. The sensor data would be labeled with a listening effort score, just as Wei’s other sensor data is labeled. Accordingly, Wei’s model would predictably be able to recognize audio/visual sensor data that corresponds to high listening effort scenarios. For the foregoing reasons, the combination of the Krol, the Ballagas, the Donley, the Wei and the Wexler references makes obvious all limitations of the claim.
Claim 11 is drawn to “a non-transitory computer-readable storage medium.” The following table illustrates the correspondence between the claimed medium and the Krol reference.
Claim 11
The Krol Reference
“11. A non-transitory computer-readable storage medium comprising stored instructions, the instructions when executed by a processor of a device, causing the device to:
The Krol reference similarly describes a non-transitory computer-readable storage medium storing instructions that when executed by a processor causes the processor to perform a method that modifies audio in artificial reality environments, including an augmented reality (AR) or a virtual reality (VR) environment. Krol at Abs., col. 1 l. 10 to col. 3 l. 5, claim 8.
“receive audio data comprising speech at a first client device executing an application in an artificial reality environment, wherein the first client device is associated with a speaking user;
Krol’s method includes operating a system 300 that includes a server 302, network 304 and multiple devices 306A and 306B operating together in a virtual video conference. Id. at col. 7 l. 8 to col. 9 l. 4, FIG.3. Each device 306A, 306B records audio and transmits the audio to server 302, which transmits the audio to the other devices in the virtual conference. Id. at col. 9 l. 5 to col. 13 l. 35, FIGs.4A-4D. The audio from a particular device includes the speech produced by the user of the particular device. Id.
“determine whether an object associated with the speaking user is within a field of view of a listening user of a second client device executing the application;
The Krol reference describes a virtual video conference, where each user associated with one of devices 306A, 306B is further associated with a virtual avatar 102 that is displayed visually by devices 306A, 306B. Id. at col. 4 ll. 20–44, FIG.1. The user’s move their avatars through a shared virtual space that has different zones, designated by virtual constructs, like walls. Id. at col. 15 ll. 36–51, FIG.8B.
The system determines if a first user’s avatar is visible to a second user based on the second user’s field of view. Id. at col. 5 ll. 20–33, col. 6 ll. 50–61, FIGs.1, 2. A first user’s avatar may become invisible to a second user because the relative position of the first user’s avatar and the second user’s field of view are not coincident. See id. Additionally, a first user’s avatar may be invisible because the first user’s avatar is behind a wall that obscures the second user’s field of view. See id. at col. 15 ll. 36–51, FIG.8B.
“modifying the audio data based at least in part on a set of rules, the set of rules comprising a rule for determining a listening effort of the listening user by applying a trained machine learning model to predict the listening effort of the listening user based on receiving a set of attributes associated with a plurality of actions of the listening user, and whether the object associated with the speaking user is within the field of view of the listening user, by changing a spatialization of the audio data corresponding to a percentage that is proportional to the listening effort of the listening user received from an output of the model predicting the listening effort of the listening user in a manner that causes the speech to appear to originate from a region closer to the listening user; and
Krol’s system similarly applies multiple rules to determines how sound should be modified and reproduced for each user. Server 302 tracks where each user’s virtual avatar is located in relation to each other in the virtual conference and in relation to the virtual conference environment. Id. at col. 7 ll. 14–28, col. 9 l. 46 to col. 10 l. 23, FIGs.3, 4B, 4C. Krol’s system then modifies the audio produced by each user to simulate distance and location effects on the speech audio produced by a user. Id. at col. 10 l. 65 to col. 11 l. 15. For example, if a first user’s avatar and a second user’s avatar are located in different rooms, designated by walls, audio from the first user will be modified based on a wall transmission factor. Id. at col. 15 l. 36 to col. 16 l. 3, FIG.8B. The wall transmission factor may attenuate the audio or completely prevent its transmission. Id. This is one example of modifying audio based on whether a speaking avatar is visible to a listening avatar.
Additional rules may apply, such as distance, position, whitelists and other rules. For instance, audio is modified to simulate left-right positions so that when a first speaking avatar is not visible because the avatar is to the left of the second user’s field of view, the first avatar’s audio is adjusted to come from the left. Id. at col. 20 l. 26 to col. 21 l. 34, FIG.11.
Krol, however, does not describe the claimed detection of a listening user’s listening effort using a trained machine learning model.
“communicate the modified audio data to the listening user of the second client device.”
Server 302 transmits modified audio to other devices through network 304. Id. at col. 9 ll. 46–61, col. 25 l. 56 to col. 26 l. 27, FIGs.4C, 13.
Table 2
The table above shows that the Krol reference describes a a non-transitory computer-readable storage medium that corresponds closely to the claimed non-transitory computer-readable storage medium. The Krol reference does not anticipate the claimed non-transitory computer-readable storage medium because Krol does not describe the claimed modification of audio based on a rule for determining a listening effort of the listening user by applying a trained machine learning model to predict the listening effort of the listening user based on receiving a set of attributes associated with a plurality of actions of the listening user. Krol also does not describe changing spatialization of the audio data (i.e., to cause speech to appear to originate from a region closer to the listening user) corresponding to a percentage that is proportional to the listening effort of the listening user received from an output of the model predicting the listening effort of the listening user.
The differences between the claimed invention and the Krol reference are such that the invention as a whole would have been obvious to one of ordinary skill in the art at the time this Application was effectively filed. The Krol reference describes a system and method for modifying audio in a virtual conference according to a set of rules to improve the realism and privacy of communicating in a virtual conference. Krol at col. 7 ll. 14–28, col. 9 l. 46 to col. 10 l. 23, col. 10 l. 65 to col. 11 l. 15. col. 15 l. 36 to col. 16 l. 3, FIGs.3, 4B, 4C, 8B. In modifying audio, Krol tracks whether a listening user can see an audio source object. Id. But Krol does not use a trained machine learning model to determine (e.g., as a percentage) the listening user’s listening effort and making the audio object sound closer based on the listening effort and whether the user is looking at the audio object.
The Ballagas reference describes an extended reality system, similar to the way Krol describes a virtual conference. Ballagas at ¶ 10. Ballagas teaches and suggests that users may experience a high cognitive load in a virtual environment and struggle to hear sounds in a spatial area of attention. Id. at ¶ 13. The spatial area of attention corresponds to an area within a user’s gaze—namely, the area and associated audio objects in the user’s view. Id. at ¶¶ 10–11, 24. Ballagas teaches tracking physiological signals (i.e., biometric signals) of a user in order to estimate the user’s cognitive load. Id. at 27–28, 51. If the cognitive load is too high, audio in the spatial area of attention is amplified. Id. One of ordinary skill would have understood that amplifying audio will cause it to appear closer to a user in the same way a sound emitted close to a person sound louder than the same sound emitted far from a person. See Krol at col. 14 ll. 16–45, FIG.6.
Further, the Donley reference, like Krol and Ballagas, is related to extended reality systems. Donley at Abs., ¶ 14. The Donley reference teaches and suggests tracking a user’s cognitive load in terms of listening effort and to adjust audio performance (e.g., sensor selection) in an extended reality system to maintain listening effort at a desired level. Id. at ¶¶ 60–62.
The Wei reference describes a machine learning model that is trained on participant tasks/actions labeled with subjective cognitive load (i.e., listening effort) values for physiological sensor signals. Wei at ¶¶ 23–27. The model is trained from (1) sensor signals collected while people perform a set of tasks and (2) labeled data corresponding to a user-perceived amount of cognitive load. Id. During an operational (i.e., inference) phase, the trained model receives sensor signals to output a continuous value (e.g., from 0 to 1), or a percentage, that is proportional to the amount of a user’s experienced cognitive load. Id. at ¶ 28. Thus, Wei teaches the idea of determining a user’s cognitive load based on sensor signals, or attributes, associated with participant tasks/actions. And the cognitive load will correspond to a continuous value, or percentage, that is proportional to cognitive load.
Taken collectively, Krol, Ballagas, Donley and Wei teach and suggest the whole of the claimed invention. Krol describes a base augmented-reality device capable of detecting a user’s position, gaze direction and the relative position of other sources and obstacles in a virtual environment. Krol also describes modifying audio based on these parameters to simulate spatial audio characteristics, such as the distance between a listening user and a speaking user. Ballagas teaches and suggests further modifying desired audio corresponding to a desired speaking user that the listening user is looking at to reflect the listener’s cognitive load. In particular, Ballagas teaches amplifying the desired speaking user audio, making the desired speaking user audio appear closer than it is. Donley teaches the link between estimating cognitive load and estimating listening effort as measures of the ease in which a user may hear a sound. And Wei teaches a specific mechanism for training and operating a machine learning model to predict a user’s cognitive load, or listening effort as suggested by Donley.
Accordingly, it would have been obvious to modify Krol to estimate a user’s listening effort by collecting sensor data, or listening user attributes, and inputting the data into Wei’s trained machine learning model that is trained to predict listening effort, and to use the listening effort in combination with a user’s looking direction to spatially modify audio to make it appear closer than it is. As suggested by Ballagas, Krol’s system would be modified to estimate the user’s cognitive load/listening effort. One of ordinary skill would have reasonably chosen to make the cognitive load/listening effort estimation based on the teachings of Donley and Wei, where a trained machine learning model would be trained by a set of training users that perform a set of tasks while also trying to listen to sounds in order to estimate listening effort. As taught by Wei, the users would provide a set of labeled data corresponding to their perceived amount of listening effort during a set of tasks. Simultaneously, the machine learning system will collect sensor data as the user’s perform the tasks. The machine learning system will then be trained to recognize the relationship between sensor data, or user attributes, and listening effort. During operation, Krol’s system will receive sensor signals from a listening user that will cause the machine learning system to predict an amount of the user’s listening effort by recognizing the correspondence between the sensor signals and the learned amount of listening effort experienced by the training users. The listening effort will be reported as a continuous value between 0 and 1, or a percentage between 0% and 100%, that is proportional to the amount of listening effort predicted from the received sensor signals, or listening user attributes. As suggested by Ballagas, this listening effort will be analyzed alongside a user’s look direction, to determine whether to amplify audio from a desired audio source. When the listening user is looking at a speaking user and experiencing a high-degree of listening effort, audio from the speaking user will be amplified to simulate the speaking user being located closer to the listening user than it is in reality.
One of ordinary skill would have reasonably expected that the resulting system would exhibit an improved user experience since the audibility of a desired speaking user’s audio will be enhanced when the user is experiencing a high-degree of listening effort. For the foregoing reasons, the combination of the Krol, the Ballagas, the Donley and the Wei references makes obvious all limitations of the claim.
Claim 12 depends on claim 11 and further requires the following:
“wherein the stored instructions to modify the audio data further comprise stored instructions that, when executed, cause the device to: responsive to determining the object associated with the speaking user is not within the field of view of the listening user, diminish the audio data.”
In Krol, audio from users that cannot be seen because they are in a different room (i.e., behind a wall) is attenuated by wall transmission factors. Krol at col. 15 l. 36 to col. 16 l. 3, FIG.8B. Audio from users that cannot be seen because they are to the left or right of another user’s field of view is attenuated in one of the left and right channels to simulate positional audio. Id. at col. 20 l. 26 to col. 21 l. 34, FIG.11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 13 depends on claim 11 and further requires the following:
“wherein the stored instructions to modify the audio data further comprise stored instructions that, when executed, cause the device to: responsive to determining the object associated with the speaking user is within the field of view of the listening user, enhance the audio data.”
When a user’s field of view is centered on a speaking avatar, the avatar’s audio is enhanced by reproducing it equally in two binaural channels. Id. at col. 20 l. 26 to col. 21 l. 34, FIG.11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 14 depends on claim 11 and further requires the following:
“wherein the stored instructions to modify the audio data further comprise stored instructions that, when executed, cause the device to: identify one or more noises included in the audio data; and execute, on the one or more noises included in the audio data, one or more selected from the group consisting of: noise blocking, noise cancelling, noise masking, noise filtering, and noise amplification
Claim 15 depends on claim 14 and further requires the following:
“wherein the set of rules comprises: enhancing the speech included in the audio data if one or more of: the speaking user is within the field of view of the listening user of the second client device, the speech is associated with a volume that is at least a threshold volume, one or more words associated with safety are included in the speech, a whitelist associated with the speaking user identifies the listening user, an additional object associated with the listening user is within a boundary around the object associated with the speaking user, and a gaze point of the speaking user matches a location of the additional object associated with the listening user.”
Claims 14 and 15 are treated together. The claimed set of rules includes are presented as alternative rules for enhancing speech. In Krol, when a user’s field of view is centered on a speaking avatar, the avatar’s audio is enhanced by reproducing it equally in two binaural channels. Krol at col. 20 l. 26 to col. 21 l. 34, FIG.11. Krol also describes amplifying audio that meets a volume threshold, a listening avatar is whitelisted (i.e., allowed) to participate in a private conversation with the speaking avatar, the speaking and listening avatars are in a boundary space, the speaking avatar is looking at the listening avatar so that the speech audio is binaurally enhanced relative to the listening avatar. Id. at col. 10 l. 65 to col. 11 l. 27, col. 15 l. 36 to col. 16 l. 3, col. 17 ll. 28–41, col. 20 l. 65 to col. 21 l. 21, col. 25 ll. 11–30, FIGs.8B, 11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claims.
Claim 16 depends on claim 15 and further requires the following:
“wherein the gaze point of the speaking user is determined by the first client device.”
A first user determines the gaze point (i.e., field of view) of his avatar through inputs 106/1312 on his device. Krol at col. 24 ll. 8–15, FIG.1. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 17 depends on claim 14 and further requires the following:
“wherein the set of rules comprises: diminishing the speech included in the audio data if one or more of: the speaking user is not within the field of view of the listening user of the second client device, the speech is associated with a volume that is less than a threshold volume, a whitelist associated with the speaking user does not identify the listening user, an additional object associated with the listening user is outside a boundary around the object associated with the speaking user, and a gaze point of the speaking user does not match a location of the additional object associated with the listening user.”
The claimed rules are presented as alternative rules for diminishing speech. In Krol, when a user’s field of view does not include a speaking avatar, the avatar’s audio is diminished by reproducing it only in one ear. Krol at col. 20 l. 26 to col. 21 l. 34, FIG.11. Krol also describes attenuating audio that is below a volume threshold, a listening avatar is not whitelisted (i.e., not included in an allowed list) to participate in a private conversation with the speaking avatar, the speaking and listening avatars are not in a boundary space, the speaking avatar is not looking at the listening avatar so that the speech audio is produced in one ear of the listening avatar. Id. at col. 10 l. 65 to col. 11 l. 27, col. 15 l. 36 to col. 16 l. 3, col. 17 ll. 28–41, col. 20 l. 65 to col. 21 l. 21, col. 25 ll. 11–30, FIGs.8B, 11. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 18 depends on claim 11 and further requires the following:
“wherein the stored instructions to modify the audio data further comprise stored instructions that, when executed, cause the device to: identify one or more noises included in the audio data; send a prompt to the second client device to select one or more options from a set of options for modifying each of the one or more noises included in the audio data; receive, from the second client device, the one or more options for modifying each of the one or more noises included in the audio data; and modify the audio data based at least in part on the one or more options received from the second client device.”
The Krol reference does not describe any mechanism for handling noise in audio data. The Mulliken reference, like Krol, is drawn to extended reality systems. Mulliken at Abs., ¶¶ 3, 4, 155–158, 178, FIG.10. Mulliken teaches and suggests adding the ability to remove distractions by identifying sounds in the environment and modifying them. Id. For example, the system will identify noises and alert the user through a GUI. Id. The user inputs a noise removing preference through the GUI. Id. The system then removes the identified noise based on the user’s input preference. Id. These teachings taken in conjunction with the Krol reference, would have reasonably suggested modifying Krol’s system and method of operation to similarly include a noise identification and modification feature. One of ordinary skill would have reasonably recognized that the noise identification and modification feature would allow for distracting sounds to be removed. See id. For the foregoing reasons, the combination of the Krol, the Ballagas and the Mulliken references makes obvious all limitations of the claims.
Claim 19 depends on claim 11 and further requires the following:
“wherein modify the audio data is further based at least in part on one or more selected from the group consisting of: a set of preferences associated with the listening user, a setting associated with the application, and a predicted listening effort of the listening user.”
Krol describes modifying audio based on a application settings, such as wall positions. Krol at col. 15 l. 36 to col. 16 l. 3, FIG.8B. For the foregoing reasons, the combination of the Krol and the Ballagas references makes obvious all limitations of the claim.
Claim 20 depends on claim 19 and further requires the following:
“wherein the stored instructions to predict the listening effort of the listening user
“
“receiving a set of attributes associated with the plurality of actions of the listening user, the set of attributes selected from the group consisting of a gaze point of the listening user, a position of a head or at least one hand of the listening user, an orientation of the head or the at least one hand of the listening user, a direction of motion of the head or the at least one hand of the listening user, or a speed at which the head or the at least one hand of the listening user is moving,
“receiving, for each action of the plurality of actions, a label indicating a listening effort of the listening user, the plurality of actions selected from the group consisting of moving towards an object associated with audio data, leaning towards the object, turning the head of the listening user, wherein at least one ear of the head of the listening user is directed towards the object, cupping the at least one hand of the listening user around an outer portion of the at least one ear of the listening user, and
“training the machine learning model based at least in part on the set of attributes and the label for each action of the plurality of actions; and
“apply the machine learning model to a set of attributes associated with an action of the listening user to predict the listening effort of the listening user.”
The obviousness rejection of claim 1, incorporated herein, shows the obviousness of modifying the Krol reference’s method and system to use a trained machine learning model to predict a listening user’s listening effort based on user attributes derived from a set of sensors. During a training phase, the Wei reference teaches training the model with attribute data from a set of sensors and a set of labels indicating cognitive load/listening effort for a set of user actions. Further, Wei teaches that during an inference phase, the trained model is fed sensor data to predict a listening user’s listening effort. However, Wei does not describe sensing the claimed attributes and labeling the claimed actions.
The Wexler reference teaches the use of a learning model that is trained to analyze audio and images corresponding to actions taken by a user when having a hard-time hearing. Wexler at ¶ 500. For example, the model is trained to recognize when a user is leaning towards a speaker, putting his hand up to his ear or asking the speaker to repeat themselves. Id. Read in light of Wei, Wexler reasonably suggests training a model with sensor data, such as images and audio. The images and audio would represent actions typical of high-listening effort, such as when a listening user leans towards a speaking user, puts his hand up to his ear or asking the speaking user to repeat himself. The sensor data would be labeled with a listening effort score, just as Wei’s other sensor data is labeled. Accordingly, Wei’s model would predictably be able to recognize audio/visual sensor data that corresponds to high listening effort scenarios. For the foregoing reasons, the combination of the Krol, the Ballagas, the Donley, the Wei and the Wexler references makes obvious all limitations of the claim.
Summary
Claims 1–20 are rejected under at least one of 35 U.S.C. §§ 102 and 103 as being unpatentable over the cited prior art. In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Additional Citations
The following table lists additional references that are relevant to the invention described and claimed in this Application.
Citation
Relevance
US 2022/0360891
Audio zoom on target based on gaze.
Table 3
Response to Applicant’s Arguments
Applicant’s Reply (06 March 2026) has substantively amended all the claims. This Office action has been updated accordingly.
Applicant’s Reply at 12–14 further includes comments pertaining to the amended claims. The Examiner has taken those comments under consideration; however, the comments are moot in light of the new grounds of rejection presented in this Office action.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 C.F.R. § 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 C.F.R. § 1.17(a)) pursuant to 37 C.F.R. § 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WALTER F BRINEY III whose telephone number is (571)272-7513. The examiner can normally be reached M-F 8 am-4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carolyn Edwards can be reached on 571-270-7136. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Walter F Briney III/
Walter F Briney IIIPrimary ExaminerArt Unit 2692
4/23/2026
1 This claim (and claim 15) recites a list of the form “one or more of A, B, C, D and E.” While the list uses the conjunctive “and”, the claim list is construed in the disjunctive sense since each element of the list is not suitable for duplication; rather each element is a condition, not a duplicable item.
Prosecution Insights