DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the response to this office action, the Examiner respectfully requests that support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line numbers in the specification and/or drawing figure(s). This will assist the Examiner in prosecuting this application.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention..
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 18-23 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lovitt et al. (US 20200134026 A1, hereinafter Lovitt).
Claim 1: Lovitt teaches a method (title and abstract, ln 1-14, the method steps in fig. 5) for spatial audio rendering of a conversation channel (spatializing playback of the spoken words from a speaking user, para 8, 58, e.g., processing by utilizing user’s HRTF, para 64) in a first audio system having a headworn device (one of the AR headsets 630A or 630B in fig. 6), the method comprising the following operations performed by a digital processor in the first audio system (at least one processor of a computing device, para 16 and in an AR or a VR system 200, 300 in figs. 2, 3, respectively):
receiving an opt-in indication that a wearer of the head worn device join a conversation channel (DOA analysis on microphone signals to a sound source, para 59-61, and gaze detector to determine sound source the listener is looking at, para 62, or eye movement tracker 615 to track listener’s eye movement to the speaking user, and actively listen to the spoken words, para 83, or using beamforming at a focus of the target sound source, para 46 and in a conversation communication environment, para 4), wherein the conversation channel comprises a first target audio signal that contains an isolated voice of a first talker (received signal representing spoken words from one of the other of 630A and 630B, para 75, by using beamforming, or DOA, or gazed/eye focused sound source discussed above, i.e., isolated voice), the first target audio signal being
i) produced by processing an output of one or more microphones in a second audio system, and then received by the first audio system over-the-air from the second audio system (e.g., AR headsets 630B, having a microphone array with acoustic sensors 220A-220J, para 39 and the captured voice signals processed and provided an audio input 605 representing voice modeled/vocal characteristics of the spoken words of the spoken person 606 in figs. 7-8, para 78), or
ii) produced by the first audio system processing an output of a microphone array in the first audio system (or using microphone array at the listener’s AR or VR to detect sounds in a wide range of directions surrounding the user wearing the AR system 200, para 42; e.g., the microphones used for pickup sounds in the user’s environment and playback the sounds by using HRTF, para 61 or the listener looking at the speaker user to actively listen to them, para 81-83. Note: the prior art does not have to meet both claimed limitation “i) …” and “ii) …” because claim recited “i) … or ii) … ” at this point according to MPEP 2117); and
in response to receiving the opt-in indication, spatial audio rendering the first target audio signal into a left speaker driver signal and a right speaker driver signal that are to drive a left speaker and a right speaker, respectively, of the first audio system (playback of spoken sounds captured by the microphones of the listener’s AR system, element 630A, by using acoustic transfer function or HRTF, para 63-64, or spatializing the received spoken signal based on GPS location information or within a given room, dance hall, etc., determined by the direction identifying module 610, and the spatialization is performed by the spatialization module 611 so that listener perceives sound from behind, left, right, or front, etc., by using both speakers of the VR/AR, para 81).
Claim 19 has been analyzed and rejected according to claim 1 above and Lovitt further teaches an audio system (a VR system 200, 300 in figs. 2, 3) comprising a processor (microprocessors, FPGAs, CPUs, etc., in a controller 225, para 47, and in a controller 250, para 40, 93 and controller 250 in fig. 2 or personalization engine 600 is part of an AR headset or in communication with the AR head set, para 83 and speech differentiation module 1103 is part of personalization engine 600, para 85) and memory having stored therein instructions (computer-readable medium, para 96, including magnetic-storage media, optical-stroage media, etc., and having computer-readable instructions, para 96) that program the processor to perform the following operations the method steps of claim 1 (executed by the at leastone processor of the computing device, para 89).
Claim 2: Lovitt further teaches, according to claim 1 above, wherein the headworn device is an augmented reality headset or an AR headset (AR 630A and 630B in fig. 6), and the first talker is in a real environment of the wearer (AR 630A and 630B in the computing environment in fig. 6 and in figs. 7-9).
Claim 3: Lovitt further teaches, according to claim 2 above, wherein the left speaker and the right speaker are extra-aural speakers in a housing of the AR headset (output audio transducers 306A/306B for the left and right ears, para 49).
Claim 4: Lovitt further teaches, according to claim 2 above, wherein the first audio system further comprises, in addition to the AR headset, a left headphone housing and right headphone housing in which the left speaker and the right speaker, respectively, are integrated (output audio transducers 306A and 306B for the left and the right ears of the wearer with the housing in fig. 3).
Claim 18: Lovitt further teaches, further teaches, according to claim 1 above, the method further comprising:
receiving an opt-out indication that the wearer of the headworn device leave the conversation channel (Lovitt, indicated from speech differentiation module 1103 to detect stopping talking of the first talker for a specified amount of time, para 86); and in response to the opt-out indication, cease rendering the first target audio signal in the first audio system (the speech of other speakers is played back, and no rendering for the first talker is inherency, para 86).
Claim 20 has been analyzed and rejected according to claims 19, 2 above.
Claim 21: Lovitt further teaches, according to claim 20 above, wherein the processor comprises a second microprocessor in a companion device to the AR headset (neckband containing a controller 225, 235 as coupled with the eyewear device 202 in fig. 2).
Claim 22: Lovitt further teaches, according to claim 19 above, wherein the processor comprises a first microprocessor in a headphone (the controller 250 on the eyewear device 202 and containing the microprocessor, para 93).
Claim 23: Lovitt further teaches, according to claim 22 above, wherein the processor comprises a second microprocessor in a companion device to the headphone (Smartphone implementing the application for performing language translation, para 3 and smartphone as neckband 205, para 43).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5-7, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lovitt (above) and in view of reference Spivak et al. (US 11563855 B1, hereinafter Spivak).
Claim 5: Lovitt teaches, according to claim 2 above, the method further comprising:
detecting that the wearer is gazing at a face of the first talker for selecting the first talker (gaze detector, determining where the listener is look at), except explicitly teaching wherein in the response to gazing at the face of the first talker, enable the wearer to manually adjust a playback level of the first target audio signal via a virtual variable level control element shown on a display panel or via a physical variable level control element in the first audio system.
Spivak teaches an analogous field of endeavor by disclosing a method for spatial audio rendering (title and abstract, ln 1-13 and method steps in figs. 4A/4B) and wherein a wearer is disclosed in a first audio system (in virtual conference environment, a head-mounted display HMD is carried by a participant, col 4, ln 33-40, as the first audio system) and tracked to gaze at a face of a first talker (gazing at a participant image is tracked so that the sound parameters related to the gazed participant, e.g., volume level, may be updated, col 6, ln 46-59), and in response to gazing at the face of the first talker, enable the wearer to manually adjust a playback level of the first target audio signal via a virtual variable level control element shown on a display panel or via a physical variable level control element in the first audio system (via a displayed menu 504 after selecting the interesting participant image, e.g., 512 in fig. 5, and increasing/decreasing volume of the selected participant through button 506/508, respectively is performed for the selected participant, col 12, ln 19-32) for the benefits of improving personal experiences in multi-media environment (clearly discerning among speakers and flexibly controlling acoustic parameters individually as user’s desire, col 1, ln 18-22, e.g., through menu 504 after selecting the participant, e.g., by gazing at the participant image in the display 500 in fig. 5, col 12, ln19-32 and by localizing sound sources, col 5, ln 58-62).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have applied gazing at the face of the first talker, and enabling the wearer to manually adjust the playback level of the first target audio signal via the virtual variable level control element shown on the display panel or via the physical variable level control element in the first audio system, as taught by Spivak, to the detecting that the wearer is gazing at the face of the first talker for selecting the first talker in the method, as taught by Lovitt, for the benefits discussed above.
Claim 6: the combination of Lovitt and Spivak further teaches, according to claim 2 above, the method further comprising: detecting that the wearer is gazing at a face of the first talker (Lovitt, the discussion in claim 1, 5 above, and Spivak, the discussion in claim 5 above) and in response making an automatic change, without input from the wearer, to raise or lower a playback volume of the first target audio signal (Lovitt, selecting the interesting person by tracking user’s gaze, discussed in claim 1 above, and Spivak, initial selection of the participant by gazing and with automatic volume adjustment, col 6, 53-59); and enabling the wearer to manually override the automatic change via a virtual control element shown on a display panel or via a physical control element in the first audio system (Spivak, combined with displayed volume adjustment menu after the participant has been selected, the discussion in claim 5 above).
Claim 7: the combination of Lovitt and Spivak further teaches, according to claim 2 above, the method further comprising: determining a direct sound path parameter from the first talker to the AR headset (Lovitt, HRTF, para 60, and updated based on the location of the user, para 64, and HRTF, and Spivak, HRTFs as direct sound path parameter, col 5, ln 60-67, col 6, ln 1-5); and using the direct sound path parameter to adjust a playback volume of the first target audio signal (Lovitt, audio signals altered upon the user’s HRTF, para 64, and Spivak, the HRTF as audio cues is used for sound spatialization, when the headphones are used, col 5, ln 60-67, col 6, ln 1-5).
Claim 17: the combination of Lovitt and Spivak further teaches, further teaches, according to claim 1 above, wherein the headworn device is a pair of headphones (Lovitt, AR headset with a left and a right transducer 306A/306B in fig. 3 and Spivak, rendering audio signal on headphones by speakers 310 in fig. 3, col 9, ln 46-47).
Claims 8-12, 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Lovitt (above) and in view of reference Laaksonen et al. (US 20200145778 A1, hereinafter Laaksonen).
Claim 8: Lovitt further teaches, according to claim 2 above, the method further comprising: determining a changing position of the wearer of the AR headset by processing an output of one or more sensors in the first audio system using a visual odometry technique or a visual simultaneous localization and mapping technique (Lovitt, SLAM technique, para 56, or one or more sensors used to identify a location of a user, para 52, e.g., using cameras to indicate the location of a user or location of the device, location of other sound source, para 62), and wherein the spatial audio rendering comprises spatially rendering, using a first spatial audio processing (Lovitt, practice of the user’s HRTF with detected DOA parameter, para 61 and Spivak, HRTFs as direct sound path parameter, col 5, ln 60-67, col 6, ln 1-5), the first target audio signal as a point source that is positioned on a face of the first talker (Lovitt, as sound source and the related DOA determined, para 61 and Spivak, gazing at the participant relative to the listener in fig. 5), the first spatial audio filter being configured, while spatially rendering the first target audio signal, according to the changing position of the wearer of the AR headset (Lovitt, HRTF modified according to instantly detected DOA, para 61 and the user’s HRTF is inherently related to the listener’s relative position to the sound source, i.e., acoustic path, para 64 and Spivak, the spatialization is upon the sound parameters including relative locations of the sound sources to the listener, e.g., participant on the left 512 and right side 514 relative to the listener positioned on the front of the display in fig. 5), except explicitly teaching wherein the first spatial audio processing is a first spatial audio filter.
Laaksonen teaches an analogous field of endeavor by disclosing a method (title and abstract, ln 1-16 and method steps in figs. 5-6 and implemented on an apparatus in fig. 1) and wherein an VR headset by a wearer is disclosed (capture user 114 with headphones 115, headphones by using a head-mounted device HMD such as head-mounted array 102, para 79 and in a virtual reality environment and display, para 62 and generating augmented reality through AR apparatus 112, para 85) and wherein determining a changing position of the wearer of the AR headset by processing an output of a sensor in a first audio system (rotation of capture user’s head is tracked by a sensor as captured direction data, para 79), and wherein the spatial audio rendering comprises spatially rendering (via AR apparatus 112 and headphones 111 of the recipient user 107 in fig. 1, para 86, 112), using a first spatial audio filter (by using HRTF filtering performed by the apparatus 112, para 68), a first target audio signal as a point source that is positioned on a face of the first talker (face of the capture user 114 in a scene 103 in fig. 1, para 79), the first spatial audio filter being configured, while spatially rendering the first target audio signal, according to the changing position of the wearer of the AR headset (via transmission 501 of the metadata including the capture direction 506 and used by rendering 508 through the capture direction metadata 510 in fig. 5, para 148 and generating spatial audio content modified by at least the capture direction metadata, para 148) for benefits of improving the quality of the user experiences (by providing user desired stylistic effect in the virtual reality content, para 2, and providing user with an immersive virtual experience, para 62 and by perceiving sound change based on both movements of the both users via headtracking on both sides in figs. 1, 5).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have applied wherein the first spatial audio filter, as taught by Laaksonen, to the first spatial audio processing in the method, as taught by Lovitt, for the benefits discussed above.
Claim 9: the combination of Lovitt and Laaksonen further teaches, according to claim 8 above, wherein the conversation channel further comprises a second target audio signal that contains isolated voice of a second talker, the second target audio signal i) produced by processing an output of one or more microphones in a third audio system and then received by the first audio system over-the-air from the third audio system, or ii) produced by the first audio system processing an output of the microphone array in the first audio system, the method further comprising in response to receiving the opt-in indication, spatial audio rendering the second target audio signal into the left speaker driver signal and the right speaker driver signal (Lovitt, the second speaking user 1107 and the first speaking user 1105 relative to the listener 1101 and with speech differentiation module 1103 sequentially delivering the translated words from different speaking users to the listening user through personalization engine in fig. 11, para 86, and thus, similar spatial audio capture and playback to the first speech signal as described in claim 1 above, para 86-88).
Claim 10: the combination of Lovitt and Laaksonen further teaches, according to claim 9 above, determining a changing position of the face of the first talker (Laaksonen, by using head tracker 113 in fig. 1, to track movement of the user’s head, para 95), and wherein the spatial audio rendering comprises configuring the first spatial audio filter according to the changing position of the face of the first talker (Laaksonen, the head orientation and movement information is taken by rendering 508 so that perceived spatial audio content is modified, para 95).
Claim 11: the combination of Lovitt and Laaksonen further teaches, further teaches, according to claim 10 above, wherein the first target audio signal is produced by one or more microphones in the second audio system (Lovitt, the input transducers or microphones of the AR system 100 having the headsets 630B and worn by a speaking user 606 and providing audio input signal 605 to the personalization engine 600 in fig. 6 and similar to the computer system 401, para 77) and then received by the first audio system over-the-air (via a communicatoi module 404 in fig. 4, e.g., via radios in the air, para 66), and the determining the changing position of the face of the first talker (Laaksonen, via the head tracker at capture user 503 in fig. 5 and discussed in claim 8 above), except explicitly teaching comprises: using an ultra-wideband time of flight localization technique to sense a position of the first talker.
An Official Notice is taken that the ultra-wideband UWB time of flight localization technique is notoriously well-known in the art and using such technique for sense a position of a talker is also notoriously well-known in the art for benefits of higher localization precision with less noise-prone and low-latency for real-time tracking object location, high secured in data transmission, and less power consumed, etc.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have applied the ultra-widband time of flight localization technique to sense the position of the talker, as taught by well-know in the art above, to determining the changing position of the face of the first talker in the method, as taught by the combination of Lovitt and Laaksonen, for the benefits discussed above.
Claim 12: the combination of Lovitt and Laaksonen further teaches, further teaches, according to claim 10 above, wherein the second talker is in a real environment of the wearer (Lovitt, the second speaking user 1107 in fig. 11 and discussion in claim 9 above) and is i) depicted in a camera image of the real environment that is being displayed by a display panel of an AR headset (Lovitt, enable more immersive interactions with other people in a virtual word to enhance the interactions with other people in the real world, para 55 and see virtual elements generated by the AR headset, para 75, i.e., displayed speaking user image on the listening user HMD device 630A in fig. 6), or ii) visible by the wearer through the display panel of the AR headset (Lovitt, see elements of the real word in the virtual environment and discussed above), and the spatial audio rendering further comprises spatially rendering, using a second spatial audio filter, the second target audio signal as another point source that is positioned on a face of the second talker (Lovitt, similar to the first spatial audio filter as discussed in claim 8 above and applied to the second speaking user 606 in fig. 11), wherein the face of the second talker is displayed by or is visible through the display panel of the AR headset (Lovitt, the elements of the real world can be seen in the virtual word or display and discussed above), the second spatial audio filter being configured, while spatially rendering the second target audio signal, according to the changing position of the wearer (Lovitt, the discussed in claim 8-9 above and Laaksonen, via the head tracker and discussed in claims 9-10 above).
Claim 14: Lovitt further teaches, further teaches, according to claim 1 above, wherein the first target audio signal is produced by processing the output of the microphone array in the first audio system (processed by the controller 225 in fig. 2, para 47 and the discussed in claim 1 above) which processing comprises performing a direction detection (estimating the DOA from the sound source to the microphone in fig. 2, para 47) and including performing a direction detection (the DOA estimated and discussed above or using sensors if a distance among microphones is greater than a threshold value, para 46), and outputting a target direction of the voice source relative to the head of the wearer (the DOA is estimated in measurement of angle of the microphone array with respect to the sound source or talker, i.e., azimuth and elevation coordinate system, para 60), providing the target direction to a voice isolation algorithm (e.g., beamforming performed by the microphone array, para 46) that processes the output of the microphone array to produce the first target audio signal (the DOA is used to spatialize the sound signals as if it coming from the DOA, para 64), and wherein the spatial audio rendering uses the target direction to render the first target audio signal so that the wearer perceives the isolated voice as coming from the target direction (the discussed above, para 64), except azimuth and elevation to be outputted.
Laaksonen teaches an analogous field of endeavor by disclosing a method (title and abstract, ln 1-16 and method steps in figs. 5-6 and implemented on an apparatus in fig. 1) and wherein outputting an azimuth and an elevation representing the determined direction to the audio sources (para 88) for the benefits discussed in claim 8 above.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have applied the azimuth and the elevation representing direction detection of the target direction, as taught by Laaksonen, to the target direction determined in the method, as taught by Lovitt, for the benefits discussed above.
Claim 15: the combination of Lovitt and Laaksonen further teaches, further teaches, according to claim 14 above, the method further comprising determining whether a wearer head direction of the wearer of the headworn device is in the target direction (Lovitt, using microphone signals or sensor signals, para 61-62 and Laaksonen, through the head tracking 503 in fig. 5), based on i) detecting gaze of the wearer using an inward-facing camera of an AR headset (Lovitt, using the eye tracker or gaze detector, including a camera, to determine user’s or device’s location, or location of the sound source, para 62 and Laaksonen, a camera to visible source location, para 125), ii) processing images from a front-facing camera of the AR headset (Lovitt, the camera 104 in fig. 1 and the camera above), or both i) and ii) (discussed above); and in response to determining that the wearer head direction is in the target direction, generating the opt-in indication (Lovitt, tracking listening user’s eye movement to determine direction of the speaker user and start determination of the sound source, para 62 ).
Claim 16: the combination of Lovitt and Laaksonen further teaches, further teaches, according to claim 15 above, the method further comprising:
after generating the opt-in indication, tracking the target direction of the voice source using only acoustical-based processing of the output of the microphone array, while tracking the wearer head direction, to inform the spatial audio rendering (Lovitt, the detected DOA at the listening user and used for modifying the rendered spatial audio signal and discussed in claim 12 above, and Laaksonen, the position tracking at listener side 512 to modify the decoded audio signal rendering in fig. 5).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Lovitt (above) and in view of reference Laaksonen et al. (US 20210127224 A1, hereinafter Laaksonen24).
Claim 13: Lovitt further teaches, according to claim 1 above, method of claim 1 further comprising: receiving location information (612 in fig. 9, via GPS associated with the speaking user 606 in fig. 9, para 81), except
receiving metadata from the second audio system, wherein the metadata includes dynamic range of or an average speech level of voice of the first talker and using the metadata to adjust a playback level of first target audio signal.
Laaksonen24 teaches an analogous field of endeavor by disclosing a method (title and abstract, ln 1-16 and method steps in fig. 2 and implemented on an apparatus in fig. 1) and wherein a second audio system is disclosed (analysis side 121 in fig. 1) from which metadata is received (metadata as part of data stream 102 and received by the synthesis side 131 in fig. 1), wherein the metadata includes dynamic range of or an average speech level of voice of the first talker (metadata extensions such as MPEG-H dynamic range control, loudness, and peak limiter parameter for volume and gain control of the headset 107 and spatial audio 6DoF, para 65 and audio signal from user’s voice in the 6DoF scene in fig. 6, para 133) and using the metadata to adjust a playback level of first target audio signal (forming spatial audio 6DoF and rendered by headset 107 in fig. 1 and in an addition to user provided volume control for augmentation to spatial audio 3DoF to 6DoF, para 66) for benefits of providing higher quality of spatial sound (by rendering 3DoF spatial audio signal to 6DoF rendered, para 60, and without loss in quality, para 62 and maintaining the perceptional quality of the augmentation audio, para 118 and fitting the volume effect based on the 6DoF position, para 66).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have applied receiving the metadata from the second audio system, wherein the metadata includes the dynamic range of or an average speech level of the voice of the first talker and using the metadata to adjust the playback level of the first target audio signal, as taught by Laaksonen24, to method step of receiving the location information in the method, as taught by Lovitt, for the benefits discussed above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LESHUI ZHANG whose telephone number is (571)270-5589. The examiner can normally be reached Monday-Friday 6:30amp-4:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vivian Chin can be reached at 571-272-7848. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LESHUI ZHANG/
Primary Examiner,
Art Unit 2695
Prosecution Insights