DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04 March 2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 22 – 29 and 32 – 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over Usher et al. (hereinafter Usher, U.S. Patent Application Publication 2014/0270200; note Usher at [0045] references incorporation in its entirety of U.S. Patent Application Publication 2015/0172814 to Usher et al., hereinafter Usher 2) in view of Kwon (U.S. Patent Application Publication 2022/0060572).
Regarding Claim 22, Usher discloses:
An audio processing method (e.g. processing audio and acoustic signals; [0002]), comprising:
capturing, using a first set of microphones on a mobile device (e.g. mobile device 150 includes one or more microphones; [0028] see also microphones 22 in a separate device such as a mobile device; [0022]; alternatively consider microphones 161/162 present in second media device 160 [depicted as a smart watch in Fig. 2A]; further in the alternative, consider the Fig. 3A embodiment having microphones located on glasses 140), a first audio signal from an audio scene (e.g. capture of a sound source; [0028] note microphones in any of the devices work together [0028]; also see discussion of reception of audio by various microphones discussed in [0022] as well as capturing of audio by the various microphones depicted in Figs. 1A-B);
capturing, using a second set of microphones on a pair of earbuds (e.g. see also microphones placed in earphone 130 depicted in Fig. 3A), a second audio signal from the audio scene (e.g. capture of a sound source; [0028] note microphones in any of the devices work together [0028]; also see discussion of reception of audio by various microphones discussed in [0022] as well as capturing of audio by the various microphones depicted in Figs. 1A-B);
capturing, using a camera on the mobile device, a video signal from a video scene (e.g. capturing images and/or video [0050]);
determining, with at least one processor, context information for the mobile device (e.g. gains individually adjusted depending on user/environment context; [0026]; see further compensating volume levels to spatialize; [0027]).
Usher fails to explicitly disclose:
the context information including whether a microphone on the mobile device generates a mono audio stream or a stereo audio stream and whether the camera is being used in a selfie mode.
In a related field of endeavor (e.g. mobile device operation), Kwon discloses a foldable mobile phone that detects and sets operational parameters of various elements of the mobile phone, including which cameras are active depending on the state of the phone (Figs. 1A-D, 15A-D [0090] – [0093], [100]), which speakers are active [0152], and which microphones are used [0153].
Modifying Usher’s system to incorporate the features of Kwon, namely using Kwon’s mobile device in Usher, further discloses:
determining, with at least one processor, context information for the mobile device, the context information including whether a microphone on the mobile device generates a mono audio stream or a stereo audio stream (e.g. Usher’s context information, now including the ability to determine open or closed state, in the open state outgoing sound is received at both microphone 1 and microphone 2 [“stereo”] and in the closed state, only microphone 1 is activated and microphone 2 is deactivated [“mono”]; Kwon [0153]) and whether the camera is being used in a selfie mode (e.g. see Figs. 1A-D, particularly 1D which shows the various operating modes and when they are active, specifically note the selfie modes in 3 of the modes and non-self, or forward direction shooting in the 4th mode; see also paras [0090] – [0093]).
It would have been obvious to one of ordinary skill in the art at the time the invention was filed to apply the teachings of Kwon to the system of Usher. Using Kwon’s mobile device as the mobile device in Usher would address a number of known issues in the art as well as provide a number of advantages, including those detailed in the background [0003] – [0005] of Kwon, and further also providing an electronic device that can not only share at least a portion of the image output on the display with an external device but also handle a different action on the display in a multitasking fashion; Kwon [0006].
The combination further discloses:
determining, with the at least one processor, an audio processing profile based on the context information and a configuration of the first set of microphones (e.g. Usher’s user and mobile phone now implementing the features of Kwon, namely the ability to use predetermined criterion which may be an initial or user setting [or “profile”]; [0205] of Kwon; these settings are used to determine control in a number of aspects, including screen settings; [0251] of Kwon; further note the flow charts, and the numerous configuration tables throughout the figures, for example in Figs. 1D, 8C, 15C which detail operation of the cameras and speakers/microphone for each configuration/setting);
generating, with the at least one processor, a processed audio signal according to the audio processing profile (e.g. Usher’s incoming signals processed and output, see Fig. 1B, and/or Figs. 2B, D, particularly element 283; and note Usher’s gains individually adjusted depending on user/environment context; [0026]; see further compensating volume levels to spatialize; [0027]; now as modified by Kwon’s mobile device), and
combining, with the at least one the processed audio signal and the captured video signal as multimedia output for playback on a playback device, wherein rendering of the multimedia output on the playback device is guided based on the audio processing profile (e.g. producing a composite signal, and directional sound processing and localization of the audio; [0045] of Usher; further note providing/presenting audio content and displaying images/video; [0050] of Usher, now modified by Kwon as above which detail operations of the mobile device based on configuration/orientation and the like, see the flow charts, and the numerous configuration tables throughout the figures, for example in Figs. 1D, 8C, 15C which detail operation of the cameras and speakers/microphone for each configuration/setting).
Regarding Claim 23, in addition to the elements stated above regarding claim 22, Usher in the combination further discloses:
wherein the processed audio signal is obtained by at least one of mixing the first audio signal and the second audio signal or selecting one of the first audio signal or the second audio signal based on the context information (e.g. adjusting gains of the signals and mixing them together to produce a combined audio signal; [0024]).
Regarding Claim 24, in addition to the elements stated above regarding claim 22, Usher in the combination further discloses:
wherein the context information includes at least one of speech location information, a camera identifier for the camera used for video capture or at least one channel configuration of the first audio signal (e.g. implementing appropriate gain modification based on the direction of the source of the voice activity; [0027]), wherein the channel configuration includes at least a microphone layout and an orientation of the mobile device used to capture the first audio signal (e.g. note the device distinguishes left from right earphones based on where the individual speaking is located; [0034]).
Regarding Claim 25, in addition to the elements stated above regarding claim 24, Usher in the combination further discloses:
wherein the speech location information indicates the presence of speech in a plurality of regions of the audio scene (e.g. see for example, “close” proximity of a user speaking; [0027]; and determination of a relative direction to the earphone user; [0031]; as well as the left/right side delineation provided in [0034]; and location determination in general in [0037]).
Regarding Claim 26, in addition to the elements stated above regarding claim 25, Usher in the combination further discloses:
wherein the plurality of regions include self area (e.g. person 1, earphone user; Fig. 2A and proximity for directing conversation to person 1 [0026]), frontal area (e.g. proximity in “front” of a wearer of the earphones; [0026]) and side area (e.g. to one side; [0027]; see also left/right [0034]), a first speech from the self area is a self-speech of a first speaker wearing the earbuds (e.g. see [0026] detailed aspects of voice conversation with a second individual; and also two-way conversation between a user wearing the earphone and a second individual [0035]), a second speech from the frontal area is a speech of a second speaker not wearing the earbuds in the frontal area of the camera used for video capture (e.g. front voice activity detection ; [0031] – [0032]), and a third speech from the side area is a speech of a third speaker to the left or right of the first speaker wearing the earbuds (e.g. determining if second individual to the side and speaking; [0034]).
Regarding Claim 27, in addition to the elements stated above regarding claim 24, Usher in the combination further discloses:
wherein the camera used for video capture is one of a front-facing camera or rear-facing camera (e.g. cameras situated at the front or other directions [0045]).
Regarding Claim 28, in addition to the elements stated above regarding claim 24, the combination further discloses:
wherein the at least one channel configuration includes a mono channel configuration and a stereo channel configuration (e.g. Usher’s microphones, now modified to operate as Kwon discloses and is explained above, namely in one configuration [open], both microphones 1 and 2 are active [“stereo”], and in another configuration [folded], only microphone 1 is activated while microphone 2 is deactivated [“mono”]; [0152]-[0153] of Kwon).
Regarding Claim 29, in addition to the elements stated above regarding claim 24, Usher in the combination further discloses:
wherein the speech location information is detected using at least one of audio scene analysis or video scene analysis (e.g. monitoring the environment [“scene] for sounds and location; [0034])
Regarding Claim 32, in addition to the elements stated above regarding claim 29, Usher in the combination further discloses:
wherein the video scene analysis includes speaker detection and localization (e.g. see further detail of the spatialization provided in Usher 2, U.S. Patent Application Publication 2015/0172814 which is incorporated in its entirety into Usher as noted in [0045]; Usher 2 at [0081] which details spatial filtering for sounds originating from one direction; and incorporating the acoustic sensor in combination with video monitoring; Usher 2 at [0096], also see [0037]).
Regarding Claim 33, in addition to the elements stated above regarding claim 32, Usher in the combination further discloses:
wherein the speaker detection is implemented by facial recognition, the speaker localization is implemented by estimating speaker distance from the camera based on a face area provided by the facial recognition and focal length information from the camera used for video signal capture (e.g. distinguishing a user’s voice based on speaker characteristics or sound patterns; [0030] see further detail of this provided in Usher 2, U.S. Patent Application Publication 2015/0172814 which is incorporated in its entirety into Usher as noted in [0045]; Usher 2 at [0036] using the camera to detect and track a talker; see further Usher 2 at [0037] detailing face tracking of a user based on captured images or video)
Regarding Claim 34, in addition to the elements stated above regarding claim 23, Usher in the combination further discloses:
wherein the mixing or selection of the first and second audio signal further comprises a pre-processing step that adjusts one or more aspects of the first and second audio signal (e.g. see Fig. 1B and gain adjustments 185 and 187, which can be said to be “pre-processing” as further processing is performed afterwards, for example summing in 190).
Regarding Claim 35, in addition to the elements stated above regarding claim 34, Usher in the combination further discloses:
wherein the one or more aspects includes at least one of timbre, loudness or dynamic range (e.g. audibly compensating volume levels [“loudness”]; [0027])
Regarding Claim 36, in addition to the elements stated above regarding claim 23, Usher in the combination further discloses:
further comprising a post-processing step that adjusts one or more aspects of the mixed or selected audio signal (e.g. see Fig. 1B and summer adjustment 19, which can be said to be “post-processing” as previous processing is performed prior to this stage, for example gain adjustment in 185 and 187).
Regarding Claim 37, in addition to the elements stated above regarding claim 36, Usher in the combination further discloses:
wherein the one or more aspects of the mixed or selected audio signal include adjusting a width of the mixed or selected audio signal by attenuating a side channel component of the mixed or selected audio signal (e.g. upon detecting side voice activity (see step 255 FIG. 2B), the increasing/decreasing steps can be supplemented with frequency dependent filtering to selectively attenuate/amplify voice characteristics; [0034])
Regarding Claim 38, in addition to the elements stated above regarding claim 22, Usher in the combination further discloses:
An audio processing system, comprising:
a first set of microphones on a mobile device (e.g. mobile device 150 includes one or more microphones; [0028] see also microphones 22 in a separate device such as a mobile device; [0022]; alternatively consider microphones 161/162 present in second media device 160 [depicted as a smart watch in Fig. 2A]; further in the alternative, consider the Fig. 3A embodiment having microphones located on glasses 140) for capturing a first audio signal from an audio scene (e.g. capture of a sound source; [0028] note microphones in any of the devices work together [0028]; also see discussion of reception of audio by various microphones discussed in [0022] as well as capturing of audio by the various microphones depicted in Figs. 1A-B);
a second set of microphones on a pair of earbuds (e.g. see also microphones placed in earphone 130 depicted in Fig. 3A) for capturing a second audio signal from the audio scene (e.g. capture of a sound source; [0028] note microphones in any of the devices work together [0028]; also see discussion of reception of audio by various microphones discussed in [0022] as well as capturing of audio by the various microphones depicted in Figs. 1A-B);
a camera on the mobile device (e.g. cameras situated at the front or other directions [0045]) for capturing a video signal from a video scene (e.g. capturing images and/or video [0050]);
at least one processor (e.g. processor 103); and
a non-transitory, computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the one or more processors to perform the operations of claim 22 (e.g. in addition to the elements mapped in the rejection of claim 22 above, Usher teaches implementation within a digital signal processor or other software programmable device; [0048]; and utilizing a microprocessor and/or DSP with associated storage memory for controlling operations of the components; [0061]).
Regarding Claim 39, in addition to the elements stated above regarding claim 22, Usher in the combination further discloses:
A non-transitory, computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the operations of claim 22 (e.g. in addition to the elements mapped in the rejection of claim 22 above, Usher teaches implementation within a digital signal processor or other software programmable device; [0048]; and utilizing a microprocessor and/or DSP with associated storage memory for controlling operations of the components; [0061]).
Regarding Claim 40, in addition to the elements stated above regarding claim 22, Usher in the combination further discloses:
wherein the mobile device and the playback device are the same device (e.g. see example embodiment depicting the mobile device as a media device; [0046])
Claim(s) 30 and 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Usher et al. (hereinafter Usher, U.S. Patent Application Publication 2014/0270200; note Usher at [0045] references incorporation in its entirety of U.S. Patent Application Publication 2015/0172814 to Usher et al., hereinafter Usher 2) in view of Kwon (U.S. Patent Application Publication 2022/0060572) in further view of Bryan et al. (hereinafter Bryan, U.S. patent Application Publication 2019/0272842).
Regarding Claim 30, in addition to the elements stated above regarding claim 29, Usher in the combination further discloses:
wherein the audio scene analysis comprises at least one of self-external speech segmentation or external speech direction-of-arrival (DOA) estimation (e.g. implementing gain modification based on the direction of the source [“DOA”] of the voice activity [0037]),
the external speech DOA estimation takes inputs from the first and second audio signal, and extracts spatial audio features from the inputs (e.g. emphasizing the location of Person 2, for example, in front of the user; [0027] and beamforming for localization of a user’s voice; [0028] see further detail of this provided in Usher 2, U.S. Patent Application Publication 2015/0172814 which is incorporated in its entirety into Usher as noted in [0045]; Usher 2 at [0081] which details spatial filtering for sounds originating from one direction).
The combination fails to explicitly disclose:
wherein the self-external speech segmentation is implemented using bone conduction measurements from a bone conduction sensor embedded in at least one of the earbuds.
In a related field of endeavor (e.g. speech enhancement in electronic devices), Bryan details using bone conduction detection using accelerometers to detect vibrations of the user’s vocal chords and filtering them; [0022], [0028] and [0029].
Modifying the combination to incorporate the features of Bryan further discloses:
wherein the self-external speech segmentation is implemented using bone conduction measurements from a bone conduction sensor embedded in at least one of the earbuds (e.g. Usher’s earbuds, now adapted to include the accelerometers placed in earbuds 110 for the purpose of detecting bone conduction vibrations; [0028], which assists in determination of both types of speech, voice and unvoiced [differentiation between these indicates some sort of delineation, or “segmentation”].)
It would have been obvious to one of ordinary skill in the art at the time the invention was filed to apply the teachings of Bryan to the device disclosed by the combination. Using the accelerometer and microphone signals together leverages the accelerometer signal’s natural robustness against external or acoustic noise (e.g., babble, wind, car noise, interfering speech, etc.) to improve speech quality; [0019] of Bryan.
Regarding Claim 31, in addition to the elements stated above regarding claim 30, Usher in the combination further discloses:
wherein the spatial audio features include at least inter-channel level difference (e.g. spatialize the secondary sound source; [0027]; note also that the audio signals from the first microphone 141 and second microphone 142 are multiplexed and for analysis of a phase angle of the inter-microphone coherence for directional sensitivity [0045]; see further detail of this provided in Usher 2, U.S. Patent Application Publication 2015/0172814 which is incorporated in its entirety into Usher as noted in [0045]; Usher 2 at [0097] detailing coherence function which identifies localization information related to the magnitude differences).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew C Flanders whose telephone number is (571)272-7516. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights