DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 5-6, 8, 11, 13, 17-18, and 20 is/are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Boykin et al. (US 2018/0050800 A1 and hereafter Boykin).
Regarding claim 1, Boykin anticipates:
“A camera” (see Boykin, figure 6, unit 16, figures 26A-D, unit 2202, and ¶ 0089, 0139, and 0218)
“a multi-directional microphone configured to receive audio” by teaching a microphone array (see Boykin, figure 6, unit 16 and 1-4, and ¶ 0089);
“a processor configured to extract audio information about the audio from the multi-directional microphone” by teaching a processor for processing the audio captured by the microphones, such as processing the audio to determine the direction of the sound (see Boykin, figure 27, units 2513 and 2713 and ¶ 0066, 0089, and 0145-0146); and
“a memory configured to store instructions executable by the processor” by teaching internal memory, or storage, for the computer program instructions of the device (see Boykin, figure 27, units 2511 and 2711, and ¶ 0056, 0066, and 0145-0146)
“wherein, by executing the instructions stored on the memory, the processor is configured to control:
a direction information calculation module to determine geo-orientation information about the audio based on the audio information” by teaching that the captured audio data is processed to determine a direction of a sound source and/or determine whether designated audio content is present (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0089 and 0233), and
“an audio information providing module to map information to metadata including audio clip information about an audio clip from the audio, and the geo-orientation information” by teaching that metadata, such as text, or keywords, determined from voice-to-text translation, timing of a detected sound, and/or the type of event detected, such as a gunshot event, are included or associated with the recorded audio, and the geo-orientation information, such as GPS, direction, location, and/or vector coordinates are included or associated with the recorded audio (see Boykin, ¶ 0059, 0148-0149, 0174-0175, 0177, 0181, and 0233).
Regarding claim 5, see the preceding rejection with respect to claim 1 above. Boykin anticipates the “camera of claim 1, wherein the processor is further configured to control a type classification module to classify the audio into a type based on the audio information” by teaching that the audio is classified, or identified, such that different sounds, voices, and noises are identified (see Boykin, ¶ 0174).
Regarding claim 6, see the preceding rejection with respect to claim 5 above. Boykin anticipates the “camera of claim 5, wherein the processor is further configured to control the type classification, based on the type of the audio being a voice, to:
generate language information of the voice and text information of the voice” by teaching voice-to-text translation of detected speech (see Boykin, ¶ 0175), and
“detect a pre-stored keyword included in the text information” by teaching that detected keywords are marked in the text (see Boykin, ¶ 0175).
Regarding claim 8, Boykin anticipates:
“A system for providing audio information, the system comprising:
a multi-directional microphone configured to receive audio” by teaching a microphone array (see Boykin, figure 6, unit 16 and 1-4, and ¶ 0089);
“a camera configured to:
determine geo-orientation information about the audio received by the multi-directional microphone” by teaching that the captured audio data is processed to determine a direction of a sound source and/or other geo-orientation information, such as GPS, direction, location, and/or vector coordinates associated with the recorded audio (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0059, 0089, 0148-0149, 0174-0175, 0177, 0181, and 0233), and
“map information to metadata including audio clip information about an audio clip from the audio, and the geo-orientation information” by teaching that metadata, such as text, or keywords, determined from voice-to-text translation, timing of a detected sound, and/or the type of event detected, such as a gunshot event, are included or associated with the recorded audio, and the geo-orientation information, such as GPS, direction, location, and/or vector coordinates are included or associated with the recorded audio (see Boykin, ¶ 0059, 0148-0149, 0174-0175, 0177, 0181, and 0233); and
“a server configured to receive the metadata of the audio from the camera and control the camera based on the metadata” by teaching that the system includes uploading the captured data and metadata to remote computer storage (see Boykin, figure 1, units 15 and 16, ¶ 0061, 0066, 0193, and 0197).
Regarding claim 11, see the preceding rejection with respect to claim 8 above. Boykin anticipates the “system of claim 8, wherein the camera is further configured to classify the audio into a type based on the audio information” by teaching that the audio is classified, or identified, such that different sounds, voices, and noises are identified (see Boykin, ¶ 0174)
Regarding claim 13, Boykin anticipates:
“A method of providing audio information by using a multi-directional microphone provided in a camera” by teaching a microphone array in a camera (see Boykin, figure 6, unit 16 and 1-4, and ¶ 0089),
“the method comprising:
extracting audio information about audio from the multi-directional microphone” by teaching that the captured audio data from the microphone array is processed to determine a direction of a sound source and/or determine whether designated audio content is present (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0089 and 0233);
“determining geo-orientation information about the audio based on the audio information” by teaching that the captured audio data is processed to determine a direction of a sound source and/or other geo-orientation information, such as GPS, direction, location, and/or vector coordinates associated with the recorded audio (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0059, 0089, 0148-0149, 0174-0175, 0177, 0181, and 0233); and
“mapping information to metadata including audio clip information about an audio clip from the audio, and the geo-orientation information” by teaching that metadata, such as text, or keywords, determined from voice-to-text translation, timing of a detected sound, and/or the type of event detected, such as a gunshot event, are included or associated with the recorded audio, and the geo-orientation information, such as GPS, direction, location, and/or vector coordinates are included or associated with the recorded audio (see Boykin, ¶ 0059, 0148-0149, 0174-0175, 0177, 0181, and 0233).
Regarding claim 17, see the preceding rejection with respect to claim 13 above. Boykin anticipates the “method of claim 13, further comprising classifying the audio into a type based on the audio information” by teaching that the audio is classified, or identified, such that different sounds, voices, and noises are identified (see Boykin, ¶ 0174).
Regarding claim 18, see the preceding rejection with respect to claim 17 above. Boykin anticipates the “method of claim 17, wherein the classifying the audio into a type comprises, based on the type of the audio being a voice:
generating language information of the voice and text information of the voice” by teaching voice-to-text translation of detected speech (see Boykin, ¶ 0175), and
“detecting a pre-stored keyword included in the text information” by teaching that detected keywords are marked in the text (see Boykin, ¶ 0175).
Regarding claim 20, see the preceding rejection with respect to claim 13 above. Boykin anticipates a “non-transitory computer-readable storage medium storing a computer program which, when executed, causes a processor to execute the method of claim 13” by teaching internal memory, such as a hard disk, for the computer program instructions of the device (see Boykin, figure 27, units 2511 and 2711, and ¶ 0056, 0066, and 0145-0146).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2-4, 9-10, and 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boykin as applied to claims 1, 8, and 13 above, and further in view of Benattar et al. (US 2018/0146284 A1 and hereafter Benattar).
Regarding claim 2, see the preceding rejection with respect to claim 1 above. Boykin anticipates the camera of claim 1, wherein the captured audio data from the microphone array is processed to determine a direction of a sound source and/or determine whether designated audio content is present (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0089 and 0233). However, Boykin does not appear to teach the features where the geo-orientation information is determined “based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone”.
Benattar discloses a beamformer direction of arrival and orientation analysis system (see Benattar, abstract and figures 1 and 6-7). Herein, Benattar teaches beamforming techniques that detect a relative time delay between audio signals detected by the microphones and determine a direction of the desired audio source, such as a voice signal (see Benattar, ¶ 0028 and 0080). It would have been obvious to one of ordinary skill in the art at the time of the effective filing date to modify Boykin with the teachings of Benattar to improve the captured audio (see Boykin, ¶ 0089 and 0144, in view of Benattar, ¶ 0013-0017, 0027-0028, and 0080).
Therefore, the combination of Boykin and Benattar makes obvious the “camera of claim 1, wherein the processor is further configured to control the direction information calculation module to determine the geo-orientation information based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone” by teaching that the captured audio data is processed to determine a direction of a sound source and/or other geo-orientation information, such as GPS, direction, location, and/or vector coordinates associated with the recorded audio (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0059, 0089, 0148-0149, 0174-0175, 0177, 0181, and 0233) and makes obvious that the geo-orientation information based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone (see Boykin, ¶ 0015-0017 and 0028).
Regarding claim 3, see the preceding rejection with respect to claim 2 above. The combination makes obvious the “camera of claim 2, wherein the geo-orientation information comprises:
direction information about a yaw corresponding to a rotation about a vertical axis of the multi-directional microphone, a roll corresponding to a rotation about a front-back axis of the multi-directional microphone, and a pitch corresponding to a rotation about a left-right axis of the multi-directional microphone” because Benattar makes obvious a microphone array with at least 4 microphones for detecting audio signal direction in three-dimensional space (see Benattar, ¶ 0028), and makes obvious the use of a position sensor for detecting yaw, roll, and pitch information (see Benattar, ¶ 0032-0034 and 0095); and
“location information about a latitude, a longitude, and an elevation of the multi-directional microphone” because GPS information is included and GPS information includes latitude, longitude, and elevation (see Boykin, ¶ 0059, 0148-0149, and 0188).
Regarding claim 4, see the preceding rejection with respect to claim 3 above. The combination makes obvious the “camera of claim 3, wherein the audio clip information comprises a uniform resource locator (URL) of the audio clip or data in which the audio clip is encoded” because Boykin teaches that the recorded audiovisual data is uploaded to network servers and is accessible through the internet, which makes obvious that a webpage, or URL, is provided for accessing the recorded data (see Boykin, ¶ 0061, 0091, 0188, 0193, and 0197).
Regarding claim 9, see the preceding rejection with respect to claim 8 above. Boykin anticipates the system of claim 8, wherein the captured audio data from the microphone array is processed to determine a direction of a sound source and/or determine whether designated audio content is present (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0089 and 0233). However, Boykin does not appear to teach the features where the geo-orientation information is determined “based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone”. For the same reasons as stated above with respect to claim 2, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date to modify Boykin with the teachings of Benattar to improve the captured audio (see Boykin, ¶ 0089 and 0144, in view of Benattar, ¶ 0013-0017, 0027-0028, and 0080).
Therefore, the combination of Boykin and Benattar makes obvious the “system of claim 8, wherein the camera is further configured to determine the geo-orientation information based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone” by teaching that the captured audio data is processed to determine a direction of a sound source and/or other geo-orientation information, such as GPS, direction, location, and/or vector coordinates associated with the recorded audio (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0059, 0089, 0148-0149, 0174-0175, 0177, 0181, and 0233) and makes obvious that the geo-orientation information based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone (see Boykin, ¶ 0015-0017 and 0028).
Regarding claim 10, see the preceding rejection with respect to claim 9 above. The combination makes obvious the “system of claim 9, wherein the audio clip information comprises a uniform resource locator (URL) of the audio clip or data in which the audio clip is encoded” because Boykin teaches that the recorded audiovisual data is uploaded to network servers and is accessible through the internet, which makes obvious that a webpage, or URL, is provided for accessing the recorded data (see Boykin, ¶ 0061, 0091, 0188, 0193, and 0197).
Regarding claim 14, see the preceding rejection with respect to claim 13 above. Boykin anticipates the method of claim 13, wherein the captured audio data from the microphone array is processed to determine a direction of a sound source and/or determine whether designated audio content is present (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0089 and 0233). However, Boykin does not appear to teach the features where the geo-orientation information is determined “based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone”. For the same reasons as stated above with respect to claim 2, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date to modify Boykin with the teachings of Benattar to improve the captured audio (see Boykin, ¶ 0089 and 0144, in view of Benattar, ¶ 0013-0017, 0027-0028, and 0080).
Therefore, the combination of Boykin and Benattar makes obvious the “method of claim 13, wherein the determining the geo-orientation information comprises determining the geo-orientation information based on an arrival time difference, an amplitude, and an intensity of the audio received from the multi-directional microphone” by teaching that the captured audio data is processed to determine a direction of a sound source and/or other geo-orientation information, such as GPS, direction, location, and/or vector coordinates associated with the recorded audio (see Boykin, figure 9, units 1-4 and steps 72 and 74, and ¶ 0059, 0089, 0148-0149, 0174-0175, 0177, 0181, and 0233) and makes obvious that the geo-orientation information based on an arrival time difference, an amplitude, and an intensity of the audio received by the multi-directional microphone (see Boykin, ¶ 0015-0017 and 0028).
Regarding claim 15, see the preceding rejection with respect to claim 14 above. The combination makes obvious the “method of claim 14, wherein the determining the geo-orientation information further comprises:
obtaining direction information about a yaw corresponding to a rotation about a vertical axis of the multi-directional microphone, a roll corresponding to a rotation about a front-back axis of the multi-directional microphone, and a pitch corresponding to a rotation about a left-right axis of the multi-directional microphone” because Benattar makes obvious a microphone array with at least 4 microphones for detecting audio signal direction in three-dimensional space (see Benattar, ¶ 0028), and makes obvious the use of a position sensor for detecting yaw, roll, and pitch information (see Benattar, ¶ 0032-0034 and 0095);
“obtaining location information about a latitude, a longitude, and an elevation of the multi-directional microphone” because GPS information is included and GPS information includes latitude, longitude, and elevation (see Boykin, ¶ 0059, 0148-0149, and 0188); and
“determining the geo-orientation information based on the direction information and the location information” because Boykin teaches the GPS location and estimated direction (see Boykin, ¶ 0148-0149, 0177, and 0233), and Benattar makes obvious three-dimensional directional information of desired sounds (see Benattar, ¶ 0080 and 0095).
Regarding claim 16, see the preceding rejection with respect to claim 14 above. The combination makes obvious the “method of claim 15, wherein the mapping the audio clip information to the metadata comprises: mapping the information about the audio clip comprising a uniform resource locator (URL) of the audio clip to the metadata, or mapping data in which the audio clip is encoded to the metadata” because Boykin teaches that the recorded audiovisual data is uploaded to network servers and is accessible through the internet, which makes obvious that a webpage, or URL, is provided for accessing the recorded data (see Boykin, ¶ 0061, 0091, 0188, 0193, and 0197).
Claim(s) 7, 12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Boykin as applied to claims 6, 11, and 18 above, and further in view of Xiong et al. (US 2022/0172700 A1 and hereafter Xiong).
Regarding claim 7, see the preceding rejection with respect to claim 6 above. Boykin anticipates the camera of claim 6, where the detected sound, such as a voice, is classified and marked with metadata, including event tags and marked start and end points (see Boykin, ¶ 0084, 0086, 0174-0175, and 0177-0181). However, Boykin does not appear to teach the features to “divide the audio based on a speaker included in the voice”.
Xiong discloses systems and methods for audio privacy in network video surveillance systems, such as detecting a human speaking and masking the human voice component from surveillance video (see Xiong, abstract). In particular, Xiong teaches the analysis of the video data and teaches event detection algorithms, such as audio detection, voice recognition, speech recognition, etc., for mapping events with tags or metadata (see Xiong, ¶ 0003-0004 and 0044 and figure 1, unit 156). Xiong further teaches identifying multiple different voices in the video and supports voice diarization (see Xiong, ¶ 0059). It would have been obvious to one of ordinary skill in the art at the time of the effective filing date to modify Boykin with the teachings of Xiong for the purpose of tracking multiple voices in a recorded video and allowing the text of a conversation to be searched for pertinent information (see Boykin, ¶ 0084, 0086, and 0175, in view of Xiong, ¶ 0044 and 0059).
Therefore, the combination of Boykin and Xiong makes obvious the “camera of claim 6, wherein the processor is further configured to control the type classification module, based on the type of the audio being the voice, to:
divide the audio based on a speaker included in the voice” because Boykin teaches a voice-to-text translation of the captured audio that can be searched for keywords (see Boykin, ¶ 0086 and 0175), and Xiong makes obvious segmenting, or dividing, the audio based on voice diarization techniques (see Xiong, ¶ 0044, 0059, and 0087), and
“map identification information from the voice to the metadata” because the combination makes obvious adding metadata including event tags and other appropriate tags for detected voices (see Boykin, ¶ 0175 and see Xiong, ¶ 0044, 0059, and 0087).
Regarding claim 12, see the preceding rejection with respect to claim 11 above. Boykin anticipates the “system of claim 11, wherein the camera is further configured to, based on the type of the audio being a voice:
generate language information of the voice and text information of the voice” by teaching voice-to-text translation of detected speech (see Boykin, ¶ 0175), [and]
“detect a pre-stored keyword included in the text information” by teaching that detected keywords are marked in the text (see Boykin, ¶ 0175).
However, Boykin does not appear to teach the features to “divide the audio based on a speaker included in the voice”. For the same reasons as stated above with respect to claim 7, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date to modify Boykin with the teachings of Xiong for the purpose of tracking multiple voices in a recorded video and allowing the text of a conversation to be searched for pertinent information (see Boykin, ¶ 0084, 0086, and 0175, in view of Xiong, ¶ 0044 and 0059).
Therefore, the combination of Boykin and Xiong makes obvious the additional features to “divide the audio based on a speaker included in the voice” because Boykin teaches a voice-to-text translation of the captured audio that can be searched for keywords (see Boykin, ¶ 0086 and 0175), and Xiong makes obvious segmenting, or dividing, the audio based on voice diarization techniques (see Xiong, ¶ 0044, 0059, and 0087), and
“map identification information from the voice to the metadata” because the combination makes obvious adding metadata including event tags and other appropriate tags for detected voices (see Boykin, ¶ 0175 and see Xiong, ¶ 0044, 0059, and 0087).
Regarding claim 19, see the preceding rejection with respect to claim 18 above. Boykin anticipates the method of claim 18, where the detected sound, such as a voice, is classified and marked with metadata (see Boykin, ¶ 0084, 0086, 0174-0175, and 0177-0181). However, Boykin does not appear to teach the features to “divide the audio based on a speaker included in the voice”. However, Boykin does not appear to teach the features to “divide the audio based on a speaker included in the voice”. For the same reasons as stated above with respect to claim 7, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date to modify Boykin with the teachings of Xiong for the purpose of tracking multiple voices in a recorded video and allowing the text of a conversation to be searched for pertinent information (see Boykin, ¶ 0084, 0086, and 0175, in view of Xiong, ¶ 0044 and 0059).
Therefore, the combination of Boykin and Xiong makes obvious the “method of claim 18, wherein the classifying the audio into a type further comprises, based on the type of the audio being a voice:
dividing the audio based on a speaker included in the voice” because Boykin teaches a voice-to-text translation of the captured audio that can be searched for keywords (see Boykin, ¶ 0086 and 0175), and Xiong makes obvious segmenting, or dividing, the audio based on voice diarization techniques (see Xiong, ¶ 0044, 0059, and 0087); and
“mapping identification information from the voice to the metadata” because the combination makes obvious adding metadata including event tags and other appropriate tags for detected voices (see Boykin, ¶ 0175 and see Xiong, ¶ 0044, 0059, and 0087).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daniel R Sellers whose telephone number is (571)272-7528. The examiner can normally be reached Mon - Fri 10:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fan S Tsang can be reached at (571)272-7547. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daniel R Sellers/ Primary Examiner, Art Unit 2694
Prosecution Insights