DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 5 and 8 to 18 are rejected under 35 U.S.C. 103 as being unpatentable over Etchart; Tyler Jerel et al. (US 20230068798 A1), hereinafter ETCHART, in view of KALINLI; OZLEM (US 20120259638 A1), hereinafter KALINLI.
Regarding claim 1, ETCHART teaches:
(Original) A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with an audio sensor and an image sensor,
ETCHART [0235] “Each of these devices (110/120/125) may include one or more controllers/processors (1804/1904), which may each include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (1806/1906) for storing data and instructions of the respective device. The memories (1806/1906) may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive memory (MRAM), and/or other types of memory. Each device (110/120/125) may also include a data storage component (1808/1908) for storing data and controller/processor-executable instructions. Each data storage component (1808/1908) may individually include one or more non-volatile storage types such as magnetic storage, optical storage, solid-state storage, etc. Each device (110/120/125) may also be connected to removable or external non-volatile memory and/or storage (such as a removable memory card, memory key drive, networked storage, etc.) through respective input/output device interfaces (1802/1902).”
ETCHART [0238] “Referring to FIG. 18, the device 110 may include input/output device interfaces 1802 that connect to a variety of components such as an audio output component such as a speaker 1812, a wired headset or a wireless headset (not illustrated), or other component capable of outputting audio. The device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 1820 or array of microphones, a wired headset or a wireless headset (not illustrated), etc. If an array of microphones is included, approximate distance to a sound's point of origin may be determined by acoustic localization based on time and amplitude differences between sounds captured by different microphones of the array. An array of microphones may also be used to perform beamforming/other techniques to determine a direction of a sound's point of origin relative to the device 110. Data from the array of microphones as well as other components may be used to track a sound's source as it moves around an environment of a device 110. The device 110 may additionally include a display 1816 for displaying content. The device 110 may further include a camera 1818."
the one or more programs including instructions for: detecting: audio data via the audio sensor;
ETCHART [0041] “The system 100 may operate using various components as described in FIG. 2. The various components may be located on same or different physical devices. Communication between various components may occur directly or across a network(s) 199. The device 110 may include audio capture component(s), such as a microphone or array of microphones of a device 110, captures audio 11 and creates corresponding audio data. Once speech is detected in audio data representing the audio 11, the device 110 may determine if the speech is directed at the device 110/system 120. In at least some embodiments, such determination may be made using a wakeword detection component 220. The wakeword detection component 220 may be configured to detect various wakewords. In at least some examples, each wakeword may correspond to a name of a different digital assistant. An example wakeword/digital assistant name is “Alexa.” In another example, input to the system may be in form of text data 213, for example as a result of a user typing an input into a user interface of device 110. Other input forms may include indication that the user has pressed a physical or virtual button on device 110, the user has made a gesture, etc. The device 110 may also capture images using camera(s) 1818 of the device 110 and may send image data 112 representing those image(s) to the system 120. The image data 112 may include raw image data or image data processed by the device 110 before sending to the system 120. The image data 112 may also include individual still images and/or a video feed of multiple images.”
and video data via the image sensor, wherein the video data represents a scene; ETCHART [0041] “The system 100 may operate using various components as described in FIG. 2. The various components may be located on same or different physical devices. Communication between various components may occur directly or across a network(s) 199. The device 110 may include audio capture component(s), such as a microphone or array of microphones of a device 110, captures audio 11 and creates corresponding audio data. Once speech is detected in audio data representing the audio 11, the device 110 may determine if the speech is directed at the device 110/system 120. In at least some embodiments, such determination may be made using a wakeword detection component 220. The wakeword detection component 220 may be configured to detect various wakewords. In at least some examples, each wakeword may correspond to a name of a different digital assistant. An example wakeword/digital assistant name is “Alexa.” In another example, input to the system may be in form of text data 213, for example as a result of a user typing an input into a user interface of device 110. Other input forms may include indication that the user has pressed a physical or virtual button on device 110, the user has made a gesture, etc. The device 110 may also capture images using camera(s) 1818 of the device 110 and may send image data 112 representing those image(s) to the system 120. The image data 112 may include raw image data or image data processed by the device 110 before sending to the system 120. The image data 112 may also include individual still images and/or a video feed of multiple images.”
ETCHART [0125] “Image matching algorithms, such as those used by image processing component 240, may take advantage of the fact that an image of an object or scene contains a number of feature points. Feature points are specific points in an image which are robust to changes in image rotation, scale, viewpoint or lighting conditions. ..."
and in response to detecting the audio data via the audio sensor and the video data via the image sensor: in accordance with a determination, based on the audio data and the video data, that the scene includes a user whose attention is directed to the electronic device while the user is speaking and that a set of initiation criteria is satisfied: determining, based on the audio data and the video data, whether the audio data includes speech that is intended for the electronic device; ETCHART [0116] “The feature vector 436 may be passed to the user detector 425. The user detector 425 (which may use various components/operations of image processing component 240, user recognition component 295, etc.) may be configured to process image data 112 and/or feature vector 436 to determine information about the user's behavior which in turn may be used to determine if an input is system directed. For example, the user detector 425 may be configured to determine the user's position/behavior with respect to device 110/system 100. The user detector 425 may also be configured to determine whether a user's mouth is opening/closing in a manner that suggests the user is speaking. The user detector 425 may also be configured to determine whether a user is nodding or shaking his/her head. The user detector 425 may also be configured to determine whether a user's gaze is directed to the device 110, to another user, or to another object. For example, the use detector 425 may include, or be configured to use data from, a gaze detector 565. The user detector 425 may also be configured to determine gestures of the user such as a shoulder shrug, pointing toward an object, a wave, a hand up to indicate an instruction to stop, or a fingers moving to indicate an instruction to continue, holding up a certain number of fingers, putting a thumb up, etc. The user detector 425 may also be configured to determine a user's position/orientation such as facing another user, facing the device 110, whether their back is turned, etc. The user detector 425 may also be configured to determine relative positions of multiple users that appear in image data (and/or are speaking in audio data 211 which may also be considered by the user detector 425 along with feature vector 431), for example which users are closer to a device 110 and which are farther away. The user detector 425 (and/or other component) may also be configured to identify other objects represented in image data and determine whether objects are relevant to a dialog or system interaction (for example determining if a user is referring to an object through a movement or speech).”
ETCHAR does not teach, but KALINLI teaches:
and in accordance with a determination, based on the audio data and the video data, that the scene does not include a user whose attention is directed to the electronic device while the user is speaking: forgoing determining whether the audio data includes speech that is intended for the electronic device.
KALINLI [0021] “After facial orientation characteristics associated with a user's speech have been obtained, the relevancy of the user's speech may be characterized according to those facial orientation characteristics as indicated at 119. By way of example, and not by way of limitation, a user's speech may be characterized as irrelevant where one or more of the facial orientation characteristics obtained falls outside of an allowed range. For example, a program may set a maximum allowable head tilt angle of 45.degree., and so any speech made outside of a 45.degree. head tilt angle will be characterized as irrelevant and discarded prior to processing. Similarly, the program may set a maximum angle of divergence from a specified target of 10.degree. for the user's eye gaze direction, and so any speech made outside of a 10.degree. divergent eye gaze direction will be characterized as irrelevant and discarded prior to processing. Relevance may also be characterized based on a combination of facial orientation characteristics. For example, speech made by a user whose head tilt angle falls outside of an allowed range, but whose eye gaze direction falls within the maximum angle of divergence may be characterized as relevant or speech made by a user whose head looks straight to the target, but whose eye gaze direction falls outside of the maximum angle of divergence may be characterized as irrelevant.”
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of ETCHART the capability to forgoing determining whether the audio data includes speech that is intended for the electronic device if the user attention is not directed to the device. The benefit and motivation of such modification is discussed by KALINLI in the following portion: KALINLI [0025] “By weighing the relevance of detected user speech prior to speech recognition processing, a system may save considerable hardware resources as well as improve the overall accuracy of speech recognition. Discarding irrelevant voice inputs decreases the workload of the processor and eliminates confusion involved with processing extraneous speech.”
Regarding claim 2, the rejection of claim 1 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 1, wherein the determination, based on the audio data and the video data, that the scene includes the user whose attention is directed to the electronic device while the user is speaking includes: a first type of determination, made by a first process, that the scene includes the user whose attention is directed to the electronic device while the user is speaking;
ETCHART [0117] “The user detector 425 may operate one or more models (e.g., one or more classifiers) to determine if certain situations are represented in the image data 112. For example the user detector 425 may employ a visual directedness classifier that may determine, for each face detected in the image data 112 whether that face is looking at the device 110 or not. For example, a light-weight convolutional neural network (CNN) may be used which takes a face image cropped from the result of the face detector as input and output a [0,1] score of how likely the face is directed to the camera or not. Another technique may include to determine a three-dimensional (3D) landmark of each face, estimate the 3D angle of the face and predict a directness score based on the 3D angle.”
and a second type of determination, made by a second process, that the scene includes the user whose attention is directed to the electronic device while the user is speaking, wherein the second type of determination is different from the first type of determination.
"ETCHART [0117] “The user detector 425 may operate one or more models (e.g., one or more classifiers) to determine if certain situations are represented in the image data 112. For example the user detector 425 may employ a visual directedness classifier that may determine, for each face detected in the image data 112 whether that face is looking at the device 110 or not. For example, a light-weight convolutional neural network (CNN) may be used which takes a face image cropped from the result of the face detector as input and output a [0,1] score of how likely the face is directed to the camera or not. Another technique may include to determine a three-dimensional (3D) landmark of each face, estimate the 3D angle of the face and predict a directness score based on the 3D angle.”
ETCHART [0119] “The user detector 425 (or other component(s) such as those in user recognition component 295) may be configured to determine whether a face represented in image data belongs to a person who is speaking or not, thus performing active speaker detection. The system may take the output from the face tracker and aggregate a sequence of face from the same person as input and predict whether this person is speaking or not. Lip motion, user ID, detected voice data, and other data may be used to determine whether a user is speaking or not."
Regarding claim 5, the rejection of claim 1 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 1, wherein the determination that the scene includes the user whose attention is directed to the electronic device while the user is speaking is based on a determination that a gaze of the user is directed to the electronic device while the user is speaking.
ETCHART [0162] “Object tracking component 560 may also include a gaze detection component 565. The gaze detection component 565 may also be located elsewhere in the system design. The gaze detection component 565 may include a classifier or other component (for example including one or more trained model(s)) that is configured to track the gaze of a user using image data and/or feature data corresponding to image data. The gaze detection component 565 may output data indicating that a user is looking at a device or looking elsewhere (for example at another user). If operated on device 110, the gaze detection component 565 may process image data (which may including raw image data captured by a camera or may include feature data representing raw image data) to determine that the user is gazing at a camera of the device. If operated on another device, the gaze detection component 565 may process image data to determine a user is looking at device 110, for example looking at device 110 while speaking an utterance. In this manner processing by a second device may be used to determine that a user is looking at a first device while speaking to the first device. Thus, for example, in a smart-home situation image data from a camera removed from a speech-capture device (e.g., device 110) may be used to determine a user was looking at the speech-capture device when speaking. Data from the gaze detector 565 may be used, for example, by system directed input detector 385.”
ETCHART [0116] “The feature vector 436 may be passed to the user detector 425. The user detector 425 (which may use various components/operations of image processing component 240, user recognition component 295, etc.) may be configured to process image data 112 and/or feature vector 436 to determine information about the user's behavior which in turn may be used to determine if an input is system directed. For example, the user detector 425 may be configured to determine the user's position/behavior with respect to device 110/system 100. The user detector 425 may also be configured to determine whether a user's mouth is opening/closing in a manner that suggests the user is speaking. The user detector 425 may also be configured to determine whether a user is nodding or shaking his/her head. The user detector 425 may also be configured to determine whether a user's gaze is directed to the device 110, to another user, or to another object. For example, the use detector 425 may include, or be configured to use data from, a gaze detector 565. ..."
Regarding claim 8, the rejection of claim 1 is incorporated, furthermore ETCHART does not teach, but KALINLI teaches:
(Original) The non-transitory computer-readable storage medium of claim 1, wherein the determination that the scene includes the user whose attention is directed to the electronic device while the user is speaking is based on a determination that a pose of the user faces the electronic device while the user is speaking.
KALINLI [0028] “By way of example and not by way of limitation, the pose of a user's head may be estimated using five reference points, the outside corners 128 of each of the eyes, the outside corners 124.sub.M of the mouth, and the tip of the nose (not shown). A facial symmetry axis may be found by connecting a line between a midpoint of the eyes (e.g., halfway between the eyes' outside corners 128) and a midpoint of the mouth (e.g., halfway between the mouth's outside corners 124.sub.M). A facial direction can be determined under weak-perspective geometry from a 3D angle of the nose. Alternatively, the same five points can be used to determine the head pose from the normal to the plane, which can be found from planar skew-symmetry and a coarse estimate of the nose position. …”
KALINLI [0018] “Once the presence of the user's face has been identified, one or more facial orientation characteristics associated with the user's face during speech can be obtained during the interval of time as indicated at 117. Again, suitably configured image analysis software may be used to analyze one or more images of the user's face to determine the facial orientation characteristics. By way of example, and not by way of limitation, one of these facial orientation characteristics may be a user's head tilt angle. The user's head tilt angle refers to the angular displacement between a user's face during speech and a face that is directed exactly at the specified target (e.g., visual display, camera, etc.). The user's head tilt angle may refer to the vertical angular displacement, horizontal angular displacement, or a combination of the two. A user's head tilt angle provides information regarding his intent during speech. In most situations, a user will directly face his target when speaking, and as such the head tilt angle at which the user is speaking will help determine who/what the target of his speech is."
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of ETCHART the capability to detect a head pose to determine the intended target of the speech. The benefit and motivation of such modification is discussed by KALINLI in the following portion: KALINLI [0014] “In embodiments of the present invention the relevancy of a given voice input may be determined without relying on a user's deliberate or conscious control over the capturing of speech. The relevance of a user's voice input may be characterized based on certain detectable cues that are given unconsciously by a speaker during speech. For example, the direction of the speaker's speech and the direction of the speaker's sight during speech may both provide tell-tale signs as to who or what is the target of the speaker's voice.”
Regarding claim 9, the rejection of claim 1 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 1, wherein the user is a first user, wherein the scene includes a second user different from the first user, and wherein the one or more programs further include instructions for: determining, based on the audio data and the video data, whether an attention of the first user is directed to the electronic device while the first user is speaking;
ETCHART [0116] “The feature vector 436 may be passed to the user detector 425. The user detector 425 (which may use various components/operations of image processing component 240, user recognition component 295, etc.) may be configured to process image data 112 and/or feature vector 436 to determine information about the user's behavior which in turn may be used to determine if an input is system directed. For example, the user detector 425 may be configured to determine the user's position/behavior with respect to device 110/system 100. The user detector 425 may also be configured to determine whether a user's mouth is opening/closing in a manner that suggests the user is speaking. The user detector 425 may also be configured to determine whether a user is nodding or shaking his/her head. The user detector 425 may also be configured to determine whether a user's gaze is directed to the device 110, to another user, or to another object. For example, the use detector 425 may include, or be configured to use data from, a gaze detector 565. The user detector 425 may also be configured to determine gestures of the user such as a shoulder shrug, pointing toward an object, a wave, a hand up to indicate an instruction to stop, or a fingers moving to indicate an instruction to continue, holding up a certain number of fingers, putting a thumb up, etc. The user detector 425 may also be configured to determine a user's position/orientation such as facing another user, facing the device 110, whether their back is turned, etc. The user detector 425 may also be configured to determine relative positions of multiple users that appear in image data (and/or are speaking in audio data 211 which may also be considered by the user detector 425 along with feature vector 431), for example which users are closer to a device 110 and which are farther away. The user detector 425 (and/or other component) may also be configured to identify other objects represented in image data and determine whether objects are relevant to a dialog or system interaction (for example determining if a user is referring to an object through a movement or speech).”
ETCHART [0117] “The user detector 425 may operate one or more models (e.g., one or more classifiers) to determine if certain situations are represented in the image data 112. For example the user detector 425 may employ a visual directedness classifier that may determine, for each face detected in the image data 112 whether that face is looking at the device 110 or not. For example, a light-weight convolutional neural network (CNN) may be used which takes a face image cropped from the result of the face detector as input and output a [0,1] score of how likely the face is directed to the camera or not. Another technique may include to determine a three-dimensional (3D) landmark of each face, estimate the 3D angle of the face and predict a directness score based on the 3D angle.”
ETCHART [0067] “The user recognition component 295 determines scores indicating whether user input originated from a particular user. For example, a first score may indicate a likelihood that the user input originated from a first user, a second score may indicate a likelihood that the user input originated from a second user, etc. The user recognition component 295 also determines an overall confidence regarding the accuracy of user recognition operations. The user recognition component 295 may also be configured to determine (or assist another component in determining) that a particular voice matches a particular face for purposes of user identification and/or following a user in an environment if the user is not visible in image data)."
and determining, based on the audio data and the video data, whether an attention of the second user is directed to the electronic device while the second user is speaking. ETCHART [0201] “The system 100 may perform active speaker detection 1254 to the selected image data 1275 to generate label data, illustrated in FIG. 12B as label graph 1280. For example, the system 100 may perform the active speaker detection 1254 to determine first measurement data corresponding to a first user 5a and second measurement data corresponding to a second user 5b. The system 100 may determine a first variation associated with the first measurement data, determine a second variation associated with the second measurement data, and then compare the first variation to the second variation. Based on this comparison, the system 100 may generate the label data indicating whether the first user 5a is speaking or the second user 5b is speaking.”
ETCHART [0116] “The feature vector 436 may be passed to the user detector 425. The user detector 425 (which may use various components/operations of image processing component 240, user recognition component 295, etc.) may be configured to process image data 112 and/or feature vector 436 to determine information about the user's behavior which in turn may be used to determine if an input is system directed. For example, the user detector 425 may be configured to determine the user's position/behavior with respect to device 110/system 100. The user detector 425 may also be configured to determine whether a user's mouth is opening/closing in a manner that suggests the user is speaking. The user detector 425 may also be configured to determine whether a user is nodding or shaking his/her head. The user detector 425 may also be configured to determine whether a user's gaze is directed to the device 110, to another user, or to another object. For example, the use detector 425 may include, or be configured to use data from, a gaze detector 565. ..."
Regarding claim 10, the rejection of claim 9 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 9, wherein: determining whether the attention of the first user is directed to the electronic device while the first user is speaking includes: cropping the video data to obtain a portion of the video data that represents a face of the first user;
ETCHART [0117] “The user detector 425 may operate one or more models (e.g., one or more classifiers) to determine if certain situations are represented in the image data 112. For example the user detector 425 may employ a visual directedness classifier that may determine, for each face detected in the image data 112 whether that face is looking at the device 110 or not. For example, a light-weight convolutional neural network (CNN) may be used which takes a face image cropped from the result of the face detector as input and output a [0,1] score of how likely the face is directed to the camera or not. Another technique may include to determine a three-dimensional (3D) landmark of each face, estimate the 3D angle of the face and predict a directness score based on the 3D angle.”
and determining whether the attention of the second user is directed to the electronic device while the second user is speaking includes: cropping the video data to obtain a portion of the video data that represents a face of the second user.
ETCHART [0117] “The user detector 425 may operate one or more models (e.g., one or more classifiers) to determine if certain situations are represented in the image data 112. For example the user detector 425 may employ a visual directedness classifier that may determine, for each face detected in the image data 112 whether that face is looking at the device 110 or not. For example, a light-weight convolutional neural network (CNN) may be used which takes a face image cropped from the result of the face detector as input and output a [0,1] score of how likely the face is directed to the camera or not. Another technique may include to determine a three-dimensional (3D) landmark of each face, estimate the 3D angle of the face and predict a directness score based on the 3D angle.”
ETCHART [0119] “The user detector 425 (or other component(s) such as those in user recognition component 295) may be configured to determine whether a face represented in image data belongs to a person who is speaking or not, thus performing active speaker detection. The system may take the output from the face tracker and aggregate a sequence of face from the same person as input and predict whether this person is speaking or not. Lip motion, user ID, detected voice data, and other data may be used to determine whether a user is speaking or not.”
ETCHART [0159] “FIG. 6F illustrates another advantage of using an approach such as that described with respect to FIG. 6B to determine the position of various features on a user's face. In this example, it can be seen that the features of a head of a second user 604 have a different relative position and separation. Thus, a computing device also can not only determine positions and changes of facial features for a specific user, but can distinguish between different users. Also, the device can be configured to detect how close a user is to the device based on, for example, the amount and ratio of separation of various features 625, such that the device can detect movement towards, and away from, the device. This can help to improve the accuracy of facial tracking."
Regarding claim 11, the rejection of claim 9 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 9, wherein the one or more programs further include instructions for: selecting, from the first user and the second user, the first user, wherein the set of initiation criteria is satisfied when the first user is selected. "ETCHART [0030] In some examples, the system may be configured to process audio data to detect an utterance and determine a beginning and/or ending of the utterance. To improve a user experience, the system may perform active speaker detection to determine which user is speaking based on the image data. Using the techniques described above, the system may process a portion of the image data corresponding to the utterance to determine an amount of variation in mouth movements for each face represented in the image data. For example, the system may compare a first amount of variation associated with a first user to a second amount of variation associated with a second user in order to determine whether the first user is speaking or the second user is speaking.
ETCHART [0077] The device 110 may also include a system directed input detector 385. (The system 120 may also include a system directed input detector 285 which may operate in a manner similar to system directed input detector 385.) The system directed input detector 385 may be configured to determine whether an input to the system (for example speech, a gesture, etc.) is directed to the system or not directed to the system (for example directed to another user, etc.). The system directed input detector 385 may work in conjunction with the wakeword detector 220. If the system directed input detector 385 determines an input is directed to the system, the device 110 may “wake” and begin sending captured data for further processing (for example, processing audio data using the language processing 292/392, processing captured image data using image processing component 240/340 or the like). If data is being processed the device 110 may indicate such to the user, for example by activating or changing the color of an illuminated output (such as an LED ring), displaying an indicator on a display (such as a light bar across the display), outputting an audio indicator (such as a beep) or otherwise informing a user that input data is being processed. If the system directed input detector 385 determines an input is not directed to the system (such as a speech or gesture directed to another user) the device 110 may discard the data and take no further action for processing purposes. In this way the system 100 may prevent processing of data not directed to the system, thus protecting the users' privacy. As an indicator to the user, however, the system may output an audio, visual, or other indicator when the system directed input detector 385 is determining whether an input is potentially device directed. For example, the system may output an orange indicator while considering an input, and may output a green indicator if a system directed input is detected. Other such configurations are possible."
Regarding claim 12, the rejection of claim 1 is incorporated, furthermore ETCHART teaches:
(Currently Amended) The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further include instructions for: determining a start time of when the attention is directed to the electronic device while the user is speaking.
ETCHART [0035] “The system 100 may use computer vision (CV) techniques operating on image data to perform active speaker detection. The system 100 may thus use image data to determine when a user 5 is speaking and/or which user is speaking. The system 100 may use face detection techniques to detect a human face represented in image data (for example using object detection component 530 as discussed below). The system 100 may use a classifier or other model configured to determine whether a face is looking at a device 110 (for example using object tracking component 560 as discussed below). The system 100 may also be configured to track a face in image data to understand which faces in the video are belonging to the same person and where they may be located in image data and/or relative to a device 110 (for example using user recognition component 295 and/or object tracking component 560 as discussed below). The system 100 may also be configured to determine an active speaker, for example by determining which face(s) in image data belong to the same person and whether the person is speaking or not (for example using image data of a user's lips to see if they are moving and matching such image data to data regarding a user's voice and/or audio data of speech and whether the words of the speech match the lip movement). The system 100 may use components such as user recognition component 295, object tracking component 560, and/or other components to perform such operations.”
ETCHART [0194] “Using the label data, the system 100 may determine an utterance boundary 1225, such as identifying a beginning and/or end of an utterance. In some examples, the system 100 may perform the active speaker detection 1202 to determine a beginning of the utterance and may perform other techniques to detect the ending of the utterance without departing from the disclosure. For example, the system 100 may generate the label data (e.g., label graph 1220), detect a beginning of the utterance based on the label data (e.g., first portion of the label data corresponding to “Begin” as illustrated in FIG. 12A), and generate a first utterance boundary 1225a indicating that an utterance has begun. In response to the first utterance boundary 1225a, the system 100 may generate audio data corresponding to the utterance and perform audio processing to determine an end of the utterance.”
ETCHART [0195] “In other examples, the system 100 may detect a beginning of the utterance (e.g., using audio processing and/or active speaker detection 1202) at a first time and then separately perform the active speaker detection 1202 to determine an ending of the utterance. For example, while generating the audio data corresponding to the utterance, the system 100 may generate the label data (e.g., label graph 1220), detect an ending of the utterance based on the label data (e.g., second portion of the label data corresponding to “End” as illustrated in FIG. 12A), and generate a second utterance boundary 1225b indicating that the utterance has ended. In response to the second utterance boundary 1225b, the system 100 may stop generating the audio data and may perform additional processing to detect a voice command represented in the audio data."
Regarding claim 13, the rejection of claim 12 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 12, wherein determining, based on the audio data and the video data, whether the audio data includes speech that is intended for the electronic device includes:determining whether the audio data includes speech that is intended for the electronic device based on a portion of the audio data that is identified based on the start time.
ETCHART [0114] “As shown in FIG. 4, the system directed input detector 285/385 may simply user audio data to determine whether an input is system directed (for example, system directed audio detector 440 may output an SDD result 442). This may be true particularly when no image data is available (for example for a device without a camera). If image data 112 is available, however, the system may also be configured to use image data 112 to determine if an input is system directed. The image data 112 may include image data captured by device 110 and/or image data captured by other device(s) in the environment of device 110. The audio data 211, image data 112 and other data 481 may be timestamped or otherwise correlated so that the system directed input detector 285/385 may determine that the data being analyzed all relates to a same time window so as to ensure alignment of data considered with regard to whether a particular input is system directed. For example, the system directed input detector 285/385 may determine system directedness scores for every frame of audio data/every image of a video stream and may align and/or window them to determine a single overall score for a particular input that corresponds to a group of audio frames/images.”
ETCHART [0193] “As illustrated in FIG. 12A, the frame-based active speaker detection 1200 begins with the system 100 performing active speaker detection 1202 on image data 1210. For example, the system 100 may perform the steps described above with regard to FIG. 7 to generate label data corresponding to the image data 1210, illustrated as label graph 1220. In some examples, the system 100 may generate an individual label for each video frame in the image data 1210, although the disclosure is not limited thereto. The label graph 1220 illustrates an example in which the system 100 determines label data that includes a first label (e.g., Label0=No Speech) for video frames that are not associated with an utterance and a second label (e.g., Label1=Speech) for video frames that are associated with an utterance.”
ETCHART [0194] “Using the label data, the system 100 may determine an utterance boundary 1225, such as identifying a beginning and/or end of an utterance. In some examples, the system 100 may perform the active speaker detection 1202 to determine a beginning of the utterance and may perform other techniques to detect the ending of the utterance without departing from the disclosure. For example, the system 100 may generate the label data (e.g., label graph 1220), detect a beginning of the utterance based on the label data (e.g., first portion of the label data corresponding to “Begin” as illustrated in FIG. 12A), and generate a first utterance boundary 1225a indicating that an utterance has begun. In response to the first utterance boundary 1225a, the system 100 may generate audio data corresponding to the utterance and perform audio processing to determine an end of the utterance."
Regarding claim 14, the rejection of claim 12 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 12, wherein determining, based on the audio data and the video data, whether the audio data includes speech that is intended for the electronic device includes: determining whether the audio data includes speech that is intended for the electronic device based on a portion of the video data that is identified based on the start time.
ETCHART [0194] “Using the label data, the system 100 may determine an utterance boundary 1225, such as identifying a beginning and/or end of an utterance. In some examples, the system 100 may perform the active speaker detection 1202 to determine a beginning of the utterance and may perform other techniques to detect the ending of the utterance without departing from the disclosure. For example, the system 100 may generate the label data (e.g., label graph 1220), detect a beginning of the utterance based on the label data (e.g., first portion of the label data corresponding to “Begin” as illustrated in FIG. 12A), and generate a first utterance boundary 1225a indicating that an utterance has begun. In response to the first utterance boundary 1225a, the system 100 may generate audio data corresponding to the utterance and perform audio processing to determine an end of the utterance.”
ETCHART [0200] “After generating the utterance boundary 1265, the system 100 may perform active speaker detection 1254 to determine which user was speaking during the time window indicated by the utterance boundary 1265. For example, the system 100 may process input image data 1270 to generate selected image data 1275 that corresponds to the utterance boundary 1265. In some examples, the utterance boundary 1265 may only indicate a beginning of the utterance and the system 100 may continue generating the selected image data 1275 until the system 100 detects an ending of the utterance. However, the disclosure is not limited thereto and in other examples the utterance boundary 1265 may indicate both a beginning and an ending of the utterance and the system 100 may generate the selected image data 1275 corresponding to the entire utterance."
Regarding claim 15, the rejection of claim 1 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further include instructions for: in response to detecting the audio data via the audio sensor and the video data via the image sensor: in accordance with a determination, based on the audio data and the video data, that the scene includes the user whose attention is directed to the electronic device while the user is speaking: outputting an indication;
ETCHART [0077] “The device 110 may also include a system directed input detector 385. (The system 120 may also include a system directed input detector 285 which may operate in a manner similar to system directed input detector 385.) The system directed input detector 385 may be configured to determine whether an input to the system (for example speech, a gesture, etc.) is directed to the system or not directed to the system (for example directed to another user, etc.). The system directed input detector 385 may work in conjunction with the wakeword detector 220. If the system directed input detector 385 determines an input is directed to the system, the device 110 may “wake” and begin sending captured data for further processing (for example, processing audio data using the language processing 292/392, processing captured image data using image processing component 240/340 or the like). If data is being processed the device 110 may indicate such to the user, for example by activating or changing the color of an illuminated output (such as an LED ring), displaying an indicator on a display (such as a light bar across the display), outputting an audio indicator (such as a beep) or otherwise informing a user that input data is being processed. If the system directed input detector 385 determines an input is not directed to the system (such as a speech or gesture directed to another user) the device 110 may discard the data and take no further action for processing purposes. In this way the system 100 may prevent processing of data not directed to the system, thus protecting the users' privacy. As an indicator to the user, however, the system may output an audio, visual, or other indicator when the system directed input detector 385 is determining whether an input is potentially device directed. For example, the system may output an orange indicator while considering an input, and may output a green indicator if a system directed input is detected. Other such configurations are possible.”
and in accordance with a determination, based on the audio data and the video data, that the scene does not include a user whose attention is directed to the electronic device while the user is speaking: forgoing outputting the indication.
ETCHART [0077] “The device 110 may also include a system directed input detector 385. (The system 120 may also include a system directed input detector 285 which may operate in a manner similar to system directed input detector 385.) The system directed input detector 385 may be configured to determine whether an input to the system (for example speech, a gesture, etc.) is directed to the system or not directed to the system (for example directed to another user, etc.). The system directed input detector 385 may work in conjunction with the wakeword detector 220. If the system directed input detector 385 determines an input is directed to the system, the device 110 may “wake” and begin sending captured data for further processing (for example, processing audio data using the language processing 292/392, processing captured image data using image processing component 240/340 or the like). If data is being processed the device 110 may indicate such to the user, for example by activating or changing the color of an illuminated output (such as an LED ring), displaying an indicator on a display (such as a light bar across the display), outputting an audio indicator (such as a beep) or otherwise informing a user that input data is being processed. If the system directed input detector 385 determines an input is not directed to the system (such as a speech or gesture directed to another user) the device 110 may discard the data and take no further action for processing purposes. In this way the system 100 may prevent processing of data not directed to the system, thus protecting the users' privacy. As an indicator to the user, however, the system may output an audio, visual, or other indicator when the system directed input detector 385 is determining whether an input is potentially device directed. For example, the system may output an orange indicator while considering an input, and may output a green indicator if a system directed input is detected. Other such configurations are possible.”
Regarding claim 16, the rejection of claim 1 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further include instructions for: in accordance with a determination, based on the audio data and the video data, that the audio data includes speech that is intended for the electronic device: initiating a task based on the speech that is intended for the electronic device;
ETCHART [0119] “The user detector 425 (or other component(s) such as those in user recognition component 295) may be configured to determine whether a face represented in image data belongs to a person who is speaking or not, thus performing active speaker detection. The system may take the output from the face tracker and aggregate a sequence of face from the same person as input and predict whether this person is speaking or not. Lip motion, user ID, detected voice data, and other data may be used to determine whether a user is speaking or not.”
ETCHART [0120] “The system directed image detector 450 may then determine, based on information from the user detector 425 as based on the image data whether an input relating to the image data is system directed. The system directed image detector 450 may also operate on other input data, for example image data including raw image data 112, image data including feature data 436 based on raw image data, other data 481, or other data. The determination by the system directed image detector 450 may result in a score indicating whether the input is system directed based on the image data. If no audio data is available, the indication may be output as SDD result 442. If audio data is available, the indication may be sent to system directed detector 470 which may consider information from both system directed audio detector 440 and system directed image detector 450. The system directed detector 470 may then process the data from both system directed audio detector 440 and system directed image detector 450 to come up with an overall determination as to whether an input was system directed, which may be output as SDD result 442.... “
ETCHART [0123] “If an input is determined to be system directed, the data related to the input may be sent to downstream components for further processing (e.g., to language processing 292). If an input is determined not to be system directed, the system may take no further action regarding the data related to the input and may allow it to be deleted. In certain configurations, to maintain privacy, the operations to determine whether an input is system directed are performed by device 110 (or home server(s) 120) and only if the input is determined to be system directed is further data (such as audio data 211 or image data 112) sent to a remote system 120 that is outside a user's home or other direct control.”
ETCHART [0040] “When the system 100 determines that the user is speaking (e.g., detects an utterance), the system 100 may generate (168) first audio data representing the utterance, may perform (170) speech processing on the first audio data to determine a voice command, and may cause an action to be performed based on the voice command."
and providing an output indicative of the initiated task.
ETCHART [0074] “In at least some embodiments, the system 120 may receive the audio data 211 from the device 110, to recognize speech corresponding to a spoken input in the received audio data 211, and to perform functions in response to the recognized speech. In at least some embodiments, these functions involve sending directives (e.g., commands), from the system 120 to the device 110 (and/or other devices 110) to cause the device 110 to perform an action, such as output an audible response to the spoken input via a loudspeaker(s), and/or control secondary devices in the environment by sending a control command to the secondary devices.”
ETCHART [0040] “When the system 100 determines that the user is speaking (e.g., detects an utterance), the system 100 may generate (168) first audio data representing the utterance, may perform (170) speech processing on the first audio data to determine a voice command, and may cause an action to be performed based on the voice command."
ETCHART [0049] “The speech processing system 292 may further include a NLU component 260. The NLU component 260 may receive the text data from the ASR component. The NLU component 260 may attempts to make a semantic interpretation of the phrase(s) or statement(s) represented in the text data input therein by determining one or more meanings associated with the phrase(s) or statement(s) represented in the text data. The NLU component 260 may determine an intent representing an action that a user desires be performed and may determine information that allows a device (e.g., the device 110, the system(s) 120, a skill component 290, a skill system(s) 125, etc.) to execute the intent. For example, if the text data corresponds to “play the 5.sup.th Symphony by Beethoven,” the NLU component 260 may determine an intent that the system output music and may identify “Beethoven” as an artist/composer and “5th Symphony” as the piece of music to be played. For further example, if the text data corresponds to “what is the weather,” the NLU component 260 may determine an intent that the system output weather information associated with a geographic location of the device 110. ..."
Regarding claim 17, the rejection of claim 16 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 16, wherein the one or more programs further include instructions for: identifying, based on the audio data and/or the video data, the user, wherein the output is personalized for the identified user.
ETCHART [0063] “Upon receipt by the system(s) 120, the image data 112 may be sent to an orchestrator component 230. The orchestrator component 230 may send the image data 112 to an image processing component 240. The image processing component 240 can perform computer vision functions such as object recognition, modeling, reconstruction, etc. For example, the image processing component 240 may detect a person, face, etc. (which may then be identified using user recognition component 295). The image processing component 240 is described in greater detail below with regard to FIG. 5. The device 110 may also include an image processing component 340 which operates similarly to image processing component 240.”
ETCHART [0069] “The system 100 (either on device 110, system 120, or a combination thereof) may include profile storage for storing a variety of information related to individual users, groups of users, devices, etc. that interact with the system. As used herein, a “profile” refers to a set of data associated with a user, group of users, device, etc. The data of a profile may include preferences specific to the user, device, etc.; input and output capabilities of the device; internet connectivity information; user bibliographic information; subscription information, as well as other information.”
ETCHART [0072] “The profile storage 270 may include one or more device profiles. Each device profile may be associated with a different device identifier. Each device profile may include various device identifying information. Each device profile may also include one or more user identifiers, representing one or more users associated with the device. For example, a household device's profile may include the user identifiers of users of the household.”
ETCHART [0119] “The user detector 425 (or other component(s) such as those in user recognition component 295) may be configured to determine whether a face represented in image data belongs to a person who is speaking or not, thus performing active speaker detection. The system may take the output from the face tracker and aggregate a sequence of face from the same person as input and predict whether this person is speaking or not. Lip motion, user ID, detected voice data, and other data may be used to determine whether a user is speaking or not.”
ETCHART [0067] “The user recognition component 295 determines scores indicating whether user input originated from a particular user. For example, a first score may indicate a likelihood that the user input originated from a first user, a second score may indicate a likelihood that the user input originated from a second user, etc. The user recognition component 295 also determines an overall confidence regarding the accuracy of user recognition operations. The user recognition component 295 may also be configured to determine (or assist another component in determining) that a particular voice matches a particular face for purposes of user identification and/or following a user in an environment if the user is not visible in image data)."
Regarding claim 18, the rejection of claim 1 is incorporated, furthermore ETCHART teaches:
(Original) The non-transitory computer-readable storage medium of claim 1, wherein the audio data is determined to include speech that is intended for the electronic device without detecting a spoken trigger in the audio data.
ETCHART [0077] The device 110 may also include a system directed input detector 385. (The system 120 may also include a system directed input detector 285 which may operate in a manner similar to system directed input detector 385.) The system directed input detector 385 may be configured to determine whether an input to the system (for example speech, a gesture, etc.) is directed to the system or not directed to the system (for example directed to another user, etc.). The system directed input detector 385 may work in conjunction with the wakeword detector 220. If the system directed input detector 385 determines an input is directed to the system, the device 110 may “wake” and begin sending captured data for further processing (for example, processing audio data using the language processing 292/392, processing captured image data using image processing component 240/340 or the like). If data is being processed the device 110 may indicate such to the user, for example by activating or changing the color of an illuminated output (such as an LED ring), displaying an indicator on a display (such as a light bar across the display), outputting an audio indicator (such as a beep) or otherwise informing a user that input data is being processed. If the system directed input detector 385 determines an input is not directed to the system (such as a speech or gesture directed to another user) the device 110 may discard the data and take no further action for processing purposes. In this way the system 100 may prevent processing of data not directed to the system, thus protecting the users' privacy. As an indicator to the user, however, the system may output an audio, visual, or other indicator when the system directed input detector 385 is determining whether an input is potentially device directed. For example, the system may output an orange indicator while considering an input, and may output a green indicator if a system directed input is detected. Other such configurations are possible.
Regarding claim 19, arguments analogous to the rejection of claim 1 are applicable, furthermore ETCHART teaches:
(Original) An electronic device, comprising: one or more processors; a memory; an audio sensor; an image sensor; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
ETCHART [0245] “Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture such as a memory device or non-transitory computer readable storage medium. The computer readable storage medium may be readable by a computer and may comprise instructions for causing a computer or other device to perform processes described in the present disclosure. The computer readable storage medium may be implemented by a volatile computer memory, non-volatile computer memory, hard drive, solid-state memory, flash drive, removable disk, and/or other media. In addition, components of system may be implemented as in firmware or hardware.”
ETCHART [0238] “Referring to FIG. 18, the device 110 may include input/output device interfaces 1802 that connect to a variety of components such as an audio output component such as a speaker 1812, a wired headset or a wireless headset (not illustrated), or other component capable of outputting audio. The device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 1820 or array of microphones, a wired headset or a wireless headset (not illustrated), etc. If an array of microphones is included, approximate distance to a sound's point of origin may be determined by acoustic localization based on time and amplitude differences between sounds captured by different microphones of the array. An array of microphones may also be used to perform beamforming/other techniques to determine a direction of a sound's point of origin relative to the device 110. Data from the array of microphones as well as other components may be used to track a sound's source as it moves around an environment of a device 110. The device 110 may additionally include a display 1816 for displaying content. The device 110 may further include a camera 1818.”
Regarding claim 20, arguments analogous to the rejection of claim 1 are applicable, furthermore ETCHART teaches:
(Original) A method, comprising: at an electronic device with one or more processors, memory, an audio sensor, and an image sensor:
ETCHART [0245] “Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture such as a memory device or non-transitory computer readable storage medium. The computer readable storage medium may be readable by a computer and may comprise instructions for causing a computer or other device to perform processes described in the present disclosure. The computer readable storage medium may be implemented by a volatile computer memory, non-volatile computer memory, hard drive, solid-state memory, flash drive, removable disk, and/or other media. In addition, components of system may be implemented as in firmware or hardware.”
ETCHART [0238] “Referring to FIG. 18, the device 110 may include input/output device interfaces 1802 that connect to a variety of components such as an audio output component such as a speaker 1812, a wired headset or a wireless headset (not illustrated), or other component capable of outputting audio. The device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 1820 or array of microphones, a wired headset or a wireless headset (not illustrated), etc. If an array of microphones is included, approximate distance to a sound's point of origin may be determined by acoustic localization based on time and amplitude differences between sounds captured by different microphones of the array. An array of microphones may also be used to perform beamforming/other techniques to determine a direction of a sound's point of origin relative to the device 110. Data from the array of microphones as well as other components may be used to track a sound's source as it moves around an environment of a device 110. The device 110 may additionally include a display 1816 for displaying content. The device 110 may further include a camera 1818.”
Claims 3 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over ETCHART in view of KALINLI in further view of Singer; Steven Mark et al. (US 20130223635 A1.), hereinafter SINGER.
Regarding claim 3, the rejection of claim 2 is incorporated, furthermore ETCHART in view of KALINLI does not teach but SINGER teaches:
(Original) The non-transitory computer-readable storage medium of claim 2, wherein: the first process consumes less processing power than the second process;
SINGER [Abstract] “Devices and methods of detecting a predetermined audio signal in audio signals are provided. A device includes a processor coupled to a clock signal generator, a power controller and an audio detector. The power controller controls a clock rate provided to the processor by the clock signal generator, to control the device to operate in a low power mode having a relatively low power consumption or in a normal power mode having a relatively high power consumption. The audio detector receives audio signals and detects, in the low power mode, probable presence of a predetermined audio signal in the audio signals. The power controller controls the device to switch from the low power mode to the normal power mode responsive to the detected presence of the predetermined audio signal by the audio detector.”
and the first type of determination has a lower accuracy than the second type of determination.
SINGER [0018] “According to an exemplary embodiment, the device may provide more than one level of audio processing, with the audio detector detecting, in the low power mode, the probable presence of the predetermined signal and a DSP detecting, in the normal power mode, the predetermined signal. Thus, the audio detector may perform detection with a lower accuracy with reduced power consumption (under the low power mode) while the DSP may perform higher accuracy detection with higher power consumption (under the normal power mode), responsive to the audio detector.”
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of ETCHART in view of KALINLI the capability to have two different power levels, where one low power mode consist of slower processing power and lower accuracy power efficient detection. The benefit and motivation of such modification is discussed by SINGER in the following portion: SINGER [0013] “Various devices may include a low power mode and a normal power mode. In the low power mode, the energy consumption is typically reduced (compared to the normal power mode) by disabling some of the functions of the device. The low power mode may be useful, for example, for battery-powered devices.”
Regarding claim 4, the rejection of claim 2 is incorporated, furthermore ETCHART teaches:
ETCHART [0162] “Object tracking component 560 may also include a gaze detection component 565. The gaze detection component 565 may also be located elsewhere in the system design. The gaze detection component 565 may include a classifier or other component (for example including one or more trained model(s)) that is configured to track the gaze of a user using image data and/or feature data corresponding to image data. The gaze detection component 565 may output data indicating that a user is looking at a device or looking elsewhere (for example at another user). If operated on device 110, the gaze detection component 565 may process image data (which may including raw image data captured by a camera or may include feature data representing raw image data) to determine that the user is gazing at a camera of the device. If operated on another device, the gaze detection component 565 may process image data to determine a user is looking at device 110, for example looking at device 110 while speaking an utterance. In this manner processing by a second device may be used to determine that a user is looking at a first device while speaking to the first device. Thus, for example, in a smart-home situation image data from a camera removed from a speech-capture device (e.g., device 110) may be used to determine a user was looking at the speech-capture device when speaking. Data from the gaze detector 565 may be used, for example, by system directed input detector 385.”
ETCHART in view of KALINLI does not teach but SINGER teaches:
(Original) The non-transitory computer-readable storage medium of claim 2, wherein the second process is initiated in response to the first type of determination, made by the first process,
SINGER [Abstract] “Devices and methods of detecting a predetermined audio signal in audio signals are provided. A device includes a processor coupled to a clock signal generator, a power controller and an audio detector. The power controller controls a clock rate provided to the processor by the clock signal generator, to control the device to operate in a low power mode having a relatively low power consumption or in a normal power mode having a relatively high power consumption. The audio detector receives audio signals and detects, in the low power mode, probable presence of a predetermined audio signal in the audio signals. The power controller controls the device to switch from the low power mode to the normal power mode responsive to the detected presence of the predetermined audio signal by the audio detector.”
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of ETCHART in view of KALINLI the capability to have two different power levels, where the low power mode determines if the normal mode needs to be engaged, based on the speaker detection disclosed by ETCHART. The benefit and motivation of such modification is discussed by SINGER in the following portion: SINGER [0013] “Various devices may include a low power mode and a normal power mode. In the low power mode, the energy consumption is typically reduced (compared to the normal power mode) by disabling some of the functions of the device. The low power mode may be useful, for example, for battery-powered devices.”
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over ETCHART in view of KALINLI in further view of MAEHARA AKIRA et al. (JP 2023139924 A), hereinafter MAEHARA.
Regarding claim 6, the rejection of claim 5 is incorporated, furthermore ETCHART in view of KALINLI does not teach but MAEHARA teaches:
(Original) The non-transitory computer-readable storage medium of claim 5, wherein the determination that the gaze of the user is directed to the electronic device while the user is speaking is based on: a first gaze tracking process that is selected based on a first distance between the user and the electronic device;
MAEHARA [0005] “An occupant state recognition device according to one aspect of the present invention is an occupant state recognition device that recognizes the occupant state by determining the visual state of the occupant of a vehicle, and comprises: an occupant information acquisition unit that acquires the occupant's face orientation and gaze direction based on imaging information from an in-vehicle camera of the vehicle; a distance acquisition unit that acquires the distance from the in-vehicle camera to the occupant's head based on distance measurement information from an in-vehicle sensor of the vehicle; and a visual state determination unit that determines the occupant's visual state based on the distance and at least one of the face orientation and gaze direction, wherein the visual state determination unit determines the occupant's visual state based on the gaze direction when the distance is less than or equal to a predetermined long-distance threshold for acquiring the gaze direction, and determines the occupant's visual state based on the face orientation when the distance is greater than the long-distance threshold.”
And a second gaze tracking process that is selected based on a second distance between the user and the electronic device, wherein the first distance is different from the second distance, and wherein the first gaze tracking process is different from the second gaze tracking process. MAEHARA [0009] “Another aspect of the present invention relates to an occupant state recognition method, which recognizes the occupant state by determining the visual state of the occupant of a vehicle. The method involves acquiring the occupant's face orientation and gaze direction based on imaging information from an in-vehicle camera, acquiring the distance from the in vehicle camera to the occupant's head based on distance measurement information from an in-vehicle sensor, and performing a visual state determination to determine the occupant's visual state based on the distance and at least one of the face orientation and gaze direction. In the visual state determination, if the distance is less than or equal to a predetermined long distance threshold for acquiring the gaze direction, the occupant's visual state is determined based on the gaze direction, and if the distance is greater than the long-distance threshold, the occupant's visual state is determined based on the face orientation.”
MAEHARA [0025] “The occupant information acquisition unit 11 uses the captured image of the occupant taken by the vehicle's driver monitor camera 2 to acquire the occupant's line of sight. ""Direction of gaze"" refers to the direction in which the crew members' eyes are directed. The direction of the occupant's gaze can be obtained by known image processing techniques, such as using the relative positional relationships of multiple points set relative to the occupant's eyes in the image captured by the driver monitor camera 2. Multiple points may include reference points set at the inner corner of the eye, etc., and moving points set at the iris, etc. A reference point is a point that serves as the reference for the relative position of a moving point. The crew information acquisition unit 11 may acquire angle information in the line of sight direction for both the yaw angle direction and the pitch angle direction.”
MAEHARA [0024] “The occupant information acquisition unit 11 acquires the orientation of the occupant's face using the image captured by the vehicle's driver monitor camera 2. ""Face direction"" refers to the direction that the crew members' faces are actually pointing. The orientation of the occupant's face can be obtained, for example, by using the arrangement of the eyes, nose, and mouth in the region corresponding to the occupant's face in the image captured by the driver monitor camera 2, using known image processing techniques such as pattern matching. The crew information acquisition unit 11 may acquire angle information of the direction of the crew's face in both the yaw angle direction and the pitch angle direction."
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of ETCHART in view of KALINLI the capability to detect the gaze or line of sight of the user with two different processes depending on the distance between the user and the device. The benefit and motivation of such modification is discussed by MAEHARA in the following portion: MAEHARA [0006] “In one aspect of the present invention, in a passenger state recognition device, if the distance from the in-vehicle camera to the passenger's head is large and the accuracy of the imaging information from the in-vehicle camera affects the accuracy of acquiring the direction of gaze, the passenger's visual state is determined based on the orientation of the face rather than the direction of gaze. Therefore, according to one aspect of the present invention, the occupant's visual state can be determined by switching whether or not to use the direction of gaze depending on the accuracy of the imaging information from the in-vehicle camera.”
Regarding claim 7, the rejection of claim 6 is incorporated, furthermore ETCHART in view of KALINLI does not teach but MAEHARA teaches:
(Original) The non-transitory computer-readable storage medium of claim 6, wherein: the first gaze tracking process tracks a first respective user gaze with a first precision level; MAEHARA [0007] “In one embodiment, the visual status determination unit calculates the reliability of acquiring the line of sight direction based on the imaging information, and when the distance is greater than the long-distance threshold and the reliability is equal to or greater than a predetermined reliability threshold, the visual status of the occupant may be determined based on the line of sight direction. Thus, even if the distance from the in-vehicle camera to the occupant's head is large, if the reliability of acquiring the direction of gaze is above the confidence threshold, the occupant's visual state is determined based on the direction of gaze. Therefore, by appropriately utilizing the direction of gaze according to the accuracy of the image information captured by the in-vehicle camera, the occupant's visual state can be determined.”
MAEHARA [0034] “The visual state determination unit 13 may calculate the reliability of acquiring the line of sight direction based on the imaging information. ""Reliability"" refers to an index that indicates whether the direction of gaze can be obtained from the image captured by the driver monitor camera 2 with sufficient accuracy to appropriately determine the occupant's visual condition. ""Reliability"" is one of the indicators used to determine whether it is appropriate to use the direction of the crew member's line of sight when assessing the crew member's visual condition. The visual status determination unit 13 calculates the reliability level according to the image quality of the image captured by the driver monitor camera 2, for example. The reliability can also be measured by, for example, the degree of image quality degradation (degree of blur) of the captured image. The degree of image quality degradation can be calculated using, for example, known techniques used in camera autofocus. The reliability may be calculated based on the brightness and contrast of the captured image, etc."
and the second gaze tracking process tracks a second respective user gaze with a second precision level different from the first precision level.
MAEHARA [0010] “In another aspect of the present invention, in a method for recognizing the occupant's state, if the distance from the in-vehicle camera to the occupant's head is large and the accuracy of the imaging information from the in-vehicle camera affects the accuracy of acquiring the direction of gaze, the occupant's visual state is determined based on the orientation of the face rather than the direction of gaze. Therefore, according to another aspect of the present invention, the occupant's state recognition method can determine the occupant's visual state by switching whether or not to use the line of sight depending on the accuracy of the imaging information from the in-vehicle camera.”
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of ETCHART in view of KALINLI the capability to detect the gaze or line of sight of the user with two different processes depending on the distance between the user and the device, wherein the selection is based on the precision level of the measurement. The benefit and motivation of such modification is discussed by MAEHARA in the following portion: MAEHARA [0006] “In one aspect of the present invention, in a passenger state recognition device, if the distance from the in-vehicle camera to the passenger's head is large and the accuracy of the imaging information from the in-vehicle camera affects the accuracy of acquiring the direction of gaze, the passenger's visual state is determined based on the orientation of the face rather than the direction of gaze. Therefore, according to one aspect of the present invention, the occupant's visual state can be determined by switching whether or not to use the direction of gaze depending on the accuracy of the imaging information from the in-vehicle camera.”
Summary of references used as prior art
ETCHART is used as the main reference because it provides an invention with similar features to the one claimed, where the system using a microphone and a camera detects if the speech is directed to the device, can detect one speaker/user from another second user, have profiles for multiple user, process the received audio (if is directed to the device) to provide a response, timestamps and processing audio without the need of a spoken trigger in the audio data.
KALINLI supplement ETCHART’s attention determination, especially about forgoing the determination of whether the audio data contains speech if the user attention is not directed to the device and head pose limitation.
SINGER teaches a two-stage power mode, wherein the low power mode provides a low accuracy detection and can determine if the higher accuracy/ higher power mode needs to be enabled.
MAEHARA teaches a two-stage gaze/attention detection, wherein the selection is dependent on the distance and the detection accuracy of the image sensor.
Prior art not relied in this rejection
Rastrow; Ariya et al. (US 20210295833 A1) DEVICE-DIRECTED UTTERANCE DETECTION: Teaches determining whether a speech is directed to a device without requiring a wake-word and provides an interrupt event architecture from a low-latency operation to a higher-accuracy classifier.
Konzelmann; Jaclyn et al. (US 20200349966 A1) HOT-WORD FREE ADAPTATION OF AUTOMATED ASSISTANT FUNCTION(S): Teaches using gaze-based measures to control and determine aspects of the user speech processing.
MATSUMOTO KENJI (JP 2015022700 A) VISUAL LINE DIRECTION DETECTION DEVICE AND VISUAL LINE DIRECTION DETECTION METHOD: Teaches different methods of gaze estimation based on user’s distance from the camera judged by head size and pupil-based gaze estimation to account for estimation accuracy.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HECTOR J. CRESPO FEBLES whose telephone number is (571)272-4512. The examiner can normally be reached Mon - Fri 7:30 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HECTOR J. CRESPO FEBLES/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
