DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed December 17, 2025 have been fully considered but they are not persuasive. Claims 1, 6, 11 and 16 have been amended.
Regarding claim 6:
Applicant argument: Applicant argues on page 9 of the Remark that “the Office relies on Ekambaram as allegedly teaching "wherein the facial muscle is a muscle that moves an eye pupil of the first user of the first headset." Applicant respectfully disagrees. Ekambaram merely describes generic "facial muscle movement," and provides examples such as "the strain of a facial muscle can indicate a smile or a scowl." Nowhere does Ekambaram identify, detect, or otherwise reference any muscle responsible for moving the eye or the eye's pupil. In contrast, original claim 6 claim expressly requires that the detected movement be in a muscle that moves an eye pupil of the first user. This limitation is anatomically specific beyond general facial-expression muscles. Ekambaram fails to teach or suggest the detail of "a muscle that moves an eye pupil of the user," and the combination of references fail to disclose "a signal indicative of a movement in a facial muscle from a first user of the first headset, wherein the facial muscle is a muscle that moves an eye pupil of the first user of the first headset."
Examiner’s response: Examiner respectfully disagrees with the argument because Ekambaram fairly discloses highlight claim invention “wherein the facial muscle is a muscle that moves an eye pupil of the first user of the first headset”. Ekambaram, see par. [0033] discloses wearable device 110 has sensors to measure the rate of eye movement, pupil size, or facial muscle movement. For example, eyes moving rapidly back and forth can indicate confusion, a dilated pupil can indicate surprise, and the strain of a facial muscle can indicate a smile or a scowl, and par. [0023] clearly discloses muscle movement and eye movement the sensor measures the user eye movement or muscle movement. Expression analysis program 112 accesses rating database 114 to retrieve known facial expressions, facial movements, or eye movements associated with each sentiment (e.g., happiness, frustration, confusion, attention, boredom, neutrality, anger, laughter, or polarity such as positive reaction and negative reaction) for use in the sentiment analysis. In some embodiments, the range of sentiment is converted to a scaled rating of one to five, wherein a one is highly negative, two is negative, three is neutral, four is positive, and five is highly positive. Therefore, the arguments are not persuasive.
Regarding claims 11 and 20, are similar to claim 1 (please see argument of claim 1 above).
Regarding claims 1-10, 12-15, 17-20, depend on either independent claims 1, 11 or 16. Therefore, for the reasons stated above and presented detail action below, the rejection form first Office Action are maintained.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
a communications module configured to transmit the signals and the facial gesture to a remote server hosting an immersive reality application that includes an avatar of the headset user in claims 11-15.
Paragraphs 22 and 43 of the specification recite corresponding structure.
Because these claim limitation is being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this limitation interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
Claims 1-3, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over AIMONE et al. (US 20140347265 A1) in view of NDUKA (US 20200034608 A1) and further in view of Zimmermann et al. (US 20210312684 A1).
Regarding claim 1. AIMONE discloses a computer-implemented method (AIMONE, see at least par. [0038], This invention describes a method, performed by a wearable computing device comprising a display, and at least one bio-signal measuring sensor, comprising: acquiring at least one bio-signal measurement from a user using the at least one bio-signal measuring sensor; processing the at least one bio-signal measurement in accordance with a profile associated with the user; determining a correspondence between the processed at least one bio-signal measurement and a predefined display control action; and in accordance with the correspondence determination, modifying an image displayed on the display. Optionally, the display may be part of the wearable computing device itself, or it may be provided on a separate computing device that is connected to or otherwise in communication with the wearable computing device. The separate computing device may also be a wearable device worn by the user.), comprising:
receiving, from a sensor on a facial interface of a first headset (AIMONE, see at least par. [0099] FIGS. 1A-1C further show a bio-signal sensor 130 positioned at the temporalis muscle. In one example, sensor 130 may measure activity in the temporal lobe, such as firing of muscles usable by the wearable computing device 100 to determine muscle activity or muscle movement. In the embodiment shown in FIGS. 10A-10C, a bio-signal sensor, not shown, similar to sensor 130 may be positioned in the arm portion of the frame 1002 along the user's temple and obscured from view underneath the frame 1002. This sensor may be of a continuous shape disposed along an arm portion of the frame 1002 or the sensor may be a discreet shape positioned at a particular spot along the frame 1002.), a signal indicative of a movement in a facial muscle from a first user of the first headset (AIMONE, see at least par. [0106] The user's expression may be determined using electrical bio-signal sensors (a) that measure activation of the user's facial muscles. The avatar may be a simplified representation of the user or a more accurate or life-like representation.);
determining a facial expression of the first user with the signal from the sensor in the facial interface of the first headset (AIMONE, see at least par. [0114] Optionally, measurements of the user's facial expressions or emotional response may be taken by performing ectromyogram ("EMG") measurements, and associated expressions or emotions may be determined by the wearable computing device.),
adjusting a subject avatar of the first user of the first headset based on the facial expression (AIMONE, see at least par. [0151] FIG. 28 shows the user's avatar (b) appearing on the display interface. In an exemplary implementation of the wearable computing device, this avatar may reflect the user's current facial expression or emotional state. The user's expression may be determined using electrical bio-signal sensors (a) that measure activation of the user's facial muscles. The avatar may be a simplified representation of the user or a more accurate or life-like representation.); and
AIMONE does not disclose receiving, from a sensor on a facial interface of a first headset, a signal indicative of a movement in a facial muscle from a first user of the first headset; wherein the facial muscle is a muscle that moves an eye pupil of the first user of the first headset; determining a facial expression of the user with the signal from the sensor in the facial interface of the headset based on a machine learning algorithm trained to associate the facial expression to the movement of the facial muscle based at least in part of degree of activation of the facial muscle, and providing, from a remote server hosting an immersive reality application, the adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application. However,
Ekambaram discloses:
receiving, from a sensor on a facial interface of a first headset, a signal indicative of a movement in a facial muscle from a first user of the first headset; wherein the facial muscle is a muscle that moves an eye pupil of the first user of the first headset (Ekambaram, see at least pars. the video feed captures the user facial expressions and facial movements. In another embodiment, expression analysis program 112 analyzes sensor data (e.g., data from an eye tracking sensor), captured by a sensor in sensor unit 116, to determine the user sentiment or reaction to a screenshot of application 108. For example, the sensor measures the user eye movement or muscle movement. Expression analysis program 112 accesses rating database 114 to retrieve known facial expressions, facial movements, or eye movements associated with each sentiment (e.g., happiness, frustration, confusion, attention, boredom, neutrality, anger, laughter, or polarity such as positive reaction and negative reaction) for use in the sentiment analysis. [0033] In step 204, sensor unit 116 captures the user facial expression. In some embodiments, sensors in the wearable device monitor facial features. In these embodiments, wearable device 110 has sensors to measure the rate of eye movement, pupil size, or facial muscle movement. For example, eyes moving rapidly back and forth can indicate confusion, a dilated pupil can indicate surprise, and the strain of a facial muscle can indicate a smile or a scowl. In other embodiments, a picture is taken of a portion of the user face via a camera in sensor unit 116 of wearable device 110. In these embodiments, the picture is analyzed by expression analysis program 112.),
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with receiving, from a sensor on a facial interface of a first headset, a signal indicative of a movement in a facial muscle from a first user of the first headset; wherein the facial muscle is a muscle that moves an eye pupil of the first user of the first headset, as suggested by Ekambaram. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to enable the continued integration of computer system functionality into everyday life. For example, small mobile computing systems, such as miniaturized computers, input devices, sensors, detectors, image displays, wireless communication devices as well as image and audio processors, can be integrated into a device that can be worn by a user. Such small and potentially wearable computing systems can be used in conjunction with mobile devices, for example via wireless networking. Wearable computing devices used in conjunction with a mobile device can expand the functionality of applications for mobile devices (Ekambaram, see par. [0004]).
AIMONE in view of Ekambaram does not discloses determining a facial expression of the user with the signal from the sensor in the facial interface of the headset based on a machine learning algorithm trained to associate the facial expression to the movement of the facial muscle based at least in part of degree of activation of the facial muscle, and providing, from a remote server hosting an immersive reality application, the adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application. However,
NDUKA discloses:
determining a facial expression of the first user with the signal from the sensor in the facial interface of the first headset based on a machine learning algorithm trained to associate the facial expression to the movement of the facial muscle based at least in part of degree of activation of the facial muscle (NDUKA, see pars.[0158-0161] Furthermore, the direction of maximal skin movement can vary according to the degree of activation of associated facial muscles and other muscles involved in a particular facial expression. For example a subtle smile may not create activation of orbicularis oculi with consequent movement of the overlying skin. Whereas a smile approaching maximal intensity will result in co-contraction of the orbicularis oculi. It can therefore be advantageous to configure processor 208 to identify time-varying patterns of skin movements from the set of optical flow sensors which correlate to the progression of a facial expression. This (along with detected magnitude of muscle activations) can help to provide information regarding the strength of a facial expression (e.g. the ‘size’ of smile). Such information may be provided by processor 208 (e.g. via API 210) for use by suitable applications at a computer system. In particular, the ability to include the time varying direction of skin movement during facial expressions can enable system 200 to capture natural facial expressions—e.g. for recreation at an avatar in a VR environment or AR representation.);
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with determining a facial expression of the first user with the signal from the sensor in the facial interface of the first headset based on a machine learning algorithm trained to associate the facial expression to the movement of the facial muscle based at least in part of degree of activation of the facial muscle, as suggested by NDUKA. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to accurate system for detecting facial muscle activity which can be provided as a low power, portable system that can be used whilst the user is moving about (NDUKA, see par. [0003]).
AIMONE in view of Ekambaram, and further in view of NDUKA does not discloses providing, from a remote server hosting an immersive reality application, the adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application. However,
Zimmermann discloses:
providing, from a remote server hosting an immersive reality application (Zimmermann, see pars. [0071]), the adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application (Zimmermann;, see at least par. [0045] The wearable system can extract intent of a user's interaction based on contextual information associated with the user's environment, the user's movements, the user's intentions, and so forth. The wearable system can accordingly map the world motion of the user's interaction to an avatar's action based on the avatar's environment and map the local action of the user's interaction directly to the avatar. The mapping of the world motion can include adjusting one or more characteristics of the avatar such as, e.g., the movement, position, orientation, size, facial expression, pose, eye gaze, etc., to be compatible with the physical environment in which the avatar is rendered (rather than simply mapping the characteristics in a direct one-to-one fashion).)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with providing, from a remote server hosting an immersive reality application), the adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application, as suggested by Zimmermann. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to automatically scale an avatar or to render an avatar based on a determined intention of a user, an interesting impulse, environmental stimuli, or user saccade points. The disclosed systems and methods may apply discomfort curves when rendering an avatar. The disclosed systems and methods may provide a more realistic interaction between a human user and an avatar. (Zimmermann, see par. [0008]).
Regarding claim 2. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1 (as rejected above), and AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann further discloses wherein receiving a signal from a sensor comprises receiving a signal from one of an inertial motion unit, from an electrical sensor, from a capacitive sensor, from a contact microphone, from an optical sensor, from a haptic sensor, from a moisture sensor, and from a haptic sensor (AIMONE, see at least par. [0113] the wearable computing device may include an optical sensor and transmitter placed facing the skin to measure the user's heart rate.).
Regarding claim 3. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann iscloses the computer-implemented method of claim 1 (as rejected above), AIMONE in view of NDUKA and further in view of Lombardi further discloses wherein the signal is an electrical signal from a neural activation of the facial muscle, further comprising identifying the facial muscle by a location of the sensor on the facial interface (AIMONE, see at least par. [0151] FIG. 28 shows the user's avatar (b) appearing on the display interface. In an exemplary implementation of the wearable computing device, this avatar may reflect the user's current facial expression or sensors (a) that measure activation of the user's facial muscles. The avatar may be a simplified representation of the user or a more accurate or life-like representation.).
Regarding claim 9. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1 (as rejected above), AIMONE in view of NDUKA and further in view of Lombardi further discloses further comprising displaying in the first headset a feedback message for the first user based on the facial expression (AIMONE, see at least par. [0054] The bio-signal correspondence determination may determine a correspondence between the at least one bio-signal measurement and a predefined bio-signal measurement stored in the user profile, the predefined bio-signal measurement associated with at least one emotional response type. The at least one predefined display control action may comprise tagging the one at least one displayed item with the corresponding at least one emotional response type. The at least one predefined display control action may comprise displaying a predefined message associated with at least one emotional response type.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over AIMONE et al. (US 20140347265 A1) in view of Ekambaram et al. (US 20160260143 A1), in view of NDUKA US 20200034608 A1, further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 1 above, and further in view of Laszlo (US 20180160982 A1).
Regarding claim 4. . AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1 (as rejected above), but AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann does not disclose wherein receiving a signal from a sensor comprises receiving multiple signals from multiple sensors disposed on the facial interface of the first headset, further comprising correlating the signals to assess whether and to what extent one or more facial muscles from the first user of the first headset are moved. However,
Laszlo discloses:
wherein receiving a signal from a sensor comprises receiving multiple signals from multiple sensors disposed on the facial interface of the first headset, further comprising correlating the signals to assess whether and to what extent one or more facial muscles from the first user of the first headset are moved (Laszlo, see at least par. [0068] the system can identify that a user moved their head based on data from accelerometers on a wearable device on the user's head. The system can identify that a user moved either eyes based on data from an eye tracking sensor or by processing image data with image processing algorithms (e.g., object detection and tracking algorithms). The system can identify that a user moved their facial muscles based by processing image data (e.g., frames of video) using facial detection algorithms. The system can identify a user's heartbeat based on data from a pulse sensor or by processing images of a user using pulse detection algorithms.
[0069] The system identifies a signal pattern that is representative of the physiological action within the brain activity data (406). For example, the system can use a machine learning model to identify signal patterns associated with the identified type of user physiological action. For example, the system can correlate identified user physiological actions with signal patterns within the brainwave data that are related to the identified actions. For example, the system can identify noise patterns generated in brainwave sensors when a user moves their head. As another example, the system can identify interference patterns within the brainwave data caused by a user's heartbeat based on heart rate data such as data indicating a user's pulse rate and timing.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with wherein receiving a signal from a sensor comprises receiving multiple signals from multiple sensors disposed on the facial interface of the first headset, further comprising correlating the signals to assess whether and to what extent one or more facial muscles from the first user of the first headset are moved, as suggested by Laszlo. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby improve the signal quality of brainwave sensors and brainwave sensor systems. Implementations may permit the acquisition of high quality brainwave data while a user is ambulatory. Implementations may enable transparent co-registration of eye movements with EEG activity. (Laszlo, see par. [0021])
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over AIMONE et al. (US 20140347265 A1) in view of Ekambaram US 20160260143 A1), in view of NDUKA (US 20200034608 A1), further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 1 above, and further in view of GRAU et al. (US 20180158246 A1, hereinafter GRAU).
Regarding claim 5. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1, but AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann does not disclose wherein adjusting the subject avatar for the first user comprises molding a three-dimensional solid model of a face of the first user of the first headset based on the facial expression. However,
GRAU discloses:
disclose wherein adjusting the subject avatar for the first user comprises molding a three-dimensional solid model of a face of the first user of the first headset based on the facial expression (GRAU, see at least par. [0032] To resolve these further difficulties, the internal cameras mounted on the internal side of the HMD and facing the eyes of the user may be infra-red (IR) cameras that do not require significant visible light to form the images of the occluded area, and will not interfere with the visibility of the display(s) in the HMD. With IR cameras, the color data is lost and the luminance or shading data is distorted due to the enclosed space on the HMD that may block all other light from entering the internal space. This results in great difficult in converting the IR image data to color data. However, this can be accomplished by learning an appearance model based on color video images and a 3D model to provide the position (or landmarks), color, brightness, and so forth for the occluded area. The images of the occluded area (whether from the IR images, color images of the face taken without the HMD, or both) are warped to the model for different facial expressions on the user.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with disclose wherein adjusting the subject avatar for the first user comprises molding a three-dimensional solid model of a face of the first user of the first headset based on the facial expression, as suggested by GRAU. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to improve robustness and accuracy any information from the HMD such as orientation or position can be used for this purpose if available (Oculus and HTC provide this information). Since the registration from the internal camera coordinate system to the 3D model is already accomplished, the registration from the internal camera to the external camera is now complete. A transformation matrix (hand-eye transformation) for converting coordinates of the external images to the 3D model is computed and then may be used to register the external cameras to the internal cameras, and in turn the occluded face parts shown on the internal images. (GRAU, see par. [0074]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over AIMONE et al. (US 20140347265 A1) in view of Ekambaram et al. (US 20160260143 A1), in view of NDUKA (US 20200034608 A1), further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 1 above, and further in view of Azuma et al. (US 20240382125 A1).
Regarding claim 6. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1 (as rejected above), AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann does not disclose further comprising
determining the facial expression of the first user comprises determining a gaze direction of the user of the first headset. However,
Azuma discloses:
determining the facial expression of the first user comprises determining a gaze direction of the user of the first headset (Azuma, see par. [0072], identify facial expressions can also be used in order to determine to facial expressions and gaze direction of the user.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with determining the facial expression of the first user comprises determining a gaze direction of the user of the first headset, as suggested by Azuma. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to assist the user or his or her family member or community member, the prediction result may be also used to provide more accurate health assessments for the user. (Azuma, see par. [0237]).
Claims 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over AIMONE et al. (US 20140347265 A1) in view of Ekambaram et al. (US 20160260143 A1), in view of NDUKA (US 20200034608 A1), further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 1 above and further in view of Azuma et al. (US 20240382125 A1).
Regarding claim 7. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1 (as rejected above), AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann does not disclose further comprising determining a user physical condition based on the facial expression of the first user. However,
Azuma discloses:
further comprising determining a user physical condition based on the facial expression of the first user (Azuma, see at least pars. [0102] In the present disclosure, the autonomic response of the user includes reflex actions of the autonomic nervous system of the user. These reflex actions include dilation of the pupils of the user's eyes, increase or decrease in the user's pulse or heart rate, increase or decrease in skin conductance (e.g. as a result of sweating), increase or decrease in blood pressure, transient changes in facial expressions or the like. Any reflex action of the autonomic nervous system of the user in response to the display of the first image can be acquired as the autonomic response of the user in accordance with embodiments of the disclosure..).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with further comprising determining a user physical condition based on the facial expression of the first user, as suggested by Azuma. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to assist the user or his or her family member or community member, the prediction result may be also used to provide more accurate health assessments for the user. (Azuma, see par. [0237]).
Regarding claim 8. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1 (as rejected above), AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann does not disclose further comprising determining a user psychological condition based on the facial expression of the first user and an environment in the immersive reality application. However,
Azuma discloses:
further comprising determining a user psychological condition based on the facial expression of the first user and an environment in the immersive reality application (Azuma, see at least par. [0239] As a specific example, such a facial expression recognition method may include first tracking points on the skin (e.g. key anchor points such as the location of the corners of the user's mouth). Then, the method may comprise measuring deformation of the face by movement of the tracked points. These movement patterns can then be compared to a known data set of facial expressions in order to recognised specific facial expressions. In particular, specific facial expressions can be recognised by a combination of movements from multiple tracked points. In some examples, all facial expressions may be defined as positive, negative or neutral valence. Then, the facial response of the user following the display of the first image (and the emotionally salient stimuli which are located therein) can be used as user response data to determine the state of the mental condition of the user in accordance with embodiments of the disclosure.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with further comprising determining a user psychological condition based on the facial expression of the first user and an environment in the immersive reality application, as suggested by Azuma. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to assist the user or his or her family member or community member, the prediction result may be also used to provide more accurate health assessments for the user. (Azuma, see par. [0237]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over AIMONE et al. (US 20140347265 A1) ) in view of Ekambaram et al. (US 20160260143 A1), in view of NDUKA (US 20200034608 A1) and further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 1 above, and further in view of Libin (US 11677575 B1).
Regarding claim 10. AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann discloses the computer-implemented method of claim 1 (as rejected above), AIMONE in view of Ekambaram, further in view of NDUKA and further in view of Zimmermann does not disclose further comprising modifying the facial expression of the first user based on an audience in the immersive reality application. However,
Libin discloses:
further comprising modifying the facial expression of the user based on an audience in the immersive reality application (see col. 9, lines 50-67 and col. 10, lines 1-10, FIG. 4 is a schematic illustration 400 of processing negative audience feedback and choosing an associated system action. Similarly to the previous drawings, the immersive presenter 110 is talking to the audience 140 about presentation material 120b opened in a virtual channel over the virtual backdrop 130. Most of the video conference participants shown in FIG. 4 display a negative attitude towards the current phase of the presentation. Thus, facial expressions of the participants 410, 420 reveal anger, a participant 430 displays disappointment, a participant 440 makes an angry exclamation (not necessarily broadcasted to the conference but nevertheless captured by the system), a participant 450 writes an angry text comment and a participant 460 makes an angry voice comment. The system processes the audience feedback using the workflow explained in detail in conjunction with FIG. 3. Specifically, voice comments are recognized by the STT component 231 and sent to the NLP component 234 along with a text message from the participant 450; voice emotions are analyzed by the component 232, facial expressions by the component 233, and the outputs of the components 232, 233, 234 are aggregated by the sentiment recognition engine 235, which determines that the PEI 250 is abnormal and the angry status 330 prevails, as shown by the height of a first column 470 of the PEI 250, significantly exceeding the summary height of the columns corresponding to the productive state 340 and indifferent state 350.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with further comprising modifying the facial expression of the first user based on an audience in the immersive reality application, as suggested by Libin. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to develop mechanisms and systems for assessment and non-invasive facilitation of participant engagement and enhancing presenter performance in video conferences (Libin, see col. 3, lines 40-44).
Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over NDUKA (US 20200034608 A1) and further in view of el Kaliouby (US 20110263946 A1), and further in view of Zimmermann et al. (US 20210312684 A1).
Regarding claim 11. NDUKA discloses a headset (NDUKA, see par. [0192] Optical flow sensors are compact and readily provided at wearable apparatus (e.g. a pair of glasses or VR headset) which can be worn whilst the user moves about, and which does not need to be large enough to capture the whole of the face of a user.), comprising:
a facial interface including one or more sensors and configured to contact a skin of a face of a headset user around two eyes and a nose in the face (NDUKA, see FIG. 13 and at least par. [0171] For example, FIG. 13 illustrates an advantageous sensor arrangement for measuring the activity of the frontalis muscle ‘A’ in which an optical flow sensor is arranged to detect the movement 1301 of skin in the sub-brow region ‘B’ which overlies an antagonistic muscle to the frontalis. The sub-brow region is that area on or below the supraorbital ridge and above the eye. An optical sensor in the sub-brow region detects stretching of the skin due to contraction of the frontalis muscle. ), wherein the one or more sensors are geometrically disposed on the facial interface to identify a motion of facial muscles of the headset user (NDUKA, see par. [0172] Depending on the shape and size of the apparatus, such as the glasses shown in FIG. 3, sensors 301 (or their equivalent on other apparatus) may in fact be positioned so as to detect movement of skin in the sub-brow region rather than the movement of skin overlying the frontalis. The processor 208 may be configured to interpret elevation of the area of skin in the sub-brow region as activation of the frontalis muscle. In the manner described above this can enable the activity of the frontalis to be indirectly captured by optical flow sensors mounted on apparatus having a compact form factor which does not allow the direct measurement of skin movement overlying the frontalis.) user and a degree of activation of the facial muscles (NDUKA, see par. [0161] Furthermore, the direction of maximal skin movement can vary according to the degree of activation of associated facial muscles and other muscles involved in a particular facial expression.)
a processor configured to at least partially receive multiple signals from the one or more sensors (NDUKA, see par. [0182] The multi-modal arrangement shown in FIG. 14 exploits the characteristics of the optical flow, EMG and/or proximity sensors so as to provide a combined sensor unit which has a very wide dynamic range. The signals from the two or more sensor types may be combined so as to provide a compound signal. Suitable scaling of the signals captured by each sensor type may be empirically determined for a given facial muscle. In combining the signals from the different sensors for a given muscle, the processor 208 may be configured to predominantly use the signal contribution from the sensor type which is best or better suited to the size of the muscle activity. For example, for large activations the processor may derive a compound signal reflecting the degree of activation of a muscle where the primary component is from the proximity sensor and the contributions from the optical flow and/or EMG sensor are damped or omitted so as to prevent the saturated, flat optical flow and/or EMG sensor dominating and concealing the variation expressed in the proximity sensor. At small activations the contribution from the EMG sensor may be promoted or selected over the other sensors. At intermediate activations the contribution from the optical flow sensor may be promoted or selected over the other sensors. Suitable points at which to switch between sensors or reduce the contribution of one sensor in favour of another may be determined empirically.),
a memory storing instructions (NDUKA, see at least par. [0158] A machine learning system for learning facial muscle activations and/or facial expressions may be provided at a computer system to which wearable apparatus including system 200 is connected, and the wearable apparatus may be provided with a learning mode in which the outputs of the optical flow sensors 201 are passed through to the machine learning system at the computer system. A dataset formed by such a machine learning system may be stored at memory 200 for use by the processor. Such a dataset could include, for example, an algorithm or set of parameters for a predefined algorithm, executable code for processor 208, etc.)
NDUKA does not discloses a facial interface including one or more sensors and configured to contact a skin of a face of a headset user around two eyes and a nose in the face, wherein the one or more sensors are geometrically disposed on the facial interface to identify a motion of facial muscles of the headset user and a degree of activation of the facial muscles, wherein the facial muscles includes at least a muscle that moves an eye pupil of the headset user; a memory storing instructions and a chart, the chart including a map of a facial expression to the motion of facial muscles of the headset user; identify a facial gesture of the headset user based on the signals and the chart and a communications module configured to transmit the signals and the facial gesture to a remote server hosting an immersive reality application that includes an avatar of the headset user. However,
Ekambaram discloses:
a facial interface including one or more sensors and configured to contact a skin of a face of a headset user around two eyes and a nose in the face, wherein the one or more sensors are geometrically disposed on the facial interface to identify a motion of facial muscles of the headset user and a degree of activation of the facial muscles, wherein the facial muscles includes at least a muscle that moves an eye pupil of the headset user (Ekambaram, see at least pars. the video feed captures the user facial expressions and facial movements. In another embodiment, expression analysis program 112 analyzes sensor data (e.g., data from an eye tracking sensor), captured by a sensor in sensor unit 116, to determine the user sentiment or reaction to a screenshot of application 108. For example, the sensor measures the user eye movement or muscle movement. Expression analysis program 112 accesses rating database 114 to retrieve known facial expressions, facial movements, or eye movements associated with each sentiment (e.g., happiness, frustration, confusion, attention, boredom, neutrality, anger, laughter, or polarity such as positive reaction and negative reaction) for use in the sentiment analysis. [0033] In step 204, sensor unit 116 captures the user facial expression. In some embodiments, sensors in the wearable device monitor facial features. In these embodiments, wearable device 110 has sensors to measure the rate of eye movement, pupil size, or facial muscle movement. For example, eyes moving rapidly back and forth can indicate confusion, a dilated pupil can indicate surprise, and the strain of a facial muscle can indicate a smile or a scowl. In other embodiments, a picture is taken of a portion of the user face via a camera in sensor unit 116 of wearable device 110. In these embodiments, the picture is analyzed by expression analysis program 112.),
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of NDUKA, with receiving, from a sensor on a facial interface of a first headset, a signal indicative of a movement in a facial muscle from a first user of the first headset; wherein the facial muscle is a muscle that moves an eye pupil of the first user of the first headset, as suggested by Ekambaram. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to enable the continued integration of computer system functionality into everyday life. For example, small mobile computing systems, such as miniaturized computers, input devices, sensors, detectors, image displays, wireless communication devices as well as image and audio processors, can be integrated into a device that can be worn by a user. Such small and potentially wearable computing systems can be used in conjunction with mobile devices, for example via wireless networking. Wearable computing devices used in conjunction with a mobile device can expand the functionality of applications for mobile devices (Ekambaram, see par. [0004]).
NDUKA in view of Ekambaram does not discloses a memory storing instructions and a chart, the chart including a map of a facial expression to the motion of facial muscles of the headset user; identify a facial gesture of the headset user based on the signals and the chart and a communications module configured to transmit the signals and the facial gesture to a remote server hosting an immersive reality application that includes an avatar of the headset user.
el Kaliouby discloses:
a memory storing instructions and a chart, the chart including a map of a facial expression to the motion of facial muscles of the headset user (el Kaliouby, see FIGs. 20 par. [0099], the bar graphs are color coded to displayed a high likelihood or confidence that the gesture is observed on the person's face. The line graphs 1006 on the bottom show the probability of the mental states over time. The graphs are dynamic and move as the video moves. On the right, a radial chart 990 summarizes the most likely mental state at any point in time. FIG. 22 shows instantaneous output 1010 of just the mental state levels, shown as bubbles 1012, 1014, 1016, 1018, 1020 that increase in radius (proportional of probability) depending on the mental state, for example agreeing, disagreeing, concentrating, thinking interested or confused. The person's face 1022 is shown to the left, with the main facial feature points highlighted on the face. In FIG. 26, there is shown instantaneous output of just the mental state levels at any point in time. The person's face is shown to the left, with the main facial feature points highlighted on the face. The probability of each gesture and/or mental state is mapped to the radius of a bubble/circle, called an Emotion Bubble, which is computed as a percentage of a maximum radius size);
identify a facial gesture of the headset user based on the signals and the chart (el Kaliouby, see at least par. [0099], This ranges from no temporal information, where the graph provides a static snapshot of what is detected at a specific point in time (e.g., bar charts in FIG. 20, showing the gestures at a certain point in time), to views that offer temporal information or history (e.g., radial chart 990 in FIG. 21, showing history of a person's over an extended period of time); (3) the window size and sliding factor. In FIG. 20, there is shown a snapshot of one visual output of head and facial analysis system and the plots that are output. On the upper left of the screen the person's video is shown along with the feature point locations. Below the frame is information relating to the confidence of the face finder, the frame rate, the current frame being displayed, as well as eye aspect ratio and face size. On the lower left, the currently recognized facial and head action units are highlighted. The line graphs on the right show the probabilities of the various head gestures, facial expressions as well as mental states. FIG. 25 shows different graphical output given by the system 1000, including a radial chart 990. In the center, the person's video 1002 is shown. In FIG. 21, there is shown another possible output of the system being a radial view that shows the person's most likely mental state over an extended period of time, giving a bird's eye view or general sentiment of a person's state. The probability of the head gestures and facial expressions are displayed as bar graphs 1004 on the left; the bar graphs are color coded to displayed a high likelihood or confidence that the gesture is observed on the person's face. The line graphs 1006 on the bottom show the probability of the mental states over time. The graphs are dynamic and move as the video moves. On the right, a radial chart 990 summarizes the most likely mental state at any point in time); and
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of NDUKA, with a chart, the chart including a map of a facial expression to the motion of facial muscles of the headset user; identify a facial gesture of the headset user based on the signals and the chart and a communications module configured to transmit the signals, as suggested by el Kaliouby. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to providing to a user information relating to said at least one mental state and processing data reflective of input from a user, and based in least in part on said input, confirm or modify said determination. (el Kaliouby, see par. [0008]).
NDUKA in view of Ekambaram and further in view of el Kaliouby does not disclose a communications module configured to transmit the signals and the facial gesture to a remote server hosting an immersive reality application that includes an avatar of the headset user, and provide, from the remote server, an adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application. However,
Zimmermann discloses:
a communications module configured to transmit the signals and the facial gesture to a remote server hosting an immersive reality application (Zimmermann, see at least par. [0133] FIG. 9A schematically illustrates an overall system view depicting multiple user devices interacting with each other. The computing environment 900 includes user devices 930a, 930b, 930c. The user devices 930a, 930b, and 930c can communicate with each other through a network 990. The user devices 930a-930c can each include a network interface to communicate via the network 990 with a remote computing system 920 (which may also include a network interface 971). The network 990 may be a LAN, WAN, peer-to-peer network, radio, Bluetooth, or any other network. The computing environment 900 can also include one or more remote computing systems 920. The remote computing system 920 may include server computer systems that are clustered and located at different geographic locations. The user devices 930a, 930b, and 930c may communicate with the remote computing system 920 via the network 990.) that includes an avatar of the headset user (Zimmermann, see at least par. [0175] a plurality of participants (e.g., tens, hundreds, thousands, or more) with normal vision wearing a particular wearable headset (e.g., a particular model and brand of AR/VR/MR headset) view an avatar in a custom testing application.), and
provide, from the remote server, an adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application (Zimmermann;, see at least par. [0045] The wearable system can extract intent of a user's interaction based on contextual information associated with the user's environment, the user's movements, the user's intentions, and so forth. The wearable system can accordingly map the world motion of the user's interaction to an avatar's action based on the avatar's environment and map the local action of the user's interaction directly to the avatar. The mapping of the world motion can include adjusting one or more characteristics of the avatar such as, e.g., the movement, position, orientation, size, facial expression, pose, eye gaze, etc., to be compatible with the physical environment in which the avatar is rendered (rather than simply mapping the characteristics in a direct one-to-one fashion).)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of NDUKA, with a communications module configured to transmit the signals and the facial gesture to a remote server hosting an immersive reality application that includes an avatar of the headset user, and provide, from the remote server, an adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application, as suggested by Zimmermann. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to automatically scale an avatar or to render an avatar based on a determined intention of a user, an interesting impulse, environmental stimuli, or user saccade points. The disclosed systems and methods may apply discomfort curves when rendering an avatar. The disclosed systems and methods may provide a more realistic interaction between a human user and an avatar. (Zimmermann, see par. [0008]).
Regarding claim 12. NDUKA in view of Ekambaram, further in view of el Kaliouby and further in view of Zimmermann discloses the headset of claim 11 (as rejected above), and NDUKA in view of Ekambaram, further in view of el Kaliouby and further in view of Zimmermann further discloses wherein the one or more sensors include at least one of an inertial motion sensor, an electric sensor, a capacitance sensor, a contact microphone, an optical sensor, a haptic sensor, a moisture sensor, and a temperature sensor (NDUKA, see at least par. [0125] A motion sensor 211 may be used to filter out movements of a wearable device that are not indicative of muscle activity. For example, processor 208 may be configured to damp or ignore signals from the optical flow sensors that are acquired during periods of lame acceleration (e.g. above some predetermined threshold) as measured at the motion sensor. Processor 208 may be configured to damp or ignore components of skin movement detected by the optical flow sensors which are aligned in direction with a motion (e.g. an acceleration) measured by the motion sensor. Processor 208 may be configured to subtract a component from skin movements detected by the optical flow sensors that are aligned in direction with a motion (e.g. an acceleration) measured by the motion sensor. For a given magnitude and direction of motion (e.g. acceleration) measured at the motion sensor, an appropriate size component to subtract, or the amount of damping to introduce into movement vectors captured by the optical flow sensors may be determined by trial and error. A particularly advantageous approach is to use machine learning to determine an appropriate set of damping factors and/or vector components for subtraction from a skin movement vector determined by each optical flow sensor.).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over NDUKA (US 20200034608 A1) in view of Ekambaram et al. (US 20160260143 A1), further in view of el Kaliouby (US 20110263946 A1) and further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 11 above, and further in view of Toth et al. (US 20170231490 A1, hereinafter Toth).
Regarding claim 13. NDUKA in view of Ekambaram, further in view of el Kaliouby and further in view of Zimmermann the headset of claim 11(as rejected above), but NDUKA in view of Ekambaram, further in view of el Kaliouby and further in view of Zimmermann does not discloses wherein the one or more sensors include a photoplethysmography sensor to determine a cardiovascular activity across the nose of the headset user. However,
Toth discloses:
wherein the one or more sensors include a photoplethysmography sensor to determine a cardiovascular activity across the nose of the headset user (Toth, see at least par. [0094] The HMD may include one or more photoplethysmographic (PPG) sensors. The PPG sensor may be directed towards one or more tissue sites on the face, neck, head, of the subject (e.g. an eye, a retina, an ocular tissue, a nose, a nostril, a nasal lining, an ear lobe, etc.). The PPG sensor may be advantageous for capturing one or more cardiovascular parameters like blood oxygen saturation level, heart pulse rate, respiratory rate, bilirubin, or the like in the target tissues.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of NDUKA, with wherein the one or more sensors include a photoplethysmography sensor to determine a cardiovascular activity across the nose of the headset user, as suggested by Toth. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to provide systems, devices, methods, and kits for monitoring physiologic and/or physical signals from a subject. Another objective is to provide systems and methods for assessing the autonomic nervous system (ANS), and/or neuroendocrine system of a subject for the purposes of patient selection for a treatment, treatment feedback, treatment outcome prediction, and treatment follow-up. Yet another objective is to provide emotional, visual, and/or autonomic feedback of a subject immersed in a virtual and/or augmented reality environment. (Toth, see par. [0010]).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over NDUKA (US 20200034608 A1) in view of Ekambaram et al. (US 20160260143 A1), further in view of el Kaliouby (US 20110263946 A1) and further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 11 above, and further in view of BILLINGHURST (US 20190354334 A1).
Regarding claim 14. NDUKA in view of Ekambaram, further in view of el Kaliouby, and further in view of Zimmermann discloses the headset of claim 11 (as rejected above), but NDUKA in view of Ekambaram, further in view of el Kaliouby, and further in view of Zimmermann does not discloses wherein the one or more sensors include electrical sensors symmetrically disposed around the two eyes of the headset user, and configured to assess a gaze direction of the headset user. However,
BILLINGHURST discloses:
wherein the one or more sensors include electrical sensors symmetrically disposed around the two eyes of the headset user, and configured to assess a gaze direction of the headset user (BILLINGHURST, see par. [0104], The physiological sensors may be distributed around the body, be worn or held, or may be integrated in the head mounted display. These sensors may be standalone sensors, or integrated with other components (ie components may be used to provide more than one function). For example a camera in an eye tracking system could also be used to detect and report pupil size (in addition to tracking gaze). Absolute pupil size and temporal changes in pupil size could be used or processed to estimate the physiological state of the local user. For example as a person gets scared their pupil size widens. Similarly in addition to providing the audio stream from the microphone in a headset to the remote user, the audio stream could be processed to detect stress or other emotions in the local users voice. Various physiological sensors will now be described.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of NDUKA, with wherein the one or more sensors include electrical sensors symmetrically disposed around the two eyes of the headset user, and configured to assess a gaze direction of the headset user, as suggested by BILLINGHURST. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to (BILLINGHURST, see par. [0003]).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over NDUKA (US 20200034608 A1) in view of Ekambaram et al. (US 20160260143 A1), further in view of el Kaliouby (US 20110263946 A1), and further in view of Zimmermann et al. (US 20210312684 A1), as applied claim 11 above and further in view of Jin et al. (US 20090156925 A1).
Regarding claim 15. NDUKA in view of Ekambaram in view of el Kaliouby, and further in view of Zimmermann the headset of claim 11 (as rejected above), but NDUKA in view of Ekambaram in view of el Kaliouby, and further in view of Zimmermann does not wherein a portion of the facial interface including the one or more sensors is detachable from the headset. However,
Jin discloses:
wherein a portion of the facial interface including the one or more sensors is detachable from the headset (Jin, see at least par. [0039] The cap 19 is used when the sensor module 10 is fixed to a headset. When the cap 19 is rotated, the height of the cap 19 is changed so as to firmly fix the module 10 to the headset. That is, the sensor module 10 of the present invention is simply and firmly attached to and detached from the headset by rotating the cap 19. Preferably, the cross section of the central part of the cap 19 has a regular ring shape such that an upper protrusion 19u of the cap 19 is protruded outwardly.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of NDUKA, with wherein a portion of the facial interface including the one or more sensors is detachable from the headset, as suggested by Jin. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to improves a general flat-type passive electrode, and more particularly to an active dry sensor module for measurement of bioelectricity, which filters the bioelectricity plural times at a rated capacity and shields interference and noise components due to a power line so as to increase reliability of the bioelectricity, and omits the use of a conductive gel so as to suppress discomfort supplied to a reagent. (Jin, see par. [0002]).
Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Zimmermann et al. (US 20210312684 A1) in view of CHAND et al. (US 20240104859 A1) and further in view of NDUKA (US 20200034608 A1) and further in view of in view of Ekambaram et al. (US 20160260143 A1).
Regarding claim 16. Zimmermann does a computer-implemented method (Zimmermann, see par. [0054], systems and methods may provide for a much more realistic interaction between a user of the wearable system and avatars in the user's environment.), comprising:
providing, from a remote server to a one or more headsets used by multiple participants in an immersive reality application, a subject avatar of a first participant (Zimmermann, see FIG. 9A-9B, see FIG. 17 and pars. [0007, 0131-0134], a particular VR headset may be considered in determining avatar characteristics for one or more avatars to be rendered on the VR headset. Accordingly, a VR headset with higher graphics capabilities, for example, may use a high photorealism avatar (e.g., allowing higher gaze discrimination accuracy by the user) as a default, while a VR headset with lower graphics capabilities may use a lower photorealism avatar as a default.);
provide, from the remote server, an adjusted subject avatar to one or more headsets used by multiple users in the immersive reality application (Zimmermann;, see at least par. [0045] The wearable system can extract intent of a user's interaction based on contextual information associated with the user's environment, the user's movements, the user's intentions, and so forth. The wearable system can accordingly map the world motion of the user's interaction to an avatar's action based on the avatar's environment and map the local action of the user's interaction directly to the avatar. The mapping of the world motion can include adjusting one or more characteristics of the avatar such as, e.g., the movement, position, orientation, size, facial expression, pose, eye gaze, etc., to be compatible with the physical environment in which the avatar is rendered (rather than simply mapping the characteristics in a direct one-to-one fashion).)
Zimmermann does not disclose receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle of the first participant, participant, wherein the facial muscle is a muscle that moves an eye pupil of the first participant; determining a facial gesture of the first participant based on the signal; and updating the subject avatar of the first participant with the facial gesture. However,
CHAND discloses:
receiving from a first headset, with the first participant (CHAND, see at least par. [0279], first computer system 700 optionally receives user inputs (e.g., facial expressions detected by one or more cameras of first computer system) to control the facial expressions of avatar 1010 of the first user.), a signal indicative of a movement of a facial muscle of the first participant (CHAND, see at least par. [0075], determine attention or gaze position and/or gaze movement which can optionally be used to detect gaze-only inputs based on gaze movement and/or dwell. A combination of the various sensors described above can be used to determine user facial expressions and/or hand movements for use in generating an avatar or representation of the user such as an anthropomorphic avatar or representation for use in a real-time communication session where the avatar has facial expressions, hand movements, and/or body movements that are based on or similar to detected facial expressions, hand movements, and/or body movements of a user of the device. Gaze and/or attention information is, optionally, combined with hand tracking information to determine interactions between the user and one or more user interfaces based on direct and/or indirect inputs);
determining a facial gesture of the first participant based on the signal (CHAND, see at least par. [0256], a determination that the movement of the user of the computer system (e.g., 700, 750, and/or X700) includes a second expression (e.g., a smile, a wink, a frown, a first facial expression, and/or a first movement of facial features of the user of the computer system) that is different from the first expression.); and
updating the subject avatar of the first participant with the facial gesture (CHAND, see at least par. [0256] updating display of the self-view representation (e.g., 712 and/or 714) of the avatar of the user of the computer system (e.g., 700, 750, and/or X700) based on (e.g., using and/or to reflect) the second expression.).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Zimmermann, with receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle of the first participant; determining a facial gesture of the first participant based on the signal; and updating the subject avatar of the first participant with the facial gesture, as suggested by CHAND. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to increased significantly in recent years. Example augmented reality environments include at least some virtual elements that replace or augment the physical world (CHAND, see par. [0003]).
Zimmermann in view of CHAND does not discloses receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle and a degree of activation of the facial muscle of the first participant. However,
NDUKA discloses:
receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle and a degree of activation of the facial muscle of the first participant (NDUKA, see par. [0161] Furthermore, the direction of maximal skin movement can vary according to the degree of activation of associated facial muscles and other muscles involved in a particular facial expression. For example a subtle smile may not create activation of orbicularis oculi with consequent movement of the overlying skin. Whereas a smile approaching maximal intensity will result in co-contraction of the orbicularis oculi. It can therefore be advantageous to configure processor 208 to identify time-varying patterns of skin movements from the set of optical flow sensors which correlate to the progression of a facial expression. This (along with detected magnitude of muscle activations) can help to provide information regarding the strength of a facial expression (e.g. the ‘size’ of smile). Such information may be provided by processor 208 (e.g. via API 210) for use by suitable applications at a computer system. In particular, the ability to include the time varying direction of skin movement during facial expressions can enable system 200 to capture natural facial expressions—e.g. for recreation at an avatar in a VR environment or AR representation.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Zimmermann, with receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle and a degree of activation of the facial muscle of the first participant, as suggested by NDUKA. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to accurate system for detecting facial muscle activity which can be provided as a low power, portable system that can be used whilst the user is moving about (NDUKA, see par. [0003]).
Zimmermann in view of CHAND, and further in view of NDUKA does not discloses receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle and a degree of activation of the facial muscle of the first participant, wherein the facial muscle is a muscle that moves an eye pupil of the first participant. However,
Ekambaram discloses:
receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle and a degree of activation of the facial muscle of the first participant, wherein the facial muscle is a muscle that moves an eye pupil of the first participant (Ekambaram, see at least pars. the video feed captures the user facial expressions and facial movements. In another embodiment, expression analysis program 112 analyzes sensor data (e.g., data from an eye tracking sensor), captured by a sensor in sensor unit 116, to determine the user sentiment or reaction to a screenshot of application 108. For example, the sensor measures the user eye movement or muscle movement. Expression analysis program 112 accesses rating database 114 to retrieve known facial expressions, facial movements, or eye movements associated with each sentiment (e.g., happiness, frustration, confusion, attention, boredom, neutrality, anger, laughter, or polarity such as positive reaction and negative reaction) for use in the sentiment analysis. [0033] In step 204, sensor unit 116 captures the user facial expression. In some embodiments, sensors in the wearable device monitor facial features. In these embodiments, wearable device 110 has sensors to measure the rate of eye movement, pupil size, or facial muscle movement. For example, eyes moving rapidly back and forth can indicate confusion, a dilated pupil can indicate surprise, and the strain of a facial muscle can indicate a smile or a scowl. In other embodiments, a picture is taken of a portion of the user face via a camera in sensor unit 116 of wearable device 110. In these embodiments, the picture is analyzed by expression analysis program 112.),
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of AIMONE, with receiving from a first headset, with the first participant, a signal indicative of a movement of a facial muscle and a degree of activation of the facial muscle of the first participant, wherein the facial muscle is a muscle that moves an eye pupil of the first participant, as suggested by Ekambaram. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to enable the continued integration of computer system functionality into everyday life. For example, small mobile computing systems, such as miniaturized computers, input devices, sensors, detectors, image displays, wireless communication devices as well as image and audio processors, can be integrated into a device that can be worn by a user. Such small and potentially wearable computing systems can be used in conjunction with mobile devices, for example via wireless networking. Wearable computing devices used in conjunction with a mobile device can expand the functionality of applications for mobile devices (Ekambaram, see par. [0004]).
Regarding claim 17. Zimmermann in view of CHAND and further in view of NDUKA and further in view of Ekambaram discloses the computer-implemented method of claim 16 (as rejected above), and Zimmermann in view of CHAND and further in view of NDUKA and further in view of Ekambaram further discloses wherein receiving a signal indicative of a movement of a facial muscle of the first participant comprises receiving a signal from at least one of an inertial motion sensor, a contact microphone, an electric sensor, a capacitive sensor, an optical sensor, a haptic sensor, a moisture sensor, and a temperature sensor (Zimmermann, see at least [0098] The images obtained by the inward-facing imaging system 466 may be analyzed to determine the user's eye pose or mood, which can be used by the wearable system 400 to decide which audio or visual content should be presented to the user. The wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as IMUs, accelerometers, gyroscopes, etc.).
Claims 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zimmermann, et al. (US 20210312684 A1) in view of CHAND et al. (US 20240104859 A1).further in view of NDUKA (US 20200034608 A1), further in view of in view of in view of Ekambaram et al. (US 20160260143 A1), as applied claim 16 above, and further in view of BILLINGHURST (US 20190354334 A1).
Regarding claim 18. Zimmermann in view of CHAND and further in view of NDUKA, and further in view of Ekambaram discloses the computer-implemented method of claim 16 (as rejected above), but Zimmermann in view of CHAND and further in view of NDUKA, and further in view of Ekambaram does not disclose wherein receiving the signal integrating of a movement of a facial muscle comprises integrating multiple signals from one or more sensors in a facial interface of the first headset. However,
BILLINGHURST discloses:
wherein receiving the signal integrating of a movement of a facial muscle comprises integrating multiple signals from one or more sensors in a facial interface of the first headset (BILLINGHURST, see at least par. [0106] Photo-sensors mounted on glasses or a similar frame can be used to measure facial muscle movement such as skin deformation around the eye caused by facial expression change. Photo reflective sensors measure the distance between the module and skin surface on face and are small enough to fit on a wearable device, are unobtrusive, and the signals can be the processed fast enough for real-time prediction of the facial expression or emotional state. The facial expression or emotional state can then be displayed or visualised by the remote user. Estimation of the facial expression or emotional state may be performed locally (for example by an Arduino microcontroller running a machine learning algorithm trained on the user) in which case only data representing the emotional state needs to be sent to the remote user, or sensor data may be sent to the remote user for processing and estimation of the facial expression or emotional state).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Zimmermann, with wherein receiving the signal integrating of a movement of a facial muscle comprises integrating multiple signals from one or more sensors in a facial interface of the first headset, as suggested by BILLINGHURST. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to (BILLINGHURST, see par. [0003]).
Regarding claim 19. Zimmermann in view of CHAND and further in view of NDUKA, and further in view of Ekambaram discloses the computer-implemented method of claim 16 (as rejected above), and Zimmermann in view of CHAND and further in view of NDUKA, and further in view of Ekambaram further discloses further comprising updating the subject avatar of the second participant with the facial gesture (CHAND, see at least par. [0256] updating display of the self-view representation (e.g., 712 and/or 714) of the avatar of the user of the computer system (e.g., 700, 750, and/or X700) based on (e.g., using and/or to reflect) the second expression.), but Zimmermann in view of CHAND and further in view of NDUKA does not disclose receiving from a headset with a second participant, a signal indicative of a movement of a facial muscle from the second participant. However,
BILLINGHURST discloses:
receiving from a headset with a second participant, a signal indicative of a movement of a facial muscle from the second participant (BILLINGHURST, see at least par. [0105] Motion or orientation sensors such accelerometers, tilt sensors, gyroscopes, vibration sensors, stretch/linear extension sensors, strain sensors, photo-sensors (including light based emitter/receiver pairs) etc, can be used in a variety of ways. Motion or orientation sensors can be used to measure muscle movements around the face and eyes can also be used to infer facial expressions and thus emotional state. Motion sensors may also capture gross body movements such as shaking of the head, movements of the arms, as well as finer scale movements such as involuntary tremor or fine shaking of hands or muscles that may indicate fatigue or another physiological and/or emotional state.), determining a facial gesture of the first participant based on the signal (BILLINGHURST, see at least par. [0104] A range of physiological sensors 22 may be used to measure a range of physiological parameters, or to provide multiple measurements of the same parameter. The physiological sensor data is used to obtain an estimate of the emotional state of the local user. In the context of this specification, emotional state is used in an inclusive sense to include both physiological state and emotional state inferred from physiological data. That is the physiological state may be directly indicative of an emotional state, or an emotional state may be determined or inferred from physiological data. In this context the physiological sensors could broadly be considered emotion monitoring sensors. That is they collect data from a person (user/wearer) which can be processed/analysed to estimate or determine an emotional state of the person (user/wearer). The physiological sensors 22 may include one or more of a heart rate sensor, a blood pressure sensor, a temperature sensor, an electrodermal activity sensor (also known as a skin conductance or galvanic skin response sensor), a pH sensor, a sweat composition sensor, an accelerometer, a motion sensor, an orientation sensor, a microphone, a camera, an electroencephalogram (EEG) sensor, an electromyography (EMG) sensor, etc.),
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Zimmermann, with receiving from a headset with a second participant, a signal indicative of a movement of a facial muscle from the second participant, determining a facial gesture of the first participant based on the signal, as suggested by BILLINGHURST. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to (BILLINGHURST, see par. [0003]).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Zimmermann et al. (US 20210312684 A1) in view of CHAND et al. (US 20240104859 A1) and further in view of NDUKA (US 20200034608 A1), further in view of in view of in view of Ekambaram et al. (US 20160260143 A1), as applied claim 16, and further in view YAO et al. (US 20170286759 A1).
Regarding claim 20. Zimmermann in view of CHAND, further in view of NDUKA, and further in view of Ekambaram discloses the computer-implemented method of claim 16 (as rejected above), but Zimmermann in view of CHAND, further in view of NDUKA, and further in view of Ekambaram does not disclose wherein determining a facial gesture of the first participant comprises conferring with a chart associating the movement of the facial muscle with the facial gesture. However,
YAO discloses:
wherein determining a facial gesture of the first participant (YAO, see at least par. [0034] The image processing device 100 also may have a facial expression recognition system or unit 104 to determine the facial expression class for each detected face and at a single frame or over a sequence of frames.) comprises conferring with a chart associating the movement of the facial muscle with the facial gesture (YAO, see at least par. [0074] Referring to FIG. 8, by another example, the DGRBFD has favorable capability in capturing diverse movements of facial muscles as demonstrated in charts 802 which present some representative DGRBFDs over aligned face shape samples 800 concerning six popular facial expression classes).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Zimmermann, with wherein determining a facial gesture of the first participant comprises conferring with a chart associating the movement of the facial muscle with the facial gesture, as suggested by YAO. The modification provides an improved system and method for generating subject avatars based on headset users that have a true social presence in an immersive reality environment, thereby to accurate facial expression (YAO, see par. [0003]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KIM THANH THI TRAN whose telephone number is (571)270-1408. The examiner can normally be reached Monday-Friday 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ALICIA HARRINGTON can be reached on 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KIM THANH T TRAN/ Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615