DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 521-532, 534-536 and 538-540 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by USPN 2019/0268660 to el Kaliouby et al.
With regard to claim 1, el Kaliouby discloses a non-transitory computer readable medium containing instructions that when executed by at least one processor cause the at least one processor to perform (paragraph [0049], See computer program product) operations for extracting reactions to content based on facial skin micromovements (paragraphs [0092]-[0093], facial micro-expressions), the operations comprising:
during a time period when an individual is consuming content, determining the facial skin micromovements of the individual based on reflections of coherent light from a facial region of the individual (paragraphs [0008] and [0092]-[0093], media presentation content is presented to a user while their facial micro-expressions are monitored and analyzed from images acquired using a camera which is using light reflections and can use various wavelengths such as near-infrared);
determining at least one specific micro-expression from the facial skin micromovements (paragraphs [0092]-[0095], micro-movements of the facial muscles are tracked to identify micro-expressions);
accessing at least one data structure containing correlations between a plurality of micro-expressions and a plurality of non-verbalized perceptions (paragraphs [0095], The FACS Facial Action Coding System is used to relate identifying facial muscle movements to specific facial expressions and to score the related emotions, moods, or cognitive states which are interpreted as non-verbalized perceptions);
based on the at least one specific micro-expression and the correlations in the data structure, determining a specific non-verbalized perception of the content consumed by the individual (paragraph [0008], “The analysis server is used to correlate the cognitive state data which was captured for the vehicle occupant to cognitive state data collected from other people who experienced the first media presentation.” The presented media is rankled according to the viewer observed reaction in the form of facial expressions); and
initiating an action associated with the specific non-verbalized perception (paragraph [0008], “One or more further media presentation selections are recommended to the vehicle occupant, based on the ranking and the correlating. The further media presentation selections can be similar to or dissimilar to recommendations made to the other people.” The system recommends or modifies the presentation content based on the observed reactions or emotional cognitive states to the presentation content).
With regard to claim 522, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the at least one specific micro-expression is imperceptible to a human eye (paragraphs [0092]-[0093], The micro-expressions are only perceptible with a high frame rate camera for example).
With regard to claim 523, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the facial skin micromovements used for determining the at least one specific micro-expression correspond to recruitment of at least one muscle from a group of muscles including:
a zygomaticus muscle, a genioglossus muscle, an orbicularis oris muscle, a risorius muscle, or a levator labii superioris alaeque nasi muscle (paragraphs [0041] and [0095], Numerous facial muscles are tracked and individual muscle movements are encoded including eyes, eyebrows, nose, mouth, etc. “The AUs include so-called main codes (inner brow raiser, lid tightener, etc.), head movement codes (head turned left, head up, etc.), eye movement codes (eyes turned left, eyes up, etc.), visibility codes (eyes not visible, entire face not visible, etc.), and gross behavior codes (sniff, swallow, etc.). Emotion scoring can be included where intensity, as well as specific emotions, moods, or cognitive states, are evaluated.”).
With regard to claim 524, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the at least one specific micro-expression includes a sequence of micro-expressions associated with the specific non-verbalized perception (paragraphs [0109]-[0110], Facial features are identified and tacked in order to determine action units or AUs that are plotted for a sequence of images or frames in order to determine if a facial expression is present).
With regard to claim 525, el Kaliouby discloses the non-transitory computer readable medium of claim 524, wherein the operations further include determining a degree of the specific non-verbalized perception based on the sequence of micro-expressions, and determining an action to initiate based on the degree of the specific non-verbalized perception (paragraphs [0079]-[0080], [0087] and [0095],The FACS system determines and factors in intensity or degree or facial features both as a function of time and a function of facial feature movement).
With regard to claim 526, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the at least one data structure includes past non-verbalized perceptions of previously consumed content, and wherein the operations further comprise determining a degree of the specific non-verbalized perception relative to the past non-verbalized perceptions, and determining an action to initiate based on the degree of the specific non-verbalized perception (paragraph [0008]: “An analysis server is used to rank the first media presentation relative to another media presentation. The ranking of the first media presentation is based on the cognitive state data which was captured for the vehicle occupant. The ranking is for the vehicle occupant. The ranking can include “similar” or “dissimilar”, a numerical value, a scale, a threshold, etc. The analysis server is used to correlate the cognitive state data which was captured for the vehicle occupant to cognitive state data collected from other people who experienced the first media presentation. The correlating can determine whether the vehicle occupant experienced one or more cognitive states similar to or different from other people. One or more further media presentation selections are recommended to the vehicle occupant, based on the ranking and the correlating. The further media presentation selections can be similar to or dissimilar to recommendations made to the other people.” The content presented to the user is scored based on user reaction/perception of the content. See also paragraph [0043]: “The anticipated preferences for the vehicle occupant can be based on an occupant ID or profile, on previous occurrences of the occupant within the vehicle, on a time of day or day of week, on demographic data relating to the vehicle occupant, and so on. The flow 100 further includes ranking the first media presentation relative to another media presentation based on the cognitive state data which was aggregated 152 from the multiple people. The ranking can be based on a “most liked” media presentation, a “least liked” media presentation, a numerical score, etc.” The user’s preferences of content are tracked and stored over time in order to maintain a user profile for recommending and presenting preferable content).
With regard to claim 527, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the non-verbalized perceptions include an emotional state of the individual (paragraphs [0054], [0079] and [0090], Emotions are inferred from the analyzed facial expressions).
With regard to claim 528, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the operations further include determining an action to initiate based on the consumed content and the specific non-verbalized perception (paragraph [0008], “One or more further media presentation selections are recommended to the vehicle occupant, based on the ranking and the correlating. The further media presentation selections can be similar to or dissimilar to recommendations made to the other people.” The system recommends or modifies the presentation content based on the observed reactions or emotional cognitive states to the presentation content).
With regard to claim 529, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the action initiated includes causing a transmission of a message reflecting a correlation between the specific non-verbalized perception and the consumed content (paragraph [0048], When a recommendation is given to change the presented content a message may be sent as an indication that a different content is ready).
With regard to claim 530, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the action initiated includes storing in memory a correlation between the specific non-verbalized perception and the consumed content (paragraph [0043]: “The anticipated preferences for the vehicle occupant can be based on an occupant ID or profile, on previous occurrences of the occupant within the vehicle, on a time of day or day of week, on demographic data relating to the vehicle occupant, and so on. The flow 100 further includes ranking the first media presentation relative to another media presentation based on the cognitive state data which was aggregated 152 from the multiple people. The ranking can be based on a “most liked” media presentation, a “least liked” media presentation, a numerical score, etc.” The user’s preferences/reactions/non-verbalized perception of content are tracked and stored over time in order to maintain a user profile for recommending and presenting preferable content).
With regard to claim 531, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the action includes determining additional content to be presented to the individual based on the specific non-verbalized perception and the consumed content (paragraph [0043]: “The anticipated preferences for the vehicle occupant can be based on an occupant ID or profile, on previous occurrences of the occupant within the vehicle, on a time of day or day of week, on demographic data relating to the vehicle occupant, and so on. The flow 100 further includes ranking the first media presentation relative to another media presentation based on the cognitive state data which was aggregated 152 from the multiple people. The ranking can be based on a “most liked” media presentation, a “least liked” media presentation, a numerical score, etc.” The user’s preferences of content are tracked and stored over time in order to maintain a user profile for recommending and presenting preferable content).
With regard to claim 532, el Kaliouby discloses the non-transitory computer readable medium of claim 531, wherein the consumed content is of a first type and the additional content is of a second type differing from the first type (paragraph [0048], When a recommendation is given to change the presented content a message may be sent as an indication that a different content is ready to be viewed).
With regard to claim 534, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the action includes selecting an alternative manner for presenting the consumed content (paragraph [0048], When a recommendation is given to change the presented content a message may be sent as an indication that a different content is ready to be viewed).
With regard to claim 535, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the action varies based on a type of the consumed content paragraph [0008], “One or more further media presentation selections are recommended to the vehicle occupant, based on the ranking and the correlating. The further media presentation selections can be similar to or dissimilar to recommendations made to the other people.” The system recommends or modifies the presentation content based on the observed reactions or emotional cognitive states to the presentation content).
With regard to claim 536, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the operations further include operating at least one wearable coherent light source in a manner enabling illumination of a non-lip portion of a face of the individual, and receiving signals indicative of coherent light reflections from the non-lip portion of the face (paragraphs [0139]-[0140], Wearable camera such as glasses or head worn camera is disclosed. See paragraphs [0036] and [0039], near-infrared camera is disclosed).
With regard to claim 538, el Kaliouby discloses the non-transitory computer readable medium of claim 521, wherein the reflections of coherent light are received by a wearable light detector (paragraphs [0139]-[0140], Wearable camera such as glasses or head worn camera is disclosed. See paragraphs [0036] and [0039], near-infrared camera is disclosed).
With regard to claim 539, the discussion of claim 521 applies. El Kaliouby discloses the method of claim 539 in the operation of the computer program discussed in claim 521.
With regard to claim 540, the discussion of claim 521 applies. El Kaliouby discloses a system for performing the computer program discussed in claim 521. See Fig. 2. And paragraphs [0044] and [0068].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 533 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPN 2019/0268660 to el Kaliouby et al. and USPN 2022/0171960 to Nelson et al.
.
With regard to claim 533, el Kaliouby discloses the non-transitory computer readable medium of claim 521, but does not explicitly disclose wherein the consumed content is part of a chat with at least one other individual and the action includes generating a visual representation of the specific non-verbalized perception in the chat.
Nelson discloses a system for monitoring users as they spectate content similar to el Kaliouby and further teaches that the user’s reaction to the content as well as an associated chat can be tracked in the form of facial expression, and further teaches generating emojis representing the user’s emotional reaction (Fig. 1 and paragraphs [0004], [0075], [0140] and [0146]). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to enable the facial tracking of a user in the context of a chat and generate emojis representing the emotional state of the user based on their facial expression taught by Nelson in combination with the facial expression reaction tracking of el Kaliouby ion order to generate effective representations of observed emotional data.
Claim 537 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPN 2019/0268660 to el Kaliouby et al. and USPN 2021/0174934 to Kilmer et al.
With regard to claim 537, el Kaliouby discloses the non-transitory computer readable medium of claim 536, but does not explicitly disclose wherein the facial skin micromovements are determined based on speckle analysis of the coherent light reflections.
Kilmer discloses a system that monitors facial muscle movements for the purpose of identifying expressions similar to el Kaliouby and further teaches that speckle detection and tracking is used to track the facial movement (paragraphs [0055]-[0057] and [0062]-[0066]).
Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the speckle image processing and facial tracking taught by Kilmer in combination with the facial feature tracking of el Kaliouby in order to track and identify facial movements for identifying expressions.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY J TUCKER whose telephone number is (571)272-7427. The examiner can normally be reached 9AM-5PM Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J TUCKER/Primary Examiner, Art Unit 2661
Prosecution Insights