DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 201-203, 205-208 and 219-220 are rejected under 35 U.S.C. 102(a) as being anticipated by USPN 2019/0268660 to el Kaliouby et al.
With regard to claim 201, el Kaliouby discloses a non-transitory computer readable medium containing instructions that when executed by at least one processor cause the at least one processor to perform (paragraph [0049], computer program product) operations for attention-associated interactions based on facial skin micromovements (paragraphs [0092]-[0093], facial micro-expressions), the operations comprising:
determining facial skin micromovements of an individual based on reflections of coherent light from a facial region of the individual (paragraphs [0008], [0036] and [0092]-[0093], A user’s facial expressions are tracked and monitoring using images captured from reflected light which can be near infra-red for example or any type of several different types of cameras);
using the facial skin micromovements to determine a specific engagement level of the individual (paragraphs [0011], [0036], [0041] and [0092]-[0093], A user’s engagement level is one of several cognitive states determined by monitoring facial expressions including micro-expressions);
receiving data associated with a prospective interaction with the individual (paragraph [0008]: “The first media presentation includes one of a YouTube™ video, a Vimeo™ video, and a Netflix™ video. Cognitive state data for the vehicle occupant is captured. The cognitive state data includes video facial data from the vehicle occupant during the first media presentation playing.” Cognitive video facial data is recorded for the interaction or the user viewing a specific presentation content.);
accessing a data structure correlating information reflective of alternative engagement levels with differing presentation manners (paragraph [0008]: “An analysis server is used to rank the first media presentation relative to another media presentation. The ranking of the first media presentation is based on the cognitive state data which was captured for the vehicle occupant. The ranking is for the vehicle occupant. The ranking can include “similar” or “dissimilar”, a numerical value, a scale, a threshold, etc. The analysis server is used to correlate the cognitive state data which was captured for the vehicle occupant to cognitive state data collected from other people who experienced the first media presentation.” The user’s cognitive state linked to the content is used to rank the content based on the user’s reaction);
based on the specific engagement level and the correlating information, determining a specific presentation manner for the prospective interaction (paragraph [0008]: “An analysis server is used to rank the first media presentation relative to another media presentation. The ranking of the first media presentation is based on the cognitive state data which was captured for the vehicle occupant. The ranking is for the vehicle occupant. The ranking can include “similar” or “dissimilar”, a numerical value, a scale, a threshold, etc. The analysis server is used to correlate the cognitive state data which was captured for the vehicle occupant to cognitive state data collected from other people who experienced the first media presentation.” The user’s engagement level or reaction is used to rank the content or user interaction of viewing the content.); and
associating the specific presentation manner with the prospective interaction for subsequent engagement with the individual (paragraph [0008], "One or more further media presentation selections are recommended to the vehicle occupant, based on the ranking and the correlating. The further media presentation selections can be similar to or dissimilar to recommendations made to the other people." The system recommends or modifies the presentation content based on the observed reactions or emotional cognitive states to the presentation content. The continent presentations are also accordingly ranked based on the observed cognitive state data or user interaction. Certain presentation manners including the type of content, video, audio, etc. are accordingly associated with the user’s cognitive state or interaction of watching the content).
With regard to claim 202, el Kaliouby discloses the non-transitory computer readable medium of claim 201, wherein the operations further include generating an output reflecting the prospective interaction according to the determined specific presentation manner (paragraph [0008], "One or more further media presentation selections are recommended to the vehicle occupant, based on the ranking and the correlating. The further media presentation selections can be similar to or dissimilar to recommendations made to the other people." The system recommends or modifies the presentation content based on the observed reactions or emotional cognitive states to the presentation content. The output is based on the interaction of the user observing the presentation content).
With regard to claim 203, el Kaliouby discloses the non-transitory computer readable medium of claim 201, wherein the operations further include:
operating at least one coherent light source in a manner enabling illuminating a non-lip portion of a face of the individual, and receiving signals indicative of the reflections of coherent light from the non-lip portion of the face (paragraph [0104], Facial regions including non-lip portions, such as eyes, nose, ears, etc. are imaged in order to determine facial expressions).
With regard to claim 205, el Kaliouby discloses the non-transitory computer readable medium of claim 201, wherein the specific engagement level is a category of engagement (paragraphs [0011], [0036], [0041] and [0092]-[0093], A user’s engagement level is one of several cognitive states determined by monitoring facial expressions including micro-expressions. Additionally, categories of engagement include the detected reaction emotions determined from the FACS categories of facial expressions).
With regard to claim 206, el Kaliouby discloses the non-transitory computer readable medium of claim 201, wherein the specific engagement level includes a magnitude of engagement (paragraphs [0041] and [0095], The FACS facial expression tracking system categorizes the facial expressions and links the facial expressions to intensity scores which are interpreted as engagement levels).
With regard to claim 207, el Kaliouby discloses the non-transitory computer readable medium of claim 201, wherein the specific engagement level is reflective of an extent to which the individual is engaged in an activity including at least one of a conversation, thoughts, or rest (paragraphs [0041] and [0095], The FACS facial expression tracking system categorizes the facial expressions and links the facial expressions to intensity scores which are interpreted as engagement levels. From paragraph [0041]: “One or more cognitive states can be inferred from the cognitive state data. In embodiments, the cognitive state data can include one or more of drowsiness, fatigue, distraction, impairment, sadness, stress, happiness, anger, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, curiosity, humor, depression, envy, sympathy, embarrassment, poignancy, or mirth. The cognitive states that can be inferred can be based on one or more codings or action units from the facial coding action system (FACS).” The various emotions listed correspond, for example thoughts or rest of the individual based on their determined facial expression).
With regard to claim 208, el Kaliouby discloses the non-transitory computer readable medium of claim 207, wherein the operations further include determining the extent to which the individual is engaged in the activity based on facial skin micromovements that correspond with recruitment of at least one muscle out of a group of muscles including: a zygomaticus muscle, an orbicularis oris muscle, a risorius muscle, or a levator labii superioris alaeque nasi muscle (paragraphs [0054], [0083]-[0085], [0090], [0094]-[0095], The FACS system tracks numerous facial muscles for the purpose of identifying the facial expression).
See also FACS Action Units below. FACS is an industry standard. Action Units are listed below as related to facial elements and the underlying muscular basis controlling the facial elements (table citation from Wikipedia).
A.U. number
F.A.C.S. name
Muscular basis
0
Neutral face
1
Inner brow raiser
frontalis (pars medialis)
2
Outer brow raiser
frontalis (pars lateralis)
4
Brow lowerer
depressor glabellae, depressor supercilii, corrugator supercilii
5
Upper lid raiser
levator palpebrae superioris, superior tarsal muscle
6
Cheek raiser
orbicularis oculi (pars orbitalis)
7
Lid tightener
orbicularis oculi (pars palpebralis)
8
Lips toward each other
orbicularis oris
9
Nose wrinkler
levator labii superioris alaeque nasi
10
Upper lip raiser
levator labii superioris, caput infraorbitalis
11
Nasolabial deepener
zygomaticus minor
12
Lip corner puller
zygomaticus major
13
Sharp lip puller
levator anguli oris (also known as caninus)
14
Dimpler
buccinator
15
Lip corner depressor
depressor anguli oris (also known as triangularis)
16
Lower lip depressor
depressor labii inferioris
17
Chin raiser
mentalis
18
Lip pucker
incisivii labii superioris and incisivii labii inferioris
19
Tongue show
20
Lip stretcher
risorius with platysma
21
Neck tightener
platysma]
22
Lip funneler
orbicularis oris
23
Lip tightener
orbicularis oris
24
Lip pressor
orbicularis oris
25
Lips part
depressor labii inferioris, or relaxation of mentalis or orbicularis oris
26
Jaw drop
masseter; relaxed temporalis and internal pterygoid
27
Mouth stretch
pterygoids, digastric
28
Lip suck
orbicularis oris
With regard to claim 219, the discussion of claim 201 applies. The method steps of claim 219 are disclosed in the discussion of the computer program of claim 201.
With regard to claim 220, the discussion of claim 201 applies. El Kaliouby discloses a processor for performing the steps of the computer program of claim 201 (paragraph [0036] and Fig. 2).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 204 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPNs 2019/0268660 to el Kaliouby et al. and 2021/0174934 to Kilmer et al.
With regard to claim 204, el Kaliouby discloses the non-transitory computer readable medium of claim 203, but does not explicitly disclose wherein the operations further include performing a speckle analysis on the coherent light reflections from the non-lip portion of the face to determine the facial skin micromovements.
Kilmer discloses a similar system to el Kaliouby of tracking facial expressions using the FACS muscle tracking system (paragraphs [0045]-[0047] and further discloses that the facial image features are identified and tracked using projected speckle pattern analysis (paragraphs [0055]-[0057] and [0062]-[0066]). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the speckle pattern analysis taught by Kilmer in combination with the facial feature tracking of el Kaliouby in order to determine facial muscle movement and expression analysis
Claims 209-215 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPNs 2019/0268660 to el Kaliouby et al. and 2022/0144312 to Kwatra et al.
With regard to claim 209, el Kaliouby discloses the non-transitory computer readable medium of claim 201, and further discloses that the system may operate using a phone (paragraphs [0008], [0038], [0044] and [0051], but does not explicitly disclose wherein the received data associated with the prospective interaction includes an incoming call, and wherein the associated differing presentation manners include notifying the individual of the incoming call, and directing the incoming call to voicemail.
Kwatra discloses facial expression determination in a similar vehicle environment to that of el Kaliouby, and further teaches determining priority of incoming communications to the system (paragraphs [0015]-[0017] and [0023]). Kwatra teaches that incoming communications such as calls, or text messages are prioritized using machine learning over time and then routes the incoming communications depending on the determined priority. For example, depending on the priority of the call, the call may be routed to the driver or to the passenger Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the communication prioritization taught by Kwatra to route the incoming calls to the driver or passenger or likewise to voicemail, in combination with the vehicle environment of el Kaliouby in order to present communications based on the determined priority.
With regard to claim 210, el Kaliouby discloses the non-transitory computer readable medium of claim 201, but does not explicitly disclose wherein the received data associated with the prospective interaction includes an incoming text message, and wherein the associated differing presentation manners include presenting the text message to the individual in real time and deferring presentation of the text message to a later time.
Kwatra discloses receiving a text message and determining the priority of the text message and then presenting or delaying the text message presentation based on the determined priority or time sensitive nature of the message (paragraphs [0021], [0024]).
Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the communication prioritization taught by Kwatra to route the incoming texts to the driver or passenger or likewise not immediately deliver the text, in combination with the vehicle environment of el Kaliouby in order to present communications based on the determined priority.
With regard to claim 211, el Kaliouby discloses the non-transitory computer readable medium of claim 201, but does not explicitly disclose wherein determining the specific presentation manner for the prospective interaction includes determining how to notify the individual of the prospective interaction.
Kwatra discloses receiving a text message or incoming call and determining the priority of the communication and then presenting or delaying the text message presentation based on the determined priority or time sensitive nature of the communication. The determined priority is then used to choose how to notify the driver or the passenger, etc.(paragraphs [0015]-[0017] and [0021]-[0024]).
Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the communication prioritization taught by Kwatra to route the incoming communication to the driver or passenger or likewise not immediately deliver the communication, in combination with the vehicle environment of el Kaliouby in order to present communications based on the determined priority.
With regard to claim 212, el Kaliouby discloses the non-transitory computer readable medium of claim 211, but does not explicitly disclose wherein determining how to notify the individual of the prospective interaction is based least in part on an identification of a plurality of electronic devices currently used by the individual.
Kwatra discloses receiving a text message or incoming call and determining the priority of the communication and then presenting or delaying the text message presentation based on the determined priority or time sensitive nature of the communication. The determined priority is then used to choose how to notify the driver or the passenger, etc. The plurality of electronic devices currently used by the individual is interpreted as the car the individual is driving at the time of communication (paragraphs [0015]-[0017] and [0021]-[0024]).
Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the communication prioritization while the user is operating a vehicle taught by Kwatra to route the incoming communication to the driver or passenger or likewise not immediately deliver the communication, in combination with the vehicle environment of el Kaliouby in order to present communications based on the determined priority and based on the driver operating the vehicle.
With regard to claim 213, el Kaliouby discloses the non-transitory computer readable medium of claim 201, but does not explicitly disclose wherein the received data associated with the prospective interaction is indicative of an importance level of the prospective interaction, and wherein the specific presentation manner is determined based at least in part on the importance level.
Kwatra discloses receiving a text message or incoming call and determining the priority of the communication and then presenting or delaying the text message presentation based on the determined priority/importance or time sensitive nature of the communication. The determined priority is then used to choose how to notify the driver or the passenger, etc.(paragraphs [0015]-[0017] and [0021]-[0024]).
Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the communication prioritization taught by Kwatra to route the incoming communication to the driver or passenger or likewise not immediately deliver the communication, in combination with the vehicle environment of el Kaliouby in order to present communications based on the determined priority.
With regard to claim 214, el Kaliouby discloses the non-transitory computer readable medium of claim 201, but does not explicitly disclose wherein the received data associated with the prospective interaction is indicative of an urgency level of the prospective interaction, and wherein the specific presentation manner is determined based at least in part on the specific urgency level.
Kwatra discloses receiving a text message or incoming call and determining the priority/urgency of the communication and then presenting or delaying the text message presentation based on the determined priority/urgency or time sensitive nature of the communication. The determined priority is then used to choose how to notify the driver or the passenger, etc.(paragraphs [0015]-[0017] and [0021]-[0024]).
Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the communication prioritization taught by Kwatra to route the incoming communication to the driver or passenger or likewise not immediately deliver the communication, in combination with the vehicle environment of el Kaliouby in order to present communications based on the determined priority/urgency level.
With regard to claim 215, el Kaliouby discloses the non-transitory computer readable medium of claim 201, but does not explicitly disclose wherein the specific presentation manner includes deferring presentation of content until a time period of detected low engagement and wherein the operations further include detecting low engagement at a subsequent time and presenting the content at the subsequent time.
Kwatra discloses receiving a text message or incoming call and determining the priority/urgency of the communication and then presenting or delaying the text message presentation based on the determined priority/urgency or time sensitive nature of the communication. The determined priority is then used to choose how to notify the driver or the passenger, etc.(paragraphs [0015]-[0017] and [0021]-[0024]). Low engagement is interpreted as delaying presentation of the communication to presumably a time when the user is no longer driving.
Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the communication prioritization taught by Kwatra to route the incoming communication to the driver or passenger or likewise not immediately deliver the communication, in combination with the vehicle environment of el Kaliouby in order to present communications based on the determined priority/urgency level.
Allowable Subject Matter
Claims 216-218 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
No found prior art of record teaches the specific combination of limitations recited in claim 216 regarding determining relevance to a conversation of an incoming communication. Claims 217 and 218 are indicated as allowable subject matter because they depend from claim 216.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY J TUCKER whose telephone number is (571)272-7427. The examiner can normally be reached 9AM-5PM Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J TUCKER/Primary Examiner, Art Unit 2661
Prosecution Insights