DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
In the response to this Office Action, the Examiner respectfully requests that support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line numbers in the specification and/or drawing figure(s). This will assist the Examiner in prosecuting this application.
Election/Restrictions
Applicant's election without traverse of Species II: Figure 8 in the reply filed on 09/16/2025 is acknowledged.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-7, 9, 10-11, and 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2009/0079816 A1 to Qvarfordt et al. (hereinafter "Qvarfordt") in view of U.S. Patent Application Publication 2013/0120522 A1 to Lian et al. (hereinafter "Lian").
Regarding Claim 1, Qvarfordt teaches an apparatus comprising: machine readable instructions; and at least one programmable circuit (Claim 12; Figs. 2-5; Para. 41-79 of Qvarfordt; conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214… analysis module 214 can utilize the cultural model data 210 and non-verbal behavior history and state data 212 in analyzing the user's behavior. The term module, as used in the present specification can include, but is not limited to compiled software applications, applets, programming objects such as classes and functions and other constructs) to instantiate: timing circuitry to calculate a duration of time for which an eye gaze of a first user of a video conference is directed toward a second user of the video conference (Figs. 2-5; Para. 26, 41-78 of Qvarfordt; video conferencing system 200 can receive video input data from the image capture device 202. In one embodiment, the system includes a video/audio cultural appropriateness module 214, which receives the data from the image capture device 202, and performs a cultural analysis on the data by analyzing the gaze, gesture, timing, emphasis, emotion and other culturally relevant aspects of the user's actions captured on video), the second user presented as an image via a video conference interface, the video conference interface presented by a display of an electronic device (Figs. 2-5; Para. 26, 41-78 of Qvarfordt; video conferencing system 200 can receive video input data from the image capture device 202. In one embodiment, the system includes a video/audio cultural appropriateness module 214, which receives the data from the image capture device 202, and performs a cultural analysis on the data by analyzing the gaze, gesture, timing, emphasis, emotion and other culturally relevant aspects of the user's actions captured on video… if the system determines that, according to the cultural model, it would be more appropriate for the user to be gazing down instead of looking up, the eye portion of the user's video image can be replaced with a pre-recorded calibrated image of the user's eyes gazing downward); and social feedback control circuitry to: select an eye gaze time threshold based on content associated with the video conference interface (Figs. 2-5; Para. 26, 41-78 of Qvarfordt; video conferencing system 200 can receive video input data from the image capture device 202. In one embodiment, the system includes a video/audio cultural appropriateness module 214, which receives the data from the image capture device 202, and performs a cultural analysis on the data by analyzing the gaze, gesture, timing, emphasis, emotion and other culturally relevant aspects of the user's actions captured on video… the amount of mutual gaze, i.e. when two participants (speaker and listener) are looking at each other at the same time, should not be more than 40 percent or less than 30 percent of the total conversation time. The duration of mutual gaze should be around 1 second, but not longer than 2 seconds. When listening, a person should look at the speaker around 60-70 percent of the duration of the utterance. When speaking, a person should look at the speaker 30-50 percent of the duration of the utterance. The duration of an utterance can be determined by monitoring the audio channels from the different speaker's locations. Appropriate length of a glance on the conversational partner is 2-3 seconds… direction of an averted gaze can be included in the cultural model. Native English speakers should more often glance away upwards or sideways and not downwards. Short downward glances are appropriate if they do not become too frequent or too long (more than 1 second). Other averted glances can vary by up to 3 seconds).
Qvarfordt does not explicitly disclose in response to the duration of time satisfying the eye gaze time threshold, cause an alert to be presented on the display with the video conference interface.
However, Lian teaches in response to a duration of time satisfying eye gaze time threshold, causing an alert to be presented on a display with a video conference interface (Figs. 2-4; Para. 21-48 of Lian; gaze/speech analyzer 36 may analyze video signals during the video conference and determine that the active speaker is staring (somewhat continually) at a target location for a period of time (e.g., two to three seconds)… when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Therefore, at the time when the invention was filed, it would have been obvious to a person of ordinary skill in the art to include in response to the duration of time satisfying the eye gaze time threshold, cause an alert to be presented on the display with the video conference interface using the teachings of Lian in order to modify the system taught by Qvarfordt. The motivation to combine these analogous arts would have to provide a system and a method for alerting a participant in a video conference and offer a lifelike meeting experiences for their end users (Para. 1-2 of Lian).
Regarding Claim 2, the combination of Qvarfordt and Lian teaches that the eye gaze is a corrected eye gaze associated with gaze corrected frames output for presentation to the second user (Para. 26, 43 of Qvarfordt; system can store calibrated image data of the user's eye portion, which can be extracted and used to modify the eye image portion of the video image data in accordance with the alternative behavior… conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214 in order to modify the non-verbal behavior of the user).
Regarding Claim 3, the combination of Qvarfordt and Lian teaches determining a direction of the eye gaze relative to a portion of the video conference interface; and cause the gaze corrected frames to be output for presentation to the second user based on the direction of the eye gaze relative to the portion of the video conference interface (Figs. 2-4; Para. 21-48 of Lian; Participant 40 may see participant 42 on screen 15 a, participant 46 on screen 15 b, and participants 48 on screen 15 c… Participant 40 may direct his gaze at target screen 15 c and (as he speaks) fix his gaze on target participant 48 a: located at a target location (with corresponding coordinates L(x, y, z)) on target screen 15 c… Para. 26, 43 of Qvarfordt; system can store calibrated image data of the user's eye portion, which can be extracted and used to modify the eye image portion of the video image data in accordance with the alternative behavior… conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214 in order to modify the non-verbal behavior of the user).
Regarding Claim 4, the combination of Qvarfordt and Lian teaches instantiating speaker identification circuitry to identify the second user or a third user as a speaker, the social feedback control circuitry to select the eye gaze time threshold based on the identified speaker (Figs. 2-4; Para. 21-48 of Lian; when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Regarding Claim 6, the combination of Qvarfordt and Lian teaches that the social feedback control circuitry is to select the eye gaze time threshold based on a number of users in the video conference (Figs. 2-4; Para. 21-48 of Lian; when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)… Participant 40 may see participant 42 on screen 15 a, participant 46 on screen 15 b, and participants 48 on screen 15 c… Participant 40 may direct his gaze at target screen 15 c and (as he speaks) fix his gaze on target participant 48 a: located at a target location (with corresponding coordinates L(x, y, z)) on target screen 15 c).
Regarding Claim 7, the combination of Qvarfordt and Lian teaches that the alert includes a bounding box around at least a portion of the image of the second user (Figs. 3A-3B; Claim 8; Para. 41-48 of Lian; Multipoint manager element 20 may generate any suitable notification that alerts participant 48 a that participant 40 is speaking to her. As used herein in this Specification, the term ‘notification’ includes any suitable visual, audio, textual information. Such notifications may include a text message (e.g., an instant message), a blinking light, a colored light, any illumination feature, a muted sound, a beep, a proprietary sound, a vibration, an icon, a text, a symbol, an avatar… a picture, a proprietary graphic, or any other suitable notification that is conducive to alerting a given participant in a video conference).
Regarding Claim 9, the combination of Qvarfordt and Lian teaches that the second user presented in the image is presented as an avatar in the image (Fig. 3; Para. 45-48 of Qvarfordt; web server 300 can provide the functionality for enabling video conferencing via avatar display 314 for the users. Each user can be visually represented in the video conference by his/her respective avatar).
Regarding Claim 10, Qvarfordt teaches a non-transitory machine readable storage medium comprising instructions to cause at least one programmable circuit (Claim 22; Para. 41-79 of Qvarfordt; analysis module 214 can utilize the cultural model data 210 and non-verbal behavior history and state data 212 in analyzing the user's behavior. The term module, as used in the present specification can include, but is not limited to compiled software applications, applets, programming objects such as classes and functions and other constructs) to at least: determine a direction of an eye gaze of a first participant of a video conference relative to a video conference interface presented on a display of a first electronic device (Figs. 2-5; Para. 26, 41-78 of Qvarfordt; if the system determines that, according to the cultural model, it would be more appropriate for the user to be gazing down instead of looking up, the eye portion of the user's video image can be replaced with a pre-recorded calibrated image of the user's eyes gazing downward… conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214… modification of the behavior can consist of modifying the output stream to the video output device 204… analysis module 214 can utilize the cultural model data 210 and non-verbal behavior history and state data 212 in analyzing the user's behavior. The term module, as used in the present specification can include, but is not limited to compiled software applications, applets, programming objects such as classes and functions and other constructs); cause a first gaze corrected frame to be output for transmission to a second electronic device associated with a second participant of the video conference based on the direction of the eye gaze, the video conference interface including an image of the second participant (Figs. 2-5; Para. 26, 41-78 of Qvarfordt; system can store calibrated image data of the user's eye portion, which can be extracted and used to modify the eye image portion of the video image data in accordance with the alternative behavior… conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214 in order to modify the non-verbal behavior of the user), the first gaze corrected frame including a corrected eye gaze for the first participant (Figs. 2-5; Para. 26, 41-78 of Qvarfordt; system can store calibrated image data of the user's eye portion, which can be extracted and used to modify the eye image portion of the video image data in accordance with the alternative behavior… conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214 in order to modify the non-verbal behavior of the user… If the gaze position is deemed inappropriate 508, the system can determine an alternative behavior for the user, as shown in step 522. Once the alternative behavior is calculated, the output stream can be modified accordingly, as shown in step 526. In one embodiment, the cultural model data 524 can be used to modify the stream, such as by splicing a selected culturally appropriate gaze sample onto the video signal).
Qvarfordt does not explicitly disclose monitoring a duration of time for which the corrected eye gaze is associated with the second participant; and causing an alert to be output at the first electronic device when the duration of time exceeds an eye gaze threshold.
However, Lian teaches monitoring a duration of time for which eye gaze is associated with a second participant; and causing an alert to be output at a first electronic device when the duration of time exceeds an eye gaze threshold (Figs. 2-4; Para. 21-48 of Lian; gaze/speech analyzer 36 may analyze video signals during the video conference and determine that the active speaker is staring (somewhat continually) at a target location for a period of time (e.g., two to three seconds)… when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Therefore, at the time when the invention was filed, it would have been obvious to a person of ordinary skill in the art to include monitoring a duration of time for which the corrected eye gaze is associated with the second participant; and causing an alert to be output at the first electronic device when the duration of time exceeds an eye gaze threshold using the teachings of Lian in order to modify the system taught by Qvarfordt. The motivation to combine these analogous arts would have to provide a system and a method for alerting a participant in a video conference and offer a lifelike meeting experiences for their end users (Para. 1-2 of Lian).
Regarding Claim 11, the combination of Qvarfordt and Lian teaches identifying one of the second participant or a third participant of the video conference as a speaker; assign a first value to the eye gaze threshold when the second participant is identified as the speaker; and assign a second value to the eye gaze threshold when the third participant is identified as the speaker, the second value different than the first value (Para. 57-59 of Qvarfordt; the amount of mutual gaze, i.e. when two participants (speaker and listener) are looking at each other at the same time, should not be more than 40 percent or less than 30 percent of the total conversation time. The duration of mutual gaze should be around 1 second, but not longer than 2 seconds. When listening, a person should look at the speaker around 60-70 percent of the duration of the utterance. When speaking, a person should look at the speaker 30-50 percent of the duration of the utterance. The duration of an utterance can be determined by monitoring the audio channels from the different speaker's locations. Appropriate length of a glance on the conversational partner is 2-3 seconds… direction of an averted gaze can be included in the cultural model. Native English speakers should more often glance away upwards or sideways and not downwards. Short downward glances are appropriate if they do not become too frequent or too long (more than 1 second). Other averted glances can vary by up to 3 seconds…Figs. 2-4; Para. 21-48 of Lian; gaze/speech analyzer 36 may analyze video signals during the video conference and determine that the active speaker is staring (somewhat continually) at a target location for a period of time (e.g., two to three seconds)).
Regarding Claim 13, the combination of Qvarfordt and Lian teaches detecting an eye gaze of a third participant of the video conference relative to the first participant based on image data of the third participant, the image data associated with the video conference interface; and cause the alert to be output at the first electronic device based on the eye gaze of the third participant. (Figs. 2-4; Para. 21-48 of Lian; when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Regarding Claim 14, the combination of Qvarfordt and Lian teaches detecting a change in the direction of the eye gaze of the first participant relative to the video conference interface; and cause a second gaze corrected frame to be output for transmission to a third electronic device associated with a third participant of the video conference based on the change in the direction of the eye gaze, the video conference interface including an image of the third participant (Figs. 2-4; Para. 21-48 of Lian; Participant 40 may see participant 42 on screen 15 a, participant 46 on screen 15 b, and participants 48 on screen 15 c… Participant 40 may direct his gaze at target screen 15 c and (as he speaks) fix his gaze on target participant 48 a: located at a target location (with corresponding coordinates L(x, y, z)) on target screen 15 c… Para. 26, 43 of Qvarfordt; system can store calibrated image data of the user's eye portion, which can be extracted and used to modify the eye image portion of the video image data in accordance with the alternative behavior… conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214 in order to modify the non-verbal behavior of the user).
Regarding Claim 15, the combination of Qvarfordt and Lian teaches causing the processor circuitry to cause the second gaze corrected frame to be output for transmission to respective electronic devices associated with each participant of the video conference based on the change in the direction of the eye gaze (Figs. 2-4; Para. 21-48 of Lian; Participant 40 may see participant 42 on screen 15 a, participant 46 on screen 15 b, and participants 48 on screen 15 c… Participant 40 may direct his gaze at target screen 15 c and (as he speaks) fix his gaze on target participant 48 a: located at a target location (with corresponding coordinates L(x, y, z)) on target screen 15 c… Para. 26, 43 of Qvarfordt; system can store calibrated image data of the user's eye portion, which can be extracted and used to modify the eye image portion of the video image data in accordance with the alternative behavior… conferencing system 200 also includes an output modification module 216 that operates in conjunction with the analysis module 214 in order to modify the non-verbal behavior of the user).
Regarding Claim 16, the combination of Qvarfordt and Lian teaches that the alert includes an audio alert or a visual alert presented via the display (Figs. 2-4; Para. 21-48 of Lian; when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Claims 19-20, 23, and 25 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication 2023/0119205 A1 to Agrawal et al. (hereinafter "Agrawal") in view of U.S. Patent Application Publication 2013/0120522 A1 to Lian et al. (hereinafter "Lian").
Regarding Claim 19, Agrawal teaches an apparatus comprising: at least one memory; machine readable instructions; and at least one programmable circuit to at least one of instantiate or execute the machine readable instructions (Fig. 1; Para. 24 of Agrawal; communication device 100 has device memory 102, communication subsystem 104, data storage subsystem 106, and input/output (I/O) subsystem 108. Device memory 102 and each subsystem (104, 106, and 108) are managed by controller 110. Device memory 102 includes program code for applications, such as video conference application 114, and other application(s) 116) to: output an indicator of a direction of an eye gaze of a first participant of a video conference relative to content presented via a video conference interface, the indicator to be presented with the video conference interface on a display of an electronic device (Figs. 1-9; Para. 24-52 of Agrawal; Communication devices 100 a - 100 b are sufficiently spaced apart that local participant 123 a, whose image is incorporated in transmitted image stream 193 t is easily recognized as gazing away from image capturing device 130 a when gazing at display device 133 b. Controller 110 can detect that local participant 123 a is producing speech 314 or that an environmental sound is detected that will result in transmitted image stream 193 t being viewed by remote participant 123 b at second communication device(s) 144 a. To improve an apparent eye-to-eye contact between local participant and the viewers on the video communication session, controller 110 causes one or more prompts to be presented to local participant 123 a, such as displayed prompt 316 presented at/on display device 133 b… Controller receives image stream 193 a from image capturing device 130 a and communicates transmitted image stream 193 t to remote environment 304 of communication environment. In response to detecting speech 314 by local participant 123 a or ambient sound sufficient to trigger the image capture device to begin to transmit transmitted image stream 193 t, controller 110 a prompts local participant 123 a to gaze at image capturing device 130 a by one or more of: (i) presenting displayed prompt 316 on display device 133 b).
Agrawal does not explicitly disclose adjusting a property of the indicator based on a change associated with one or more of content of the video conference interface or the direction of the eye gaze of the first participant relative to the content.
However, Lian teaches adjusting a property of an indicator based on a change associated with one or more of content of a video conference interface or a direction of eye gaze of a participant relative to the content (Figs. 2-4; Para. 21-48 of Lian; gaze/speech analyzer 36 may analyze video signals during the video conference and determine that the active speaker is staring (somewhat continually) at a target location for a period of time (e.g., two to three seconds)… when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Therefore, at the time when the invention was filed, it would have been obvious to a person of ordinary skill in the art to include adjusting a property of the indicator based on a change associated with one or more of content of the video conference interface or the direction of the eye gaze of the first participant relative to the content using the teachings of Lian in order to modify the system taught by Agrawal. The motivation to combine these analogous arts would have to provide a system and a method for alerting a participant in a video conference and offer a lifelike meeting experiences for their end users (Para. 1-2 of Lian).
Regarding Claim 20, the combination of Agrawal and Lian teaches identifying a second participant of the video conference as a speaker; and adjust the property of the indicator in response to identifying the second participant as the speaker (Figs. 2-4; Para. 21-48 of Lian; when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Regarding Claim 23, the combination of Agrawal and Lian teaches adjusting the property of the indicator when a duration of time of the eye gaze exceeds an eye gaze threshold for maintaining the direction of the eye gaze of the first participant relative to the content (Figs. 2-4; Para. 21-48 of Lian; when the active speaker stops speaking with a questioning tone, with his head and eyes directed at a target location on a target screen for a certain time interval, then the target participant (whose image is positioned at the target location) may be notified via an appropriate notification (e.g., blinking icon on display, beep, text message, etc.)).
Regarding Claim 25, the combination of Agrawal and Lian teaches that the content of the video conference interface includes an avatar representing a second participant of the video conference (Figs. 3A-3B; Claim 8; Para. 41-48 of Lian; Multipoint manager element 20 may generate any suitable notification that alerts participant 48 a that participant 40 is speaking to her. As used herein in this Specification, the term ‘notification’ includes any suitable visual, audio, textual information. Such notifications may include a text message (e.g., an instant message), a blinking light, a colored light, any illumination feature, a muted sound, a beep, a proprietary sound, a vibration, an icon, a text, a symbol, an avatar).
Allowable Subject Matter
Claims 5, 12, and 21 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
None of the references, either singularly or in combination, teach or fairly suggest the apparatus of claim 1, wherein one or more of the at least one programmable circuit is to instantiate pupil dilation monitoring circuitry to determine a pupil dilation rate of the second user, the social feedback control circuitry to select the eye gaze time threshold based on the pupil dilation rate.
None of the references, either singularly or in combination, teach or fairly suggest the non-transitory machine readable storage medium of claim 10, wherein the instructions are to cause one or more of the at least one programmable circuit to: determine a pupil dilation rate of the second participant based on image data of the second participant, the image data associated with the video conference interface; and select the eye gaze threshold based on the pupil dilation rate.
None of the references, either singularly or in combination, teach or fairly suggest the apparatus of claim 19, wherein one or more of the at least one programmable circuit is to determine a pupil dilation rate for a second participant of the video conference, the video conference interface including an image of the second participant; and adjust the property of the indicator based on the pupil dilation rate.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABHISHEK SARMA whose telephone number is (571)272-9887. The examiner can normally be reached on Mon - Fri 8:00-5:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr Awad can be reached on 571-272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABHISHEK SARMA/
Primary Examiner, Art Unit 2621