DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-13, 15-21 are pending. Claims 1, 15, and 18 stand amended to incorporate the subject matter of claim 14, now canceled. Claim 21 is newly presented.
The previously indicated allowability of the subject matter of canceled claim 13, now incorporated into the independent claims 1, 15, and 18, is withdrawn in view of the newly discovered reference Kaliouby (US 20110263946 A1) and Baranton (WO 2024227576 A1, effectively filed 5/4/2023), and similarly for the dependent claims 5-12, 17, and 20. Rejections based on the newly cited reference(s) follow. Accordingly, this Official Action is made NON-FINAL.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4, 9-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kaliouby (US 20110263946 A1, newly cited) in view of Swital (US 20170188807 A1).
Regarding claim 1, Kaliouby discloses an apparatus (Fig. 1, apparatus associated with display, Fig. 2, components thereof, Fig. 6A, recording and analysis of subject; ¶39, “The apparatus constitutes one or more video cameras that record one or more person's facial and head activity as well as one or more person's field of vision (what the person(s) are looking at), which could be on a computer, a laptop, other portable devices such as camera phones, large/small displays such as those used in advertising, TV monitors, or whatever other object the person is looking at … By way of example, the video camera may include an eye tracker for tracking a persons gaze location, or otherwise gaze location tracking may be provided with any other suitable means.”) comprising:
at least one processing device comprising a processor coupled to a memory (e.g. Abstract, “A digital computer and method for processing data indicative of images of facial and head movements of a subject to recognize at least one of said movements and to determine at least one mental state of said subject is provided.”; see also ¶5-6; a digital computer carrying out the disclosed method is considered to disclose computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs);
the at least one processing device being configured:
to obtain eye tracking data associated with a computing device (data as obtained from eye tracker noted above, and see ¶48, table of detected head and facial actions data including eye movements 43, 45, 46, 61, 63, 64, 65, 66);
to identify, based at least in part on the obtained eye tracking data, presence of a user (106) viewing a display screen (110) associated with the computing device (considered disclosed by detection of eye movements as described by ¶48; see also ¶92, “The hierarchical models goes from low-level inferences about the presence of a face in the video”);
to determine, based at least in part on the obtained eye tracking data, eye usage data of the identified user (¶48, the table including a list of action patterns including eye usage pattern data discussed supra);
to process the determined eye usage data to generate at least one data structure (i.e. the table of data of ¶48; see also ¶98, “The Action Unit log 954 includes a line for each action unit for each frame; each line contains the Action Unit name and the number of instances detected of this Action Unit and the length of each instance (start frame and End Frame), so it is essentially a memory dump of the action unit buffer”; plainly the data is a structure in memory for the log to be generated) characterizing conformance of the identified user with a set of one or more eye usage patterns (Fig. 6A, action unit analysis i.e. those eye usage patterns of the table in ¶48 discussed supra) for viewing the display screen associated with the computing device (as in Fig. 1);
to generate one or more recommendations (518) based at least in part on the generated at least one data structure (Fig. 6A, step 518, notifying subject with analysis feedback based on the action unit analysis which is based on those of the ¶48 table; ¶47 and ¶100 provide detail of the feedback, including light, audio, and vibration based feedback); and
to provide the generated one or more recommendations (the feedback 518 of Fig. 6A) to the computing device (considered disclosed since the action unit analysis is used by the computing device to generate the feedback instructions that is provided optically/audibly/tactilely by the device);
wherein providing the generated one or more recommendations to the computing device comprises activating one or more indicators (e.g. LED controller 186) associated with the computing device, the one or more indicators comprising at least one of: a visual indicator separate from the display screen associated with the computing device (¶47, “The action handler 178 may further interface with LED controller 186 which maps certain gesture or mental probabilities of mental states to a green, yellow or red LED which can be mounted on the frame of an eyeglass or any other wearable or ambient object, for example, to give the person wearing the system real-time feedback”); a designated portion of the display screen associated with the computing device; and one or more audio indicators associated with the computing device (ibid. “action handler 178 may interface with any suitable device to indicate the status of mental states or otherwise. By way of example, a high probability of "confusion" that persists over a certain amount of time may trigger a pre-recorded sound file that informs the person using the system that this state has occurred and may provide advice on the course of action to take”).
Kaliouby does not explicitly show that the one or more eye usage patterns specifying at least one of (i) a recommended screen distance range, (ii) a recommended viewing angle range, (iii) a recommended uninterrupted screen time duration, and (iv) a recommended blinking rate range, or to generate one or more recommendations for altering an eye usage of the identified user based at least in part on the generated at least one data structure.
Swital drawn to an eye monitoring system for a user wearing a head mounted display (Abstract) explicitly shows that the one or more eye usage patterns specifying at least one of (i) a recommended screen distance range, (ii) a recommended viewing angle range, (iii) a recommended uninterrupted screen time duration (¶115-117, “… the wearer information data generating unit may be configured to further provide a recommendation data based, at least, on at least one of: wearer's visual behavior data or the generated wearer information … For example, recommendation data includes … activity of the wearer such as taking a break or removing the head mounted device.”), and (iv) a recommended blinking rate range (¶74, “The wearer's visual behavior data may comprise wearer's eye data relating to one or both of the eyes of the wearer such as blinking frequency … ” showing that the blinking rate is monitored and ¶125, “In another example, if the wearer information data generating unit does not identify any reaction of the eye (no blinking, no movement of the eye for example), an alert may be triggered by calling the emergency teams”, showing that the blinking rate is acted upon), and to generate one or more recommendations for altering an eye usage of the identified user based at least in part on the generated at least one data structure (i.e. take a break or seek medical attention as disclosed by Swital).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made screen use recommendations as disclosed by Swital using the eye usage patterns disclosed by both Kaliouby and Swital for the purpose of providing helpful user feedback in the form of advice or alerts.
Regarding claim 2, the modified Kaliouby teaches the apparatus of claim 1 and further discloses wherein the eye tracking data (¶39) associated with the computing device (stored video frame 506) is obtained from a video stream captured utilizing one or more imaging sensors (by camera 116 as in Fig. 1A, and 504 as in Fig. 6A) associated with the computing device (desktop computer 116).
Regarding claim 4, the modified Kaliouby teaches the apparatus of claim 1 and further discloses wherein identifying the presence of the user (106) viewing the display screen (110) associated with the computing device comprises performing facial recognition of one or more faces (of the user) detected in a video stream capture (by camera 116) utilizing one or more imaging sensors (of the camera) associated with the computing device (¶91, “The hierarchical models goes from low-level inferences about the presence of a face in the video …”; the video being generated by the camera imaging sensors as noted above).
Regarding claim 9 and 10, the modified Kaliouby teaches the apparatus of claim 1, Kaliouby explicitly shows that action duration is monitored against threshold durations (¶49), and Swital explicitly shows monitoring state of the wearer during a screen activity (¶123), but the modified Kaliouby does not explicitly show wherein determining the eye usage data of the identified user comprises determining a duration of time that one or more eyes of the identified user have been viewing the display screen associated with the computing device without an interruption lasting at least a threshold duration of time.
Furthermore, Official Notice is taken that stopwatch and alarm functions are well known, i.e. determining duration and providing an alarm alerting based on threshold duration.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have implemented the stopwatch monitoring and duration threshold functions of Kaliouby to carry out the screen viewing duration recommendation known to Swital in order to provide feedback to the wearer (as known both from Kaliouby and Swital as discussed above) toward preventing fatigue (Swital, ¶122-123).
Regarding claim 11, the modified Kaliouby teaches the apparatus of claim 1 and further discloses wherein determining the eye usage data of the identified user comprises determining a blinking rate of one or more eyes of the identified user (Swital, ¶74, ¶163, visual behavior imaging sensor detecting frequency of blinking).
Regarding claim 12, the modified Kaliouby does not explicitly show the apparatus of claim 11, but does not explicitly show wherein responsive to the generated at least one data structure indicating that the determined blinking rate is outside the recommended blinking rate range, at least one of the generated one or more recommendations comprises instructions for altering the determined blinking rate to be within the recommended blinking rate range.
However, as discussed above Kaliouby and Swital are drawn to providing feedback about complex mental states, and include blinking rate detection. Swital explicitly provides a recommendation with rest stop location and rest instructions (¶122).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to, in response to a parameter outside of range, such as the indication of sleepiness as disclosed by Swital, generate recommendations including getting some rest and thereby alter the detected blinking rate by remedying the user’s tiredness.
Regarding claim 13, the modified Kaliouby teaches the apparatus of claim 1 and further discloses wherein the at least one processing device is part of the computing device (Fig. 2, ¶46-47, “interconnected as a system”).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over the modified Kaliouby as applied to claim 1 above, and further in view of Li (US 20220413602 A1).
Regarding claim 3, the modified Kaliouby teaches the apparatus of claim 1 but does not explicitly show wherein the eye tracking data associated with the computing device comprises data obtained from one or more depth sensors associated with the computing device.
Li explicitly shows wherein the eye tracking data associated with the computing device comprises data obtained from one or more depth sensors associated with the computing device (¶63, Fig. 3, eye tracking 120 using 3D face map generation 170, i.e. depth sensing, using camera 114 i.e. depth sensor, of a desktop computer 116).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have utilized the eye tracking method of Li to generate accurate eye tracking estimates in the device of the modified Kaliouby.
Claim(s) 5-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over the modified Kaliouby as applied to claim 1 above, and further in view of Baranton (WO 2024227576 A1, effectively filed 5/4/2023, newly cited).
Regarding claim 5, the modified Kaliouby teaches the apparatus of claim 1, but does not explicitly show wherein determining the eye usage data of the identified user comprises determining a screen distance between one or more eyes of the identified user and the display screen associated with the computing device.
Baranton drawn to screen ergonomics explicitly shows determining the eye usage data of the identified user comprises determining a screen distance (“d”) between one or more eyes of the identified user (4) and the display screen (10) associated with the computing device (depicted as a phone) (Fig. 2, “The positional parameter may be a distance of observation d separating the area of interest 10 from the user 11, or a direction or an angle a of observation of the area of interest by the user, taken between the gaze direction 12 and at least one reference direction or reference plane RP.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have followed the ergonomic recommendations of Baranton to improve the feedback provided by the modified Kaliouby and thereby improve consequences of poor posture (Baranton, p. 1, Background).
Regarding claim 6, the modified Kaliouby teaches the apparatus of claim 5, and further discloses wherein responsive to the generated at least one data structure (Baranton, Fig. 4 showing the data structure generation system implemented on calculation module 3) indicating that the determined screen distance (“d”) is outside the recommended screen distance range (viz. an inadequate screen distance), at least one of the generated one or more recommendations comprises instructions for altering the determined screen distance to be within the recommended screen distance range (Baranton, p. 3, “This feature allows a control over viewing distance to a display and allows the calculation module to issue recommendation to the user to adjust this distance if inadequate.”).
Regarding claim 7, the modified Kaliouby teaches the apparatus of claim 1, but does not explicitly show wherein determining the eye usage data of the identified user comprises determining a viewing angle between one or more eyes of the identified user and the display screen associated with the computing device.
Baranton drawn to screen ergonomics explicitly shows wherein determining the eye usage data of the identified user comprises determining a viewing angle (12, labeled as “alpha”) between one or more eyes of the identified user (4) and the display screen (10) associated with the computing device (depicted as a phone) (Fig. 2, “The positional parameter may be a distance of observation d separating the area of interest 10 from the user 11 , or a direction or an angle [alpha] of observation of the area of interest by the user, taken between the gaze direction 12 and at least one reference direction or reference plane RP.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have followed the ergonomic recommendations of Baranton to improve the feedback provided by the modified Kaliouby and thereby improve consequences of poor posture (Baranton, p. 1, Background).
Regarding claim 8, the modified Kaliouby teaches the apparatus of claim 7, and further discloses wherein responsive to the generated at least one data structure (Baranton, Fig. 4 showing the data structure generating system implemented on calculation module 3) indicating that the determined viewing angle (alpha) is outside the recommended viewing angle range, at least one of the generated one or more recommendations comprises instructions for altering the determined viewing angle to be within the recommended viewing angle range (Baranton, end of p. 3 into p. 4, “According to another implementation, the at least one positional parameter comprises at least one head or neck position angle, the at least one reference parameter comprising a corresponding recommended angular range, the at least one reference parameter further comprising a maximal duration spend out of the recommended angular range over a predetermined duration. This feature allows a control over improper body posture, notably head and neck posture, for example when watching a screen.”).
Claims 15-21 recite essentially the same limitations as those addressed above by Kaliouby, Swital, and Baranton and are rejected for the same reasons under 35 U.S.C. 103.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to COLLIN X BEATTY whose telephone number is (571)270-1255. The examiner can normally be reached M - F, 10am - 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pinping Sun can be reached at 5172701284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COLLIN X BEATTY/Primary Examiner, Art Unit 2872
