DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-23 are pending.
Title
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed: SYSTEMS AND METHODS OF INITIATING COMPUTERIZED ACTIONS USING EYE GAZE AND EYE GESTURES BASED ON ESTABLISHED EYE GAZE PATTERNS.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-8, 10-14, 16-17, 19-21 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Lemelson et al. (US 6,421,064 B1, IDS) in view of Zhao et al. (US 2022/0206575 A1) and Hsieh et al. (US 2024/0370086 A1).
As to claim 1, Lemelson teaches a method of initiating computerized actions based on eye gaze (Lemelson, Abs., a “system for controlling the automatic scrolling of information includes a screen, a computer system, gimbaled sensor system for following and tracking the position and movement of the user's head and user's eye”), comprising:
detecting, via an eye tracker (Lemelson, e.g., FIG. 1A, “gimbaled sensor system 16 with camera 26 ”) of a computing device (Lemelson, e.g., FIG. 1A, “computer system 14”), eye gaze data comprising a plurality of eye gaze data points of an eye movement of a user (Lemelson, FIG. 2, “step at block 58 for obtaining sensor gaze coordinates from the user 22”), the plurality of eye gaze data points comprising X, Y, and Z eye gaze coordinates (Lemelson, FIG. 4A, e.g., “C (Xc, Yc, Zc”)”, “D (Xd, Yd, Zd)” and “screen gaze coordinates at point B (Xb, Yb, 0) 146”);
training a machine learning model using obtained eye gaze data points (Lemelson, FIG. 5C, “At block 140, the screen gaze coordinates B(Xb,Yb) 146 are determined through a neural network (see FIG. 5C), and these coordinates are sent to the software application program at block 142”);
determining a context of a computing environment (Lemelson, e.g., see FIGS. 6-7), comprising:
identifying an object (Lemelson, e.g., FIGS. 6A-6C, “pop-up menu (not shown)”) of the computing environment (Lemelson, e.g., see FIGS. 6-7);
establishing a location of the object (Lemelson, e.g., FIGS. 6A-6C, “the pop-up menu selection region 240 may become highlighted”) within at least one bounding box, rectangular area or zone having a designated size in the computing environment (Lemelson, see e.g., FIGS. 6-7); and
determining a computerized action (Lemelson, e.g., FIGS. 6-7, “If the pop-up menu 240 has been selected either by dwell time or some other selection protocol, then a pop-up menu algorithm is run at block 332 and then ends at block 344 via connector E 342”) that is associated with the object (Lemelson, e.g., FIG. 7D, “scroll image in horz. And vert. directions by calculated amount 340”);
based on the eye gaze patterns of the user as established by the machine learning model (Lemelson, FIG. 5C, “At block 140, the screen gaze coordinates B(Xb,Yb) 146 are determined through a neural network (see FIG. 5C), and these coordinates are sent to the software application program at block 142”), determining, by the machine learning model, a location of the plurality of eye gaze data points of the user in the computing environment (Lemelson, FIG. 5A, “calculate X,Y screen gaze coordinates with neural network 140”); and
initiating the computerized action associated with the object (Lemelson, e.g., FIGS. 6A-6D, “scroll text down/up/left/right at a rate that is a function of screen gaze coordinate 264 266 272 276”) when the plurality of eye gaze data points of the user and the object are both located within the at least one bounding box, rectangular area or zone of the computing environment (Lemelson, e.g., FIGS 6A-6D, “gaze on pop-up menu? 250” → Yes → “selected? 252” → Yes → “gaze in upper/lower/right/left region? 262 266 270 274” → Yes).
Lemelson does not explicitly teach “sampling the detected eye gaze data points at a prescribed sampling interval (Zhao, FIGS. 1-2, [0066], e.g., “In order to further improve the accuracy, there is a need for reasonably selecting a sample number in a sampling window according to an interval time of the sampling time points”; “acquire eye movement data of two eyes of a user S101”);
collecting the sampled eye gaze data points”; and training the machine learning model using the “sampled eye gaze data points”.
However, Zhao teaches the concept of collecting the sampled eye gaze data points (Zhao, FIG. 1, “selecting eye movement data for identifying a gaze behavior from the acquired eye movement data S102”; [0066], “in order to further improve the accuracy, there is a need for reasonably selecting a sample number in a sampling window according to an interval time of the sampling time points”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the obtained eye gaze data points taught by Lemelson to be those with “sampling time points”, so that the “neural network” taught by Lemelson is trained based on the sampled eye gaze data points, as taught by Zhao, in order to provide “a method and apparatus for identifying a gaze behavior in a three-dimensional space and a memory medium so as to overcome the defect that the prior art cannot determine a visual depth and be used for completing identification of the gaze behavior in the three-dimensional space” (Zhao, [0005])
Lemelson in view of Zhao does not teach “the machine learning model configured for establishing eye gaze patterns of the user”.
However, Hsieh teaches the concept of the machine learning model configured for establishing eye gaze patterns of the user (Hsieh, e.g., FIG. 8, [0021], “user eye gaze patterns 802”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the “neural network” taught by Lemelson to further establish the “user eye gaze patterns 802”, as taught by Hsieh, in order to provide “dynamic and selective illumination of the display panel that reduces power consumption and enhances user productivity relative to other solutions” (Hsieh, [0016]).
As to claim 2, Lemelson teaches the method of claim 1, wherein the one or more objects (Lemelson, e.g., FIGS. 6A-6C, “pop-up menu (not shown)”) of the computing environment (Lemelson, see e.g., FIGS. 6-7) comprises one or more words or text, colors, symbols, numbers, alphabet characters, shapes, user interfaces (UI), UI elements, buttons, menu buttons, action menus, controls (Lemelson, e.g., FIGS. 6A-6C, “pop-up menu (not shown)”), images/photos, icons, music or sound icons, avatars, multimedia, videos, video clips, web links, hyperlinks, advertisements, text messages, or multimedia messages.
As to claim 3, Lemelson teaches the method of claim 1, wherein the eye tracker (Lemelson, e.g., FIG. 1A, “gimbaled sensor system 16 with camera 26 ”) comprises one of a camera (Lemelson, e.g., FIG. 1A, “camera 26 ”), a web cam, any type of camera that faces the user's face (Lemelson, e.g., see FIG. 1A), a head mounted display, virtual reality (VR) equipment or head mounted display, augmented reality (AR) equipment or head mounted display, an eye tracker associated with a XR (Extended Reality) headset, a camera configured to record video, a sensor configured to record video, a security camera, a camera of a building, a camera coupled to a window or a door, a camera mounted on a vehicle or a motorcycle, a screen-based eye tracker, and a remote eye tracker.
As to claim 4, Lemelson teaches the method of claim 3, wherein the eye tracker (Lemelson, e.g., FIG. 1A, “gimbaled sensor system 16 with camera 26”) is configured to track:
the user's view on a display of the computing environment (Lemelson, see FIG. 1A, “according to the position of the user's head 18 and centers the focus of the gimbaled sensor system 16 with camera 26 generally onto the user's eye 20 via camera line of sight 30”); or the user's view of a real world in the computing environment utilizing an augmented reality or a virtual reality.
As to claim 5, Lemelson teaches the method of claim 1, wherein the computing environment (Lemelson, see e.g., FIGS. 1 and 6-7) comprises an environment displayed to the user (Lemelson, see, e.g., FIG. 1A) on a display screen (Lemelson, FIG. 1A, “display or screen 12”) of a computing device (Lemelson, FIG. 1A, “computer system 14”), a desktop environment (Lemelson, see FIG. 1A), a virtual reality environment, an augmented reality environment, a computing environment that incorporates real world elements or objects, a 2-dimensional computing environment or a three-dimensional computing environment.
As to claim 7, Lemelson teaches the method of claim 1, wherein the sampled plurality of eye gaze data points further comprises a plurality of face projection points corresponding to various positions on the user's face as the user is viewing the objects of the computing environment (Lemelson, e.g., see FIG. 4B, “A segmentation image processing algorithm may then be used to segment the profile of the closest user 188 from a wide camera field of view and segment the user's eyes 20 thereby identifying the user's eyes 20 by determining the relative size of the segmented region around each of the user's eyes 20”).
As to claim 8, Lemelson teaches the method of claim 1, wherein the computerized action comprises one of an action of scroll up, scroll down (Lemelson, e.g., FIGS. 6A-6D, “scroll text down/up/left/right at a rate that is a function of screen gaze coordinate 264 266 272 276”), scroll previous, scroll next, move up, move down, move left, move right, move forward, move backward, an action to control the object, an action to select the object (Lemelson, e.g., FIG. 6D, “selected? 252” → Yes), an action to deselect the objection, and an action to control the computing environment (Lemelson, e.g., FIG. 6D, “jump to hypertext location 260”).
As to claim 10, it differs from claim 1 only in that it is the system performing the method of claim 1. It recites substantially the same limitations as in claim 1, and Lemelson in view of Zhao and Shieh teaches them. Examiner renders the same motivations as in claim 1. Please see claim 1 for detailed analysis.
As to claims 11-14 and 16-17, they recite substantially the same limitations as in claims 1-5 and 7-8, respectively, and Lemelson in view of Zhao and Shieh teaches them. Examiner renders the same motivations as in claim 1. Please see claims 1-5 and 7-8 for detailed analysis.
As to claim 19, it differs from claim 1 only in that it is the same the method of claim 1, recites substantially the same limitations as in claim 1, and further recites “detecting, via the machine learning model, an eye gesture made by the user; storing the eye gesture and state in a historical eye gesture data repository”. Lemelson in view of Zhao and Shieh teaches them, and Hsieh further teaches detecting, via the machine learning model, an eye gesture made by the user; storing the eye gesture and state in a historical eye gesture data repository (Hsieh, FIGS. 1 and 8, [0017], “Over a period of time, the processor identifies a user eye gaze pattern, which, as used herein, means the manner in which the user moves her eyes over the period of time to view areas of the display panel 104. … In some examples, the training period continues indefinitely”). Examiner renders the same motivations as in claim 1. Please see claim 1 for detailed analysis.
As to claim 20, Hsieh teaches the method of claim 19, further comprising:
storing the eye gesture in the machine learning model (Hsieh, FIG. 6, [0019], “determine a relationship between conditions of the electronic device and a user eye gaze pattern 610”); and
utilizing the eye gesture to retrain the machine learning model and update eye gaze data patterns of the user (Hsieh, FIGS. 1 and 8, [0017], “Over a period of time, the processor identifies a user eye gaze pattern, which, as used herein, means the manner in which the user moves her eyes over the period of time to view areas of the display panel 104. … In some examples, the training period continues indefinitely”). Examiner renders the same motivation as in claim 1.
As to claim 21, Lemelson teaches the method of claim 19, wherein the eye gesture comprises one or more of the following: a stare, a fixation (Lemelson, e.g., FIG. 6A, “if a user dwells on a hypertext region 238 at point P5234 … for a specified dwell time”), a single blink, and a double blink.
As to claim 23, Lemelson teaches the method of claim 19, wherein the computerized action comprises one of an action of swipe up, swipe down, swipe left, swipe right, move up, move down, move left, move right, move forward, move backward, an action to control the object (Lemelson, e.g., FIGS. 6A-6D, “scroll text down/up/left/right at a rate that is a function of screen gaze coordinate 264 266 272 276”), an action to select the object (Lemelson, e.g., FIG. 6D, “selected? 252” → Yes), an action to deselect the objection, and an action to control the computing environment (Lemelson, e.g., FIG. 6D, “jump to hypertext location 260”).
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lemelson et al. (US 6,421,064 B1, IDS) in view of Zhao et al. (US 2022/0206575 A1), Hsieh et al. (US 2024/0370086 A1) and Keane et al. (US 2014/0347391 A1).
As to claim 6, Lemelson teaches the method of claim 1, further comprising one or more of the following:
selecting the object upon detection of the eye gaze on the object for a period of time (Lemelson, e.g., FIGS. 6-7, “If the pop-up menu 240 has been selected either by dwell time”);
notifying the user that the computerized action associated with the selected object is about to initiated (Lemelson, e.g., FIGS. 6-7, “if a user 22 gazes at point P6236 on pop-up menu selection region 240, then the pop-up menu selection region 240 may become highlighted”).
Lemelson in view of Zhao and Shieh does not teach “upon detection of the eye gaze away from the object for a period of time, deselecting the object and ending the computerized action associated with the selected object”.
However, Keane teaches the concept of, upon detection of the eye gaze away from the object for a period of time, deselecting the object and ending the computerized action associated with the selected object (Keane, FIG. 13, [0133], “a de-selection gesture may for example be … the user looking away from the one or more dynamic virtual objects for a predetermined period of time.
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify, e.g., the “run pop-up menu algorithm 254” taught by Lemelson to further perform the step of the “de-selection gesture”, as taught by Keane, in order to provide more intuitive user interaction with menu.
As to claim 15, it recites substantially the same limitations as in claim 6, and Lemelson in view of Keane teaches them. Examiner renders the same motivations as in claim 6. Please see claim 6 for detailed analysis.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Lemelson et al. (US 6,421,064 B1, IDS) in view of Zhao et al. (US 2022/0206575 A1), Hsieh et al. (US 2024/0370086 A1) and Davies et al. (US 2020/0035218 A1).
As to claim 9, Lemelson in view of Zhao and Shieh does not teach the method of claim 1, wherein when the object comprises one or more words, colors, symbols, numbers, alphabet characters, or shapes, the computerized action to be initiated comprises pronouncing, speaking, verbalizing, or articulating to the user at least one of the following: the one or more words, colors, symbols, numbers, alphabet characters, or shapes, in a same language or in a different language; a definition of each of the one or more words, colors, symbols, numbers, alphabet characters, or shapes to the user; and a sentence including the one or more words, colors, symbols, numbers, alphabet characters, or shapes.
However, Davies teaches the concept that when the object (Davies, e.g., FIGS. 2-4, [0093], “application Voice Drop”) comprises one or more words, colors, symbols (Davies, e.g., FIGS. 2-4, [0093], smile symbol), numbers, alphabet characters (Davies, e.g., FIGS. 2-4, [0093], “VOICEDROP”), or shapes, the computerized action to be initiated comprises pronouncing, speaking, verbalizing, or articulating to the user (Davies, e.g., FIGS. 2-4, [0093], “TTS system 116 can generate speech data indicative of TTS system 116 being activated and playback an explanatory or educational message 206 for the user … via speaker(s) 124 (not shown)”) at least one of the following:
the one or more words, colors, symbols, numbers, alphabet characters, or shapes, in a same language or in a different language;
a definition of each of the one or more words, colors, symbols, numbers, alphabet characters, or shapes to the user; and
a sentence including the one or more words, colors, symbols, numbers, alphabet characters, or shapes (Davies, e.g., FIGS. 2-4, [0093], “You discovered VoiceDrop! Simply drag and drop me and I will read content aloud for you”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify the “pop-up menu 240” taught by Lemelson to be further associated with “generating speech data”, as taught by Davies, in order to provide “a computer-implemented method for selecting content to be read aloud” (Davies, [0005]).
As to claim 18, it recites substantially the same limitations as in claim 9, and Davies teaches them. Examiner renders the same motivations as in claim 9. Please see claim 9 for detailed analysis.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Lemelson et al. (US 6,421,064 B1, IDS) in view of Zhao et al. (US 2022/0206575 A1), Hsieh et al. (US 2024/0370086 A1) and Mullins et al. (US 2016/0246384 A1).
As to claim 22, Lemelson in view of Zhao and Hsieh does not explicitly teach the method of claim 21, wherein the double blink comprises a confirmation or a trigger gesture.
However, Mullins teaches the concept of the double blink comprising a confirmation or a trigger gesture (Mullin, FIG. 5, [0022], “Visual gesture … For example, double blinking triggers a particular function or command in the AR application. Double blinking while looking at a particular location or object (physical or virtual) may trigger other predefined functions”).
At the time of effective filing date, it would have been obvious to one of ordinary skill in the art to modify, e.g., the menu selection by dwelling eye gaze taught Lemelson to be further triggered by the “double blinking”, as taught by Mullins, in order to provide “detecting visual gestures from a computing device” (Mullin, [0021]).
Conclusion
The prior arts made of record and not relied upon are considered pertinent to applicant’s disclosure: Wang (US 2021/0263586 A1) teaches the concept of “adjusting a movement of the portion of the displayed content in response to the captured motion signal being higher than a predetermined threshold” (Abs.);
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICHARD J HONG whose telephone number is (571) 270-7765. The examiner can normally be reached on 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chanh Nguyen can be reached on (571) 272-7772. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Sep. 29, 2026
/RICHARD J HONG/Primary Examiner, Art Unit 2623
***