Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al (US Publication No. 20220244781 A1) in view of Gebauer et al (US Publication No. 20200348755 A1).
Regarding Claim 1, Wu discloses A method comprising: accessing an uncorrected gaze vector computed based on a center point of a pupil of a user (Reference “pupil center 308”, See Specification paragraph 0072 where a process for identifying a gaze is described and cites pupil centers in the appearance based computations followed by geometry based computations which cite the gaze as a gaze vector specifically. These structures are further defined in Specification paragraph 0077 and Figure 3 showing the eyeball model along with pupil center 308); the machine learning model trained to establish a relationship between a plurality of ground truth gaze vectors and uncorrected gaze vectors (Reference “GT data 268” and “network output data 264”, see Specification paragraph 0084 where the machine learning model is trained with the error data which is a difference between the ground truth or GT data and the network output data 264) for a plurality of pupil parameters, the pupil parameters including diameters, gaze angles, or gaze eccentricities ( Reference “gaze vector 238”, see Specification paragraph 0075 where the gaze vector 238 can be described by polar and azimuth angles and therefore contains the gaze angles as pupil parameters. Paragraph 0076 continues to describe the glints detected by the system as potential gaze eccentricities as well, describing them as “The detected glints are used to estimate important geometric quantities in the eye which are not directly observable from the eye camera images. As shown in FIG. 2, there can be a large angle between the user's gaze and the camera axis”) ;
However, Wu fails to disclose processing an image of the pupil by a machine learning model to predict an estimated error in a gaze vector [and] generating a corrected gaze vector by applying the estimated error in the gaze vector predicted by the machine learning model to the uncorrected gaze vector that has been computed based on the center point of the pupil of the user. Instead, Gebauer discloses processing an image of the pupil by a machine learning model to predict an estimated error in a gaze vector (Reference “convolutional neural network 920”, see Specification paragraph 0105 where a gaze tracking neural network which is a type of machine learning model receives an image an as part of its calculations includes mean squared error on x and y coordinates of glint performance) [and] generating a corrected gaze vector by applying the estimated error in the gaze vector predicted by the machine learning model to the uncorrected gaze vector (Note “initial pupil estimate” and “estimated error”, see Specification paragraph 0111 where an error is added to the estimate producing new x-y coordinates which would read as a vector. This error is computed by a neural network as described in paragraph 0110 and 0111) that has been computed based on the center point of the pupil of the user (Note “pupil estimate”, see Specification paragraph 0023 where the neural network determines initial gaze characteristics with a pupil center). Motivation for this modification is also disclosed by Gebauer (See Specification paragraph 0107) where the structure of the machine learning models or neural networks described produce pupil location results more efficiently and more accurately. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wu in view of Gebauer.
Regarding Claim 2, Wu discloses The method of claim 1, but fails to disclose further comprising performing one or more augmented reality operations based on the corrected gaze vector. Instead, Gebauer discloses performing one or more augmented reality operations based on the corrected gaze vector (Note “gaze tracking”, see Specification paragraph 0056 where the user selects an option on the display by looking at it). Motivation for this modification is also disclosed by Gebauer (See Specification paragraph 0056) where the display provides further functionality based on the user gaze such as providing higher resolution for the area the user is looking at and lower resolution elsewhere or reducing distortion. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wu in view of Gebauer.
Regarding Claim 3, Wu discloses The method of claim 2, but fails to disclose further comprising: identifying an object depicted in a display of an augmented reality device that corresponds to the corrected gaze vector and performing the one or more augmented reality operations in relation to the identified object.
Instead, Gebauer discloses further comprising: identifying an object depicted in a display of an augmented reality device that corresponds to the corrected gaze vector and performing the one or more augmented reality operations in relation to the identified object (Note “gaze tracking”, see Specification paragraph 0056 where the gaze tracking described is used to allow a user to select an item in the VR display which would be an augmented reality operation. Also note this operation is specifically performed by the user “looking at it”). Motivation for this modification is also disclosed by Gebauer (See Specification paragraph 0056) where the display provides further functionality based on the user gaze such as providing higher resolution for the area the user is looking at and lower resolution elsewhere or reducing distortion. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wu in view of Gebauer.
Regarding Claim 4, Wu discloses The method of claim 1, but fails to disclose wherein the machine learning model is further trained to establish the relationship between the plurality of ground truth gaze vectors and the uncorrected gaze vectors based on gaze eccentricity. Instead, Gebauer discloses wherein the machine learning model is further trained to establish the relationship between the plurality of ground truth gaze vectors and the uncorrected gaze vectors based on gaze eccentricity (Note “glint location” and “gaze direction”, see Specification paragraph 0022 where glint location and gaze direction are part of the neural network process. In Specification paragraph 0059 we see for example the glint location would be specific to the direction of light reflected off of the eyeball itself which would describe a gaze eccentricity rather than purely a gaze angle—that is an angle describing the eyeball angle relative to the head or amount a person might have to strain their eyes to glance to the side for example).
Regarding Claim 5, Wu discloses The method of claim 1, further comprising: training a first component of the machine learning model to predict a first estimated error in a first angular component; training a second component of the machine learning model to predict a second estimated error in a second angular component; and training a third component of the machine learning model to predict a third estimated error in a third angular component (Reference “multi-task neural network 256”, see Specification paragraph 0116 where the multi-task neural network is trained using three types of error data in three separate phases: First, eye segmentation data in a first training step is used. Then in a second training step only glint detection data is used. Finally, a third training step is described which uses cornea center data 280 to train the multi-task neural network. It is noted these are all separate angular components of a glance or gaze. Specification paragraph 0117 further describes the weight modification or result of the training from these steps).
Regarding Claim 6, Wu discloses The method of claim 5, wherein the first, second, and third components are trained separately and independently of each other (Reference “multi-task neural network 256”, see Specification paragraph 0116 where the multi-task neural network is trained using three types of error data in three separate phases: First, eye segmentation data in a first training step is used. Then in a second training step only glint detection data is used. Finally, a third training step is described which uses cornea center data 280 to train the multi-task neural network. It is noted these are all separate angular components of a glance or gaze. Note that these training steps use completely separate and independent data sets and are trained in completely separate stages as well).
Regarding Claim 15, Wu discloses The method of claim 1, wherein the uncorrected gaze vector is computed as a function of corneal reflection and a center position of the pupil (See rejection of Claim 1 above. Also note paragraph 0097 describing the corneal reflection as part of the glint and pupil center calculations).
Regarding Claim 16, Wu discloses The method of claim 1, further comprising detecting changes to the detected point of the pupil based on different environmental conditions, the change in the detected point causing errors in the computed uncorrected gaze vector, the errors comprising at least one of error in gaze position, direction, or vergence depth estimation (Note “environment illumination”, see Specification paragraph 0076 where challenges in gaze estimation result from the increased eccentricity of the pupils as well as due to environment illumination. The light emitters and detected glints are used for estimations. As noted in paragraph 0075, the virtual image light may also be adjusted for this).
Regarding Claim 17, Wu discloses The method of claim 1, but fails to disclose further comprising adding the estimated error in the gaze vector to the uncorrected gaze vector to generate the corrected gaze vector. Instead, Gebauer discloses further comprising adding the estimated error in the gaze vector to the uncorrected gaze vector to generate the corrected gaze vector (See rejection of Claim 1 above. Also note in Specification paragraph 0111 where the pupil center is incorporated with the error describes an addition of 10 and 2 resulting in 12 for the x-dimension of this vector or coordinate: “the initial pupil estimate 1180 may identify a pupil center at x, y: 10, 10 and the estimated error 1175 from the refinement network 1145 may indicate that the x of pupil center should be 2 greater, resulting in a refined pupil estimate of x, y: 12, 10”). Motivation for this modification is also disclosed by Gebauer where the structure of the machine learning models or neural networks described produce pupil location results more efficiently and more accurately. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to modify Wu in view of Gebauer.
Regarding Claim 18, Wu discloses A system comprising: at least one storage device; and at least one processor coupled to the at least one storage device and configured to perform operations (Reference “processing units 202”, “memory 220”, and “communication buses 204” See Figure 2 and Specification paragraph 0029 showing these linked as a system as an embodiment to this invention) comprising [the method described in the rejection of Claim 1 above].
Claim 19 is rejected for containing similar limitations to those described in rejection of Claim 2 above.
Regarding Claim 20, Wu discloses A non-transitory machine-readable storage medium comprising instructions (Reference “memory 220”, see Specification paragraph 0031 that, when executed by one or more processors of a machine, cause the machine to perform operations (Reference “processing units 202”, “memory 220”, and “communication buses 204” See Figure 2 and Specification paragraph 0029 showing these linked as a system as an embodiment to this invention) comprising [the method described in the rejection of Claim 1 above].
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Wu et al (US Publication No. 20220244781 A1) in view of Gebauer et al (US Publication No. 20200348755 A1) further in view of Yasuda et al (US Publication No. 20220129080 A1).
Regarding Claim 7, Wu discloses The method of claim 5, wherein the first, second, and third components are trained (See rejection of Claim 5 above describing the error components the network was trained on for the images. Reference “camera 260” and “multi-task neural network 256”, see Specification paragraph 0086 where the multi-task neural network receives an image of the eye and the paragraph 0088 describing the training mode) for each eye of a plurality of eyes of the user (Reference “camera 260”, see Specification paragraph 0083 describing the setup of two cameras, one for each eye). However, Wu fails to disclose to enable corrections to be applied to a combined uncorrected gaze vector associated with left and right eyes of a user or to be applied separately to a first uncorrected gaze vector associated with the left eye and a second uncorrected gaze vector associated with the right eye.
Instead, Yasuda teaches to enable corrections to be applied to a combined uncorrected gaze vector associated with left and right eyes (Note “combined” and “left and right eyeballs”, see Specification paragraph 0076 where a single vector is estimated by combining the sightline or gaze vectors. These sightline vectors are corrected later in paragraphs 0083 and 0084) of a user or to be applied separately to a first uncorrected gaze vector associated with the left eye and a second uncorrected gaze vector associated with the right eye (Note “respective sightline vectors”, see Specification paragraph 0076 where either the vectors may be combined as above or using the respective estimated sightline vectors of the left and right eyeballs.). There is motivation described later by Yasuda where depending on the selected mode the user is able to correct the sightline vector or gaze position as described above by rotating their head (See Specification paragraph 0103 describing the ability to select to correct the gaze vector and paragraph 0108 describing the correction process where the user is able to correct gaze by rotating their head). Therefore, it would have been obvious to modify Wu in view of Yasuda to allow correction of the respective gaze vectors or a combined gaze vector.
Allowable Subject Matter
Claims 8-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 8, as noted above in rejections of Claims 1 and 5 respectively, Wu in view of Yasuda discloses the method of claim 5. However, while Wu discloses many varieties of GT or ground truth data as described in paragraph 0114, none of those describe a ground truth associated with a pupil diameter and therefore Wu fails to disclose: “accessing a plurality of training data comprising the uncorrected gaze vectors for the plurality of pupil diameters and the plurality of ground truth gaze vectors associated with the uncorrected gaze vectors for the plurality of pupil diameters; obtaining a first batch of the training data comprising a first uncorrected gaze vector for a first pupil diameter; processing the first pupil diameter by the machine learning model to predict a first estimated error in the first uncorrected gaze vector;”. No reference appeared to combine with Wu or Yasuda as disclosed above in an obvious manner to teach the above limitations. Therefore the claim is objected to, but contains allowable subject matter if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 9-14 are also objected to as being dependent claims to the potentially allowable Claim 8 above with no further rejections.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER JOHN RODGERS whose telephone number is (703)756-1993. The examiner can normally be reached 5:30AM to 2:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXANDER JOHN RODGERS/ Examiner, Art Unit 2661
/JOHN VILLECCO/ Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights