DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
35 USC § 101
Regarding independent claims 1, 11 and 19, the act of performing an entrance pupil reconstruction based upon feature measurements from the image data reflects the improvement of pupil localization as it demonstrates the reconstruction process based upon live data, rather than stored parameters that would lead to inaccuracy. This is outlined in the specification at paragraph 0016. As such, this integrates the abstract idea into a practical application because the claim improves the functioning of a computer or technical field.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
Claim(s) 1-8 and 11-17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Smyth (US 8,824,779).
Regarding claim 1, Smyth discloses a device for reconstructing entrance pupils, the device comprising:
a camera system (“The light sensors 16, 18 receive the reflected light (i.e., light rays), which is used to determine the gaze direction of the eye. More particularly, the light sensors 16, 18 generate multiple images including two-dimensional "stereo" images of the eye 10 that can be used to calculate the optical axis and visual axis of the eye” at col. 10, line 24; “To this purpose, the invention may be a dual camera system with stereo views of the eye, either direct or from a mirror system” at col. 15, line 27); and
processing circuitry coupled to memory storing instructions that, when executed by the processing circuitry (“Further, the components of the visual axis calculation system may consist of digital processors in the form of computer routines with operating systems comprising dedicated digital computer programs” at col. 11, line 18), causes the processing circuitry to:
receive image data from the camera system (“The light sensors 26, 27 detect the light reflected from the user's eye via the optical systems from the corresponding light collecting optics, and output analog image signals to the opto-electronic device 30, under control of the controller via control line 31; in turn, the opto-electronic device 30 processes the image signals with digital output to the image processor 34 under control via control line 32” at col. 11, line 6);
extract a set of features from the image data, wherein extracting the set of features comprises classifying the features into a plurality of feature types (“In one embodiment, these parameters comprise the image locations of the source specular points, and the apparent pupil centers and elliptical axes isolated for the images. In other embodiments, the parameters comprise the origins and directional cosines of the optical axes. In the calibration process, the digital computer receives as input the optical features (e.g., the origins and directional cosines of the optical axes of sight) in lieu of that for the visual lines of sight as the user looks at visual cues presented under computer control. The computer determines the relation between these features and the visual lines of sight and the relating values are stored in the digital processor of the eye-tracking system by the computer. This result may be in the form of an artificial neural network relating the optical features such as the source specular points and the apparent pupil centers and elliptical axes, or the origin and directional cosines of the optical line of sight, to the location and direction of the visual axis of the eye for each apparatus of the invention” at col. 52, line 57);
calculate a set of feature measurements from the extracted set of features (the features are represented by the directional cosines for the measured pupil centers); and
perform an entrance pupil reconstruction process for determining entrance pupil position data using the set of feature measurements (“In such a case, the digital computer can use the various visual parameters provided by the tracking system to compute the visual gaze point with regard to task workspace coordinates from an embedded knowledge of the workspace geometry and the head location and orientation in that space determined from a head position and orientation sensor” at col. 52, line 44).
Regarding claim 11, Smyth discloses a method for reconstructing entrance pupils, the method comprising:
receiving image data (“The light sensors 26, 27 detect the light reflected from the user's eye via the optical systems from the corresponding light collecting optics, and output analog image signals to the opto-electronic device 30, under control of the controller via control line 31; in turn, the opto-electronic device 30 processes the image signals with digital output to the image processor 34 under control via control line 32” at col. 11, line 6);
extracting a set of features from the image data, wherein extracting the set of features comprises classifying the features into a plurality of feature types (“In one embodiment, these parameters comprise the image locations of the source specular points, and the apparent pupil centers and elliptical axes isolated for the images. In other embodiments, the parameters comprise the origins and directional cosines of the optical axes. In the calibration process, the digital computer receives as input the optical features (e.g., the origins and directional cosines of the optical axes of sight) in lieu of that for the visual lines of sight as the user looks at visual cues presented under computer control. The computer determines the relation between these features and the visual lines of sight and the relating values are stored in the digital processor of the eye-tracking system by the computer. This result may be in the form of an artificial neural network relating the optical features such as the source specular points and the apparent pupil centers and elliptical axes, or the origin and directional cosines of the optical line of sight, to the location and direction of the visual axis of the eye for each apparatus of the invention” at col. 52, line 57);
calculating a set of feature measurements from the extracted set of features (the features are represented by the directional cosines for the measured pupil centers); and
performing an entrance pupil reconstruction process for determining entrance pupil position data using the set of feature measurements (“In such a case, the digital computer can use the various visual parameters provided by the tracking system to compute the visual gaze point with regard to task workspace coordinates from an embedded knowledge of the workspace geometry and the head location and orientation in that space determined from a head position and orientation sensor” at col. 52, line 44).
Regarding claims 2 and 12, Smyth discloses a device and method wherein the camera system comprises a first camera for capturing images of a first eye of a user (“The light sensors 26, 27 detect the light reflected from the user's eye via the optical systems from the corresponding light collecting optics, and output analog image signals to the opto-electronic device 30, under control of the controller via control line 31; in turn, the opto-electronic device 30 processes the image signals with digital output to the image processor 34 under control via control line 32” at col. 11, line 6) and a second camera for capturing images of a second eye of the user (“The invention may be used with head-mounted displays including retinal scanning displays such as those developed for virtual reality, stereographic displays, monocular or binocular vision helmet mounted displays, and night vision goggles used in piloted helicopters, vehicles, and teleoperated robotics control stations” at col. 54, line 64; this implies that there are separate camera systems for each eye display), and wherein the entrance pupil position data comprises a first entrance pupil position for the first eye and a second entrance pupil position for the second eye (“In such a case, the digital computer can use the various visual parameters provided by the tracking system to compute the visual gaze point with regard to task workspace coordinates from an embedded knowledge of the workspace geometry and the head location and orientation in that space determined from a head position and orientation sensor” at col. 52, line 44; each system calculates the respective pupil position).
Regarding claims 3 and 13, Smyth discloses a device and method wherein the camera system comprises a first camera and a second camera for capturing images of a first eye of a user, and wherein the received image data comprises stereoscopic image data of the first eye (“The light sensors 16, 18 receive the reflected light (i.e., light rays), which is used to determine the gaze direction of the eye. More particularly, the light sensors 16, 18 generate multiple images including two-dimensional "stereo" images of the eye 10 that can be used to calculate the optical axis and visual axis of the eye” at col. 10, line 24; “To this purpose, the invention may be a dual camera system with stereo views of the eye, either direct or from a mirror system” at col. 15, line 27).
Regarding claims 4, 5 and 14, Smyth discloses a device and method wherein:
the set of features measurements comprises one or more pupil feature measurements and one or more glint feature measurements (“In a further embodiment, the stereo reconstructed glint locations and that of the apparent pupil image at the corneal surface may be used in this process; however, as noted the pupil image is refracted by the cornea from the actual location and these approximations are at best useful as bounds on the final computations from the inner eye structure” at col. 39, line 32);
performing the entrance pupil reconstruction process comprises reconstructing a cornea model using the one or more glint feature measurements and reconstructing a pupil model using the one or more pupil feature measurements and the cornea model (“Referring to FIG. 33, showing a schematic section 600 of the eye cornea 642 as a spherical surface with center 644 and radius, an iris defining the pupil, the pupil center 646 located within the anterior surface, an optical axis 650 for the eye located by the corneal spherical center 644 and pupil center 646, a camera with optical axis 654 offset from that of the eye, the pupil center seen in the camera image corresponding to a corneal surface point 652, and the glint point seen in the camera image corresponding to corneal surface point 670 with glint from a light source with bearing offset 672, where the corneal surface points for the pupil center and glint may be determined from the consideration of appropriate angles, as follows” at col. 44, line 44); and the entrance pupil position data is determined based on a ray-tracing process using the pupil model (“FIG. 24 illustrates an example ray-tracing technique 300 performed by the locator 37. In particular, FIG. 24 shows a cross-sectional view of an eye 306 with the geometry of the ray-tracing technique. Here, emitted rays 303, 305 from a light source element 308 are reflected from the corneal surface 310 (as reflected rays 307, 309) to the sensors 336A and 336B from surface reflection points 312 and 314, respectively. The emitted (303, 305) and reflected (307, 309) rays are coplanar with surface normal 316 and 318 drawn from the corneal center 320. The locations of the reflection points 312, 314 on the corneal surface 310 are computed from the source and sensor pairing relations for the emitted (303, 305) and reflected (307, 309) rays. In this manner, the locator 37 computes a geometrical representation of the corneal anterior surface, here represented in the figure by a spherical section with a center and radius, from the set of reflection points (i.e., glint specular reflections) indexed by the light source and sensor pairing over a source activation cycle” at col. 36, line 37).
Regarding claims 6 and 15, Smyth discloses a device and method wherein:
the set of features measurements further comprises one or more limbus feature measurements (“Referring back to the image processing, the pupil image (or that of the limbus), may be approximated as an ellipsoid resulting from a weak-perspective orthographic projection of a circle in three dimensional space followed by scaling” at col. 40, line 37); and
performing the entrance pupil reconstruction process further comprises:
reconstructing a limbus model using the one or more limbus feature measurements (“For a limbus figure of known radius the originating circle may be located in the camera space from the camera focal length, since the center of the figure has been located in the image and the depth distance along the camera optical axis to the limbus is related by the perspective proportion, d=r.sub.L*f.sub.c/r.sub.x, in terms of the major radius; here, the limbus radius r.sub.L=5.5 mm for the standard cornea and f.sub.c is the focal length of the camera” at col. 40, line 64); and
reconstructing a cornea refractive index based on the limbus model and the pupil model, wherein the entrance pupil position data is determined further based on the cornea refractive index (“However, the limbus defined by the iris contained behind the cornea is subject to similar distortions caused by refraction as is the pupil” at col. 41, line 5; “Here, r=sqrt(x.sup.2+y.sup.2), is the radius of the cornea arc about the surface in a plane parallel to the limbus; the limbus is closely circular with a radius of approximately r.sub.L=5.5 mm, and a distance 2.18 mm behind the cornea apex. The iris lies slightly behind the limbus. In a further refinement, the cornea surface may be represented as an elongated ellipsoid” at col. 41, line 30; “In a further embodiment, the accuracy of the simple perspective line-of-sight direction estimate made from the pupil image may be improved by adjusting for the refractive distortions caused by the offset from the camera. FIG. 31 shows corrections to line-of sight estimate made from the pupil image offset that have been computed assuming a spherical corneal surface as approximation. This difference between the corrected and the image perspective derived offset can be expressed in terms of the apparent pupil width seen in a frontal view of the eye” at col. 43, line 36).
Regarding claims 7 and 16, Smyth discloses a device and method wherein the set of feature measurements comprises a pupil feature measurement, and wherein performing the entrance pupil reconstruction process comprises:
estimating a pupil position based on the pupil feature measurement (“Further, the apparent image features of the eye: pupil image and those of inner structures of the eye including the sphincteral pattern of the iris and the retinal capillary network in the image may be matched across views by the template indices” at col. 32, line 31);
determining a set of candidate entrance pupil positions based on the estimated pupil position (“In one embodiment, the process individually compares the isolated features to those of the template and finds candidate matching features based on the Euclidean distance of their feature vectors” at col. 32, line 35); and
selecting a candidate entrance pupil position from the set of candidate entrance pupil positions based on loss values resulting from applying a loss function to each of the candidate entrance pupil positions (“An iterative search through all possible matches selects the most likely feature matching set with the minimal sum of Euclidean distances for the corresponding feature vectors. In this embodiment, the features are individually compared across the image and template and the Euclidean distance is computed for the feature vectors. The most likely candidate provides a linear least square sum as a solution for an accurate fit” at col. 32, line 38; “The computer determines the relation between these features and the visual lines of sight and the relating values are stored in the digital processor of the eye-tracking system by the computer. This result may be in the form of an artificial neural network relating the optical features such as the source specular points and the apparent pupil centers and elliptical axes, or the origin and directional cosines of the optical line of sight, to the location and direction of the visual axis of the eye for each apparatus of the invention” at col. 52, line 57” at col. 52, line 66; notably a neural network utilizes a loss function during training).
Regarding claims 8 and 17, Smyth discloses a device and method wherein the set of feature measurements further comprises a limbus feature measurement (“Referring back to the image processing, the pupil image (or that of the limbus), may be approximated as an ellipsoid resulting from a weak-perspective orthographic projection of a circle in three dimensional space followed by scaling” at col. 40, line 37), and wherein the pupil position is estimated based on the limbus feature measurement (“For a limbus figure of known radius the originating circle may be located in the camera space from the camera focal length, since the center of the figure has been located in the image and the depth distance along the camera optical axis to the limbus is related by the perspective proportion, d=r.sub.L*f.sub.c/r.sub.x, in terms of the major radius; here, the limbus radius r.sub.L=5.5 mm for the standard cornea and f.sub.c is the focal length of the camera” at col. 40, line 64; “In a further embodiment, the accuracy of the simple perspective line-of-sight direction estimate made from the pupil image may be improved by adjusting for the refractive distortions caused by the offset from the camera. FIG. 31 shows corrections to line-of sight estimate made from the pupil image offset that have been computed assuming a spherical corneal surface as approximation. This difference between the corrected and the image perspective derived offset can be expressed in terms of the apparent pupil width seen in a frontal view of the eye” at col. 43, line 36).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Smyth and Miller et al. (US 2019/0243448).
Regarding claims 10 and 18, Smyth discloses the elements of claims 1 and 11 above.
Smyth does not explicitly disclose that the instructions, when executed by the processing circuitry, further causes the processing circuitry to: store data resulting from the entrance pupil reconstruction process as one or more user profile parameters.
Miller et al. teaches a device and method in the same field of endeavor of gaze determination, wherein the instructions, when executed by the processing circuitry, further causes the processing circuitry to:
store data resulting from the entrance pupil reconstruction process as one or more user profile parameters (“To determine the CoR, the display system may identify a location (e.g., three-dimensional location) at which the normal vectors intersect, converge, or are in close proximity, for example, most of the vectors intersect, converge, or are in close proximity or on average the vectors intersect, converge, or are in close proximity. For example, a root mean squared process may be employed. Thus, and as described in FIG. 7A, the identified location may represent a location at which the optical axes intersect. This intersection point may be assigned as the eye's CoR. In some implementations, the distance from the center of the limbus circle to the CoR may be determined and stored for future use. In at least some of these implementations, the system may store such a distance in association with a particular user, and later rely upon the stored distance to determine the CoR for that particular user (e.g., by identifying a position along the normal vector that is the stored distance away from the center of the limbus)” at paragraph 0733, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to store the data of Smyth with a user’s profile as taught by Miller et al. for quick access in subsequent device use.
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Smyth and Ludusan (US 10,061,383).
Regarding claim 19, Smyth discloses an extended reality device for reconstructing entrance pupils, the device comprising:
a camera system (“The light sensors 16, 18 receive the reflected light (i.e., light rays), which is used to determine the gaze direction of the eye. More particularly, the light sensors 16, 18 generate multiple images including two-dimensional "stereo" images of the eye 10 that can be used to calculate the optical axis and visual axis of the eye” at col. 10, line 24; “To this purpose, the invention may be a dual camera system with stereo views of the eye, either direct or from a mirror system” at col. 15, line 27); and
processing circuitry coupled to memory storing instructions that, when executed by the processing circuitry (“Further, the components of the visual axis calculation system may consist of digital processors in the form of computer routines with operating systems comprising dedicated digital computer programs” at col. 11, line 18), causes the processing circuitry to:
receive image data from the camera system (“The light sensors 26, 27 detect the light reflected from the user's eye via the optical systems from the corresponding light collecting optics, and output analog image signals to the opto-electronic device 30, under control of the controller via control line 31; in turn, the opto-electronic device 30 processes the image signals with digital output to the image processor 34 under control via control line 32” at col. 11, line 6);
extract a set of features from the image data, wherein extracting the set of features comprises classifying the features into a plurality of feature types (“In one embodiment, these parameters comprise the image locations of the source specular points, and the apparent pupil centers and elliptical axes isolated for the images. In other embodiments, the parameters comprise the origins and directional cosines of the optical axes. In the calibration process, the digital computer receives as input the optical features (e.g., the origins and directional cosines of the optical axes of sight) in lieu of that for the visual lines of sight as the user looks at visual cues presented under computer control. The computer determines the relation between these features and the visual lines of sight and the relating values are stored in the digital processor of the eye-tracking system by the computer. This result may be in the form of an artificial neural network relating the optical features such as the source specular points and the apparent pupil centers and elliptical axes, or the origin and directional cosines of the optical line of sight, to the location and direction of the visual axis of the eye for each apparatus of the invention” at col. 52, line 57);
perform a feature measurement process to calculate a set of feature measurements the features are represented by the directional cosines for the measured pupil centers), the feature measurement process comprising:
select a process from a set of entrance pupil reconstruction processes to be performed based on the set of feature measurements (“Referring to FIG. 26, the expanded process operating with the routine 400 computes 402 line-of sight estimates by triangulation from the distribution of glints about the pupil image center for the images; computes 401 estimates of the sight direction for all images from the pupil image offsets; checks 403 that the pupil offset estimates are bounded by the glint gaze estimates and if so, selects 405 the pupil offset estimates as bounds, otherwise 404 uses the glint sight estimates; computes 406 the sight estimates from the internal eye structure the locations of which are determined by template matching; checks 407 that the template estimates are bounded, and if not selects 409 the weighted average of the bounds as the sight direction, otherwise 408 uses the template estimate as such” at col. 39, line 52); and
perform the selected process (the conditions above are checked to determine how to establish the sight directions).
Smyth does not explicitly disclose that the feature measurement process comprising: performing a quantity assessment on the features for each of the feature types; and performing a quality assessment for each of the features.
Ludusan teaches a device in the same field of endeavor of point of gaze estimation, wherein the feature measurement process comprising:
performing a quantity assessment on the features for each of the feature types (“For the no-glint-occlusion case 2A, for each eye, the following primary eye components are used for the PoG estimation: bright pupil 201, iris 203 (the used gaze aspects), glint 202, nasal eye corner 204 and temporal eye corner 205 (the used reference aspects)” at col. 6, line 13; “(h′) if the glint 202 was not detected 309” at col. 7, line 4; therefore, the determination of the glint presence constitutes a quantification assessment of how many features are present); and
performing a quality assessment for each of the features (“(j) if the error ε between the PCCR PoG and the PCNL/PCTC PoGs is higher than a predetermined threshold 315” at col. 6, line 56);
select a process from a set of entrance pupil reconstruction processes to be performed based on the set of feature measurements (“(k) using a predefined combination scheme, combine 317 the PoG stack into a final PoG estimate” at col. 6, line 62; “(k′) if the discrepancy between the virtual glint PCCR PoG estimation is above a predefined threshold value, use only the PCTC 208 and PCNL 207 PoG estimation to compute 317 the final PoG estimate. If the PCCR PoG estimation is below the threshold value, employ the same combination scheme as the one used for the 2A case” at col. 7, line 9).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the feature assessment as taught by Ludusan in the process of Smyth to be able to choose the appropriate gaze estimation method based upon the presence or absence of particular feature data.
Allowable Subject Matter
Claims 9 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art does not teach or disclose that performing the entrance pupil reconstruction process comprises using one or more user profile parameters to compensate for missing feature measurements as required by claim 9; upon determining that the effective count for the pupil feature type is below a predetermined threshold number, the selected process is a mono-based entrance pupil reconstruction process; and upon determining that the effective count for the pupil feature type is above a predetermined threshold number, the selected process is a stereo-based entrance pupil reconstruction process as required by claim 20.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672