DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 7-20 are rejected under 35 U.S.C. 103 as being unpatentable over Haeberling et al. (US 20230316810, hereinafter Haeberling) in view of Stafford et al. (US 20190384381, hereinafter Stafford).
Regarding Claim 1, Haeberling discloses a method for enabling projection of images from a digital display ([0022] FIG. 1, 3D content system 100 for capturing and displaying content in a stereoscopic display device used by multiple users to, for example, conduct videoconference communications in 3D (e.g., 3D telepresence sessions)), the method comprising:
obtaining face image data and eye region image data for one or more viewers within a field of view of at least one camera in proximity to a 3D-enabled digital display ([0021] FIG. 1, track facial features of a user (e.g., a user of a 3D telepresence system) without using wearable devices (e.g., a HMD, markers, and/or the like using at least one camera to capture images at regular intervals);
detecting face and eye landmarks for the one or more viewers in one or more image frames based on the face image data ([0019], having an accurate up-to-date 3D pose of the user's facial features (e.g., eyes and ears) used to modify the displayed scene);
determining head pose information based on the face image data and eye region image data ([0031], enable accurately determining user position (e.g., user eyes) to enable generation of realistic 3D. The systems and techniques described herein may reconfigure the image content projected from the display to ensure that the user can move around, but still experience proper parallax, low rates of distortion, and realistic 3D images in real time);
determining eye tracking information for each of the one or more viewers based on the face image data, eye region image data, and head pose information ([0068] FIG. 3, tracking facial features and position determining position(s) based on the facial features with the location of facial landmarks (or key points) of a face that includes eyes, mouth, ears), the eye tracking information including
a) a point of regard (PoR) of each eye of each of the one or more viewers ([0022] FIG. 1, video can be projected to render 3D video based on the position of a viewer and to project audio based on the position of participants [one or more viewers] in the video conference),
b) eye state of each eye of each of the one or more viewers ([0070] FIG. 4 , facial feature positions for the pupils of the left eye and the right eye respectively), and
e) a position of each eye of each of the one or more viewers relative to the 3D- enabled digital display ([0068] FIG. 3, tracking facial features and position determining position(s) based on the facial features with the location of facial landmarks (or key points) of a face that includes eyes, mouth, ears); and
determining a number of projections and a distribution of projections for each eye of each of the one or more viewers based on the eye tracking information ([0022] FIG. 1, project video and audio content in such a way to improve a video conference such as projecting 3D video based on the position of a viewer and to project audio based on the position of participants in the video conference; [0031], image content projected from the display to ensure that the user can move around, but still experience proper parallax, low rates of distortion, and realistic 3D images in real time).
Haeberling does not explicitly disclose c) gaze direction of each eye of each of the one or more viewers, and d) eye region illumination information for each eye of each of the one or more viewers.
Stafford teaches c) gaze direction of each eye of each of the one or more viewers ([0033] analyze image may to identify eye location based on corneal reflections in the image data, and determine gaze direction based on a relative location of the pupils in the image), and d) eye region illumination information for each eye of each of the one or more viewers ([0034] Bright Pupil tracking involves illumination of the eyes with a light source that is substantially in line with the optical axis of the camera, causing the emitted light to be reflected off of the retina and back to the camera through the pupil).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of gaze direction and eye region illumination as taught by Stafford ([0033]) into the eye tracking system of Haeberling in order to enable utilizing eye tracking to reduce errors in eye tracking so that greater radius of high resolution on a display is theoretically needed to preserve high fidelity for the user with reduced computation complexity (Stafford, [0023]).
Regarding Claim 2, Haeberling in view of Stafford discloses the method of claim 1,
Stafford discloses wherein the obtaining face image data and eye region image data comprises receiving at least one digital intensity image, wherein the at least one digital intensity image includes at least one visible eye region ([0034], Bright Pupil tracking involves illumination of the eyes with a light source that is substantially in line with the optical axis of the camera, causing the emitted light to be reflected off of the retina and back to the camera through the pupil. The pupil presents in the image as an identifiable bright spot at the location of the pupil, similar to the red eye effect which occurs in images during conventional flash photography. In this method of gaze tracking, the bright reflection from pupil itself helps the system locate the pupil if contrast between pupil and iris is not enough). The same reason or rational of obviousness motivation applied as used above in claim 1.
Regarding Claim 7, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses wherein the detecting face and eye landmarks for the one or more viewers in one or more image frames comprises applying a deep learning inference algorithm to image input to provide a bounding box for each detected face in the one or more image frames ([0033] The 3D systems 106, 108 can include face finder/recognition tools such as machine learned (ML) tools (e.g., software) configured to identify faces in an image and extract facial features and the position (or x, y, z location) of the facial features).
Regarding Claim 8, Haeberling in view of Stafford discloses the method of claim 7,
Stafford discloses further comprising applying a deep learning inference algorithm to the bounding box for each detected face to provide a set of face and eye landmarks for each bounding box ([0075], he foveal region may be eliminated completely when it determined that a saccade is either occurring or about to occur, and a new foveal region and peripheral/transition region boundaries may be established based on gaze tracking data 200 obtained during the saccade; [0087] FIG. 6A) The same reason or rational of obviousness motivation applied as used above in claim 1.
Regarding Claim 9, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses wherein the eye tracking information is determined based on a) mapping the eye region image data to a Cartesian coordinate system, and b) unprojecting the pupil and limbus of both eyeballs onto the Cartesian coordinate system to give 3D contours of each eyeball ([0068], the 2D facial feature extraction identifies a face of a user (e.g., a participant in a 3D telepresence communication) and extract the facial features of the identified face; [0070] FIG. 4, numbers 0 and 1 can be the pupils of the left eye and the right eye respectively. The facial feature extractor can output two variables including the landmark number and a pixel number. The pixel number can be based on the resolution (e.g., number of pixels) of the image. For example, the image can have a resolution of 2048×1536 (e.g., a 3-megapixel camera). Therefore, the image can have 3,145,728 pixels with the first pixel in the upper left corner of the image and the last pixel being in the lower right corner of the image).
Regarding Claim 10, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses wherein the 3D-enabled digital display comprises one or more autostereoscopic displays ([0028], multiple types of 3D display technology to provide an autostereoscopic view for the respective viewer).
Regarding Claim 11, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses wherein the one or more autostereoscopic displays comprises at least one of a holographic display, a volumetric display, a compressive light field display, or an integral imaging display ([0028], multiple types of 3D display technology to provide an autostereoscopic view for the respective viewer).
Regarding Claim 13, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses wherein the obtaining face image data and eye region image data for one or more viewers within a field of view of at least one camera in proximity to a 3D- enabled digital display is performed by at least one of a laptop camera, a tablet camera, a smartphone camera, or a digital external camera ([0022] FIG. 1, 3D content system 100 for capturing and displaying content in a stereoscopic display device used by multiple users to, for example, conduct videoconference communications in 3D (e.g., 3D telepresence sessions)).
Regarding Claim 14, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling in view of Stafford discloses wherein the obtaining face image data and eye region image data for one or more viewers within a field of view of at least one camera in proximity to a 3D- enabled digital display is performed using only ambient light ([0022] FIG. 1, 3D content system 100 for capturing and displaying content in a stereoscopic display device used by multiple users to, for example, conduct videoconference communications in 3D (e.g., 3D telepresence sessions)).
Regarding Claim 15, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses wherein the obtaining face image data and eye region image data for one or more viewers within a field of view of at least one camera in proximity to a 3D-enabled digital display is performed without active illumination ([0031] 3D displays, there may be a single location that provides a 3D view of image content (e.g., users, objects, etc.) provided by such displays. A user may be seated in the single location to experience proper parallax, little distortion, and realistic 3D images).
Regarding Claim 16, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses further comprising: detecting degradation in the eye region image data of a viewer; and switching to a different camera based on the degradation in the eye region image data([0026], plurality of cameras can be used to capture two or more images such as two or more images can be captured sequentially by the same camera (e.g., for tracking). Two or more images can be captured at the same time by two or more cameras (e.g., to be used for triangulation)).
Regarding Claim 17, Haeberling in view of Stafford discloses the method of claim 16,
Haeberling discloses wherein the switching to a different camera based on the degradation in the eye region image data comprises switching to a different camera that can capture eye region image data of both eyes of the viewer at or above a minimum resolution level ([0026], plurality of cameras can be used to capture two or more images such as two or more images can be captured sequentially by the same camera (e.g., for tracking). Two or more images can be captured at the same time by two or more cameras (e.g., to be used for triangulation)).
Regarding Claim 18, Haeberling in view of Stafford discloses the method of claim 1,
Haeberling discloses further comprising analyzing the eye region image data for at least one of engagement with the 3D-enabled digital display, fixation, or saccade ([0031], 3D displays, there may be a single location that provides a 3D view of image content (e.g., users, objects, etc.) provided by such displays. A user may be seated in the single location to experience proper parallax, little distortion, and realistic 3D images)
Regarding Claim 19, system claim 66 of using the corresponding method claimed in claim 1, and the rejections of which are incorporated herein for the same reasons as used above.
Regarding Claim 20, computer program product claim 67 of using the corresponding method claimed in claim 1, and the rejections of which are incorporated herein for the same reasons as used above.
Claims 3-6 are rejected under 35 U.S.C. 103 as being unpatentable over Haeberling et al. (US 20230316810, hereinafter Haeberling) in view of Stafford et al. (US 20190384381, hereinafter Stafford) and Das et al. (US 20230418054, hereinafter Das).
Regarding Claim 3, Haeberling in view of Stafford discloses the method of claim 1, but does not explicitly disclose wherein the obtaining face image data further comprises associating at least one digital user identifier with each face in the face image data.
Das discloses wherein the obtaining face image data further comprises associating at least one digital user identifier with each face in the face image data ([0230] At block 1406, the wearable system may attempt to identify the current user by performing an identification process. As one example, the wearable system may estimate the IPD of the current user to determine if the current user's IPD matches a calibrated user's IPD (e.g., if the two IPDs are within some threshold of each other). In some embodiments, the wearable system may determine that the current user is a calibrated user if the calibrated IPD and the current user's IPD are within a threshold of, e.g., 0.5 mm, 1.0 mm, 1.5 mm, or 2.0 mm, of the calibrated user's IPD. In general, larger thresholds may facilitate faster determinations and help to ensure that calibrated users are identified and their calibration parameters used; for example, larger thresholds are biased towards finding that the current user is the calibrated user. In at least some embodiments, the wearable system may be able to determine a current user's IPD with relatively high accuracy (e.g., 95%) within a relatively short time frame (e.g., 5-7 seconds of eye tracking data, which may correspond to between about 150 and 200 frames of eye tracking images).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of digital user identifier as taught by Das ([02303]) into the eye tracking system of Haeberling & Stafford in order to provide the wearable system to render content on a depth plane that generally reduces or minimizes any vergence-accommodation mismatches, which could otherwise lead to user discomfort and fatigue (Das, [0227]).
Regarding Claim 4, Haeberling in view of Stafford and Das discloses the method of claim 3,
Das discloses wherein the at least one digital user identifier comprises at least one unique digital user identifier ([0230] At block 1406, the wearable system may attempt to identify the current user by performing an identification process. As one example, the wearable system may estimate the IPD of the current user to determine if the current user's IPD matches a calibrated user's IPD (e.g., if the two IPDs are within some threshold of each other). In some embodiments, the wearable system may determine that the current user is a calibrated user if the calibrated IPD and the current user's IPD are within a threshold of, e.g., 0.5 mm, 1.0 mm, 1.5 mm, or 2.0 mm, of the calibrated user's IPD. In general, larger thresholds may facilitate faster determinations and help to ensure that calibrated users are identified and their calibration parameters used; for example, larger thresholds are biased towards finding that the current user is the calibrated user. In at least some embodiments, the wearable system may be able to determine a current user's IPD with relatively high accuracy (e.g., 95%) within a relatively short time frame (e.g., 5-7 seconds of eye tracking data, which may correspond to between about 150 and 200 frames of eye tracking images).
Regarding Claim 5, Haeberling in view of Stafford and Das discloses the method of claim 3,
Das discloses wherein the at least one digital user identifier comprises at least one anonymized digital user identifier ([0230] At block 1406, the wearable system may attempt to identify the current user by performing an identification process. As one example, the wearable system may estimate the IPD of the current user to determine if the current user's IPD matches a calibrated user's IPD (e.g., if the two IPDs are within some threshold of each other). In some embodiments, the wearable system may determine that the current user is a calibrated user if the calibrated IPD and the current user's IPD are within a threshold of, e.g., 0.5 mm, 1.0 mm, 1.5 mm, or 2.0 mm, of the calibrated user's IPD. In general, larger thresholds may facilitate faster determinations and help to ensure that calibrated users are identified and their calibration parameters used; for example, larger thresholds are biased towards finding that the current user is the calibrated user. In at least some embodiments, the wearable system may be able to determine a current user's IPD with relatively high accuracy (e.g., 95%) within a relatively short time frame (e.g., 5-7 seconds of eye tracking data, which may correspond to between about 150 and 200 frames of eye tracking images).
Regarding Claim 6, Analogous rejection as the rejection of Claim 3 applies.
Claim 12 are rejected under 35 U.S.C. 103 as being unpatentable over Haeberling et al. (US 20230316810, hereinafter Haeberling) in view of Stafford et al. (US 20190384381, hereinafter Stafford) and Swedish et al. (US 20160320837, hereinafter Swedish).
Regarding Claim 12, Haeberling in view of Stafford discloses the method of claim 1, but does not explicitly disclose wherein the obtaining face image data and eye region image data for one or more viewers within a field of view of at least one camera in proximity to a 3D-enabled digital display is performed by a camera at a distance of at least 0.2 meters from at least one of the plurality of viewers.
Swedish discloses the method of claim 48, but does not explicitly disclose wherein the obtaining face image data and eye region image data for one or more viewers within a field of view of at least one camera in proximity to a 3D-enabled digital display is performed by a camera at a distance of at least 0.2 meters from at least one of the plurality of viewers ([0099], the camera acquires the RR images at a distance from the eye that is: (a) at least 10 cm; (b) at least 50 cm; (c) at least 1 meter; (d) at least 3 meters; (e) at least 10 meters; (f) at least 100 meters; or (g) at least 1000 meters; [0100] the camera acquires clear, detailed images of a subject's retina even when the subject is so far away from the camera that the rest of the face is below the diffraction threshold of the camera, and thus the rest of the face is not discernable in images captured by the camera)
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of digital display is performed by a camera at a distance as taught by Swedish ([0099]) into the eye tracking system of Haeberling & Stafford in order to provide clear, detailed images of the retina may be acquired at a distance of thousands of meters, when the subject's face (other than the retina) is not discernable (Swedish, [0023]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samuel D Fereja whose telephone number is (469)295-9243. The examiner can normally be reached 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DAVID CZEKAJ can be reached on (571) 272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMUEL D FEREJA/Primary Examiner, Art Unit 2487
Prosecution Insights