DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8, 10-12, 14-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over BIRDWELL (US 2005/0250579 A1) in view of Martinez (US 2023/0116638 A1 and O’CONNOR et al. (US 2015/0302658 A1).
RE claim 1, Birdwell in view of Martinez and O’Connor teaches a method comprising:
(a)
obtaining tracking data for a subject comprising a set of markers associated with an eye region;
Birdwell teaches a method for generating relatively realistic virtual eyes for a character in a virtual environment [abstract, 0018]. Fig. 4A, Birdwell teaches sclera texture (404) is positioned in the openings bounded by the virtual upper and lower eyelids for each eye of the character [0055]. Fig. 6, Birdwell teaches intersecting spheres are employed to determine the position of virtual corneas and virtual eyeballs in the openings bounded by the virtual upper and lower eyelids (said eye region) of the character [0057]. Based in part on the direction that the character is looking, the spheres are positioned in the openings. The subsequent direction/orientation of the spheres is employed to determine the position for each virtual iris in the opening bounded by the virtual upper and lower eyelids (said eye region) [0058]. Thus, Birdwell teaches determining an eye region but fails to disclose obtaining tracking data for a subject associated with an eye region.
Martinez teaches a system/method for locating a point of gaze onto a screen based on an image of a face of a user (said subject) [abstract]. Figs. 1, 3a, an image is acquired, and landmarks are determined [0011-0012, 0018, 0208]. Martinez teaches an eye landmark, which is a point of an eye which, together with other eye landmarks, marks the shape of said eye [0023, 0209, Fig. 3b]. As an example, an eye landmark (said set of markers associated with an eye region) may be the left or right corner of an eye, a point of the eyelid of the center of the eyeball [0023]. Martinez further teaches determining the first gaze vector (said tracking data) to determine where the first eye is looking with respect to the eyeball center (said marker) [0033].
Thus, Martinez teaches determining eye-border landmark locations of a face of a user based on an image. Birdwell also determines the location of the eyes based on the upper and lower eyelids of a virtual character which is also an image. Therefore, it would have been obvious before the effective filing date of the claimed invention to utilize the landmark locations of Martinez with the eye detection and ultimately the gaze vector of Birdwell because Martinez’s eye landmark detection would improve gaze estimation accuracy.
(b)
obtaining, for each eye of a subject, an iris region based on the set of markers associated with the eye region;
In further view of Martinez, Martinez further teaches the eye landmark may be an iris landmark [0023]. Furthermore, Martinez teaches a first and/or second face landmark [0024]. Thus, each iris can be determined.
(c)
determining a viewing direction of the subject;
In further view of Martinez, Martinez teaches determining a gaze vector that estimates the 3D direction, in which the eye contained in the first ROI is looking [0033].
(d)
obtaining a lighting map for an environment; and
Birdwell teaches graphics engine (308) may include an ambient light component for rendering ambient light effects [0050]. Additional components may exist that are employable for managing and storing such information such as environment data files [0050]. Birdwell additionally teaches external light source (602) provides light that is incident upon spheres (506, 508) [0067]. Line segment (610A) represents light incident on virtual cornea sphere (606) and line segment (610B) represents reflected light from external light source (602) that is visible to character (608) in the virtual environment [0067]. However, Birdwell is silent to using a lighting map.
In further view of O’Connor, O’Connor teaches an augmented reality system that can deliver augmented reality content to the user [0234]. The augmented reality system may include physical space or room-based sensor systems [0257, Fig. 7]. The room-based sensor systems (704) detect and/or collect information from a physical environment [0257]. O’Connor further teaches ambient light sensing [0258-0259]. The AR system may additionally or alternatively optimize rendered virtual content based at least in part on one or more characteristics of the particular ambient environment [0380]. O’Connor teaches virtual content may be dynamically altered based on the detected light of the ambient environment [0380], i.e., emulate the lighting system incident to the environment in which it is super-imposed [0382]. O’Connor teaches Image Based Lighting (IBL) where IBL captures real world lighting photographically to render virtual content that appears indistinguishable from real objects [0384]. IBL light maps can be selected to illuminate a particular virtual scene [0384]. Fig. 20, the method of O’Connor teaches capturing an image of the ambient surroundings through the user’s FOV cameras [0387]. A light map is selected having the closest approximation (1808) (said obtaining a lighting map for an environment) and renders the virtual object based on the selected light map (1810) [0388].
It would have been obvious before the effective filing date of the claimed invention to utilize the obtained light map of O’Connor as the information from the external light source discussed by Birdwell because the light map of O’Connor provides the light information of the environment which can provide a more accurate glint to the eye of Birdwell.
(e)
generating virtual representation data for a virtual representation of the subject by applying a lighting effect to the iris region in accordance with the lighting map and the viewing direction.
Birdwell teaches a method for generating relatively realistic virtual eyes for a character in a virtual environment [abstract, 0018]. Fig. 4A, Birdwell teaches sclera texture (404) is positioned in the openings bounded by the virtual upper and lower eyelids for each eye of the character [0055]. Fig. 6, Birdwell teaches intersecting spheres are employed to determine the position of virtual corneas and virtual eyeballs in the openings bounded by the virtual upper and lower eyelids (said eye region) of the character [0057]. Based in part on the direction that the character is looking, the spheres are positioned in the openings. The subsequent direction/orientation of the spheres is employed to determine the position for each virtual iris in the opening bounded by the virtual upper and lower eyelids (said eye region) [0058]. Birdwell teaches eye engine (312) that is configured to determine and provide the textures, and glint for rendering virtual eyes for a character in a virtual environment (said generating virtual representation data for a virtual representation of the subject by applying a lighting effect to the iris region) [0052]. A relatively light spot or additive “glint” texture is layered in a position on top of the virtual iris for the transparent texture that is based at least in part of a summation of light sources that are illuminating the character in a scene of the virtual environment [0019]. The additive glint texture is also positioned on top of the virtual iris based at least in part on the direction that the character is looking (said viewing direction) [0019, 0059]. Graphics engine (308) may also include an ambient light component for rendering ambient light effects [0050]. In the combined invention, the light map of O’Connor would be relied upon for providing the illumination of Birdwell (see claim 1(d)). Martinez provides a means for also determining the iris (see claim 1(b)).
RE claim 3, Birdwell in view of Martinez and O’Connor teaches
(a)
wherein the environment corresponds to an environment in which the virtual representation of the subject is to be presented.
Birdwell teaches displaying realistic virtual eyes for a character in a virtual environment [0018]. In further view of O’Connor, O’Connor teaches the augmented reality system may include physical space or room-based sensor systems [0257, Fig. 7]. The room-based sensor systems (704) detect and/or collect information from a physical environment [0257]. O’Connor further teaches ambient light sensing [0258-0259]. The AR system may additionally or alternatively optimize rendered virtual content based at least in part on one or more characteristics of the particular ambient environment [0380]. O’Connor teaches virtual content may be dynamically altered based on the detected light of the ambient environment (said wherein the environment corresponds to an environment in which the virtual representation of the subject is to be presented) [0380].
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 4, Birdwell in view of Martinez and O’Connor teaches
(a)
wherein the environment corresponds to a physical environment in which the subject is located.
Birdwell teaches displaying realistic virtual eyes for a character in a virtual environment [0018]. In further view of O’Connor, O’Connor teaches the augmented reality system may include physical space or room-based sensor systems [0257, Fig. 7]. The room-based sensor systems (704) detect and/or collect information from a physical environment [0257]. O’Connor further teaches ambient light sensing [0258-0259]. The AR system may additionally or alternatively optimize rendered virtual content based at least in part on one or more characteristics of the particular ambient environment [0380]. O’Connor teaches virtual content may be dynamically altered based on the detected light of the ambient environment (said wherein the environment corresponds to a physical environment in which the subject is located) [0380].
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 5, Birdwell in view of Martinez and O’Connor teaches
(a)
lighting effect is further applied in accordance with an additional lighting map for an additional environment.
Birdwell teaches displaying realistic virtual eyes for a character in a virtual environment [0018]. Additive glint texture (616) (said lighting effect) is disposed at the point where the light incident upon virtual eyeball sphere (604) is reflected back to character (608) [0068].
In further view of O’Connor, O’Conner teaches the AR system may additionally or alternatively optimize rendered virtual content based at least in part on one or more characteristics of the particular ambient environment [0380]. O’Connor teaches virtual content may be dynamically altered based on the detected light of the ambient environment [0380]. A light map is selected having the closest approximation (1808) (said lighting map) and renders the virtual object based on the selected light map (1810) [0388]. Additionally, O’Connor teaches a passable world model (said additional environment) that allows a user to effectively pass over a piece of the user’s world (e.g., ambient surroundings, interactions, etc.) to another user. Each user's respective individual AR system captures information as the user passes through or inhabits an environment, which the AR system processes to produce a passable world model [0269]. Therefore, each user’s respective world model would include the lighting map that pertains to their model (said additional lighting map). Thus, in the situation when a user enters another user’s world model, the information associated with the other world would be ap9plied (said lighting effect is further applied in accordance with an additional lighting map for an additional environment).
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 6, Birdwell in view of Martinez and O’Connor teaches
(a)
wherein the lighting effect comprises adjusting a brightness to one or more regions of the eye of a virtual representation of the subject in accordance with the lighting map.
Birdwell teaches graphics engine (308) include an ambient light component for rendering ambient light effects [0050]. Eye engine (312) is configured to determine and provide the textures, and glint for rendering virtual eyes for a character in the virtual environment [0052]. Fig. 6, line segment (612A) represents light incident upon virtual eyeball sphere (604) and line segment (612B) represents reflected light from external light source (602) that is visible to character (608) [0068]. Additive glint texture (616) (said lighting effect adjusting a brightness) is disposed at the point where the light incident upon virtual eyeball sphere (604) is reflected back to character (608) [0068]. As seen in Fig. 4D, the additive glint texture (410) would be light, which inherently would be a change of brightness, to indicate the light incident upon the virtual eyeball sphere.
O’Connor is relied upon as teaching the art of applying a light map to a virtual environment determined by the physical environment [0257-0259, 0380, 0382, 0384, 0388]. O’Connor teaches virtual content may be dynamically altered based on the detected light of the ambient environment [0380]. A light map is selected having the closest approximation (1808) (said lighting map) and renders the virtual object based on the selected light map (1810) [0388]. The virtual content may be dynamically altered based on the detected light of the ambient environment [0380], i.e., emulate the lighting system incident to the environment in which it is super-imposed [0382].
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 7, Birdwell in view of Martinez and O’Connor teaches
(a)
wherein the lighting effect comprises reflecting a portion of the lighting map onto the iris region in accordance with the viewing direction.
Birdwell teaches graphics engine (308) include an ambient light component for rendering ambient light effects [0050]. Eye engine (312) is configured to determine and provide the textures, and glint for rendering virtual eyes for a character in the virtual environment [0052]. Fig. 6, line segment (612A) represents light incident upon virtual eyeball sphere (604) and line segment (612B) represents reflected light from external light source (602) that is visible to character (608) [0068]. Additive glint texture (616) (said lighting effect) is disposed at the point where the light incident upon virtual eyeball sphere (604) is reflected back to character (608). The orientation of spheres (606, 608) in the direction that the character is looking and the direction of reflected light from these spheres is employed to project the positions of additive glint textures (616, 614) on the sclera texture and virtual iris, respectively (said in accordance with the viewing direction) [0068].
O’Connor is relied upon as teaching the art of applying a light map to a virtual environment determined by the physical environment [0257-0259, 0380, 0382, 0384, 0388]. The virtual content may be dynamically altered based on the detected light of the ambient environment [0380], i.e., emulate the lighting system incident to the environment in which it is super-imposed 0382]. A light map is selected having the closest approximation (1808) (said lighting map) and renders the virtual object based on the selected light map (1810) [0388].
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 8, claim 8 recites similar limitations as claim 1 but in manufacturing form. Therefore, the same rationale used for claim 1 is applied. Furthermore, Birdwell teaches computer-readable media for storage of computer readable instructions [0039].
RE claim 10, claim 10 recites similar limitations as claim 3 but in manufacturing form. Therefore, the same rationale used for claim 3 is applied.
RE claim 11, claim 11 recites similar limitations as claim 4 but in manufacturing form. Therefore, the same rationale used for claim 4 is applied.
RE claim 12, claim 12 recites similar limitations as claim 5 but in manufacture form. Therefore, the same rationale used for claim 5 is applied.
RE claim 14, claim 14 recites similar limitations as claim 6 but in manufacturing form. Therefore, the same rationale used for claim 6 is applied.
RE claim 15, Birdwell teaches
(a)
apply an additional lighting effect to a portion of the virtual representation of the subject comprising an eye region and excluding the iris region.
Fig. 4D, Birdwell displays a character with relatively realistic virtual eyes rendered by the combined layering of sclera texture (404), transparent texture with virtual iris (406), additive glint texture (408) (over the virtual iris), and another additive glint texture (410) (over the sclera texture) in the openings bounded by the virtual upper and lower eyelids for the character. Additive glint texture (410) is layered on top of transparent texture (406) at a position that is over sclera texture (404), not over the virtual iris (said applying addition lighting effect to a portion … excluding the iris region) [0060].
RE claim 16, Birdwell in view of Martinez and O’Connor teaches
(a)
wherein the viewing direction is determined based on a head pose.
In further view of Martinez, Martinez teaches constructing a head pose estimation vector by using at least the first and second face landmark location [0073]. Martinez further teaches locating the first point of gaze onto the screen is based on the head pose estimation vector [0074].
The same motivation to combine as taught in the rationale of claim 8 is incorporated herein.
RE claim 17, Birdwell in view of Martinez and O’Connor teaches
(a)
wherein the viewing direction is determined based on a gaze vector.
Birdwell teaches determining the direction the character is looking (said viewing direction) [0057]. However, Birdwell does not discuss gaze vectors.
In further view of Martinez, Martinez teaches the first gaze vector estimates the three-dimensional direction, in which the eye contained in the first ROI is looking. The first gaze vector may be a 3D unit vector [0033].
The same motivation to combine as taught in the rationale of claim 8 is incorporated herein.
RE claim 18, claim 18 recites similar limitations as claim 1 but in system form. Therefore, the same rationale used for claim 1 is applied. Furthermore, Birdwell teaches processing unit (212) [0038] and computer-readable media for storage of computer readable instructions [0039].
RE claim 20, claim 20 recites similar limitations as claim 6 but in system form. Therefore, the same rationale used for claim 6 is applied.
Claims 2, 9, 19 are rejected under 35 U.S.C. 103 as being unpatentable over BIRDWELL (US 2005/0250579 A1) in view of Martinez (US 2023/0116638 A1 and O’CONNOR et al. (US 2015/0302658 A1) as applied to claims 1, 8 and 18 respectively, and in further view of KRISHNA et al. (US 10,617,294 B1).
RE claim 2, Birdwell in view of Martinez and O’Connor teaches where obtaining the iris region comprises:
(a)
determining an eye region based on the set of markers, wherein the set of markers correspond to a set of points on the eye opening, wherein the set of markers are each associated with location information; and
Birdwell teaches determining eye regions based on the upper and lower eyelids. Birdwell also teaches determining the position of each virtual iris [0057].
Martinez is relied upon as teaching the art of determining eye regions based on markers. Figs. 1, 3a, an image is acquired, and landmarks are determined [0011-0012, 0018, 0208]. Martinez teaches an eye landmark, which is a point of an eye which, together with other eye landmarks, marks the shape of said eye [0023, 0209, Fig. 3b]. As an example, an eye landmark (said set of markers associated with an eye region) may be the left or right corner of an eye, a point of the eyelid of the center of the eyeball (said markers correspond to a set of points on the eye opening, wherein the set of markers are each associated with location information) [0023]. Martinez further teaches the eye landmark may be an iris landmark [0023]. Furthermore, Martinez teaches a first and/or second face landmark [0024]. Thus, each iris can be determined. The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
(b)
identifying the iris region within the eye region based on a color differential among pixels in image data comprising the eye region.
As taught in the rationale of claim 2(a), Birdwell in view of Martinez teaches determining the iris region based on markers. However, Birdwell in view of Martinez does not discuss identifying the iris based on a color differential.
Krishna teaches extracting the eyes from a facial image based on the facial landmarks [abstract]. An eye extraction module (206) extracts the eyes from the facial image of the subject (102) by identifying the eyes based on the position of the plurality of the facial landmarks [9:16-17]. The red reflex determination module (210) determines the red reflex in the iris of the eye (said based on a color differential) by (a) automatically estimating an average lightness value of all the pixels in an eye bounding box associated with the iris of the eye; (b) automatically identifying a plurality of iris shades in the eye by classifying an angle representing hue; (c) automatically segmenting, using a sliding-window algorithm, the iris as the region with the highest concentration of dark-colored pixels, and (d) automatically trimming the iris region of the image based on the red channel in the eye to obtain the red reflex image of the iris [9:61-10:5].
It would have been obvious before the effective filing date of the claimed invention to apply Krishna’s pixel-based iris segmentation to the eye image region located using the landmarks of Martinez. Martinez already obtains eye and iris landmark locations for gaze tracking. Krishna teaches a way to distinguish the iris pixels within a landmark-located eye region. By utilizing Krishna within the combined invention of Birdwell in view of Martinez and O’Connor, it would provide a pixel-defined iris region for the iris of Birdwell.
RE claim 9, claim 9 recites similar limitations as claim 2 but in manufacture form. Therefore, the same rationale used for claim 2 is applied.
RE claim 19, claim 19 recites similar limitations as claim 2 but in system form. Therefore, the same rationale used for claim 2 is applied.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over BIRDWELL (US 2005/0250579 A1) in view of Martinez (US 2023/0116638 A1 and O’CONNOR et al. (US 2015/0302658 A1) as applied to claim 8, and in further view of KATZ et al. (2017/0352183 A1).
RE claim 13, in further view of Martinez, Martinez is relied upon as determining landmarks (said markers) from a captured image [0011-0012, 0018, 0208]. Martinez teaches an eye landmark, which is a point of an eye which, together with other eye landmarks, marks the shape of said eye [0023, 0209, Fig. 3b]. However, Birdwell in view of Martinez and O’Connor does not discuss the set of markers that are obtained from sensors worn by the subject.
Katz teaches
(a)
wherein the set of markers are obtained from sensor data captured by one or more sensors of a device worn by the subject, and
Fig. 1, Katz teaches a VR system (100) that operates in augmented reality and/or mixed reality environments. The system (100) comprises an HMD (105), imaging device (135), and a VR input interface (140) [0012]. The HMD (105) presents content to a user [0013]. The HMD (105) includes a facial tracking system (160) (said sensors of a device worn by the subject) [0014]. The facial tracking system (160) tracks portions of a face of a user (e.g., including eyes of the user) [0023]. The facial tracking system (160) collects calibration attributes, which describes landmarks of the face covered by the HMD (105) (said markers obtained from sensor data … worn by the subject) [0023].
(b)
wherein the virtual representation data is generated based on the tracking data.
Katz teaches the facial tracking system (160) generates tracking information (said tracking data) based on, e.g., tracked portions of a face of the user (may include eye position). Tracking information is information passed to the console (110) that may be used for virtual animation of a portion of a user’s face (said virtual representation data) [0023].
It would have been obvious before the effective filing date of the claimed invention to utilize the sensors of the HMD of Katz with the eye tracking of Martiez and virtual eye rendering of Birdwell because the HMD of Katz using the data obtained from the sensors and tracking data to animate the eyes of the user’s virtual representation [Katz: 0023]. This would allow for the movement of the user to be replicated in the virtual world.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE L SAMS:
direct telephone number:
(571) 272-7661
email:
michelle.sams@uspto.gov
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M. Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE L SAMS/
Primary Examiner, Art Unit 2611
1 October 2026
