Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
2. Applicant's election with traverse of group III claims 16-20 in the reply filed on 4/2/2026 is acknowledged. The traversal is on the ground(s) that no serious search burden exists because the claims share a common core regarding a multi-camera system. This is not found persuasive because, while the camera system is mutual to each group, each camera system requires details specific to each group. Said details include: particular types of cameras, lenses, FOVs, and models. Finding a particular camera with a particular lens that operates with the particular FOV different for each group does comprise a search burden.
Applicant requests consideration for combining groups II and III. Applicant has amended the independent claims of each group to be closer in subject matter and are found to overlap in such a manner to not place a search burden on the examiner.
In view of applicant’s amendments and arguments, a combination of original group II claims into original group III is considered proper for the election of group III. The election of group III is considered to include claims 9-15 and 16-20.
It should be noted that while additional claims have been added, applicant’s traversal is maintained to preserve applicant’s right of petition.
The requirement is still deemed proper (in view of groups I and IV) and is therefore made FINAL.
3. Claims 1-25 are pending with claims 1-8 and 21-25 withdrawn.
Examiner Note
4. Regarding claim 16 amendments to “a first camera system” including at least three infrared cameras per eye and at least one
Also, it is noted the claims in general comprise terms which are only utilized in the original discloser of the claims itself (such as “cognitive load” of claim 16). While this does not make the subject matter indefinite it forces the terms to comprise the scope of interpretation by its inherent definition.
Further, the claims comprise a plurality of interpretive ranges without specifically claimed acceptable values. Such examples are low, precise, subtle, infer, appropriate, etc. While not indefinite they provide an interpretable range that is considerably broad even limited in view of the scope of the technology implementation.
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 9, 11, and 13-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rueckner (US Patent 10,567,641), in view of El-Monajjed et al. (US Patent Application Publication 2022/0342478), herein after referred to as El-Monajjed, in view of Ninan (US Patent Application Publication 2024/0272712), herein after referred to as Ninan.
Regarding independent claim 9, Rueckner discloses a visual processing system (Figure 6 system 601+602 described in column 13 line 56 to column 14 line 37 to include glasses 100/1000 (figures 1A-1C and 10A-10C), other devices 625 (panoramic strap 200 with cameras 201 figures 2A-3 and 5A-5C), and video processing 604 of cameras 611-613 (visual output).), comprising:
a first camera system (612/102) configured (column 14 lines 38-41 describes 611-613 can correspond to 101, 102, 201, 203 and other cameras discussed in the disclosure) to be mounted on, within, or inside a portion (Figures 1B/10B depicts 102 respectively mounted on an inner (eye facing) lens or frame of glasses 100, herein after referred to as inner portion 100.) of a wearable visor (Figures 5A-5C helmet 501+100 interpreted as the wearable visor. Column 13 lines 34-37 describes the entire system can be incorporated into a helmet or hate as a single unit.), comprising a plurality of cameras (102) configured to capture images of a user’s eyes (Column 11 lines 19-26 describes infrared LEDs 104 illuminate the eyes which are tracked with rear-facing cameras 102. Figure 1B/10B depicts two cameras 102. Figure 22 column 22 lines 22-41 detail the eye tracking (while mistakenly referring to externally facing camera 1002 instead of rear-facing camera 102).);
a second camera system (200) configured to be mounted on, within or inside anther portion (501) (Figures 5A-5C panoramic strap 200 (comprising cameras 201 figures 2A-2C) mounted on helmet 501 as described in column 13 lines 23-27.) of the wearable visor (501+100), comprising a plurality of cameras (201) configured to capture a spherical or panoramic field of view (Column 11 lines 38-454 describes panoramic cameras 201 as high-definition with overlapping fields of view to capture a 360 degree view. Column 12 lines 7-27 describes the panoramic camera system 200 utilizes 3D models, depth-maps, and stereoscopic images and video.);
a processor (607) configured to:
analyze the eye images to construct a three-dimensional model of each eye's movements and deformations (Figure 22 and column 22 lines 22-67 describes using IR light 2201 reflected off of the eye 2220 to measure phakometry via projected spatial patterns (figure 23A-23D), Purkinje images 2204-2207 relates to a shape of the lens and how far away the user is focusing their eyes over time, and generate a 3D model of the eye.),
map the three-dimensional eye model to the field of view (201) to determine a precise dual-pupil (2402) gaze point (2401) and focal depth (Column 5 line 54-column 6 line 7 and 11 lines 19-26 describes gaze tracking utilizing IR eye reflected light (104) captured by a camera (102) and software to track the movements over time. The system can utilize pupil and iris parameters such as center, shape, angle, and oculomotor signals from the eye image to be mapped to a gaze target in the world image (via cameras 101+201). Figure 24 and column 23 lines 1-17 details the system to incorporate 3D world models, gaze direction vector models, gaze depth, gaze vector mapping, 3D eye model or depth image, object detection, and motion to determine that target at varying depths. Figures 7A and 16A-16C described in column 16 line 41 to column 17 line 11 a process 700 of object and pattern search within the field of view to create a 3D model of the world to refine the mapping. Figure 16E and column 18 lines 18-42 describes the processes 800+850 of gaze targets 1642-1644 mapped onto a panoramic video 1641 at particular times.), and
track [ ] and subtle eye movements to infer user intent (Figure 7B and column 17 lines 26-52 describes a captured eye gaze over time determining a user to look at multiple objects or scan over a scene effectively defining the region or objects of interest (user intent).); and
a [ ] display system (1903) integrated into the wearable visor (501+100), configured to present context-aware information (2103 and/or 2101 and/or 2102) at the determined dual-pupil gaze point with depth-appropriate focus (Figure 21 and column 30 lines 10-26 describes the augmented reality AR display 903 to depict the current gaze target (dual-pupil gaze point) by a reticle 2101 (context aware additional information), detect portion of the field of view that are in focus and represent those portions on the AR display with focus peaking 2102 (context aware additional information), and display the bounding box or crop marks 2103 of a remote camera currently in the field of view of the user (context aware additional information).).
Rueckner does not specifically disclose to track micro-saccades in addition to subtle eye movements to infer user intent.
El-Monajjed discloses to track micro-saccades in addition to subtle eye movements to infer user intent (paragraphs [0103] and [0112] describes to detect microsaccades and sensitive small rapid motions for gaze tracking to filter said motions as noise, thereby improving the estimation of intended user gaze position and/or behavior.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner’s gaze tracking with the known technique of to track micro-saccades in addition to subtle eye movements to infer user intent yielding the predictable results of filtering said motions as noise, thereby improving the estimation of intended user gaze position and/or behavior as disclosed by El-Monajjed (paragraphs [0103] and [0112]).
Rueckner does not specifically disclose the display system to be a low-latency display system.
Ninan discloses low-latency display system (Paragraphs [0108]-[0115] describes a low latency real-time display for displaying AR images on AR 3D glasses to correspond to the user’s real time focus vision in the field of view of the user.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner’s display with the known technique of a low-latency display yielding the predictable results of displaying AR images on AR 3D glasses to correspond to the user’s real time focus vision in the field of view of the user as disclosed by Ninan (paragraph [0112]).
Regarding claim 11, Rueckner discloses the system of claim 9, wherein the second camera system comprises at least six cameras arranged to provide a full 360-degree spherical view around the user's head (Figures 2A-2C and 5A-5C depict six cameras 201, Column 11 lines 38-454 describes panoramic cameras 201 as high-definition with overlapping fields of view to capture a 360-degree view.).
Regarding claim 13, Rueckner discloses the system of claim 9, further comprising an illumination system (104) configured to project structured light patterns onto the user’s eyes, wherein the illumination system comprises a plurality of infrared light-emitting diodes (Column 11 lines 19-26 describes infrared LEDs 104 illuminate the eyes which are tracked with rear-facing cameras 102.) configured to project different structured light patterns in rapid succession (Figure 22 and column 22 lines 22-67 describes using IR light 2201 reflected off of the eye 2220 to measure phakometry via projected spatial patterns (figure 23A-23D), Purkinje images 2204-2207 relates to a shape of the lens and how far away the user is focusing their eyes over time, and generate a 3D model of the eye.).
Regarding claim 14, Rueckner discloses the system of claim 9, wherein the processor is further configured to analyze eyelid movements and blink patterns to infer user fatigue levels (Column 6 lines 36-51 describes detecting blinking (eyelid movement) to help the system distinguish between irrelevant and intentionally actions (inferred fatigue levels).).
Regarding claim 15, Rueckner discloses the system of claim 9, wherein the low-latency display system comprises a foveated rendering engine configured to present high-resolution imagery only within the determined dual-pupil gaze point and lower resolution imagery in peripheral areas (Figure 19E and column 29 lines 42-55 describes to blur regions to help alleviate eye-strain and fatigue.).
6. Claim(s) 10, 16-17, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rueckner - El-Monajjed - Ninan in view of Khan et al. (US Patent Application Publication 2022/0151489), herein after referred to as Khan.
Regarding claim 10, Rueckner discloses the system of claim 9, wherein the first camera system comprises at least one cameras per eye, positioned to capture the entire visible surface of each eye (Column 11 lines 19-26 describes infrared LEDs 104 illuminate the eyes which are tracked with rear-facing cameras 102.).
Rueckner does not specifically disclose the first camera system comprises at least four cameras per eye.
Khan discloses a camera system comprises at least four cameras per eye (Figure 8 808 and paragraphs [0069] and [0099] describes at least four eye tracking cameras for each eye so as to image each quadrant of the user’s eye.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner first camera system of one camera per eye with the known technique of four cameras per eye yielding the predictable results of imaging each quadrant of the user’s eye as disclosed by Khan (paragraph [0099]).
Regarding independent claim 16, Rueckner discloses a visual processing system (Figure 6 system 601+602 described in column 13 line 56 to column 14 line 37 to include glasses 100/1000 (figures 1A-1C and 10A-10C), other devices 625 (panoramic strap 200 with cameras 201 figures 2A-3 and 5A-5C), and video processing 604 of cameras 611-613 (visual output).), comprising:
a first camera system (612/102) configured (column 14 lines 38-41 describes 611-613 can correspond to 101, 102, 201, 203 and other cameras discussed in the disclosure) to be mounted on, within, or inside a portion (Figures 1B/10B depicts 102 respectively mounted on an inner (eye facing) lens or frame of glasses 100, herein after referred to as inner portion 100.) of a wearable visor (Figures 5A-5C helmet 501+100 interpreted as the wearable visor. Column 13 lines 34-37 describes the entire system can be incorporated into a helmet or hate as a single unit.), comprising:
at least one infrared cameras (102) per eye positioned equidistantly around each eye's central rotation point (Column 11 lines 19-26 describes infrared LEDs 104 illuminate the eyes which are tracked with rear-facing cameras 102. Figure 1B/10B depicts two cameras 102. Figure 22 column 22 lines 22-41 detail the eye tracking (while mistakenly referring to externally facing camera 1002 instead of rear-facing camera 102).), and
at least one camera (102) per eye configured to capture images of a user’s eyes (see examiner note above) (Column 11 lines 19-26 describes infrared LEDs 104 illuminate the eyes which are tracked with rear-facing cameras 102. Figure 1B/10B depicts two cameras 102. Figure 22 column 22 lines 22-41 detail the eye tracking (while mistakenly referring to externally facing camera 1002 instead of rear-facing camera 102).);
a second camera system (200) configured to be mounted on, within, or inside another portion (501) (Figures 5A-5C panoramic strap 200 (comprising cameras 201 figures 2A-2C) mounted on helmet 501 as described in column 13 lines 23-27.) of the wearable visor (501+100), comprising:
at least four wide-angle (column 12 lines 19-22) cameras (figure 2A depicts 6 cameras 201) arranged to capture a 360-degree spherical or panoramic field of view (Column 11 lines 38-454 describes panoramic cameras 201 as high-definition with overlapping fields of view to capture a 360 degree view. Column 12 lines 7-27 describes the panoramic camera system 200 utilizes 3D models, depth-maps, and stereoscopic images and video.), and
at least one depth-sensing camera (Figure 6 ranging sensors 621 described in column 14 lines 57-60 to include depth-sensing cameras.);
a processor (607) configured to:
construct a three-dimensional model of each eye's position, rotation, and deformation based on data from the first camera system (102) (Figure 22 and column 22 lines 22-67 describes using IR light 2201 reflected off of the eye 2220 (led 104 and cameras 102 column 11 lines 19-26) to measure phakometry via projected spatial patterns (figure 23A-23D), Purkinje images 2204-2207 relates to a shape of the lens and how far away the user is focusing their eyes over time, and generate a 3D model of the eye.),
generate a high-resolution (column 11 lines 4-8 discloses external cameras 101 used to capture the exterior field of view may not be “very” high resolution describing a range of high resolution for the camera.), three-dimensional environmental model based on data from the second camera system (200) (Figures 7A and 16A-16C described in column 16 line 41 to column 17 line 11 a process 700 of object and pattern search within the field of view to create a 3D model of the world to refine mapping. Figure 16E and column 18 lines 18-42 describes the processes 800+850 of gaze targets 1642-1644 mapped onto a panoramic video 1641 at particular times.),
determine a gaze vector and focal depth by mapping the eye model to the environmental model (Column 5 line 54-column 6 line 7 and 11 lines 19-26 describes gaze tracking utilizing IR eye reflected light (104) captured by a camera (102) and software to track the movements over time. The system can utilize pupil and iris parameters such as center, shape, angle, and oculomotor signals from the eye image to be mapped to a gaze target in the world image (via cameras 101+201). Figure 24 and column 23 lines 1-17 details the system to incorporate 3D world models, gaze direction vector models, gaze depth, gaze vector mapping, 3D eye model or depth image, object detection, and motion to determine that target at varying depths.),
track [ ] and pupil dilation to infer user intent and cognitive load (Figure 7B and column 17 lines 26-52 describes a captured eye gaze over time determining a user to look at multiple objects ( cognitive load) or scan over a scene effectively defining the region or objects of interest (user intent).), and
dynamically update the gaze vector and focal depth in real-time (Column 23 lines 1-17 describes to utilize the gaze vector and gaze depth in correspondence with the users gaze. Column 8 lines 54-62 describes processing gaze data in real time.); and
a [ ] see-through (Figure 21 and column 30 lines 10-26 describes the information to be overlaid on an augmented reality display as a using is focusing on real world objects (describing see through).) display (1903) integrated into the visor (501+100), configured to present context-aware augmented reality content (2103 and/or 2101 and/or 2102) at the determined gaze vector and focal depth (Figure 21 and column 30 lines 10-26 describes the augmented reality AR display 903 to depict the current gaze by a reticle 2101 (context aware additional information), detect portion of the field of view that are in focus and represent those portions on the AR display with focus peaking 2102 (context aware additional information), and display the bounding box or crop marks 2103 of a remote camera currently in the field of view of the user (context aware additional information). Column 23 lines 1-17 details the system to incorporate 3D world models, gaze direction vector models, gaze depth, gaze vector mapping, 3D eye model or depth image, object detection, and motion to determine that target at varying depths.).
Rueckner does not specifically disclose at least three infrared cameras per eye positioned equidistantly around each eye's central rotation point
Khan discloses at least three infrared cameras per eye positioned equidistantly around each eye's central rotation point (Figure 8 808 and paragraphs [0069] and [0099]-[0100] describes at least four infrared eye tracking cameras for each eye so as to image each quadrant of the user’s eye, depicted in the figure as equidistant.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner first camera system of one camera per eye with the known technique of four infrared cameras per eye positioned equidistantly around each eye's central rotation point yielding the predictable results of imaging each quadrant of the user’s eye as disclosed by Khan (paragraph [0099]).
Rueckner does not specifically disclose to track micro-saccades in addition to subtle eye movements to infer user intent and cognitive load.
El-Monajjed discloses to track micro-saccades in addition to subtle eye movements to infer user intent and cognitive load (paragraphs [0103] and [0112] describes to detect microsaccades and sensitive small rapid motions for gaze tracking to filter said motions as noise, thereby improving the estimation of intended user gaze position and/or behavior (cognitive load).).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner’s gaze tracking with the known technique of to track micro-saccades in addition to subtle eye movements to infer user intent yielding the predictable results of filtering said motions as noise, thereby improving the estimation of intended user gaze position and/or behavior as disclosed by El-Monajjed (paragraphs [0103] and [0112]).
Rueckner does not specifically disclose the display system to be a low-latency display system.
Ninan discloses low-latency display system (Paragraphs [0108]-[0115] describes a low latency real-time display for displaying AR images on AR 3D glasses to correspond to the user’s real time focus vision in the field of view of the user.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner’s display with the known technique of a low-latency display yielding the predictable results of displaying AR images on AR 3D glasses to correspond to the user’s real time focus vision in the field of view of the user as disclosed by Ninan (paragraph [0112]).
Regarding claim 17, Rueckner discloses the system of claim 16, further comprising an eye illumination system configured to project structured light patterns onto the user's eyes, wherein the processor is further configured to analyze distortions in the structured light patterns to enhance the three-dimensional eye model (Figure 22 and column 22 lines 22-67 describes using IR light 2201 reflected off of the eye 2220 to measure phakometry via projected spatial patterns (figure 23A-23D), Purkinje images 2204-2207 relates to a shape of the lens and how far away the user is focusing their eyes over time, and generate a 3D model of the eye.).
Regarding claim 19, Rueckner discloses the system of claim 16, further comprising an inertial measurement unit, wherein the processor is further configured to use data from the inertial measurement unit to compensate for head movements when determining the gaze vector (Column 24 lines 31-47 describes gimbal stabilization/compensation with gaze and head motion control.).
7. Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rueckner - El-Monajjed - Ninan in view of Nizampatnam et al. (US Patent 12,210,676), herein after referred to as Nizampatnam.
Regarding claim 12, Rueckner discloses the system of claim 9.
Rueckner does not specifically disclose further comprising a neural processing unit configured to apply machine learning algorithms to the eye movement data to predict user attention patterns.
Nizampatnam discloses a neural processing unit configured to apply machine learning algorithms to the eye movement data to predict user attention patterns (Figure 8 840 and column 19 line 65 to column 20 line 9 describes a machine learning algorithm to predict and classify the user’s eye movement and retention state (attention pattern) to identify specific interactions with virtual/real-world objects.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner’s system with the known technique of a neural processing unit configured to apply machine learning algorithms to the eye movement data to predict user attention patterns yielding the predictable results of identifying specific interactions with virtual/real-world objects as disclosed by Khan (column 19 line 65 to column 20 line 9).
8. Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rueckner - El-Monajjed - Ninan – Khan in view of Nizampatnam.
Regarding claim 18, Rueckner discloses the system of claim 16.
Rueckner does not specifically disclose wherein the processor comprises a neural processing unit configured to apply machine learning algorithms to historical eye-tracking data to predict future gaze patterns and pre-render augmented reality content.
Nizampatnam discloses wherein the processor comprises a neural processing unit configured to apply machine learning algorithms to historical eye-tracking data to predict future gaze patterns and pre-render augmented reality content (Figure 8 840 and column 19 line 65 to column 20 line 9 describes a machine learning algorithm trained (historical) to predict and classify the user’s eye movement and retention state (attention pattern) to identify specific interactions with virtual/real-world objects.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Rueckner’s system with the known technique of wherein the processor comprises a neural processing unit configured to apply machine learning algorithms to historical eye-tracking data to predict future gaze patterns and pre-render augmented reality content yielding the predictable results of identifying specific interactions with virtual/real-world objects as disclosed by Khan (column 19 line 65 to column 20 line 9).
Allowable Subject Matter
9. Claim 20 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 20, Rueckner discloses the system of claim 16, wherein the see-through display comprises a foveated display system configured to present high-resolution imagery [ ] and progressively lower resolution imagery in peripheral areas (Figure 19E and column 29 lines 42-55 describes to blur regions to help alleviate eye-strain and fatigue.).
However, none of the cited art discloses to present high-resolution imagery within a 5-degree radius of the determined gaze vector and progressively lower resolution imagery in peripheral areas.
Conclusion
10. Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER E LEIBY whose telephone number is (571)270-3142. The examiner can normally be reached 11-7.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr Awad can be reached at 571-272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRISTOPHER E LEIBY/Primary Examiner, Art Unit 2621
Prosecution Insights