Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Response to Applicant’s Arguments
Applicant’s arguments filed 08/13/2026 have been fully considered, but they are not deemed to be persuasive. Applicant argues that the cited references do not teach “displaying to the user a video stream comprising camera- captured foundational images and adding or modifying a virtual object in that displayed video stream based at least in part on current vantage parameters that include the user's eye state” which is not persuasive based on the new reference of KABARA et al (A Stereoscopic Video See-through Augmented Reality System Based on Real-time Vision-based Registration); specifically, Kabara teaches the claimed “displaying to the user a video stream comprising camera- captured foundational images and adding or modifying a virtual object in that displayed video stream based at least in part on current vantage parameters” (Kabara, 1. Introduction - There exist two methods of showing a user images in which the real and virtual environments are merged. An optical see-through shows a user the real environment through a half-transparent mirror and the virtual environment reflected on the half-transparent mirror. A video see-through shows a user image sequences of the real environment captured by a camera and the virtual environment overlaid upon the image sequences… We focus on an augmented reality system that adopts the combination of vision sensor and video see-through system; Figure 5 – First, only the background image of a real scene is rendered on the frame buffer as in Figure 5 (a). Z-buffer value is also set to the farthest value through out the screen. Then, Z-buffer of the real objects are set to the depth of the real objects in Figure 5 (c). The regions of real objects are illustrated with gray levels of depth in Figure 5 (b). Finally, the virtual objects are rendered by using the model-view matrix in Figure 5 (d),(e)). Furthermore, based on the teaching of Garwin’s eyeball analyzer, configured to monitor an eye state of the user and to determine a set of current vantage parameters of the user including the eye state of the user (Garwin, column 7, EYETRACKING DEVICE PROGRAMS - The eyetracking device programs consist of two subroutines which are identified as "EYETRACK" and "AIMPOINT". They are shown as simplified flowcharts in FIGS. 7 and 8. The purpose of the eyetrack subroutine is to find the center of the pupil and glint in such a way as to eliminate or minimize the effects of any errors of measurement. The AIMPOINT subroutine uses the results of eyetrack to compute the direction of gaze. The separation into two routines is purely for convenience in programming; column 5, lines 29-32 - Having established in the DETECT subroutine of FIG. 11 a selection of a viewing area so as to convey information, the apparatus in response then performs an execution of an action). Accordingly, the claimed invention as represented in the claims does not represent a patentable distinction over the art of record.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA 35 U.S.C. 103(c) and potential pre-AIA 35 U.S.C. 102(e), (f) or (g) prior art under pre-AIA 35 U.S.C. 103(a).
Claims 21-40 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over LIU et al (An Optical See-Through Head Mounted Display with Addressable Focal Planes) in view of HOLLERER et al (Mobile Augmented Reality) and GARWIN et al (4,595,990), and further in view of KABARA et al (A Stereoscopic Video See-through Augmented Reality System Based on Real-time Vision-based Registration).
As per claim 21, Liu teaches the claimed "system," comprising: "a heads-up viewing device wearable by a user" (Liu, 2 SYSTEM SETUP - Figure 2 illustrates a schematic design of an optical see-through HMD with addressable focal planes), comprising: "one or more cameras configured to capture foundational images of a scene" (Liu, 3.1 Variable focal plane approach - A digital camcorder (DCR-TRV20 NTSC, SONY) was mounted at the exit pupil location of the HMD to capture demonstration videos); "a display unit configured to display, to the user, a video stream comprising the foundational images" (Liu, Figure 3 - The image source is a 0.59" full- color organic light emitting diode (OLED) microdisplay with 800x600 pixels and a refresh rate up to 85Hz (eMagin Inc.); 3.2.1 Method - Secondly, instead of addressing each pixel by a laser scanning mechanism in the RSD, our approach uses a 2D microdisplay device to present a high-resolution, full-color image time-sequentially to generate multiple focal planes); and "add or modify a virtual object in the video stream of foundational images based at least in part on the current vantage parameters" (Liu, 3.1 Variable focal plane approach - Figure 5 shows a simple mixed-reality application in a variable focal plane mode. The real scene consists of two real cups, one located at 40cm from the viewer and the other at 100cm away. Focusing the digital camera at 40cm and 100cm, respectively, Figure 5(a) shows a virtual COKE can clearly rendered at a 40cm depth, while in Figure 5(c) the can was rendered at a 100cm depth. In comparison, the virtual COKE can appears blurred while the digital camera was focused at 100cm (Figure 5(b)) and 40cm (Figure 5 (d)), respectively).
Applicant’s arguments filed 08/13/2026 have been fully considered, but they are deemed to moot based on the new reference of Kabara. Specifically, Kabara teaches the claimed “displaying to the user a video stream comprising camera- captured foundational images and adding or modifying a virtual object in that displayed video stream based at least in part on current vantage parameters” (Kabara, 1. Introduction - There exist two methods of showing a user images in which the real and virtual environments are merged. An optical see-through shows a user the real environment through a half-transparent mirror and the virtual environment reflected on the half-transparent mirror. A video see-through shows a user image sequences of the real environment captured by a camera and the virtual environment overlaid upon the image sequences… We focus on an augmented reality system that adopts the combination of vision sensor and video see-through system; Figure 5 – First, only the background image of a real scene is rendered on the frame buffer as in Figure 5 (a). Z-buffer value is also set to the farthest value through out the screen. Then, Z-buffer of the real objects are set to the depth of the real objects in Figure 5 (c). The regions of real objects are illustrated with gray levels of depth in Figure 5 (b). Finally, the virtual objects are rendered by using the model-view matrix in Figure 5 (d),(e)). Furthermore, based on the teaching of Garwin’s eyeball analyzer, configured to monitor an eye state of the user and to determine a set of current vantage parameters of the user including the eye state of the user (Garwin, column 7, EYETRACKING DEVICE PROGRAMS - The eyetracking device programs consist of two subroutines which are identified as "EYETRACK" and "AIMPOINT". They are shown as simplified flowcharts in FIGS. 7 and 8. The purpose of the eyetrack subroutine is to find the center of the pupil and glint in such a way as to eliminate or minimize the effects of any errors of measurement. The AIMPOINT subroutine uses the results of eyetrack to compute the direction of gaze. The separation into two routines is purely for convenience in programming; column 5, lines 29-32 - Having established in the DETECT subroutine of FIG. 11 a selection of a viewing area so as to convey information, the apparatus in response then performs an execution of an action) (see also Hollerer, page 26 - A technology with potential for mobile AR is gaze tracking. Eye trackers observe a person's pupils with tiny cameras to determine where that person's gaze is directed). Thus, it would have been obvious, in view of Hollerer, Kabara, and Garwin, to configure Liu's system as claimed by using a gaze tracking, or an eyeball analyzer, configured to monitor an eye state of the user. The motivation is to interact with the display without using hands or gestures.
Claim 22 adds into claim 21 "wherein the eyeball analyzer is configured to monitor a gaze direction of an eye of the user" (Garwin, INTERFACE, column 7 - These programs, in combination, compute the spatial coordinates which define the subject's direction of gaze; Hollerer, page 26 - A technology with potential for mobile AR is gaze tracking. Eye trackers observe a person's pupils with tiny cameras to determine where that person's gaze is directed). Thus, it would have been obvious, in view of Hollerer, Kabara, and Garwin, to configure Liu's system as claimed by using a gaze tracking, or an eyeball analyzer, configured to monitor a gaze direction of an eye of the user. The motivation is to interact with the display without using hands or gestures.
Claim 23 adds into claim 22 "wherein the virtual object includes a sight point in a field of view of the user that indicates the gaze direction" (Garwin, FIXATION DETECTION PROGRAM - This program uses the previously described subroutines EYETRACK and AIMPOINT to obtain repeated measurements of the direction of the gaze. Let the display surface be divided into cells which contain the various targets After each measurement, the aiming coordinates, Xs, Yₛ obtained from the AIMPOINT subroutine are compared against the list, and a cell number associated if a match is found to an active cell within the tolerance range. A successful match may be considered as a "hit"; Figure 8 - step 73: calculate X and Y coordinates of aiming point of gaze on viewing area). Thus, it would have been obvious, in view of Hollerer, Kabara, and Garwin, to configure Liu's system as claimed by using the tracked gaze direction to define a sight point in a field of view which is the intersection of the gaze direction and the display screen. The motivation is to interact with the display without using hands or gestures.
Claim 24 adds into claim 21 "wherein the eyeball analyzer is configured to monitor a depth-of-focus of an eye of the user" (Liu, Figure 5 - Two real cups, one at 40cm and the other at 100cm away from the eye, are realistically augmented with a virtual COKE can rendered at two different depths: (a)-(b) 40cm; and (c)-(d) 100cm. The digital camera was focused at (a), (d) 40cm and (b), (c) 100cm, respectively; 2 SYSTEM SETUP - Two examples are shown in Figure 4 (a): at 38Vrms of applied voltage, the liquid lens produces 0 diopter of optical power indicated by the flat shape of the liquid interface; while at 49Vrms the liquid lens delivers 10.5 diopters of optical power indicated by the strongly curved liquid interface).
Claim 25 adds into claim 24 "wherein the depth-of-focus is determined based at least in part on a detected radius of a lens of the eye" (Garwin, column 5 - For a given subject there will also be pupil expansion and contraction) (Noted: pupil expansion (dilation) and contraction (constriction) affect focus distance by changing the eye's depth of focus, similar to how an aperture works on a camera) (see also Liu, 2 SYSTEM SETUP - Two examples are shown in Figure 4 (a): at 38Vrms of applied voltage, the liquid lens produces 0 diopter of optical power indicated by the flat shape of the liquid interface; while at 49Vrms the liquid lens delivers 10.5 diopters of optical power indicated by the strongly curved liquid interface). Thus, it would have been obvious, in view of Hollerer and Garwin, to configure Liu's system as claimed by monitoring the depth of focus based on the changing lens of the pupil. The motivation is to interact with the display without using hands or gestures.
Claim 26 adds into claim 25 "wherein the eyeball analyzer is calibrated based on a calibration sequence of the heads-up viewing device that relates the radius to selected depths of focus" (Garwin, column 5 - For a given subject there will also be pupil expansion and contraction; column 9, CALIBRATION - In general, the calibration constants are stable with time given for a given subject. However, occasional shifts do occur... It uses the EYETRACK and AIMPOINT subroutines of FIGS. 7 and 8, respectively, and the stored calibration constants from the calibration routine of FIG. 9 to determine the apparent location of the target as seen by the user, and then modifies the constants C1 and D1 so that the computed location agrees with the known location. Again, a number of measurements are taken and averaged, and screened for excess noise) (Noted: pupil expansion (dilation) and contraction (constriction) affect focus distance by changing the eye's depth of focus, similar to how an aperture works on a camera) (see also Liu, 2 SYSTEM SETUP - Two examples are shown in Figure 4 (a): at 38Vrms of applied voltage, the liquid lens produces 0 diopter of optical power indicated by the flat shape of the liquid interface; while at 49Vrms the liquid lens delivers 10.5 diopters of optical power indicated by the strongly curved liquid interface). Thus, it would have been obvious, in view of Hollerer and Garwin, to configure Liu's system as claimed by performing calibration sequences of the heads-up viewing device on the depth of focus based on the changing lens of the pupil. The motivation is to interact with the display without using hands or gestures.
Claim 27 adds into claim 21 "wherein the eyeball analyzer comprises one or more sensors configured to capture sensor data about the eye, the sensor data is stored to a memory device" (Garwin, INTERFACE, column 6 - At the end of each scan line the contents of the counter 60 and the values stored in the temporary registers are transferred to the buffer memory 32 of FIG. 4...; column 7 - This arrangement ensures that the computer 33 can always capture the most recent field of data. Access to the memory sections is controlled by memory control logic micro-processor 62), and "the eyeball analyzer further comprises software executed by a central processing unit (CPU) of the heads-up viewing device that accesses the memory device" (Garwin, INTERFACE, column 7 - These programs, in combination, compute the spatial coordinates which define the subject's direction of gaze). Thus, it would have been obvious, in view of Hollerer, Kabara, and Garwin, to configure Liu's system as claimed by storing the collected information from the sensors for tracking the user's eye state. The motivation is to interact with the display without using hands or gestures.
Claim 28 adds into claim 21 "wherein the current vantage parameters include parameters that indicate two or more of: "an x-y-z position of the user in an environment, a viewing direction of the user, a head tilt of the user, a gaze direction of one or more eyes of the user, or a depth-of-focus of one or more eyes of the user" (Hollerer, page 26 - A technology with potential for mobile AR is gaze tracking. Eye trackers observe a person's pupils with tiny cameras to determine where that person's gaze is directed; page 29 - For coarse position tracking, we would use an A-GPS receiver, employing the new chip technology that promises to achieve signal reception even in commonly untracked areas, such as beneath thick foliage, or indoors. Position tracking using WiFi signal quality measures is also a possibility. Higher-precision position tracking indoors is dependent on the availability of special purpose tracking infrastructures... For orientation tracking, we would choose fiber optic gyroscope (FOG) sensors (Sawada, 2001) in combination with a magnetometer, if the budget allows it. Ideally, these position and orientation tracking technologies should be used to provide first guess for state-of-the-art vision-based tracking, which requires the addition of one or more tiny cameras). Thus, it would have been obvious, in view of Hollerer, Kabara, and Garwin, to configure Liu's system as claimed by collecting the vantage parameters of the user to use with the eye tracking. The motivation is to interact with the display without using hands or gestures.
Claim 29 adds into claim 21 "wherein the virtual object comprises a textual description of a physical object in the scene determined based at least in part on the current vantage parameters" (Hollerer, page 7 - Figure 9.3(b); page 9 - Figure 9.4(b); page 17 - Figure 9.7 (a)). Thus, it would have been obvious, in view of Hollerer and Garwin, to configure Liu's system as claimed by inserting the virtual object with its associated textual description. The motivation is to enhance the visual representation of object with its description.
Claim 30 adds into claim 21 "wherein, to add the virtual object to a foundation image in the video stream, the one or more processors is configured to: generate a modifying image of the virtual object; and superimpose the modifying image onto the foundational image" (Hollerer, page 8 - Tourism. Taking navigational Uls one step further by including information about objects in a mobile person's environment that might be of interest to a traveler, leads naturally to applications for tourism (Feiner et al., 1997; Cheverst et al., 2000). In this case, AR is not only used to find destinations, but also to display background information. For example, instead of looking up a description and historic account of a famous cathedral in a guide book, (or even on a wirelessly connected palm-sized computer in front of the site, AR can make the air around the church come alive with information: 3D models of related art or architecture, the life and work of the architect, or architectural changes over the centuries can be documented in situ with overlays Geographical field work. Field workers in geography and regional sciences could use AR techniques to collect, compare, and update survey data and statistics in the field (Nusser et al., 2003). By assisting data collection and display, an AR system could enable discovery of patterns in the field, not just the laboratory. Instant verification and comparison of information with data on file would be possible). Thus, it would have been obvious, in view of Hollerer, Kabara, and Garwin, to configure Liu's system as claimed by modifying the virtual object on display. The motivation is to update the visual representation of the virtual object.
Claims 31-38 and 39-40 claim a method and one or more computer-readable media storing program instructions based on the system of claims 21-30; therefore, they are rejected under a similar rationale.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/Primary Examiner, Art Unit 2616
Prosecution Insights