DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements on 04/25/2025; 12/16/2025; 01/13/2026; 01/28/2026; and 02/20/2026 have been acknowledged and considered by the examiner. An initialed copy of the PTO-1449 is included in this correspondence.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-7, 11-14, and 18-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Hu et al. (US Pub. 2025/0199607 A1).
Regarding claim 1; Hu teaches a method (a method of controlling autofocus cameras for generating mixed reality, para. [0002]) comprising:
at an electronic device (an artificial reality system 100, Fig.1A) having a processor (a processor 602, Fig.6), a display (a head-mounted display (HMD) 104, Fig.1A), and one or more sensors (a plurality of sensors, for example, forward-facing cameras 105A-B, upward- and an eye-tracking system; see Fig.1A, para. [0010 and 0035]):
obtaining a plurality of sensor-based distance signals based on sensor data from a plurality of sensors (para. [0010, 0015-0018, 0030, 0034, and 0037], the system is configured to obtain scene depth information using the sensors. For example, Fig.1B, para. [0034], the user 102 may be interested in seeing or interacting with any virtual or physical objects at any given time. The user 102 may be playing a game of chess through a virtual chessboard anchored to the top of table 150. While the user 102 is analyzing the chessboard, it would be reasonable to overlay the virtual chessboard over a sharp passthrough image of the table 150. As such, in this example, the user's desired scene depth would correspond to the distance between the user and the table 150), the plurality of sensors comprising one or more eye sensors (the eye-tracking system comprises one or more inward-facing cameras, para. [0035 and 0038]) and one or more environment sensors (para. [0035], Fig.1A, the system comprises external-facing cameras configured to capture the physical environment around the user (e.g., cameras 105A and 105B, Fig.1A));
determining a fusion characteristic based on the sensor data (para. [0010, 0030, 0041, and 0048], the autofocus system is configured to obtain eye-tracking data, vergence information, scene depth information, phase detection autofocus; and an objection detection. The artificial reality system 100 may enable the user 102 to provide inputs which may provide context to help the system 100 determine regions in which the user might be interested in seeing. The input may include tracking a position of a controller 106, the user’s gaze, and the user’s hand gesture…);
determining a focus adjustment of at least one of the one or more sensors based on fusing the sensor-based distance signals using the fusion characteristic (para. [0010, 0030, and 0048], the autofocus system may adjust the cameras’ focal distances based on the combination of eye-tracking data, scene depth information, phase detection autofocus, and objection detection…); and
adjusting a focus of the at least one of the one or more sensors based on the focus adjustment (para. [0010, 0030, 0034, and 0048], the autofocus system adjusts a focus distance based on scene depth and object detection).
Regarding claim 2; Hu teaches the method of claim 1 as discussed above. Hu further teaches determining the fusion characteristic is based on confidence associated with the sensor-based distance signals (para. [0041 and 0047], a virtual laser pointer may extend from the controller and allow the user to aim at and/or interact with virtual or physical objects. When the user is also looking at the same objects, the system 100 could conclude with high confidence that the user is interested in seeing the region in which the objects are located and compute a corresponding desired scene depth for adjusting the autofocus cameras. In addition, the controller 215 may use contextual information to infer the subject of the user's interest. For example, when the user is playing chess using a virtual chessboard anchored to a physical table, the user would likely want the physical region corresponding to the physical table to be in focus in the passthrough image. In another example, if the user has been sequentially viewing or interacting with a series of virtual artifacts based on their relative distance to the user, the controller 215 may be able to predict, with higher confidence, where the user would likely look next. In some cases, the controller 215 may also anticipate that the user would likely pay attention to particular content designed to catch the user's attention. For example, if the controller 215 knows that a virtual alert has surfaced on the user's real-world refrigerator, the controller 215 would have more confidence that the user would likely look at the alert, especially if the user's gaze falls in the general direction of the refrigerator).
Regarding claim 3; Hu teaches the method of claim 1 as discussed above. Hu further teaches determining the fusion characteristic comprises: determining a context based on the sensor data (para. [0047], the controller 215 may classify, based on eye-tracking data, whether the user is in a vergence movement or in fixation); determining confidence associated with the sensor-based distance signals based on the context (para. [0047], if the user is fixated on an object (i.e., fixed vergence), the controller 215 may choose to perform autofocusing so that the user could clearly see the objects of interest. If the user is currently in vergence movement, however, the controller 215 may choose to suspend autofocusing until the user starts fixating on a physical or virtual object. In particular embodiments, the controller 215 may also have a default or fallback focus state); and determining the fusion characteristic based on the confidence associated with the sensor-based distance signals (para. [0047]).
Regarding claim 4; Hu teaches the method of claim 1 as discussed above. Hu further teaches determining the fusion characteristic comprises: determining a vergence distance based on the sensor data based on an intersection of gaze directions (para. [0038], the vergence distance is computed based on the point of intersection of the gaze of the right eye and the gaze of the left eye); determining confidence associated with the sensor-based distance signals based on the vergence distance (para. [0047], if the user is fixated on an object (i.e., fixed vergence), the controller 215 may choose to perform autofocusing so that the user could clearly see the objects of interest. If the user is currently in vergence movement, however, the controller 215 may choose to suspend autofocusing until the user starts fixating on a physical or virtual object. In particular embodiments, the controller 215 may also have a default or fallback focus state); and determining the fusion characteristic based on the confidence associated with the sensor-based distance signals (para. [0047]).
Regarding claim 5; Hu teaches the method of claim 1 as discussed above. Hu further teaches determining the fusion characteristic comprises: determining an operational mode; and determining the fusion characteristic based on the operational mode (para. [0047], if the user is fixated on an object (i.e., fixed vergence), the controller 215 may choose to perform autofocusing so that the user could clearly see the objects of interest. If the user is in vergence movement or when eye-tracking data is missing or has a low confidence score, controller 215 may fall back to using a default scene depth. In other words, when the user is in the vergence movement, the system is configured to operate in a fallback mode).
Regarding claim 6; Hu teaches the method of claim 5 as discussed above. Hu further teaches the operational mode is selected from a plurality of operational modes comprising at least two of: a nominal mode; a passthrough video mode; a virtual reality (VR) mode; spatial photo capture mode; a spatial video capture mode; a persona enrollment or avatar enrollment mode; an APE calibration mode; an in-field calibration mode; an object capture mode configured to generate a model of an object; and a fallback mode (para. [0047], if the user is fixated on an object (i.e., fixed vergence), the controller 215 may choose to perform autofocusing so that the user could clearly see the objects of interest. If the user is in vergence movement or when eye-tracking data is missing or has a low confidence score, controller 215 may fall back to using a default scene depth. In other words, when the user is in the vergence movement, the system is configured to operate in a fallback mode. Therefore, Hu further teaches a nominal/normal mode (i.e., vergence fixation) and a fallback mode (i.e., vergence movement or when eye-tracking data is missing or has a low confidence score)).
Regarding claim 7; Hu teaches the method of claim 5 as discussed above. Hu further teaches the determined operational model is a nominal mode and the fusion characteristic produces the focus adjustment using only a vergence-based distance signal (Figs. 1A-1B; para. [0034]; Hu discloses that the system is operating in a normal mode in which the system only uses scene depth information to adjust the focus of the camera. In particular; the autofocus cameras may initially have a focal distance that gives them a depth of field corresponding to DoF (Depth of Field) A, as shown in FIG. 1B. Since the table 150 is not located within DoF A, the table 150 would appear blurry and out-of-focus in images captured by the cameras. The blurriness of the table 150 would in turn be reflected in the passthrough images 150A-B. After the HMD 104 determines the desired scene depth, it may instruct the autofocus cameras to adjust their focal distances so that the cameras' updated depth of field corresponds to DoF B. Now that the table 150 is within DoF B, the table 150 would appear sharp and in focus in the images captured by the cameras, which in turn would be reflected in the passthrough images 150A-B. Therefore, in the normal mode, the autofocus camera adjusts the focal distance based only on the scene depth which is determined by a vergence of the user’s gazes).
Regarding claim 11; Hu teaches the method of claim 5 as discussed above. Hu further teaches the determined operational model is an object capture mode and the fusion characteristic produces the focus adjustment by identifying a target object and determining a distance of the target object from the electronic device (e.g., Fig.1B, para. [0009, 0030, 0033, and 0034], Hu discloses that the HMD comprises auto-focus cameras to capture images for passthrough. The captured image is re-projected to the user 102 based on his viewpoints. The HMD determines a distance between the user and an object of interest (e.g., table 150, Fig.1B). The auto-focus camera is instructed to adjust focal distances according to the distance).
Regarding claim 12; Hu teaches the method of claim 5 as discussed above. Hu further teaches the determined operational model is a fallback mode and the fusion characteristic produces the focus adjustment based on determining signal loss characteristics of the plurality of sensor-based distance signals (para. [0047], the controller 215 may also have a default or fallback focus state. For example, when the user is in vergence movement or when eye-tracking data is missing or has a low confidence score, controller 215 may fall back to using a default scene depth).
Regarding claim 13; Hu teaches the method of claim 1 as discussed above. Hu further teaches the fusion characteristic produces the focus adjustment based on combining: a vergence-based distance signal corresponding to a distance at which gaze directions intersect (para. [0038], the vergence distance is computed based on the point of intersection of the gaze of the right eye and the gaze of the left eye); and a point-of-regard-based distance signal corresponding to one or more distances at which one or more rays associated with a gaze direction intersect one or more environment objects (para. [0053], the system may use the eye gazes of the user to determine the desired scene depth for the user based on, e.g., vergence of the user's eye gazes, a point of intersection between the user's gazes or view direction and one or more physical objects in the scene (e.g., by computing an intersection between the user's gazes or view direction and a three-dimensional model representation of the real-world environment), a point of intersection between the user's gazes or view direction and one or more virtual objects positioned relative to the real-world environment).
Regarding claim 14; Hu teaches the method of claim 13 as discussed above. Hu further teaches the one or more environment objects comprise one or more real objects or virtual objects of an extended reality (XR) environment (para. [0053 and 0055], the HMD is used to interact with physical objects and virtual objects in a mixed reality environment).
Regarding claim 18; Hu teaches the method of claim 1 as discussed above. Hu further teaches the fusion characteristic changes over time based on changes in context occurring over time (para. [0041], the artificial reality system 100 may enable the user 102 to provide inputs which may provide context to help the system 100 determine regions in which the user might be interested in seeing. The controller 106 may have an IMU so that the position of the controller 106 may be tracked. The controller 106 may further be tracked based on predetermined patterns on the controller. For example, the controller 106 may have several infrared LEDs or other known observable features that collectively form a predetermined pattern. Using a sensor or camera, the system 100 may be able to capture an image of the predetermined pattern on the controller. Based on the observed orientation of those patterns, the system may compute the controller's position and orientation relative to the sensor or camera. The orientation of the controllers 106 and/or any explicit input provided through the controllers 106 may provide the system 100 additional contextual information about the user's interest. For example, a virtual laser pointer may extend from the controller and allow the user to aim at and/or interact with virtual or physical objects. When the user is also looking at the same objects, the system 100 could conclude with high confidence that the user is interested in seeing the region in which the objects are located and compute a corresponding desired scene depth for adjusting the autofocus cameras. In another example, the system 100 may use its external-facing cameras to track the hands and/or body of the user. The tracked positions and/or motions of the user's hands and body could also be used to infer the user's interest and intent).
Regarding claim 19; Hu teaches a head-mounted device (a head-mounted device 100, Fig.1) comprising:
one or more sensors (a plurality of sensors, for example, forward-facing cameras 105A-B, upward- and an eye-tracking system; see Fig.1A, para. [0010 and 0035]);
a display (a head-mounted display (HMD) 104, Fig.1A);
a non-transitory computer-readable storage medium (a memory 604, Fig.6); and
one or more processors (a processor 602, Fig.6) coupled to the non-transitory computer-readable storage medium (Fig.6), wherein the non-transitory computer-readable storage medium comprises program instructions (para. [0062]) that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:
obtaining a plurality of sensor-based distance signals based on sensor data from a plurality of sensors, the plurality of sensors comprising one or more eye sensors and one or more environment sensors; determining a fusion characteristic based on the sensor data; determining a focus adjustment of at least one of the one or more sensors based on fusing the sensor-based distance signals using the fusion characteristic; and adjusting a focus of the at least one of the one or more sensors based on the focus adjustment (similar to the analysis of claim 1).
Regarding claim 20; Hu teaches a non-transitory computer-readable storage medium (a memory 604, Fig.6), storing program instructions executable via a processor to perform operations (para. [0062]) comprising:
obtaining a plurality of sensor-based distance signals based on sensor data from a plurality of sensors, the plurality of sensors comprising one or more eye sensors and one or more environment sensors; determining a fusion characteristic based on the sensor data; determining a focus adjustment of at least one of the one or more sensors based on fusing the sensor-based distance signals using the fusion characteristic; and adjusting a focus of the at least one of the one or more sensors based on the focus adjustment (similar to the analysis of claim 1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (US Pub. 2025/0199607 A1) in view of Linde (US Patent No. 9,589,348).
Regarding claim 8; Hu teaches the method of claim 5 as discussed above. Hu further teaches fusion characteristic produces the focus adjustment (see the analysis of claim 1).
Hu does not teach that the determined operational module is a VR mode and the focus adjustment is a fixed focus.
Linde teaches that a VR camera is a fixed focus camera (Fig.1, col.3||32-34; a VR camera 110 is a fixed focus camera).
At the time of invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify the system of Hu of adjusting focus of cameras according to a vergence distance to include the teaching of Linde of providing a fixed focus VR camera. Accordingly, the system of Hu as modified by Linde would render a method of adjusting focus of the camera to be a fixed focus in a VR mode. The motivation would have been in order to enable the user to see multiple objects within the external environment and to reduce power consumption.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (US Pub. 2025/0199607 A1) in view of Usami (US Pub. 2023/0186449 A1).
Regarding claim 9; Hu teaches the method of claim 5 as discussed above. Hu further teaches fusion characteristic produces the focus adjustment (see the analysis of claim 1).
Hu does not teach the determined operational model is a spatial photo capture mode and the focus adjustment using bracketed focus stacking.
Usami teaches the determined operational model is a spatial photo capture mode and the focus adjustment using bracketed focus stacking (para. [0018 and 0022], Usami discloses a digital camera configured to operate in a photograph mode. The digital camera comprises a focus stacking unit 25 configured to output in-focus pixels in each of a plurality of captured images obtained by a focus bracketing of an imaging unit 22. In particular, a system control unit 50 performs driving control of the focus lens 103 and the shutter 101 during the focus bracketing, thus sequentially capturing a plurality of images different in focus position. A focus position variation (focus step) between adjacent captured images obtained by such imaging processing is set based on a value calculated by the system control unit 50).
At the time of invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify the system of Hu of adjusting focus of cameras according to a vergence distance to include the technique of Usami of focus stacking in which a plurality of images different in focus position in an optical axis direction is captured (focus bracketing), and in-focus regions of the respective images are extracted to combine an image extended in depth of field. The motivation would have been in order to enable the user to improve the image quality with extended depth of field.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (US Pub. 2025/0199607 A1) in view of Agrawal et al. (US Pub. 2023/0298197 A1).
Regarding claim 10; Hu teaches the method of claim 5 as discussed above. Hu further teaches fusion characteristic produces the focus adjustment (see the analysis of claim 1).
Hu does not teach the determined operational model is a persona enrollment avatar enrollment mode in which the device is held out in front of a face of the user and the focus adjustment based on detecting the face of the user and determining a distance of the face of the user from the electronic device.
Agrawal teaches the determined operational model is a persona enrollment avatar enrollment mode in which the device is held out in front of a face of the user and the focus adjustment is based on detecting the face of the user (Fig.3B, para. [0066], Agrawal discloses a method of enabling a user to hold an electronic device 100 in front of a user’s face; detecting the user’s face; determining the user’s gaze 260B corresponding to a location 262B on the electronic device; and adjusting focus of rear facing camera of the electronic device 100 such that the rear facing camera focuses on a region of interest 250B corresponding to the gaze location 262B).
At the time of invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify the HMD system of Hu of adjusting focus of a camera in the HMD device according to a distance between the HMD device and an object of interest to include the teaching of Agrawal of enabling the user to hold an electronic device in front of the user; detecting the user’s face; determining a gaze location on the electronic device; and adjusting focus of the rear facing camera of the electronic device on a region of interest corresponding to the gaze location on the electronic device. Accordingly, in the system of Hu as modified by Agrawal, the HMD system would be configured to detect a distance between the HMD device and the electronic device so as to adjust focus of the camera of the HMD device based on the distance between the HMD device and the electronic device. Therefore, a combination of Hu and Agrawal teaches that the focus adjustment is based on determining a distance of the face of the user from the electronic device. The motivation would have been in order to improve the mixed reality experience to the user.
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (US Pub. 2025/0199607 A1) in view of Lehmuskallio (US Pub. 2025/0238078 A1).
Regarding claim 15; Hu teaches the method of claim 13 as discussed above. Hu further teaches the point of regard distance signal comprises distances determined by sampling rays at a point of regard identified based on the gaze direction (para. [0046], the controller 215 may use the eye gaze of the user to determine what the user may be looking at. In particular, the controller 215 may cast rays corresponding to the user's gaze vectors and determine whether they intersect a virtual or physical object).
Hu does not teach the point of regard distance signal comprises distances determined by rays around a point of regard identified based on the gaze direction and identifying a distribution based on distances of intersections of the rays with the one or more objects.
Lehmuskallio teaches the point of regard distance signal comprises distances determined by rays around a point of regard identified based on the gaze direction and identifying a distribution based on distances of intersections of the rays with the one or more objects (para. [0053-0054], Lehmuskallio discloses a method of determining a gaze region by determining distances from a user’s eyes to a plurality of pixels around a gaze point. In particular, only pixels representing objects at a very similar optical distance to the gaze point would be included in a gaze-contingent region).
At the time of invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify the HMD system of Hu to include the teaching of Lehmuskallio of determining a gaze region by including pixels having very similar optical distance to the gaze point. The motivation would have been in order to enhance the clarity of the gaze region and to reduce visual noise (Lehmuskallio, para. [0054]).
Regarding claim 16; Hu in view of Lehmuskallio teaches the method of claim 15 as discussed above. Hu does not teach the fusion characteristic is determined based on optimizing the vergence-based distance signal and the distribution of distances of the point-of-regard-based distance signal.
Lehmuskallio teaches a gaze region is determined based on optimizing the vergence-based distance signal and the distribution of distances of the point-of-regard-based distance signal (para. [0053-0054], a gaze region is determined based on selecting pixels around the gaze point having the same optical distance as the gaze point. Therefore, Lehmuskallio further teaches optimizing the method of determining the vergence-based distance and the distribution of distances of the gaze region).
The motivation is the same as the rejection of claim 15.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (US Pub. 2025/0199607 A1) in view of NG (US Pub. 2025/0187903 A1).
Regarding claim 17; Hu teaches the method of claim 1 as discussed above. Hu does not teach the focus adjustment is determined based on an optimization function determined or configured based on the fusion characteristic.
NG teaches the focus adjustment is determined based on an optimization function determined or configured based on the fusion characteristic (Para. [0176-0178]; NG discloses a method of adjusting focus of a MEMS camera. In particular, the camera may be operating in different focus modes: single-shot autofocus (i.e., focus once and locks the focus until a new command is given) and continuous autofocus (i.e., keep adjusting focus for moving objects). The camera comprises a feedback mechanism including a feedback loop which continuously evaluates the image's focus quality and makes fine adjustments as needed until the subject is sharp. In addition, the camera further comprises software enhancements, such as face detection, object tracking, and scene recognition, to optimize focus for different shooting scenarios).
At the time of invention was effectively filed, it would have been obvious to one of ordinary skill in the art to modify the HMD system of Hu to include the method of NG of providing a feedback mechanism and software enhancement for optimizing the focus adjustment of the camera. The motivation would have been in order to improve the image sharpness.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Borah et al. (US Pub. 2026/0136098 A1) discloses a method for optimizing auto-focus functionality for capturing a multimedia content.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NGUYEN H TRUONG whose telephone number is (571)270-1630. The examiner can normally be reached M-F: 10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chanh Nguyen can be reached at 571-272-7772. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NGUYEN H TRUONG/Examiner, Art Unit 2623
/CHANH D NGUYEN/Supervisory Patent Examiner, Art Unit 2623
Prosecution Insights