Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Allowable Subject Matter
Claim 7 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3-4, 8, 11, 17, 19-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Bagheri et al. (Pub No. US 2024/0353925 A1).
As per claim 1, Bagheri teaches the claimed:
1. A method comprising:
capturing, by a head-mounted display (HMD) device operating in a first mode, first sensor data and performing a first autoexposure (AE) process on the first sensor data (Figure 1 shows an HMD worn by the user. Also, please see [0035] “If the temporal stability of the user's gaze is determined to be in a stable state, then the computing system may employ an automatic exposure control (AEC) algorithm to adjust camera exposure based on the user ROI. If, conversely, the temporal stability of the user's gaze is determined to be in an unstable state, then the computing system will adjust the camera's exposure based on a default ROI (e.g. frame-average ROI or center-weighted ROI)” and [0025] “… Cameras commonly use automatic exposure control (AEC) algorithms to adjust the brightness of captured image frames. These AEC algorithms are commonly optimized for the purpose of allowing human eyes to visualize details of a captured environment, but these algorithms modify the exposure of an image frame by taking into account the brightness level of the overall image or the brightness level of the center field of view of the image.”
In this instance, the claimed “first mode” corresponds to a HMD that performs automatic exposure control (AEC) algorithms using a default ROI due to temporal instability of the user. The first sensor data is captured frames of the forward-facing cameras 105A-B in figure 1 located on the HMD);
determining that the HMD is operating in a second mode that is different than the first mode (At the end of [0026] “… It would be useful to provide a technique for optimizing frame exposure in which the AEC algorithm considers a user region of interest of a frame and thereby prioritize that region of interest for exposure decisions” and [0035] “If the temporal stability of the user's gaze is determined to be in a stable state, then the computing system may employ an automatic exposure control (AEC) algorithm to adjust camera exposure based on the user ROI. If, conversely, the temporal stability of the user's gaze is determined to be in an unstable state, then the computing system will adjust the camera's exposure based on a default ROI (e.g. frame-average ROI or center-weighted ROI).”
In this instance, the “AEC algorithm that considers a user region of interest” due to temporal stability of the user’s gaze corresponds to the claimed “second mode”);
capturing, by the HMD operating in the second mode, second sensor data ([0031] “FIG. 5 is a block diagram illustrating an example embodiment of a gaze based automatic exposure control (AEC) algorithm. At block 510, a computing system may detect a gaze of a user. Gaze detection may involve detecting an area of an input image which the user is currently looking at. In particular embodiments, the computing system may receive an eye gaze of a user, such as that determined by an eye tracker technique. A gaze detection algorithm may determine a user's eye gaze … Gaze detection may involve predicting a future user gaze. For example, a predicted eye gaze may be determined using a head position of the user, a predicted or estimated motion of the user, or eye tracking prediction technique”, [0032] “… The point of interest may be a predicted point of interest determined based on, for example, the predicted user gaze”, and [0033] “the user ROI may be determined by expanding the gaze point to encompass an object captured on the image plane by utilizing information about a determined location of the object on the image plane”.
In this instance, the claimed “second sensor data” is sensor data used to predict the ROI based on the user’s gaze or further predicted eye gaze);
determining face location data for a subject detected in the second sensor data ([0031] “FIG. 5 is a block diagram illustrating an example embodiment of a gaze based automatic exposure control (AEC) algorithm … the computing system may receive an eye gaze of a user, such as that determined by an eye tracker technique. A gaze detection algorithm may determine a user's eye gaze”. The detection of gaze includes determining face location data because the eyes are located on the user’s face.
[0033] “the user ROI may be determined by expanding the gaze point to encompass an object captured on the image plane by utilizing information about a determined location of the object on the image plane” and also in [0031] “… Gaze detection may involve predicting a future user gaze. For example, a predicted eye gaze may be determined using a head position of the user, a predicted or estimated motion of the user, or eye tracking prediction technique””);
performing a second AE process on the second sensor data that is different than the first AE process ([0031] “FIG. 5 is a block diagram illustrating an example embodiment of a gaze based automatic exposure control (AEC) algorithm”, [0032] “… The point of interest may be a predicted point of interest determined based on, for example, the predicted user gaze”, and [0033] “the user ROI may be determined by expanding the gaze point to encompass an object captured on the image plane by utilizing information about a determined location of the object on the image plane”.
In this instance, the claimed “a second AE process” corresponds to the gaze based automatic exposure control (AEC) algorithm where the automatic exposure is based upon the user’s gaze and looking at a particular ROI. The “second sensor data” correspond to sensor data used to track the user and their eyes. This is different from the first AE process because, as mentioned above, the first AE process performs auto-exposure using a default ROI when the user’s gaze is unstable), wherein the second AE process is based, at least in part, on the determined face location data ([0031] “FIG. 5 is a block diagram illustrating an example embodiment of a gaze based automatic exposure control (AEC) algorithm … the computing system may receive an eye gaze of a user, such as that determined by an eye tracker technique. A gaze detection algorithm may determine a user's eye gaze … Gaze detection may involve predicting a future user gaze. For example, a predicted eye gaze may be determined using a head position of the user, a predicted or estimated motion of the user, or eye tracking prediction technique”.
In this instance, the gaze based automatic exposure control (AEC) algorithm (second AE process) is based upon detecting the user’s eyes and gaze. The detection of gaze includes determining face location data because the eyes are located on the user’s face. In addition, the end portion of [0031] also teaches of detecting a predicted eye gaze that uses the determined head position of the user. Determining the head position of the user for future gaze control involves determining face location data because, again, the eyes are located on the user’s face); and
generating a graphical representation for the subject, based, at least in part, on the second sensor data that has had the second AE process performed on it ([0035] “… If the temporal stability of the user's gaze is determined to be in a stable state, then the computing system may employ an automatic exposure control (AEC) algorithm to adjust camera exposure based on the user ROI” and in [0036] “As previously explained, a user ROI may be configured based on information … A camera or cameras associated with an artificial reality device may be instructed to capture an image using the output of the AEC algorithm. The resultant image or images may be rendered and presented to the user with an artificial reality head mounted device (HMD).”)
As per claim 3, Bagheri teaches the claimed:
3. The method of claim 1, wherein the first AE process comprises a scene average AE process ([0021] “FIG. 2 illustrates exemplary frame data captured from sensors on an artificial reality display device … Information about the physical environment surrounding the user may be captured using, for example, one or more cameras 105 such as external-facing cameras 105A-B (illustrated in FIG. 1)” and [0035] “… If, conversely, the temporal stability of the user's gaze is determined to be in an unstable state, then the computing system will adjust the camera's exposure based on a default ROI (e.g. frame-average ROI or center-weighted ROI).”
In this instance, the “frame-average ROI” corresponds to the claimed “scene average AE” because the frame represents the captured scene located in front of the user wearing the HMD).
As per claim 4, Bagheri teaches the claimed:
4. The method of claim 1, further comprising: displaying, on a display of the HMD, the first sensor data that has had the first AE process performed on it (Figure 1 shows an HMD with a display and first sensor data as outward facing cameras 105A-B. Also, please see [0035] “… If, conversely, the temporal stability of the user's gaze is determined to be in an unstable state, then the computing system will adjust the camera's exposure based on a default ROI (e.g. frame-average ROI or center-weighted ROI)”) and in [0036] “As previously explained, a user ROI may be configured based on information … A camera or cameras associated with an artificial reality device may be instructed to capture an image using the output of the AEC algorithm. The resultant image or images may be rendered and presented to the user with an artificial reality head mounted device (HMD).”)
As per claim 8, Bagheri teaches the claimed:
8. The method of claim 1, wherein the second AE process comprises a region of interest (ROI)-weighted AE process ([0035] “If the temporal stability of the user's gaze is determined to be in a stable state, then the computing system may employ an automatic exposure control (AEC) algorithm to adjust camera exposure based on the user ROI” and [0036] “… For example, the user ROI may be associated with a weight, indicating an amount of influence the user ROI has over the AEC algorithm.”)
As per claim 11, Bagheri teaches the claimed:
11. The method of claim 8, wherein the second AE process further comprises a blending between a scene average AE process and the ROI-weighted AE process (Bagheri in [0036] “… if the user ROI weight is 50%, then the AEC algorithm may base its output half on information from the user ROI and half on information of the frame as a whole”. In this instance, blending occurs because the ROI-weighted AE process is 50% while the rest of the scene is 50% as well.
The frame as a whole undergoes a scene average process, e.g. please see Bagheri in [0029] “an application of a frame average automatic exposure control (AEC) algorithm to frame 400”).
As per claim 17, the reasons and rationale for the rejection of claim 1 is incorporated herein.
Bagheri teaches the claimed:
A non-transitory computer readable medium ([0061] “Herein, a computer-readable non-transitory storage medium or media may include …”).
As per claim 19, this claim is similar in scope to limitations recited in claim 8, and thus is rejected under the same rationale.
As per claim 20, the reasons and rationale for the rejection of claims 1 and 17 are incorporated herein.
Bagheri teaches the claimed:
A head-mounted display (HMD) device, comprising: one or more processors; a display; one or more image sensors (Figure 1 shows a HMD comprising a display 135 and image sensors 105A-B. Figure 9, piece 902 shows the claimed processor).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Lohr et al. (Pub No. US 2020/0342673 A1).
As per claim 2, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Lohr teaches the claimed:
2. The method of claim 1, wherein the first mode comprises a passthrough video generation mode (Lohr in [0022] “… That is, the HMD may include one or more modes, such as a pass-through mode where the real-world environment is presented to the user”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the pass-through mode as taught by Lohr with the system of Bagheri in order to “allow the user to interact with and view objects in the real-world environment, such as co-workers, computer screens, mobile devices, etc.” (beginning of [0022] in Lohr).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Park (US 20240312150 A1).
As per claim 5, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Park teaches the claimed:
5. The method of claim 1, wherein the second mode comprises an enrollment mode (As mentioned above for claim 1, Bagheri teaches of the second mode corresponding to a user achieving eye gaze stability by focusing on a ROI in the scene in front of them. Bagheri is silent about an enrollment mode per se. Park teaches this feature because they teach of the eye gaze ROI as being a person’s face located in front of the user where the HMD and in response cropping the image to focus on that user’s face, e.g. please see Park in the abstract “For example, if the smart glasses detect a human face at the gaze point, then the ROI is cropped to the human face. In this manner, the smart glasses may leverage specific capabilities of the smart glasses to augment the user experience”. Also, this is shown in figure 1 of Park as well.
In this instance, the claimed “enrollment mode” is being interpreted as specific type of image processing performed to the HMD display in response to the HMD user’s ROI looking at a person’s face in front of them (e.g. please see the spec in the beginning of paragraph [0046]). In particular, the enrollment mode is being interpreted as when the HMD display is altered to draw attention or increase focus on the user’s face or facial features in particular.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the enrollment mode as taught by Park with the system of Bagheri in order to allow the user to better visualization the person’s face as the ROI that they are looking at (e.g. by cropping the rest of the scene around that person’s face – abstract of Park and in figure 1 of Park).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Lemoff et al. (Pub No. US 2018/0335835 A1).
As per claim 6, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Lemoff teaches the claimed:
6. The method of claim 1, wherein determining that the HMD is operating in a second mode that is different than the first mode comprises: determining that the subject detected in the second sensor data is within a threshold difference of a target pose (As mentioned above for claim 1, Bagheri teaches of determining that the HMD is operating a second mode when the user’s gaze is temporal stable. The second sensor data is data used to detect eye gaze and motion. However, Bagheri does not mention the claimed: “determining that the subject detected in the second sensor data is within a threshold difference of a target pose”. Lemoff teaches that this was known in the art for determining temporal stability, e.g. please see Lemoff in their claim 25 which recites: “determining that the user's gaze has stabilized based on whether the sensed motion has fallen below a threshold.” In this instance, a user with eye gaze that is a sensed motion before a threshold corresponds to the claimed “determining that the subject detected in the second sensor data is within a threshold difference of a target pose”. Thus, in this instance, a target pose is a pose of the user where their eyes have little to no sensed motion).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the target pose and threshold as taught by Lemoff with the system of Bagheri in order mathematically have a way to measure exactly when eye gaze stability is achieved for the eye tracking software.
Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Park.
As per claim 9, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Park teaches the claimed:
9. The method of claim 8, wherein the ROI comprises a face of the subject (As mentioned above for claim 1, Bagheri teaches of the second mode corresponding to a user achieving eye gaze stability by focusing on a ROI in the scene in front of them. Bagheri is silent about the ROI comprising a face of the subject per se. Park teaches this feature because they teach of the eye gaze ROI as being a person’s face located in front of the user where the HMD and in response cropping the image to focus on that user’s face, e.g. please see Park in the abstract “For example, if the smart glasses detect a human face at the gaze point, then the ROI is cropped to the human face”. Also, this is shown in figure 1 of Park as well).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the ROI to be a face as taught by Park with the system of Bagheri in order to allow the user to better visualize the person’s face as the ROI that they are looking at (e.g. by cropping the rest of the scene around that person’s face – abstract of Park).
As per claim 10, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Park teaches the claimed:
10. The method of claim 9, wherein performing the second AE process comprises determining a location of the face of the subject in the second sensor data (This feature is taught when the second sensor data instead corresponds to forward looking camera data 102 in figure 1 of Park. For example, forward looking camera data to determine a face located in front of the user wearing the HMD. Figure 1 of Park also teaches that the location of the face as shown with circle 108. In this interpretation, the claimed “subject” instead corresponds to the person located in front of the user wearing the HMD as shown in figure 1 of Park).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Park with the system of Bagheri. The motivation of claim 9 is incorporated herein.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Lee et al. (Pub No. US 2024/0045943 A1).
As per claim 12, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Lee teaches the claimed:
12. The method of claim 1, wherein the second sensor data comprises sensor data captured from at least a first image sensor and a second image sensor (As mentioned above for claim 1, the second sensor data in Bagheri corresponds to eye tracking sensor data. Bagheri is silent about a first and second image sensor per se. Lee teaches this feature in [0085] “In an embodiment, the first eye tracking camera 270a and the second eye tracking camera 270b may detect and track pupils.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the first and second sensor as taught by Lee with the system of Bagheri in order to have one sensor dedicated to capturing eye gaze information for each eye. Thus, one sensor can be placed spatially closer to each eye in order to capture its eye gaze movement better.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Lee in further view of Motta et al. (Pub No. US 2018/0082482 A1) and Ogawa (Pub No. US 2010/0007766 A1).
As per claim 13, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Motta and Ogawa teaches the claimed:
13. The method of claim 12, wherein the first image sensor and the second image sensor are driven with different exposure settings (Motta teaches of the first sensor data (forward looking cameras) having a capture frame rate of 60 FPS (e.g. Motta in [0026] “… video see through cameras 210 may include high quality, high resolution RGB video cameras, for example 10 megapixel (e.g., 3072×3072 pixel count) cameras with a frame rate of 60 frames per second”) and teaches of the second sensor data (eye tracking camera) having a capture frame rate of 120 FPS (e.g. Motta in [0035] “… each eye tracking sensor 214 may include an IR light source and IR camera, for example a 400×400 pixel count camera with a frame rate of 120 FPS”.
Ogawa teaches of the frame rate affecting the exposure settings as well, e.g. please see Ogawa in [0182] “Further, in the above-mentioned camera apparatus 100, the frame rate is changed to thereby change the exposure time for the image-pickup element 130”. Thus, having different frame rates for the first and second image sensors results in these sensors also having different exposure settings as well).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the different frame rates as taught by Motta with the system of Bagheri because accurately tracking eye movements often requires a higher capture frame rate due to the rapid movements of eye gaze changing over time. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have different exposure settings as taught by Ogawa with the system of Bagheri as modified by Motta because cameras operating at different frame rates often require different exposure times (so that images may be captured at that given frame rate is a manner that is satisfactory for using for later image processing purposes).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Motta.
As per claim 14, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Motta teaches the claimed:
14. The method of claim 1, wherein the first sensor data and the second sensor data are captured with different frame rates (Motta teaches of the first sensor data (forward looking cameras) having a capture frame rate of 60 FPS (e.g. Motta in [0026] “… video see through cameras 210 may include high quality, high resolution RGB video cameras, for example 10 megapixel (e.g., 3072×3072 pixel count) cameras with a frame rate of 60 frames per second”) and teaches of the second sensor data (eye tracking camera) having a capture frame rate of 120 FPS (e.g. Motta in [0035] “… each eye tracking sensor 214 may include an IR light source and IR camera, for example a 400×400 pixel count camera with a frame rate of 120 FPS”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the different frame rates as taught by Motta with the system of Bagheri because accurately tracking eye movements often requires a higher capture frame rate due to the rapid movements of eye gaze changing over time.
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Park.
As per claim 15, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Park teaches the claimed:
15. The method of claim 1, wherein generating a graphical representation for the subject further comprises: fusing at least two image frames captured as part of the second sensor data (This feature is taught when the second sensor data instead corresponds to forward looking camera data in figures 1-4 of Park. For example, forward looking camera 102 or 302 or 402 data to determine a face located in front of the user wearing the HMD as shown in figure 1 of Park. Figure 1 of Park also teaches that the location of the face as shown with circle 108. In this interpretation, the claimed “subject” instead corresponds to the person located in front of the user wearing the HMD as shown in figure 1 of Park). Park teaches of fusing the two images from second sensor data in [0064] and in [0071]. Park shows “generating a graphical representation for the subject” in image 410 in figure 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to fuse the second sensor data as taught by Park with the system of Bagheri in order to form a wider field of view in which to find relevant feature in the scene in front of the user (Park in [0064] and [0071]-[0072]).
As per claim 16, Bagheri alone does not explicitly teach the claimed limitations.
However, Bagheri in combination with Park teaches the claimed:
16. The method of claim 1, further comprising: performing, in response to no face location data being detected for the subject in the second sensor data, a third AE process on the second sensor data that is different than the second AE process (Park teaches this feature in figure 2 where the ROI does not detect a face but instead detects a gauge object. In response, a third AE process is used to emphasize a graph in image 210. Also, please see Park in [0059] and [0061] as well).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the third AE process as taught by Park with the system of Bagheri in order to customize the display to optimize for different types of objects. For example, in Park the third AE process helps emphasis visual data about the detected gauge in particular.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Bagheri in view of Lohr in further view of Park.
As per claim 18, this claim is similar in scope to limitations recited in claims 2 and 5, and thus is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL F HAJNIK whose telephone number is (571) 272-7642. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights