DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Argument
Applicant’s arguments filed in the response on Mar 17, 2026 have been considered and are persuasive. Therefore, the previous rejection is withdrawn. However, the claims are now rejected under a new grounds of rejection using different prior art.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on Feb 25, 2026 is in compliance with theprovisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by theexaminer.
Allowable Subject Matter
Claim 24 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention..
Claims 1-5, 10-14, 17, 21, and 25 are rejected under 35 U.S.C. 102(a)(1) or 102(a)(2) as being unpatentable over Kuang et al. (US-20210358187-B1).
Regarding claim 1, Kuang disclose A method, comprising: determining, by an electronic device, one or more local lighting conditions for one or more respective local portions of a physical environment (Kuang discloses in para. [0007] “… The method includes receiving light information from a light sensor. The light information describing at least light values for a plurality of different color channels. The method also includes estimating environmental matching information for the target area within the local area based in part on the received light information”; also please see [0017] as well), wherein the one or more respective local portions of the physical environment are within and smaller than a field-of-view of one or more cameras of the electronic device (Kuang in para. [0062] “A light sensor of the electronic device determines light information for the local area. The light information includes light values for the local area that describe the color and brightness of the local area. In FIG. 3, the light values comprise values that measure color and brightness of the local area based on light emitted from a first light source 330 and light emitted from a second light source 335. The first light source 330 describes light from the moon. The second light source 335 describes light from an artificial light source—e.g., a low pressure sodium light”.
Kuang in figure 3 discloses a first light local area 330 and second light local area 335 are within and are smaller than the field-of-view of the “Artificial Content Feed 300”. The “Artificial Content Feed 300” has a field-of-view based on the field-of-view of the camera and the user’s field of view wearing the HMD (head-mounted-display), e.g. please see Kuang in para. [0004] “A display assembly is configured to customize artificial reality content (e.g., augmented reality, mixed reality, and/or virtual reality) for display based in part on estimated environmental matching information (e.g., white point, color, brightness, etc.) of a surrounding environment. The artificial reality content may comprise virtual content and real-world content presented on a display simultaneously. The surrounding environment of the display assembly is the real-world content. The display assembly uses a light sensor to generate environmentally matched virtual content included in the artificial reality content” and in para. [0079] “… For example, the field of view of the displayed content is such that the displayed content is presented using almost all (e.g., approximately 110 degrees diagonal), and in some cases all, of the user's field of view.”); and
generating, using the one or more local lighting conditions and at least one image from the one or more cameras, a three-dimensional scene for display by the electronic device, the three-dimensional scene including: a view of a region of the physical environment that is based on the at least one image from the one or more cameras, and virtual content overlaid on the view of the region of the physical environment (Kuang discloses in figure 3. Also, Kuang discloses at the end of para. [0004] “… The display assembly uses a light sensor to generate environmentally matched virtual content included in the artificial reality content” and Kuang discloses in para. [0063]-[0064]: "A target area 315 is a set of locations in the artificial content feed 300. The display controller may estimate environmental matching information for each location within the target area 315. In one embodiment, the display controller may determine where the target area 315 is to be located within the artificial content feed 300 based on information provided by a separate device (e.g., a mobile device) … The portion of the local area 320 is a set of locations surrounding the periphery of the target area 315. The portion of the local area 320 includes the virtual object 310 and real-world objects (e.g., the grass, the sky, etc.) … The display controller updates display instructions for the artificial content feed 300 based at least in part on the determined environmental matching information about the target area 315 and the portion of the local area 320. The display instructions comprise instructions for presenting the virtual object 310 at the target area 315." In this passage, the virtual object 310 corresponds to the claimed “virtual content overlaid on the view of the region of the physical environment”. The scene is three-dimensional because depth information is considered as well in Kuang in para. [0032]-[0033]).
Regarding claim 2, Kuang also discloses The method of claim 1, wherein each of the one or more local lighting conditions includes an ambient light level in the respective local portion of the physical environment (Kuang discloses in para. [0025] ”For the presentation of artificial reality content, to be believable, a display assembly that uses a light sensor to generate environmentally matched artificial reality content is needed. The display assembly customizes the generated artificial reality content (e.g., augmented reality content and/or mixed reality content) to environmentally match the ambient lighting and ambient color.”)
Regarding claim 3, Kuang also discloses The method of claim 2, wherein at least one of the one or more local lighting conditions includes a direction corresponding to a light source (Kuang discloses in para. [0062] “A light sensor of the electronic device determines light information for the local area. The light information includes light values for the local area that describe the color and brightness of the local area. In FIG. 3, the light values comprise values that measure color and brightness of the local area based on light emitted from a first light source 330 and light emitted from a second light source 335. The first light source 330 describes light from the moon. The second light source 335 describes light from an artificial light source—e.g., a low pressure sodium light”. Also, please see in figure 3 where directional light rays are determined from second light source 335).
Regarding claim 4, Kuang also discloses The method of claim 1, wherein generating the three-dimensional scene comprises providing the one or more local lighting conditions to a processing stage in a processing chain that includes at least one subsequent processing stage after the processing stage (Kuang discloses in para. [0070]-[0072] and in figure 5 where the lighting conditions from step 510 are provided to processing stages steps 520-530. This includes at least one subsequent processing stage after the processing stages 530-540. In this figure, the series of steps being performed makes up a processing chain).
Regarding claim 5, Kuang also discloses The method of claim 4, wherein the processing chain is configured to perform at least one of: image pre-processing of the at least one image from the one or more cameras, computer vision operations using the at least one image from the one or more cameras, three-dimensional immersion effect generation for the three-dimensional scene, gesture detection, surface texture estimation, scene reconstruction, object tracking, spatial computing, six
degree-of-freedom display of the virtual content, or spatial audio processing (Kuang discloses in figure 5 where the lighting conditions from step 510 are provided to the processing chain in stages 520-540. These stages include pre-processing in stage 530, 3D immersion effect generation in stages 520-540, and scene reconstruction and spatial computing in stages 520-540).
Claims 10 and 17, are similar in scope to claim 1 thus rejected under the same rationale. Kuang discloses a non-transitory machine readable medium, and an electronic device comprising a memory, and a processor in [0095].
Claim 11, is similar in scope to claim 2, thus rejected under the same rationale.
Claim 12, is similar in scope to claim 3, thus rejected under the same rationale.
Regarding claim 13, Kuang also discloses The non-transitory machine readable medium of claim 10, wherein the operations further comprise determining one of a local color or a local texture using at least one of the plurality of local lighting conditions. (Kuang discloses in para. [0025]” For the presentation of artificial reality content, to be believable, a display assembly that uses a light sensor to generate environmentally matched artificial reality content is needed. The display assembly customizes the generated artificial reality content (e.g., augmented reality content and/or mixed reality content) to environmentally match the ambient lighting and ambient color” and discloses in para. [0007] “In some embodiments, a method is disclosed for customizing artificial reality content for display based in part on estimated environmental matching information for a target area within a local area. The method includes receiving light information from a light sensor. The light information describing at least light values for a plurality of different color channels.”)
Claim 14, is similar in scope to claim 4, thus rejected under the same rationale.
Regarding claim 21, Kuang also discloses The method of claim 1, wherein determining the one or more local lighting conditions comprises determining, using a light sensor of the electronic device (Kuang discloses in the abstract “The display controller is configured to estimate environmental matching information for a target area within a local area based in part on light information received from a light sensor” and in figure 1A light sensor 135 being part of electronic device 100), the one or more local lighting conditions of the one or more respective local portions of a physical environment (Kuang discloses in para. [0062] “A light sensor of the electronic device determines light information for the local area. The light information includes light values for the local area that describe the color and brightness of the local area. In FIG. 3, the light values comprise values that measure color and brightness of the local area based on light emitted from a first light source 330 and light emitted from a second light source 335. The first light source 330 describes light from the moon. The second light source 335 describes light from an artificial light source—e.g., a low pressure sodium light”).
Regarding claim 25, Kuang also discloses The method of claim 4, wherein: the processing chain includes a plurality of subsequent processing stages after the processing stage (Kuang discloses in figure 5 a plurality of subsequent processing stages after the processing stage 520-530 after the processing stage 510); and generating, by another subsequent processing stage subsequent to the plurality of subsequent processing stages, the three-dimensional scene based on an output of the plurality of subsequent processing stages (Kuang discloses in figure 5 generating, by another subsequent processing stage 540 subsequent to the plurality of subsequent processing stages 520-530, the three-dimensional scene based on an output of the plurality of subsequent processing stages).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kuang in view of Goto (US-20240233392-A1).
Regarding claim 6, Kuang discloses all the elements of claim 1 as discussed above.
However, Kuang does not fully disclose wherein the plurality of local lighting conditions include a first local lighting condition on a first side of a light discontinuity in the physical environment and a second local lighting condition on a second, opposing, side of the light discontinuity.
Goto does disclose wherein the plurality of local lighting conditions include a first local lighting condition on a first side of a light discontinuity in the physical environment and a second local lighting condition on a second, opposing, side of the light discontinuity (para. [0058] “The shadow region detection unit 67 detects a shadow present in the image data on the basis of the image data transmitted from the surrounding environment sensor 31. In the present embodiment, a “shadow” refers to a patch of shade on, for example, a wall or the ground caused by an object such as a moving body or a stationary object blocking the travel of the light. For example, it is possible for the shadow region detection unit 67 to identify the shadow by obtaining an outline along which brightness or hue greatly changes within a predetermined distance, on the basis of brightness data and hue data regarding each pixel in the image data.” In this passage, the portions where the brightness or hue greatly changes along a shadow outline corresponds to the claimed light discontinuity with two sides (e.g. a bright and dark side along the shadow outline).
Goto in figure 7 discloses that the detected shadow along its boundaries has a first local lighting condition (bright color) on a first side of a light discontinuity in the physical environment and a second local lighting condition (dark color) on a second, opposing, side of the light discontinuity). It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Goto into the teachings of Kuang in order to a have a system that is able to detect shadowed regions in the local environment. This may help the system determine local light source directions. This also helps determine and understand local lighting conditions for Kuang.
Claims 15 and 20, which are similar in scope to claim 6, thus rejected under the same rationale.
Claim(s) 7, 9, 16, 18-19, and 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over Kuang in view of Grasset et al. (US-20130314441-A1).
Regarding claim 7, Kuang discloses all the elements of claim 1 as discussed above.
However, Kuang does not fully disclose The method of claim 1, further comprising, prior to obtaining the one or more local lighting conditions, determining that the one or more respective local portions of the physical environment are one or more portions of the physical environment that are salient to a user of the electronic device.
Grasset does disclose The method of claim 1, further comprising, prior to obtaining the one or more local lighting conditions, determining that the one or more respective local portions of the physical environment are one or more portions of the physical environment that are salient to a user of the electronic device (the end of para. [0052] “For the salient-relative approach, the average lightness (or saturation) of the salient regions is determined, so the labels can be more prominent with respect to the saliency information on the image. For example, the salient regions may be determined based on the highest salience level or two highest levels in the saliency map 208. FIG. 4, by way of example, illustrates Label 5 with a low contrast background with respect to the surrounding pixels and illustrates Label 6 with a relatively high contrast background.” These are salient to the user holding/wearing the mobile device for the augmented reality display system. The salient map 208 is determined before the local lighting conditions for salient regions. This is because the highest salience level or two highest levels in the saliency map 208 can only be determined once the saliency map 208 itself is determined. Thus, the saliency map 208 for local portions is determined first before local lighting conditions are determined (the highest salient regions)). It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Grasset into the teachings of Kuang in order to a have a system that is able to determine prominent portions of the scene that are in different lighting. This helps with virtual graphics overlay (Grasset in para. [0052]).
Regarding claim 9, the combination of Kuang and Grasset disclose all the elements of claim 7 as discussed above. However, Kuang alone does not fully disclose the additional features of claim 9.
Kuang and Grasset together also disclose The method of claim 7, wherein determining that the one or more respective local portions of the physical environment are one or more portions of the physical environment that are salient to the user of the electronic device comprises determining that at least one of the one or more respective local portions of the physical environment is associated with a gaze of the user (Grasset discloses at the end of para. [0052] and in para. [0010] teaches of determining salient portions of the physical environment using a mobile device (e.g. a cell phone) with a camera. The claimed feature is taught when the camera 130 on the headset 100 in figure 1A of Kuang is used to capture the scene instead of the handheld mobile device in Grasset. In this instance, the camera is forward facing on the headset 100 in figure 1A of Kuang that is worn by the user and thus the camera captures images associated with the gaze of the user). It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Grasset into the teachings of Kuang. The motivation of claim 7 is incorporated herein.
Claims 16, 18-19, and 22-23, which are similar in scope to claim 7, thus rejected under the same rationale. With regards to claims 19 and 22, the salient portions in Grasset are determined based upon the user actions of wearing or holding the mobile device, pointing it towards the region, and activating the camera on the mobile device to capture an image of that region. With regards to claim 23, the salient portions are determined based upon the saliency map in Grasset discloses at the end of para. [0052] and Grasset’s lighting from the camera sensor in para. [0010].
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Kuang in view of Grasset in further view of Clavin et al. (US-20130147686-A1).
Regarding claim 8, the combination of Kuang and Grasset disclose all the elements of claim 7 as discussed above. Kuang alone does not teach the additional claimed features of claim 8.
Clavin discloses The method of claim 7, wherein determining that the one or more respective local portions of the physical environment are one or more portions of the physical environment that are salient to the user of the electronic device comprises detecting a body part of the user in at least one of the one or more respective local portions of the physical environment (para. [0154] “The message could alternatively be located elsewhere in the user's field of view such as laterally of the background image 1402 … Furthermore, the user provides a command using a hand gesture. In this case, the hand 1438 and its gesture (e.g., a flick of the hand) are detected by a forward facing camera 113 with a field of view indicated by dashed lines 1434 and 1436." In this passage, the detection of the user’s hand corresponds to a detected body part of the user in one or more respective local portions of the physical environment, e.g. local portions located in front of the user wearing the HMD). It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Clavin into the teachings of Kuang in order to detect user hand gestures for providing input commands to the system while the user is wearing an HMD. The hand gestures are a common and useful way for the user to provide input to the system when the user is wearing the HMD.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRIS ALEJANDRO PUNTIER whose telephone number is (703)756-1893. The examiner can normally be reached M-F 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRIS ALEJANDRO PUNTIER/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights