Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention
was made.
Claim(s) 1, 2, 4, and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Miao (Made of reference: US 20130113973 A1) in view of Szeto (US-20200090368-A1).
Regarding claim 1 Miao teaches An extended reality (XR) system (hmd 100), comprising:a lens through which a user is to view a physical scene and onto which virtual content is displayed (Eye piece 101) ;a frame to retain the lens in front of the user (Frame 105, 110,115);an imaging system to present the virtual content on the lens (140 and 150) ;a switch layer disposed over the lens, wherein the switch layer has a selectively alterable transparency (Para 17, 24 and 45: variable transparency layer 150) ;a front-facing sensor to measure an ambient brightness of the physical scene ( para 21- 22,and 27: 145 light ambient sensor) ;a tracking system to record a position and orientation of objects in the physical scene; and a controller to, based on a measured brightness of the physical scene (Para 19, 29, 33: controller 155 monitors ambient light sensor 145 and determines brightness), apply an electrical voltage to the switch layer to adjust a strength of light passing through the lens ( Para 24: controller 155 controls variable transparency layer 150 via control signal 250; signal may be a voltage signal. Para 17, 33-37, 45: Transparency of layer 150 is adjusted according to ambient brightness thereby controlling ambient scene light transmitted through the eyepiece).
Miao fails to teach a tracking system to record a position and orientation of objects in the physical scene.
SZETO teaches a tracking system to record a position and orientation of objects in the physical scene (Para 102: the method acquires from camera 60 a camera data sequence including a first image frame of a real object in a scene and the processer of HMD 100 “ tracks a pose of the real object with respect to the camera along the camera data. It would have been obvious to one of ordinary skill I the art to modify the HMD system of the Miao to include the object tracking of Szeto in order to enable the HMD to detect and track the pose of real world object, thereby allowing virtual content to be accurately positioned relative to objects in the physical)
Regarding claim 2, Miao in view of Szeto The XR system of claim 1, wherein the controller is to adjust a brightness of the virtual content based on the measured brightness of the physical scene ( Miao, Para 17, 19, and 30-31: explains that controller 155 controls the brightness of CGI light 135).
Regarding claim 4, Miao in view of Szeto teaches The XR system of claim 1, wherein the tracking system comprises an optical sensor ( Szeto Para.100: teaches a tracking system comprising an optical sensor, wherein a camera is used to capture images of a real world object for detecting and tracking the object).
Regarding claim 7, Miao in view of Szeto teaches The XR system of claim 1, wherein the controller is integrated in the frame ( Miao, Para fig.1 para 14, 18 and 22: teaching an HMD having an frame assembly and controller 155 incorporated into the HMD/frame structure controlling the display system).
Claim(s) 3 is rejected under 35 U.S.C. 103 as being unpatentable over Miao (Made of reference: US 20130113973 A1) in view of Szeto (US-20200090368-A1) in further view of Bhardwaj (US-20150260991-A1).
Regarding claim 3, Miao in view of Szeto teaches The XR system of claim 1, but fails to teach wherein the switch layer is disposed on a side of the lens opposite the user.
Bhardwaj teaches wherein the switch layer is disposed on a side of the lens opposite the user ( Para 17: “ electro chromic film 150 is coated over the ambient facing side of eyepiece 150.It would have been obvious to one of ordinary skill in the art to dispose the switch layer of Miao in view of Szeto on the ambient facing side of the lens as taught by Bhardwaj in order to selectively control transmission of ambient light through the lens and thereby improve the contrast and visibility of displayed virtual content in bright ambient environments).
Claim(s) 5 - 6 are rejected under 35 U.S.C. 103 as being unpatentable over Miao (Made of reference: US 20130113973 A1) in view of Szeto (US-20200090368-A1) in further view of Allen (US-20180025521-A1).
Regarding claim 5, Miao in view of Szeto teaches The XR system of claim 1, wherein:the controller is to determine a brightness, and a value of the electrical voltage is based on a measured brightness ( Miao, Para 29-30, 33: teaches determining a brightness value of the ambient environment and controlling an electrically operated display/transparency component based upon the determined brightness value).
Allen teaches a boundary region of the physical scene visible through the lens ( Para 15-16: teaches determining ambient light measurement for different sub regions of a real world image and determining whether individual sub regions require modulation. It would have been obvious to modify the system of Miao in view of Szeto to determine brightness for individual regions of the physical scene as taught by Allen, in order to provide localized light modulation responsive to variations in brightness across the real world scene.).
Regarding claim 6, Miao in view of Szeto teaches The XR system of claim 1, wherein a value of the electrical voltage is based on an averaged, weighted, or targeted brightness of the physical scene visible through the lens ( Miao, Para 29-30: teaches measuring/calculating a brightness value for the ambient environment and then using that brightness value to determine the electrical control/bias. Allen, Para 16: expressly states “ the control determination block may take an average of the ambient light measurements of the entire real world image).
Claim(s) 8 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Nourai (US-20180268611-A1) in view of Mathur (US-20200074724-A1) in further view of Miao (US 20130113973 A1).
Regarding claim 8, Nourai teaches A method, comprising: obtaining pixel coordinates of virtual content projected from a perspective of a front-facing sensor of an extended reality (XR) system ( Nourai, Para 37,42 48-49 and 66: forward facing camera; rendering virtual content from its pose/perspective; projecting the content into the camera image frame coordinate space).
Mathur Teaches determining, based on the pixel coordinates of the virtual content projected from the perspective of the front-facing sensor, a boundary region of the Virtual content( Mathur, Para 126 and 139 and examples 52-66: obtains virtual content pixel locations, identifies dimmer pixels based on the virtual object’s size, shape and position, and establishes a dimmed region corresponding to the virtual image );determining, with the front-facing sensor facing a physical scene visible through s lens, an ambient brightness of a portion of the physical scene within the boundary region (Mathur, Para 139 – 142 and examples 70-76: front facing world cameras/optical sensors obtain spatially resolved brightness values; the relevant physical scene portion is specifically the portion perceived as occluded by the virtual object. It would have been obvious to modify Nourai with Mathur’s spatially resolved dimming technique to identify the physical-scene region corresponding to the camera aligned virtual content and selectively reduce ambient light within that region, thereby improving the visibility and contrast of the virtual content).
Miao Teaches applying an electrical voltage to a switch layer overlying the lens to adjust a transparency of light passing through the lens wherein a value of the electrical voltage is based on a measured brightness of the physical scene ( Miao, para 17,19,24,29, 33-34: brightness determines a transparency control setting; a controllable voltage source applies bias voltage to an LCD/ electrochromic transparency layer over the eyepiece. It would have been obvious to implement Mathur’s adjusting dimming layer using Miao’s known voltage controlled LCD or electrochromic transparency layer because doing so provides a predictable and controllable mechanism for adjusting light transmission based on the measured ambient brightness).
Regarding claim 11, Nourai in view of Mathur and in further view of Miao teaches The method of claim 8, further comprising adjusting a brightness of the virtual content based on the measured brightness of the physical scene within the boundary region ( Mathur, para 137 and 142-144, fig 4: the ambient light sensor obtains spatially resolved light values corresponding to two dimensional positions within the field of view; the world light brightness and virtual image brightness at each location are used to determine visibility; and the projector is adjusted to increase or decrease the brightness of the virtual image light to obtain the desired visibility) .
Claim(s) 9 -10 are rejected under 35 U.S.C. 103 as being unpatentable over Nourai (US-20180268611-A1) in view of Mathur (US-20200074724-A1) in further view of Miao (US 20130113973 A1) and Zhang (Hansong Zhang, Dinesh Manocha, Tom Hudson, and Kenneth E. Hoff. 1997. Visibility culling using hierarchical occlusion maps. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., USA, 77–88. https://doi.org/10.1145/258734.258781).
Regarding claim 9, Nourai in view of Mathur and in further view of Miao teaches The method of claim 8, wherein determining the boundary region comprises: converting three-dimensional (3D) coordinates of the virtual content from the perspective of a tracking system to 3D coordinates of the virtual content from a perspective of the front-facing sensor ( Para 45,66: IMU obtains head eye and camera poses, expressly including XYZ coordinates and orientation; the camera’s position is known relative to the head/eyes; and “a transformation from the head and /or eyes … to the camera” is determined so virtual content can be generated from the camera perspective);converting the 3D coordinates of the virtual content from the perspective of the front-facing sensor to two-dimensional (2D) coordinates ( Para 42, 48-49, and 66: teaches that its GPU translates a 3D scene into a 2D image and projects camera perspective virtual content into the coordinate space of the camera image frame); but fails to teach generating a bounding rectangle around the virtual content based on the 2D coordinates.
Zhang teaches generating a bounding rectangle around the virtual content based on the 2D coordinates ( Zhang teaches projected object using a ‘ screen space bounding rectangle’. The rectangle encloses the pixels covered by the object, and its extremal values are calculated by projecting the corners of the objects bounding box onto the screen. It would have been obvious to apply Zhang’s screen space bounding rectangle technique to the camera perspective virtual content image generated by Nourai, because Nourai already projects the virtual object into the camera image coordinate space, While Zhang teaches that a bounding rectangle provides a computationally efficient, conservate identification of the screen pixels occupied by the projected object).
Regarding claim 10, Nourai in view of Mathur and in further view of Miao and Zhang teaches The method of claim 9, wherein converting the 3D coordinates of the virtual content from the perspective of the tracking system to 3D coordinates of the virtual content from the perspective of the front-facing sensor is based on physical dimensions of a frame of XR system and relative position of components of the XR system ( Miao, Para 14-15, 20 and 22, fig.1: the frame assembly is shaped and sized to position the eyepieces and supports components including the controller and forward facing ambient light sensor/camera at specified locations on the frame; Nourai, para 54, 60, 70 and 78: the camera pose is determined or extrapolated using a known and fixed offset between the camera and frame structure, and the known spatial relationship between the tracked head/eyes and camera is used to determine a transformation into the camera perspective).
Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over Nourai (US-20180268611-A1) in view of Mathur (US-20200074724-A1) in further view of Miao (US 20130113973 A1) and Glynn (US-20180190019-A1).
Regarding claim 12, Nourai in view of Mathur and in further view of Miao teaches The method of claim 8, but fails to teach further comprising adjusting a color profile of the virtual content based on the measured brightness of the physical scene within the boundary region.
Glynn, teaches adjusting a color profile of the virtual content based on the measured brightness of the physical scene within the boundary region ( para 43-47 and 52-53 and claim 17: a background region sensor identifies a region corresponding to the displayed virtual GUI; the regions visual properties include brightness and color; and the controller adjust the color, theme, brightness, and other visual properties of the virtual content based on the measured background-region properties and ambient light. It would have been obvious to incorporate Glynn’s adaptive color adjustment into the XR system of Mathur because adjusting the color profile of virtual content according to the brightness and visual properties of the corresponding physical scene region would predictably increase contrast readability, and visibility of the virtual content under changing environment conditions.).
Claim(s) 13 is rejected under 35 U.S.C. 103 as being unpatentable over Nourai (US-20180268611-A1) in view of Mathur (US-20200074724-A1) in further view of Miao (US 20130113973 A1) and Sears (US-20200111258-A1).
Regarding claim 13, Nourai in view of Mathur and in further view of Miao teaches The method of claim 8, but fails to teach wherein contrast is adjusted to achieve a 5:1 ambient contrast ratio (ACR).
Sears teaches wherein contrast is adjusted to achieve a 5:1 ambient contrast ratio (ACR). ( Para 41-44: a portion of the ambient image underlying an AR component is selectively dimmed to increase visibility, and the degree of dimming is selected to achieve a desired contrast ratio within a range of 1.5;1-6;1, expressly including 5;1. It would have been obvious to configure the contrast adjustment of Mathur to achieve Sear’s expressly disclosed 5;1 contrast ratio because selecting that known contrast target would predictably provide adequate visibility and readability of virtual content displayed over an ambient physical scene).
Claim(s) 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Mathur (US-20200074724-A1) in view of Nourai (US-20180268611-A1) in further view of Miao (US 20130113973 A1).
Regarding claim 14, Mathur teaches A non-transitory machine-readable storage medium encoded with instructions executable by a processor, the machine-readable storage medium comprising instructions to (Mathur, 23-37, examples 18-32: expressly discloses a non transitory computer readable medium):present virtual content on a lens of an extended reality (XR) system, wherein a user views a physical scene through the lens ( Mathur, Para 120-124, figs 1-2a: the user views real world objects through transparent semitransparent eyepiece 202, while projector 214 projects virtual image light onto the eyepiece so the user observes the virtual content together with world light);determine, based on the pixel coordinates of the virtual content, a boundary region of the virtual content( Mathur, Para 126 and 139-141, figs 2c and 18a-19b and examples 53-66: identifies the location of virtual image light within the dimmer and determines a dimmed portion corresponding to the virtual object based on its size, shape, position, sampled perimeter points and projected locations);measure, with a front-facing sensor facing the physical scene, an ambient brightness of the physical scene within the boundary region( Mathur para 124, 128-130 and 137, examples 67-71: teaches front facing world cameras and optical sensors obtain spatially resolved world light values associated with respective two dimensional positions in the field of view including the portion corresponding to the virtual content); and based on a measured ambient brightness of the physical scene within the boundary region(para 124, 126, 137, and 142-143:the world light values corresponding to respective pixel locations; the dimmed region corresponds to the location of the virtual content and the dimming values for that region are determined using the detected world light brightness), adjust a contrast between the physical scene visible through the lens and the virtual content ( Para 121, 126, and 143: the dynamic dimmer selectively changes the amount of world light transmitted through spatially selected portion of the field of view).
Mathur fails to teach obtaining pixel coordinates of the virtual content and further fails to teach by selectively altering a transparency of a switch layer overlying the lens.
Nourai teaches obtaining pixel coordinates of the virtual content (Nourai para 72 and 78: virtual content is rendered from the camera perspective using the camera pose, with a transformation into the camera perspective, thereby locating the rendered virtual content in the camera image coordinate space. It would have been obvious to combine Nourai with the system of Mathur to accurately align the spatially controlled dimming region with the virtual content as represented from the front facing camera perspective).
Miao teaches by selectively altering a transparency of a switch layer overlying the lens (Para 15, 19 and 23 and figs 1-2 and 4: variable transparency layer 150/427 is an lcd electrochromic or switchable mirror layer coated or mounted over the ambient scene side of the eyepiece viewing region. It would have been obvious to implement Mathur’s regional contrast adjustment using a known electrically switchable using a known electrically switchable transparency layer over the eyepiece, predictably improving virtual content visibility under changing ambient brightness)
Regarding claim 15, Mathur in view of Nourai and Miao teaches The non-transitory machine-readable storage medium of claim 14, wherein the measured ambient brightness within the boundary region comprises an averaged, weighted, or targeted brightness of the physical scene within the boundary region( Mathur 124,126, 137, 139-143: the ambient light sensor measures spatially resolved brightness values at respective pixel locations, and the system targets the brightness values associated with the selected region corresponding to the virtual content when determining regional dimming values. Thereby teaching a targeted brightness of the physical scene within the boundary region).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
J. David Hincapié-Ramos, L. Ivanchuk, S. K. Sridharan and P. P. Irani, "SmartColor: Real-Time Color and Contrast Correction for Optical See-Through Head-Mounted Displays," in IEEE Transactions on Visualization and Computer Graphics, vol. 21, no. 12, pp. 1336-1348, 1 Dec. 2015, doi: 10.1109/TVCG.2015.2450745.: discloses correcting the color and contrast of virtual content displayed over a physical background to maintain visibility and legibility in an optical see through hmd.
Bell (WO-2016028828-A1): Discloses an optical see through HMD that measures ambient light and adjusts the light transmission of an electrochromic dimming layer based on the ambient light value and displayed image brightness
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LATRELL ANTHONY CREARY whose telephone number is (703)756-1219. The examiner can normally be reached Mon - Fri 7:30am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LATRELL ANTHONY CREARY/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
