DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application is being examined under the pre-AIA first to invent provisions.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claim(s) 1-5, 8, 11-14, 17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Stahl et al. (US Patent/PGPub. No. 10914957) in view of Lee (US Patent/PGPub. No. 11792352).
Regarding Claim 1,
Stahl et al. teach
a method (Col. 2, Ln. 47-51, FIG. 4, i.e. method of operation for a mixed reality system), comprising:
determining depth information (Col. 5, Ln. 35-37, FIG. 1, i.e. world sensors 140 that collect information … (video, depth information, lighting information, etc.)) associated with one or more objects (Col. 5, Ln. 49-52, FIG. 1, i.e. different virtual objects may be displayed at different depths) in a physical environment (Col. 5, Ln. 35-37, FIG. 1, i.e. user 190's environment) of a head-wearable device (Col. 5, Ln. 35-37, FIG. 1, i.e. HMD 100);
receiving first image data (Col. 7, Ln. 52-59, FIG. 2, i.e. collect information about the user 290's environment (video, depth information, lighting information, etc.)) from peripheral cameras (Col. 7, Ln. 52-59, FIG. 2, i.e. two scene cameras 220);
receiving second image data (Col. 8, Ln. 23-30, FIG. 2, i.e. capture information about the position, orientation, and/or motion of the user and/or the user's head in the environment) from forward-facing (FIG. 2, i.e. as shown by the figure(s)) camera components (Col. 8, Ln. 23-30, FIG. 2, i.e. pose sensors 222 (e.g., IR or RGB cameras)),
and
tracking movement of a user wearing the head-wearable device (Col. 8, Ln. 23-30, FIG. 2, i.e. capture information about the position, orientation, and/or motion of the user) and/or the one or more objects (Col. 8, Ln. 4-12, FIG. 2, i.e. depth of real objects) based on the depth information (Col. 8, Ln. 4-12, FIG. 2, i.e. depth of real objects), the first image data, and the second image data (i.e. please see above citation(s)).
However, Stahl et al. do not explicitly teach
wherein each of the forward-facing camera components includes a first type of camera and a second type of camera.
In the same field of endeavor, Lee teaches
wherein each of the forward-facing camera components (Col. 3, Ln. 37-42, FIG. 2, i.e. side or tracking cameras 220(1)˜220(4)) includes a first type of camera and a second type of camera (Col. 3, Ln. 37-42, FIG. 2, i.e. include a wide-view camera, a fisheye camera, a depth sensor, or a combination thereof).
It would have been obvious to a person having ordinary skill in the art at the time the invention’s effective date was filed to replace Stahl et al. teaching a method of tracking user and objects in a physical environment with plurality of cameras with Lee teaching of a method of tracking user and objects in a physical environment with plurality of cameras each including combination of different types to effectively a see-through view that would be generated by a single non-existent (or virtual) frontal see-through camera (Lee’s Col. 3, Ln. 37-42).
Regarding Claim 2,
the method of claim 1,
Stahl et al. teach
further comprising:
receiving at least a portion of structured light from an illumination source (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. IR illumination source) reflected from (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. emitters and receivers/detectors) the one or more objects (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. objects) in the physical environment (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. user's environment) of the head-wearable device (i.e. please see above citation(s)).
Regarding Claim 3,
the method of claim 1, wherein
Stahl et al. teach
the first type of camera is a monochrome camera (Col. 8, Ln. 45-55, FIG. 2, i.e. 222 may provide low latency monochrome) and the second type of camera is a color camera (Col. 8, Ln. 45-55, FIG. 2, i.e. pose sensors 222 (e.g., IR or RGB cameras)).
Regarding Claim 4,
the method of claim 3, wherein
Stahl et al. teach
the monochrome camera has a higher refresh rate (Col. 8, Ln. 45-55, FIG. 2, i.e. head pose sensors 222 … with a frame rate of 120 frames per second (FPS)) than the color camera (Col. 7, Ln. 59-65, FIG. 2, i.e. cameras 220 may include high quality, high resolution RGB video cameras … with a frame rate of 60 frames per second (FPS)).
It would have been obvious to a person having ordinary skill in the art at the time the invention’s effective date was filed, from Stahl et al. teaching, to replace one of the plurality of cameras 222s having a 120 refresh rate with one of cameras 220s having a 60 refresh rate to effectively accommodate designs, manufactures, and cost requirements.
Regarding Claim 5,
the method of claim 3, wherein
Stahl et al. teach
the color camera (Col. 9, Ln. 2-9, FIG. 2, i.e. light sensor 223 may include an RGB high dynamic range (HDR) video camera, for example a 500×500 pixel count camera) has a higher resolution than the monochrome camera (Col. 8, Ln. 45-55, FIG. 2, i.e. 400×400 pixel count … sensors 222 may provide low latency monochrome).
It would have been obvious to a person having ordinary skill in the art at the time the invention’s effective date was filed, from Stahl et al. teaching, to replace one of the plurality of cameras 223s having a higher resolution with one of cameras 220s having a lower resolution to effectively render content in important regions with more detail/at higher resolution than content in less important regions (Stahl et al.’s Col. 23, Ln. 55-59).
Regarding Claim 8,
the method of claim 1,
Stahl et al. teach
further comprising:
performing hand (Col. 6, Ln. 35-45, FIG. 2, i.e. user sensors 224-227 that collect information … (e.g., the user's expressions, eye movement, hand gestures, etc.)) and/or controller tracking using the peripheral cameras (i.e. please see above citation(s)).
It would have been obvious to a person having ordinary skill in the art at the time the invention’s effective date was filed, from Stahl et al. teaching, to replace one of the plurality of cameras 222s for capturing user’s environment with one of cameras 224s for capturing user’s hand gestures to effectively accommodate designs, manufactures, and cost requirements.
Regarding Claim 11,
Stahl et al. teach
a system (Col. 5, Ln. 23-29, FIG. 1, i.e. mixed reality system 10), comprising:
one or more processors (Col. 6, Ln. 23-29, FIG. 2, i.e. processors 204); and
one or more memories (Col. 11, Ln. 40-47, FIG. 3, i.e. memory 330) storing instructions (Col. 11, Ln. 51-59, FIG. 3, i.e. instruction set architecture) that, when executed by the one or more processors, cause the system (i.e. please see above citation(s)) to:
determine depth information (Col. 5, Ln. 35-37, FIG. 1, i.e. world sensors 140 that collect information … (video, depth information, lighting information, etc.)) associated with one or more objects (Col. 5, Ln. 49-52, FIG. 1, i.e. different virtual objects may be displayed at different depths) in a physical environment (Col. 5, Ln. 35-37, FIG. 1, i.e. user 190's environment) of a head-wearable device (Col. 5, Ln. 35-37, FIG. 1, i.e. HMD 100);
receive first image data (Col. 7, Ln. 52-59, FIG. 2, i.e. collect information about the user 290's environment (video, depth information, lighting information, etc.)) from peripheral cameras (Col. 7, Ln. 52-59, FIG. 2, i.e. two scene cameras 220);
receive second image data (Col. 8, Ln. 23-30, FIG. 2, i.e. capture information about the position, orientation, and/or motion of the user and/or the user's head in the environment) from forward-facing (FIG. 2, i.e. as shown by the figure(s)) camera components (Col. 8, Ln. 23-30, FIG. 2, i.e. pose sensors 222 (e.g., IR or RGB cameras)), and
track movement of a user wearing the head-wearable device (Col. 8, Ln. 23-30, FIG. 2, i.e. capture information about the position, orientation, and/or motion of the user) and/or the one or more objects (Col. 8, Ln. 4-12, FIG. 2, i.e. depth of real objects) based on the depth information (Col. 8, Ln. 4-12, FIG. 2, i.e. depth of real objects), the first image data, and the second image data (i.e. please see above citation(s)).
However, Stahl et al. do not explicitly teach
wherein each of the forward-facing camera components includes a first type of camera and a second type of camera.
In the same field of endeavor, Lee teaches
wherein each of the forward-facing camera components (Col. 3, Ln. 37-42, FIG. 2, i.e. side or tracking cameras 220(1)˜220(4)) includes a first type of camera and a second type of camera (Col. 3, Ln. 37-42, FIG. 2, i.e. include a wide-view camera, a fisheye camera, a depth sensor, or a combination thereof).
It would have been obvious to a person having ordinary skill in the art at the time the invention’s effective date was filed to replace Stahl et al. teaching a system of tracking user and objects in a physical environment with plurality of cameras with Lee teaching a system of tracking user and objects in a physical environment with plurality of cameras each including combination of different types to effectively a see-through view that would be generated by a single non-existent (or virtual) frontal see-through camera (Lee’s Col. 3, Ln. 37-42).
Regarding Claim 12,
the system of claim 11, wherein
Stahl et al. teach
the instructions when executed by the one or more processes further cause the system (i.e. please see above citation(s)) to:
receive at least a portion of structured light from an illumination source (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. IR illumination source) reflected from (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. emitters and receivers/detectors) the one or more objects (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. objects) in the physical environment (Col. 7, Ln. 66-67, Col. 8, Ln. 1-4, FIG. 2, i.e. user's environment) of the head-wearable device (i.e. please see above citation(s)).
Regarding Claim 13,
the system of claim 11 (i.e. please see above citation(s)), wherein
Stahl et al. teach
the first type of camera is a monochrome camera (Col. 8, Ln. 45-55, FIG. 2, i.e. 222 may provide low latency monochrome) and the second type of camera is a color camera (Col. 8, Ln. 45-55, FIG. 2, i.e. pose sensors 222 (e.g., IR or RGB cameras)).
Regarding Claim 14, the system of claim 13, wherein
the monochrome camera (i.e. please see above citation(s)) has at least one of (i) a higher refresh rate (Col. 8, Ln. 45-55, FIG. 2, i.e. head pose sensors 222 … with a frame rate of 120 frames per second (FPS)) than the color camera (Col. 7, Ln. 59-65, FIG. 2, i.e. cameras 220 may include high quality, high resolution RGB video cameras … with a frame rate of 60 frames per second (FPS)), and/or (ii) a lower resolution than the color camera.
It would have been obvious to a person having ordinary skill in the art at the time the invention’s effective date was filed, from Stahl et al. teaching, to replace one of the plurality of cameras 222s having a 120 refresh rate with one of cameras 220s having a 60 refresh rate to effectively accommodate designs, manufactures, and cost requirements.
Regarding Claim 17,
the system of claim 11, wherein
Stahl et al. teach
the instructions when executed by the one or more processes further cause the system (i.e. please see above citation(s)) to:
perform hand (Col. 6, Ln. 35-45, FIG. 2, i.e. user sensors 224-227 that collect information … (e.g., the user's expressions, eye movement, hand gestures, etc.)) and/or controller tracking using the peripheral cameras (i.e. please see above citation(s)).
Regarding Claim 20,
Stahl et al. teach
a non-transitory computer-readable storage medium (Col. 11, Ln. 40-47, FIG. 3, i.e. memory 330) having instructions (Col. 11, Ln. 51-59, FIG. 3, i.e. instruction set architecture) embodied thereon, wherein the instructions, when executed by one or more processors (i.e. please see above citation(s)), cause a system (Col. 5, Ln. 23-29, FIG. 1, i.e. mixed reality system 10) to:
determine depth information (Col. 5, Ln. 35-37, FIG. 1, i.e. world sensors 140 that collect information … (video, depth information, lighting information, etc.)) associated with one or more objects (Col. 5, Ln. 49-52, FIG. 1, i.e. different virtual objects may be displayed at different depths) in a physical environment (Col. 5, Ln. 35-37, FIG. 1, i.e. user 190's environment) of a head-wearable device (Col. 5, Ln. 35-37, FIG. 1, i.e. HMD 100);
receive first image data (Col. 7, Ln. 52-59, FIG. 2, i.e. collect information about the user 290's environment (video, depth information, lighting information, etc.)) from peripheral cameras (Col. 7, Ln. 52-59, FIG. 2, i.e. two scene cameras 220);
receive second image data (Col. 8, Ln. 23-30, FIG. 2, i.e. capture information about the position, orientation, and/or motion of the user and/or the user's head in the environment) from forward-facing (FIG. 2, i.e. as shown by the figure(s)) camera components (Col. 8, Ln. 23-30, FIG. 2, i.e. pose sensors 222 (e.g., IR or RGB cameras)), and
track movement of a user wearing the head-wearable device (Col. 8, Ln. 23-30, FIG. 2, i.e. capture information about the position, orientation, and/or motion of the user) and/or the one or more objects (Col. 8, Ln. 4-12, FIG. 2, i.e. depth of real objects) based on the depth information (Col. 8, Ln. 4-12, FIG. 2, i.e. depth of real objects), the first image data, and the second image data (i.e. please see above citation(s)).
However, Stahl et al. do not explicitly teach
wherein each of the forward-facing camera components includes a first type of camera and a second type of camera.
In the same field of endeavor, Lee teaches
wherein each of the forward-facing camera components (Col. 3, Ln. 37-42, FIG. 2, i.e. side or tracking cameras 220(1)˜220(4)) includes a first type of camera and a second type of camera (Col. 3, Ln. 37-42, FIG. 2, i.e. include a wide-view camera, a fisheye camera, a depth sensor, or a combination thereof).
It would have been obvious to a person having ordinary skill in the art at the time the invention’s effective date was filed to replace Stahl et al. teaching a system of tracking user and objects in a physical environment with plurality of cameras with Lee teaching a system of tracking user and objects in a physical environment with plurality of cameras each including combination of different types to effectively a see-through view that would be generated by a single non-existent (or virtual) frontal see-through camera (Lee’s Col. 3, Ln. 37-42).
Allowable Subject Matter
3. Claim(s) 6-7, 9-10, 15-16, and 18-19 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
4. The following is an examiner’s statement of reasons for allowance:
Stahl et al. (US Patent/PGPub. No. 10914957) teach a mixed reality system including a head-mounted display (HMD) and a base station. Information collected by HMD sensors may be transmitted to the base via a wired or wireless connection. On the base, a rendering engine renders frames including virtual content based in part on the sensor information, and an encoder compresses the frames according to an encoding protocol before sending the frames to the HMD over the connection. Instead of using a previous frame to estimate motion vectors in the encoder, motion vectors from the HMD and the rendering engine are input to the encoder and used in compressing the frame. The motion vectors may be embedded in the data stream along with the encoded frame data and transmitted to the HMD over the connection. If a frame is not received at the HMD, the HMD may synthesize a frame from a previous frame using the motion vectors.
Lee (US Patent/PGPub. No. 11792352) teaches camera view synthesis on head-mounted display (HMD) for virtual reality (VR) and augmented reality (AR) are described. A method involves receiving, from a plurality of tracking cameras disposed around a HMD, image data of a scene which is on a first side of the HMD. The method also involves performing, using the image data and depth information pertaining to the scene, view synthesis to generate a see-through effect of viewing the scene from a viewing position on a second side of the HMD opposite the first side thereof.
The subject matter of the independent claims could either not be found or was not suggested in the prior art of record. The subject matter not found was a HMD device including
“…wherein the first type of camera and the second type of camera area co-located such that a field of view of the first type of camera substantially overlaps with a field of view of the second type of camera.” (Claim 6; Claim 15 is similar),
“…wherein the forward-facing camera components are co-located behind a cover window composed of glass.” (Claim 7; Claim 16 is similar),
“…colorizing image data from the first type of camera using color information from the second type of camera.” (Claim 9; Claim 18 is similar),
“…projecting structured light onto a controller operating with the head-wearable device; and
tracking a position of the controller using at least a portion of the first image data and the second image data.” (Claim 10; Claim 19 is similar),
in combination with the other elements (or steps) of the device or apparatus and method recited in the claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINH TANG LAM whose telephone number is (571) 270-3704. The examiner can normally be reached Monday to Friday 8:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nitin K Patel can be reached at (571) 272-7677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VINH T LAM/Primary Examiner, Art Unit 2628