Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 7-9 and 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mantzel (US Patent Pub. No.: US 2017/0318222 A1), hereinafter Mantzel, in view of Fortin-Deschenes (US Patent No.: US 11,199,706 B2), hereinafter Fortin-Deschenes, further in view of Song (Optimization of the Progressive Image Mosaicing Algorithm in Fine Art Image Fusion for Virtual Reality, IEEEAccess, date of publication September 7, 2020, date of current version May 17, 2021), hereinafter Song.
Regarding claim 1, Mantzel teaches a method comprising, by a computing system: receiving a color image captured by a color camera and a monochrome image captured by a monochrome camera (A device comprising a monochrome camera, a color camera and a processor may be configured to perform the techniques. Abstract); computing, for each of the color image and the monochrome image, histogram statistics (Camera processor 14 may then compare the features (which may be stored as histograms) to identify horizontal matches and thereby determine the parallax value indicative of the level of parallax remaining after registration (62). [0081]); performing, based on the histogram statistics (Camera processor 14 may then compare the features (which may be stored as histograms) to identify horizontal matches and thereby determine the parallax value indicative of the level of parallax remaining after registration (62). [0081]), tone map matching to normalize the monochrome image with respect to the color image (The processor may be configured to perform intensity equalization with respect to a luma component of either the color image data or the monochrome image data to correct for differences in intensity between the color camera and the monochrome camera. Abstract); performing local motion estimation to calculate motion vectors based on the first pyramid of images corresponding to the normalized monochrome image and the second pyramid of images corresponding to the color image (As shown in the example of FIG. 8, camera processor 14 may register combined color image 15A ( denoted as "Bayer" or "B") (which is the image at the bottom of the pyramid of the color images.) to combined mono image 15N (denoted as "Mono" or "M") (130) (which is the image at the bottom of the pyramid of the mono images.). Camera processor 14 may next compute shift values (i.e., the motion vectors), as described in more detail with respect to FIG. 9 (132). Camera processor 14 may output the shift values as an array of shift values, each shift value specified to shift a variable (x) number of pixels horizontally. [0092].
PNG
media_image1.png
670
1456
media_image1.png
Greyscale
), wherein the motion vectors indicate pixel correspondence between the normalized monochrome image and the color image (As shown in the example of FIG. 8, camera processor 14 may register combined color image 15A ( denoted as "Bayer" or "B") to combined mono image 15N (denoted as "Mono" or "M") (130). Camera processor 14 may next compute shift values, as described in more detail with respect to FIG. 9 (132). Camera processor 14 may output the shift values (i.e., the motion vectors) as an array of shift values, each shift value specified to shift a variable (x) number of pixels horizontally. [0092]); and generating a mono-color merged image by adding, for each pixel in the normalized monochrome image, color information extracted from corresponding pixel in the color image (Camera processor 14 may then register the chroma component of enhanced color image 17 to the luma component of combined mono image 15N to generate the fused image (which is another way by which to refer to enhanced color image 17) (140). [0093]) using the motion vectors (As shown in the example of FIG. 8, camera processor 14 may register combined color image 15A ( denoted as "Bayer" or "B") to combined mono image 15N (denoted as "Mono" or "M") (130). Camera processor 14 may next compute shift values, as described in more detail with respect to FIG. 9 (132). Camera processor 14 may output the shift values as an array of shift values, each shift value specified to shift a variable (x) number of pixels horizontally. [0092].
PNG
media_image2.png
640
1466
media_image2.png
Greyscale
).
Mantzel does not teach the following limitations as further recited, but Fortin-Deschenes further teaches wherein the color camera and the monochrome camera are associated with an artificial reality system (In accordance with an aspect of the disclosure, there is provided a Head-Mounted Display (HMD) device used for applications that immerse a user in a virtual reality (VR) or an augmented/mixed reality (MR) environment, comprising: a pair of RGB camera sensors and associated lenses with infrared (IR) cut-off filters; a pair of mono camera sensors with near infrared (NIR) bandpass filters and associated lenses. Column 2 line 59); generating image for display on the artificial reality system (The processing units (21) render graphic content (game view, video, virtual objects, etc.) to be displayed in the HMD (7). Column 7 line 8).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mantzel to incorporate the teachings of Fortin-Deschenes to utilize the color camera and the monochrome camera in an artificial reality system to generate image for display on the artificial reality system in order to improve the HMD camera system performance such as the image quality (noise, dynamic range, resolution, absence of artifacts).
The combination of Mantzel and Fortin-Deschenes does not teach the following limitations as further recited, but Song further teaches performing gaussian pyramid decomposition (In each layer of the Gaussian pyramid, a Gaussian (a, b, c) with different blur parameters is used to convolve the source image I (a, b) to obtain images with different degrees of blur in the same layer, namely, the Gaussian scale space. Page 69563 left column last paragraph) to transform normalized monochrome image (Since the original image has considerable noise, the image needs to be preprocessed, and the result after equalization through the histogram is shown in Figure 8. Page 69566 right column 5th paragraph.
PNG
media_image3.png
602
624
media_image3.png
Greyscale
) into a first pyramid of images ((4) Pyramid fusion method: This method first performs a series of multiresolution decompositions on two images to be mosaicked. This technology first decomposes the resolution of images and then performs image fusion. Page 69563 left column 2nd paragraph) and the color image (Since the original image has considerable noise, the image needs to be preprocessed, and the result after equalization through the histogram is shown in Figure 8. Page 69566 right column 5th paragraph. It is common knowledge the image preprocessing equalization is performed on two images.) into a second pyramid of images (This method first performs a series of multiresolution decompositions on two images to be mosaicked. This technology first decomposes the resolution of images and then performs image fusion. Page 69563 left column 2nd paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mantzel and Fortin-Deschenes to incorporate the teachings of Song to perform gaussian pyramid decomposition to transform normalized monochrome image into a first pyramid of images and the color image into a second pyramid of images in order to effectively fuse the contents of images and make the fusion result very smooth.
Regarding claim 2, Mantzel in the combination teaches the method of Claim 1, further comprising: applying one or more post processing functions to the mono-color merged image to remove one or more noise artifacts from the mono-color merged image (Camera processor 14 may also perform noise reduction and image sharpening, as additional examples. [0059]); and generating a de-noised mono-color merged image (The techniques may, in this way, leverage the higher signal-to-noise ratio (SNR) and resolution of the mono image (as a result of not having a color filter) to create a better (in terms of clarity, artifacts, and/or noise) fused image. [0123]) based on applying the one or more post processing functions (Camera processor 14 may also perform noise reduction and image sharpening, as additional examples. [0059]).
Regarding claim 4, Mantzel in the combination teaches the method of Claim 2, wherein the one or more noise artifacts comprise: luma noise from the monochrome camera; color noise from the color camera (Also, color filters block some of the light that hits the sensor and lower light level typically translates to higher noise level. [0037]); or low frequency chroma noise from the color camera.
Regarding claim 5, Fortin-Deschenes in the combination teaches the method of Claim 2, further comprising: displaying the de-noised mono-color merged image as a passthrough image on a display of the artificial reality system (In virtual reality, the user's body needs to be drawn and seen to achieve immersion. To this end, the field of view over which the tracking is performed should match the field of view of the passthrough cameras. Column 9 line 50. at least one processing unit operatively connected to the pair of RGB camera sensors, the pair of mono cameras sensors, the IMU, the ToF camera sensor and associated IR emitter, speckle projector and display via at least one communication link, the at least one processing unit generating graphic content using data streams from the pair of RGB camera sensors, the pair of mono cameras sensors, the IMU and the ToF camera sensor and displaying the graphic content through the display. Column 3 line 5).
Regarding claim 7, Song in the combination teaches the method of Claim 1, wherein the histogram statistics comprise one or more properties associated with color image and the monochrome image (
PNG
media_image3.png
602
624
media_image3.png
Greyscale
. The x axis in the histogram shown in FIGURE 8 is luminance level.).
Regarding claim 8, Song in the combination teaches the method of Claim 7, wherein the one or more properties comprise brightness or luminance levels (
PNG
media_image3.png
602
624
media_image3.png
Greyscale
. The x axis in the histogram shown in FIGURE 8 is luminance level.).
Regarding claim 9, Mantzel in the combination teaches the method of Claim 7, wherein normalizing the monochrome image with respect to the color image comprises: adjusting the one or more properties in the monochrome image to align with the one or more properties of the color image (The processor may be configured to perform intensity equalization with respect to a luma component of either the color image data or the monochrome image data to correct for differences in intensity between the color camera and the monochrome camera. Abstract).
Regarding claim 11, Fortin-Deschenes in the combination teaches the method of Claim 1, wherein a resolution of each of the monochrome and color cameras (In accordance with an aspect of the disclosure, there is provided a Head-Mounted Display (HMD) device used for applications that immerse a user in a virtual reality (VR) or an augmented/mixed reality (MR) environment, comprising: a pair of RGB camera sensors and associated lenses with infrared (IR) cut-off filters; a pair of mono camera sensors with near infrared (NIR) bandpass filters and associated lenses. Column 2 line 59) is the same (Depending on the cost target, type of market, technological approach, device resolution, performance and feature-set, different choices in the embodiments can be made and several functional blocks can be combined. Column 7 line 66. A person with ordinary skill in the art would recognize a resolution of each of the monochrome and color cameras can be chosen to be the same.).
Regarding claim 12, Fortin-Deschenes in the combination teaches the method of Claim 1, wherein color and monochrome cameras are mounted next to each other on a head-mounted device (HMD) of the artificial reality system (In a first exemplary embodiment, shown in FIG. 4A, the HMD (7) has two RGB camera sensors (62, 64) and lenses (63, 65) with IR cut-off filters for better pass-through quality. It also integrates two mono camera sensors with near infrared (NIR) bandpass filters (66, 68) and lenses (67, 69) optimized for computer vision analysis. Column 8 line 20.
PNG
media_image4.png
644
1014
media_image4.png
Greyscale
).
Regarding claim 13, Mantzel in the combination teaches the method of Claim 1, wherein the color image and the monochrome image are synchronously captured at similar times (Camera processor 14 may coordinate the concurrent capture of color image data 13A and mono image data 13N by interfacing with each of color camera 12A and mono camera 12N concurrently to capture the scene at approximately the same time. [0070]).
Regarding claim 14, Fortin-Deschenes in the combination teaches the method of Claim 1, wherein the artificial reality system is a mixed reality headset (In accordance with an aspect of the disclosure, there is provided a Head-Mounted Display (HMD) device used for applications that immerse a user in a virtual reality (VR) or an augmented/mixed reality (MR) environment, comprising: a pair of RGB camera sensors and associated lenses with infrared (IR) cut-off filters; a pair of mono camera sensors with near infrared (NIR) bandpass filters and associated lenses. Column 2 line 59).
Claims 15-17 are drawn to a non-transitory computer-readable storage medium having executable instructions stored for carrying out the method of claims 1-3. Therefore, claims 15-17 correspond to method claims 1-3, and are rejected for the same reasons of obviousness as used above.
Apparatus claims 18-20 are drawn to the apparatus corresponding to the method of using same as claimed in claims 1-3. Therefore apparatus claims 18-20 correspond to method claims 1-3, and are rejected for the same reasons of obviousness as used above.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Mantzel (US Patent Pub. No.: US 2017/0318222 A1), hereinafter Mantzel, in view of Fortin-Deschenes (US Patent No.: US 11,199,706 B2), hereinafter Fortin-Deschenes, further in view of Song (Optimization of the Progressive Image Mosaicing Algorithm in Fine Art Image Fusion for Virtual Reality, IEEEAccess, date of publication September 7, 2020, date of current version May 17, 2021), hereinafter Song, further in view of Hsu (US Patent Pub. No.: US 2013/0128061 A1), hereinafter Hsu.
Regarding claim 3, Mantzel, Fortin-Deschenes and Song teach all of the elements of the claimed invention as stated in claim 2 except for the following limitations as further recited. However, Hsu teaches wherein the one or more post processing functions comprise: temporal noise reduction; and spatial noise reduction (The image processing apparatus includes an image capturing module, an image separation module, an image stabilization module, a temporal noise reduction module, and a spatial noise reduction module. Abstract).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mantzel, Fortin-Deschenes and Song to incorporate the teachings of Hsu to incorporate temporal noise reduction and spatial noise reduction in the post processing functions in order to improve the image quality.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Mantzel (US Patent Pub. No.: US 2017/0318222 A1), hereinafter Mantzel, in view of Fortin-Deschenes (US Patent No.: US 11,199,706 B2), hereinafter Fortin-Deschenes, further in view of Song (Optimization of the Progressive Image Mosaicing Algorithm in Fine Art Image Fusion for Virtual Reality, IEEEAccess, date of publication September 7, 2020, date of current version May 17, 2021), hereinafter Song, further in view of Macken (US Patent Pub. No.: US 2022/0206299 A1), hereinafter Macken, further in view of Solh (US Patent No.: US 9,414,037 B1), hereinafter Solh.
Regarding claim 6, Mantzel, Fortin-Deschenes and Song teach all of the elements of the claimed invention as stated in claim 1 except for the following limitations as further recited. However, Macken teaches determining (Determining one or more lighting conditions in the field of view may comprise selecting at least one appropriate lighting data source based on the field of view. Determining one or more lighting conditions in the field of view may comprise receiving data from the lighting data source indicating the current lighting conditions. [0018]) that lighting conditions associated with the artificial reality system fall within a certain luminance range (According to the present invention, the luminance profile is adjusted according to the user's visual field of view, reducing the possibility that glow or low visibility of images impacts the user's perception of either their surroundings or the images on the HMD. [0082]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mantzel, Fortin-Deschenes and Song to incorporate the teachings of Macken to determine that lighting conditions associated with the artificial reality system fall within a certain luminance range in order to detect different or changing environments of the HMDs so not to impair visibility for the user.
The combination of Mantzel, Fortin-Deschenes, Song and Macken does not teach the following limitations as further recited, but Solh further teaches performing a mono-color fusion process to generate the mono-color merged image responsive to determining that the lighting conditions associated with the artificial reality system fall within the certain luminance range (In order to reduce noise and increase the quality of images captured in low lighting conditions, the luma ( or brightness) information Y from mono image 302 is fused with U and Y (chrominance) information from color image 304. Column 6 line 37).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Solh to perform a mono-color fusion process to generate the mono-color merged image responsive to determining that the lighting conditions associated with the artificial reality system fall within the certain luminance range in order to enhance images in low lighting conditions.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Mantzel (US Patent Pub. No.: US 2017/0318222 A1), hereinafter Mantzel, in view of Fortin-Deschenes (US Patent No.: US 11,199,706 B2), hereinafter Fortin-Deschenes, further in view of Song (Optimization of the Progressive Image Mosaicing Algorithm in Fine Art Image Fusion for Virtual Reality, IEEEAccess, date of publication September 7, 2020, date of current version May 17, 2021), hereinafter Song, further in view of Musatenko (US Patent No.: US 9,344,639 B2), hereinafter Musatenko.
Regarding claim 10, Mantzel, Fortin-Deschenes and Song teach all of the elements of the claimed invention as stated in claim 1 except for the following limitations as further recited. However, Musatenko teaches wherein a resolution of the monochrome camera is higher than a resolution of the color camera (Additionally, a recombination process to generate the enhanced, high-resolution HDR color image will have a resolution as determined from the HDR mono image sensors, which is going to be much higher than the resolution of the Bayer sensor (both in HDR mode). Column 2 line 35).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Mantzel, Fortin-Deschenes and Song to incorporate the teachings of Musatenko to utilize a monochrome camera with higher resolution than a color camera in order to generate the enhanced, high-resolution color image.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHAO whose telephone number is (703)756-1922. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VU LE can be reached at (571)272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEI ZHAO/Examiner, Art Unit 2668
/VU LE/Supervisory Patent Examiner, Art Unit 2668