DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because:
Fig. 2 does not depict a flowchart listing steps involved in a method for static distortion correction for a head mounted display (HMD) as disclosed in the specifications.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The abstract of the disclosure is objected to because:
In line 8, “(400)” should be removed. Figure numbers that point to a certain item in the present invention should be used in the specification.
A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claim 1 and 8 are objected to because of the following informalities:
In line 16 (of claim 1) and line 19 (of claim 8), “(320)” should be removed. Figure numbers that point to a certain item in the present invention should be used in the specification.
Claim 4 is objected to because of the following informalities:
The use of the phrase “collected by one of” suggests selection from the following group of elements. However, the listings do not use “or” or “and” to indicate how the listing should be read. This creates ambiguity as to whether the claim requires a single element from the list or all listed in combination and therefore fails to particularly point out the invention. For the purposes of examination, the examiner has interpreted that every element in the list must be required.
Claim 7 and 13 are objected to because of the following informalities:
The use of the phrase “comprising one of” suggests selection from the following group of elements. However, the listing uses “and” which implies that all of the elements must be present. This creates ambiguity as to whether the claim requires a single element from the list or all listed in combination and therefore fails to particularly point out the invention. For the purposes of examination, the examiner has interpreted that every element in the list must be required.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5, 8, and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bates et al (US 10156723 B2) in view of Kanehara et al (US 20190377270 A1) and Hiroi et al (“Neural Distortion Fields for Spatial Calibration of Wide Field-of-View Near-Eye Displays”), hereinafter Bates, Kanehara, and Hiroi respectively.
Regarding claim 1, Bates teaches a method for static distortion correction for a head mounted display (HMD) (“Display pre-distortion methods and apparatus for HMDs are disclosed” – Col 1, Lines 19-20), the method comprising: collecting distortion data (“For example, if a wearer's pupil is off center optically, as compared or relative to an average pupil position, then a display can be pre-distorted (e.g., modified)” – Par 22, Lines 6-8 [NOTE: in order to perform the correction modification, the distortion must inherently be collected for proper correction.]), of an optical arrangement of the HMD (“The example HMD 200 of FIGS. 2A-C has a pair of apertures 205 and 206 defined through a rear surface 210 for respective ones of a pair of lenses 215 and 216, and respective ones of a pair of displays 220 and 221” – Par 27, Lines 7-10, Figs. 2A-C), corresponding to a plurality of gaze directions wherein each of the plurality of gaze directions is a direction of sight defined by a pupil-position or a head-position with respect to a display screen of the HMD (“A disclosed example HMD includes a lens, an eye tracker to determine a position of a pupil relative to the lens” – Col 1 Lines 20-22), and wherein the distortion data comprises display pixel coordinates for display pixels and corresponding image plane coordinates (“A disclosed example HMD includes a lens, an eye tracker to determine a position of a pupil relative to the lens” – Col 1 Lines 20-22“An example pre-distortion map shown in FIG. 13 defines that a blue pixel at a first pixel location (X1, Y1) should instead be displayed at a second pixel location (X2, Y2).” – Col 2, Lines 51-54, note: the pixels form on the display is image plane); generating a distortion map corresponding to each of the plurality of gaze directions (“The pre-distortion map(s) can be dynamically adapted (e.g., for each frame) as the user wears an HMD according to their changing pupil location.” – Col 3, Lines 1-4), wherein the distortion map comprises virtual plane coordinates of the corresponding display pixels determined using the distortion data (“An example pre-distortion map shown in FIG. 13 defines that a blue pixel at a first pixel location (X1, Y1) should instead be displayed at a second pixel location (X2, Y2).” – Col 2, Lines 51-54).
Bates does not teach combining the distortion map, corresponding to each of the plurality of gaze directions, to compute a single multi-view distortion map, wherein the single multi-view distortion map is defined by calculating a weighted average of each virtual plane coordinates on the distortion map for the plurality of gaze directions.
However, Kanehara teaches combining the distortion map, corresponding to each of the plurality of gaze directions, to compute a single multi-view distortion map (“The control unit may combine (or synthesize) the fine distortion map and the coarse distortion map to create a combined distortion map, which may also be referred to as a composite distortion map or an integrated distortion map.” – Par 123, Lines 9-12).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Bates to incorporate the teachings of Kanehara to combine multiple distortion maps to create a single combined distortion map. The distortion maps generated from the gaze directions provide information how the correction should look at that particular gaze direction. Since not all of that information is relevant, by combining the distortion maps, there would be a more consistent view of how the undistorted image should look across multiple gazes.
Bates in view of Kanehara still does not teach wherein the single multi-view distortion map is defined by calculating a weighted average of each virtual plane coordinates on the distortion map for the plurality of gaze directions.
However, Hiroi teaches wherein the single multi-view distortion map is defined by calculating a weighted average of each virtual plane coordinates on the distortion map for the plurality of gaze directions (“NDF takes spatial position and gaze direction as input and outputs the display pixel coordinate and its intensity as perceived in the input gaze direction. We synthesize the distortion map from a novel viewpoint by querying points on the ray from the viewpoint and computing a weighted sum to project output display coordinates into an image.” - Abstract, Par 2, Lines 2-6; note: under special situation, when the total weight =1, the weight average is the weighted sum, also as previously discussed, each gaze/viewing direction onto the display plane is a virtual plane and each have its own distortion map, all the individual distortion map combined distortion map from a single Multiview distortion map).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Bates to incorporate the teachings of Hiroi to define the multi-view distortion map by calculating a weighted average of each virtual plane coordinates on the distortion map. The virtual plane coordinates provide an undistorted view of what the display pixels should look like. One of ordinary skill in the art would know that taking the weighted average of each virtual plane coordinate would build a consistent distortion map which would improve the potency of the distortion correction. [NOTE: After the combination, the single multi-view distortion map as disclosed by Kanehara can then be applied on a display content to be rendered on the display screen for the static distortion correction for an entire field of view associated with the display screen of the HMD as taught by Bates (“Once all pixels have been processed, the pre-distorted (e.g., modified) image is sent to the display 720, 721 for output to a wearer of the HMD 700” – Lines 25-28)].
Regarding claim 8, the claim describes a head mounted display (HMD) that performs the steps of a method described in claim 1. Therefore, the HMD apparatus claim 8 corresponds to the method disclosed in claim 1 and is rejected for the same reasons above.
Regarding claim 5, Bates in view of Kanehara and Hiroi teach the method of claim 1. Bates further teaches wherein the distortion data is collected for a pre-defined set of pixels on the display screen or every pixel on the display screen (“If all pixels and colors have been processed (block 930), control returns from the example method of FIG. 9 to, for example, block 830 of FIG. 8. If not all pixels and colors have been processed (block 930), control returns to block 905 to select another pixel.” – Col 7, Lines 53-57. [NOTE: the process disclosed by Bates show that all the pixels must be processed first implying that the distortion data collection considers all of the pixels on the display screen.]). It would have been obvious to one of ordinary skill in the art before the effective filing date to further include the teaching of Bates to collect the distortion data for all of the pixels on the display screen. One of ordinary skill in the art would recognize that collecting the distortion data for each pixel would create a consistent visual that shows how the distorted image would appear.
Regarding claim 11, the claim describes a head mounted display (HMD) that performs the steps of a method described in claim 5. Therefore, the HMD apparatus claim 11 corresponds to the method disclosed in claim 5 and is rejected for the same reasons above.
Claim(s) 3 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bates, Kanehara, Hiroi, and Park (US 20210271880 A1), hereinafter Park.
Regarding claim 3, Bates in view of Kanehara and Hiroi teach the method of claim 1. Bates does not teach wherein a weight of the corresponding virtual plane coordinate (410A) on the distortion map is based on a type of the gaze direction encompassing the corresponding virtual plane coordinate, and wherein the type of the gaze direction comprises a critical gaze direction having a higher weight based a higher probability of being viewed, a moderate gaze direction having a moderate weight based a moderate probability of being viewed and a non-critical gaze direction having a lower weight based a low probability of being viewed. However, Park teaches wherein a weight of the corresponding virtual plane coordinate (410A) on the distortion map is based on a type of the gaze direction encompassing the corresponding virtual plane coordinate (“In operation 120, the prediction apparatus generates the plurality of weight filters. The prediction apparatus may generate a first filter which is based on a gaze location of the user”, Par 48), and wherein the type of the gaze direction comprises a critical gaze direction having a higher weight based a higher probability of being viewed (“Thus, a high weight (for example, a first weight) may be assigned to a class estimation probability for pixel(s) corresponding to the first region 220”, Par 63), a moderate gaze direction having a moderate weight based a moderate probability of being viewed and a non-critical gaze direction having a lower weight based a low probability of being viewed (“a relatively low weight (for example, a second weight), when compared to the first region 220, may be assigned to a class estimation probability for pixel(s) corresponding to the second region 230”, Par 63). [NOTE: Adding a moderate weight for a moderate gaze direction would be simple for one of ordinary skill in the art given that a higher and lower weight can be assigned based on the probability of the gaze direction being viewed.]. It would have been obvious to one of ordinary skill in the art to modify bates to incorporate the teachings of Park to base the weights off the probability of the gaze directions. One of ordinary skill in the art would recognize that distortion data would be more important where the gaze direction lies and less prevalent in the periphery. With the weighted distortion data, the merged distortion maps provide accurate correction to the image with an emphasis on the gaze direction.
Regarding claim 10, the claim describes a head mounted display (HMD) that performs the steps of a method described in claim 3. Therefore, the HMD apparatus claim 10 corresponds to the method disclosed in claim 3 and is rejected for the same reasons above.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bates, Kanehara, Hiroi, Richards et al (US 10551879 B1), and Ye et al (US 20240177637 A1), hereinafter Richards and Ye respectively.
Regarding claim 4, Bates in view of Kanehara and Hiroi teach the method of claim 1. Bates does not teach wherein the distorted data is collected by one of: using software to simulate propagation of light through the optical arrangement), using a camera setup to capture images through the optical arrangement, using software to simulate propagation of light through an inverted optical system, wherein the display screen is used as an image plane. However, Richards teaches wherein the distorted data is collected by one of: using software (“Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules” – Col 10, Lines 11-13) to simulate propagation of light through the optical arrangement), using a camera setup to capture images through the optical arrangement (“In some embodiments, the eye tracking unit 260 includes one or more emitters to illuminate the eye 220 with light and a camera to capture images of the eye 220.” - Col 4, Lines 25-28). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Bates to incorporate the teachings of Richards to use software to use software to simulate the propagation of light through the optical arrangement or use a camera set up to capture images through the optical arrangement. One of ordinary skill would recognize that both the software simulation or camera set up are known methods of data collection for optical arrangements/systems. Applying these methods to distortion correction would yield predictable results. Bates in view of Kanehara, Hiroi, and Richards still does not teach using software to simulate propagation of light through an inverted optical system, wherein the display screen is used as an image plane. However, Ye teaches using software to simulate propagation of light through an inverted optical system, wherein the display screen is used as an image plane (“The display panel 202 is disposed at an object plane 203 of the collimator 206, and the eyebox 204A is disposed at an image plane 205 of the collimator 206. Sets of parallel rays 207 are traced backwards, i.e. from the eyebox 204A” – Par 49, Lines 9-13. [NOTE: optical design software, par 47, is used to trace of light rays set toward the display panel 102.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Bates to incorporate the teachings of Ye to simulate propagated light through an inverted optical system, wherein the display screen is used as an image plane. This is an alternative strategy that is known in the art and would serve the same purpose with predictable results as the other listed forms of collecting distorted data.
Claim(s) 6-7 and 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bates, Kanehara, Hiroi, and Blackmon et al (US 20170169602 A1), hereinafter Blackmon.
Regarding claim 6, Bates in view of Kanehara and Hiroi teach the method of claim 1. Bates does not teach wherein the plurality of gaze directions is associated with arbitrary regions of the display screen or pre-defined regions of interest for the display screen. However, Blackmon teaches wherein the plurality of gaze directions is associated with arbitrary regions of the display screen or pre-defined regions of interest for the display screen (“The processing system may further comprise gaze tracking logic configured to determine a gaze position for an image to be rendered, wherein a region of interest of the image is based on the determined gaze position” - Par 10, Lines 17-20). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Bates to incorporate the teachings of Blackmon to have the gaze directions be associated to a region of interest. Blackmon specifically teaches that the gaze directions are associated to a region of interest which can benefit the system by maintain high image quality on that region of interest and less on the periphery. This advantage can be applied to the present application by localizing the distortion correction at the focus of attention to prioritize the visually important areas.
Regarding claim 12, the claim describes a head mounted display (HMD) that performs the steps of a method described in claim 6. Therefore, the HMD apparatus claim 12 corresponds to the method disclosed in claim 6 and is rejected for the same reasons above.
Regarding claim 7, Bates in view of Kanehara and Hiroi teach the method of claim 1. Bates further teaches wherein generating the distortion map comprise one of: generating the distortion map for a full field of view corresponding to each of the plurality of gaze directions (“a pre-distortion map for an image to be displayed on a display screen of the HMD associated with the pupil” - Col 1, Lines 23-25), for the entire field of view of the display screen (“displaying the image on the display screen of the HMD, wherein the image is modified by the pre-distortion map” - Col 1, Lines 25-27). Bates in view of Kanehara and Hiroi do not teach generating the distortion map for a reduced field of view corresponding to each or some of the plurality of gaze directions. However, Blackmon teaches generating the distortion map for a reduced field of view corresponding to each or some of the plurality of gaze directions (“As an example, the size of the region of interest may correspond to a 5° field of view from the user's viewpoint. In this way, the region of interest would correspond to a typical foveal region of the user's field of view” – Par 50, Lines 1-4). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application generate the distortion map for a full field of view or reduced field of view. One of ordinary skill in the art would know that distortion correction relies on the distortion map to point out the disparities between the distorted image and what should be seen. It would be a logical step to have the distortion map be generated for the entire field of view or the reduced field of view based on a region of interest on the HMD display screen.
Regarding claim 13, the claim describes a head mounted display (HMD) that performs the steps of a method described in claim 7. Therefore, the HMD apparatus claim 13 corresponds to the method disclosed in claim 7 and is rejected for the same reasons above.
Allowable Subject Matter
Claims 2 and 9 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 2, Bates in view of Kanehara and Hiroi teach the method of claim 1. Bates does not teach wherein a weight of a corresponding virtual plane coordinate (410A) on the distortion map is based on a distance (D) between the corresponding virtual plane coordinate and a gaze direction (310A) of the distortion map, and wherein the weight decreases with increase of the distance. However, Smith (US 20180183986 A1), hereinafter Smith, teaches wherein the weight decreases with increase of the distance (“In some examples, similar to the center metering weight array 230, weights of the eye gaze weight array 690 may decrease as distance increases from a location that the gaze 670 is looking” – Par 102, Lines 5-9). Bates in view of Kanehara, Hiroi, and Smith still does not teach wherein a weight of a corresponding virtual plane coordinate (410A) on the distortion map is based on a distance (D) between the corresponding virtual plane coordinate and a gaze direction (310A) of the distortion map. None of the prior art searched, alone or in combination, renders obvious to the limitations of claim 2.
Regarding claim 9, the claim describes a head mounted display (HMD) that performs the steps of a method described in claim 2. Therefore, the HMD apparatus claim 9 corresponds to the method disclosed in claim 2 and would be allowable for the same reasons above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID V. NGUYEN whose telephone number is (571)272-6111. The examiner can normally be reached M-F 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID VAN NGUYEN/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617