DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required: “identifying an augmentation area in the second frame based on a location of the physical object relative to the MR device at the second pose, and the predefined position relative to the physical object” (recited in claims 1, 11, and 20); and
“applying a warping transformation of the virtual object based on the location of the physical object relative to the MR device at the second pose, and the predefined position relative to the physical object;
and identifying a projected location of the virtual object in the second frame based on the warping transformation, wherein identifying the augmentation area is based on the projected location of the virtual object in the second frame” (recited in claims 2 and 12)
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites:
“identifying an augmentation area in the second frame based on a location of the physical object relative to the MR device at the second pose, and the predefined position relative to the physical object”
It is unclear how the location of the physical object relative to the MR device and the predefined position relative to the physical object are used to identify an augmentation area in the second frame. Applicant’s Specification (Para 61) discloses identifying an augmentation area (e.g. region of interest) based on an original/rendered location of the virtual object in a current frame and the latest pose data identifying the pose of the AR device, which differs from the claimed feature. Correction is required.
Accordingly, claims 11 and 20 are similarly rejected.
Claims 2-10 and 12-19 are rejected based on dependency from a rejected base claim.
Claim 2 recites:
“applying a warping transformation of the virtual object based on the location of the physical object relative to the MR device at the second pose, and the predefined position relative to the physical object;
and identifying a projected location of the virtual object in the second frame based on the warping transformation, wherein identifying the augmentation area is based on the projected location of the virtual object in the second frame”
It is unclear how the location of the physical object relative to the MR device and the predefined position relative to the physical device are used to apply a warping transformation of the virtual object. Applicant’s Specification (Para 42) discloses the warping is based on a latest pos of the AR device, which differs from the claimed feature. Correction is required.
Accordingly, claim 12 is similarly rejected.
Claim 20 recites the limitation "the graphical processing unit" in line 15. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yoshinori Ohashi et al., US 2021/0390782 A1.
Independent clam 1, Ohashi discloses a method comprising:
determining a first pose of a mixed reality (MR) device (i.e. recognize display of a virtual object in an AR region at a position of the HMD/user viewpoint – Para 77; Fig. 5C “512”);
generating, based on the first pose, a first frame (i.e. the generated virtual object is rendered in line-of-sight direction and viewpoint position of the HMD user – Para 62, 66);
rendering, in the first frame, a virtual object that appears coupled to a physical object at a predefined position relative to the physical object (i.e. virtual object displayed positioned atop a physical table – Fig. 5A “530”; Fig. 6A “512”);
determining, after the first pose, a second pose of the MR device (i.e. tracker senses updated position that corresponds to the user viewpoint – Para 38, 67; HMD/user viewpoint moves from an initial position to a subsequent position – Fig. 5B-D);
generating, based on the second pose, a second frame (Fig. 7B/C);
identifying an augmentation area in the second frame based on a location of the physical object relative to the MR device at the second pose (i.e. image generation apparatus includes an image generation section – Fig. 2 - that includes an AR region determination section – Para 58 – for determining the AR region corresponding to the overlay/augmentation region – Para 60; augmentation region corresponds to the display position of a virtual object - Fig. 5 “512, 522“; Para 80), and the predefined position relative to the physical object (Fig. 6A);
determining depth information limited to the augmentation area in the second frame (i.e. determine depth or he AR region as the HMD/ user viewpoint position changes – Para 72, 80);
and rendering, in the second frame, the virtual object based on the depth information (i.e. following detected movement of the HMD, a graphical processing unit, e.g. GPU, performs rendering – Para 68; display a different, e.g. wider, view of the virtual object and other effects based on the change in depth– Fig. 5D; Para 80).
Ohashi suggests the virtual object appearing coupled to the physical object at the predefined position as Ohashi discloses displaying a virtual object augmenting a physical surface based on translational repositioning of the mixed reality device (Fig. 7 8).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention at the time the invention was made to include the virtual object appearing coupled to the physical object at the predefined position with the method of Ohashi because translational movement of the mixed reality device repositions the device relative to the physical object, which correspondingly adjusts the view of the virtual object positioned relative to the physical object, and achieves predictable results.
Claim 2, Ohashi discloses the method of claim 1, further comprising: applying a warping transformation of the virtual object based on the location of the physical object relative to the MR device at the second pose, and the predefined position relative to the physical object (i.e. The reprojection section receives the latest position and orientation information of the head-mounted display from the HMD position and orientation acquisition section and performs reprojection processing on the rendered augmented reality image on which the post-processing has been performed, to convert the image into an image that is visible from the latest viewpoint position and line-of-sight direction of the head-mounted display – Para 66, 67);
and identifying a projected location of the virtual object in the second frame based on the warping transformation, wherein identifying the augmentation area is based on the projected location of the virtual object in the second frame (i.e. image generation apparatus includes an image generation section – Fig. 2 - that includes an AR region determination section – Para 58 – for determining the AR region corresponding to the overlay/augmentation region – Para 60; augmentation region corresponds to the display position of a virtual object - Fig. 5 “512, 522“; Para 80);
Claim 3, Ohashi discloses the method of claim 1, further comprising: identifying a projected path of the virtual object based on a preconfigured dynamics behavior of the virtual object, wherein the augmentation area in the second frame comprises a projected location based on the projected path of the virtual object in the second frame (i.e. display virtual object in an updated position according to position of recognized moving object in an augmentation region - Fig. 5C, 5D “512“; Para 77, 78, 89).
Claim 4, Ohashi discloses the method of claim 1, wherein a size of the augmentation area corresponds to a size of the virtual object in the second frame (i.e. the virtual object size corresponds to the augmentation region in the physical space – Fig. 5-8; Para 77, 78).
Claim 5, Ohashi discloses the method of claim 1, wherein determining the first pose and the second pose is determined from a tracker measuring rotational, inclination movements of a user (Para 31, 44), wherein the 6DOF tracker comprises a visual-inertial odometry (VIO) system or a SLAM system (i.e. acquire inertial measurement unit data and perform simultaneous localization and mapping – Para 52).
Ohashi suggests a six-degrees of freedom (6DOF) tracker as he discloses measuring rotational, inclination, forward, rearward, leftward, rightward, upward and downward movements of the user device (Para 44).
It would have been obvious to one of skill in the art before the effective filing date of the claimed invention at the time the invention was made to combine a six-degrees of freedom tracker with the method of Ohashi because tracking movements to measure motion in various directions is exemplary of tracking motion.
Claim 6, Ohashi discloses the method of claim 1, wherein determining the second pose comprises: accessing a latest IMU data from the MR device (i.e. receive the latent position and orientation information – Para 66); and predicting the second pose based on the latest IMU data and the first pose (i.e. use position and orientation of the tracker to estimate a position and orientation of the body part – Para 63; the body part position corresponds to the viewpoint of the user and the AR region – Para 53, 62, 77).
Claim 7, Ohashi discloses the method of claim 1, wherein determining the depth information comprising: accessing a depth sensor of the MR device, the depth sensor comprising at least one a structured-light sensor, a time-of-flight sensor, passive stereo sensor, and an ultrasound device (i.e. depth sensors such as structured light or time of flight provides depth information from the real world – Para 102).
Claim 8, Ohashi discloses the method of claim 7, further comprising: configuring a setting of the depth sensor to limit depth sensing to the augmentation area (i.e. range sensor provided depth information - Para 46; apply post processing such as depth of filed adjustment to the AR region – Para 65; determine change in distance between the user and the moving object to determine the AR effect to apply - Para 82, 83).
Claim 9, Ohashi discloses the method of claim 1, wherein determining the depth information comprising: 27Docket No. 4218.C86US1 computing a depth based on a monocular image (i.e. determine depth based on features captured from a camera image – Para 65) or a 3D reconstructed scene (i.e. determine depth information based on information from a stereo camera – Para 102).
Claim 10, Ohashi discloses the method of claim 1, wherein the MR device comprises augmented reality glasses, (i.e. the display device may be glasses – Para 33).
Ohashi suggests wherein the first frame and the second frame do not depict the physical object as Ohashi discloses the view window size may be modified to change the degree of the external world outside the physical space that is displayed – Fig. 7C, 9; Para 81, 98, 104).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention at the time the invention was made to include wherein the first frame and the second frame do not depict the physical object with the known method of Ohashi because altering the size/shape of the view region correlates to narrowing the view of the physical space to correspondingly fill the view window with the display of the virtual object, which achieves predictable results.
Independent claim 11, the claim is similar in scope to claim 1. Therefore, similar rationale as applied in the rejection of claim 1 applies herein.
Claims 12-19, the corresponding rationale as applied in the rejection of claims 2-9 apply herein.
Independent claim 20, the claim is similar in scope to claim 1. Therefore, similar rationale as applied in the rejection of claim 1 applies herein.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHANTE HARRISON whose telephone number is (571)272-7659. The examiner can normally be reached Monday - Friday 8:00 am to 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 571-272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHANTE E HARRISON/Primary Examiner, Art Unit 2615
Prosecution Insights