DETAILED ACTION
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new grounds of rejection.
Regarding to the 35 U.S.C 112 (b) rejection of claim 7, the amendment has cured the basis of the 35 U.S.C 112 (b) rejection. Therefore, the 35 U.S.C 112 (b) rejection of claim 7 is hereby withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 10-12, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Couche (US 20240112411 A1) in view of Edwards (US 20230360333 A1).
As per claim 1, Couche teaches the claimed:
1. A computer-implemented method of determining a virtual pose for view of
content, the computer-implemented method comprising:
identifying a real-world environment of a client device;
(Couche [0019]: “For example, a physical environment is visible through a transparent portion of the display generation component (e.g., true or real passthrough) of the electronic device.”)
obtaining a 3D map corresponding to the real-world environment;
(Couche [0019]: “In some examples, a representation of the physical environment is displayed in the three-dimensional environment via the display generation component.”)
obtaining a physical pose of a camera of the client device;
(Couche [0027]: “In some examples, device 200 uses image sensor(s) 206 to detect the position and orientation of device 200”)
Couche alone does not explicitly teach the remaining claim limitations.
However, Couche in combination with Edwards teaches the claimed:
determining the virtual pose by applying an offset to the physical pose; and
(Edwards [0064]: “The reference point 310 denotes the relative location of where the physical environment should appear to be within the computer-generated environment. In the example shown here, an X,Y offset 310 indicates the positioning of the location of the physical location within this terrain map, providing a frame of reference for the rendering.”
Edwards [0075]: “The virtual camera’s position within the virtual environment may be offset from the corresponding position of the physical camera 308 within the physical environment”
Edwards teaches the reference point or offsets that is used to indicate where to position the virtual location when rendering based on the physical location.)
causing display, by the client device, of a view of the real-world environment from the virtual pose, the view of the real-world environment generated from the 3D map using the virtual pose.
(Edwards [0009]: “The AR application may then display the captured video of the physical object, such that the view of the physical background is replaced with a view of the virtual environment from the second position or orientation of the virtual camera (e.g., that was created by rendering or ray tracing the 3D Model from the new selected position of a virtual camera).”
Edwards [0062]: “The AR application may establish a reference point 310 that denotes the spatial relationship between the 3D model 302 and physical environment where the physical camera 308 is located.”
Edwards [0075]: “The virtual camera’s position within the virtual environment may be offset from the corresponding position of the physical camera 308 within the physical environment, but move in a synchronized way as the physical camera 308 is moved … This process may be repeated frame-by-frame as either the camera 308 is moved or reoriented, or as the 3D model is updated.”
Edwards teaches the displaying of the virtual environment, which is the view of the real-world environment, that is based on the physical environment with an offset. This indicates that the virtual environment is generated from the 3D model, which is based on the physical location, and the offset or reference point, which is the virtual pose. Thus, Edwards teaches the view of the virtual environment that is different than the actual physical environment.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the offset as taught by Edwards with the system of Couche in order to adjust a rendered viewpoint of the device’s actual position and orientation for more stable and flexible visuals that the user desires.
As per claim 11, this claim is similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale.
As per claim 2, Couche teaches the claimed:
2. The computer-implemented method of claim 1, wherein obtaining the 3D map
comprises generating the 3D map based on images captured by the camera of the client device.
(Couche [0020]: “Conversely, in some examples, the electronic device 100 captures one or more images of the physical environment 110 around the electronic device 100 and displays the representations of the physical environment 110 in the three-dimensional environment 102.”)
As per claim 12, this claim is similar in scope to limitations recited in claim 2, and thus is rejected under the same rationale.
As per claim 10, Couche teaches the claimed:
10. The computer-implemented method of claim 1, further comprising causing display of a virtual item in conjunction with the view of the real-world environment generated from the 3D map.
(Couche [0020]: “The three-dimensional environment 102 includes a virtual pointer element 106 that is offset from a portion of the input device 104 and one or more generated virtual objects that are generated in accordance with the movement of the input device 104.”)
As per claim 20, this claim is similar in scope to limitations recited in claim 10, and thus is rejected under the same rationale.
Claims 3-5 and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Couche in view of Edwards and in further view of Li (US 20200250889 A1).
As per claim 3, Couche and Edwards alone do not explicitly teach the claimed limitations.
However, Couche and Edwards in combination with Li teaches the claimed:
3. The computer-implemented method of claim 1, wherein obtaining the physical
pose of the camera comprises determining the physical pose of the camera by comparing an
image captured by the camera of the client device to the 3D map.
(Li [0070]: “At operation 402, the client device is configured to control an image capture module to capture at least one current image of the real-world object. The at least one current image is processed to identify features (for example straight edges, patches of lighter-than- (or darker-than-) surrounding pixels, etc.), operation 404.”
Li [0072]: “The tracked changes in the location and orientation of the identified features are then processed, operation 408, to derive a camera pose (in three dimensions) that, when projected onto a two-dimensional plane best matches the changes in location and orientation of the identified features.” Li teaches the projection onto a 2D plane, indicating that there is a 3D representation of the image, thus comparing the changes in location and orientation of the identified features will help derive a camera pose.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the capture module as taught by Li with the system of Couche as modified by Edwards in order to be able to find changes in features of the captured images with a 3D projection to determine and derive a camera pose.
As per claim 13, this claim is similar in scope to limitations recited in claim 3, and thus is rejected under the same rationale.
As per claim 4, Couche and Edwards alone do not explicitly teach the claimed limitations.
However, Couche and Edwards in combination with Li teaches the claimed:
4. The computer-implemented method of claim 1, wherein the 3D map is a topographical mesh of the real-world environment.
(Li [0057]: “The client device includes at least one processor and the processor operates to access a UV mapping and a 3D mesh corresponding to a real-world object or environment, operation 202.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the 3D mesh as taught by Li with the system of Couche as modified by Edwards in order to represent the real world as a continuous surface with accurate alignments of objects.
As per claim 14, this claim is similar in scope to limitations recited in claim 4, and thus is rejected under the same rationale.
As per claim 5, Couche and Edwards alone do not explicitly teach the claimed limitations.
However, Couche and Edwards in combination with Li teaches the claimed:
5. The computer-implemented method of claim 1, wherein the offset comprises at least one of a predetermined rotation or a predetermined translation.
(Li [0089]: “Examples of other transformations include … translations (in which part or all of an original 3D mesh is mapped to an undistorted 3D mesh at a spatial offset in virtual content space from the location of the original real-world object).”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the translation offset as taught by Li with the system of Couche as modified by Edwards in order to provide for controlled transformations, allowing room for noise, precise alignment with objects, and stable rendering of the scene.
As per claim 15, this claim is similar in scope to limitations recited in claim 5, and thus is rejected under the same rationale.
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Couche in view of Edwards and in further view of Ma (WO 2023003558 A1) and in further view of Rowell (US 20200342652 A1).
As per claim 6, Couche and Edwards alone do not explicitly teach the claimed limitations.
However, Couche and Edwards in combination with Ma and Rowell teaches the claimed:
6. The computer-implemented method of claim 1, wherein the offset is determined
by a calibration process that comprises:
directing a user to position the client device in a desired position; (Ma [0042]: “In some embodiments, the user is first guided to adjust a FOV of the display until the size of the rendered virtual object 360 matches that of real-world counterpart … Then the user is guided to adjust the FOV or convergence display settings until the rendered virtual object 360 is at the same distance and/or depth as its real object”.)
collecting sensor data while the client device is in the desired position; and (Ma [0030]: “In some situations, both video or static visual data captured by the image sensor and the inertial sensor data measured by the one or more inertial sensors are applied to determine and predict device poses.”)
calculating at least one of a rotation or translation of the offset that converts the physical
pose while the client device is in the desired position to a desired virtual pose.
(Rowell [0141]: “the rotational offset angles between the geometric coordinate system of the synthetic image scene and the image plane of the virtual camera.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the user guidance and sensor data as taught by Ma with the system of Couche as modified by Edwards in order to adjust the device to a desired position to match the display of the rendering to the real world. Also to use the rotational offset angles as taught by Rowell with the system of Couche in order to determine how much rotation is needed to adjust the orientation of the virtual pose independent from the actual orientation to enable better alignment with virtual content.
As per claim 16, this claim is similar in scope to limitations recited in claim 6, and thus is rejected under the same rationale.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Couche in view of Edwards and in further view of Ma in further view of Rowell and in further view of Mandala (US 11577159 B2).
As per claim 7, Couche and Edwards alone do not explicitly teach the claimed limitations.
However, Couche and Edwards in combination with Rowell and Mandala teaches the claimed:
7. The computer-implemented method of claim 6, wherein the desired virtual pose is equivalent to the camera being at eye level of the user pointing along an axis parallel to a ground plane.
(Rowell [0065]: “The camera coordinates 210 may also include one or more rotation position parameters describing the Euler angles of rotation (e.g., a-pitch, n-roll, y-yaw) for each virtual camera module.” Rowell teaches the angles of rotation for the virtual camera module that includes a pitch parameter, and a pitch parameter at an initial position of 0 degrees is parallel to the ground. Thus, it would be obvious to say that having the angles of rotation at an initial position would cause the virtual pose to be parallel to the ground.
Mandala (col 18, line 55-58): “Specifically, as shown in FIG. 5A, user's viewpoint 154 is perfectly aligned with virtual object 120 as shown by optical or alignment axis represented by arrow 164 emanating from viewpoint 154.” Mandala teaches the user’s view that is looking at the virtual object at eye level, also shown in FIGs 5a and 5b.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the virtual pose as taught by Mandala and Rowell with the system of Couche as modified by Lai, Rowell, and Mandala in order to determine an initial position of the device in order to have an anchor point of the virtual environment relative to the user’s position and gaze.
As per claim 17, this claim is similar in scope to limitations recited in claim 7, and thus is rejected under the same rationale.
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Couche in view of Edwards and in further view of Goodrich (US 20220206572 A1) and in further view of Assouline (US 11928783 B2).
As per claim 8, Couche and Edwards alone do not explicitly teach the claimed limitations.
However, Couche and Edwards in combination with Goodrich and Assouline teaches the claimed:
8. The computer-implemented method of claim 1, wherein the offset is a function of an angle of the physical pose relative to a ground plane.
(Goodrich [Abstract]: “The subject technology identifies a surface corresponding to a ground plane in the field of view. The subject technology determines a distance from the identified surface to the anchor point. The subject technology generates AR content based at least in part on the determined distance.”
Goodrich teaches the ground plane detection and the offset that is computed by the distance from the plane to the anchor.
Assouline (col 26, line 61-63): “the image modification module 518 can modify the orientation of the first portion of the AR representation to correspond to the orientation of the second vertical plane while maintaining an orientation of other portions of the AR representation as corresponding to the orientation of the first vertical surface.”
Assouline teaches the modification of the orientation relative to the plane. Thus, substituting the distance offset in Goodrich with the angle or orientation offset in Assouline is a standard geometric substitution in AR and would be obvious to one of ordinary skill in the art.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the identification of a ground plane as taught by Goodrich and Assouline with the system of Couche as modified by Edwards in order to provide a stable spatial reference to allow the system to anchor change relative to the real-world environment. Also to use the modification of an orientation corresponding to a plane as taught by Assouline with the system of Couche in order to adjust the orientation of the virtual environment relative to the real-world in a stable and consistent way.
As per claim 18, this claim is similar in scope to limitations recited in claim 8, and thus is rejected under the same rationale.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Couche in view of Edwards and in further view of Goodrich and in further view of Assouline and in even further view of Doran (US 10140736 B2).
As per claim 9, Couche, Lai, Goodrich, and Assouline alone do not explicitly teach the claimed limitations.
However, Couche, Lai, Goodrich, and Assouline in combination with Doran teaches the claimed:
9. The computer-implemented method of claim 8, wherein the function is a function whose first derivative is continuous.
(Doran (col 6, line 14-17): “In an embodiment each smooth curve is a curve that is continuous and for which the first derivative of the curve (of the path of the curve) is piecewise continuous.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the first derivative continuous function as taught by Doran with the system of Couche as modified by Lai, Goodrich, and Assouline in order to determine the offset based on angles in AR systems to have visually smooth transitions.
As per claim 19, this claim is similar in scope to limitations recited in claim 9, and thus is rejected under the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA SUO whose telephone number is (571) 272-8387. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA SUO/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights