Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 1/20/2026 has been entered.
Response to Amendment
The amendment filed on 1/20/2026 has been entered and made of record. Claims 2-5, 7-9, 11, 13-17 and 19 are amended. Claims 1, 18 and 20 are cancelled. Claim 21 is new. Claims 2-17, 19 and 21 are pending.
Response to Arguments
Applicant’s arguments with respect to the rejections of independent claim 21 have been fully considered but they are not persuasive.
Applicant asserts that the cited reference of Totty fails to disclose or teach a method with ordered steps where "acquiring, after the surface is created, a photographic image of the physical space by the camera" (p. 6-7 of Remarks).
Examiner notices that the above argued limitation is directed to two dataset: one is the surface created from depth points of a physical space captured by depth sensor or LIDAR, another is from the photographic image of the same physical space. Since they are from two different datasets, there is no temporal dependency between these two datasets. Examiner also notices that the provisional patent application No 63/294,609 discloses some steps (i.e. Step 2A-3 Generate a 3D Surface of the Data and step 2B-1 Obtain Initial Photography Data). In this case, it is clearly that applicant uses references step “2A” and “2B” to indicate there is no time relationship between step 2A and step 2B because step 2A is processing data from LIDAR sensor, while step 2B is processing image data from camera. Considering applicant explicitly contends that the cited new limitation requests “acquiring a photographic image after the surface is created, which is not supported by the written description in the specification.
Examiner also notices that the above argued limitation actually describes a fusing operation to register a surface from 3D points (i.e. LIDAR points cloud or 3D model from depth sensor) with a photographic image. Totty discloses “S220 can include: filtering and denoising geometric representations based on known characteristics of the depth sensor… registering depth sensor data to scene imagery (e.g., visual RGB data of each image and/or frame)… using the camera intrinsics and extrinsics to project points of a point cloud back into the RGB camera to register the points with RGB images and/or frames to determine a particular depth map; and/or creating geometric meshes from depth sensor data” in [0082]. Therefore, Totty does teaches acquiring both surface data and photographic image.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 2-17, 19 and 21 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
Independent claim 21 recites the limitation “acquiring, after the surface is created, a photographic image of the physical space by the camera”. Here the surface is created from point cloud from LIDAR sensor, while a photographic image is from a camera. There is no time sequence relationship between these two datasets. Further, claim 21 also recites “optimizing, by the internal positioning system, the photographic image and the additional photographic image to create an optimized image”. Here, the photographic image and the additional photographic image are two individual images. There is no written description in the specification to describe how to optimize two images into one optimized image.
Claims 2-17 and 19 depend on claim 21 and are rejected under the similar rationale.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-17, 19 and 21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Independent claim 21 recites the limitation “optimizing, by the internal positioning system, the photographic image and the additional photographic image to create an optimized image”. Here, there are two photographic images before optimizing step, but there is only one optimized image output. For the purpose of the examination of patent application, the limitation “optimizing, by the internal positioning system, the photographic image and the additional photographic image to create an optimized image” can be interpreted as optimizing the photographic image and the additional photographic image to create optimized images.
Claims 2-17 and 19 depend on claim 21 and are rejected under the similar rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-17, 19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Totty et al. (US 2020/0302686) in view of Chande et al. (US 2022/0408019 A1).
As to Claim 21, Totty teaches A method of conveying a realistic depiction of a physical space in a virtual space for a user (Totty, [0005] and Fig 7), the method comprising steps of:
providing a capturing device including a screen, a camera, a gyrometer, a sensor, and an internal positioning system including a three-dimensional planar coordinate system for identifying a position and an orientation of the capturing device within the physical space (Totty discloses “The capture device can include one or more sensors, such as optical sensors (e.g., cameras), depth sensors (e.g., LIDAR, radar, projected light, stereocameras, etc.), inertial sensors (e.g., IMU, gyroscope), light sensors, color temperature sensors, location sensors (e.g., GPS), and/or other sensors” in [0034]; see also pose of the capture device (3D) in [0049]);
establishing, by the internal positioning system at an initial coordinate, a starting point of the capturing device within the physical space (Totty discloses “Examples of geometric representations that can be generated and/or used include: scene scan and/or position representations from S120-S150 ( e.g., 3D scene points and/ or 3D camera poses from SLAM/ARKit/ARCore, etc.; sparse and medium accuracy); depth sensor geometric representation from S220 ( e.g., sparse or dense, high accuracy)…” in [0055]; “The point clouds can include (x, y, z) coordinates of points of the scene” in [0057]);
determining, by the internal positioning system and the gyrometer, a pose of the capturing device, the pose including a direction of the sensor and a location where the capturing device is positioned; obtaining, at the capturing device and by the sensor, a plurality of depth points and pose data (Totty discloses “The associated capture data can include: poses of the capture device (3D)… capture device inertial data ( e.g. IMU, gyroscopes, accelerometers, positions, velocities, acclerations, etc.), geolocation, 2D feature points, 3D scene points, planes, objects, a set of scene scan geometric representation(s), a set of additional scene scan geometric representation(s), camera poses, and/or any other suitable data” in [0049], see also [0046, 0056]);
storing the pose data for each of the plurality of depth points (Totty, [0036]);
estimating, using localization, a new pose location (Totty discloses simultaneous localization and mapping (SLAM) in [0142]);
generating, by the internal positioning system, a plurality of vectors that connect a coordinate for each of the plurality of depth points, at least one of the plurality of depth points including the initial coordinate, thereby creating a surface (Totty discloses “using depth sensor intrinsics and extrinsics to project depth data into a point cloud; using the camera intrinsics and extrinsics to project points of a point cloud back into the RGB camera to register the points with RGB images and/or frames to determine a particular depth map; and/or creating geometric meshes from depth sensor data” in [0082]; “detecting, matching, and/or triangulating keypoints and line segments S230 (e.g., reading and/or writing 2D and 3D keypoints and/or line segments to the datastore)” in [0154]. Here, keypoints triangulating refers to connect points to mesh. For example, Chande discloses “According to various embodiments, designated the three-dimensional model may include points in a three-dimensional space. The points may be connected by edges that together form surfaces. The designated three-dimensional model may be determined using one or more of a variety of techniques” in [0076]);
acquiring, after the surface is created, a photographic image of the physical space by the camera (Totty discloses “S220 can include: filtering and denoising geometric representations based on known characteristics of the depth sensor… registering depth sensor data to scene imagery (e.g., visual RGB data of each image and/or frame)… using the camera intrinsics and extrinsics to project points of a point cloud back into the RGB camera to register the points with RGB images and/or frames to determine a particular depth map; and/or creating geometric meshes from depth sensor data” in [0082]);
acquiring, by the capturing device, an additional photographic image with the pose data (Totty discloses “Capturing scene data from a second vantage point S150 functions to determine second scene imagery and/or second associated capture data” in [0075]);
optimizing, by the internal positioning system, the photographic image and the additional photographic image to create an optimized image (Totty, [0086, 0108]);
mapping, by the internal positioning system, the optimized image onto the surface to create a textured surface (Totty discloses “stitching the 5 images into a wide angle panorama, using coarse alignment from poses and matches, and fine alignment using 2D and 3D content preserving local warps S310 (e.g., reading images from and writing large FOV images to the datastore); and mapping previously generated data into panorama image formats S320 (e.g., reading data from and writing panoramic depthmaps, panoramic normal maps, and panoramic segment maps to the datastore )” in [0155]; see also [0091-0093]); and
confirming, by the internal positioning system, at least one of the plurality of depth points to verify an accuracy of the textured surface (Totty discloses misalignment in [0039]; low accuracy, middle accuracy and high accuracy in [0063]; “mapping previously generated data into panorama image formats S320 (e.g., reading data from and writing panoramic depthmaps, panoramic normal maps, and panoramic segment maps to the datastore )” in [0155], see also [0082, 0157]), wherein the user is permitted to have a first-person walkthrough experience through the virtual space by using the screen of the capturing device (Totty discloses “The VSVR can be static and/or dynamic (e.g., wherein a user can move between different vantage points, pan within a VSVR, zoom into a VSVR region, etc.)… free viewpoint walk throughs; VR experiences; or other representations of the visual appearance of a space from one or more viewpoints” in [0040]; “Determining one or more geometric representations using structure from motion (SfM) S250 functions to determine: 3D points; updated camera poses, which can include updated positions and orientations of the camera for every image and/or frame (or a subset thereof)” in [0085].)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Totty with the teaching of Chande so as to connect depth points with edges to generate geometric meshes (Chande, [0076]).
As to Claim 2, Totty in view of Chande teaches The method of Claim 21, wherein the pose of the capturing device is determined based on the position and the orientation of the capturing device within the physical space before scanning the physical space(Totty discloses “poses of the capture device (3D)… capture device inertial data (e.g. IMU, gyroscopes, accelerometers, positions, velocities, acclerations, etc.), geolocation…” in [0049]; “S130 can additionally or alternatively include collecting 3D camera poses for each video frame ( or a subset thereof) using the SLAM engine and/or any other suitable element” in [0073]; “updated camera poses, which can include updated positions and orientations of the camera for every image and/or frame” in [0085].)
As to Claim 3, Totty in view of Chande teaches The method of Claim 21, wherein the capturing device is moved through the physical space to acquire a three-dimensional scan of the physical space (Totty discloses “Generating 3D camera poses and 3D scene points using the SLAM engine S120 functions to determine the SLAM data when moving the user device in a pre-determined configuration (e.g., FIG. 8, circle, square, etc.), such as during Sll0. However, camera poses and scene points can be otherwise captured in-situ” in [0072]; see also [0073-0074, 0153].)
As to Claim 4, Totty in view of Chande teaches The method of Claim 21, wherein scanning the physical space includes collecting lidar depth and red, green, and blue depth data (Totty discloses “The capture device can include one or more sensors, such as optical sensors (e.g., cameras), depth sensors (e.g., LID AR, radar, projected light, steereocameras, etc.), inertial sensors…” in [0034]; “the depth sensor geometric representation from S220, the RGB and/or RGBD SLAM geometric representation from S240” in [0088].)
As to Claim 5, Totty in view of Chande teaches The method of Claim 21, wherein the capturing device includes a hand-held device (Totty discloses “Examples of the device 100 include: smartphones, tablets, smart watches, cameras, and/or other devices” in [0034], see also [0072].)
As to Claim 6, Totty in view of Chande teaches The method of Claim 5, wherein the capturing device comprises one of a mobile device, a smartphone, a tablet, a digital camera, an action camera, a wearable computer, a smart watch, and a drone (Totty discloses “Examples of the device 100 include: smartphones, tablets, smart watches, cameras, and/or other devices” in [0034].)
As to Claim 7, Totty in view of Chande teaches The method of Claim 6, wherein scanning the physical space acquiring a three-dimensional scan of the physical space and acquiring a view of a complete panoramic image of the physical space, is performed using an application of the capturing device (Totty discloses “Providing the user with guidelines to capture imagery from a vantage point Sll0 functions to determine the scene data based on pre-determined capture process including a predetermined set of camera motion guidelines and/or a random camera motion pattern, and to ensure that the user captures sufficient data for subsequent virtual model generation.” in [0071], see also [0050].)
As to Claim 8, Totty in view of Chande teaches The method of Claim 7, wherein processing the three-dimensional scan of the physical space to generate a three-dimensional surface reconstruction of the physical space includes generating a three-dimensional point cloud and model, wherein a color image is mapped to a computed geometry to produce a textured model (Totty discloses “Examples of geometric representations that can be generated and/or used include: scene scan and/or position representations from S120-S150 (e.g., 3D scene points and/or 3D camera poses…); depth sensor geometric representation from S220… final geometric representation from S590 (e.g., fused point cloud or depth map” in [0055]; “S220 can include: …registering depth sensor data to scene imagery (e.g., visual RGB data of each image and/or frame); using depth sensor intrinsics and extrinsics to project depth data into a point cloud; using the camera intrinsics and extrinsics to project points of a point cloud back into the RGB camera to register the points with RGB images and/or frames to determine a particular depth map; and/or creating geometric meshes from depth sensor data” in [0082]; “mapping previously generated data into panorama image formats S320 (e.g., reading data from and writing panoramic depthmaps, panoramic normal maps, and panoramic segment maps to the datastore )” in [0155], see also [0157].)
As to Claim 9, Totty in view of Chande teaches The method of Claim 21, wherein panoramic images are acquired using the capturing device (Totty, [0055, 0091, 0153-0155].)
As to Claim 10, Totty in view of Chande teaches The method of Claim 9, wherein the panoramic images include a three-hundred-and-sixty-degree image of the physical space (Totty, [0027, 0040]).
As to Claim 11, Totty in view of Chande teaches The method of Claim 10, wherein the panoramic images are continuously acquired to generate a complete panoramic image of the physical space (Totty, [0071-0073]).
As to Claim 12, Totty in view of Chande teaches The method of Claim 11, wherein the complete panoramic image is generated using a plurality of panoramic images (Totty, [0091, 0155]).
As to Claim 13, Totty in view of Chande teaches The method of Claim 12, wherein the complete panoramic image is mapped onto a three-dimensional surface reconstruction by matching a feature from the panoramic images to or from a three-dimensional surface reconstruction color image and using the pose of the capturing device to project the complete panoramic image onto the three-dimensional surface reconstruction (Totty discloses “S220 can include… registering depth sensor data to scene imagery (e.g., visual RGB data of each image and/or frame); using depth sensor intrinsics and extrinsics to project depth data into a point cloud; using the camera intrinsics and extrinsics to project points of a point cloud back into the RGB camera to register the points with RGB images and/or frames to determine a particular depth map; and/or creating geometric meshes from depth sensor data” in [0082]; “… and mapping previously generated data into panorama image formats S320 (e.g., reading data from and writing panoramic depthmaps, panoramic normal maps, and panoramic segment maps to the datastore )” in [0155]; see also [0091, 0093] and Fig 8.)
As to Claim 14, Totty in view of Chande teaches The method of Claim 13, wherein visual feedback at the capturing device guides the user to acquire the panoramic images of the physical space (Totty discloses “Providing the user with guidelines to capture imagery from a vantage point S110 functions to determine the scene data based on pre-determined capture process including a predetermined set of camera motion guidelines and/or a random camera motion pattern, and to ensure that the user captures sufficient data for subsequent virtual model generation… S110 can include guiding the user through the capture process (e.g., guiding the user though each guideline, such as to ensure the still images of the scene are overlapping, to ensure the video footage is captured from a particular vantage point and/or angle, to ensure high quality associated capture data, etc.)” in [0071], see also [0073].)
As to Claim 15, Totty in view of Chande teaches The method of Claim 14, wherein mapping the complete panoramic image onto the three-dimensional surface reconstruction is at the capturing device (Totty, [0064-0065]).
As to Claim 16, Totty in view of Chande teaches The method of Claim 8, wherein the three-dimensional scan is uploaded to a server to process the three-dimensional scan of the physical space to generate the three-dimensional surface reconstruction of the physical space (Totty discloses “All or portions of the method can be performed: locally on the user device, by a remote computing system (e.g., server system)” in [0065]; “capturing scene video by scanning a scene S130, capturing scene images S140, capturing scene data from a second vantage point S150, uploading the scene imagery and associated capture data to the platform S190, and/or any other suitable process” in [0070];see also [0153-0158].)
As to Claim 17, Totty in view of Chande teaches The method of Claim 16, further comprising labeling an item of the three-dimensional scan (Totty discloses “For example, S420 can determine a class label (e.g., "floor", "wall", "couch", etc.) for every pixel… the geometric representations (e.g., associated based on the labels determined for source pixels from source images underlying the geometric representation voxel)” in [0098]; “Refining segmentation masks S530, which can function to determine refined segmentation labels per pixel (or a subset thereof) in one or more models” in [0118].)
Claim 19 is rejected based upon similar rationale as Claim 5.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIMING HE whose telephone number is (571)270-1221. The examiner can normally be reached on Monday-Friday, 8:30am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEIMING HE/
Primary Examiner, Art Unit 2611