DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 9-10, 15-16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (US 20260039948 A1).
Regarding Claim 1, Li teaches A computer-implemented method of generating reconstructed three-dimensional representations (“The image capture plan may include obtaining a motion plan that, when performed by the user, enables the display free body wearable computing device to capture a set of images that meet criteria for generating a three-dimensional model with a desired level of quality.” Li ¶ 16.), the computer-implemented method (Li Fig. 5; Li ¶¶ 159-160, 162-166) comprising:
receiving, by a computing system comprising one or more processors, image data comprising at least one image (Fig. 3A 300 “identify a scope of interest in a scene” based on at least one image, because “ Identifying the scope of the interest may include: (i) obtaining, using at least one camera of the display free body wearable computing device, a first image depicting: (a) a first portion of the scene, . . ..” Li ¶ 20.) associated with a path (Fig. 3A 302 “image capture plan”; Fig. 4B Motion Plan 410) through a physical space (Fig. 4B Scene 400) (
PNG
media_image1.png
616
492
media_image1.png
Greyscale
);
determining, by the computing system, based on the image data (“Identifying the scope of the interest may include: (i) obtaining, using at least one camera of the display free body wearable computing device, a first image depicting: (a) a first portion of the scene, . . ..” Li ¶ 20.), a plurality of scan nodes associated with the path (selected locations along Fig. 3B 314 “motion path”), wherein the plurality of scan nodes comprise locations at which to capture a plurality of two-dimensional scanned images of the physical space (
PNG
media_image2.png
512
738
media_image2.png
Greyscale
“Based on the number of images, the locations, and the fields of view. display free body wearable computing device 50 may obtain motion plan 410 (shown in long-dashed lines). Motion plan 410 may include a path that transverses each location of the identified locations. When performed by the user, motion plan 410 may enable display free body wearable computing device 50 to capture a set of images that allow for the three-dimensional model to be generated with the desired level of quality.” Li ¶ 133.
“At operation 314, a motion plan may be obtained based on the number of images, the location, and the fields of view. The motion plan may be obtained by: (i) aggregating the locations and fields of view for obtaining each respective image of a set of images, (ii) projecting a path that may include each location of the locations, and/or any other processes. Refer to FIG. 4A for additional details.” Li ¶ 104.);
generating, by the computing system, based on the plurality of scan nodes, a plurality of instructions (Li Fig. 4E 412D “Focus on object 404”; or instructions that director a camera to captures images at selected locations) associated with capturing the plurality of two-dimensional scanned images of the physical space (
PNG
media_image3.png
500
744
media_image3.png
Greyscale
“Based on the number of images, the locations, and the fields of view. display free body wearable computing device 50 may obtain motion plan 410 (shown in long-dashed lines). Motion plan 410 may include a path that transverses each location of the identified locations. When performed by the user, motion plan 410 may enable display free body wearable computing device 50 to capture a set of images that allow for the three-dimensional model to be generated with the desired level of quality.” Li ¶ 133.);
generating, by the computing system, based on the plurality of instructions (Li Fig. 4E 412D “Focus on object 404”; or instructions that director a camera to captures images at selected locations), the plurality of two-dimensional scanned images associated with the plurality of scan nodes (selected locations to capture images) (Li ¶ 133); and
generating, by the computing system, based on the image data and the plurality of two-dimensional scanned images, a reconstructed three-dimensional representation of the physical space (
“When performed by the user, motion plan 410 may enable display free body wearable computing device 50 to capture a set of images that allow for the three-dimensional model to be generated with the desired level of quality.” Li ¶ 133.
Li provides an example, stating “Consider a scenario in which a user of display free body wearable computing device 50 desires to generate a three-dimensional (3D) interactive model of a room that the user is present. Once a request for the 3D interactive model is identified, display free body wearable computing device 50 may: (i) provide instruction to the user (e.g., to move around the room), (ii) capture images using the camera at a certain frequency (e.g., while the user is moving around the room), and/or perform any other actions.” Li ¶ 90.).
Li’s abovementioned embodiment does not explicitly disclose that the image data comprising a plurality of two-dimensional images to be captured by Li’s wearable computing device.
Li’s another embodiment teaches the image data comprising a plurality of two-dimensional images to be captured by Li’s wearable computing device (“The pair of cameras may comprise lenses configured to: (i) establish a camera line of sight that is parallel to a line of sight of the user; and (ii) establish a camera field of view that comprises the field of view of the user. ” Li ¶ 34. “The stereo image may include a pair of images of the scene, each of the images being captured at different angles and/or positions with respect to the scene by the pair of cameras.” Li ¶ 35.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Li’s stereoscopic image data with Li’s motion plan based on image data. One of ordinary skill in the art would be motivated to provide more information to identify scope of interest, because stereoscopic data may provide depth information. In addition, stereoscopic information may also assist gesture identification to determine scope of interest.
Regarding Claim 2, Li further teaches The computer-implemented method of claim 1, further comprising: generating, by the computing system, the image data (images taken to identify the scope of interest) based on one or more directions to traverse the path through the physical space (
PNG
media_image4.png
496
740
media_image4.png
Greyscale
The images taken to determine the scope of interest have to be facing the direction towards the object 404 (scope of interest), which also correspond to the direction of the motion plan to traverse the motion plan through the physical scene.).
Regarding Claim 3, Li further teaches The computer-implemented method of claim 1, further comprising: determining, by the computing system, based on the image data, the path through the physical space (Fig. 3A 300 “identify a scope of interest in a scene” based on at least one image, because “ Identifying the scope of the interest may include: (i) obtaining, using at least one camera of the display free body wearable computing device, a first image depicting: (a) a first portion of the scene, . . ..” Li ¶ 20. Fig. 3A 302 “image capture plan” is further determined based on the scope of the interest. Fig. 4C provides an example for the “image capture plan”: “Motion Plan 410.”).
Regarding Claim 9, Li further teaches The computer-implemented method of claim 1, wherein the generating, by the computing system, based on the plurality of instructions, the plurality of two-dimensional scanned images associated with the plurality of scan nodes (repeating limitations already recited in Claim 1) comprises:
determining, by the computing system, that a velocity of an image capture device that captures the plurality of two-dimensional scanned images does not exceed a scanned image capture velocity threshold (Fig. 4D “Walk slower to reduce blur.” “For example, user movement 422B by the user may be faster than a movement required to capture an image of a desired level of quality. The image captured while user movement 422B is performed may subsequently have a level of quality that does not meet the quality standard (e.g., is too blurry).” Li ¶ 144.);
determining, by the computing system, that an image capture rate of the image capture device that captures the plurality of two-dimensional scanned images does not exceed a scanned image capture rate threshold; or
determining, by the computing system, one or more directions in which to position the image capture device to capture the plurality of two-dimensional scanned images (Fig. 4E “Focus on object 404,” so that the camera would be facing the object to be captured. Figs. 4A-E.).
Regarding Claim 10, Li further teaches The computer-implemented method of claim 1, wherein the generating, by the computing system, based on the plurality of instructions, the plurality of two-dimensional scanned images associated with the plurality of scan nodes (repeating limitations already recited in Claim 1) comprises:
determining, by the computing system, a portion of a predetermined field of view (Fig. 4B: Camera Field of View 130 of selected location along motion plan) of the physical space from each of the plurality of scan nodes (selected locations along Fig. 4B motion plan 410) that has been captured (
PNG
media_image2.png
512
738
media_image2.png
Greyscale
); and
generating, by the computing system, one or more indications (status communication) of the portion (part of all the scope of interest) of the predetermined field of view (from the users) of the physical space from each of the plurality of scan nodes that has been captured (
“Consider a scenario in which a user of display free body wearable computing device 50 desires to generate a three-dimensional (3D) interactive model of a room that the user is present. Once a request for the 3D interactive model is identified, display free body wearable computing device 50 may: (i) provide instruction to the user (e.g., to move around the room), (ii) capture images using the camera at a certain frequency (e.g., while the user is moving around the room), and/or perform any other actions. Display free body wearable computing device 50 may provide the captured images along with metadata regarding each of the captured images to a second service platform (e.g., 204B) of service platforms 204. Using image data provided by display free body wearable computing device 50, service platform 204B may perform, for example, 3D rendering service, video editing service, video storage services, and/or any other services to generate the 3D interactive model desired by the user. Display free body wearable computing device 50 may subsequently communicate a status (e.g., completion, instructions for access, etc.) of the desired 3D interactive model to the user.” Li ¶ 90.
“The audio output may include, for example, spatial audio cues (e.g., sounds, beeps, etc.) to indicate a motion plan, instructions (e.g., verbal instructions) to direct movement of a portion of the user while capturing an image, information regarding a status of the images captured, and/or any other information. ” Li ¶ 53.).
Regarding Claim 15, Li further teaches The computer-implemented method of claim 1, wherein the plurality of two-dimensional scanned images are captured by one or more image capture devices comprising one or more cameras (,
PNG
media_image5.png
346
442
media_image5.png
Greyscale
where there are two cameras), a smartphone, or an augmented reality headset.
Claims 16 and 18 are substantially similar to Claim 1. The rejection analyses based on Li for Claim 1 are also applied to Claims 16 and 18. In addition:
Claim 16 recites, “One or more tangible non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising: . . .” (Li Fig. 5; Li ¶¶ 159-160, 162-166).
Claim 18 recites, “A computing system comprising: one or more processors; one or more non-transitory computer-readable media storing instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising: . . .” (Li Fig. 5; Li ¶¶ 159-160, 162-166).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Li as applied to Claim 1, in further view of Sones et al. (US 20110188741 A1).
Regarding Claim 4, Li teaches The computer-implemented method of claim 1, wherein the determining, by the computing system, a plurality of scan nodes associated with the path, wherein the plurality of scan nodes comprise locations at which to capture a plurality of two-dimensional scanned images of the physical space (repeating the limitations from Claim 1, which analyses have already been provided).
Li does not explicitly disclose; however, Sones teaches comprises determining, by the computing system, based on the image data, estimated dimensions of the physical space (“FIG. 1 illustrates an embodiment of an image acquisition and processing configuration for performing dimension estimation using stereoscopic imaging techniques and showing an example of an imaging geometry.” Sones ¶ 30.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Sones’s with Li’s motion plan based on image data. One of ordinary skill in the art would be motivated to more accurately and/reasonably determine the motion plan. Li’s Figs. 4A-E teach a motion path that surrounds an object/area of interest. The dimension information about the object has an impact, for example, of the size the motion path.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Sones as applied to Claim 4, in further view of Castillo et al. (US 20210243362 A1).
Regarding Claim 5, Li in view of Sones teaches The computer-implemented method of claim 4.
Li in view of Sones does not explicitly disclose; however, Castillo teaches wherein the estimated dimensions of the physical space are based on inputting the image data into one or more machine-learned models that are configured to determine three-dimensional features based on detection of two-dimensional features of the two-dimensional images (
“For example, a feature match (otherwise referred to as a feature correspondence) of the one or more feature matches can indicate that the first 3D position associated with the first pixel of the first 2D image matches the second 3D position associated with the second pixel of the second 2D image. The computer-implemented method can include determining a 3D reconstruction condition based on the one or more feature matches between the first 2D image and the second 2D image. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.” Castillo ¶ 10.
“The images stored in 2D image data store 220 and/or the 3D models stored in 3D model data store 210 may serve as inputs to machine-learning or artificial-intelligence models. The images and/or the 3D models may be used as training data to train the machine-learning or artificial-intelligence models or as test data to generate predictive outputs. Machine-learning or artificial-intelligence models may include supervised, unsupervised, or semi-supervised machine-learning models.” Castillo ¶ 77. Castillo ¶ 11 provides a detailed example.
“Disclosed are techniques for enhancing two-dimensional (2D) image capture of subjects (e.g., a physical structure, such as a residential building) to maximize the feature correspondences available for three-dimensional (3D) model reconstruction.” Castillo Abstract.
“Server 120 can analyze the complete set of 2D images to automatically detect or compute the 3D dimensions of house 150 by evaluating the feature correspondences detected between images of the set of images.” Castillo ¶ 73.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Castillo’s machine learning and feature matching with Li in view of Sones. One of ordinary skill in the art would be motivated to improve image processing speed and/or accuracy.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Li as applied to Claim 1, in further view of HARVIAINEN (WO 2020139766 A2).
Regarding Claim 6, Li teaches The computer-implemented method of claim 1, wherein the determining, by the computing system, a plurality of scan nodes associated with the path, wherein the plurality of scan nodes comprise locations at which to capture a plurality of two-dimensional scanned images of the physical space (repeating limitations from Claim 1, which has already been analyzed).
Li does not explicitly disclose; however, HARVIAINEN teaches
comprises determining, by the computing system, based on the image data, the plurality of scan nodes comprising locations from which a field of view to capture the plurality of two-dimensional scanned images is not occluded by one or more objects (
“In some embodiments, an object is determined to be visible if the bounding volume of the object is not entirely obscured from the viewpoint by the occlusion volumes of the other objects. In some embodiments, an object is determined to be visible if no more than a threshold amount of the bounding volume of the object is obscured from the viewpoint by the occlusion volumes of the other objects. In some embodiments, an object is determined to be visible if there is at least one viewpoint in a plurality of selected viewpoints at which the bounding volume of the object is not obscured by the occlusion volumes of the other objects. The selected viewpoints may be viewpoints within a defined navigation volume.” HARVIAINEN ¶ 162. “Not transmitting visual data for objects not visible may be one method to improve content delivery efficiency.” HARVIAINEN ¶ 63.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine HARVIAINEN’s determining object occlusion with Li. One of ordinary skill in the art would be motivated to reduce data processing/transmission. “Not transmitting visual data for objects not visible may be one method to improve content delivery efficiency.” HARVIAINEN ¶ 63.
Regarding Claim 7, Li in view of HARVIAINEN teaches The computer-implemented method of claim 1, wherein the determining, by the computing system, a plurality of scan nodes associated with the path, wherein the plurality of scan nodes comprise locations at which to capture a plurality of two-dimensional scanned images of the physical space (repeating limitations from Claim 1, which has already been analyzed) comprises:
determining, by the computing system, based on the image data, the plurality of scan nodes comprising locations from which capture of the plurality of two-dimensional scanned images is not obstructed by one or more objects (
“In some embodiments, an object is determined to be visible if the bounding volume of the object is not entirely obscured from the viewpoint by the occlusion volumes of the other objects. In some embodiments, an object is determined to be visible if no more than a threshold amount of the bounding volume of the object is obscured from the viewpoint by the occlusion volumes of the other objects. In some embodiments, an object is determined to be visible if there is at least one viewpoint in a plurality of selected viewpoints at which the bounding volume of the object is not obscured by the occlusion volumes of the other objects. The selected viewpoints may be viewpoints within a defined navigation volume.” HARVIAINEN ¶ 162. “Not transmitting visual data for objects not visible may be one method to improve content delivery efficiency.” HARVIAINEN ¶ 63.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine HARVIAINEN’s determining object occlusion with Li. One of ordinary skill in the art would be motivated to reduce data processing/transmission. “Not transmitting visual data for objects not visible may be one method to improve content delivery efficiency.” HARVIAINEN ¶ 63.
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Li as applied to Claim 1 or 18, in further view of Tezaur et al. (US 20210012454 A1).
Regarding Claim 8, Li teaches The computer-implemented method of claim 1.
Li does not explicitly disclose; however, Tezaur teaches wherein the plurality of instructions comprise an instruction to capture the plurality of two-dimensional scanned images comprising a substantially omnidirectional field of view from each of the plurality of scan nodes (
“Referring to FIGS. 1-2, omnidirectional cameras including wide-angle or ultra-wide angle (or fisheye) lens cameras form curved or circular images 100 of a scene captured by the camera. The image data of these circular or 360 degree images are often projected, such as with equirectangular projections (ERPs), to form a more practical 2D representation or image 200 that is capture device independent, easier to understand, and often used for further image processing or analysis.” Tezaur ¶ 23.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Tezaur’s omnidirectional field of view with Li. One of ordinary skill in the art would be motivated to acquire more space information for 3D reconstruction.
Claim 19 is substantially similar to Claim 8. The rejection analyses based on Li in view of Tezaur for Claim 8 are also applied to Claim 19.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Li as applied to Claim 1, in further view of Kim et al. (US 20220343613 A1).
Regarding Claim 11, Li teaches The computer-implemented method of claim 1.
Li does not explicitly disclose; However, Kim teaches further comprising: generating, by the computing system, an augmented reality environment based on the reconstructed three-dimensional representation ( “The apparatus may include: an environment reconstruction thread unit performing 3D reconstruction of a real environment in for the 3D augmented reality; a moving object selection unit being input from the user with a moving object which is a real object to be moved from a 3D reconstruction image; . . ..” Kim ¶ 14. ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kim’s environment reconstruction with Li. One of ordinary skill in the art would be motivated to place virtual object(s) accurately in an augmented reality environment and/or to allow a user the interact with the physical environment through augmented reality.
Claims 12-14, 17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li as applied to Claim 1, 16, or 18, in further view of Zhang et al. (“RaDe-GS: Rasterizing Depth in Gaussian Splatting”).
Regarding Claim 12, Li teaches The computer-implemented method of claim 1.
Li does not explicitly disclose; However, Zhang teaches wherein the reconstructed three-dimensional representation is based on performance of one or more Gaussian splatting techniques on the plurality of two-dimensional images or the plurality of two-dimensional scanned images (Zhang Fig. 1 “We present a rasterized method to compute the depth and surface normal maps of general Gaussian splats. Our method achieves high-quality 3D shape reconstruction and maintains excellent training and rendering efficiency.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s Gaussian splatting techniques with Li. One of ordinary skill in the art would be motivated to enhance reconstruction efficiency. “Gaussian Splatting (GS) has proven to be highly effective in novel view synthesis, achieving high-quality and real-time rendering. However, its potential for reconstructing detailed 3D shapes has not been fully explored.” Zhang Abstract.
Regarding Claim 13, Li in view of Zhang teaches The computer-implemented method of claim 1, wherein the reconstructed three-dimensional representation is based on inputting the image data (which is used to determine the scope of interest for the 3D reconstruction, and which also provides an image from a viewpoint for 3D reconstruction) and the plurality of two-dimensional scanned images into one or more machine-learned models configured to generate the reconstructed three-dimensional representation (“3D reconstruction from multi-view images is a classic problem with numerous applications in computer vision and graphics.” Zhang 1 Introduction. “Another important direction in NeRF research aims to improve the accuracy of reconstructed 3D shapes. Notable works[Li et al. 2023; Wang et al. 2022, 2021; Yariv et al. 2021; Yu et al. 2022] have in tegrated a Signed Distance Function (SDF) with the radiance field to create high-fidelity 3D models.” Zhang 2.2.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s Gaussian splatting techniques with Li. One of ordinary skill in the art would be motivated to enhance reconstruction efficiency/convenience/speed by utilizing machine learning.
Regarding Claim 14, Li in view of Zhang teaches The computer-implemented method of claim 13, wherein the one or more machine-learned models comprise a neural radiance field (NeRF) model (“3D reconstruction from multi-view images is a classic problem with numerous applications in computer vision and graphics.” Zhang 1 Introduction. “Another important direction in NeRF research aims to improve the accuracy of reconstructed 3D shapes. Notable works[Li et al. 2023; Wang et al. 2022, 2021; Yariv et al. 2021; Yu et al. 2022] have in tegrated a Signed Distance Function (SDF) with the radiance field to create high-fidelity 3D models.” Zhang 2.2.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s Gaussian splatting techniques with Li. One of ordinary skill in the art would be motivated to enhance reconstruction efficiency/convenience/speed by utilizing machine learning.
Claims 17 and 20 are substantially similar to Claim 12. The rejection analyses based on Li in view of Zhang for Claim 12 are also applied to Claims 17 and 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Schindler et al. "Real-Time Camera Guidance for 3d Scene Reconstruction." ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 1 (2012): 69-74. (Year: 2012). Schindler discloses the general concept of the claimed invention as indicated by the title and figs. 1-2.
PNG
media_image6.png
356
316
media_image6.png
Greyscale
PNG
media_image7.png
282
312
media_image7.png
Greyscale
However, Schindler does not appear to explicitly disclose determining, by the computing system, based on the image data, a plurality of scan nodes associated with the path, wherein the plurality of scan nodes comprise locations at which to capture a plurality of two-dimensional scanned images of the physical space.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611
Prosecution Insights