Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-9 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-9 of U.S. Patent No. 12159358.
U.S. App. #18932606
1
2
3
4
5
6
7
8
9
U.S. Patent #12159358
1
2
3
4
5
6
7
8
9
U.S. App. #18932606 Claim 1
U.S. Patent #12159358 Claim 1
1. A computer-implemented method comprising:
receiving image data of a scene captured by a camera;
inputting the image data of the scene into a scene reconstruction model;
receiving, from the scene reconstruction model, a final spatial model of the scene, wherein the scene reconstruction model generates the final spatial model by:
predicting a depth map for each image of the image data,
extracting a feature map for each image of the image data,
generating a first spatial model based on the predicted depth maps of the images,
generating a second spatial model based on the extracted feature maps of the images, and
determining the final spatial model by combining the first spatial model and the second spatial model; and
providing functionality on a computing device related to the scene and based on the final spatial model.
1. A computer-implemented method comprising:
receiving real-time image data of a scene captured by a camera assembly of a mobile device;
inputting the real-time image data of the scene into a scene reconstruction model;
receiving, from the scene reconstruction model, a final heightfield of the scene comprising a height value at each 2D position of the scene, wherein the scene reconstruction model generates the final heightfield by:
for each image of the image data, predicting a depth map based on the image,
for each image of the image data, extracting a feature map based on the image,
generating a raw heightfield based on the predicted depth maps of the images,
generating an aggregate feature map based on the feature maps of the images,
regressing a refined heightfield based on the aggregate feature map, and
determining the final heightfield as a combination of the raw heightfield and the refined heightfield; and
providing functionality on the mobile device related to the scene and based on the final heightfield.
U.S. App. #18932606 Claim 2
U.S. Patent #12159358 Claim 2
2. The computer-implemented method of claim 1, wherein the image data comprises a plurality of images and a camera pose for each image.
2. The computer-implemented method of claim 1,
wherein the real-time image data comprises a plurality of images and a camera pose for each image.
U.S. App. #18932606 Claim 3
U.S. Patent #12159358 Claim 3
3. The computer-implemented method of claim 2, wherein the camera pose for each image is captured by a position sensor coupled to the camera.
3. The computer-implemented method of claim 2,
wherein the camera pose for each image is captured by a position sensor of the mobile device.
U.S. App. #18932606 Claim 4
U.S. Patent #12159358 Claim 4
4. The computer-implemented method of claim 2, wherein the camera pose for each image is estimated by a pose estimation model based on the images.
4. The computer-implemented method of claim 2, wherein the camera pose for each image is estimated by a pose estimation model based on the images.
U.S. App. #18932606 Claim 5
U.S. Patent #12159358 Claim 5
5. The computer-implemented method of claim 1, wherein predicting the depth map comprises applying a depth estimation model to the image to determine the depth map.
5. The computer-implemented method of claim 1,
wherein predicting the depth map comprises applying a depth estimation model to the image to determine the depth map.
U.S. App. #18932606 Claim 6
U.S. Patent #12159358 Claim 6
6. The computer-implemented method of claim 1, wherein extracting the feature map comprises applying a convolutional network to the image to determine the feature map.
6. The computer-implemented method of claim 1,
wherein extracting the feature map comprises applying a convolutional network to the image to determine the feature map.
U.S. App. #18932606 Claim 7
U.S. Patent #12159358 Claim 7
7. The computer-implemented method of claim 1,
wherein the feature map comprises a first tensor for a first feature type and a second tensor for a second feature type.
7. The computer-implemented method of claim 1, wherein the feature map comprises a first tensor for a first feature type and a second tensor for a second feature type.
U.S. App. #18932606 Claim 8
U.S. Patent #12159358 Claim 8
8. The computer-implemented method of claim 1,
wherein generating the raw heightfield comprises:
generating a 3D model using truncated signed distance field with the predicted depth maps; and
ray casting the 3D model to generate the raw heightfield.
8. The computer-implemented method of claim 1, wherein the first spatial model is a heightfield, and wherein generating the first spatial model comprises:
generating a 3D model using truncated signed distance field with the predicted depth maps; and
ray casting the 3D model to generate the heightfield.
U.S. App. #18932606 Claim 9
U.S. Patent #12159358 Claim 9
9. The computer-implemented method of claim 8, wherein ray casting the 3D model to generate the heightfield comprises, for each position of the heightfield, casting a ray downward to a surface of the 3D model to determine a height of the surface at that position.
9. The computer-implemented method of claim 8,
wherein ray casting the 3D model to generate the raw heightfield comprises, for each position of the raw heightfield, casting a ray downward to a surface of the 3D model to determine a height of the surface at that position.
Although the claims at issue are not identical, they are not patentably distinct from each other. For example, Claim 1 of Patent 12159358 discloses "receiving, from the scene reconstruction model, a final heightfield of the scene" while Application 18932606 Claim 1 discloses "receiving, from the scene reconstruction model, a final spatial model of the scene". A heightfield is a raster image. Each pixel stores surface elevation data for display in 3D computer graphics. Spatial model describes the spatial arrangement of object, features, or properties in a given space. Spatial model can include raster models such as heightfields. Therefore, Patent 12159358 Claim 1 discloses all limitations of Application 18932606 Claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6, 10 are rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al (US20130060540) in view of Borer et al (US20210233300) further in view of Al-Faris et al. ("Deep learning of fuzzy weighted multi-resolution depth motion maps with spatial feature fusion for action recognition." Journal of Imaging 5.10 (2019): 82).
Regarding Claim 1. Frahm teaches A computer-implemented method comprising:
receiving image data of a scene captured by a camera (Frahm, abstract, the invention describes methods of generating a three-dimensional representation of an object in a reference plane from a depth map including distances from a reference point to pixels in an image of the object taken from a reference point. Weights are assigned to respective voxels in a three-dimensional grid along rays extending from the reference point through the pixels in the image based on the distances in the depth map from the reference point to the respective pixels, and a height map including an array of height values in the reference plane is formed based on the assigned weights. An n-layer height map may be constructed by generating a probabilistic occupancy grid for the voxels and fanning an n-dimensional height map comprising an array of layer height values in the reference plane based on the probabilistic occupancy grid.
[0086] Height map and texture generation according to some embodiments can be implemented using a parallel processing graphics processing unit (GPU). This may speed up computation by approximately 150 times on an Nvidia GTX 280 graphics card. Additional computation gains are anticipated with future generation video processors due to the scalability of some embodiments. The speed of these systems/methods may enable real-time generation of city models from ground-based range data.
[0106] FIGS. 2A and 2B are schematic diagrams that illustrate systems for generating vertical height map models and/or three-dimensional models according to some embodiments.
[0107] Referring to FIGS. 2A and 3, the system includes an image capture system 20 that includes a processor 22, a camera 24, a depth map generator 26 and a position estimator 28.
[0111] The image capture system 20 may be mobile and/or transportable, so that it may be used to capture images of an object 10 or other scene from multiple reference points. The camera may include a digital camera configured to capture and record high resolution digital images, which are well known in the art.);
inputting the image data of the scene into a scene reconstruction model (Frahm, [0107] Referring to FIGS. 2A and 3, the system includes an image capture system 20 that includes a processor 22, a camera 24, a depth map generator 26 and a position estimator 28. The output of the image capture system 20, which includes at least one depth map and an indication of the location of the reference point corresponding to the depth map, is stored in a database 30, which may be local to or remote from the image capture system 20.
[0108] The output of the image capture system 20 stored in the database 30 may be accessed by a three dimensional modeling system 50, which generates a height map model in response to the depth map information stored in the database 30.
[0110] The height map model and/or the textured or untextured three-dimensional mesh representation can be output to a display server 52 and rendered as a three dimensional rendering 56 on a monitor screen 54 connected to the display server 52.);
receiving, from the scene reconstruction model, a final spatial model of the scene (Frahm, [0098] Similar to volumetric reconstruction approaches, in systems/methods according to some embodiments, a 2D horizontal grid is defined over the region of interest. To avoid the excessive memory usage of the volumetric approaches, for every 2D grid cell, a height value is computed that reduces or minimizes the amount of free space below and the amount of filled space above the value.
[0108] The output of the image capture system 20 stored in the database 30 may be accessed by a three-dimensional modeling system 50, which generates a height map model in response to the depth map information stored in the database 30. The three-dimensional modeling system 50 may also generate a textured or untextured, three dimensional mesh representation from the height map model.)
wherein the scene reconstruction model generates the final spatial model by:
predicting a depth map for each image of the image data (Frahm, [0105] Systems/methods according to some embodiments are directed to the dense part of three dimensional urban reconstruction, and may employ the redundancy of the estimated depths of the scene typically delivered through depth estimation for every video frame, as for example delivered by a system along the lines of the system disclosed in M. Pollefeys and et al., Detailed Real-Time Urban 3d Reconstruction From Video, Intl. Journal of Computer Vision (2008), which is incorporated herein by reference. The inputs to these systems/methods are one or more video sequences along with the estimated camera poses and the intrinsic calibration for every frame, a depth map for every frame, and an estimate of the world's vertical or gravity direction.),
Frahm fails to explicitly teach, however, Borer teaches extracting a feature map for each image of the image data (Borer, abstract, the invention describes methods for generating image data of a three-dimensional (3D) animatable asset. A rendering module executing on a computer system accesses a machine learning model that has been trained via first image data of the 3D animatable asset generated from first rig vector data. The rendering module receives second rig vector data. The rendering module generates, via the machine learning model, a second image data of the 3D animatable asset based on the second rig vector data.
[0058] FIG. 7 illustrates a more detailed view of the machine learning model 320 of FIG. 6, according to various embodiments. As shown, the machine learning model 320 includes a progressive learning stage 1 (PL1) 710 that generates 16x16 feature maps. The machine learning model 320 further includes a progressive learning stage 2 (PL2) 720 that generates 32x32 feature maps. The machine learning model 320 further includes additional progressive learning stages 3-6 (PL3-PL6) (not shown) that generate feature maps at increasingly higher spatial resolutions. The machine learning model 320 further includes a progressive learning stage 7 (PL7) 710 that generates feature maps at the final 1024x1024 spatial resolution.),
Frahm and Borer are analogous art because they both teach method of generating 3D scene model from input images using machine learning. Borer further teaches method of extracting feature map from input images. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm), to further use the feature map extracting method (taught in Borer), so as to capture the 3D object motion in real-time (Borer, [0006-0007]).
The combination of Frahm and Borer further teaches generating a first spatial model based on the predicted depth maps of the images (Frahm, [0108] The output of the image capture system 20 stored in the database 30 may be accessed by a three dimensional modeling system 50, which generates a height map model in response to the depth map information stored in the database 30. The three dimensional modeling system 50 may also generate a textured or untextured three dimensional mesh representation from the height map model.),
The combination of Frahm and Borer fails to explicitly teach, however, Al-Faris teaches generating a second spatial model based on the extracted feature maps of the images (Al-Faris, abstract, the paper describes a novel method for creating spatial temporal human action recognition (HAR) model. First, fuzzy weight functions are used in computations of depth motion maps (DMMs). Multiple length motion information is also used. These features are referred to as fuzzy weighted multi-resolution DMMs (FWMDMMs). This formulation allows for various aspects of individual actions to be emphasized. It also helps to characterise the importance of the temporal dimension. This is important to help overcome, e.g., variations in time over which a single type of action might be performed. A deep convolutional neural network (CNN) motion model is created and trained to extract discriminative and compact features. Transfer learning is also used to extract spatial information from RGB and depth data using the AlexNet network. Different late fusion techniques are then investigated to fuse the deep motion model with the spatial network. The result is a spatial temporal HAR model. The developed approach is capable of recognizing both human action and human–object interaction. Three public domain datasets are used to evaluate the proposed solution. The experimental results demonstrate the robustness of this approach compared with state-of-the art algorithms.
Page 3, par 1, the paper proposes a novel framework for learning of action models by combining together a deep and handcrafted hybrid feature model to get discriminative information. The key contribution of this paper is as follows:
In order to learn information in the temporal dimension with diverse applicability, we develop a novel spatial temporal deep learning model. It includes a new method that improves traditional depth motion maps (DMMs) [17,18] called fuzzy weighted multi-resolution depth motion maps (FWMDMMs). The FWMDMM includes a number of different temporal model instances. These are used to help overcome the inherent variability in time associated for each individual action. Furthermore it can help overcome difficulties with self-occlusions and actions that might have similar types of movements.
Page 8, par 4-5, CNNs can automatically achieve feature extraction directly from input data. This can help solve the exhaustive search of hand-crafted methods. Furthermore, CNN operations include the local receptive field, shared weights and pooling. These processes are adopted to attain shift, scale and distortion invariance that can improve recognition accuracy. As shown in Fig 1, a pre-trained AlexNet model proposed in [47] containing eight pre-trained layers is used here for initialization. Two parallel AlexNet networks are used here to compute features of the RGB and depth information. In addition, extra CNNs are used for the motion model to fuse and learn the information of the two parallel networks and DMM hand-crafted features.), and
Frahm, Borer and Al-Faris are analogous art because they all teach method of generating 3D scene model from input images using machine learning. Borer and AI-Faris both teach method of extracting feature map from input images. Al-Faris further teaches generating spatial model from feature maps. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm and Borer), to further use the feature map extracting method to generate spatial model (taught in Al-Faris), so as to provide robust human action recognition (HAR) model to recognize both human action and human-object interaction (Al-Faris, abstract).
The combination of Frahm, Borer and Al-Faris further teaches determining the final spatial model by combining the first spatial model and the second spatial model (Al-Faris, page 9, par 5, In our work, we propose multiple different fusion techniques at different positions of the spatial two-stream networks. Information fusion is implemented partially between the RGB and depth stream networks combining multiple layers of two trained parallel networks including several CNN architectures such as early, middle and late fusion. This helps to find the best position and technique for RGB and depth fusion that can optimize the recognition rate. In addition, hand-crafted features are exploited in the deep motion model using improved DMMs as an auxiliary source of features that represent motion information of an action. Fusion position via motion model is specified based on the best recognition result of the fused spatial two-stream networks. Then, fused spatial information is utilized with the motion information in the deep motion model using the same fusion position. By adding hand-crafted features, our approach can incorporate two explicitly different types of features such as spatial and temporal information into the classification process.
Page 9, par 7, A number of fusion techniques that are used between the two spatial networks are described here. Moreover, the consequences of each technique are highlighted in the experiments section. Let
PNG
media_image1.png
35
517
media_image1.png
Greyscale
be a fusion function which fuses two feature maps
PNG
media_image2.png
30
72
media_image2.png
Greyscale
PNG
media_image3.png
33
127
media_image3.png
Greyscale
and
PNG
media_image4.png
38
215
media_image4.png
Greyscale
that belong to two different networks to produce an output
PNG
media_image5.png
32
76
media_image5.png
Greyscale
PNG
media_image6.png
32
128
media_image6.png
Greyscale
, where H, W and D are the height, width and number of channels of the feature maps, respectively. The number of feature maps are based on the specific architecture of the network (in our case, there are
PNG
media_image7.png
32
187
media_image7.png
Greyscale
for convolutional layer 5). Function f can be employed at various stages in the networks to achieve early, mid or late fusion.
Page 10, par 3, After spatial fusion stages, we propose to go deeper to represent the temporal information in a proper way to help utilise the highly discriminative motion features. Our deeper motion model consists of a CNN based architecture that employs multiple distinct convolution operations to help identify discriminative features.); and
providing functionality on a computing device related to the scene and based on the final spatial model (Borer, [0047] After rendering module 130 applies the rigging poses 210 to the 3D asset model 220, the surface geometry of the 3D asset model 220 changes to match the corresponding rigging pose 210. Rendering module 130 modifies a position and/or orientation of a virtual camera to view the 3D asset model 220 from various angles and positions. Additionally or alternatively, rendering module 130 rotates the 3D asset model 220 in 3D space while keeping the virtual camera in the same position and orientation. Rendering module 130 repeats this process for various rigging poses 210 to generate 3D asset poses and views 230.).
Regarding Claim 6. The combination of Frahm, Borer and Al-Faris further teaches The computer-implemented method of claim 1, wherein extracting the feature map comprises applying a convolutional network to the image to determine the feature map (Al-Faris, page 8, par 2-3, convolutional neural networks (CNNs) can automatically achieve feature extraction directly from input data. This can help solve the exhaustive search of hand-crafted methods. Furthermore, CNN operations include the local receptive field, shared weights and pooling. These processes are adopted to attain shift, scale and distortion invariance that can improve recognition accuracy.).
The reasoning for combination of Frahm, Borer and Al-Faris is the same as described in Claim 1.
Regarding Claim 10. The combination of Frahm, Borer and Al-Faris further teaches The computer-implemented method of claim 1, wherein the scene reconstruction model generates the final spatial model by:
generating an aggregate feature map from the feature maps of the images,
wherein generating the second spatial model comprises generating the second spatial model further based on the aggregate feature map (Al-Faris, page 9, par 7, A number of fusion techniques that are used between the two spatial networks are described here. Moreover, the consequences of each technique are highlighted in the experiments section. Let
PNG
media_image1.png
35
517
media_image1.png
Greyscale
be a fusion function which fuses two feature maps
PNG
media_image2.png
30
72
media_image2.png
Greyscale
PNG
media_image3.png
33
127
media_image3.png
Greyscale
and
PNG
media_image4.png
38
215
media_image4.png
Greyscale
that belong to two different networks to produce an output
PNG
media_image5.png
32
76
media_image5.png
Greyscale
PNG
media_image6.png
32
128
media_image6.png
Greyscale
, where H, W and D are the height, width and number of channels of the feature maps, respectively. The number of feature maps are based on the specific architecture of the network (in our case, there are
PNG
media_image7.png
32
187
media_image7.png
Greyscale
for convolutional layer 5). Function f can be employed at various stages in the networks to achieve early, mid or late fusion.).
The reasoning for combination of Frahm, Borer and Al-Faris is the same as described in Claim 1.
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al in view of Borer et al, Al-Faris et al further in view of Mahendran et al (US20210407125).
Regarding Claim 2. The combination of Frahm, Borer and Al-Faris fails to explicitly teach, however, Mahendran teaches The computer-implemented method of claim 1, wherein the image data comprises a plurality of images and a camera pose for each image (Mahendran, abstract, the invention describes methods for object recognition neural network for amodal center prediction. One of the methods includes receiving an image of an object captured by a camera. The image of the object is processed using an object recognition neural network that is configured to generate an object recognition output. The object recognition output includes data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image.
[0010] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The object recognition neural network predicts a two-dimensional (2-D) amodal center of an object in an input image, along with a bounding box of the object and a category of the object. The 2-D amodal center of an object is a projection of a predicted 3-D center of the object under a camera pose of the camera that captured the input image. The 2-D amodal centers can be a very sparse representation of the objects in the input image and can efficiently store information of the number of objects in the scene and their corresponding locations. The 2-D amodal center can be employed by users or application developers as an efficient and effective substitute for other 2-D or 3-D object representations that might be computationally more expensive. For example, a 2-D amodal center can be a substitute for a 3-D object bounding box, a 3-D point cloud representation, or a 3-D mesh representation, etc. The number and locations of the 3-D
objects recognized in the scene can be efficiently stored, and can be efficiently accessed and queried by the application developers. In some implementations, multiple 2-D amodal centers of the same object predicted from multiple input images captured under different camera poses can be combined to determine a 3-D center of the object.).
Frahm, Borer, Al-Faris and Mahendran are analogous art because they all teach method of generating 3D scene model from input images using machine learning. Mahendran further teaches input images with plurality of camera poses to the object recognition neural model. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm, Borer and Al-Faris), to further input images with plurality of camera poses (taught in Mahendran), so as to accurately recognize moving objects/users (Mahendran, [0005]).
Regarding Claim 3. The combination of Frahm, Borer, Al-Faris and Mahendran further teaches The computer-implemented method of claim 2, wherein the camera pose for each image is captured by a position sensor coupled to the camera (Mahendran, [0031] The AR system 100 may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction of the physical world, which may serve as an abstract layer for the map and provide spatial information.).
The reasoning for combination of Frahm, Borer, Al-Faris and Mahendran is the same as described in Claim 2.
Regarding Claim 4. The combination of Frahm, Borer, Al-Faris and Mahendran further teaches The computer-implemented method of claim 2, wherein the camera pose for each image is estimated by a pose estimation model based on the images (Frahm, [0105] Systems/methods according to some embodiments are directed to the dense part of three dimensional urban reconstruction, and may employ the redundancy of the estimated depths of the scene typically delivered through depth
estimation for every video frame, as for example delivered by a system along the lines of the system disclosed in M. Pollefeys and et al., Detailed Real-Time Urban 3d Reconstruction From Video, Intl. Journal of Computer Vision (2008), which is incorporated herein by reference. The inputs to these systems/methods are one or more video sequences along with the estimated camera poses and the intrinsic calibration for every frame, a depth map for every frame, and an estimate of the world's vertical or gravity direction. Camera parameters can be recovered with Structure from Motion (SfM) techniques, as well as from inertial sensor data and/or GPS measurements. Depth maps can be computed robustly and efficiently using GPU-accelerated multi-view stereo, as discussed in S. Kim et al., Gain Adaptive Real-Time Stereo Streaming, International Conference on Computer Vision Systems (2007), which is incorporated herein by reference. The vertical or gravity direction can be easily obtained from the inertial sensors or from the vertical vanishing point in each image, as discussed in S. Teller, Automated Urban Model Acquisition: Project Rationale And Status, Image Understanding Workshop (1998), pp. 455-462, which is incorporated herein by reference. The output of these systems/methods is a textured three-dimensional polygonal mesh.).
Claims 5 are rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al in view of Borer et al, Al-Faris et al further in view of Guo et al (US 20220057806).
Regarding Claim 5. The combination of Frahm, Borer and Al-Faris fails to explicitly teach, however, Guo teaches The computer-implemented method of claim 1, wherein predicting the depth map comprises applying a depth estimation model to the image to determine the depth map (Guo, abstract, the invention describes methods for detecting obstacles from unknown objects during automated driving by a vehicle. In one embodiment, a method includes generating, from an image that includes an unknown object, a depth map from a depth estimation component and processed data from a semantic segmentation component in parallel by using a neural network model. The method also includes detecting that the unknown object is an obstacle when the unknown object satisfies criteria using an optical model according to the depth map and a segmentation map. The method also includes determining a height of the obstacle and a distance to the obstacle according to the optical model and the criteria. The method also includes adapting a vehicle plan of the automated driving according to the height.
[0006] In one embodiment, a detection system for detecting obstacles from unknown objects during automated driving by a vehicle is disclosed. The detection system includes one or more processors and a memory communicably coupled to the one or more processors. The memory stores a detection module including instructions that when executed by the one or more processors cause the one or more processors to generate, from an image that includes an unknown object, a depth map from a depth estimation component and processed data from a semantic segmentation component in parallel by using a neural network model. The detection module also includes instructions to detect that the unknown object is an obstacle when the unknown object satisfies criteria using an optical model according to the depth map and a segmentation map.).
Frahm, Borer, Al-Faris and Guo are analogous art because they all teach method of generating 3D scene model from input images using machine learning. Guo further teaches using a depth estimator to generate depth map. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm, Borer and Al-Faris), to further use the depth estimator (taught in Guo), so as to provide effective obstacle detection model for scenario such as automated driving (Guo, [0002-0004]).
Claims 7 are rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al in view of Borer et al, Al-Faris et al further in view of Morariu et al (US20200175095).
Regarding Claim 7. The combination of Frahm, Borer and Al-Faris fails to explicitly teach, however, Morariu teaches The computer-implemented method of claim 1, wherein the feature map comprises a first tensor for a first feature type and a second tensor for a second feature type (Morariu, abstract, the invention describes methods for identifying sets of objects for the electronic document by applying a set of object-recognition rules to the electronic document, with each object-recognition rule generating a set of identified objects. The computing system generates feature maps that represent a set of identified objects. The computing system generates a heat map that identifies attributes of the electronic document including object candidates of the electronic document by applying a page-segmentation machine learning model to the electronic document. The computing system computes a tag by applying a fusion deep learning module to the feature map and the heat map to correlate a document object identified by the feature map with an attribute of the electronic document identified by the heat map. The computing system generates the tagged electronic document by applying the tag to the electronic document.
[0031] In a non-limiting example, a rule feature map for a particular object recognition rule can have a tensor with a shape of (HOR, WOR, DOR). In this example, the state feature map has three dimensions. The term HOR is an integer representing the size of a first dimension, which corresponds to the height of input electronic document 120 in this example. The term WOR is an integer representing the size of a second dimension, which corresponds to the width of input electronic document 120 in this example. The term DOR is an integer representing the size of a third dimension, which corresponds to a number of object types in this example. This three-dimensional rule feature map for a particular object-recognition rule is indexed by a three-dimensional index (h, w, d), where h is an integer having possible values [1 ... HOR], w is integer having possible values [1 ... WOR], and d is an integer having possible value integer having possible values [1 ... DOR]. For instance, if a state feature map is a three-dimensional array of numbers having a height, width, and depth, then (h, w, d) refers to the real number value in row h, column w, and depth d. For instance, the real number value in row h, column w, and depth d may be a binary value that indicates presence "1" or absence "0" of an object type stored at a particular index. A tensor size of the state feature map (HOR, WOR, DOR) can be the number of values contained with the state feature map, i.e., HOR *WOR *DOR).
Frahm, Borer, Al-Faris and Morariu are analogous art because they all teach method of feature detecting using machine learning. The combination of Frahm, Borer and Al-Faris further teaches generating 3D model from input images. Morariu further teaches tensors corresponding to feature map. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm, Borer and Al-Faris), to further uses the multi-dimensional tensors for feature map generation (taught in Morariu), so as to accurately detect objects using a rule-based feature map (Morariu, [0004-0005]).
Claims 11-13, 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al in view of Borer et al, Al-Faris et al further in view of Tsai et al (US20100188397).
Regarding Claim 11. The combination of Frahm, Borer and Al-Faris fails to explicitly teach, however, Tsai teaches The computer-implemented method of claim 1, wherein providing functionality on the computing device related to the scene and based on the final spatial model comprises:
receiving a request from the computing device to view the final spatial model of the scene;
responsive to the request, obtaining the final spatial model of the scene from a map database; and
transmitting the final spatial model to the computing device for viewing the final spatial model of the scene, wherein the computing device is configured to render a perspective of the final spatial model based on a pose of the computing device (Tsai, abstract, the invention describes methods are providing for navigating a three-dimensional model using deterministic movement of an electronic device. An electronic device can load and provide an initial display of a three-dimensional model (e.g., of an environment or of an object). As the user moves the electronic device, motion-sensing components, positioning circuitry, and other components can detect the device movement and adjust the displayed portion of the three-dimensional model to reflect the movement of the device. By walking with the device in the user's real environment, a user can virtually navigate a representation of a three-dimensional environment. In some embodiments, a user can record an object or environment using an electronic device, and tag the recorded images or video with movement information describing the movement of the device during the recording. The recorded information can then be processed with the movement information to generate a three-dimensional model of the recorded environment or object.
[0028] Electronic device 100 can include a processor or control circuitry 102, storage 104, memory 106 input/output circuitry 108, and communications circuitry 112, as typically found in an electronic device of the type of electronic device 100, and operative to enable any of the uses expected from an electronic device of the type of electronic device 100 (e.g., connect to a host device for power or data transfers).
[0046] In some embodiments, a user can instead or in addition navigate a three-dimensional model of an object. FIGS. 4A-4E are a series of schematic views of a three-dimensional object seen from different perspectives as the user moves the device in accordance with one embodiment of the invention. Displays 400, 410, 420, 430 and 440 can each include representations 402, 412, 422, 432 and 442, respectively, showing different perspectives of a three-dimensional model. The particular orientation at which the electronic device is held can be indicated by the perspective view of devices 404, 414, 424, 434 and 444, respectively. As the user moves the electronic device, the perspective of the three-dimensional model shown can change to reflect the orientation of the device relative to the model.).
Frahm, Borer, Al-Faris and Tsai are analogous art because they all teach method of viewing 3D model. The combination of Frahm, Borer and Al-Faris further teaches generating 3D model from input images. Tsai further teaches user interface to view the 3D model/scene in different viewpoint. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm, Borer and Al-Faris), to further uses the GUI to navigate the generated 3D model/scene (taught in Tsai), so as to provide user with intuitive method to view a displayed 3D object/environment (Tsai, [0003-0004]).
Regarding Claim 12. The combination of Frahm, Borer, Al-Faris and Tsai further teaches The computer-implemented method of claim 1, wherein providing functionality on the computing device related to the scene and based on the final spatial model comprises generating, using the final spatial model, virtual content for display in conjunction with the image data of the scene (Tsai, [0006] In some embodiments, a user can direct an electronic device to display information that can be associated with three-dimensional navigation such as, for example, three-dimensional models of environments or objects. For example, a user can direct a user to access a mapping application that provides images of what can be seen in some or any direction from a particular location. As another example, a user can direct an electronic device to display a video game in which a user may navigate a virtual world and see, in any direction, what the virtual world resembles (e.g., rendered images of the virtual world's appearance from any location in the virtual world). As still another example, a user can direct the electronic device to display a three-dimensional object (e.g., an object for sale) that the user can manipulate or view from different angles).
The reasoning for combination of Frahm, Borer, Al-Faris and Tsai is the same as described in Claim 11.
Claim 13 is similar in scope as Claim 1, 11, and thus is rejected under same rationale.
Claim 18 is similar in scope as Claim 10, and thus is rejected under same rationale.
Claim 20 is similar in scope as Claim 13, and thus is rejected under same rationale.
Claims 14 are rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al in view of Borer et al, Al-Faris et al, Tsai et al further in view of Mahendran et al (US20210407125).
Claim 14 is similar in scope as Claim 2, and thus is rejected under same rationale.
Frahm, Borer, Al-Faris, Tsai and Mahendran are analogous art because they all teach method of generating and/or viewing 3D model. The combination of Frahm, Borer, Al-Faris and Tsai further teaches generating 3D model from input images and viewing the 3D model/scene in different viewpoint. Mahendran further teaches input images with plurality of camera poses to the object recognition neural model. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation and viewing method (taught in Frahm, Borer, Al-Faris and Tsai), to further input images with plurality of camera poses (taught in Mahendran), so as to accurately recognize moving objects/users (Mahendran, [0005]).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al in view of Borer et al, Al-Faris et al, Tsai et al further in view of Morariu et al (US20200175095).
Claim 15 is similar in scope as Claim 7, and thus is rejected under same rationale.
Frahm, Borer, Al-Faris, Tsai and Morariu are analogous art because they all teach method of generating and/or viewing 3D model. The combination of Frahm, Borer, Al-Faris and Tsai further teaches generating 3D model from input images and viewing the 3D model/scene in different viewpoint. Morariu further teaches tensors corresponding to feature map. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm, Borer, Al-Faris and Tsai), to further uses the multi-dimensional tensors for feature map generation (taught in Morariu), so as to accurately detect objects using a rule-based feature map (Morariu, [0004-0005]).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Frahm et al in view of Borer et al, Al-Faris et al, Tsai et al further in view of Sanor et al (US20220189124).
Regarding Claim 19. The combination of Frahm, Borer, Al-Faris and Tsai fails to explicitly teach, however, Sanor teaches The computer-implemented method of claim 13, further comprising:
augmenting the spatial model with one or more virtual elements, wherein transmitting the spatial model comprises transmitting the spatial model with the augmented one or more virtual elements to the computing device (Sanor, abstract, the invention describes methods for displaying a real-world vehicle in an augmented reality environment. The system employs a user device camera to obtain image data of an environment that includes a real-world vehicle. The system analyzes the image data to identify the vehicle within the environment. A wireframe model of the vehicle is then generated and registered to the vehicle. The image data is displayed on the user device. In response to user input, the system may then attach a virtual vehicle accessory to the wireframe model. The accessory is then displayed on the user device display in an augmented reality environment such that the vehicle appears to seamlessly incorporate the accessory.
[0032] As used herein, identifying a real-world object may refer to both the detection and classification of an object within the image data. Furthermore, a machine learning model may be implemented during registration of the multidimensional wireframe model of the identified object and the identified object captured in the image data being captured by the imaging device. For example, the machine learning model may be trained to identify whole objects and isolated features of an object so that registration of the isolated features of the object and the multidimensional wireframe model may be accurately and precisely mapped to each other. The multidimensional wireframe model may be a predefined computer aided model of the object. For example, the multi-dimensional wireframe model may be a three-dimensional spatial model that may appear as overlaying portions of an image (e.g., still image, live video stream, recorded video, etc.) of an object (e.g., a vehicle) within a physical environment. Moreover, the augmented reality system described herein may utilize the trained machine learning model to associate a virtual object (e.g., a virtual accessory) with a portion of the virtual wireframe model such that the virtual object may appear precisely positioned on or integrated with the object (e.g., a vehicle) within the physical environment.).
Frahm, Borer, Al-Faris, Tsai and Sanor are analogous art because they all teach method of generating and/or viewing 3D model. The combination of Frahm, Borer, Al-Faris and Tsai further teaches generating 3D model from input images and viewing the 3D model/scene in different viewpoint. Sanor further teaches augment the 3D virtual model with extra virtual object. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the 3D scene model generation method (taught in Frahm, Borer, Al-Faris and Tsai), to further augment the generated 3D model with extra object (taught in Sanor), so as to provide user with extra accessory data in scenario such as display extra feature for a vehicle (Sanor, [0025]).
Allowable Subject Matter
Claims 8-9, 16-17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 8, it recites “wherein the first spatial model is a heightfield, and wherein generating the first spatial model comprises:
generating a 3D model using truncated signed distance field with the predicted depth maps; and
ray casting the 3D model to generate the heightfield” in the context of Claim 8.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 8. Therefore, Claim 8 is allowable over prior art.
Claim 9 depends from Claim 8 with respective additional limitations. Therefore, Claim 9 is allowable over prior art.
Claim 16 recites similar limitations as discussed above with regard to claim 8. Therefore, claim 16 is allowable over prior art.
Claim 17 depends from Claim 16 with respective additional limitations. Therefore, Claim 17 is allowable over prior art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Zienkiewicz et al ("Real-time height map fusion using differentiable rendering.", IEEE, 2016), abstract, the paper describes a robust real-time method which performs dense reconstruction of high quality height maps from monocular video. By representing the height map as a triangular mesh, and using efficient differentiable rendering approach, our method enables rigorous incremental probabilistic fusion of standard locally estimated depth and colour into an immediately usable dense model. We present results for the application of free space and obstacle mapping by a lowcost robot, showing that detailed maps suitable for autonomous navigation can be obtained using only a single forward-looking camera.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN SHENG whose telephone number is (571)272-5734. The examiner can normally be reached M-F 9:30AM-3:30PM 6:00PM-8:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 5712723022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Xin Sheng/ Primary Examiner, Art Unit 2619
Prosecution Insights