Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/03/2026 has been entered.
Response to Amendment
This is in response to applicant’s amendment/response filed on 02/03/2026, which has been entered and made of record. Claims 1, 7, 11, 17, 21 and 27 have been amended. No claim has been cancelled. No claim has been added. Claims 1-30 are pending in the application.
Response to Arguments
Applicant's arguments filed on 02/03/2026 regarding claims rejection under 35 U.S.C 103 have been fully considered but they are not persuasive.
Applicant submits “creating a mesh does not associate the created mesh with a segmentation class, much less associate the created mesh with a segmentation class for a new object. Further, segmenting the mesh and then identifying the segments does not teach associating the mesh with a segmentation class for a new object, which, by definition, would not be identified.” (Remarks, Page 11)
The examiner disagrees with Applicant’s premises and conclusion. Foco teaches in ¶0042, “take a point cloud or mesh as input, and can associate embeddings with point cloud segments that were identified by the semantic segmentation module 416”. Point cloud or mesh as input can associate with segments.
Foco also teaches in ¶0064, “This image data may already be registered, or may be registered using a process such as identifying and correlating feature points in the image data. A three-dimensional (3D) representation, such as a mesh or point cloud, can then be generated 704 for the environment based at least on this image data, as well as other data that may be provided with respect to the environment. The 3D representation can be analyzed 706 to determine one or more segments, segmentations, or portions of the 3D representation that correspond to one or more objects in the environment. An identity of these one or more objects can then be determined 708, based at least on the data corresponding to the one or more segments for those objects.” Once identity of the object is determined, the object and the segment are registered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-30 are rejected under 35 U.S.C. 103 as being unpatentable over Foco et al. (US Pub 2024/0203052 A1) in view of Gowda et al. (US Patent 11,640,692 B1) and Li et al. (US Pub 2022/0139037 A1).
As to claim 1, Foco discloses an apparatus for image processing, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory (Fig.4 and 9, 10A and B, Fig. 11-15), the at least one processor being configured to:
generate a segmentation class for a first object in a received first image (Fig. 8, ¶0023, “generate a representation of an environment, in which objects in the environment can be segmented and identified based on the captured image data.” ¶0023, “For objects that are classified, such as objects that match images and other representations from a catalog or repository of three-dimensional (3D) models, stored 3D geometry representing the actual shape or dimensions of the objects can be used, and for other objects a new estimated 3D reconstruction/approximation (e.g., mesh) can be generated.”),
the segmentation class is generated based on semantic segmentation of the received first image (¶0038, “The initial representation of the environment can be passed to a semantic segmentation module 416. This module may include at least one segmentation algorithm or neural network, for example, that is trained to segment the environment into individual objects, where possible or at least a minimum confidence is satisfied.”)
generate a first three-dimensional (3D) model of the first object (¶0023, “The image and/or video data can be analyzed to generate a 3D model or representation of the environment, which can then be analyzed using one or more models (that may be class-specific) to identify segments of this model that correspond to individual objects of those classes. Vector or latent representations of these objects can be generated and compared to a library or repository of objects in order to accurately identify these objects.”);
compare the first object against a set of registered objects based on the generated segmentation class to determine that the first object is not in the set of registered objects (¶0023, “For objects that are classified, such as objects that match images and other representations from a catalog or repository of three-dimensional (3D) models, stored 3D geometry representing the actual shape or dimensions of the objects can be used, and for other objects a new estimated 3D reconstruction/approximation (e.g., mesh) can be generated.” “The image and/or video data can be analyzed to generate a 3D model or representation of the environment, which can then be analyzed using one or more models (that may be class-specific) to identify segments of this model that correspond to individual objects of those classes. Vector or latent representations of these objects can be generated and compared to a library or repository of objects in order to accurately identify these objects.” ¶0040, “compare object segmentations against data stored in an object database 418 or other such location. This module can find a set of the most suitable 3D models in an existing database of 3D shapes and materials for either all or a subset of objects that were found in either an image or a point cloud by, for example, the semantic segmentation module 416.” ¶0041, “An object representation may be generated using a mesh generator 422 for objects that are unable to be successfully identified, or for objects that are able to be identified but for which models, point clouds, meshes, or other representations do not already exist in the database and must be created or generated if they are to be included in a representation of an environment.”);
associate the generated segmentation class with the first 3D model based on the determination that the first object is not in the set of registered objects (¶0041, “An object representation may be generated using a mesh generator 422 for objects that are unable to be successfully identified, or for objects that are able to be identified but for which models, point clouds, meshes, or other representations do not already exist in the database and must be created or generated if they are to be included in a representation of an environment.” ¶0042, “the object classification module 420 may return the object that is determined to be closest based on the proximity of its embedding in the latent space. A network, such as a transformer, ResNet for images, or Minokowski Engine for point clouds, may be utilized that can take a point cloud or mesh as input, and can associate embeddings with point cloud segments that were identified by the semantic segmentation module 416.” ¶0063, “If it is determined that one or more objects are unable to be accurately classified, or if no stored representation exists for an identified object, then a surface representations, such as a mesh, point cloud, or other free-form representation, can be utilized 614 that was generated or reconstructed for those objects, as may be based at least on the points in the respective representations for those objects, as well as potentially image features in a corresponding portion of the scan data.” ¶0064, “This image data may already be registered, or may be registered using a process such as identifying and correlating feature points in the image data.”)
register the first 3D model of the first object and associated segmentation class based on the determination that the first object is not in the set of registered objects (¶0023, “for other objects a new estimated 3D reconstruction/approximation (e.g., mesh) can be generated. This representation can be generated by capturing video data, such as RGB-D data, for a scene (without the necessity of slower, user or compute intensive techniques such as motion capture, etc.), multiple images, or even a single image.” ¶0058, “corner, edge, or extrema points, among others, that should be easily identifiable in different views, and that enable those views to be registered together into a single representation.” ¶0064, “This image data may already be registered, or may be registered using a process such as identifying and correlating feature points in the image data” “An identity of these one or more objects can then be determined 708, based at least on the data corresponding to the one or more segments for those objects.”); and
output the first 3D model of the first object (¶0023, “A result can then be a 3D representation of a scene or environment in which objects are identified and segmented as individual objects, and representations (e.g., images) of the scene or environment can be viewed and interacted with through various viewports, positions, and perspectives.” ¶0024, “Such approaches can also be beneficial in scanning and classifying real world objects to add to a virtual environment, such as an item that is to be added as a usable virtual object in a video game, or an item that is available through an e-commerce platform and may be viewable in various potential placements.”).
Foco does not explicitly disclose the segmentation class is generated based on semantic segmentation of the received first image, and wherein the first image is a two-dimensional (2D) image.
Gowda teaches the segmentation class is generated based on semantic segmentation of the received first image, and wherein the first image is a two-dimensional (2D) image (Gowda, Col 6, lines 35-50, “The segmentation unit 244 is configured with instructions executable by a processor to generate segmentation data of the physical environment using one or more of the techniques disclosed herein. For example, the segmentation unit 244 obtains a sequence of light intensity images (e.g., RGB) from a light intensity camera (e.g., a live camera feed) and performs a semantic segmentation algorithm to assign semantic labels to recognized features (e.g., walls, doors, floor, windows, etc.) and/or objects (e.g., furniture, appliances, people, etc.) in the image data. The segmentation unit 244 can then generate segmented data, i.e., semantic image data (e.g., RGB-S), using one or more of the techniques disclosed herein. In some implementations, the segmentation includes confidence levels for each identified feature and/or object for each pixel location.” Col 11, lines 15-37, “the method 400 obtains a segmentation mask associated with a second image, where the segmentation mask identifies portions of the second image associated with an object. The second image includes pixels each having a value provided by a camera (e.g., RGB cameras with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor, monochrome cameras, IR cameras, event-based cameras, or the like). For example, the device (e.g., device 120) may include a light intensity camera (e.g., RGB camera) that captures the second image, a segmentation machine learning model to generate the segmentation mask identifying objects of a particular type (e.g., person, cat) associated with motion (e.g., people frequently move/couches do not). In some implementations, the mask may indicate objects using values 0 or 1. In some implementations, the segmentation machine learning model may be a neural network executed by a neural engine/circuits on the processor chip tuned to accelerate AI software. In some implementations, the segmentation mask may include confidence values at the pixel level. For example, a pixel location may be labeled as 0.8 chair, thus, the segmentation machine learning model is 80% confident that the x,y,z coordinates for that pixel location is a chair. As additional data is obtained, the confidence level at each pixel location may be adjusted.” Col 14, lines 25-45, “The example environment 600 further includes a segmentation unit 620 that is configured with instructions executable by a processor to obtain the light intensity image data (e.g., light intensity data 607) and identify and segment wall structures (wall, doors, windows, etc.) and objects (e.g., person, table, teapot, chair, vase, etc.) using one or more known techniques. For example, the segmentation unit 620 (e.g., segmentation unit 244 of FIG. 2, and/or segmentation unit 344 of FIG. 3) receives intensity image data 607 from the image sources (e.g., light intensity camera 606), and generates segmentation data 622 (e.g., semantic segmentation data such as RGB-S data). For example, the semantic segmentation data 624 illustrates an example semantically labelled image of the physical environment 105 in FIG. 1. In some implementations, segmentation unit 620 uses a machine learning model, where a semantic segmentation model may be configured to identify semantic labels for pixels or voxels of image data. In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), decision tree, support vector machine, Bayesian network, or the like.”).
Foco and Gowda are considered to be analogous art because all pertain to generating three-dimensional geometric representations of physical environments. It would have been obvious before the effective filing date of the claimed invention to have modified Foco with the features of “the segmentation class is generated based on semantic segmentation of the received first image, and wherein the first image is a two-dimensional (2D) image” as taught by Gowda. The suggestion/motivation would have been in order to provide a more useful, realistic, or physically meaningful model (Gowda, Col 1, lines 60-65). The claim would have been obvious because the technique for improving a particular class of devices was part of the ordinary capabilities of a person of ordinary skill in the art, in view of the teaching of the technique for improvement in other situations.
Assuming arguendo that Foo does not teaches associate the generated segmentation class with the first 3D model based on the determination that the first object is not in the set of registered objects and register the first 3D model of the first object and associated segmentation class based on the determination that the first object is not in the set of registered objects.
Li teaches associate the generated segmentation class with the first 3D model based on the determination that the first object is not in the set of registered objects and register the first 3D model of the first object and associated segmentation class based on the determination that the first object is not in the set of registered objects (Li, ¶0081, “The model can represent objects as a collection of deformable parts, in which each part may be semantically coherent across different instances of the same category (e.g., wings on birds and wheels on cars). Therefore, by leveraging self-supervised learned part segmentation of a large collection of category-specific images, semantic consistency can be enforced within the model between the reconstructed meshes and the original images.” “The model can easily generalize to various object categories without such labels, such as horses, penguins, and the like.” ¶0084, “reconstructed object mask 134 corresponds to a semantic part segmentation that indicates which pixels in the input image correspond to sematic parts of the object in the image.” ¶0094, “self-supervised co-part segmentation is leveraged to decompose 2D images into a collection of semantic parts. The semantic parts of different object instances can be associated with each other and a category-level canonical semantic UV map can be built through semantic part invariance” “the self-supervised reconstruction model is learned by encouraging the consistency of semantic part labels in both the 2D and 3D space” ¶0096, “Self-supervised co-parts segmentation may be utilized to enforce semantic consistency” Fig. 7, ¶0114, “FIG. 7 illustrates how meshes with semantic labels might be generated for input images, allowing for 3D reconstruction of objects depicted in the input 2D images and generation of 2D images from alternative camera poses” 3d reconstruction via semantic consistency teaches the associate the generated segmentation class with the first 3D model.)
Foco, Gowda and Li are considered to be analogous art because all pertain to generating three-dimensional geometric representations of physical environments. It would have been obvious before the effective filing date of the claimed invention to have modified Foco with the features of “associate the generated segmentation class with the first 3D model based on the determination that the first object is not in the set of registered objects and register the first 3D model of the first object and associated segmentation class based on the determination that the first object is not in the set of registered objects” as taught by Li. The suggestion/motivation would have been in order to reduce ambiguities during joint prediction of shape of an object, a camera pose of the object, and/or texture of the object (Li, ¶0081).
As to claim 2, claim 1 is incorporated and the combination of Foco, Gowda and Li discloses the segmentation class is generated as part of a semantic segmentation process (Foco, ¶0038, “The initial representation of the environment can be passed to a semantic segmentation module 416. This module may include at least one segmentation algorithm or neural network, for example, that is trained to segment the environment into individual objects, where possible or at least a minimum confidence is satisfied.”).
As to claim 3, claim 1 is incorporated and the combination of Foco, Gowda and Li discloses to compare the first object against the set of registered objects, the at least one processor is configured to compare the segmentation class of the first object to the set of registered objects (Foco, ¶0023, ““The image and/or video data can be analyzed to generate a 3D model or representation of the environment, which can then be analyzed using one or more models (that may be class-specific) to identify segments of this model that correspond to individual objects of those classes. ” ¶0041, “An object representation may be generated using a mesh generator 422 for objects that are unable to be successfully identified, or for objects that are able to be identified but for which models, point clouds, meshes, or other representations do not already exist in the database and must be created or generated if they are to be included in a representation of an environment.” ¶0043, “a pre-existing or generated representation of an object, once determined or generated, can be substituted into the representation of the environment in place of an estimated or inferred representation from an initial (or intermediate) representation of the environment.”).
As to claim 4, claim 1 is incorporated and the combination of Foco, Gowda and Li discloses the at least one processor is further configured to generate a set of features for the first object, and wherein, to compare the first object against the set of registered objects, the at least one processor is configured to compare features of the set of features against features of the set of registered objects (Foco, ¶0029, “These representations may include images, models, meshes, point clouds, feature vectors, embeddings, or other such representations. Once a segmentation or representation of a chair object in the environment is determined or generated, that segmentation can be compared against these stored representations of chairs in order to determine a match, or a known chair type that corresponds to the generated chair segmentation with at least a minimum level of confidence or certainty, or with less than a maximum allowable amount of deviation or error in the match.” ¶0040, “the object classification module 420 can include an algorithm or neural network trained to compare object segmentations against data stored in an object database 418 or other such location. This module can find a set of the most suitable 3D models in an existing database of 3D shapes and materials for either all or a subset of objects that were found in either an image or a point cloud by, for example, the semantic segmentation module 416.” “The comparison include, for example, an algorithm that compares shape data for 3D segmentations, or a neural network that compares feature vectors or embeddings in a latent space, among other such options.” ¶0063, “this can include generating an embedding or feature vector for these objects and searching for matching or proximate points in an embedding or latent space.”).
As to claim 5, claim 1 is incorporated and the combination of Foco, Gowda and Li discloses the first 3D model comprises a 3D model of a portion of the first object visible in the first image (Foco, ¶0023, “The image and/or video data can be analyzed to generate a 3D model or representation of the environment, which can then be analyzed using one or more models (that may be class-specific) to identify segments of this model that correspond to individual objects of those classes. ¶0037, “data from an environment scan may be separated into sections to allow a 3D model to be optimized to compensate for drift. Where depth information is not available, depth may be inferred from the input data and used to estimate the locations of specific points in this 3D representation space. The points from different views can then be fused or otherwise utilized to generate a single, consistent representation of the environment, such as a point cloud or mesh that represents the overall environment, that is consistent from multiple views. In some embodiments, a sparse 3D model can be generated based at least by using these keypoints and normals to register the different views and build a consistent sparse representation of the environment, and then the data from these various views can be used to fill in the sparse model once generated.”).
As to claim 6, claim 1 is incorporated and the combination of Foco, Gowda and Li discloses registering the first 3D model comprises storing the first 3D model in a memory (Foco, ¶0071, “any portion of code and/or data storage 1001 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.” ¶0112, “different types of registers for storing different types of data”).
As to claim 7, claim 1 is incorporated and the combination of Foco, Gowda and Li discloses the at least one processor is further configured to:
generate a segmentation class for a second object in a second image, wherein the second object comprises a different instance of the first object (Foco, ¶0058, “there will be more than one image or instance of sensor data to be analyzed. This may include, for example, images or video frames captured from different locations or orientations, with different fields of views or resolutions, and may even be captured using multiple devices.”);
generate a second 3D model of the second object (Foco, ¶0058, “multiple initial models can be generated based on this different data, and then these keypoints can be used to register, correlate, and refine these separate models into a single, consistent representation.”);
compare the second object against a set of registered objects to determine that the second object matches the first object; and incorporate a portion of the second 3D model into the first 3D model (Foco, ¶0041, “An object representation may be generated using a mesh generator 422 for objects that are unable to be successfully identified, or for objects that are able to be identified but for which models, point clouds, meshes, or other representations do not already exist in the database and must be created or generated if they are to be included in a representation of an environment. The mesh generator 422 can include any appropriate algorithm, component, network, or process that is able to take data such as point cloud or dimension data and generate a 2D or 3D virtual representation of an object as discussed herein. In at least one embodiment, a meshing and texturing technique can be used without de-lighting, while another embodiment might utilize a NeRF-based implementation from the Kaolin library, which can be extended to reconstruct objects using point cloud data, camera poses, and/or multiple views from an original RGB(-D) scan. A meshing approach might utilize point cloud and normal data to find an implicit surface representation using samples on a recursively subdivided grid, which can be converted to a mesh, and a technique to sample the color of the surface from point cloud and normal data. An example NeRF-based (or similar) implementation can utilize differentiable rendering, where the constructed surface is represented by a neural network, subject to a global optimization.”. ¶0058, “In many instances, there will be more than one image or instance of sensor data to be analyzed. This may include, for example, images or video frames captured from different locations or orientations, with different fields of views or resolutions, and may even be captured using multiple devices. In order to generate an accurate representation of the environment that is consistent both spatially and temporally, this data needs to be correlated such that the same features are identified and correlated across these various images, frames, or instances. In at least some embodiments, this can be performed by determining and registering a number of keypoints 502 in each image or representation of the environment, as illustrated in FIG. 5. These keypoints can include any unique or identifying features of the environment that can be useful in registering different instances of the data. These may include, for example, corner, edge, or extrema points, among others, that should be easily identifiable in different views, and that enable those views to be registered together into a single representation. As illustrated in FIG. 5, a visual appearance can be determined for each of these keypoints to assist with registration for data captured from different points of view, and which may represent different subsets of the environment that may have only a limited number of overlapping keypoints. Such keypoints can also be useful in stabilizing camera tracking over time. In some embodiments, these keypoints can be used to register captured data that can be used to generate a representation of the environment. In other embodiments, multiple initial models can be generated based on this different data, and then these keypoints can be used to register, correlate, and refine these separate models into a single, consistent representation.”)
As to claim 8, claim 7 is incorporated and the combination of Foco, Gowda and Li discloses to incorporate the portion of the second 3D model into the first 3D model, the at least one processor is configured to: retrieve the first 3D model (Foco, ¶0058, “In order to generate an accurate representation of the environment that is consistent both spatially and temporally, this data needs to be correlated such that the same features are identified and correlated across these various images, frames, or instances. In at least some embodiments, this can be performed by determining and registering a number of keypoints 502 in each image or representation of the environment”);
align the second 3D model with the first 3D model (Foco, ¶0058, “this data needs to be correlated such that the same features are identified and correlated across these various images, frames, or instances.” “these keypoints can be used to register, correlate, and refine these separate models into a single, consistent representation.”); and
incorporate the portion of the second 3D model into the first 3D model based on the aligning (Foco, ¶0058, “These keypoints can include any unique or identifying features of the environment that can be useful in registering different instances of the data. These may include, for example, corner, edge, or extrema points, among others, that should be easily identifiable in different views, and that enable those views to be registered together into a single representation.” “a visual appearance can be determined for each of these keypoints to assist with registration for data captured from different points of view, and which may represent different subsets of the environment that may have only a limited number of overlapping keypoints. Such keypoints can also be useful in stabilizing camera tracking over time. In some embodiments, these keypoints can be used to register captured data that can be used to generate a representation of the environment. In other embodiments, multiple initial models can be generated based on this different data, and then these keypoints can be used to register, correlate, and refine these separate models into a single, consistent representation.”).
As to claim 9, claim 8 is incorporated and the combination of Foco, Gowda and Li discloses to align the second 3D model with the first 3D model, the at least one processor is configured to align features points associated with the second 3D model with feature points associated with the first 3D model (Foco, ¶0058, “this data needs to be correlated such that the same features are identified and correlated across these various images, frames, or instances.” “these keypoints can be used to register, correlate, and refine these separate models into a single, consistent representation.”).
As to claim 10, claim 8 is incorporated and the combination of Foco, Gowda and Li discloses to align the second 3D model with the first 3D model, the at least one processor is configured to generate a transformation matrix (Foco, ¶0049, “Landmarks can be transformed or projected to be within the camera frustum in a new view.” ¶0055, “For geometric cues, points in the source vertex/normal map can transform close to corresponding points in the target vertex/normal map. For visual cues, RGB-D (3D) keypoints in the source frame can transforms close to visually matched “virtual” RGB-D keypoints (projected landmarks) in the target frame.” ¶0061).
As to claim 11, the combination of Foco, Gowda and Li discloses a method for image processing, comprising: generating a segmentation class for a first object in a first image, wherein the segmentation class is generated based on semantic segmentation of the received first image, and wherein the first image is a two-dimensional (2D) image; generating a first three-dimensional (3D) model of the first object; comparing the first object against a set of registered objects based on the generated segmentation class to determine that the first object is not in the set of registered objects; associating the generated segmentation class with the first 3D model based on the determination that the first object is not in the set of registered objects; registering the first 3D model of the first object and associated segmentation class based on the determination that the first object is not in the set of registered objects; and outputting the first 3D model of the first object (See claim 1 for detailed analysis.).
As to claim 12, claim 11 is incorporated and the combination of Foco, Gowda and Li discloses the segmentation class is generated as part of a semantic segmentation process (See claim 2 for detailed analysis.).
As to claim 13, claim 11 is incorporated and the combination of Foco, Gowda and Li discloses comparing the first object against the set of registered objects comprises comparing the segmentation class of the first object to the set of registered objects (See claim 3 for detailed analysis.).
As to claim 14, claim 11 is incorporated and the combination of Foco, Gowda and Li discloses generating a set of features for the first object, and wherein comparing the first object against the set of registered objects comprises comparing features of the set of features against features of the set of registered objects (See claim 4 for detailed analysis.).
As to claim 15, claim 11 is incorporated and the combination of Foco, Gowda and Li discloses the first 3D model comprises a 3D model of a portion of the first object visible in the first image (See claim 5 for detailed analysis.).
As to claim 16, claim 11 is incorporated and the combination of Foco, Gowda and Li discloses registering the first 3D model comprises storing the first 3D model in a memory (See claim 6 for detailed analysis.).
As to claim 17, claim 11 is incorporated and the combination of Foco, Gowda and Li discloses generating a segmentation class for a second object in a second image, wherein the second object comprises a different instance of the first object; generating a second 3D model of the second object; comparing the second object against a set of registered objects to determine that the second object matches the first object; and incorporating a portion of the second 3D model into the first 3D model (See claim 7 for detailed analysis.).
As to claim 18, claim 17 is incorporated and the combination of Foco, Gowda and Li discloses incorporating the portion of the second 3D model into the first 3D model comprises: retrieving the first 3D model; aligning the second 3D model with the first 3D model; and incorporating the portion of the second 3D model into the first 3D model based on the aligning (See claim 8 for detailed analysis.).
As to claim 19, claim 18 is incorporated and the combination of Foco, Gowda and Li discloses aligning the second 3D model with the first 3D model comprises aligning features points associated with the second 3D model with feature points associated with the first 3D model (See claim 9 for detailed analysis.).
As to claim 20, claim 18 is incorporated and the combination of Foco, Gowda and Li discloses aligning the second 3D model with the first 3D model comprises generating a transformation matrix (See claim 10 for detailed analysis.).
As to claim 21, the combination of Foco, Gowda and Li discloses a non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: generate a segmentation class for a first object in a first image, wherein the segmentation class is generated based on semantic segmentation of the first image, and wherein the first image is a two-dimensional (2D) image; generate a first three-dimensional (3D) model of the first object; compare the first object against a set of registered objects based on the generated segmentation class to determine that the first object is not in the set of registered objects; associate the generated segmentation class with the first 3D model based on the determination that the first object is not in the set of registered objects; register the first 3D model of the first object and associated segmentation class based on the determination that the first object is not in the set of registered objects; and output the first 3D model of the first object (See claim 1 for detailed analysis.).
As to claim 22, claim 21 is incorporated and the combination of Foco, Gowda and Li discloses the segmentation class is generated as part of a semantic segmentation process (See claim 2 for detailed analysis.).
As to claim 23, claim 21 is incorporated and the combination of Foco, Gowda and Li discloses to compare the first object against the set of registered objects, the instructions cause the at least one processor to compare the segmentation class of the first object to the set of registered objects (See claim 3 for detailed analysis.).
As to claim 24, claim 21 is incorporated and the combination of Foco, Gowda and Li discloses the instructions cause the at least one processor to generate a set of features for the first object, and wherein, to compare the first object against the set of registered objects, the instructions cause the at least one processor to compare features of the set of features against features of the set of registered objects (See claim 4 for detailed analysis.).
As to claim 25, claim 21 is incorporated and the combination of Foco, Gowda and Li discloses the first 3D model comprises a 3D model of a portion of the first object visible in the first image (See claim 5 for detailed analysis.).
As to claim 26, claim 21 is incorporated and the combination of Foco, Gowda and Li discloses registering the first 3D model comprises storing the first 3D model in a memory (See claim 6 for detailed analysis.).
As to claim 27, claim 21 is incorporated and the combination of Foco, Gowda and Li discloses the instructions cause the at least one processor: generate a segmentation class for a second object in a second image, wherein the second object comprises a different instance of the first object; generate a second 3D model of the second object; compare the second object against a set of registered objects to determine that the second object matches the first object; and incorporate a portion of the second 3D model into the first 3D model (See claim 7 for detailed analysis.).
As to claim 28, claim 27 is incorporated and the combination of Foco, Gowda and Li discloses to incorporate the portion of the second 3D model into the first 3D model, the instructions cause the at least one processor: retrieve the first 3D model; align the second 3D model with the first 3D model; and incorporate the portion of the second 3D model into the first 3D model based on the aligning (See claim 8 for detailed analysis.).
As to claim 29, claim 28 is incorporated and the combination of Foco, Gowda and Li discloses to align the second 3D model with the first 3D model, the instructions cause the at least one processor to align features points associated with the second 3D model with feature points associated with the first 3D model (See claim 9 for detailed analysis.).
As to claim 30, claim 28 is incorporated and the combination of Foco, Gowda and Li discloses to align the second 3D model with the first 3D model, the instructions cause the at least one processor to generate a transformation matrix (See claim 10 for detailed analysis.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/Primary Examiner, Art Unit 2613