DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 9/1/2026 has been entered.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3-10, and 12-20 is/are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Brown et al1 (“Brown”).
Regarding claim 1, Brown teaches one or more processors comprising: processing circuitry to (see Brown, paragraphs 0039-0040 teaching “instructions, when implemented by the processing unit (CPU) 538, cause the processing unit (CPU) 538 to represent objects tracked within 2D video data by 3D modeling as described above with respect to FIGS. 1 through 4”):
compute (note that the following limitations encompass computing any 3D arrangement or computation of modeled environmental surfaces including by creating or updating a new or existing surface model so long as such computing is in some way based on sensor data generated by an ego-machine in the environment and based on 3D object representations which may be a 3D object detected in the environment as represented in sensor data or may be a representation of 3D data corresponding to 3D objects in the sensor data in the sense that such 3D objects may be detected or sensed as corresponding to some pre-determined 3D object that will inform computation of a 3D surface topology of the environment such that for example a depth based representation, bounding volumes, meshes, or images positioned or associated with a 3D space or 3D information are 3D representations, and such representations may be generated from observations or selected from some pre-existing representations; further note that an environment may comprise background and foreground objects and a background or foreground of object in an environment could also separately be understood to be a 3D surfaced topology of the environment; note further that “based on…detected dynamic objects in the environment” is interpreted such that an object does not necessarily have to be acting dynamically or the like but any object which may be considered to be dynamic at any time in any manner may be considered a dynamic object where the manner of detection is not limited and such object must be able to be represented in 3D; finally note that ”sensor data generating using one or more sensors of an ego-machine” is interpreted to refer to any type of sensor data usable in generating the recited information with respect to an ego-machine which is considered to be any machine or device or the like where the sensor bearing apparatus is the apparatus from whose sensing position is being observed), based at least on sensor data generated using one or more sensors of an ego-machine in an environment (see Brown, paragraph 0025 teaching “system for representing objects tracked within 2D video data by 3D modeling” and “at 102 an object is detected and persistently tracked within the field of view environment of calibrated video camera 2D data feed. The video camera is calibrated to recover extrinsic camera parameters that place the camera within a spatial context of a 3D model of the environment of its field of view. Calibrating the camera determines the camera's height, orientation and position in the world plane using standard camera calibration techniques. For example, referring to FIG. 2, the height of a camera (h) from a ground plane and its angle of orientation (A) with respect to the ground plane may be calibrated with respect to coordinates on x, y and z-axis's in a 3D representation of the environment of its field of view through use of a reference object 154 and its known or determined location coordinates” such that here the camera apparatus is the ego-machine, its image sensor generates the 2D video data and the sensed data is of the environment being sensed with respect to the 3D model of the environment and the field of view in relation to the 3D model of the environment; ), and based at least on one or more three-dimensional (3D) representations of one or more detected dynamic objects in the environment (see Brown, paragraphs 0026-0031 teaching “detectors capable of detecting and distinguishing different object types from one another, for example people, vehicles, trucks, buses or other flora and fauna, etc. Movements of the detected objects are tracked across the field of view of the camera” and “a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment” and “Initialization of the 3D model at 106 may comprise recognizing a type or behavior of the detected object and selecting an appropriate model that matches the dynamics of the detected object type and/or behavior from a plurality of models” and “appropriate type of person model may also be selected from multiple person models, selecting a walking person model based on speed and/or location dynamics of the detected object, or a running person model instead based on a faster speed of movement. Some embodiments may distinguish size or other attributes of the detected object to select between different models, for example an automobile 3D model for a vehicle object detected moving along a roadway and having a size dimension larger than a bicycle object and smaller than a truck object” such that the objects such as persons or vehicles are detected dynamic objects and 3D representations of these are the appropriate 3D mesh models for example corresponding to the detected dynamic objects, where the 3D representations are utilized in generating the 3D surface topology of the objects in the environment for placement in the 3D surface model of the environment such that this generates a 3D surface topology of the environment which can include such 3D objects), a 3D surface topology of the environment (see Brown, paragraphs 0025-0027 and figure 3 teaching using such sensor data and the 3D representations of the detected dynamic objects as explained above to generate a 3D surface topology of the environment where “video camera is calibrated to recover extrinsic camera parameters that place the camera within a spatial context of a 3D model of the environment of its field of view” and “the height of a camera (h) from a ground plane and its angle of orientation (A) with respect to the ground plane may be calibrated with respect to coordinates on x, y and z-axis's in a 3D representation of the environment of its field of view” and “At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102” and “one may back-project a pixel (x) 204 image location of the feet 205 of a person object 207 in the 2D image 206 of the environment of its field of view to a 3D location (X) 208 of the feet 209 of a 3D mesh-model 211 selected for use in representing the person 207 within a 3D world model 210 of the field of view environment” and as in paragraphs 0037-0038, “mesh or wireframe-based 3D geometry models may be created through a wireframe transformation of voxel models of objects” and “an Augmented Virtual Environment (AVE) that takes in a 3D model of an environment and augments it with a 3D understanding of objects (people or vehicles, etc.) that are contained within the environment” and “to augment the 3D model of an environment with 3D renderings of people and vehicles, etc., within that environment that are both spatially and temporally accurate” such that here the augmented model forms a surface topology of the environment as populated with the detected object meshes including the modeled ground surface and the surface of the spatially positioned objects where as can be seen in figure 3 for example the 3D world model 210 is a model of the 3D surface topology of the environment; see also ); and
generate a visualization based at least on projecting, onto the one or more 3D representations in the 3D surface topology, at least a portion of the sensor data detected as corresponding to the one or more detected dynamic objects (see Brown, paragraphs 0027-0034 and figure 3, teaching “a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment by using a back-projection of the corresponding 2D image of the tracked object as a function of the centroid and ground-plane intersection data determined at 104. The back-projection is achieved using the calibration between the camera images and the 3D world model. For example, referring to FIG. 3, given the known position (P) 202 of the calibrated camera, one may back-project a pixel (x) 204 image location of the feet 205 of a person object 207 in the 2D image 206 of the environment of its field of view to a 3D location (X) 208 of the feet 209 of a 3D mesh-model 211 selected for use in representing the person 207 within a 3D world model 210 of the field of view environment” and “back-projecting the feet 205 of a person 207 from an image allows us to place the person in a 3D location in the world, and we can then place a 3D model 211 at that 3D location” and “At 110 texture of the 2D object is projected onto the 3D model and the 2D tracks of the object are upgraded to 3D motion to drive the 3D model in dynamic mesh-based 3D” and “at 412 a real-time 3D projection of the object from the camera feed into the 3D environment generates an AVE that is both realistic and immersive by using the motion of the 2D object to drive motion of the 3D volumetric-based object model rendered with the texture of the 2D object projected thereon, and within a 3D context” such that here the captured appearance of the detected object is projected onto the representation of that object within the environmental topology and generates a visualization such as the AVE).
Regarding claim 3, Brown teaches all that is required as applied to claim 1 above and further teaches compute a first 3D surface topology of the environment representing a static portion of the environment (see Brown, paragraphs 0025-0027 teaching “object is detected and persistently tracked within the field of view environment of calibrated video camera 2D data feed. The video camera is calibrated to recover extrinsic camera parameters that place the camera within a spatial context of a 3D model of the environment of its field of view. Calibrating the camera determines the camera's height, orientation and position in the world plane using standard camera calibration techniques. For example, referring to FIG. 2, the height of a camera (h) from a ground plane and its angle of orientation (A) with respect to the ground plane may be calibrated with respect to coordinates on x, y and z-axis's in a 3D representation of the environment of its field of view through use of a reference object 154 and its known or determined location coordinates” such that here “a spatial context of a 3D model of the environment” and “ground plane” and “3D representation of the environment” and such model and topology is of static objects such as the ground plane and for example the static mountain objects as seen in figure 3 as part of the 3D world model 210 where “At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102” means that such 3D model of the environment was determined at 102 prior to inserting dynamic objects into the “3D world model 210 of the field of view” such that this receipt and initialization of the 3D world model computes a first 3D surface topology representing a static portion of the environment such as the ground plane and static mountains which establish the spatial context and 3D world which the dynamic objects are place in); and update the first 3D surface topology based at least on inserting the one or more 3D representations of the one or more detected dynamic objects into the first 3D surface topology (see Brown, paragraphs 0027-0028 teaching “At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment by using a back-projection of the corresponding 2D image of the tracked object as a function of the centroid and ground-plane intersection data determined at 104” and “back-projecting the feet 205 of a person 207 from an image allows us to place the person in a 3D location in the world, and we can then place a 3D model 211 at that 3D location” such that this updates the first 3D surface topology by inserting the 3D representations of the dynamic objects into the first 3D surface topology which can form an environment topology together).
Regarding claim 4, Brown teaches all that is required as applied to claim 1 above and further teaches wherein the one or more detected dynamic objects include at least one rigid object of one or more classes of rigid objects, wherein the processing circuitry is further to generate one or more 3D representations of the at least one rigid object (see Brown, paragraphs 0026-0033 teaching the detected dynamic objects may be rigid objects such as a “embodiments use detectors capable of detecting and distinguishing different object types from one another, for example people, vehicles, trucks, buses or other flora and fauna, etc. Movements of the detected objects are tracked across the field of view of the camera” and “to select between different models, for example an automobile 3D model for a vehicle object detected moving along a roadway and having a size dimension larger than a bicycle object and smaller than a truck object” and “an automobile 3D model for a vehicle object detected moving along a roadway and having a size dimension larger than a bicycle object and smaller than a truck object” and 3D representations of the rigid object may be generated by initializing a 3D model representation for the rigid object and inserting it into the world model as where “At 104 a centroid of the tracked object is effectively localized and the intersection of the centroid with a ground-plane in the images is determined. At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment by using a back-projection of the corresponding 2D image of the tracked object as a function of the centroid and ground-plane intersection data determined at 104” such that this can generate an initial position and depth values for the detected object in the 3D world environment and further generation of representations of the object with respect to its movement is explained below) based at least on warping one or more detected depth values corresponding to the at least one rigid object using one or more detected trajectories corresponding to the at least one rigid object (see Brown, paragraphs 0026-0033 teaching as above that an object including a rigid object is assigned a 3D location including a depth value in the 3D world which is the inferred depth of the detected rigid object where “nonlinear dynamics of a tracked motion path of the object in the 2D image data are represented” and “Initialization of the 3D model at 106 may comprise recognizing a type or behavior of the detected object and selecting an appropriate model that matches the dynamics of the detected object type and/or behavior from a plurality of models” and “At 110 texture of the 2D object is projected onto the 3D model and the 2D tracks of the object are upgraded to 3D motion to drive the 3D model in dynamic mesh-based 3D by learning a weighted combination of the local linear models that minimizes the image re-projection error of model movement” and “given the mixing coefficients one may express either the 2D motion or the 3D motion of the object as a mixture of 2D or 3D base models, and the location of the 3D model in the world (X) may be updated as a function of time (t)” and “At 410 3D mesh models are initialized and used to populate the tracked objects with appropriate 3D models, for example, a walking person 3D model for a object person detected on a causeway, or an automobile 3D model for a vehicle object detected moving along a roadway and having a size dimension larger than a bicycle object and smaller than a truck object, as discussed generally above with respect to FIGS. 1 through 3. Thus, at 412 a real-time 3D projection of the object from the camera feed into the 3D environment generates an AVE that is both realistic and immersive by using the motion of the 2D object to drive motion of the 3D volumetric-based object model rendered with the texture of the 2D object projected thereon, and within a 3D context” such that here this changing of the model coordinates according to their calculated trajectories being determined functions to warp the detected inferred depth values corresponding to the rigid object using the trajectories which determine where to move a previously located point to a new point, thus warping it to the new point).
Regarding claim 5, Brown teaches all that is required as applied to claim 1 above and further teaches the processing circuitry further to fuse, into the 3D surface topology, at least a first 3D representation of at least a first detected dynamic object of the one or more detected dynamic objects generated based at least on: tracking a trajectory of the first detected dynamic object, identifying one or more detected depth values representing the first detected object in a previous time slice, and warping the one or more detected depth values using the trajectory (see Brown, paragraphs 0026-0033 and figure 3 teaching tracking a trajectory of the first detected dynamic object where “Movements of the detected objects are tracked across the field of view of the camera” and “nonlinear dynamics of a tracked motion path of the object in the 2D image data are represented as a collection of different local linear models. Each local linear model encodes the nonlinear dynamics in a small-neighborhood. For example, the 2D object path may be modeled as a local linear combination of several base models” and the detected dynamic object may be “a vehicle object detected moving along a roadway” and depth values for the object are inferred and identified at each point in time where “a centroid of the tracked object is effectively localized and the intersection of the centroid with a ground-plane in the images is determined. At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment by using a back-projection of the corresponding 2D image of the tracked object as a function of the centroid and ground-plane intersection data determined at 104” such that the detected depth values correspond to the depth of the object in the 3D world as positioned when inserted and such depth values continue to be detected as the depth values of the object representation are warped to follow the trajectory and used to apply the proper 2D texture image where “one may express either the 2D motion or the 3D motion of the object as a mixture of 2D or 3D base models, and the location of the 3D model in the world (X) may be updated as a function of time (t)” and “at 412 a real-time 3D projection of the object from the camera feed into the 3D environment generates an AVE that is both realistic and immersive by using the motion of the 2D object to drive motion of the 3D volumetric-based object model rendered with the texture of the 2D object projected thereon, and within a 3D context” such that here the first 3D representation is fused into the 3D surface topology based on detecting the dynamic object to show the dynamic object moving in the 3D environment from a location in a previous time slice to a new location such that its depth values are warped according to the determined moving trajectory).
Regarding claim 6, Brown teaches all that is required as applied to claim 1 above and further teaches wherein the one or more detected dynamic objects include at least one non-rigid object of one or more classes of non-rigid objects (see Brown, paragraphs 0026-0031 teaching detected dynamic objects as non-rigid objects of one or more classes of non-rigid objects such as a person object where “embodiments use detectors capable of detecting and distinguishing different object types from one another, for example people, vehicles, trucks, buses or other flora and fauna, etc. Movements of the detected objects are tracked across the field of view of the camera” and “recognizing a type or behavior of the detected object and selecting an appropriate model that matches the dynamics of the detected object type and/or behavior from a plurality of models. For example, referring again to FIG. 3, recognizing that the detected object 207 is a person may result in selecting a 3D person model 214 from a plurality of other object models, such as vehicles, trucks, dogs, etc”), and wherein the processing circuitry is further to generate one or more 3D representations of the at least one non-rigid object based at least on inserting, for the at least one non-rigid object, a 3D representation of a two-dimensional (2D) surface at a location in the 3D surface topology corresponding to a detected location of the at least one non-rigid object in the environment (see Brown, paragraph 0027 teaching “At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment” such that here a “mesh” model is recognized as corresponding to a model made up of a surface that has been meshed into 2D surfaces and which corresponds to a 3D representation of the object as each 2D face is a part of the 3D representation and for example paragraph 0037 teaches such 2D surfaces as part of a mesh where “mesh or wireframe-based 3D geometry models may be created through a wireframe transformation of voxel models of objects. Segmented training data objects may be used to create textured voxel 3D models through reconstruction processes and systems, for example through space-carving and voxel coloring approaches. Imaged surfaces may be treated as locally planar, allowing back projections of images at hypothetical planar patches. Wireframe transformations may start with a marching cubes algorithm, followed by first order neighborhood smoothing and a surface primitive reduction to obtain a low resolution wireframe model” and such meshes which include a 3D representation of a 2D at a location in the 3D surface topology are inserted in the 3D surface topology corresponding to the detected location of the non-rigid object in the environment as when the object is initialized and placed into the model for example where “at 102 an object is detected and persistently tracked within the field of view environment of calibrated video camera 2D data feed. The video camera is calibrated to recover extrinsic camera parameters that place the camera within a spatial context of a 3D model of the environment of its field of view. Calibrating the camera determines the camera's height, orientation and position in the world plane using standard camera calibration techniques. For example, referring to FIG. 2, the height of a camera (h) from a ground plane and its angle of orientation (A) with respect to the ground plane may be calibrated with respect to coordinates on x, y and z-axis's in a 3D representation of the environment of its field of view through use of a reference object 154 and its known or determined location coordinates” and “a centroid of the tracked object is effectively localized and the intersection of the centroid with a ground-plane in the images is determined. At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment by using a back-projection of the corresponding 2D image of the tracked object as a function of the centroid and ground-plane intersection data determined at 104. The back-projection is achieved using the calibration between the camera images and the 3D world model” and “one may back-project a pixel (x) 204 image location of the feet 205 of a person object 207 in the 2D image 206 of the environment of its field of view to a 3D location (X) 208 of the feet 209 of a 3D mesh-model 211 selected for use in representing the person 207 within a 3D world model 210 of the field of view environment” such that here the mesh of the non-rigid object includes a 2D surface such as the facets of the mesh model and these are inserted at a location in the 3D topology corresponding to the detected location of the non-rigid object in the environment such as the person object).
Regarding claim 7, (Original) Brown teaches all that is required as applied to claim 1 above and further teaches the processing circuitry further to fuse into the 3D surface topology at least a first 3D representation of a flat surface at a location corresponding to a detected centroid of a corresponding one of the one or more detected dynamic objects (see Brown, paragraph 0027 teaching “At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment” such that here a “mesh” model is recognized as corresponding to a model made up of flat surfaces that has been meshed into flat 2D surfaces and which corresponds to a 3D representation of the object as each 2D flat face is a part of the 3D representation and for example paragraph 0037 teaches such 2D surfaces as part of a mesh where “mesh or wireframe-based 3D geometry models may be created through a wireframe transformation of voxel models of objects. Segmented training data objects may be used to create textured voxel 3D models through reconstruction processes and systems, for example through space-carving and voxel coloring approaches. Imaged surfaces may be treated as locally planar, allowing back projections of images at hypothetical planar patches. Wireframe transformations may start with a marching cubes algorithm, followed by first order neighborhood smoothing and a surface primitive reduction to obtain a low resolution wireframe model” where such primitives of a mesh correspond to flat surfaces, and as in paragraphs 0025-0033, the fusion of this represented flat surface including the overall mesh it is a part of with the 3D surface topology occurs when placing the 3D model including the 2D flat surfaces into the 3D world model and this placement location is corresponding to a detected centroid corresponding to the dynamic object where “At 104 a centroid of the tracked object is effectively localized and the intersection of the centroid with a ground-plane in the images is determined. At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment by using a back-projection of the corresponding 2D image of the tracked object as a function of the centroid and ground-plane intersection data determined at 104. The back-projection is achieved using the calibration between the camera images and the 3D world model. For example, referring to FIG. 3, given the known position (P) 202 of the calibrated camera, one may back-project a pixel (x) 204 image location of the feet 205 of a person object 207 in the 2D image 206 of the environment of its field of view to a 3D location (X) 208 of the feet 209 of a 3D mesh-model 211 selected for use in representing the person 207 within a 3D world model 210 of the field of view environment” such that here a centroid relating to the object is used to determine where to fuse the 3D model including the flat surfaces into the 3D world model).
Regarding claim 8, Brown teaches all that is required as applied to claim 1 above and further teaches the processing circuitry further to generate the representation of the sensor data for the one or more 3D representations based at least on wherein the at least a portion of the sensor data comprises a segmented set of the sensor data classified as corresponding to the one or more detected dynamic objects (see Brown, paragraphs 0026-0027 teaching that a set of sensor samples are determined as an object-associated image region associated with a corresponding detected dynamic object where “embodiments use detectors capable of detecting and distinguishing different object types from one another, for example people, vehicles, trucks, buses or other flora and fauna, etc. Movements of the detected objects are tracked across the field of view of the camera” and “At 104 a centroid of the tracked object is effectively localized and the intersection of the centroid with a ground-plane in the images is determined. At 106 a 3D mesh-based volumetric model appropriate for the tracked object is initialized within the spatial context of the 3D model of the environment determined at 102. The initializing at 106 comprises populating the tracked object with the 3D mesh-based volumetric model within the spatial context of the 3D model of the environment by using a back-projection of the corresponding 2D image of the tracked object as a function of the centroid and ground-plane intersection data determined at 104. The back-projection is achieved using the calibration between the camera images and the 3D world model. For example, referring to FIG. 3, given the known position (P) 202 of the calibrated camera, one may back-project a pixel (x) 204 image location of the feet 205 of a person object 207 in the 2D image 206 of the environment of its field of view to a 3D location (X) 208 of the feet 209 of a 3D mesh-model 211 selected for use in representing the person 207 within a 3D world model 210 of the field of view environment” and “back-projecting the feet 205 of a person 207 from an image allows us to place the person in a 3D location in the world, and we can then place a 3D model 211 at that 3D location” and as in paragraph 0032, “At 110 texture of the 2D object is projected onto the 3D model” such that here the images sampled to provide the texture are segmented such that the relevant portions of the image are projected onto the 3D model in the projection step and the determined texture of the 2D object for mapping at the appropriate places corresponds to the segmented set of sensor data classified as corresponding to a moving object).
Regarding claim 9, Brown teaches all that is required as applied to claim 1 above and further teaches wherein the one or more processors are comprised in at least one of (note that in each “system for” limitation below it is not required that the technique be actually used or being used in such a system so long as the system is capable of being used for such a purpose in any manner as such systems do not breathe any new life or meaning into the claim limitations; note that the method and processors can be considered to be comprised in numerous of the systems below, but as the claim is recited in the alternative only one limitation will be specifically addressed):
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content (see Brown, paragraph 0038 teaching “Embodiments of the present invention thus provide for an Augmented Virtual Environment (AVE) that takes in a 3D model of an environment and augments it with a 3D understanding of objects (people or vehicles, etc.) that are contained within the environment. An end-user of the system or process may thus get a complete 3D understanding of the environment being monitored or otherwise represented by 2D video data, enabling the user to place the video feeds within a spatio-temporal context of such an environment. Embodiments leverage computer vision technologies to augment the 3D model of an environment with 3D renderings of people and vehicles, etc., within that environment that are both spatially and temporally accurate, which may enable reduction in cognitive overloads of human monitors while at the same time enhancing the visualization quality of the information being gathered” such that the output as explained above may be considered all of such types of content);
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Regarding claims 10 and 12-18, the instant claims correspond to a “system comprising one or more processors” where the system performs the same functions as recited with regard to the device of “One or more processors” as in claims 1 and 3-9, respectively. Such processors can already be considered a system for performing such functions. In light of this, the limitations of claims 10 and 12-18 correspond to the limitations of claims 1 and 3-9, respectively; thus they are rejected on the same grounds as claims 1 and 3-9, respectively.
Regarding claims 19-20, the instant claims recite a method comprising the same functions as performed by the device as in claims 1 and 9, respectively, such that the device performing such a method as explained as in claim 1 such as the device of Brown also is a device performing such a method as in claims 19-20. In light of this, the limitations of claims 19-20 correspond to the limitations of claims 1 and 9, respectively; thus they are rejected on the same grounds as claims 1 and 9, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Brown in view of Benedek et al2 (“Benedek”).
Regarding claim 2, Brown teaches all that is required as applied to claim 1 above but fails to teach to generate the visualization based at least on projecting, onto the 3D surface topology, a masked subset of the sensor data that omits the at least a portion of the sensor data detected as corresponding to the one or more detected dynamic objects during a first pass of generating the visualization. Rather Brown only teaches specifically projecting onto 3D surface topology the sensor data corresponding to the tracked dynamic object, and the 3D surface topology of static or background elements are supplied without limit on how such 3D world model is created and visualized with respect to sensor data. Thus Brown stands as a base device upon which the claimed invention can be seen as an improvement through such projecting sensor data based on a masked subset of sensor data that omits dynamic objects during a first pass of generating the visualization where for example this could improve the system by realistically texturing not only dynamic objects but background objects as well.
In the same field of endeavor relating to computing 3D surface representations and generating representations of an environment and dynamic objects in an environment using sensor data of an ego-machine in an environment (see Benedek, paragraphs 0082-0086 teaching “generating a three-dimensional model, a so-called combined three-dimensional model. The combined three-dimensional model comprises parts reconstructed on the basis of a point set corresponding to a scene generated by a scanning device, and parts modelled by three-dimensional models generated on the basis of optical recordings” where “a point set corresponding to a scene is generated by means of a scanning device where the scene comprises at least one object shape” and “a combined three-dimensional model is generated on the basis of the background three-dimensional model and the three-dimensional model of at least one substituting object shape substituting each of the at least one object shape subset, respectively” such that here the computing is based on the point set generated by a scanning device sensor of an ego-machine in an environment as the camera may be for example attached to a moving ego-machine where “LIDAR device converts the distance data into a point set corresponding to a scene in a known manner, and the LIDAR device is located in the centre of the point set” such that this device functions egocentrically with respect to the scene as the perspective is from the egocentric perspective of the device capturing the perspective of the scene and not outside cameras imaging the device and scene), Benedek teaches to generate the visualization based at least on projecting, onto the 3D surface topology, a masked subset of the sensor data that omits the at least a portion of the sensor data detected as corresponding to the one or more detected dynamic objects during a first pass of generating the visualization (see Benedek, paragraphs 0087-0091 teaching “a foreground even having dynamic characteristics may be efficiently separated from a principally static background” and “application of the DMRF method, which determines the background point set and the foreground point set according to spatial and time characteristics” and “dividing the point set corresponding to a scene into a foreground point set and a background point set” and an “object shape mask 18, which allows the separation of all points of the background” such that here are taught techniques for masking a subset of sensor data that omits a portion of sensor data detected as corresponding to detected dynamic object during a first pass of a visualization and as in paragraphs 0135-0138, “a background three-dimensional model 56 is assigned to the background subset 48. This assignment may be carried out in a way that the background associated subsets 48 of the point set corresponding to a scene are fitted with a polygon mesh. The three-dimensional model of the ground may be obtained by polygon mesh fitting” and “texturing of the background can be carried out on the basis of photos taken on site” and also for example “the texture of the walls may be prepared on the basis of the photos taken on site” such that here this corresponds to projecting onto a 3D surface topology a masked subset of the sensor data that omits the at least a portion of the sensor data detected as dynamic objects as the background has projected onto its 3D surface topology the texture from the corresponding masked background areas from photos taken on site such that the appropriate image data is projected onto the appropriate 3D model location which thus provides the imagery from the on site cameras as a mask of data and this would omit the sensor data with the detected dynamic objects as they are not present in the background data and images which are mapped to the background model). Thus Benedek teaches the above known techniques applicable to the base system of Brown above.
Therefore it would have been obvious for one of ordinary skill in the art before the effective filing date of the invention to modify Brown to incorporate the techniques of Benedek above as doing so would be no more than application of known techniques to a base device ready for improvement which would yield predictable results and result in an improved system. The predictable result of the combination would be that the background world model of Brown which is already supplied as a texturable model, would have applied sensor data omitting the dynamic objects from the images on site as in Benedek which would generate a reconstructed 3D surface topology similar to the 3D world model in Brown, which would then be supplied as in Brown as the world model, which the dynamic objects could then be inserted into as in Brown. This would result in an improved system as it would ensure that the environmental appearance of the background to which the dynamic object models are added is also realistic and would ensure that no dynamic object models appear in the static background to which the models are added in a subsequent visualization stage thus leading to a generated environment which is more spatially and temporally accurate in line with Brown’s suggested line of improvements (see Brown, paragraph 0038 teaching “get a complete 3D understanding of the environment being monitored or otherwise represented by 2D video data, enabling the user to place the video feeds within a spatio-temporal context of such an environment. Embodiments leverage computer vision technologies to augment the 3D model of an environment with 3D renderings of people and vehicles, etc., within that environment that are both spatially and temporally accurate, which may enable reduction in cognitive overloads of human monitors while at the same time enhancing the visualization quality of the information being gathered” such that here ensuring the background matches sensor data captured at the site accurately and modeling the background separately from the dynamic objects leads to greater spatio-temporal accuracy).
Regarding claim 11, parent claim 10 corresponds to claim 1 above as explained above such that the system of claim 11 also corresponds to the device acting as an equivalent system as in claim 2 such that claim 11 corresponds to claim 2. Note that claim 11 contains a further recitation that the “first pass” is “a first pass of texturizing the detected 3D surface topology” where claim 2 only requires a “first pass of generating the visualization.” This limitation is also taught by Brown as modified as in the rejection of claim 2 where the first pass may be considered a first pass of texturizing the detected 3D surface topology and this is already the manner in which the limitation is interpreted as addressed in the rejection of claim 2. In light of this, the limitations of claim 11 correspond to the limitations of claim 2; thus it is rejected on the same grounds as claim 2.
Response to Arguments
Applicant’s arguments, see “REMARKS”, filed 9/1/2026, with respect to the rejection(s) of claim(s) 1-20 under 35 U.S.C. 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Brown and Brown in view of Benedek as explained above such that claims 1-20 are now rejected on different grounds as explained above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SCOTT E SONNERS whose telephone number is (571)270-7504. The examiner can normally be reached Mon-Friday 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SCOTT E SONNERS/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
1 US PGPUB No. 20120281873
2 US PGPUB No. 20160093101
