DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Germany on 10/24/2023. It is noted, however, that applicant has not filed a certified copy of the EP 23205632 application as required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8 – 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 8 is rendered indefinite for the following reasons:
Claim 8 recites the limitation “the shape” in “fourth, fifth and sixth limitations”. There is insufficient antecedent basis for this limitation in the claim and there is no prior definition for “shape” in the claim.
Claim 8 recites the limitation “the voxel representation of the shape to the shape” in the fifth limitation. It is unclear and confusing to one of the ordinary skill in the art what the limitation means.
Claim 9 is rendered indefinite for the following reasons:
Claim 9 recites the limitation “the shape” in “seventh, eighth and ninth limitations”. There is insufficient antecedent basis for this limitation in the claim and there is no prior definition for “shape” in the claim.
Claim 9 recites the limitation “the voxel representation of the shape to the shape” in the eighth limitation. It is unclear and confusing to one of the ordinary skill in the art what the limitation means.
Claim 10 is rendered indefinite for the following reasons:
Claim 10 recites the limitation “the shape” in “fourth, fifth and sixth limitations”. There is insufficient antecedent basis for this limitation in the claim and there is no prior definition for “shape” in the claim.
Claim 10 recites the limitation “the voxel representation of the shape to the shape” in the fifth limitation. It is unclear and confusing to one of the ordinary skill in the art what the limitation means.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
(g)(1) during the course of an interference conducted under section 135 or section 291, another inventor involved therein establishes, to the extent permitted in section 104, that before such person’s invention thereof the invention was made by such other inventor and not abandoned, suppressed, or concealed, or (2) before such person’s invention thereof, the invention was made in this country by another inventor who had not abandoned, suppressed, or concealed it. In determining priority of invention under this subsection, there shall be considered not only the respective dates of conception and reduction to practice of the invention, but also the reasonable diligence of one who was first to conceive and last to reduce to practice, from a time prior to conception by the other.
A rejection on this statutory basis (35 U.S.C. 102(g) as in force on March 15, 2013) is appropriate in an application or patent that is examined under the first to file provisions of the AIA if it also contains or contained at any time (1) a claim to an invention having an effective filing date as defined in 35 U.S.C. 100(i) that is before March 16, 2013 or (2) a specific reference under 35 U.S.C. 120, 121, or 365(c) to any patent or application that contains or contained at any time such a claim.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 8 – 10 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Huang et al. (US 20240296623 A1; hereafter referred to as Huang).
Regarding Claim 8, Huang taches:
A computer implemented method for operating a computer controlled machine, the computer controlled machine including a robot or a vehicle or a domestic appliance or a power tool or a manufacturing machine or a personal assistant or an access control system (Huang, [0020] The systems and methods described herein may be used by, without limitation, non-autonomous vehicles, semi-autonomous vehicles, piloted and un-piloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater craft, remotely operated vehicles such as drones, and/or other vehicle types”; [0052] “The operations can be performed using various hardware and software as discussed elsewhere herein, as may be performed on one or more processors, including but not limited to central processing units (CPUs)”), the method comprising:
capturing a digital image with a sensor (Huang, Fig. 1, [0026] “FIG. 1A, a scanning device 102, such as may use LIDAR… obtain point data representative of the nearby geometry, as may relate to the shapes and surfaces of objects within a view of the scanning device”);
determining a point cloud representation of an object depending on the digital image, wherein the point cloud representation includes points that represent a view of the object (Huang, [0026] “obtaining three-dimensional (3D) data representative of an environment or surrounding geometry … In FIG. 1A, a scanning device 102, such as may use LIDAR ... such as that illustrated in the image 100 of FIG. 1 to attempt to obtain point data representative of the nearby geometry, as may relate to the shapes and surfaces of objects within a view of the scanning device….receiving data from such a process once performed, the returned scan data can correspond to a set of points, where each point corresponds to a scan location where a given scan…The set of points can include three-dimensional location data indicating a location of each point with respect to a default reference frame, origin, or point of reference. Collectively, the set of points corresponds to a point cloud, or point-based representation of the scanned environment in at least two or three dimensions, such as the point cloud illustrated in the image 150 of FIG. 1B”; Huang, [0170] “obtaining a point cloud representation of a physical object”);
determining a voxel representation of the object depending on the point cloud representation, wherein the voxel representation includes voxels that represent the view (Huang, [0022] “a machine learning model and associated with at voxels in a voxelized three-dimensional space, where individual points are mapped to the appropriate levels of the voxel hierarchy”; Huang, [0035] “ FIG. 2A, a trained machine learning model 204 can infer a density function (or kernel, etc.) for each oriented point in a point cloud 202 (or at least a subset of those points). The points can be mapped to a voxel hierarchy 206”; Huang, [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy”; Huang, [0171]);
mapping the voxel representation with a model including a neural network to a voxel representation of the shape (Huang, Fig. 2B, [0038] “Once the weights or coefficients a have been determined for the basic functions for the various voxels of the hierarchy 252, those coefficients can then be applied to that hierarchy or field and evaluated as illustrated in the pipeline view 250 of FIG. 2B…evaluation is performed using a kernel evaluation module 254, component, process, or operation. The kernel evaluation model can attempt to evaluate this density field… to generate an implicit surface 258, geometric mesh, or other such shape representation”; [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy… A shape generator 422 can then use this data to generate a geometric shape, mesh, or implicit surface, such as though use of a generative neural network, and can store this shape data 424 for use in a subsequent operation, such as for rendering an image including a view of this object”; Huang, [0055] “ multi-dimensional mapping systems can be used to obtain point data for a shape, surface, contour, environment, or other object or location for which surface reconstruction is to be performed”; Huang, [0172]- [0175]);
determining the shape depending on the voxel representation of the shape using an artificial neural network that is configured to map the voxel representation of the shape to the shape (Huang,[0053] “A shape generator 422 can then use this data to generate a geometric shape, mesh, or implicit surface, such as though use of a generative neural network, and can store this shape data 424 for use in a subsequent operation, such as for rendering an image including a view of this object”; Huang, [0055] “ multi-dimensional mapping systems can be used to obtain point data for a shape, surface, contour, environment, or other object or location for which surface reconstruction is to be performed”; Huang, [0172]- [0175]); and
operating the computer-controlled machine depending on the shape (Huang, [0057] “the reconstruction may need to be done in near-real time. This can include, for example, the reconstruction of objects in an environment for purposes of real-time navigation or collision avoidance, such as for autonomous vehicles or unmanned aircraft. Another example use case involves operation of a robotic assembly, where accurate 3D geometric reconstruction may be important for ensuring proper and intended operation of the robotic assembly with respect to one or more objects in a nearby environment”).
Regarding Claim 9, Huang taches:
A device for operating a computer-controlled machine, the device comprising:
at least one processor (Huang, [0060] “multiple processors within one or more computing devices, such as may include a combination of CPUs and GPUs”);
at least one memory (Huang, [0065] “data storage 701 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., Flash memory), or other storage”);
wherein the at least one processor is configured to execute instructions that, when executed by the at least one processor (Huang, Fig. 9, [0085] “a system-on-a-chip (SOC) or some combination thereof 900 formed with a processor that may include execution units to execute an instruction”), causing the device to perform the following steps:
capturing a digital image with a sensor (Huang, Fig. 1, [0026] “FIG. 1A, a scanning device 102, such as may use LIDAR…obtain point data representative of the nearby geometry, as may relate to the shapes and surfaces of objects within a view of the scanning device”);
determining a point cloud representation of an object depending on the digital image, wherein the point cloud representation includes points that represent a view of the object (Huang, [0026] “obtaining three-dimensional (3D) data representative of an environment or surrounding geometry … In FIG. 1A, a scanning device 102, such as may use LIDAR... such as that illustrated in the image 100 of FIG. 1 to attempt to obtain point data representative of the nearby geometry, as may relate to the shapes and surfaces of objects within a view of the scanning device….receiving data from such a process once performed, the returned scan data can correspond to a set of points, where each point corresponds to a scan location where a given scan…The set of points can include three-dimensional location data indicating a location of each point with respect to a default reference frame, origin, or point of reference. Collectively, the set of points corresponds to a point cloud, or point-based representation of the scanned environment in at least two or three dimensions, such as the point cloud illustrated in the image 150 of FIG. 1B”; Huang, [0170] “obtaining a point cloud representation of a physical object”);
determining a voxel representation of the object depending on the point cloud representation, wherein the voxel representation includes voxels that represent the view (Huang, [0022] “a machine learning model and associated with at voxels in a voxelized three-dimensional space, where individual points are mapped to the appropriate levels of the voxel hierarchy”; Huang, [0035] “ FIG. 2A, a trained machine learning model 204 can infer a density function (or kernel, etc.) for each oriented point in a point cloud 202 (or at least a subset of those points). The points can be mapped to a voxel hierarchy 206”; Huang, [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy”; Huang, [0171]);
mapping the voxel representation with a model including a neural network to a voxel representation of the shape (Huang, Fig. 2B, [0038] “Once the weights or coefficients a have been determined for the basic functions for the various voxels of the hierarchy 252, those coefficients can then be applied to that hierarchy or field and evaluated as illustrated in the pipeline view 250 of FIG. 2B…evaluation is performed using a kernel evaluation module 254, component, process, or operation. The kernel evaluation model can attempt to evaluate this density field, to generate an implicit surface 258, geometric mesh, or other such shape representation”; [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy… A shape generator 422 can then use this data to generate a geometric shape, mesh, or implicit surface, such as though use of a generative neural network, and can store this shape data 424 for use in a subsequent operation, such as for rendering an image including a view of this object”; Huang, [0055] “ multi-dimensional mapping systems can be used to obtain point data for a shape, surface, contour, environment, or other object or location for which surface reconstruction is to be performed”; Huang, [0172]- [0175]);
determining the shape depending on the voxel representation of the shape using an artificial neural network that is configured to map the voxel representation of the shape to the shape (Huang,[0053] “A shape generator 422 can then use this data to generate a geometric shape, mesh, or implicit surface, such as though use of a generative neural network, and can store this shape data 424 for use in a subsequent operation, such as for rendering an image including a view of this object”; Huang, [0055] “ multi-dimensional mapping systems can be used to obtain point data for a shape, surface, contour, environment, or other object or location for which surface reconstruction is to be performed”; Huang, [0172]- [0175]); and
operating the computer-controlled machine depending on the shape, wherein the computer-controlled machine is a robot or a vehicle or a domestic appliance or a power tool or a manufacturing machine or a personal assistant or an access control system (Huang, [0057] “the reconstruction may need to be done in near-real time. This can include, for example, the reconstruction of objects in an environment for purposes of real-time navigation or collision avoidance, such as for autonomous vehicles or unmanned aircraft. Another example use case involves operation of a robotic assembly, where accurate 3D geometric reconstruction may be important for ensuring proper and intended operation of the robotic assembly with respect to one or more objects in a nearby environment”);
wherein the at least one memory stores the instructions (Huang, Fig. 9, [0090] “memory 920 may store instruction(s) 919 and/or data 921”).
Regarding Claim 10, Huang taches:
A non-transitory computer-readable medium on which is stored a computer program including instructions for operating a computer-controlled machine, the computer-controlled machine including a robot or a vehicle or a domestic appliance or a power tool or a manufacturing machine or a personal assistant or an access control system, the instructions, when executed by a computer (Huang, [0242] “a non-transitory computer-readable storage medium that excludes transitory signals… code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media having stored thereon executable instructions (or other memory to store executable instructions) that, when executed (i.e., as a result of being executed) by one or more processors of a computer system, cause computer system to perform operations”), causing the computer to perform the following steps:
capturing a digital image with a sensor (Huang, Fig. 1, [0026] “FIG. 1A, a scanning device 102, such as may use LIDAR…obtain point data representative of the nearby geometry, as may relate to the shapes and surfaces of objects within a view of the scanning device”);
determining a point cloud representation of an object depending on the digital image, wherein the point cloud representation includes points that represent a view of the object (Huang, [0026] “obtaining three-dimensional (3D) data representative of an environment or surrounding geometry … In FIG. 1A, a scanning device 102, such as may use LIDAR... such as that illustrated in the image 100 of FIG. 1 to attempt to obtain point data representative of the nearby geometry, as may relate to the shapes and surfaces of objects within a view of the scanning device….receiving data from such a process once performed, the returned scan data can correspond to a set of points, where each point corresponds to a scan location where a given scan…The set of points can include three-dimensional location data indicating a location of each point with respect to a default reference frame, origin, or point of reference. Collectively, the set of points corresponds to a point cloud, or point-based representation of the scanned environment in at least two or three dimensions, such as the point cloud illustrated in the image 150 of FIG. 1B”; Huang, [0170] “obtaining a point cloud representation of a physical object”);
determining a voxel representation of the object depending on the point cloud representation, wherein the voxel representation includes voxels that represent the view (Huang, [0022] “a machine learning model and associated with at voxels in a voxelized three-dimensional space, where individual points are mapped to the appropriate levels of the voxel hierarchy”; Huang, [0035] “ FIG. 2A, a trained machine learning model 204 can infer a density function (or kernel, etc.) for each oriented point in a point cloud 202 (or at least a subset of those points). The points can be mapped to a voxel hierarchy 206”; Huang, [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy”; Huang, [0171]);
mapping the voxel representation with a model including a neural network to a voxel representation of the shape (Huang, Fig. 2B, [0038] “Once the weights or coefficients a have been determined for the basic functions for the various voxels of the hierarchy 252, those coefficients can then be applied to that hierarchy or field and evaluated as illustrated in the pipeline view 250 of FIG. 2B…evaluation is performed using a kernel evaluation module 254, component, process, or operation. The kernel evaluation model can attempt to evaluate this density field, to generate an implicit surface 258, geometric mesh, or other such shape representation”; [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy… A shape generator 422 can then use this data to generate a geometric shape, mesh, or implicit surface, such as though use of a generative neural network, and can store this shape data 424 for use in a subsequent operation, such as for rendering an image including a view of this object”; Huang, [0055] “ multi-dimensional mapping systems can be used to obtain point data for a shape, surface, contour, environment, or other object or location for which surface reconstruction is to be performed”; Huang, [0172]- [0175]);
determining the shape depending on the voxel representation of the shape using an artificial neural network that is configured to map the voxel representation of the shape to the shape (Huang,[0053] “A shape generator 422 can then use this data to generate a geometric shape, mesh, or implicit surface, such as though use of a generative neural network, and can store this shape data 424 for use in a subsequent operation, such as for rendering an image including a view of this object”; Huang, [0055] “ multi-dimensional mapping systems can be used to obtain point data for a shape, surface, contour, environment, or other object or location for which surface reconstruction is to be performed”; Huang, [0172]- [0175]); and
operating the computer-controlled machine depending on the shape (Huang, [0057] “the reconstruction may need to be done in near-real time. This can include, for example, the reconstruction of objects in an environment for purposes of real-time navigation or collision avoidance, such as for autonomous vehicles or unmanned aircraft. Another example use case involves operation of a robotic assembly, where accurate 3D geometric reconstruction may be important for ensuring proper and intended operation of the robotic assembly with respect to one or more objects in a nearby environment”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 – 2 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. (US 20240296623 A1; hereafter referred to as Huang) in view of Zhou et al. (Zhou, L., Du, Y., & Wu, J. (2021). 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 5826-5835); hereafter referred to as Zhou).
Regarding Claim 1, Huang teaches:
A computer implemented method for training a model including a neural network, for determining a shape of an object (Huang, [0022] “embodiments provide for the digital reconstruction of the shapes or geometries of various objects, contours, or environments”), the method comprising the following steps:
determining a first point cloud representation of the object depending on a first digital image, wherein the first point cloud representation includes points that represent a first view of the object (Huang, [0026] “obtaining three-dimensional (3D) data representative of an environment or surrounding geometry … In FIG. 1A, a scanning device 102, such as may use LIDAR .. such as that illustrated in the image 100 of FIG. 1 to attempt to obtain point data representative of the nearby geometry, as may relate to the shapes and surfaces of objects within a view of the scanning device….receiving data from such a process once performed, the returned scan data can correspond to a set of points, where each point corresponds to a scan location where a given scan…The set of points can include three-dimensional location data indicating a location of each point with respect to a default reference frame, origin, or point of reference. Collectively, the set of points corresponds to a point cloud, or point-based representation of the scanned environment in at least two or three dimensions, such as the point cloud illustrated in the image 150 of FIG. 1B”; Huang, [0170] “obtaining a point cloud representation of a physical object”);
determining a second point cloud representation of the object depending on a second digital image, wherein the second point cloud representation includes points that represent a second view of the object (Huang, [0026] “multiple scans (or continuous scanning) may be performed at multiple locations or times in order to obtain point data that provides more information about the geometry of the environment, such as may correspond to views of an object from different sides or perspectives to allow for a more accurate reconstruction of the object as a whole. When performing such a scanning process, or receiving data from such a process once performed, the returned scan data can correspond to a set of points, where each point corresponds to a scan location … The set of points can include three-dimensional location data indicating a location of each point with respect to a default reference frame, origin, or point of reference. Collectively, the set of points corresponds to a point cloud, or point-based representation of the scanned environment in at least two or three dimensions, such as the point cloud illustrated in the image 150 of FIG. 1B.”; Huang, [0170]);
determining a first voxel representation of the object depending on the first point cloud representation, wherein the first voxel representation includes voxels that represent the first view (Huang, [0022] “a machine learning model and associated with at voxels in a voxelized three-dimensional space, where individual points are mapped to the appropriate levels of the voxel hierarchy”; Huang, [0035] “ FIG. 2A, a trained machine learning model 204 can infer a density function (or kernel, etc.) for each oriented point in a point cloud 202 (or at least a subset of those points). The points can be mapped to a voxel hierarchy 206”; Huang, [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy”; Huang, [0171]);
mapping the first voxel representation using the model to a voxel representation of the shape (Huang, Fig. 2B, [0038] “Once the weights or coefficients a have been determined for the basis functions for the various voxels of the hierarchy 252, those coefficients can then be applied to that hierarchy or field and evaluated as illustrated in the pipeline view 250 of FIG. 2B…evaluation is performed,… to generate an implicit surface 258, geometric mesh, or other such shape representation”; [0053] “A voxel environment manager 414 can generate or maintain a voxel hierarchy including a number of levels and sizes of voxels selected for a reconstruction task, and can generate and maintain mapping data 416, which maps the points of the point cloud data 412 to the voxels of the voxel hierarchy… A shape generator 422 can then use this data to generate a geometric shape, mesh, or implicit surface, such as though use of a generative neural network, and can store this shape data 424 for use in a subsequent operation, such as for rendering an image including a view of this object”; Huang, [0055] “ multi-dimensional mapping systems can be used to obtain point data for a shape, surface, contour, environment, or other object or location for which surface reconstruction is to be performed”; Huang, [0172]- [0175]);
While Huang teaches providing ground-truth data (Huang, [0162] “a customer dataset 1506 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices at a facility) may be used to perform model training (which may include, without limitation, transfer learning) on initial model 1504 to generate refined model 1512. In at least one embodiment, ground truth data corresponding to customer dataset 1506 may be generated by training system 1304”), it does not explicitly teach:
providing a ground truth for training the model: (i) depending on the first point cloud representation and the second point cloud representation or (ii) depending on the first voxel representation and a second voxel representation, wherein the second voxel representation of the object is determined depending on the second point cloud representation, wherein the second voxel representation includes voxels that represent the second view, wherein the ground truth is a voxel representation that includes the voxels of the first voxel representation and the second voxel representation.
In the same field of endeavor, Zhou teaches:
providing a ground truth for training the model: (i) depending on the first point cloud representation and the second point cloud representation or (ii) depending on the first voxel representation and a second voxel representation, wherein the second voxel representation of the object is determined depending on the second point cloud representation, wherein the second voxel representation includes voxels that represent the second view, wherein the ground truth is a voxel representation that includes the voxels of the first voxel representation and the second voxel representation (Zhou, page 5830, 4.1 Shape Completion, Data “For shape completion, we use the benchmark provided by GenRe [49], which contains renderings of each shape in ShapeNet from 20 random views. We sample 200 points as our partial point clouds obtained from the provided depth images, and we evaluate shape completion on all 20 partial shapes per ground-truth sample”; Zhou, page 5827, col. 2, Point-voxel representation, “point-voxel CNN, which proposes to voxelize the point clouds for 3D convolution”; Zhou, page 5828, Col. 1, Energy-based models and denoising diffusion models, “uses a different hybrid, point-voxel representation for processing shapes”).
Huang and Zhou are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Huang with the invention of Zhou to make the invention that provides ground-truth for training the model based on point cloud representations for random views; doing so can efficiently generate high quality shapes of the objects (Zhou, 5. Conclusion); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 2, Huang in view of Zhou teaches the method according to claim 1, wherein the model is a diffusion model that is configured to remove noise from a noisy input of the diffusion model, wherein the diffusion model is configured to output the voxel representation of the shape, wherein the noisy input has a plurality of input elements, wherein the noisy input includes elements that represent the first voxel representation of the object, and elements that represent noise that is randomly sampled from a distribution, wherein the elements that represent the first voxel representation are undisturbed and include no additional noise (Zhou, page 5828, 3. Point-Voxel Diffusion, “we introduce Point-Voxel Diffusion (PVD), a denoising diffusion probabilistic model for 3D point clouds..., followed by the training objective for shape generation”; Zhou, page 5828, 3.1 Formulation, “The denoising diffusion probabilistic model is a generative model where generation is modeled as a denoising process. Starting from Gaussian noise, denoising is performed until a sharp shape is formed. In particular, the denoising process produces a series of shape variables with decreasing levels of noise… To learn our generative model, we define a ground truth diffusion distribution q(x0:T) (defined by gradually adding Gaussian noise to the ground truth shape), and learn a diffusion model pθ(x0:T), which aims to invert the noise corruption process.”; Zhou, page 5828, Col. 1, “Point clouds can then be generated by progressively sampling… to the gradual denoising of a shape from noise”).
Claims 3 – 5 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. (US 20240296623 A1; hereafter referred to as Huang) in view of Zhou et al. (Zhou, L., Du, Y., & Wu, J. (2021). 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 5826-5835); hereafter referred to as Zhou) further in view of Ansari et al. (US 20240371015 A1; hereafter referred to as Ansari).
Regarding Claim 3, Huang in view of Zhou teaches the method of claim 1, wherein the method further comprises:
determining a depth image of the object depending on the voxel representation of the shape of the object (Zhou, Fig. 4, depth images, page 5830, 4.2 Shape Completion, Data “We sample 200 points as our partial point clouds obtained from the provided depth images”; Figure5: Column1shows input depth image and ground-truth point clouds”);
While Huang in view of Zhou teaches providing pseudo ground truth data (Huang, [0141] “ground truth data may be synthetically produced (e.g., generated from computer models or renderings)”), it fails to explicitly teach:
providing a pseudo ground truth depth image; and
training the model depending on a difference between the depth image and the pseudo ground truth depth image.
In the same field of endeavor, Ansari teaches:
providing a pseudo ground truth depth image (Ansari, [0025] “generating synthetic image data and corresponding ground-truth depth information using three-dimensional modeling and rendering”; Ansari, [0031] “the synthetic training data 105 may include ground-truth depth information”); and
training the model depending on a difference between the depth image and the pseudo ground truth depth image (Ansari, [0034] “the training system 115 may use a variety of techniques to leverage synthetic training data 105 and real training data 110 while accounting for differences in domain distributions to generate depth models 120 that exhibit improved prediction accuracy”; Ansari, [0083] “the photometric loss may be generated based on the difference between the original input image and the recreated/warped image, and the depth gradient loss may be generated based on the smoothness of the generated depth output 203. If ground truth is available (e.g., the input image was synthetic), then the depth supervision loss may be generated based on the difference between the depth output 206 and the ground-truth depth information”)
Huang, Zhou and Ansari are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Huang in view of Zhou with the invention of Ansari to make the invention that trains the model depending on a difference between the depth image and the pseudo ground truth depth image; doing so can yield predictable results of high-performance models for depth estimation (Ansari, [0003]); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 4, Huang in view of Zhou further in view of Ansari teaches the method of claim 3, wherein the providing of the pseudo ground truth depth image includes determining the pseudo ground truth depth image of the object depending on the first digital image including mapping the first digital image with a first artificial neural network to the pseudo ground truth depth image of the object, wherein the first artificial neural network is configured to map the first digital image to the pseudo ground truth depth image (Ansari, [0025] “generating synthetic image data and corresponding ground-truth depth information using three-dimensional modeling and rendering”; [0028] FIG. 1 depicts an example workflow 100 for training depth models using real and synthetic training data, and using trained depth models to generate depth maps for input images; Ansar, [0029] “synthetic training data 105 and real training data 110 are accessed by a training system 115 and used to train a depth model 120”; Ansari, [0031] “the synthetic training data 105 may include ground-truth depth information”; Ansari, [0102] “to generate an inference (e.g., a depth output or depth map), a real image (captured in the physical environment) can be first processed by the target-to-source generator 420B to generate a pseudo-source image (e.g., an equivalent or approximate fake synthetic image). This pseudo-source image can then be processed using the trained depth model (which was trained primarily or entirely on synthetic data) to generate an output depth map”).
Regarding Claim 5, Huang in view of Zhou further in view of Ansari teaches the method of claim 3, wherein the providing of the pseudo ground truth depth image includes providing a training data-point including the first digital image, and the pseudo ground truth depth image (Ansari, [0102] “to generate an inference (e.g., a depth output or depth map), a real image (captured in the physical environment) can be first processed by the target-to-source generator 420B to generate a pseudo-source image (e.g., an equivalent or approximate fake synthetic image). This pseudo-source image can then be processed using the trained depth model (which was trained primarily or entirely on synthetic data) to generate an output depth map”; Ansari. [0118] “training the set of machine learning models comprises: (i) training a generator model, based on the data from the source domain and the data from the target domain, to generate pseudo-synthetic data when the data from the target domain is used as input to the generator model, and (ii) training a depth model to generate depth outputs, wherein the depth model is trained based on the data from the source domain and is not trained on the data from the target domain”).
Claims 6 – 7 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. (US 20240296623 A1; hereafter referred to as Huang) in view of Zhou et al. (Zhou, L., Du, Y., & Wu, J. (2021). 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 5826-5835); hereafter referred to as Zhou) further in view of Shin et al. (Shin, D., Fowlkes, C. C., & Hoiem, D. (2018). Pixels, voxels, and views: A study of shape representations for single view 3d object shape prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3061-3069)”; hereafter referred to as Shin).
Regarding Claim 6, Huang in view of Zhou teaches the method of claim 1, but fails to explicitly teach:
determining a silhouette image of the object depending on the voxel representation of the shape of the object;
providing a ground truth silhouette image; and
training the model depending on a difference between the silhouette image and the ground truth silhouette image.
In the same field of endeavor Shin teaches:
determining a silhouette image of the object depending on the voxel representation of the shape of the object (Shin, Page 3063, Col. 1para 3, “Our proposed method predicts 2.5D surfaces (depth image and object silhouette) of the object from a set of fixed viewpoints evenly spaced over the viewing sphere”; Shin, Page 3064, “Figure 3. Network architecture: Encoders Ed, Es, Eh learn view-specific shape features h extracted from the input depth and silhouette”);
providing a ground truth silhouette image (Sin, page 3065, Section 5. Experiments, “The k-th ground truth silhouette has associated front and back depth images. Each image is uniformly scaled to fit within 128x128 pixels”); and
training the model depending on a difference between the silhouette image and the ground truth silhouette image (Shin, page 3065, Section 5. Experiment, “Training examples for the voxel prediction network consist of input-output pairs (xd,xs) → V , where V is a grid of ground truth voxels (size 48x48x48 for the input depth experiments, and 32x32x32 for the input RGB experiments”).
Huang, Zhou and Shin are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Huang in view of Zhou with the invention of Shin to make the invention that provides ground truth silhouette image; determining a silhouette image of the object depending on the voxel representation of the shape of the object; and train the model depending on a difference between the silhouette image and the ground truth silhouette image; doing so can provide better learning and prediction of thin object parts and novel objects with shape details (Shin, 6. Discussions, 7. Comclusions); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 7, Huang in view of Zhou further in view of Shin teaches the method of claim 6, wherein the providing of the ground truth silhouette image includes providing a training data-point including the first digital image, and the ground truth silhouette image (Shin, page 3065, Section 5. Experiment, A single training example for the f ,d(k) multi-surface network is the input-output pair (xd,xs) → {(s(k), d(k) b )}k=0..9 where (xd,xs) is the input depth image and segmentation, and the orthographic depth im ages (s(k),d(k) f ,d(k) b )serve as the output ground truth. The k-th ground truth silhouette has associated front and back depth images. Each image is uniformly scaled to fit within 128x128 pixels. Training examples for the voxel prediction network consist of input-output pairs (xd,xs) → V , where V is a grid of ground truth voxels (size 48x48x48 for the input depth experiments, and 32x32x32 for the input RGB experiments”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20210103776 A1 THREE-DIMENSION (3D) ASSISTED PERSONALIZED HOME OBJECT DETECTION: The invention relates to technology for object detection in which a vision system receives training datasets including a set of two-dimensional (2D) images of the object from multiple views. A set of 3D models is reconstructed from the set of 2D images based on salient points of the object selected during reconstruction to generate one or more salient 3D models of the object that is an aggregation of the salient points of the object in the set of 3D models. A set of training 2D-3D correspondence data are generated between the set of 2D images in a first training dataset of the training datasets and the salient 3D model of the object generated using the first training dataset. A deep neural network is trained using the set of training 2D-3D correspondence data generated using the first training dataset for object detection and segmentation.
US 20210065430 A1 3D REPRESENTATION RECONSTRUCTION FROM IMAGES USING VOLUMIC PROBABILITY DATA: To generate 3D representation of a scene volume, the present invention combines the 3D skeleton approach and the shape from silhouette approach. The present invention efficiently works on complex scenes like sport events with multiple players in a stadium, with an ability to detect a wide number of interoperating 3D objects like multiple players.
US 20210012555 A1 PROCESSING POINT CLOUDS USING DYNAMIC VOXELIZATION: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data using dynamic voxelization. When deployed within an on-board system of a vehicle, processing the point cloud data using dynamic voxelization can be used to make autonomous driving decisions for the vehicle with enhanced accuracy, for example by combining representations of point cloud data characterizing a scene from multiple views of the scene.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/VAISALI RAO KOPPOLU/Patent Examiner of Art Unit 2664
Prosecution Insights