DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in the Specification on 12/18/2024. It is noted, however, that applicant has not filed a certified copy of the CN202311808802.0 application as required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Huang (US 20160012646 A1) in view of Kutliroff (US 20170243352 A1).
Regarding to claim 1, Huang discloses a three-dimensional scene reconstruction method (Fig. 2K; 2J ; [0069]: undergo 3D model reconstruction; the system assigns pixel-level color information to co-registered pixels in the depth image and points in the 3D point cloud; [0071]: the system may mesh or combine color 3D point clouds from different sensors together into a 3D color model of the object; 3D model construction and 3D model reconstruction; the system casts the depth data from each sensor into a corresponding 3D point cloud; [0072]: perform 3D model generation or reconstruction; [0075]: surface reconstruction includes vertices, edge and/or surface smoothing across at least a portion of the 3D model or mesh), comprising:
acquiring a color image and a depth image of a three-dimensional scene at the same time ([0063]: one or more sensors 200A-N capture and acquire depth and color information of a surface an object; [0065]: the sensor include one or more color or image sensors 250A-N; Fig. 2J; [0070]: acquire a depth image and a color image; acquire depth and color data of sufficient granularity; color camera captures color image and depth sensor capture depth image as illustrated in Fig. 2J
PNG
media_image1.png
206
308
media_image1.png
Greyscale
; [0092]: the sensor acquires the depth and color images/data from a first perspective, orientation or field of view; [0093]: the depth and color images are acquired simultaneously, i.e. at the same time, to ensure accurate registration between the depth and color images);
determining, based on the depth image, prior depths of object pixel points obtained after depth clustering in a detection frame of each object in the color image ([0056]: each sensor acquires depth and color information from a surface portion of the person or object; [0056]: the sensor acquires depth information but no color information; [0064]: determine depth, i.e. prior depth, or distance to a specific point; a pixel includes or is associated with a value representative of a depth or distance between the sensor and a corresponding point, feature or portion of the object; depth data is represented as (x, y, depth) across an x-y grid in a depth image; [0067]: a depth sensor 220 and a color sensor 250 are coupled or co-registered to acquire, provide, associate and/or store information that corresponds to a same or similar point, spot, feature or region on the object; determine the depth data/image from a surface portion of the object; [0069]: the system maps the depth image with prior depth to a spatial coordinate system in three dimensions that accommodates the spatial relationships between images generated by the plurality of sensors; the system assigns pixel-level color information to co-registered pixels in the depth image and/or points in the 3D point cloud; [0070]: the depth sensor generates a depth value that is associated with at least one color value or pixel corresponding to a same point or feature of an object; acquire depth and color data of sufficient granularity; [0071]: the overlapping regions are used to acquire and to determine depth and color data, e.g., to avoid missing data for 3D model generation; [0077]: the depth of holes and missing sections are different from the surrounding depth; the surrounding depth is prior depth);
performing depth diffusion on the pixel points in the detection frame based on color information of each pixel point in the detection frame and the prior depths of the object pixel points to obtain an actual depth of each pixel point in the detection frame ([0077]: fill and diffuse holes or missing sections in the 3D colored mesh, i.e. performing depth diffusion; geometric PDE image processing performs estimation, extrapolation and/or projections of a missing section, e.g., using surrounding depth and color information; the extrapolated and estimated depths are an actual depth of each pixel; [0085]: fill and diffuse a surface of each outlined feature based on color information and depth/3D information from the 3D color model; [0091]: form a second 3D distribution of colored points representing a second surface portion of the object; [0096]: a processor of the system maps color information from pixels of the first color image to pixels of the first depth image to form a first 3D distribution of colored points representing a first surface portion of the object; the 3D processing module performs noise reduction, artifact removal and smoothing on the depth image/data; the 3D processing module performs similar refinement and/or correction to the color image/data,).
Huang fails to explicitly discloses:
so as to generate a minimum bounding box of the object.
In same field of endeavor, Kutliroff teaches:
so as to generate a minimum bounding box of the object ([0038]: a plane is fitted to each cluster; Fig. 8; [0040]: the detected objects, e.g., lamp 810, table 820, sofa 830, etc., have been marked with boundaries in blue, i.e. minimum bounding box, to represent the result of the segmentation process;
PNG
media_image2.png
222
654
media_image2.png
Greyscale
Fig. 6; [0061]: a lamp 610 is contained within a boundary box 620, i.e. a minimum bounding box;
PNG
media_image3.png
506
632
media_image3.png
Greyscale
; [0063]: the 2D bounding box containing the object is scanned to analyze each pixel within the bounding box; [0099]: calculate a 2-Dimensional (2D) bounding box containing the detected object).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Huang to include so as to generate a minimum bounding box of the object as taught by Kutliroff. The motivation for doing so would have been to generate a 3D reconstruction of the scene comprising points in 3D space corresponding to structures within the scene; to improve the accuracy of the RGB values and the depth values of each individual pixel; improve or refine the computation of the 3D transformation; to improve the estimated boundary of the detected objects by removing, from each object boundary set, duplicate pixels; to calculate a 2-Dimensional (2D) bounding box containing the detected object as taught by Kutliroff in paragraphs [0027-0028], [0042], [0065], and [0099].
Regarding to claim 2, Huang in view of Kutliroff discloses the method according to claim 1, wherein the determining, based on the depth image, prior depths of object pixel points obtained after depth clustering in a detection frame of each object in the color image (same as rejected in claim 1) comprises:
determining a matching pixel point in the color image for each pixel point in the depth image and prior depths of the matching pixel points (Huang; [0067]: a depth sensor 220 and a color sensor 250 are coupled or co-registered to acquire, provide, associate and/or store information that corresponds to a same or similar point, spot, feature or region on the object; [0070]: the system identifies a co-registered pixel or point by matching pixel locations between a depth image and a color image acquired by the same sensor);
performing, for the detection frame of each object in the color image, the depth clustering on the matching pixel points in the detection frame to obtain object pixel points and non-object pixel points in the detection frame (Huang; [0057]: locate and match overlapping 3D regions, i.e. with same depth, of the plurality of surface portions, for stitching into a 3D color model of the person or object; [0077]: the depths of holes and missing sections in the 3D colored mesh are different from the depths of surrounding depth; [0085]: fill a surface of each outlined feature based on color information and depth/3D information from the 3D color model);
determining the prior depths of the object pixel points (Huang; Fig. 2B; [0067]: the data collected from a sensor include depth data/image and color data/image; [0070]: depth and color information for at least some pixels of the image; [0082]: the measurements engine scans or analyzes the 3D model's various perspectives, e.g., front and sides, to generate 2D images; the front face has fixed depths) and setting the prior depths of the non-object pixel points as a fixed value (Kutliroff; Fig. 2; [0028]: regions with darker pixels 210 are relatively far from the camera; Fig. 4; [0029]: rendering 400 of a 3D reconstruction of the scene; the background depth is fixed as illustrated in Fig. 4;
PNG
media_image4.png
408
636
media_image4.png
Greyscale
).
Same motivation of claim 1 is applied here.
Regarding to claim 3, Huang in view of Kutliroff discloses the method according to claim 2, wherein the performing, for the detection frame of each object in the color image, the depth clustering on the matching pixel points in the detection frame to obtain object pixel points in the detection frame (same as rejected in claim 1) comprises:
performing target detection on the color image to obtain the detection frame of each object in the color image (Kutliroff; Fig. 3; [0029]: the 3D points extracted from the depth map are projected and accumulated by depth pixel accumulation circuit 306; [0035]: the object detection/recognition circuit 504 process the RGB image, and the associated depth map as well, along with the 3D reconstruction, to generate a list of any objects of interest recognized in the image; the depth maps are used as additional channels in a deep convolutional network for the purpose of the object recognition and detection; Fig. 8; [0040]: the detected objects, e.g., lamp 810, table 820, sofa 830, etc., have been marked with boundaries in blue, i.e. minimum bounding box, to represent the result of the segmentation process;
PNG
media_image2.png
222
654
media_image2.png
Greyscale
Fig. 6; [0061]: a lamp 610 is contained within a boundary box 620, i.e. a minimum bounding box;
PNG
media_image3.png
506
632
media_image3.png
Greyscale
);
performing depth clustering on the matching pixel points in the detection frame based on prior depth differences between every two adjacent matching pixel points in the detection frame to obtain the object pixel points in the detection frame (Kutliroff; [0038]: calculate the cross product of differences of neighboring depth pixels; the normal vectors are then clustered into groups based on spatial proximity and the values of the vectors; a plane is fitted to each cluster; [0039]: generate clusters of pixels that are not connected to other regions and classify them as individual, segmented objects of the scene; Fig. 8; [0040]: the segmentation process generates, for each object of interest, a collection of 3D points, referred to as an object boundary set, representing the 3D boundary or contour of the object of interest).
Same motivation of claim 1 is applied here.
Regarding to claim 4, Huang in view of Kutliroff discloses the method according to claim 1, wherein the performing, based on color information of each pixel point in the detection frame and the prior depths of the object pixel points, depth diffusion on the pixel points in the detection frame to obtain an actual depth of each pixel point in the detection frame (same as rejected in claim 1) comprises:
determining a corresponding depth diffusion target based on pixel color differences and actual depth differences between every two adjacent pixel points in the detection frame as well as differences between the actual depths and the prior depths of the object pixel points in the detection frame (Kutliroff; [0038]: calculate the cross product of differences of neighboring depth pixels; the normal vectors are then clustered into groups based on spatial proximity and the values of the vectors; a plane is fitted to each cluster; [0039]: generate clusters of pixels that are not connected to other regions and classify them as individual, segmented objects of the scene; Fig. 8; [0040]: the segmentation process generates, for each object of interest, a collection of 3D points, referred to as an object boundary set, representing the 3D boundary or contour of the object of interest);
Huang in view of Kutliroff further discloses:
performing depth diffusion on the pixel points in the detection frame based on the depth diffusion target to obtain the actual depth of each pixel point in the detection frame (Huang; [0077]: fill holes or missing sections in the 3D colored mesh, i.e. performing depth diffusion; geometric PDE image processing performs estimation, extrapolation and/or projections of a missing section, e.g., using surrounding depth and color information; the extrapolated and estimated depths are an actual depth of each pixel; [0085]: fill a surface of each outlined feature based on color information and depth/3D information from the 3D color model).
Regarding to claim 5, Huang in view of Kutliroff discloses the method according to claim 4, wherein the determining a corresponding depth diffusion target based on pixel color differences and actual depth differences between every two adjacent pixel points in the detection frame as well as differences between the actual depths and the prior depths of the object pixel points in the detection frame (same as rejected in claim 4), comprises:
determining a corresponding depth diffusion smoothing term based on the pixel color differences and the actual depth differences between each pixel point and adjacent pixel points of the pixel point in the detection frame (Huang; [0075]: surface smoothing across at least a portion of the 3D model or mesh; [0076]: perform noise-reduction, smoothing, etc., on the 3D color model; [0096]: the 3D processing module performs noise reduction, artifact removal and/or smoothing on the depth image/data);
determining a corresponding depth diffusion regularization term based on the differences between the actual depths and the prior depths of the object pixel points in the detection frame (Huang; [0078]: the post-processing engine performs noise processing and smoothing to reduce or remove noise, artifacts and other irregularities; [0096]: the 3D processing module performs noise reduction, artifact removal and smoothing on the depth image/data);
determining a corresponding depth diffusion target based on the depth diffusion smoothing term and the depth diffusion regularization term (Huang; [0075]: surface smoothing across at least a portion of the 3D model or mesh; [0076]: perform noise-reduction, smoothing, etc., on the 3D color model; the post-processing process includes one or more operations, for example but not limited to morphological processing, geometric partial differential equation (PDE) image processing, and color information correction and interpolation; [0077]: fill and diffuse holes or missing sections in the 3D colored mesh, i.e. performing depth diffusion; geometric PDE image processing performs estimation, extrapolation and/or projections of a missing section, e.g., using surrounding depth and color information; the extrapolated and estimated depths are an actual depth of each pixel; [0085]: fill and diffuse a surface of each outlined feature based on color information and depth/3D information from the 3D color model; [0091]: form a second 3D distribution of colored points representing a second surface portion of the object; [0096]: the 3D processing module performs noise reduction, artifact removal and/or smoothing on the depth image/data).
Regarding to claim 6, Huang in view of Kutliroff discloses the method according to claim 1, wherein the generating a minimum bounding box of the object (same as rejected in claim 1) comprises:
determining point cloud data of the object based on pixel coordinates and the actual depth of each pixel point in the detection frame (Huang; Fig. 2A; [0068]: a 3D point cloud comprise a plurality of points spatially distributed in three dimensions relative to each other and/or relative to a coordinate system; Fig. 2K; [0069]: a plurality of point clouds are generated, which may undergo 3D model reconstruction; the system generates a color 3D point cloud by mapping color data from the corresponding sensor or field of view.);
performing principal component analysis on the point cloud data of the object (Huang; [0069]: the system assigns pixel-level color information to co-registered pixels in the depth image and/or points in the 3D point cloud; [0073-0074]: identify one or more feature points between the point clouds corresponding to the overlapping area to find or identify matching feature points; [0082]: the measurements engine determines or identifies the location and/or boundaries of various object parts or body anatomy; the measurements engine may identify or locate points or regions where limbs connect to the torso), and
determining characteristic values of the object in at least three principal axis directions to generate the minimum bounding box of the object (Huang; Fig. 6; [0061]: the object recognized and labeled as a lamp 610 is contained within a boundary box 620).
Huang in view of Kutliroff further discloses determining characteristic values of the object in at least three principal axis directions to generate the minimum bounding box of the object ([0038]: a plane is fitted to each cluster; Fig. 8; [0040]: the detected objects, e.g., lamp 810, table 820, sofa 830, etc., have been marked with boundaries in blue, i.e. minimum bounding box, to represent the result of the segmentation process;
PNG
media_image2.png
222
654
media_image2.png
Greyscale
; Fig. 6; [0061]: a lamp 610 is contained within a boundary box 620, i.e. a minimum bounding box;
PNG
media_image3.png
506
632
media_image3.png
Greyscale
; [0063]: the 2D bounding box containing the object is scanned to analyze each pixel within the bounding box; [0099]: calculate a 2-Dimensional (2D) bounding box containing the detected object).
Same motivation of claim 1 is applied here.
Regarding to claim 7, Huang in view of Kutliroff discloses the method according to claim 1, wherein the method further comprises:
generating a semantic map of the three-dimensional scene based on the minimum bounding box and semantic information of the detection frame of each object in the three-dimensional scene (Kutliroff; Fig. 8; [0040]: the detected objects, e.g., lamp 810, table 820, sofa 830, etc., have been marked with boundaries in blue, i.e. minimum bounding box, to represent the result of the segmentation process;
PNG
media_image2.png
222
654
media_image2.png
Greyscale
; Fig. 6; [0061]: a label is attached to each of the recognized objects and a 2D bounding box is generated which contains the object; the object recognized and labeled as a lamp 610 is contained within a boundary box 620; [0062]: the matching is based on a comparison of the object label and the 3D location of the center of the 2D bounding box, between the detected object and each of the existing object boundary sets).
Regarding to claim 8, Huang discloses an electronic device (Fig. 1B; [0042]: the computing device 100 is based on any of these processors;
PNG
media_image5.png
566
612
media_image5.png
Greyscale
), comprising:
a processor ([0041]: a central processing unit; Fig. 1B; [0042]: the computing device 100 is based on any of these processors); and
a memory for storing executable instructions of the processor (Fig. 1B; [0043]: main memory unit 122 may be one or more memory chips capable of storing data and allowing any storage location to be directly accessed by the microprocessor 121; [0044]: the main processor 121 communicates directly with cache memory 140 via a secondary bus);
wherein, the processor is configured to execute a three-dimensional scene reconstruction method by executing the executable instructions, the three-dimensional scene reconstruction method (Fig. 1B; [0042]: the computing device 100 is based on any of these processors; [0043]: one or more memory chips capable of storing data and allowing any storage location to be directly accessed by the microprocessor 121; Fig. 1C; [0044]: the main processor 121 communicates directly with cache memory 140 via a secondary bus; Fig. 2K; 2J ; [0069]: undergo 3D model reconstruction; the system assigns pixel-level color information to co-registered pixels in the depth image and points in the 3D point cloud; [0071]: the system may mesh or combine color 3D point clouds from different sensors together into a 3D color model of the object; 3D model construction and 3D model reconstruction; the system casts the depth data from each sensor into a corresponding 3D point cloud; [0072]: perform 3D model generation or reconstruction; [0075]: surface reconstruction includes vertices, edge and/or surface smoothing across at least a portion of the 3D model or mesh) comprising:
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 8.
Regarding to claim 9, Huang in view of Kutliroff discloses the electronic device according to claim 8,
The rest claim limitations are similar to claim limitations recited in claim 2. Therefore, same rational used to reject claim 2 is also used to reject claim 9.
Regarding to claim 10, Huang in view of Kutliroff discloses the electronic device according to claim 9,
The rest claim limitations are similar to claim limitations recited in claim 3. Therefore, same rational used to reject claim 3 is also used to reject claim 10.
Regarding to claim 11, Huang in view of Kutliroff discloses the electronic device according to claim 8,
The rest claim limitations are similar to claim limitations recited in claim 4. Therefore, same rational used to reject claim 4 is also used to reject claim 11.
Regarding to claim 12, Huang in view of Kutliroff discloses the electronic device according to claim 11,
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 12.
Regarding to claim 13, Huang in view of Kutliroff discloses the electronic device according to claim 8,
The rest claim limitations are similar to claim limitations recited in claim 6. Therefore, same rational used to reject claim 6 is also used to reject claim 13.
Regarding to claim 14, Huang in view of Kutliroff discloses the electronic device according to claim 8, wherein the method further comprises:
The rest claim limitations are similar to claim limitations recited in claim 7. Therefore, same rational used to reject claim 7 is also used to reject claim 14.
Regarding to claim 15, Huang discloses a non-transitory computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements a three-dimensional scene reconstruction method (Fig. 1B; [0042]: the computing device 100 is based on any of these processors; [0043]: one or more memory chips capable of storing data and allowing any storage location to be directly accessed by the microprocessor 121; Fig. 1C; [0044]: the main processor 121 communicates directly with cache memory 140 via a secondary bus; Fig. 2K; 2J ; [0069]: undergo 3D model reconstruction; the system assigns pixel-level color information to co-registered pixels in the depth image and/or points in the 3D point cloud; [0071]: the system may mesh or combine color 3D point clouds from different sensors together into a 3D color model of the object; 3D model construction and 3D model reconstruction; the system casts the depth data from each sensor into a corresponding 3D point cloud; [0072]: perform 3D model generation or reconstruction; [0075]: surface reconstruction includes vertices, edge and/or surface smoothing across at least a portion of the 3D model or mesh), comprising:
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 15.
Regarding to claim 16, Huang in view of Kutliroff discloses the non-transitory computer-readable storage medium according to claim 15,
The rest claim limitations are similar to claim limitations recited in claim 2. Therefore, same rational used to reject claim 2 is also used to reject claim 16.
Regarding to claim 17, Huang in view of Kutliroff discloses the non-transitory computer-readable storage medium according to claim 16,
The rest claim limitations are similar to claim limitations recited in claim 3. Therefore, same rational used to reject claim 3 is also used to reject claim 17.
Regarding to claim 18, Huang in view of Kutliroff discloses the non-transitory computer-readable storage medium according to claim 15,
The rest claim limitations are similar to claim limitations recited in claim 4. Therefore, same rational used to reject claim 4 is also used to reject claim 18.
Regarding to claim 19, Huang in view of Kutliroff discloses the non-transitory computer-readable storage medium according to claim 18,
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 19.
Regarding to claim 20, Huang in view of Kutliroff discloses the non-transitory computer-readable storage medium according to claim 15,
The rest claim limitations are similar to claim limitations recited in claim 6. Therefore, same rational used to reject claim 6 is also used to reject claim 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 20230377249 A1; Nepveu; Method And Device For Multi-Camera Hole Filling.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights