Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/10/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to because texts in FIG. 2, FIG. 9, FIG. 11, and FIG. 13 are blurry so it’s not identifiable. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claim 1 is objected to because of the following informalities:
Ciam 1 cites “A computer-implemented method for machine learning a function configured to take as input a 3D point cloud of a real scene and to output localized representations each of a respective object of the real scene and, for each respective object, a class of the respective object among a predetermined set of classes, the method comprising:” which is grammatically incorrect. Therefore, it provides difficulties on understanding clearly. The examiner will interpret it as a computer-implemented method for a function of machine learning configured to take as input a 3D point cloud of real scene and to output localized representations for each of respective object of the real scene and a class of the respective object among a determined set of classes.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 9-13, and 17-20 are rejected under 35 U.S.C. 102(a)(1)/(a)(2)as
being anticipated by Jain et al. (US 20210224516 A1; IDS REF) (hereinafter Jain).
Regarding claim 1, Jain discloses A computer-implemented method for machine learning a function configured to take as input a 3D point cloud of a real scene and to output localized representations each of a respective object of the real scene and, for each respective object, a class of the respective object among a predetermined set of classes, the method comprising: ([0006] Some implementations disclosed herein may achieve various advantages by measuring an object using multiple, class-specific machine learning models (e.g., class specific neural networks). In some implementations, multiple machine learning models are trained to determine different measurements for different object classes. For example, one model may be trained and used to determine measurements for chair type objects (e.g., determining a seat height, arm length, etc.) and another model may be trained and used to determine measurements for TV type objects (e.g., determining a diagonal screen size, greatest TV depth, etc.). Such class-specific measurements may provide more information than the simple length, width, and height of the bounding boxes that may be identified for each object.; [0008] Some implementations involve an exemplary method of providing a measurement for an object within a physical environment. The exemplary method involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with light intensity images. In some implementations, the 3D representation may be associated with 3D semantic data. For example, an algorithm may perform semantic segmentation and labeling of 3D point clouds points.)
obtaining a dataset of 3D point clouds annotated, for each 3D point cloud, with localized representations each of a respective object and, for each respective object, with a class of the respective object among the predetermined set of classes; and
training the function based on the obtained dataset,
wherein the predetermined set of classes include a plurality of semantic classes and a plurality of geometric classes. (FIG. 5;
PNG
media_image1.png
530
467
media_image1.png
Greyscale
[0069] At block 502, the method 500 obtains a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a user captures video while walking around the room to capture images of different parts of the room from multiple perspectives. The depth data can include pixel depth values from a viewpoint and sensor position and orientation data. In some implementations, the depth data is obtained using one or more depth cameras. For example, the one or more depth cameras can acquire depth based on structured light (SL), passive stereo (PS), active stereo (AS), time-of-flight (ToF), and the like. Various techniques may be applied to acquire depth image data to assign each portion (e.g., at a pixel level) of the image. For example, voxel data (e.g., a raster graphic on a 3D grid, with the values of length, width, and depth) may also contain multiple scalar values such as opacity, color, and density. In some implementations, depth data is obtained from sensors or 3D models of the content of an image. Some or all of the content of an image can be based on a real environment, for example, depicting the physical environment 105 around the device 120. Image sensors may capture images of the physical environment 105 for inclusion in the image and depth information about the physical environment 105. In some implementations, a depth sensor on the device 120 (e.g., depth camera 402) determines depth values for voxels that are determined based on images captured by an image sensor on the device 120. The physical environment 105 around the user may be 3D modeled (e.g., 3D point cloud 424) based on one or more values and subsequent depths of objects depicted in subsequent images of the physical environment can be determined based on the model and camera position information (e.g., SLAM information).; [0070] At block 504, the method 500 generates a 3D bounding box corresponding to an object in the physical environment based on the 3D representation. For example, the 3D bounding box may provide location, pose (e.g., orientation and location), and shape of each piece furniture and appliance in the room. Bounding boxes may be refined using RGB data and novel multi-network adjustment techniques (e.g., neural network based machine learning models for fine-tuning). Bounding boxes may be refined using an inflate and cut technique. In some implementations, generating a refined 3D bounding box includes generating a proposed 3D bounding box for an object using a first neural network and generating the refined 3D bounding box by inflating the proposed 3D bounding box based on a bounding box inflation scale (e.g., inflate the bounding box by 10%), identifying features of the object of the inflated proposed bounding box using a second neural network, and refining the bounding box based on the identified features. In some implementations, the first neural network generates the proposed 3D bounding box based on the 3D semantic data associated with the object. In some implementations, the second neural network identifies the features of the object based on the 3D semantic data associated with the object. In some implementations, a third neural network is trained to refine the accuracy of the identified features from the second neural network based on the 3D semantic data associated with the object and light intensity image data obtained during the scanning process, and output a further refined 3D bounding box based on the refined accuracy of the identified features from the second neural network.; [0071] At block 506, the method 500 classifies the object based on the 3D bounding box and the 3D semantic data. For example, providing a class designation or label for each generated 3D bounding box. In some implementations, classifying an object based on the 3D semantic data includes determining, based on the 3D semantic data, a class of the object using an object classification neural network, and classifying the 3D bounding box corresponding to the object based on the determined class of the object. In some implementations, a first class-specific neural network is trained to determine specific points on a first classification of objects (e.g., chairs) that are used for measurements of objects in the first classification. (e.g., arm length and seat height of a chair) In some implementations, a second class-specific neural network is trained to determine specific points on a second classification of objects (e.g., tables) that are used for measurements of objects in the second classification, wherein the second classification of objects is different than the first classification of objects. For example, table height, table top dimensions specific to round or rectangular table tops. The measurements of objects in the second classification of objects are different than the measurements of objects in the first classification of objects. For example, a chair may include more or at least different measurements than a table or a TV.)
Regarding claim 9, Jain discloses The method of claim 1, wherein the function is configured to perform 3D object detection or 3D object segmentation. ([0004] Various implementations disclosed herein include devices, systems, and methods that generate measurements using three-dimensional (3D) representations of a physical environment. The 3D representations of the physical environment may be generated based on sensor data, such as image and depth sensor data. The generation of object detection and object measurements is facilitated in some implementations using semantically-labelled 3D representations of a physical environment. Some implementations perform semantic segmentation and labeling of 3D point clouds of a physical environment.)
Regarding claim 10, Jain discloses The method of claim 1, wherein the 3D point clouds of the dataset are 3D point clouds of indoor scenes and/or obtained via scanning. ([0008] Some implementations involve an exemplary method of providing a measurement for an object within a physical environment. The exemplary method involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with light intensity images. In some implementations, the 3D representation may be associated with 3D semantic data. For example, an algorithm may perform semantic segmentation and labeling of 3D point clouds points.; [0011] In use, a user may scan a room with a device (e.g., a smartphone) and the processes described herein would identify an object (e.g., a chair), and provide specific measurements for the object (e.g., chair height, seat height, based width, etc.). In some implementations, the measurements may be automatically displayed on the user device overlaid or next to the object. In some implementations, the measurements may be provided after some type of user interaction with the identified object. For example, the user may be shown a transparent bounding box surrounding an object, and the user may select or click on the bounding box and the measurements would then be displayed. [0013] Some implementations of this disclosure involve an exemplary method of providing measurement data for an object within a physical environment. The exemplary method first involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images. In some implementations, the 3D representation is associated with 3D semantic data. For example, algorithms may be used for semantic segmentation and labeling of 3D point clouds of indoor scenes.)
Regarding claim 11, Jain discloses The method of claim 1, wherein the function has an architecture including a voxelization layer configured to voxelize the 3D point cloud taken as input and/or a convolutional neural network taking as input the 3D point cloud voxelized by the voxelization layer. ([0065] The 3D representation unit 410 includes an integration unit 420 that is configured with instructions executable by a processor to obtain the image data (e.g., light intensity data 408, depth data 404, etc.) and integrate (e.g., fuse) the image data using one or more known techniques. For example, the image integration unit 420 receives depth image data 404 and intensity image data 408 from the image sources (e.g., light intensity camera 406 and depth camera 402), and integrates the image data and generates 3D data 422. The 3D data 422 can include a dense 3D point cloud 424 (e.g., imperfect depth maps and camera poses for a plurality of image frames around the object) that is sent to the semantic 3D unit 440. The different size grey dots in the 3D point cloud 424 represent different depth values detected within the depth data. For example, image integration unit 420 fuses RGB images from a light intensity camera with a sparse depth map from a depth camera (e.g., time-of-flight sensor) and other sources of physical environment information to output a dense depth point cloud of information. The 3D data 422 can also be voxelized, as represented by the voxelized 3D point cloud 426, where the different shading on each voxel represents a different depth value. [0066] The 3D representation unit 410 further includes a semantic unit 430 that is configured with instructions executable by a processor to obtain the light intensity image data (e.g., light intensity data 408) and semantically segment wall structures (wall, doors, windows, etc.) and object type (e.g., table, teapot, chair, vase, etc.) using one or more known techniques. For example, the semantic unit 430 receives intensity image data 408 from the image sources (e.g., light intensity camera 406), and generates semantic segmentation data 432 (e.g., RGB-S data). For example, the semantic segmentation 434 illustrates a semantically labelled image of the physical environment 105 in FIG. 1. In some implementations, semantic unit 430 uses a machine learning model, where a semantic segmentation model may be configured to identify semantic labels for pixels or voxels of image data. In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), decision tree, support vector machine, Bayesian network, or the like.)
Regarding claim 12, Jain discloses A computer-implemented method of applying a function machine-learnt by machine learning the function configured to take as input a 3D point cloud of a real scene and to output localized representations each of a respective object of the real scene and, for each respective object, a class of the respective object among a predetermined set of classes, comprising: ([0006] Some implementations disclosed herein may achieve various advantages by measuring an object using multiple, class-specific machine learning models (e.g., class specific neural networks). In some implementations, multiple machine learning models are trained to determine different measurements for different object classes. For example, one model may be trained and used to determine measurements for chair type objects (e.g., determining a seat height, arm length, etc.) and another model may be trained and used to determine measurements for TV type objects (e.g., determining a diagonal screen size, greatest TV depth, etc.). Such class-specific measurements may provide more information than the simple length, width, and height of the bounding boxes that may be identified for each object.; [0008] Some implementations involve an exemplary method of providing a measurement for an object within a physical environment. The exemplary method involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with light intensity images. In some implementations, the 3D representation may be associated with 3D semantic data. For example, an algorithm may perform semantic segmentation and labeling of 3D point clouds points.)
obtaining a dataset of 3D point clouds annotated, for each 3D point cloud, with localized representations each of a respective object and, for each respective object, with a class of the respective object among the predetermined set of classes; training the function based on the obtained dataset, wherein the predetermined set of classes comprising a plurality of semantic classes and a plurality of geometric classes; (Figure 5; para [0069]-[0071])
obtaining a 3D point cloud, optionally by scanning a real scene; and applying the function to the obtained 3D point cloud. ([0008] Some implementations involve an exemplary method of providing a measurement for an object within a physical environment. The exemplary method involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with light intensity images. In some implementations, the 3D representation may be associated with 3D semantic data. For example, an algorithm may perform semantic segmentation and labeling of 3D point clouds points.; [0011] In use, a user may scan a room with a device (e.g., a smartphone) and the processes described herein would identify an object (e.g., a chair), and provide specific measurements for the object (e.g., chair height, seat height, based width, etc.). In some implementations, the measurements may be automatically displayed on the user device overlaid or next to the object. In some implementations, the measurements may be provided after some type of user interaction with the identified object. For example, the user may be shown a transparent bounding box surrounding an object, and the user may select or click on the bounding box and the measurements would then be displayed. [0013] Some implementations of this disclosure involve an exemplary method of providing measurement data for an object within a physical environment. The exemplary method first involves obtaining a 3D representation of a physical environment that was generated based on depth data and light intensity image data. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images. In some implementations, the 3D representation is associated with 3D semantic data. For example, algorithms may be used for semantic segmentation and labeling of 3D point clouds of indoor scenes.)
Regarding claim 13, Jain discloses A device comprising:
a processor; and
a non-transitory computer readable storage medium having recorded thereon: ([0021] In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.)
computer program instructions for machine learning a function configured to take as input a 3D point cloud of a real scene and to output localized representations each of a respective object of the real scene and, for each respective object, a class of the respective object among a predetermined set of classes which when executed by the processor, causes the processor to be configured to: (see supra rejection of claim 1; [0021] In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.; [0046] In some implementations, the measurement unit 246 includes a plurality of machine learning units for each specific type of object. For example, a class—1 neural network for chairs, a class—2 neural network for tables, a class—3 neural network for TVs, etc. The plurality of machine learning units can be trained for a different subset of objects such that the measurement unit 246 can provided different types of measurements for each subset of object (e.g., diameter for a round table versus length and width of a rectangular table). The measurement unit 246 is configured with instructions executable by a processor to generate and provide measurement data based on the 3D representation for the identified objects for each subset of neural networks for each class of object using one or more processes further disclosed herein with reference to FIG. 6.)
obtain a dataset of 3D point clouds annotated, for each 3D point cloud, with localized representations each of a respective object and, for each respective object, with a class of the respective object among the predetermined set of classes; and
train the function based on the obtained dataset, wherein the predetermined set of classes including a plurality of semantic classes and a plurality of geometric classes, and/or (see supra rejection of claim 1)
computer program instructions for implementing the function being machine- learnt according to the machine learning which when executed by the processor, causes the processor to be configured to: ([0006] Some implementations disclosed herein may achieve various advantages by measuring an object using multiple, class-specific machine learning models (e.g., class specific neural networks). In some implementations, multiple machine learning models are trained to determine different measurements for different object classes. For example, one model may be trained and used to determine measurements for chair type objects (e.g., determining a seat height, arm length, etc.) and another model may be trained and used to determine measurements for TV type objects (e.g., determining a diagonal screen size, greatest TV depth, etc.). Such class-specific measurements may provide more information than the simple length, width, and height of the bounding boxes that may be identified for each object.)
obtain a 3D point cloud, optionally by scanning a real scene; and apply the function to the obtained 3D point cloud. (see supra rejection of claim 12)
Regarding claim 17, Jain discloses A non-transitory computer readable medium having stored thereon a program having instructions that when executed by a computer causes the computer to implement the computer-implemented method for machine learning according to claim 1. ([0021] In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.)
Regarding claim 18, Jain discloses A non-transitory computer readable medium having stored thereon a program having instructions that when executed by a computer causes the computer to implement the computer-implemented method of applying the function machine- learnt by machine learning the function according to claim 12. ([0021] In accordance with some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions, which, when executed by one or more processors of a device, cause the device to perform or cause performance of any of the methods described herein. In accordance with some implementations, a device includes: one or more processors, a non-transitory memory, and means for performing or causing performance of any of the methods described herein.)
Regarding claims 19 and 20, claims 19 and 20 are the method and device claims of claim 11 and are accordingly rejected under same rationale.
Allowable Subject Matter
Claims 2-8 and 14-16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter over prior art:
The Examiner has further searched the prior art and has not discovered any prior art which fully teaches claims 2, 6 and 14. Regarding claim 2, the closet prior art is Jain and Xue, Fan, et al. ("From LiDAR point cloud towards digital twin city: Clustering city objects based on Gestalt principles." ISPRS Journal of Photogrammetry and Remote Sensing 167 (2020): 418-431.) (hereinafter Xue). Jain discloses obtaining 3D point clouds, localized representations for each 3D point cloud, a class of the respective object among the semantic classes, and computing a value of geometric descriptor for each remaining object of the 3D point cloud. Xue discloses clustering objects based on a matrix (page. 424).
However, one of the prior art cited above, nor any other prior art discovered by Examiner, fully teaches “clustering the remaining objects according to a partitioning of a distribution of the computed values, the plurality of geometric classes corresponding to resulting clusters.”, either singly or in an obvious combination.
Regarding claim 6, the closet prior art is Jain and Zhang, X et al. ("A Two-Stage Approach for Rare Class Segmentation in Large-Scale Urban Point Clouds." The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 43 (2022): 329-334.) (hereinafter Zhang). Jain discloses obtaining 3D point clouds for each 3D point cloud and class of the respective object among the semantic classes. Zhang discloses class segmentation or class identification (2.2 Refined Semantic Segmentation). However, one of the prior art cited above, nor any other prior art discovered by Examiner, fully teaches “for each remaining object, a class of the remaining object among a set of other semantic classes; identifying, among all the objects, at least a portion of the objects of the other semantic classes; and assigning a geometric class to each identified object.”, either singly or in an obvious combination.
Regarding claim 14, claim 14 is device claim of claim 2, and is accordingly rejected under same rationale.
Therefore, claims 2, 6, and 14 are objected to as allowable over the prior art at least due to their respective dependencies.
Claims 3-5, 7-8, and 15-16 are also objected to as allowable over the prior art at least due to their respective dependencies.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hyorim Park whose telephone number is (571)272-3859. The examiner can normally be reached Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Hyorim Park/Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
