DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noh et al. (Noh, Sangjun, et al. "Learning to place unseen objects stably using a large-scale simulation." IEEE Robotics and Automation Letters 9.3 (2024): 3005-3012.; Citation given from arXiv:2303.08387, published on Mon, 11 Sep 2023 06:32:02 UTC. This reference is part of IDS. Hereinafter Noh et al.) in view of Walter et al. (W. Wohlkinger, A. Aldoma, R. B. Rusu, and M. Vincze, “3dnet: Large-scale object class recognition from cad models,” in 2012 IEEE international conference on robotics and automation. IEEE, 2012, pp. 5384–5391. Hereinafter Walter).
[Examiner’s Note – Citations are provided from Noh, unless expressly indicated otherwise]
Regarding claim 1, Nah discloses a stable plane estimation method (A) for placing an object in a stable posture, the stable plane estimation (Abstract; Fig. 1(d); §III-C; Fig. 3. Noh states that UOP-Net detects the most stable plane from partial point clouds and enables robots to place objects stably. Fig. 1(d) specifically shows direct stable-plane detection for unseen objects) method comprising:
acquiring (§IV-A, “Simulation environment setting,” p. 3; Fig. 2. Noh uses 17,408 3D object models obtained from 3DNet [26], ShapeNet [27], and YCB [28] in the physics simulation. Fig. 2 identifies the dataset as containing 3D object models (17.4K));
disposing a flat plate model (§IV-A, pp. 2–3; Fig. 2. Noh conducts the simulation using a table and states that 64 table models were built in the simulation environment to facilitate annotation.), disposing the target object above the flat plate model (§IV-A, p. 2. After generating 512 orientations, Noh states that the object was placed on a table with a random pose along the normal direction of the table, before being dropped onto the table.), and droppingthe target object disposed above the flat plate model toward the flat plate model (Fig. 2 caption; §IV-A, p. 3. Fig. 2 states that the dataset is generated by dropping each object on a table in 512 different configurations. The text likewise says the object is dropped on the table after the orientation/pose is established.); and
sampling, as a stable plane, an area in contact with the flat plate model in the target object (Fig. 2; §IV-A, p. 3. Noh clusters sampled stable poses to identify stable planes. The z-axis represents the normal vector at the contact points of a stable plane with a horizontal surface. Noh then masks the bottom 5% of object height along the table-surface normal to identify the areas capable of supporting object stability) when the target object dropped toward the flat plate model stops (§III-B, p. 3; §IV-A, p. 2; Eqs. (1)–(2). Noh defines instability U from object movement over simulation time and expressly states that threshold ε indicates that the object has stopped. Stable poses satisfy Uᵢ < ε₁.), to generate training data for an object controlled to be placed on a specific support surface (Fig. 2; §IV-A; Fig. 3. UOP-Sim contains 17,408 object models and 69,027 stable-plane annotations. Fig. 3 expressly states UOP-Net is trained on the UOP-Sim dataset.).
Noh is not found disclosing expressly that acquiring a mesh for the target object. However, He discloses using 3D object models from three benchmark datasets, e.g., as disclosed by Walter et al. (citation # 26 in Noh). Walter converts the 3D models of a new object to meshed PLY format (see §IV.D Extensibility, steps 1-2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to implement the 3D models as disclosed in Noh using a PLY formatted meshed polygonal objects as disclosed by Walter, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I). Furthermore, Noh expressly discloses using object models from Walter.
Regarding claim 3, Noh in view of Walter discloses the stable plane estimation method of claim 1, wherein the object includes at least one untrained object as a target of manipulation placed on a plane (Noh expressly defines the task as “Unseen Object Placement (UOP)”. §III-A, Problem Statement, concerns placing an unseen object stably. Fig. 1(d) identifies UOP-Net as detecting stable planes for unseen objects. §IV-C, Experiments, evaluates generalization using objects not used for training. Abstract; Fig. 1; §III-A. Noh's task is robotic manipulation/placement: the robot detects a stable plane of the unseen object and manipulates the object into the corresponding placement configuration.).
Regarding claim 4, Noh in view of Walter discloses the stable plane estimation method of claim 1, wherein the dropping of the mesh toward the flat plate model includes changing a posture of the mesh disposed above the flat plate model and repeatedly dropping the point cloud toward the flat plate model by a predetermined number of times (§IV-A, Stable plane annotation. Noh expressly generates 512 orientations by dividing roll, pitch, and yaw into eight intervals, thereby exploring a wide range of object poses. The text explains that after generating the orientations, the object is placed relative to the table normal and dropped onto the table), and
the sampling the plurality of points as the stable plane includes sampling the stable plane for each of the plurality of areas dropped repeatedly by a predetermined number of times (Fig. 2; §IV-A. Noh expressly describes sampling stable planes. Stable poses satisfying Uᵢ < ε₁ are recorded and clustered to identify stable planes. Noh records all poses in which the object remained stable from the 512 configurations, clusters the sampled poses, and identifies the corresponding stable planes. UOP-Sim ultimately contains 69,027 stable-plane annotations for 17,408 objects).
Regarding claim 5, Noh in view of Walter discloses the stable plane estimation method of claim 1, further comprising:
labeling, as label data for the mesh, the sampled stable plane to the mesh (§III-B; §IV-A; Fig. 2. Noh expressly defines dataset D = {(O,S)ₙ}, where O is the object model and S is the corresponding stable-plane annotation. After stable-plane inspection, Noh states that this procedure permits it to “label stable planes.” UOP-Sim contains 17,408 3D object models and 69,027 stable-plane annotations); and
generating the training data for an artificial neural network that outputs the stable plane when the point cloud corresponding to the target object is input, using the mesh and the label data (§IV; Fig. 3; §IV-B. Noh expressly introduces UOP-Sim as the training dataset and UOP-Net neural network. Fig. 3 states that UOP-Net is trained on the UOP-Sim dataset.
§III-B expressly defines dataset D={(O,S)ₙ}, pairing each object model O with corresponding stable-plane annotations S. §V-A, Training Details further says UOP-Net was trained using partial point clouds sampled from UOP-Sim.
§III-B expressly defines the deep-learning function F:X→s, where X is the point cloud input and s is the most stable plane output. §IV-B describes UOP-Net's DGCNN-based architecture and stable-plane prediction.
§III-A–C; Fig. 3. The robot captures an object using a single-view RGB-D camera; the resulting partial point cloud is fed to the model. Fig. 3 expressly shows UOP-Net taking a partial point cloud to predict the stable plane. ).
Regarding claim 6, Noh in view of Walter discloses a stable plane estimation system for placing an object in a stable posture (Throughout the experiments, we instructed
the robotic system to execute object placements onto the surfaces predicted by the UOP-Net model. – §V.C, 2nd to the last page, Col. 1, Evaluation Metrics), the stable plane estimation system comprising:
an input unit configured to acquire a mesh for a target object; and a control unit configured to dispose a flat plate model, dispose the mesh above the flat plate model, and drop the mesh disposed above the flat plate model toward the flat plate model, wherein the control unit samples, as a stable plane, an area in contact with the flat plate model in the mesh when the mesh dropped toward the flat plate model stops, to generate training data for an object controlled to be placed on a specific support surface (see substantively similar claim 1 rejection above. Control unit is inherent where the simulation is run).
Claim(s) 8-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noh in view of Walter and further in view of Guler et al. (US 20240346763 A1, hereinafter Guler).
Regarding claim 8, Noh discloses a stable plane estimation method for placing an object in a stable posture, the stable plane estimation method (Abstract; Fig. 1(d); §III-C; Fig. 3. Noh addresses “Unseen Object Placement (UOP)” and directly detects the most stable plane for stable object placement. Fig. 1(d) shows UOP-Net detecting stable planes from partial observations) comprising:
acquiring a point cloud for an object (§III-A, ¶ under Assumptions; §III-B. The robot captures the object using a single-view RGB-D camera, and the resulting partial point cloud is used. Noh defines point cloud X ∈ R^(N×3) as point clouds obtained by capturing the manipulation scene);
inputting the point cloud into an artificial neural network trained using a pre-constructed training data set (Fig. 3; §§III-B, IV. Noh defines deep-learning model F:X→s and states that UOP-Net is trained on the UOP-Sim dataset and receives a partial point cloud to predict the stable plane. UOP-Sim contains 17,408 3D object models and 69,027 stable-plane annotations); and
acquiring a stable plane in which the object is placed on a support surface in a stable posture from the artificial neural network (§III-B; Fig. 3; §IV-B. Noh defines F:X→s, where s is the most stable-plane output. Fig. 3 states that the estimated stable plane is used for object placement. §IV-B selects the plane having the highest stability score and determines rotation R from its normal relative to gravity/negative table-surface normal.),
wherein the trained artificial neural network acquires a learning Fig. 2; §§III-B, IV-A. UOP-Sim contains 3D object models O. Noh uses 17,408 3D object models from 3DNet, ShapeNet, and YCB in its physics simulation), drops the learning target object onto a plane (Fig. 2; §IV-A, Stable plane annotation. Noh generates 512 orientations, places each object on a table with a random pose along the table-normal direction, and expressly states: “We dropped the object on the table” and recorded the stable poses satisfying Uᵢ < ε₁. Fig. 2 likewise states that each object is dropped on a table in 512 different configurations.), samples an area in contact with the plane in the learning target object as a learning stable plane (Fig. 2; §IV-A. After the drop simulations, Noh clusters sampled stable poses to identify stable planes. The z-axis represents the normal vector at the contact points of a stable plane with a horizontal surface. Noh then masks the bottom 5% of height along the table-surface normal to indicate areas capable of supporting object stability.), and is trained using the training data set including the learning target object and the learning stable plane (§III-B; Figs. 2–3; §IV-A; §V-A. Noh defines D={(O,S)ₙ}, where O are object models and S their corresponding stable-plane annotations. UOP-Sim contains 17,408 object models + 69,027 stable-plane annotations. Fig. 3 states UOP-Net is trained on UOP-Sim; §V-A states UOP-Net was trained using partial point clouds sampled from UOP-Sim).
Noh is not found disclosing expressly that acquiring a mesh for the target object.
However, He discloses using 3D object models from three benchmark datasets, e.g., as disclosed by Walter et al. (citation # 26 in Noh). Walter converts the 3D models of a new object to meshed PLY format (see §IV.D Extensibility, steps 1-2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to implement the 3D models as disclosed in Noh using a PLY formatted meshed polygonal objects as disclosed by Walter, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I). Furthermore, Noh expressly discloses using object models from Walter.
Although as it appears Noh in view of Walter discloses all the limitation of claim 8, one might argue that limitation of “training using the training data set including the learning mesh” is not clearly disclosed in the combination, since §V.A of Noh says the network itself is trained using partial point clouds sampled from UOP -Sim. Such conclusion obviously can be deduced from a narrower reading on the cited references.
Nevertheless, Guler discloses that during training of the mesh generation system 500, the image access component 510 accesses a database of training data to retrieve images depicting real-world objects and their corresponding ground truth 3D scans to train one or more machine learning models of the mesh generation system 500 [¶0112].
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Noh in view of Walter such that training using the training data set including the learning mesh according to the teaching of Guler, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I). Such combination would also enhances the realism of the object rendering the an AR/VR environment (¶0003).
Regarding claim 9, Noh in view of Walter and Guler discloses a stable plane estimation system for placing an object in a stable posture (Throughout the experiments, we instructed the robotic system to execute object placements onto the surfaces predicted by the UOP-Net model. – §V.C, 2nd to the last page, Col. 1, Evaluation Metrics), the stable plane estimation system comprising:
an input unit configured to acquire a point cloud for an object; and a control unit configured input the point cloud into an artificial neural network trained using a pre-constructed training data set and acquire a stable plane in which the object is placed on a support surface in a stable posture from the artificial neural network, wherein the trained artificial neural network
acquires a learning mesh for a target object, drops the learning mesh onto a plane, samples an area in contact with the plane in the learning mesh as a learning stable plane, and is trained using the training data set including the learning mesh and the learning stable plane (see substantively similar claim 1 rejection above. Control unit is inherent where the simulation is run).
Regarding claim 10, Noh in view of Walter and Guler discloses
acquiring a point cloud for an object; inputting the point cloud into an artificial neural network trained using a pre-constructed training data set; and acquiring a stable plane in which the object is placed on a support surface in a stable posture from the artificial neural network, wherein the trained artificial neural network acquires a learning mesh for a target object, drops the learning mesh onto a plane, samples an area in contact with the plane in the learning mesh as a learning stable plane, and is trained using the training data set including the learning mesh and the learning stable plane (see substantively similar claim 1 rejection above).
Noh in view of Walter and Guler as combined in claim 8 is not found disclosing expressly the limitation of, a program stored on a computer-readable recording medium, and executed by one or more processors in an electronic device, the program including instructions to execute the steps.
However, Guler discloses that in another embodiment machine 1100 can implement mesh reconstruction methods using a program stored on a computer-readable recording medium, and executed by one or more processors in an electronic device, the program including instructions to execute the mesh reconstruction methods (¶060-0163, ¶0210, fig. 11).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to execute the mesh reconstruction methods from a program stored on a computer-readable recording medium, and executed by one or more processors in an electronic device, the program including instructions to execute the steps disclosed in claim 8, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noh in view of Walter and further in view of Sundermeyer et al. (US 20220288783 A1, hereinafter Sundermeyer).
Regarding claim 7, Noh in view of Walter discloses
acquiring a mesh for a target object; disposing a flat plate model, disposing the mesh above the flat plate model, and dropping the mesh disposed above the flat plate model toward the flat plate model; and sampling, as a stable plane, an area in contact with the flat plate model in the mesh when the mesh dropped toward the flat plate model stops, to generate training data for an object controlled to be placed on a specific support surface (see substantively similar claim 1 rejection above).
Noh in view of Walter is not found disclosing expressly a program stored on a computer-readable recording medium, and executed by one or more processors in an electronic device, the program including instructions to execute the aforementioned steps.
However, Sundermeyer discloses Apparatuses, systems, and techniques to grasp objects with a robot (see abstract), wherein the system can be implemented in PPU 3500 (see fig. 35 and corresponding sections in the specification) comprising, a program (¶0480) stored on a computer-readable recording medium (3504, ¶0476, ¶0484), and executed by one or more processors (3518, ¶0476, ¶0481-484) in an electronic device (¶0474-045), the program including instructions to execute the method steps (¶0474-0476).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to implement the steps undertaken in the simulation system of Noh in view of Walter in a PPU of an electronic device as disclosed by Sundermeyer, to obtain, a program stored on a computer-readable recording medium, and executed by one or more processors in an electronic device, the program including instructions to execute the steps of claim 1, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I).
Allowable Subject Matter
Claim 2 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 2, Noh in view of Walter discloses the stable plane estimation method of claim 1, further comprising:
rotating the flat plate model disposed in a virtual space by a predetermined angle, disposing the mesh so that the sampled stable plane faces the flat plate model from above the rotated flat plate model, and dropping the disposed mesh toward the flat plate model (§IV-A, “Stable plane annotation,” p. 3; Fig. 2. Noh expressly states that it tilts the table by 10°. Before tilting, Noh states that it places the object on a flat table according to the normal vector of the candidate stable plane);
identifying the area in contact with the flat plate model in the mesh when the mesh that has been dropped toward the rotated flat plate model stops (Noh's initial stable-plane generation expressly uses dropping. The dataset is generated by dropping each object on a table in 512 different configurations and by sampling stable planes that satisfy Eq.2. The stable plane candidates are verified using a tilted table – Fig. 2.
We dropped the object on the table and recorded all poses in which it remained stable (Ui < ϵ1) to identify stable planes that support the object. Attaining stability condition (Ui < ϵ1) is understood as “flat plate model stops”
We considered only rotational motion when we evaluated object stability because rotational motion is more common than translational motion when an object placed in an unstable state falls due to the vibrations. – §V Experiments, p. 5, Evaluation metrics); and
However, the limitation of comparing the identified area with the area sampled as the stable plane, and verifying the stable plane based on a comparison result – is not reasonably taught in Noh.
Conclusion
The prior and/or pertinent art(s) made of record and not relied upon is considered pertinent to applicant's disclosure, are :
Thon et al. (US 20240253232 A1),
Yang et al. (US 20220032454 A1)
– who disclose different interaction methods with virtual object in an AR/VR environment.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NURUN FLORA whose telephone number is (571)272-5742. The examiner can normally be reached M-F 9:30 am -5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571) 272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NURUN FLORA/Primary Examiner, Art Unit 2619
Prosecution Insights