DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/21/2025 was filed after the mailing date of the application. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 15 and 16 are directed to a computer program per see, which is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because as stated in MPEP section 2106 Patent Subject Matter Eligibility: “Products that do not have a physical or tangible form, such as information (often referred to as “data per se”) or a computer program per se (often referred to as “software per se”) when claimed as a product without any structural recitations” are not directed to any of the statutory categories.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 8-10, 12-16 are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US. Patent App. Pub. No. 2018/0174325, “Fu”, hereinafter) in view of Chandler et al. (US. Patent App. Pub. No. 2022/0319043, “Chandler”).
As per claim 1, as shown in Fig. 10, ¶ [28-32], Fu teaches a computer-implemented method for tracking movements of an industrial machine (a forklift) for each image of a sequence of two-dimensional images that captures the industrial machine (captured images of the forklift) comprising:
generating a first bounding box identifying a first component of the industrial machine;
generating a second bounding box identifying a second component of the industrial machine (Fig. 10, ¶ [106-108], bounding boxes of portions of the forklift);
generating a three-dimensional model for representing the industrial machine based on the first bounding box and the second bounding box, wherein the three- dimensional model comprises a first geometric shape representing the first component a second geometric shape representing the second component (Fig. 12-13, ¶ [118-120], ¶ [123], ¶ 126-127]).
optimizing pose of the three-dimensional model based on the two- dimensional projection, and further based on the first bounding box and the second bounding box (¶ [96], ¶ [115], and ¶ [123], further addressed below); and
tracking movements of the industrial machine over time based on the optimized poses of the three-dimensional model for each image of the sequence (¶ [100], tracking the direction of travel of vehicle).
Fu does not expressly teach projecting the three-dimensional model on the image, resulting in a two-dimensional projection and optimizing pose of the three-dimensional model based on the first bounding box and the second bounding box.
However, in a very similar method of generating bounding box of a vehicle (see ¶ [16-20]), Chandler teaches the above features, i.e., projecting the three-dimensional model on the image, resulting in a two-dimensional projection (¶ [20], “Advantageously, fitting the 2D bounding box to the projection of the 3D model itself can provide a tight 2D bounding box”), and optimizing pose of the three-dimensional model based on the first bounding box and the second bounding box (¶ [228]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method as taught by Chandler into the method as taught by Fu as addressed above, the advantage of which is to provide a higher quality segmentation mask (¶ [22]).
As per claim 2, the combined teachings of Fu and Chandler also include wherein generating the first bounding box and further generating the second bounding box further comprises:
inputting the image to a first neural network trained using a first training dataset (Chandler, Fig. 4, ¶ [97], ¶ [102], and ¶ [194]); and
comprising images of the industrial machine where the first component and second component are annotated with two-dimensional bounding boxes (Chandler, ¶ [20]). Thus, claim 2 would have been obvious over the combined references for the reason above.
As per claim 8, the combined Fu-Chandler does also teach wherein optimizing the pose of the three-dimensional model comprises minimizing a misalignment of the two-dimensional projection as compared to the first bounding box and the second bounding box (Chandler, ¶ [137], by aligning the bounding box. See also Fig. 11E-G). Thus, claim 8 would have been obvious over the combined references for the reason above.
As per claim 9, the combined Fu-Chandler also teaches wherein the industrial machine is a forklift (Fu, Fig. 10, as addressed in claim 1).
As per claim 10, the combined Fu-Chandler does impliedly teach wherein the first component is a body of the forklift, and wherein the second component is a tine of the forklift (Fu, ¶ [20], ¶ [32]).
As per claim 12, the combined Fu-Chandler further impliedly teaches for each image of the sequence of two-dimensional images that captures the industrial machine:
generating at least one further bounding box identifying a respective at least one further component of the industrial machine,
generating the three-dimensional model further based on the at least one further bounding box, wherein the three-dimensional model further comprises at least one further geometric shape respectively representing the at least one further component, and
optimizing the pose of the three-dimensional model further based on the at least one further bounding box (since the combined method substantially teaches generating bounding box of different component of the forklift (such as one shown in Fig. 13 of Fu). Thus, the generation of 3D model and the optimization step is similarly addressed in claim 1). Thus, claim 12 would have been obvious over the combined references for the reason above.
Claim 13, which is similar in scope to claim 1 as addressed above, is thus rejected under the same rationale.
As per claim 14, the combined Fu-Chandler does also teach a camera configured to generate the sequence of two-dimensional images (e.g., Fu, ¶ [88]).
Claims 15 and 16, which is similar in scope to claim 1 as addressed above, is thus rejected under the same rationale.
Claims 3-5, and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US. Patent App. Pub. No. 2018/0174325) in view of Chandler et al. (US. Patent App. Pub. No. 2022/0319043) further in view of Xu et al. (US. Patent App. Pub. No. 2020/0005485, “Xu”).
As per claim 3, the combined Fu-Chandler does impliedly teach wherein generating the three-dimensional model further comprises: cropping the image according to the first bounding box to result in a region of interest (Chandler, ¶ [138], “..initially, a 3D bounding box may be automatically located and oriented in the second frame by fitting the 3D object model to the point cloud of the second frame, and the annotator may then manually tweak the 3D bounding box to minimize any visible discrepancy between the 3D model and the actual object in the second frame (thereby fine-tuning the location/orientation of the 3D bounding box in the second frame)”).
The combined Fu-Chandler does not explicitly teach estimating size and orientation of the first component based on the region of interest.
However, Xu teaches a very similar method of generating 3D bounding box (see Abstract), wherein the method further comprises the above features, i.e., estimating size and orientation of the first component based on the region of interest (see Fig. 4 (step 406), and 5, ¶ [11-12], and ¶ [48]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the method as taught by Xu to the combined Fu-Chandler method as addressed above, the advantage of which is to effectively navigate a three-dimensional environment (¶ [2]).
As per claim 4, the combined teachings of Fu, Chandler, and Xu further include wherein estimating size and orientation of the first component further comprises inputting the region of interest to a second neural network trained using a second training dataset comprising images of the industrial machine annotated with information on orientation and dimension of the first component (Xu, ¶ [35] referring to Fig. 3, “The center of the region of interest may be then unprojected into the camera frame as a ray and the rigid rotation that would rotate this ray to the z-axis of the camera frame may be found… Accordingly, the input data may be normalized for consideration by the ANN 318”). Thus, claim 4 would have been obvious over the combined references for the reason above.
As per claim 5, the combined Fu-Chandler-Xu does also teach wherein generating the three- dimensional model further comprises: generating the first geometric shape based on the estimated size; and orientation of the first component and further based on the first bounding box (Xu, ¶ [11]). Thus, claim 5 would have been obvious over the combined references for the reason above.
As per claim 7, the combined Fu-Chandler-Xu substantially teaches wherein generating the three- dimensional model further comprises generating the second geometric shape based on the first geometric shape and further based on each of the first bounding box and the second bounding box.
Allowable Subject Matter
Claims 6 and 11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The prior art taken singly or in combination does not teach or suggest, a computer-implemented method, among other things, comprising:
… wherein generating the three-dimensional model further comprises refining position of the first geometric shape by aligning a bottom center of the first geometric shape with a bottom middle point of the first bounding box (claim 6); or
…wherein the first geometric shape is a cuboid, and wherein the second geometric shape is an ellipsoid (claim 11).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hau H. Nguyen whose telephone number is: 571-272-7787. The examiner can normally be reached on MON-FRI from 8:30-5:30.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard, can be reached on (571) 272-7773.
The fax number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/HAU H NGUYEN/Primary Examiner, Art Unit 2611
Prosecution Insights