DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRIs are used for purposes of searching for prior art, but cannot be incorporated into the claims. Claim limitations must be given their plain meaning unless such meaning is inconsistent with the specification. MPEP 2111.01. BRIs for some of the claim limitations are provided below. Should Applicant believe that other interpretations are warranted, Applicant should point to the portions of the present disclosure that clearly show that a different interpretation is appropriate.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10 and 11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 10, it is unclear what is meant by the limitation “determining a similarity between a characteristic, indicated by a second external object classified as a second type, and the distribution”. It appears that the claim is directed to the manner in which the processor generates the probability distributions shown in Figs. 7A-7C of the present disclosure and uses those distributions to determine the virtual boxes, but it is unclear what the distributions are being compared to in order to determine the “similarity” recited in claim 10. The portion of the present specification that describes the distributions shown in Figs. 7A-7C does not mention determining a similarity between the distributions and anything else. Therefore, the claim cannot be understood and is open to multiple interpretations.
For these reasons, claim 10 is indefinite. Claim 11 is indefinite due to its dependence from claim 10.
The BRI for this claim, as best as can be understood, is that the second external object is classified based on the distribution and that the combination bounding box is divided into the first and second bounding boxes based at least in part on the classification of the second external object.
Claim 11 is rejected due to its dependence from claim 10.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 6, 7, 10, 12-15, 17 and 18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by U.S. Publ. Appl. No. 2022/0357453 A1 to Zhou et al. (hereinafter referred to as “Zhou”).
Regarding claim 1, Zhou discloses a vehicle control apparatus (Para. [0061] describes the system and method of Zhou being used to control operations of an autonomous vehicle (AV)) comprising:
light detection and ranging (LiDAR) device disposed on a vehicle, wherein the LiDAR device is configured to obtain sensing information corresponding to a first external object (Fig. 6, para. [0099] discloses the LiDAR system of the vehicle); and
a processor (Fig. 1, processor 146 controls operations of the AV system 120) configured to:
determine, based on the sensing information, a first virtual box corresponding to the first external object (Para. [0123]: “the perception module 402 can ‘predict’ a bounding box (e.g., using a machine learning) such that the bounding box encloses that cluster of points and another adjacent cluster of points.” In Zhou, the first external object is an object external to the vehicle sensed by the LiDAR system 123 shown in Fig. 1);
determine a candidate group comprising a combination virtual box, wherein the combination virtual box comprises the first virtual box and a second virtual box (The BRI for the term “virtual box” is based on para. [0011] of the present disclosure, which indicates that a virtual box is what is commonly referred to in the art as a “bounding box”. Para. [0123] of Zhou discloses the combination virtual box: “[m]ultiple bounding boxes can be merged together (e.g., using non-maximal suppression), such that the resulting box encloses multiple clusters of points, each corresponding to the same object.”), wherein the determining of the candidate group is based on at least one of: a driving state of the vehicle, a size of the first virtual box, or a position of the first virtual box, and wherein the combination virtual box is associated with LiDAR data representing LiDAR points (Para. [0120] discusses the perception module 402 generating the combination bounding box from LiDAR data captured by the LiDAR system 123, 502a and 602, and therefore the combination bounding box is determined based at least on the driving state of the vehicle because it is based on LiDAR data captured by the LiDAR sensors of the vehicle in whatever driving state the vehicle is in when the data is captured);
determine, based on applying the LiDAR data to a neural network model, a distribution of the LiDAR points (Fig. 13 shows the cluster segmentation and classification network 1302 that includes a neural network 1400 shown in Fig. 14 and described in, for example, paras. [0132]-[0135], which determines the distribution of the LiDAR points contained in the point cloud data);
divide, based on the distribution, the combination virtual box into an adjusted first virtual box and an adjusted second virtual box (The BRI for this limitation, which is based on paras. [0011] and [0067] of the present disclosure, is that bounding boxes that have been combined due to under-segmentation are separated into separate bounding boxes, i.e., into the first and second adjusted bounding boxes. Para. [0037] of Zhou discloses reducing under-segmentation problems by dividing the combination bounding box into first and second adjusted bounding boxes: “[t]o reduce under-segmentation, for each cluster of points, multiple bounding boxes can be “predicted” (e.g., using a machine learning) such that the bounding boxes enclose different respective portions of the cluster. Bounding boxes can be selected such that the each of the bounding boxes encloses points corresponding to a different respective object.” Para. [0140] of Zhou discloses the neural network defining boundaries between, and separating portions of, the point cloud distribution corresponding to different objects, which means dividing bounding boxes); and
control, based on at least one of the adjusted first virtual box or the adjusted second virtual box, an operation of the vehicle (Fig. 4, paras. [0091]-[0093] disclose controlling operation of the vehicle by controls 406 based on the outputs of the sensors, including the LiDAR sensors and the object classification).
Regarding claim 2, Zhou discloses that the combination virtual box is divided by determining the adjusted first virtual box classified as a first type and determining the adjusted second virtual box classified as a second type different from the first type (The BRI for this limitation, based on para. [0085] of the present disclosure, is that the first and second virtual boxes are determined by classifying the objects bounded by them as different objects. Figs. 16A-16C and paras. [0163]-[0170] of Zhou disclose dividing the combination bounding box 1602 into first and second bounding boxes 1604a and 1604b based on the point cloud distributions being classified as corresponding to different pedestrians).
Regarding claim 3, the BRI for the term “grid map”, based on para. [0024] of the present disclosure, is that it is it is a grouping of the LiDAR points. In Zhou, the points representing the first and second bounding boxes 1604a and 1604b are points of the point cloud captured by the LiDAR sensors (Paras. [0034], [0039] and [0164]), which constitute grid maps. Zhou discloses that the data that is processed by the processor 146, Fig. 1, which includes the grid map point cloud data acquired by the LiDAR sensors, is output to and stored in memory 142 and/or 144 (Para. [0064]).
Regarding claim 4, the BRI for this limitation, based on para. [0098] of the present disclosure, is that the candidate group corresponding to the combination virtual box includes a vehicle that is a distance away from the vehicle that comprises the vehicle control apparatus in the direction of travel. Zhou discloses that determining the candidate group comprises determining external objects around the vehicle that are sensed by the LiDAR system, including a vehicle that is a longitudinal distance in front of the Lidar system 602 (Paras. [0091], [0099] and [0131]: “[f]or instance, for each of the clusters of points, the classification network 1306 can determine the location of the object represented by that cluster (e.g., an absolute location, or a location relative to the AV 100) and the dimension and shape of the object. Further, for each of the clusters of points, the classification network 1306 can determine the type of object represented by that cluster. For instance, for each of the clusters of points, the classification network 1306 can determine whether the object represented by that cluster is a pedestrian, a bicycle, an automobile, a traffic sign, a barrier, a tree, a building, a bridge, or any other type of object.”).
Regarding claim 6, the BRI for this limitation, based on paras. [0061] and [0103] of the present disclosure, is that the point cloud data is projected onto a virtual two-dimensional (2-D) plane. In Zhou, the points of the point cloud data acquired by the LiDAR system are spatially distributed in at least two dimensions because they are clustered in a 2-D plane, as shown in Figs. 15A-16C and described in paras. [0154] - [0173], which means they are projected onto a designated surface according to the BRI.
Regarding claim 7, the BRI for this limitation, based on para. [0083] of the present disclosure, is that the points have spatial values associated with at least one axis of a reference frame. As is well known in the art, the points of point cloud data captured by LiDAR sensors have spatial coordinates in at least 2-D and typically 3-D. Therefore, the points of the point cloud data of Zhou meet this limitation.
Regarding claim 10, the BRI for this claim, as best as can be understood, is that the second external object is classified based on the distribution of the LiDAR points and that the combination bounding box is divided into the first and second bounding boxes based at least in part on the classification.
Zhou teaches that the combination bounding box is divided into the first and second adjusted bounding boxes based on the classification of the distribution of the LiDAR points corresponding to the first and second external objects (Figs. 16A-16C, Para. [0168]-[0170]: “[f]or each of the bounding boxes 1604a and 1604b, the cluster refinement module 1310 can determine the type of object represented by the points enclosed by that bounding box, and a confidence metric associated with that determination (e.g., using a cluster classification network and/or a neural network 1400). If the confidence metrics for both of the bounding boxes 1604a and 1604b are sufficiently high (e.g., both confidence metrics are greater than a threshold value), the cluster refinement module 1310 can split the set of points 1602 into two different sets of points in accordance with the bounding boxes 1604a and 1604b”).
Regarding claims 12-15, 17 and 18, the rejections of claims 1-4, 6 and 7 apply mutatis mutandis to claims 12-15, 17 and 18, respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou in view of U.S. Pat. No. 12,327,364 B2 to Jia (hereinafter referred to as “Jia”).
Regarding claim 8, Zhou uses a neural network with at least one hidden layer (Fig. 14, 1406b), which is considered a deep learning model. However, Zhou does not explicitly disclose that the machine learning model is a Gaussian Mixture Model (GMM). Jia, in the same field of endeavor, discloses applying LiDAR data to a GMM that determines whether or not bounding boxes are to be fused or kept separate based on probability distributions obtained by the model (Col. 3, lines 4-19, Fig. 5, Col. 7, lines 61-65, Col. 10, lines 7-8).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the AV system 120 of Zhou to use the GMM of Jia as the cluster classification network 1306 of Zhou to determine whether bounding boxes are to be divided based on the classification of objects represented by the LiDAR points. One of ordinary skill in the art would have been motivated to make the modification to take advantage of the robustness of GMMs at classifying objects given noisy, sparse and complex data. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying software executed by the processor 146 to implement a GMM for performing object classification).
Regarding claim 19, the rejection of claim 8 applies mutatis mutandis to claim 19.
Claims 9, 11 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou in view of Jia as applied to claim 8 and further in view of an article entitled “A Robust Gaussian Process-Based LiDAR Ground Segmentation Algorithm for Autonomous Driving”, by Jin et al., published in June 23, 2022 in Machines 2022, 10, 507. https://doi.org/10.3390/machines10070507 (hereinafter referred to as “Jin”).
Regarding claim 9, a hyperparameter is an inherent feature of a GMM because it is a configuration setting that is need to control the structure and behavior of the GMM. Therefore, the GMM of Jia necessarily determines the distributions based on setting a hyperparameter of the GMM to a designated value. Nevertheless, Jia does not explicitly disclose this limitation.
Jin, in the same field of endeavor, discloses determining a distribution of LiDAR points based on setting a hyperparameter of a Gaussian process model to a designated value (Section 4.1, Hyperparameter Acquisition: “[c]alculating hyperparameters in real-time for each frame of LiDAR data is a heavy burden on the computer. Therefore, it is more reasonable to calculate it in advance, which adapts to as many ground shapes as possible for Gaussian process.”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the AV system 120 of Zhou to use the GMM of Jia as the cluster classification network 1306 of Zhou based on a hyperparameter setting selected in advance as taught by Jin to determine whether bounding boxes are to be divided based on the classification of objects represented by the LiDAR points. One of ordinary skill in the art would have been motivated to make the modification to avoid the need to calculate hyperparameters in real-time and thereby reduce computational overhead as taught by Jin. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying software executed by the processor 146 to implement a GMM for performing object classification with a hyperparameter set in advance).
Regarding claim 11, Zhou does not explicitly disclose that the combination bounding box is divided into the first and second bounding boxes based on at least one of an x-axis variance of the distribution, a y-axis variance of the distribution, or a Mahalanobis variance of the distribution.
Jin discloses using the Mahalanobis variation in the Gaussian process to perform clustering and bounding box fitting of LiDAR points corresponding to detected objects (Abstract, Sections 3.1 and 3.2).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the AV system 120 of Zhou to use the Mahalanobis variance of the distributions generated by the GMM of Jia based on the teachings of Jin to determine whether bounding boxes are to be divided based on the classification of objects represented by the LiDAR points. One of ordinary skill in the art would have been motivated to make the modification to take advantage of the robustness of using the Mahalanobis variance in classifying objects and generating, merging and splitting of bounding boxes. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying software executed by the processor 146 to implement a GMM with Mahalanobis variance for performing object classification).
Regarding claim 20, the rejection of claim 9 applies mutatis mutandis to claim 20.
Allowable Subject Matter
Claims 5 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claims 5 and 16 recite determining the candidate group further based on determining that the first virtual box is located at a designated position in the combination virtual box and further based on a size of the combination virtual box being greater than or equal to a size of the first virtual box by at least a designated proportion.
None of the prior art teaches or suggests this limitation in combination with the other limitations recited in the claims from which these claims depend.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
An article entitled “Merging Scored Bounding Boxes with Gaussian Mixture Model for Object Detection”, by Gu et al., published in 2018 in Proceedings of the 6thIIAE International Conference on Intelligent Systems and Image Processing 2018, discloses a novel approach to merge all the scored bounding boxes by Gaussian Mixture Model (GMM) that takes not only the spatial information but also the score of each detection into account.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights