DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Claim(s) 1-20 do not use “means for” (or “step for”) language, or generic placeholders for "means” coupled with functional language without recitation of sufficient structure for carrying out the claimed functions and therefore do not invoke 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-9 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.[claims 1-9]
Claim 1 recites “extracting a first feature by inputting a first modality sensed by a first sensor to a student model that infers the first feature from the first modality, and converting the first feature into a first feature vector in a bird's eye view (BEV) space;
extracting a second feature extracted by inputting a multi-modality to a teacher model that infers the second feature from the modality” (emphasis added).
As written, it is unclear whether “the modality” refers to “a first modality” or “a multi-modality”.
For the purposes of applying prior art, “the modality” will be read as referring to “a multi-modality” and it is suggested that the claim be amended to recite “the multi-modality”.
Claims 2-9 are similarly rejected for their dependence on claim 1.
Clarification is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 8-10, 12-14, 16-17, 19 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. (US 2024/0096105 A1) in view of Peng et al. (“Correlation Congruence for Knowledge Distillation”) in view of Official Notice.[claim 1]
Regarding claim 1, Zhao discloses a model learning method of generating a vector map, the method performed by one or more processors (e.g. Figure 9; Paragraphs 0089-0091), the learning method comprising:
extracting a first feature by inputting a first modality sensed by a first sensor to a student model that infers the first feature from the first modality, and converting the first feature into a first feature vector in a bird's eye view (BEV) space (Figure 4A; Image 402 input to student ODCM 401-S generating BEV space at output of Student BEV FN450-S);
extracting a second feature extracted by inputting a multi-modality to a teacher model that infers the second feature from the modality, and converting the second feature into a second feature vector in the BEV space, wherein the multi-modality comprises the first modality and a second modality sensed by a second sensor (Figure 4A; Image 402 and Depth Data 404 input to Teacher ODCM 401-T and generating BEV space at output of Teacher BEF FN 450-T); and
teaching the student model to generate a vector map corresponding to the first modality by back-propagating (Figure 4A; Feature Vectors 421, Depth Distributions 422 and Detected Objects 421 used as ground truth during training of Student model; Paragraph 0071).
While Zhao teaches back propagating differences of outputs of the teacher model through the student model, Zhao does not explicitly describe back propagating cross-correlation loss by dimension, which causes the first feature vector to replicate a cross-correlation with the second feature vector to the student model.
Peng discloses a knowledge distillation system which utilizes correlation loss by dimension (note that dimension is undefined by the claim, thus any dimension may be defined to meet the claimed “dimension”), which causes a first feature vector to replicate a correlation with a second feature vector to the student model (e.g. Section 3. Correlation Congruence Knowledge Distillation; by performing knowledge distillation as described by Peng, the feature vectors would be made more similar to each other). Peng discloses that the correlation congruence knowledge distillation system significantly promote the performance of the student network (e.g. Section 5. Conclusion).
Therefore, it would have been obvious to use a correlation congruence knowledge distillation as taught by Peng when back-propagating differences between the teacher and student models of Zhao to significantly promote the performance of the student network.
While Peng describes correlation which may be any correlation metric (Section 3.3. Correlation Congruence), Peng does not explicitly disclose the use of a cross-correlation.
However, Official Notice is taken that cross-correlation is a well-known type of correlation which provides a measure of similarity between two functions. Therefore, it would have been obvious to use a cross-correlation as the correlation metric of Zhao in view of Peng to measure the similarity between the feature vectors of the teacher and student models to provide a loss between the two for training the student model.[claim 3]
Regarding claim 3, Zhao discloses wherein the teaching of the student model further comprises repeatedly back-propagating either a first ground truth (GT) loss between a first output instance of the student model and a first GT instance corresponding to the first modality or (ii) a second GT loss between a second output instance of the teacher model and a second GT instance corresponding to the multi-modality to the student model (Paragraph 0071; back propagating loss using output of teacher model such as Feature Vectors 421 or Depth Distributions 411 and GT instance of the multi-modality Detected Objects 451; alternatively see Figure 4B and note that a GT instance of Depth Data 404 may be used).[claim 8]
Regarding claim 8, Zhao discloses wherein the first modality and the second modality are input in synchronization with each other (Figure 4A; inputting Image 402 and Depth Data 404 together to teacher model).[claim 9]
Regarding claim 9, Zhao discloses wherein
the first sensor comprises a camera sensor, and the second sensor comprises a lidar sensor (Paragraphs 0020-0023, 0032-0035, 0068-0069).[claim 10]
Regarding claim 10, see the rejection of claims 1 and 9 above.[claims 12-14]
Regarding claims 12-14, see the rejection of claims 2, 8 and 9 above.[claim 16]
Regarding claim 16, Zhao discloses wherein the first modality comprises a color image and the second modality comprises a point cloud (Paragraphs 0022, 0043; note that the same types of images/lidar data may be applied to inputs of Fig. 4A).[claim 17]
Regarding claim 17, see the rejection of claim 10 above and note that Zhao discloses implementing the described method using apparatus for generating a vector map (Figure 9), the apparatus comprising:
one or more processors (Figure 9, 902; Paragraphs 0090-0091); and
memory storing instructions executed by the one or more processors (Figure 9, Main Memory 904 storing Instructions 922; Paragraphs 0090-0091).[claims 19 and 20]
Regarding claims 19 and 20, see the rejection of claims 3 and 9 above, as well as the rejection of claim 17.
Allowable Subject Matter
Claims 2 and 4-7 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Claims 11, 15 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.[claims 2, 11 and 18]
Regarding claims 2, 11 and 18, the prior art does not teach or reasonably suggest wherein the teaching of the student model comprises repeatedly back-propagating the cross-correlation loss that causes
a first correlation between the first feature vector and the second feature vector to be greater than a reference value in response to a dimension of the first feature vector and a dimension of the second feature vector being the same and that causes
a second correlation between the first feature vector and the second feature vector to be less than the reference value in response to the dimension of the first feature vector and the dimension of the second feature vector being different.
While the prior art teaches a system and method for correlation congruence knowledge distillation (see rejections above), the prior art does not teach causing first and second correlation values to be greater than or less than a reference value in response to dimensions of the feature vectors being the same or different as claimed.[claims 4-6]
Regarding claims 4-6, the prior art does not teach or reasonably suggest the model learning method of claim 1, further comprising:
augmenting the first feature vector or the second feature vector,
wherein the teaching of the student model comprises back-propagating the cross-correlation loss by dimension to the student model so that the augmented first feature vector replicates a cross-correlation with the augmented second feature vector.
While Zhao in view of Peng discloses a similar method (see rejections above), the references do not teach augmenting the first or second feature vectors and performing back-propagating of cross-correlation loss using the augmented feature vector as claimed.[claims 7 and 15]
Regarding claims 7 and 15, the prior art does not teach or reasonably suggest wherein each of the first feature vector and the second feature vector comprises a row corresponding to the dimension and a column corresponding to an instance.
While Zhao in view of Peng discloses a similar method, the references do not teach wherein each of the first feature vector and the second feature vector comprises a row corresponding to the dimension and a column corresponding to an instance as claimed.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Varma et al. EP 4345688 A1
Xu et al. CN 115496941 A
Niu et al. Efficient and Robust Knowledge Distillation from A
Stronger Teacher Based on Correlation Matching
Huang et al. Knowledge Distillation from A Stronger Teacher
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY J HENN whose telephone number is (571)272-7310. The examiner can normally be reached Monday-Friday ~10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Twyler Haskins can be reached at (571) 272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Timothy J Henn/Primary Examiner, Art Unit 2639
Prosecution Insights