DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The applicant argues that “it would not have been obvious to fine-tune the relied-upon model of Nagato using at least one labeled defect, as such a modification would run contrary to the teachings of Nagato.” Remarks at 8. The examiner disagrees. While Nagato employs non-defective learning to train its model (see Nagato ¶ 75), the POSITA would have concluded that this model, once trained, would be improved by a subsequent fine-tuning using labeled defects.
The specification of this application discloses:
The 3D profiles may be a set of 3D profiles to be used to fine-tune a pre-trained model. The pre-trained model 1806 may be input for implementing an act 1808 (e.g., by the 3D deep learning tool) of fine-tuning a deep learning model. A user may draw on the set of profiles to identify a defect such that the tool can be trained to find such a defect.
Spec. ¶ 137. In the specification, the fine-tuning is performed after training by labeling the 3D profiles. The 3D profiles are information that is extracted from the inspection scene, and are different from the information that was used to train the model. See Spec. ¶¶ 5-7. In light of the specification, claim 1 is not limited to using a model that is trained on defective products. The claim is only limited to fine-tuning based on defects.
As described above, Nagato uses non-defective learning when training its model. But the broadest reasonable interpretation of the claims encompasses a model that was trained in this manner. And the POSITA would have concluded that Nagato’s pre-trained model could be improved after training by fine-tuning the model using labeled defects. Training the model and fine-tuning the model are separate procedures that are performed using different data sets. See above, and the applicant’s Fig. 18. The person skilled in this field would appreciate that fine-tuning a model using labeled defects would not be limited to models that were pre-trained on defective examples.
Nagato’s model is used to identify defects. See Nagato ¶¶ 82-84. A transfer learning procedure, such as taught in Kaul (see the § 103 rejection below), would improve the model by fine-tuning it to identify types of defects that the model was not specifically pre-trained to detect. The POSITA would have concluded that fine-tuning Nagato’s model in this way would enable Nagato’s model to be customized for various tasks, and would provide superior results when the model is fine-tuned on labeled defect data that is specific to a particular inspection task. Kaul provides the motivation to make this modification, as described below. When Nagato-Shaubi is modified in this manner, the invention recited in the claims is rendered obvious. See the rejection below.
The assertion of official notice that was given in the rejection of claims 7, 12, 15-16, and 18-19 was not traversed, rendering this subject matter admitted prior art. See MPEP 2144.03(C).
The previous Office action described how certain language in claim 1 does not limit the claim. See the previous Office action pg. 3. The applicant has not addressed this issue.
Claim Interpretation
Claim 1 includes language that does not appear to limit the scope of the claim. Claim 1 recites providing the 2D map to the 2D deep learning model to generate an output; and providing the output to a subsystem for generating an inspection result for the 3D representation. The claim does not positively recite “generating” the output or “generating” the inspection result. Instead, the bolded language appears to describe the intended results of the “providing the 2D map” and “providing the output” limitations. Claim language which does not positively recite the performance of a process step, but rather recites an “intended result of a process step positively recited,” does not limit the scope of a process claim. MPEP 2111.04 I.
In the interest of compact prosecution, all language in claim 1 is addressed in the prior art rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-9, 11-16, and 18-24 are rejected under 35 U.S.C. 103 as being unpatentable over Nagato, US 20250026603 in view of Shaubi, US 20200294224, and further in view of Kaul, US 2024/0354929. The passages in Nagato that are relied upon in the rejection below find support in its foreign priority document.
Claim 1: Nagato discloses a method for three-dimensional (3D) inspection using a two-dimensional (2D) deep learning model (Abstract and ¶69), the method comprising:
accessing the 2D deep learning model (¶¶ 77-82 – deep learning model 37 is trained based on 2D heat maps of non-defective products);
accessing a 3D representation of a scene (¶82 – 3D data is acquired of the products under inspection. See Fig. 10.);
transforming the 3D representation to a 2D map (¶82 – the 3D data is converted into a 2D heat map), wherein the 2D map comprises a plurality of elements disposed in an array, and each of the plurality of elements comprises a vector of a geometric feature computed from the 3D representation (¶65 – the 2D heat map is an array of colors/brightness data that represent difference in height. This color or intensity is a vector that represents a geometric feature of the 3D data of the scene);
providing the 2D map to the 2D deep learning model to generate an output (¶83 – the 2D heat map is provided to the 2D deep learning model 37); and
providing the output to a subsystem for generating an inspection result for the 3D representation (¶¶ 83-84 – the output of the learning model 37 generates an inspection result for the 3D representation of the product).
Nagato fails to disclose that the 2D deep learning model was pre-trained using 2D images unrelated to an inspection task for the 3D inspection.
However, using a deep learning model that was pre-trained using images unrelated to a current task was well known in this field before the effective filing date of the claimed invention. For example, Shaubi discloses training a DNN for inspection of an image of an object, where the DNN is partly trained using a data set that is unrelated to the inspection task (¶73).
It would have been obvious to a skilled artisan before the effective filing date of the claimed invention to modify the system of Nagato with teachings in Shaubi, the rationale being to provide the deep learning model with access to a greater training set. The POSITA would conclude that this would have improved results of the model.
Nagato-Shaubi does not disclose fine-tuning the model using at least one labeled defect.
Kaul discloses fine-tuning a pre-trained model using a labeled defect (¶ 25. A pre-trained model is fine-tuned for specific defect detection tasks. The POSITA would understand this to implicitly describe labeling these defects in the fine-tuning process).
It would have been obvious to the skilled artisan before the effective filing date of the claimed invention to modify Nagato-Shaubi with these teachings in Kaul. The POSITA would have concluded that fine-tuning Nagato’s model to identify specific defects, as suggested in Kaul, would improve the model by enabling greater accuracy in identifying the specific type of defect that is identified in the fine-tuning step. The POSITA would have also appreciated that fine-tuning the pre-trained model would have enabled the model to be customized to accurately identify different types of defects in a variety of applications. See Kaul ¶ 25.
The examiner notes that Kaul’s provision application does not appear to disclose the subject matter of Kaul ¶ 25. However, the instant application’s provisional application likewise does not support the “fine-tuning” feature that is described in Spec. ¶ 137. This feature is therefore given the priority date of the filing of this application, which is 09/20/2024. Therefore Kaul, which was filed on 04/18/2024, is available as prior art for this claim limitation.
Claim 2: Nagato discloses that the 3D representation comprises a 3D point cloud, a mesh, sensor data, and/or a voxel grid (¶54 – the 3D data comprises data from sensor 17).
Claim 3: Nagato discloses that the geometric feature comprises: a distance of an associated 3D point of the 3D representation to a reference (¶65 – the features of the heat map show difference in height, which is distance to a reference of a point in the 3D representation).
Claim 4: Nagato discloses that the reference is a plane (¶¶65 and 95).
Claim 5: Nagato discloses that for each of the plurality of elements: the vector comprises a number of geometric features; and the number of geometric features is one (¶65 – height is represented in the vectors of the heat map. This is a number – one – of geometric features).
Claim 6: Nagato discloses that transforming the 3D representation to the 2D
map comprises:
determining a portion of the 3D representation that corresponds to one of the plurality of elements of the 2D map; and for the portion, computing the vector of the geometric feature for the corresponding element of the 2D map (¶¶ 64-65 – the height, i.e. vector, of a geometrical feature for each portion of the 3D data is computed when converting the 3D data into the heat map).
Claim 7: Nagato-Shaubi-Kaul fails to disclose that the 3D representation comprises a 3D point cloud comprising a plurality of 3D points. Official notice is taken that this was well known in the art before the EFD of this invention, and it would have been obvious to modify Nagato-Shaubi-Kaul to include this in order to obtain and store the 3D data in a commonly-understood and easily manipulable format. When Nagato-Shaubi is modified in this manner, the computations for converting the 3D data into the 2D heat map would then include:
for the portion, computing the vector of the geometric feature for the corresponding element comprises: computing the geometric feature for the 3D points in the portion; and determining the vector of the geometric feature for the corresponding element based on the computed geometric feature for the 3D points in the portion (Nagato ¶¶64-65).
Claim 8: Nagato discloses that transforming the 3D representation to the 2D
map comprises: projecting the computed vectors for the plurality of elements to a plane (¶64-65; Fig. 5 and Fig. 6).
Claim 9: Nagato discloses that the inspection result comprises a 3D result, and the 3D result comprises one or more of a height, surface area, center of mass, volume, or 3D bounding box in the 3D representation (¶¶ 84-85 – the inferences, i.e. the inspection results, show object defects, in height, in three dimensions. This is therefore a 3D result that comprises height in the 3D data).
Claim 11: Nagato discloses classifying an object via the subsystem (¶85. See also Fig. 19.).
Claim 12 Nagato-Shaubi-Kaul does not explicitly disclose determining, via the subsystem, whether the object is in the 3D representation of the scene. Official notice is taken that this was well known in the art of inspecting objects. Thus it would have been obvious to the POSITA to include this feature, the rationale being to reduce false positive and negative results.
Claim 13: Nagato discloses identifying, via the subsystem, a possible defect of an object (¶¶ 82-85).
Claim 14: Nagato discloses that the inspection result comprises a segment of the 3D representation associated with the possible defect (Fig. 12 and its description).
Claim 15: Nagato-Shaubi-Kaul does not explicitly disclose the 2D deep learning model and/or the subsystem comprises a back-end component that maintains an adjustable parameter. Official notice is taken that this was well known in the art of deep learning. Thus it would have been obvious to the POSITA to include this feature, the rationale being to provide greater flexibility in inspecting the objects based on the learning model.
Claim 16: Official notice is likewise taken that adjusting the parameter based on the inspection result generated by the subsystem, a training set of 2D maps, or both was well known. It would have been obvious to the POSITA to include this feature, the rationale being to provide greater flexibility in inspecting the objects based on the learning model.
Claims 18 and 19 recite that the inspection result comprises a quality metric; and the method comprises modifying the geometric feature based on the quality metric (claim 18), and modifying the geometric feature based on the quality metric comprises a brute force search, a greedy search, or a gradient-descent optimization, such that a value of the quality metric is modified (claim 19). Official notice is taken that producing a confidence (i.e., a quality) metric, and modifying operations of the model in the claimed manner, was well known and commonly practiced in this field. Therefore it would have been obvious to the POSITA to implement this in Nagato’s system, the rationale being to enable the system to continually provide improved results.
Claim 20: Nagato discloses that the 3D representation comprises one or more 3D profiles comprising a plurality of 3D points, wherein each 3D point is obtained from an initial 3D representation of the scene based on an associated polyline (¶¶64-66 – the 3D data, i.e. 3D profile, makes up the 3D representation. Each point in the 3D data is obtained in the manner claimed).
Claim 21: Nagato discloses that transforming the 3D representation comprises transforming the one or more 3D profiles to the 2D map (¶¶ 64-65).
Claim 22: Nagato discloses that the inspection result is based on the one or
more 3D profiles (¶¶83-84).
Claims 23 and 24: see rejection of claim 1. Nagato further discloses a system comprising at least one processor configured to perform the method of claim 1, and a non-transitory computer readable medium comprising program instructions that, when executed, cause at least one processor to perform the method of claim 1 (¶59).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT J HANCE whose telephone number is (571)270-5319. The examiner can normally be reached M-F 11:00am-7:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Fuelling can be reached at (571) 270-1367. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT J HANCE/Reexamination Specialist, Art Unit 3992
Prosecution Insights