Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED OFFICE ACTION
Status of Claims
Claims 1-16 are pending examination.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b) (2) (C) for any potential 35 U.S.C. 102(a) (2) prior art against the later invention.
1. Claims 1,2,3,4,5,6,7,8,9,14 and 16 are rejected under 35 U.S.C 103(a) as being unpatentable over Dal Mutto et al. (USPUB 20190096135) in view of Huy Toan Nguyen et al. ( NPL Doc. : “Defective Product Classification System for Smart Factory Based on Deep Learning,”31st March 2021, Electronics 2021, 10, 826, Pages 1- 11.).
As per claim 1, Dal Mutto et al. teaches A method of employing quality control on a product ( Paragraphs [0076-0077]- “…defect detection is a component of quality control in contexts such as manufacturing, where individual objects may be inspected and analyzed for compliance with particular quality standards. The inspection may typically be performed visually by a trained human inspector, who analyzes each manufactured object (or a representative sampling thereof) to assess compliance of the object with particular quality standards….”) , comprising: receiving a unique identifier of a previously finished product having a finished product dataset that had been evaluated against an original dataset ( Paragraph [0127]- “…the inspection agent 300 identifies an object based on its 3-D model. In various embodiments of the present invention, identification of the object is used to obtain a set of identity-specific measurements. For example, a running shoe may be evaluated based on different criteria than a hiking boot because the two types of footwear may have different shapes, colors, sizes, and quality requirements….”) ,
the finished product dataset has a finished product feature class; comparing the finished product dataset to a new dataset ( FIG. 7 and Paragraphs [0130-0131]- “…the synthesized 2-D views are supplied to a descriptor generator 314 to extract a descriptor or feature vector for each view. In operation 1316, the feature vectors for each view are combined (e.g., using max pooling, as described in more detail below) to generate a descriptor for the 3-D model and to classify the object based on the descriptor. This feature vector may contain salient and characteristic aspects of the object's shape, and is used for subsequent classification or retrieval steps. The generated descriptor may be output in operation 1318….”) ;
Dal Mutto et al. does not explicitly teach discovering a revised feature class relative to the finished product feature class based upon the new dataset; modifying the new dataset with the revised feature class to provide a revised dataset; and evaluating an unfinished product with the revised dataset.
However, within analogous art, Huy Toan Nguyen et al. teaches discovering a revised feature class relative to the finished product feature class based upon the new dataset; modifying the new dataset with the revised feature class to provide a revised dataset; and evaluating an unfinished product with the revised dataset ( Pages 2- 3- “…deep learning models is the ability to update and automatically learning new features from new data or from different datasets. Therefore, if we continuously training the network when new data is available, the system will continue to improve by learning different characteristics from new data. Deep learning-based systems are powerful because of the various types of data available that includes ground truth information to train these systems. However, in practice, it is difficult to collect and label millions of images of defective and satisfactory examples of one product to train a deep learning model…. For comparison, we re-train models on our defective product dataset and upload to cloud service. This means that, for each product, the user will be able to load the appropriate pre-trained model or retrain the deep network with new collected images from the server. Moreover, the system also uploads images from AI module to the cloud in order to train new model…”).
One of ordinary skill in the art would have been motivated to combine the teaching of Huy Toan Nguyen et al. within the modified teaching of the Systems and methods for visual inspection based on augmented reality mentioned by Dal Mutto et al. because the Defective Product Classification System for Smart Factory Based on Deep Learning mentioned by Toan Nguyen et al. provides a method and system for implementation of classification of defects within manufactured product utilizing image processing and neural network learning.
Therefore, it would have been obvious for one in the ordinary skills in the art before the effective filing date of the claimed invention to implement the Defective Product Classification System for Smart Factory Based on Deep Learning mentioned by Toan Nguyen et al. within the modified teaching of the Systems and methods for visual inspection based on augmented reality mentioned by Dal Mutto et al. for implementing a system and method for classification of defects within manufactured product utilizing image processing and neural network learning.
As per claim 2, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 1,
Dal Mutto et al. teaches wherein the receiving step includes receiving an inquiry from a customer regarding a final verdict of the previously finished product associated with the unique identifier, and comprising a step of providing a verification to the customer regarding the final verdict ( Paragraph [0214-0215]- “… In some circumstances, this means that each surface element (e.g., points in the acquired point cloud, or triangular facets in a mesh) are assigned a tag indicating whether they correspond to a feature, and if so, an identifier (ID) corresponding to the feature. Hand labeling of a surface can be accomplished using software with a suitable user interface. In some embodiments, in operation 1744-2, the locations of the surface features are combined (e.g., concatenated) to form a descriptor of the locations of the features of the target object….”) .
As per claim 3, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 2,
Dal Mutto et al. teaches wherein the final verdict includes an approval or a disapproval of the previously finished product ( Paragraph [0171]- “…if the location of the logo deviates from the location of the logo in the reference model by more than the threshold distance (where the threshold distance corresponds to an acceptable tolerance level set by the manufacturer). For example, the output of the system may include an indication of the region or regions of the scanned 3-D multi-view model containing detected defects. In addition, the particular portions of the regions representing the detected defect may also be indicated as defective (rather than the entire region). In some embodiments, a defectiveness metric is also output, rather than merely a binary “defective” or “clean” indication. The defectiveness metric may be based on the computed distances, where a larger distance indicates a larger value in the defectiveness metric….”) .
As per claim 4, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 2,
Within analogous art, Huy Toan Nguyen et al. teaches wherein the finished product dataset includes an image repository, and comprising a step of accessing the image repository, and the verification includes providing a captured product image from the image repository to the customer ( Page 2- “…The best performing model will then be selected for adopting in factory. For comparison, we re-train models on our defective product dataset and upload to cloud service. This means that, for each product, the user will be able to load the appropriate pre-trained model or retrain the deep network with new collected images from the server. Moreover, the system also uploads images from AI module to the cloud in order to train new model further or to re-train old models to enhance system performance….”) .
As per claim 5, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 4, Within analogous art, Huy Toan Nguyen et al. teaches comprising using the image repository to store a conveyed captured product image and, for each detected object, an identification of the class of the detected object and an identification of a region of the detected object in the captured product image ( Paragraph [0147]- “…data about the identified class may be retrieved from, for example, a database of metadata about the objects. The retrieved class data may include, for example, the expected dimensions of objects of the given class (e.g., size, shape, color), a reference 3-D model (e.g., a 3-D model of a canonical instance of the class (e.g., the expected shape of a manufactured part)), one or more defect detection models (e.g., models, such as convolutional neural networks, trained to detect defects in the object based on the captured 3-D model) and the like….” AND Paragraph [0155]- “FIG. 15 is a flowchart of a method for detecting defects based on descriptors of locations of features of a target object according to one embodiment of the present invention. In the embodiment shown in FIG. 15, three different types of analyses are performed on an input 3-D model of the object based on retrieved data. ..”) .
As per claim 6, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 1,
Dal Mutto et al. teaches wherein the receiving step includes scanning the unique identifier located on the previously finished product ( Paragraph [0230]- “…the objects 10 are scanned by the scanning system 99 (e.g., the analysis results are also stored or buffered in a queue (first-in-first-out) data structure, where an analysis result generated by the inspection system 300 are added to the tail of the queue and an analysis result is taken from the head of the queue when a new object is detected by the sensing components). In some embodiments of the present invention, each object 10 is associated with a visual code (e.g., a 1-D barcode, a 2-D barcode such as a QR code, or other identifier), where the visual code may be applied directly to the object itself or to a portion of the conveying system configured to carry the object (e.g., applied to a tray holding the object or printed onto the conveyor belt 12 carrying the objects). When the object 10 is scanned by the scanning system 99, the visual code is captured and the visual code (or the value encoded therein) is associated with the captured 3-D model….”) .
As per claim 7, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 6,
Dal Mutto et al. teaches wherein the discovering step includes positioning a portion of the previously finished product under a camera to obtain a captured product image ( Paragraph [0125]- “…multiple separate computer systems, some of which may be local to the scanning of the query objects (e.g., on-site and connected directly to the depth and color cameras, or connected to the depth and color cameras over a local area network), and some of which may be remote (e.g., off-site, “cloud” based computing resources connected to the depth and color cameras through a wide area network such as the Internet). For the sake of convenience, the computer systems configured using particular computer instructions to perform purpose specific operations for inspecting target objects based on captured images of the target objects are referred to herein as parts of inspection agents or inspection systems….”) .
As per claim 8, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 7,
Dal Mutto et al. teaches wherein the previously finished product is unmodified during the positioning step ( Paragraph [0105]- “…Active projection sources can also be classified as projecting static patterns, e.g., patterns that do not change over time, and dynamic patterns, e.g., patterns that do change over time. In both cases, one aspect of the pattern is the illumination level of the projected pattern. This may be relevant because it can influence the depth dynamic range of the depth camera system…”) .
As per claim 9, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 7,
Dal Mutto et al. teaches wherein a portion of the previously finished product is removed prior to performing the positioning step ( Paragraphs [0077-0078]- “…manufactured objects can automate inspection activities that might otherwise be performed manually by a human, and therefore can improve the quality control process by, for example, reducing or removing errors made by human inspectors, reducing the amount of time needed to inspect each object, and enabling the analysis of a larger number of produced objects (e.g., inspecting substantially all produced objects as opposed to sampling from the full set of the manufactured objects and inspecting only the manufactured subset)….”) .
As per claim 14, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 1,
Dal Mutto et al. teaches wherein the modifying step includes training a machine learning (ML) model with the finished product dataset ( Paragraphs [0134-0135]- “… the neural network is trained based on training data, which may include a set of 3-D models of objects and their corresponding labels (e.g., the correct classifications of the objects). A portion of this training data may be reserved as cross-validation data to further adjust the parameters of during the training process, and a portion may also be reserved as a test data to confirm that the network is properly trained.”) .
As per claim 16, Combination of Dal Mutto et al. and Huy Toan Nguyen et al. teaches claim 1,
Dal Mutto et al. teaches wherein the modifying step includes updating an inspector log of the previously finished product to reflect the revised feature class ( Paragraphs [0173-0174]- “…training set also includes input data that is synthesized by modifying the 3-D scans of the actual defective and clean objects and/or by modifying a reference model. These modifications may include introducing blemishes and defects similar to what would be observed in practice. As a specific example, one of the scanned actual defective objects may be a shoe that is missing a grommet in one of its eyelets….”) .
It is noted that any citations to specific, pages, columns, lines, or figures in the prior art references and any interpretation of the reference should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. See MPEP 2123.
Allowable Subject Matter
2. Claims 10,11,12,13 and 15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
3. The following is an examiner’s statement of reasons for objecting the claims as allowable subject matter:
As to claim 10 , prior art of record does not teach or suggest the limitation mentioned within claim 10 : “…detecting with a machine learning (ML) model at least one object in the captured product image and providing for each detected object an identification of a class of the detected object and an identification of a region of the detected object in the captured product image, wherein the class of the detected object is either an acceptable product feature class or an unacceptable product feature class; receiving for each detected object, the identification of the class of the detected object and the identification of the region of the detected object in the captured product image; and displaying at an inspection station an enhanced product image that includes a conveyed captured product image to which the identification of the class of the detected object and the identification of the region of the detected object in the captured product image for each detected object added.”
As to claim 11 , Claim 11 depends on objected allowable claim 10, therefore claim 11 is considered objected allowable over prior art of record.
As to claim 12 , Claim 12 depends on objected allowable claim 11, therefore claim 12 is considered objected allowable over prior art of record.
As to claim 13 , Claim 13 depends on objected allowable claim 12, therefore claim 13 is considered objected allowable over prior art of record.
As to claim 15, prior art of record does not teach or suggest the limitation mentioned within claim 15 : “…loading a training dataset including the finished product dataset into a server computer system, wherein the training dataset comprises training product images and, for each training product image of the training product images, an identification of a class of an object in the training product images and an identification of a region of the object in the training product images, wherein the class of the object is either an acceptable product feature class or an unacceptable product feature class; and training the ML model using the loaded training dataset.”
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Examiner’s Notes
4. The Examiner acknowledges the following prior arts below as pertinent to the current applications claim limitations and inventive concept, although the following prior arts shown below were not relied upon to address the limitations within the claim , they are analogous art mentioning the inventive concept key points on (Manufacturing object quality assessment, machine learning , training data ,feature classification , image processing , detection of object defects etc.).
1) Adriana Birlutiu et al. ," Defect Detection in Porcelain Industry based on Deep Learning Techniques," 11th November 2018, 2017 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC),Pages 263-268.
2) Daniel Weimer et al.," Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection,"30th May 2016, CIRP Annals - Manufacturing Technology 65 (2016) ,Pages 417-419.
3) Paolo Sassi et al.," A Smart Monitoring System for Automatic Welding Defect Detection," 31st July 2019, IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 66, NO. 12, DECEMBER 2019, Pages 9641-9647.
4) Xiu-Shen Wei et al.," RPC: A Large-Scale Retail Product Checkout Dataset," 22nd Jan. 2019, arXiv:1901.07249v1, Pages 1-17.
5) Aqsa Rasheed et al. ," Fabric Defect Detection Using Computer Vision Techniques: A Comprehensive Review," 16th November 2020, Hindawi Mathematical Problems in Engineering,Volume 2020, Article ID 8189403,Pages 1-21.
6) Goncalo San‑Payo et al., " Machine learning for quality control system,"16th December 2019, Journal of Ambient Intelligence and Humanized Computing (2020) 11,Pages 896-900.
7) JING YANG et al. , " Real-Time Tiny Part Defect Detection System in Manufacturing Using Deep Learning,"22nd July 2019 , IEEEAccess, VOLUME 7, 2019,Pages 89278-89286.
8) Gokturk et al. (USPUB 20080144943)
9) Ribnick et al. (USPAT 9031312)
8) CHO et al. (USPUB 20190236772)
10) Dal Mutto et al. (USPUB 20190108396 )
11) Shah et al. (USPUB 20200160497 )
12) CELLA et al. (USPUB 20200348662)
13) Liu( USPUB 20220020138 )
14) Palme et al. (USUB 20230011901 )
15) YANG (CN 117011213)
Conclusion
5. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Refer to PTO-892, Notice of Reference Cited for a listing of analogous art.
6. Any inquiry concerning this communication or earlier communications from the examiner should be directed to OMAR S ISMAIL whose telephone number is (571)272-9799 and Fax # is (571)273-9799. The examiner can normally be reached on M-F 9:00am-6:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at
http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David C. Payne can be reached on (571) 272-3024. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free)? If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OMAR S ISMAIL/
Primary Examiner, Art Unit 2635
Prosecution Insights