Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-124 and 126-129 are rejected under 35 U.S.C. 102(a)(2) as being described by US20200226420A1 by Shaubi et al.
Shaubi teaches claim 121. A method for reducing usage of processing resources when training a plurality of neural networks to perform automated visual inspection for a plurality of respective defect categories, with each of the plurality of respective defect categories being associated with a respective feature of containers or container contents, the method comprising: (Shaubi abs “obtaining by a computer a Deep Neural Network (DNN) trained for a given examination-related application within a semiconductor fabrication process, processing together one or more fabrication process (FP) images using the obtained trained DNN, wherein the DNN is trained using a training set comprising synthetic images specific for the given application; and obtaining, by the computer, examination-related data specific for the given application, and characterizing at least one of the processed one or more FP images.” The container is the semiconductor specimen, because it contains the product. Applicant claims a plurality of Neural networks, but all of Applicant’s neural networks that discriminate are represented as one model 1204 in the fig. 12. Therefore, when Shaubi teaches a neural network (DNN) that classifies different defects and trains on different defect categories (Shaubi para 89), Shaubi is teaching the “plurality of neural networks” as claimed.)
obtaining, by one or more processors, a plurality of container images; (Shaubi para 87 “obtaining (401) “real world” training samples specific for a given examination-related application within a semiconductor fabrication process;”)
generating, by one or more processors processing the plurality of container images, a plurality of training image sets each corresponding to a different one of the plurality of container images, wherein for each training image set, (Shaubi para 87 ‘using the obtained “real world” training samples to generate (402) one or more synthetic images specific for the given examination-related application;”)
generating the training image set includes generating a different training image for each of the plurality of respective defect categories, (Shaubi para 87 ‘using the obtained “real world” training samples to generate (402) one or more synthetic images specific for the given examination-related application;” The defect category is the Shaubi’s examination-related application, Shaubi para 89 “enable tailoring the generated training set to the requirements of the specific application. By way of non-limiting example, lack of FAB data related to a specific class of defects (e.g. a minority class)…”)
generating a different training image for each of the plurality of respective defect categories includes generating a first training image for a first defect category associated with a first feature, and (Shaubi para 87 ‘using the obtained “real world” training samples to generate (402) one or more synthetic images specific for the given examination-related application;” The defect category is the Shaubi’s examination-related application, Shaubi para 89 “enable tailoring the generated training set to the requirements of the specific application. By way of non-limiting example, lack of FAB data related to a specific class of defects (e.g. a minority class)…” The minority class is the class of defects.)
generating the first training image includes
identifying the first feature in the container image that corresponds to the training image set, and (Shaubi para 89 “ lack of FAB data related to a specific class of defects (e.g. a minority class) can be compensated by synthetic images presenting the respective defects. By way of a further non-limiting example, lack of appropriate FAB data related to a specific layer can be compensated by synthetic images related to this layer.”)
generating the first training image such that the first training image (i) encompasses only a subset of the container image that corresponds to the training image set, and (ii) depicts the identified first feature; and (Shaubi para 89 “lack of appropriate FAB data related to a specific layer can be compensated by synthetic images related to this layer.”)
training, by one or more processors and using the plurality of training image sets, the plurality of neural networks to perform automated visual inspection for the plurality of defect categories. (Shaubi abs “obtained trained DNN, wherein the DNN is trained using a training set comprising synthetic images…”)
Shaubi teaches claim 122. The method of claim 121, wherein training the plurality of neural networks includes training each of the plurality of neural networks to infer a presence or absence of defects in a different one of the plurality of respective defect categories. (Shaubi para 89 “lack of FAB data related to a specific class of defects (e.g. a minority class) can be compensated by synthetic images presenting the respective defects. By way of a further non-limiting example, lack of appropriate FAB data related to a specific layer can be compensated by synthetic images related to this layer. By way of yet a further non-limiting example, synthetic images can be generated to recover missing information of a specific defect. For example, if a defect was missing only a high-resolution image, the synthetic image generator might be initialized with existing data and simulate the missing high resolution image.” The abstract says that the DNN is trained on these training sets.)
Shaubi teaches claim 123. The method of claim 122, wherein training each of the plurality of neural networks includes, for each of the plurality of training image sets, using a different training image to train a different one of the plurality of neural networks. (Applicant claims a plurality of Neural networks, but all of Applicant’s neural networks that discriminate are represented as one model 1204 in the fig. 12. Therefore, when Shaubi teaches a neural network (DNN) that classifies different defects and trains on different defect categories (Shaubi para 89), Shaubi is teaching the “plurality of neural networks” as claimed.)
Shaubi teaches claim 124. The method of claim 121, wherein identifying the first feature includes (i) identifying the first feature using template matching, or (ii) identifying the first feature using blob analysis. (Shaubi para 48 “segmentation of the fabrication process image including partitioning of FP image into segments (e.g. material types, edges, pixel labeling, regions of interest, etc.)…” Segmentation and Regions of interest are blobs.)
Shaubi teaches claim 126. The method of claim 121, wherein:
generating a different training image for each of the plurality of respective defect categories further includes generating a second training image for a second defect category associated with a second feature; and
generating the second training image includes generating the second training image such that the second training image depicts the second feature. (Shaubi para 89 “lack of FAB data related to a specific class of defects (e.g. a minority class) can be compensated by synthetic images presenting the respective defects. By way of a further non-limiting example, lack of appropriate FAB data related to a specific layer can be compensated by synthetic images related to this layer. By way of yet a further non-limiting example, synthetic images can be generated to recover missing information of a specific defect. For example, if a defect was missing only a high-resolution image, the synthetic image generator might be initialized with existing data and simulate the missing high resolution image.” The abstract says that the DNN is trained on these training sets.)
Shaubi teaches claim 127. The method of claim 126, wherein generating the second image includes generating the second image by down-sampling at least a portion of the container image that corresponds to the training image set to a lower resolution. (Shaubi para 76 “Augmentation of an image from a “real world” training sample can be provided by various image processing techniques including adding noise, blurring, geometric transformation (e.g. rotating, stretching, simulating different angles, cropping, scaling, etc.)…” A blur, a stretch, a crop and scaling all include decreasing resolution of the image.)
Shaubi teaches claim 128. The method of claim 126, wherein generating the second training image includes identifying the second feature in the container image that corresponds to the training image set. (Shaubi para 89 “lack of FAB data related to a specific class of defects (e.g. a minority class) can be compensated by synthetic images presenting the respective defects. By way of a further non-limiting example, lack of appropriate FAB data related to a specific layer can be compensated by synthetic images related to this layer. By way of yet a further non-limiting example, synthetic images can be generated to recover missing information of a specific defect. For example, if a defect was missing only a high-resolution image, the synthetic image generator might be initialized with existing data and simulate the missing high resolution image.”)
Shaubi teaches claim 129. The method of claim 121, wherein:
generating the plurality of training image sets each corresponding to a different one of the plurality of container images includes digitally aligning at least some of the plurality of container images, at least in part by resampling the at least some of the plurality of container images; and
digitally aligning at least some of the plurality of container images includes (i) detecting an edge within one or more of the plurality of container edges, and (ii) comparing a position of the detected edge to a position of a reference line. (Shaubi para 48 “defect classification using attributes generated by DNN (defining classes can include modifying and/or updating preexisting class definitions and/or identifying new classes); segmentation of the fabrication process image including partitioning of FP image into segments (e.g. material types, edges, pixel labeling, regions of interest, etc.);…”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 125 is rejected under 35 U.S.C. 103 as being unpatentable over US20200226420A1 to Shaubi et al and the Aimovig user manual (Amgen).
Shaubi teaches claim 125. The method of claim 121, wherein:
Shaubi doesn’t teach a syringe.
However, Amgen teaches all of these defect categories, including:
(1) the first feature is a meniscus of a fluid within a container, and the first defect category is a presence of one or more particles in or near the meniscus; (Amgen sec. 4 “Do not use the autoinjector if the medicine is cloudy, discolored, or contains flakes or particles.”)
(2) the plurality of container images is a plurality of syringe images, and either:
the first feature is a syringe plunger and the first defect category is one or both of (i) a plunger defect or (ii) a presence of one or more particles on the syringe plunger; (Amgen sec. 4 “Do not use the autoinjector if the medicine is cloudy, discolored, or contains flakes or particles.”)
the first feature is a syringe barrel and the first defect category is one or both of (i) a barrel defect or (ii) a presence of one or more particles within the syringe barrel; (Amgen sec. 4 “Do not use the autoinjector if the medicine is cloudy, discolored, or contains flakes or particles.”)
the first feature is a syringe needle shield and the first defect category is one or both of (i) an absence of the syringe needle shield or (ii) misalignment of the syringe needle shield; or (Amgen p. sec. 5 “Do not use the autoinjector if: [Symbol font/0xB7] the white cap is missing or loose in carton. [Symbol font/0xB7] it has cracks or broken parts, or [Symbol font/0xB7] it has been dropped on a hard surface.”)
the first feature is a syringe flange and the first defect category is one or both of (i) a malformed flange or (ii) a defect on the syringe flange; (Amgen p. sec. 5 “Do not use the autoinjector if: [Symbol font/0xB7] the white cap is missing or loose in carton. [Symbol font/0xB7] it has cracks or broken parts, or [Symbol font/0xB7] it has been dropped on a hard surface.”)
(3) the plurality of container images is a plurality of cartridge images, and either:
the first feature is a cartridge piston and the first defect category is one or both of (i) a piston defect or (ii) a presence of one or more particles on the cartridge piston; (Amgen p. sec. 5 “Do not use the autoinjector if: [Symbol font/0xB7] the white cap is missing or loose in carton. [Symbol font/0xB7] it has cracks or broken parts, or [Symbol font/0xB7] it has been dropped on a hard surface.”)
the first feature is a cartridge barrel and the first defect category is one or both of (i) a barrel defect or (ii) a presence of one or more particles within the cartridge barrel; or (Amgen p. sec. 5 “Do not use the autoinjector if: [Symbol font/0xB7] the white cap is missing or loose in carton. [Symbol font/0xB7] it has cracks or broken parts, or [Symbol font/0xB7] it has been dropped on a hard surface.”)
the first feature is a cartridge flange and the first defect category is one or both of (i) a malformed flange or (ii) a defect on the cartridge flange; or (Amgen p. sec. 5 “Do not use the autoinjector if: [Symbol font/0xB7] the white cap is missing or loose in carton. [Symbol font/0xB7] it has cracks or broken parts, or [Symbol font/0xB7] it has been dropped on a hard surface.”)
(4) the plurality of container images is a plurality of vial images, and either:
the first feature is a vial body and the first defect category is one or both of (i) a body defect or (ii) a presence of one or more particles within the vial body;
the first feature is a vial crimp and the first defect category is a defective crimp; or (Amgen p. sec. 5 “Do not use the autoinjector if: [Symbol font/0xB7] the white cap is missing or loose in carton. [Symbol font/0xB7] it has cracks or broken parts, or [Symbol font/0xB7] it has been dropped on a hard surface.”)
the first feature is a lyophilized cake and the first defect category is a crack or other defect of the lyophilized cake. (Amgen sec. 4 “Do not use the autoinjector if the medicine is cloudy, discolored, or contains flakes or particles.”)
Amgen, Shaubi and the claims are all directed to image processing to detect package defects. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to use Amgen’s defect categories to detect unsafe packages before the packages are shipped to consumers.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142