DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Re claims 1, 14, 15, 30 and 59, while British English spellings for “recognise” and “neighbours” are not objectionable per MPEP 608.01, correction to “recognize” (and derivatives) and “neighbor” is suggested.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
2. Claims 1-27, 30, and 59-61 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1-4, 27, 30, and 59 recite a “reusable” recognition model. It is not clear to the extent that the model is reusable, P[0070] of the present Specification dated 10th May, 2024 provides “In one embodiment, each distinct corresponding reusable recognition model is configured to be repeatably applied to the plurality of images.”. However, “reusable” is also described in P[00111]: “the machine learning models of Lin et al., despite being applied to images with less than noise than is characteristic of those acquired in industrial applications, are not reusable between images of different ICs, or even different IC layers. That is, the systems and processes of Lin et al. require retraining for each new image to be processed”, and in P[00117]: “wherein the model is reusable on a plurality of images (i.e. is sufficiently robust to segment wires from a plurality of images, IC layers, images representative of different ICs, or the like)”, where “sufficiently robust” is a relative term of degree and indefinite, the Examiner understands this meaning of “reusable” to be commensurate with the general definition of iterative, which does not relate to the accuracy or validity of the output.
Claim 4 recites “and/or” which is indefinite, the Examiner is using the exclusive “or” in reference to claim 4, which requires only one of the plurality of features/options presented.
The term “optionally” in claim 11 is a relative term which renders the claim indefinite. The term “optionally” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. “refining” and “object merging process” in claim 11 are rendered indefinite.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
3. Claims 1-4, 9-10, 16, 25, 27, 30, and 59-61 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by “An Adaptive Deep Learning Framework for Fast Recognition of Integrated Circuit Markings” by Zhongshu Chen et al., (herein after “Chen”).
Regarding claim 1, as best understood, An image analysis method for recognising each of a plurality of object types in an image, the method to be executed by at least one digital data processor in communication with a digital data storage medium having the image stored thereon, the method comprising (Chen, §I, P[007]): “The framework is composed of four main components, each of which contains a customized deep convolutional network and some auxiliary image processing methods.”, for recognizing a plurality of object types, which are characters, §I, P[009]: “the characters to be recognized include the capital Latin alphabet and the Arabic numerals. Nonetheless, the charset can be readily extended with sufficient related data.”).
accessing a digital representation of at least a portion of the image (Chen, Fig. 2, a raw image of the chip is segmented, “Chip Segmentation Component”).;
by a first reusable recognition model associated with a first machine learning architecture, recognising objects of a first object type of the plurality of object types in the digital representation (Chen, §III, §§D, Fig. 9 discloses the reusable recognition model for a first object type of the plurality of types, see “Character Arrangement and Recognition”.);
by a second reusable recognition model associated with a second machine learning architecture, recognising objects of a second object type of the plurality of object types in the digital representation (Chen, §I, P[003]: “the deep-learning-based text detection approaches can be divided into objection-based models and segmentation based models.”, and Fig. 12 further discloses “Performance of the OCR models on the images of different IC types; the purple dashed line depicts the H-mean of our proposed framework.”.);
outputting respective first and second object datasets representative of objects of said first and second object types in the digital representation of the image (Chen, §II, §§C, “The output is a map on the same scale as the input, with each pixel classified as a label corresponding to the input image. Furthermore, each feature map in the expansive path is concatenated to its same dimensional counterpart in the contracting path. The concatenation is used to replenish some important features lost during the down sampling steps. The U-Net [19] is a typical U-shaped CNN model, and it is originally designed for segmenting biomedical images that have several same features with IC images.”, and further in §III, §§B, “an image is input into the model to get a single label, i.e., the marking orientation, as output. The labels represent the text orientations on chip images and their relationship is shown in Fig. 4(a).”).
Regarding claim 2, as best understood, wherein one or more of said first or second reusable recognition model comprises a segmentation model or an object detection model is disclosed by Chen in Fig. 2., “Chip Segmentation Component” and “Character Recognition Component”.
Regarding claim 3, as best understood, wherein said first reusable recognition model comprises a segmentation model and said second reusable recognition model comprises an object detection model is disclosed by Chen in Fig. 2., where the first model is the “Chip Segmentation Component” for segmentation and the second model is the “Character Recognition Component” for object detection.
Regarding claim 4, as best understood, wherein one or more of said first or second reusable recognition model comprises: a user-tuned parameter-free recognition model; a generic recognition model; and/or a convolutional neural network recognition model. Chen discloses a convolutional neural network in Fig. 3 which utilizes U-Net for segmentation.
Regarding claim 9, as best understood, wherein the digital representation comprises each of a plurality of image patches corresponding to respective regions of the image is disclosed by Chen in Fig. 3, where it discloses outputting the “Segmentation map”, which is a digital representation of a plurality of image patches corresponding to respective regions.
Regarding claim 10, as best understood, further comprising defining said plurality of image patches is disclosed by Chen in Fig. 3, where it discloses that the “Segmentation map” is utilized to output a defined, cropped image containing characters. The cropped image containing characters is input into the “Extraction component” of Fig. 2 and the characters are defined.
Regarding claim 16, as best understood, further comprising post-processing at least some of said objects in accordance with a refinement process is disclosed by Chen in Fig. 2 where the final output shows refined characters.
Regarding claim 25, as best understood, further comprising combining the first and second object datasets into a combined dataset representative of the image is disclosed by Chen in Fig. 2 where the output of the refined characters is a combination of the output of the segmentation process and the output of the character recognition process which is representative of the input image.
Regarding claim 27, as best understood, further comprising independently training said first and second reusable recognition models is disclosed by Chen in §III, §§, P[002]: “Fig. 5 shows the network architecture and the training process for the detection model.”, which is independently trained along with the other components shown in Fig. 2, where each component has its own model.
Regarding claim 30, as best understood, An image analysis method for recognising each of a plurality of object types of interest in an image, the method to be executed by at least one digital data processor in communication with a digital data storage medium having the image stored thereon, the method comprising (Chen, §I, P[007]): “The framework is composed of four main components, each of which contains a customized deep convolutional network and some auxiliary image processing methods.”, for recognizing a plurality of object types, which are characters, §I, P[009]: “the characters to be recognized include the capital Latin alphabet and the Arabic numerals. Nonetheless, the charset can be readily extended with sufficient related data.”).:
accessing a digital representation of the image (Chen, Fig. 2, a raw image of the chip is segmented, “Chip Segmentation Component”).;
for each object type of interest, recognising each object of interest in the digital representation by a corresponding reusable object recognition model associated with a corresponding respective machine learning architecture (Chen, Fig. 2, where the chips are the objects of interest and are segmented and their orientation is corrected for aiding in character recognition.);
outputting respective object datasets representative of respective objects of interest corresponding to each object type of interest in the digital representation of the image (Chen, §III, §§B, “an image is input into the model to get a single label, i.e., the marking orientation, as output. The labels represent the text orientations on chip images and their relationship is shown in Fig. 4(a).”).
Regarding claim 59, as best understood, An image analysis system for recognising each of a plurality of object types of interest in an image, the system comprising (Chen, Fig. 2, where the chips are the objects of interest and are segmented and their orientation is corrected for aiding in character recognition.):
a digital data processor operable to execute object recognition instructions (Chen, §II, §§C: “The architecture of the U-Net has great advantages on processing IC images and is able to yield precise segmentation with a few samples.”);
at least one digital image database comprising the image to be analysed for the plurality of object types, the at least one digital image database being accessible to the digital data processor (Chen, §II, §§C: “The architecture of the U-Net has great advantages on processing IC images and is able to yield precise segmentation with a few samples.”, where the U-Net requires storage and the ability to process images contained in a memory/database.);
a digital storage medium having stored thereon, for each of the plurality of object types, a distinct corresponding reusable recognition model deployable by the digital data processor and associated with a corresponding distinct machine learning architecture (Chen, Fig. 9 discloses a specific CNN/reusable recognition model with a distinct architecture for the character recognition component found within Fig. 2 of Chen which recognizes the specific characters found on the chips.); and
a non-transitory computer-readable medium comprising the object recognition instructions which, when executed by the digital data processor, are operable to, for each designated type of the plurality of object types of interest (where the model must be saved/stored on a memory to be operable.):
access a digital representation of at least a portion of the image from the at least one digital image database (Chen, Fig. 3 discloses segmenting the images into a plurality of portions before being used for “Character Recognition” which is disclosed in Fig. 2.);
recognise at least one object of the designated type in the digital representation by deploying the distinct corresponding reusable recognition model (Chen, Fig. 2, “Character Recognition”.);
output a respective object dataset representative of objects of said designated type in the digital representation of the image (Chen, Fig. 2, see output of “Character Recognition Component”, which is a respective object data set.).
Regarding claim 60, as best understood, further comprising a digital output storage medium accessible to the digital data processor for storing each said respective object dataset corresponding to each said designated type of the plurality of object types of interest is disclosed by Chen, §IV, §§A: “The raw dataset contains 7316 IC images scanned from a real laptop manufacturing line by an industrial camera.”, where the images are contained in a data storage accessible by the algorithm as shown in Fig. 2, where it shows “Data” which contains the raw images and the output images.
Regarding claim 61, as best understood, wherein the digital data processor is operable to repeatably execute said object recognition instructions for a plurality of images is disclosed by Chen in the §Abstract: “Experiments from the chip image dataset of a real laptop manufacturing line reached a recognition Precision of 91.73% and the Recall of 92.93%.”, and §IV, §§A: “The raw dataset contains 7316 IC images scanned from a real laptop manufacturing line by an industrial camera.”, where the process occurs for each of the images in the set, making it iterative.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of “Optical character recognition in real environments using neural networks and k-nearest neighbor” by O. Matei et al., (herein after “Matei”).
Regarding claim 17, as best understood, wherein said refinement process comprises a convolutional refinement process comprising a k-nearest neighbours (k-NN) refinement process, where Chen discloses a convolutional refinement process for character recognition in Fig. 9. Chen does not explicitly disclose utilizing a k-nearest neighbor technique for their character recognition task, which provides refined characters from optical character recognition.
However, Matei discloses utilizing a convolutional neural network and a k-nearest neighbor refinement process for optical character recognition, where the k-nearest neighbor technique is utilized to further refine the data through confirmation, Matei, §Abstract: “Our method combines two algorithms an artificial neural network on one hand, and the k-nearest neighbor as the confirmation algorithm.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chen to utilize a k-nearest neighbor technique for refinement of data, as taught by Matei, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. It is predictable that the proposed modification would have provided the benefit of improving accuracy of the recognized digits/characters.
5. Claims 21 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of “Deep Learning-Based Image Analysis Framework for Hardware Assurance of Digital Integrated Circuits” by Tong Lin et al., (herein after “Lin”).
Regarding claim 21, as best understood, Chen discloses wherein the image is representative of an integrated circuit (IC) (Chen, §Abstract: “This article develops an adaptive deep learning framework to facilitate the fast marking recognition of IC chips.”), and one or more of said first or second object type comprises a wire, a via, a polysilicon area, a contact, or a diffusion area.
However, Lin discloses and one or more of said first or second object type comprises a wire, a via, a polysilicon area, a contact, or a diffusion area in Fig. 1, where it shows the polysilicon layer/area along with the metal lines/wires and vias.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Chen to include wires, vias, and layers of the IC as object types, as taught by Lin, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. It is predictable that the proposed modification would have provided the benefit of improving the accuracy in determination of automatically proposed hardware defects.
Regarding claim 23, as best understood, wherein the image comprises an electron microscopy image is disclosed by Lin in the §Abstract: “Our aim is to examine and verify various hardware information from analyzing the Scanning Electron Microscope (SEM) images of an IC.”
Allowable Subject Matter
6. Claims 11 and 14-16 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims. This would include replacing “and, optionally,” with “wherein”.
Conclusion
7. Any inquiry concerning this communication or earlier communications from the examiner should be directed to TY M BEATTY whose telephone number is (703)756-5370. The examiner can normally be reached Mon-Fri: 8AM-4PM EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571) 272 - 3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TY MITCHELL BEATTY/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights