DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/06/2026 has been entered.
Claim Status
Claims 1-8 and 14-20 are pending for examination in the application filed 02/06/2026. Claims 1 and 14 are currently amended. Claims 9-13 were withdrawn.
Response to Arguments and Amendments
Applicant's arguments filed 02/06/2026 regarding Ferrell have been fully considered but they are not persuasive. Applicant argues on pages 10-11 of the Remarks that “the relevance feedback weights the query process more heavily towards characteristics the user is attempting to locate. Ferrell therefore does not teach or suggest ‘identifying the first substrate processing defect based on comparing the first image data with the second image data’ as recited in amended claim 1”. Ferrell teaches: [col. 3 ln. 35-38] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. [col. 5 ln 6-18] Specifically, the manufacturing image can be characterized in terms of image modality and overall characteristics, substrate-background characteristics, and anomoly-defect charcteristics. Moreover, the characteristics used to describe the modality, background, and defect are based on the texture, color, and shape of the image. In the preferred embodiment, the image feature extraction module 2 pre-processes every image in the image database 5 to generate a series of vectors having these descriptive set of features, each vector weighted to a particular characteristic of the stored image. Subsequently, the image feature extraction module 2 can store each of the series of vectors in a corresponding feature vector list 7, contained as part of the image database 5. [col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. Please see the updated 35 USC 102 and 35 USC 103 rejections below.
Applicant’s remaining arguments filed 02/06/2026 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 6, and 14-15 rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ferrell (US6751343B1).
Regarding claim 1, Ferrell teaches a method, comprising: storing, in a data storage device, a plurality of feature vectors representative of previously processed image frames that correspond to various substrate processing defects ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic. [col. 3 ln. 34-35] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved);
receiving, by a processing device, first image data (query image) comprising one or more image frames indicative of a first substrate processing defect; determining, by the processing device, a first feature vector corresponding to the first image data ([col. 3 ln. 34-41] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. More particularly, the retrieving step includes several steps. First, the query image is converted into at least one query vector corresponding to a particular characteristic of the manufacturing image. [col. 5 ln. 44-47] Significantly, the image feature extraction module 2, using the query image as an input, can generate the associated vector of numerical descriptors. [col. 6 ln. 18-23] the image feature extraction module 2 can define three unique and approximately independent feature vectors of manufacturing-specific digital imagery. The three feature vectors include an anomaly/defect characteristic, a substrate/background characteristic, and an image modality and global characteristic);
determining, by the processing device, a selection of the plurality of feature vectors based on a proximity between the first feature vector and each of the selection of the plurality of feature vectors ([col. 5 ln. 47-50] Using the vector of numerical descriptors as a guideline, a very rapid traversal of indexing tree 6 in the first-level data reduction routine can produce a preliminary selection of matching images from the image database 5. [col. 9 ln. 27-47] Having created a series of feature vector lists 7 for images 8 stored in image database 5, the manufacturing-specific CBIR system 1 can initiate an indexing process which organizes the data into a hierarchical tree structure 6 allowing for rapid retrieval of imagery during the query process. Specifically, the manufacturing-specific CBIR system 1 employes feature indexing--a process by which the manufacturing-specific CBIR system 1 organizes feature vectors contained in each feature vector list 7 to facilitate rapid access and retrieval of similar imagery. The indexing technique is critical to the efficient retrieval of similar data from the software system. Unlike existing CBIR systems, the manufacturing-specific CBIR system 1 utilizes a sequential, agglomerative, hierarchical, non-overlapping [SAHN] algorithm for sorting the features contained in the feature vector list 7. Historically, the SAHN algorithm has been used as an investigative tool for unsupervised clustering in pattern recognition problems. For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation);
determining, by the processing device, second image data (manufacturing-specific digital images) different from the first image data, the second image data comprising one or more previously processed image frames from the data storage device corresponding to embedding data associated with the selection of the plurality of feature vectors, wherein the second image data indicates one or more additional substrate processing defects matching the first substrate processing defect ([col. 3 ln. 35-38] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. [col. 5 ln 6-18] Specifically, the manufacturing image can be characterized in terms of image modality and overall characteristics, substrate-background characteristics, and anomoly-defect charcteristics. Moreover, the characteristics used to describe the modality, background, and defect are based on the texture, color, and shape of the image. In the preferred embodiment, the image feature extraction module 2 pre-processes every image in the image database 5 to generate a series of vectors having these descriptive set of features, each vector weighted to a particular characteristic of the stored image. Subsequently, the image feature extraction module 2 can store each of the series of vectors in a corresponding feature vector list 7, contained as part of the image database 5);
and performing, by the processing device, an action based on determining the second image data, wherein the action comprises identifying the first substrate processing defect based on comparing the first image data with the second image data ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Regarding claim 2, Ferrell teaches the method of claim 1. Ferrell further teaches identifying, by the processing device, the first substrate processing defect based on the second image data, wherein the action is further based on an identity of the first substrate processing defect ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Regarding claim 6, Ferrell teaches the method of claim 1. Ferrell further teaches receiving, by the processing device, a first selection of a first image frame of the first image data ([col. 5 ln. 37-40] The third module forming the manufacturing-specific CBIR system 1, a querying module 4, can accept a query image from a user and can return to the user, a collection of similar images stored in the image database 5);
generating, by the processing device, a second image frame by masking a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame ([col. 6 ln. 23-31] The defect region can undergo additional masking of morphological features which include an inner mask, which is a solid white-on-black mask identifying the defect; an annular mask, consisting of one pixel inside the inner mask border, one pixel on the border, and one pixel outside the border; and, an outer mask, surrounding the outside of the annular mask by a specified number of pixels. Generally, only the inner mask is required for the feature extraction. [col. 6 ln. 38-44] The characteristics forming each feature vector as extracted by the image feature extraction module 2 can be grouped into descriptive categories: color, texture, and shape. Each feature contributing to the vector description can be generated by masking the image 8 and plane with an appropriate mask and measuring the feature according to appropriate equations).
Regarding claim 14, Ferrell teaches a non-transitory machine-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising ([col. 15 ln. 63-64] A computer apparatus programmed with a routine set of instructions stored in a fixed medium):
store, in a data storage device, a plurality of feature vectors representative of previously processed image frames that correspond to various substrate processing defects ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic);
receive first image data (query image) comprising one or more image frames indicative of a first substrate processing defect; determine a first feature vector corresponding to the first image data ([col. 3 ln. 34-41] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. More particularly, the retrieving step includes several steps. First, the query image is converted into at least one query vector corresponding to a particular characteristic of the manufacturing image. [col. 5 ln. 44-47] Significantly, the image feature extraction module 2, using the query image as an input, can generate the associated vector of numerical descriptors. [col. 6 ln. 18-23] the image feature extraction module 2 can define three unique and approximately independent feature vectors of manufacturing-specific digital imagery. The three feature vectors include an anomaly/defect characteristic, a substrate/background characteristic, and an image modality and global characteristic);
determine a selection of the plurality of feature vectors based on a proximity between the first feature vector and each of the selection of the plurality of feature vectors ([col. 5 ln. 47-50] Using the vector of numerical descriptors as a guideline, a very rapid traversal of indexing tree 6 in the first-level data reduction routine can produce a preliminary selection of matching images from the image database 5. [col. 9 ln. 27-47] Having created a series of feature vector lists 7 for images 8 stored in image database 5, the manufacturing-specific CBIR system 1 can initiate an indexing process which organizes the data into a hierarchical tree structure 6 allowing for rapid retrieval of imagery during the query process. Specifically, the manufacturing-specific CBIR system 1 employes feature indexing--a process by which the manufacturing-specific CBIR system 1 organizes feature vectors contained in each feature vector list 7 to facilitate rapid access and retrieval of similar imagery. The indexing technique is critical to the efficient retrieval of similar data from the software system. Unlike existing CBIR systems, the manufacturing-specific CBIR system 1 utilizes a sequential, agglomerative, hierarchical, non-overlapping [SAHN] algorithm for sorting the features contained in the feature vector list 7. Historically, the SAHN algorithm has been used as an investigative tool for unsupervised clustering in pattern recognition problems. For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation);
determine second image data (manufacturing-specific digital images) different from the first image data, the second image data comprising one or more previously processed image frames from the data storage device corresponding to embedding data associated with the selection of the plurality of feature vectors, wherein the second image data indicates one or more additional substrate processing defects matching the first substrate processing defect ([col. 3 ln. 35-38] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. [col. 5 ln 6-18] Specifically, the manufacturing image can be characterized in terms of image modality and overall characteristics, substrate-background characteristics, and anomoly-defect charcteristics. Moreover, the characteristics used to describe the modality, background, and defect are based on the texture, color, and shape of the image. In the preferred embodiment, the image feature extraction module 2 pre-processes every image in the image database 5 to generate a series of vectors having these descriptive set of features, each vector weighted to a particular characteristic of the stored image. Subsequently, the image feature extraction module 2 can store each of the series of vectors in a corresponding feature vector list 7, contained as part of the image database 5);
and perform an action based on determining the second image data, wherein the action comprises identifying the first substrate processing defect based on comparing the first image data with the second image data ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Regarding claim 15, Ferrell teaches the medium of claim 14. Ferrell further teaches identify the first substrate processing defect based on the second image data, wherein the action is further based on an identity of the first substrate processing defect ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 3-4 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell in view of Sawlani (WO2020146708A1).
Regarding claim 3, Ferrell teaches the method of claim 2. Sawlani, in the same field of endeavor of substrate defect identification, teaches identifying, by the processing device, an instance of abnormality of a fabrication process associated with the first substrate processing defect ([0005] a defect analysis computational system is provided, which system may be characterized by the following features: (a) one or more processors; and (b) program instructions for executing on the one or more processors…receive manufacturing information, determine, using the first stage defect classification and the manufacturing information, one or more sources of the defects on the substrate , and output a likelihood of the defects being caused by a first source associated with the manufacturing equipment and/or fabrication process); and
causing, by the processing device, performance of a corrective action associated with fabrication process equipment based on the instance of abnormality ([0009] In various embodiments, the second stage defect classification engine is additionally configured to further classify the defects on the substrate and/or provide suggested corrective actions to reduce generation of defects on the substrate and/or reduce occurrences of defects on substrates processed in the future).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Sawlani to identify the fabrication process associated with the defect so that "a bridging defect could be corrected by etching the substrate to remove the bridged material before proceeding with the next fabrication process" [Sawlani 0081].
Regarding claim 4, Ferrell teaches the method of claim 1. Sawlani teaches preparing, by the processing device, one or more image frames of the second image data ([0069] Morphology data may include images of specific defects and can be used to classify the size and/or shape of defects. [0076] Figure 5 shows morphology data for small particle defects 501 and for large particle defects 502, I an example embodiment, a first stage defect classification engine processes all ten images to produce one or more first stage defect classifications. In general, a first stage defect classification engine may process every image taken of a defect, which could be a single image or multiple images) for presentation on a graphical user interface (GUI) ([0118] As shown, computer system 800 includes an input/output subsystem 802, which may implement an interface for interacting with human users and/or other computer systems depending upon the application. Embodiments of the invention may be implemented in program code on system 800 with I/O subsystem 802 used to receive input program statements and/or data from a human user (e.g., via a GUI or keyboard) and to display them back to the user. The I/O subsystem 802 may include, e.g., a keyboard, mouse, graphical user interface, touchscreen, or other interfaces for input, and, e.g.. an LED or other flat screen display, or other interfaces for output).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Sawlani to present the image "to receive input program statements and/or data from a human user" [Sawlani 0118].
Regarding claim 16, Ferrell teaches the medium of claim 15. Sawlani, in the same field of endeavor of substrate defect identification, teaches identify an instance of abnormality of a fabrication process associated with the first substrate processing defect ([0005] a defect analysis computational system is provided, which system may be characterized by the following features: (a) one or more processors; and (b) program instructions for executing on the one or more processors…receive manufacturing information, determine, using the first stage defect classification and the manufacturing information, one or more sources of the defects on the substrate , and output a likelihood of the defects being caused by a first source associated with the manufacturing equipment and/or fabrication process); and
and cause performance of a corrective action associated with fabrication process equipment based on the instance of abnormality ([0009] In various embodiments, the second stage defect classification engine is additionally configured to further classify the defects on the substrate and/or provide suggested corrective actions to reduce generation of defects on the substrate and/or reduce occurrences of defects on substrates processed in the future).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Sawlani to identify the fabrication process associated with the defect so that "a bridging defect could be corrected by etching the substrate to remove the bridged material before proceeding with the next fabrication process" [Sawlani 0081].
Regarding claim 17, Ferrell teaches the medium of claim 14. Sawlani teaches prepare one or more image frames of the second image data ([0069] Morphology data may include images of specific defects and can be used to classify the size and/or shape of defects. [0076] Figure 5 shows morphology data for small particle defects 501 and for large particle defects 502, I an example embodiment, a first stage defect classification engine processes all ten images to produce one or more first stage defect classifications. In general, a first stage defect classification engine may process every image taken of a defect, which could be a single image or multiple images) for presentation on a graphical user interface (GUI) ([0118] As shown, computer system 800 includes an input/output subsystem 802, which may implement an interface for interacting with human users and/or other computer systems depending upon the application. Embodiments of the invention may be implemented in program code on system 800 with I/O subsystem 802 used to receive input program statements and/or data from a human user (e.g., via a GUI or keyboard) and to display them back to the user. The I/O subsystem 802 may include, e.g., a keyboard, mouse, graphical user interface, touchscreen, or other interfaces for input, and, e.g.. an LED or other flat screen display, or other interfaces for output).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Sawlani to present the image "to receive input program statements and/or data from a human user" [Sawlani 0118].
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell in view of Gu (US20070177794A1).
Regarding claim 5, Ferrell teaches the method of claim 1. Ferrell further teaches receiving, by the processing device, a first selection of a first image frame of the first image data ([col. 5 ln. 37-40] The third module forming the manufacturing-specific CBIR system 1, a querying module 4, can accept a query image from a user and can return to the user, a collection of similar images stored in the image database 5).
Ferrell does not teach generating, by the processing device, a second image frame by cropping a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame.
Gu, in the same field of endeavor of image feature detection, teaches generating, by the processing device ([0024] As shown in FIG. 2, the image processing unit 41), a second image frame by cropping a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame ([0031] In both eyeglasses detection learning stage and actual detection of eyeglasses stage, image data passes through edge enhancement unit 401, image size shrinking unit 403, nose ridge mask unit 124 and feature vector unit 415. Face detection and cropping unit 395 crops face images. Edge enhancement unit 401 enhances edges in face images and outputs edge maps. The sizes of the edge maps image are modified by the image size shrinking unit 403. The nose ridge mask unit 124 receives the edge maps from image size shrinking unit 403, and extracts image regions located in the nose ridge region of the faces shown in the edge maps. Feature vector unit 415 extracts feature vectors from the nose ridge image regions output from nose ridge mask unit 124).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Gu to crop the image prior to determining the feature vectors to "improve eyeglasses detection by performing face registration of the faces in the face images received from face image operations unit 122" [Gu 0025].
Regarding claim 18, Ferrell teaches the medium of claim 14. Ferrell further teaches receive a first selection of a first image frame of the first image data ([col. 5 ln. 37-40] The third module forming the manufacturing-specific CBIR system 1, a querying module 4, can accept a query image from a user and can return to the user, a collection of similar images stored in the image database 5).
Ferrell does not teach generate a second image frame by at least one of cropping or masking a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame.
Gu, in the same field of endeavor of image feature detection, teaches generate a second image frame by at least one of cropping or masking a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame ([0031] In both eyeglasses detection learning stage and actual detection of eyeglasses stage, image data passes through edge enhancement unit 401, image size shrinking unit 403, nose ridge mask unit 124 and feature vector unit 415. Face detection and cropping unit 395 crops face images. Edge enhancement unit 401 enhances edges in face images and outputs edge maps. The sizes of the edge maps image are modified by the image size shrinking unit 403. The nose ridge mask unit 124 receives the edge maps from image size shrinking unit 403, and extracts image regions located in the nose ridge region of the faces shown in the edge maps. Feature vector unit 415 extracts feature vectors from the nose ridge image regions output from nose ridge mask unit 124).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Gu to crop the image prior to determining the feature vectors to "improve eyeglasses detection by performing face registration of the faces in the face images received from face image operations unit 122" [Gu 0025].
Claims 7 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell in view of Nogami (US20200118263A1).
Regarding claim 7, Ferrell teaches the method of claim 1. Ferrell further teaches a substrate processing defect ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic).
Ferrell does not teach extracting, by the processing device, a selection of text from a first image frame of the first image data; determining, by the processing device, an image scaling factor associated with the first image frame; and determining, by the processing device based on the image scaling factor, a size associated with the first defect; wherein the selection of the plurality of feature vectors is determined further using the size.
Nogami, in the same field of endeavor of defect detection, teaches extracting, by the processing device, a selection of text from a first image frame of the first image data ([0082] the second extraction unit 115 may extract a histogram feature amount using a method known as BoW (Bag of Words) or BoF (Bag of Features). In this case, a plurality of visual words are generated from multiple ROIs in advance, and the frequency with which the visual words appear in the ROI image to be processed can then be converted to a histogram);
determining, by the processing device, an image scaling factor associated with the first image frame ([0048] Depending on the position or orientation of the image capturing device when the image of the structure is captured, it may not be possible for the image capturing device to capture an image from directly opposite the concrete wall surface, but such an image can be generated by carrying out a geometric conversion process on the image. It is also assumed that the input image 101 is subjected to processing for correcting lens distortion. Furthermore, it is assumed that the input image 101 is subjected to an enlargement or reduction process so that the image resolution is constant relative to the actual space. With these processes, for example, an image in which a single pixel corresponds to 1 mm of the concrete wall surface can be obtained. Such conversion and correction can be carried out by the information processing device 100); and
determining, by the processing device based on the image scaling factor, a size associated with the first defect ([0113] In step S206, the second extraction unit 115 first calculates the size of each defect);
wherein the selection of the plurality of feature vectors is determined further using the size ([0121] The feature amount at a given pixel in the feature map 906 is expressed by an n-dimensional vector 907. [0130] The size of the ROI changes depending on the size of the defect (the length of the crack). On the other hand, the attribute determination unit 116 determines the attributes (crack width) of the defect using a defect feature amount having a predetermined dimension. Thus in the present embodiment, the second extraction unit 115 generates a defect feature amount v.sub.930 for the ROI 930).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Nogami to determine the size of a defect and then determine the selection of plurality of feature vectors because "confirming the width of the crack makes it possible to determine the severity or level of danger posed by the crack" [Nogami 0004].
Regarding claim 19, Ferrell teaches the medium of claim 14. Ferrell further teaches a substrate processing defect ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic).
Ferrell does not teach extract a selection of text from a first image frame of the first image data; determine an image scaling factor associated with the first image frame; and determine, based on the image scaling factor, a size associated with the first defect; wherein the selection of the plurality of feature vectors is determined further using the size.
Nogami, in the same field of endeavor of defect detection, teaches extract a selection of text from a first image frame of the first image data ([0082] the second extraction unit 115 may extract a histogram feature amount using a method known as BoW (Bag of Words) or BoF (Bag of Features). In this case, a plurality of visual words are generated from multiple ROIs in advance, and the frequency with which the visual words appear in the ROI image to be processed can then be converted to a histogram);
determine an image scaling factor associated with the first image frame ([0048] Depending on the position or orientation of the image capturing device when the image of the structure is captured, it may not be possible for the image capturing device to capture an image from directly opposite the concrete wall surface, but such an image can be generated by carrying out a geometric conversion process on the image. It is also assumed that the input image 101 is subjected to processing for correcting lens distortion. Furthermore, it is assumed that the input image 101 is subjected to an enlargement or reduction process so that the image resolution is constant relative to the actual space. With these processes, for example, an image in which a single pixel corresponds to 1 mm of the concrete wall surface can be obtained. Such conversion and correction can be carried out by the information processing device 100); and
determine, based on the image scaling factor, a size associated with the first defect ([0113] In step S206, the second extraction unit 115 first calculates the size of each defect);
wherein the selection of the plurality of feature vectors is determined further using the size ([0121] The feature amount at a given pixel in the feature map 906 is expressed by an n-dimensional vector 907. [0130] The size of the ROI changes depending on the size of the defect (the length of the crack). On the other hand, the attribute determination unit 116 determines the attributes (crack width) of the defect using a defect feature amount having a predetermined dimension. Thus in the present embodiment, the second extraction unit 115 generates a defect feature amount v.sub.930 for the ROI 930).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Nogami to determine the size of a defect and then determine the selection of plurality of feature vectors because "confirming the width of the crack makes it possible to determine the severity or level of danger posed by the crack" [Nogami 0004].
Claims 8 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell in view of Morariu (US20240104951A1).
Regarding claim 8, Ferrell teaches the method of claim 1. Morariu, in the same field of endeavor of image embedding, teaches dividing, by the processing device, a first image frame of the first image data into a set of image patches corresponding to the first image frame ([0028] In various embodiments, the vision encoder 206 processes the input image as a sequence of patches. In one example, the input image I∈R3×H×W is decomposed into a batch of N=HW/P2 patches of size P×P, where H and W are a height and widths after resizing such that the values are divisible by the patch size P. In an embodiment, the patches are flattened (e.g., converted) into vectors which are linearly projected to generate the embeddings);
determining, by the processing device, a set of linear embeddings corresponding to a content of each image patch of the set of image patches ([0024] the input generator 124 includes an encoder of a transformer model that processes an image of the table 122 as a sequence of patches which are flattened into vectors and linearly projected to patch embeddings);
determining, by the processing device, a set of positional embeddings corresponding to a relative position of each of the image patches of the set of image patches ([0029] In an embodiment, the N embeddings include learned positional encodings that are added to the input of the attention layers of the decoder),
wherein the first feature vector is determined based on the set of linear embedding and the set of positional embeddings ([0029] In one example, the N embeddings are transformed into an output embedding by the split decoder 208. In addition, in an embodiment, the decoders (e.g., the split decoder 208 and OCR decoder 210) utilize seeds to generate candidate predictions (e.g., set of row and/or column split predictions 214). In an example, the seeds include a fixed length feature vector).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Morariu to determine linear and positional embeddings to determine "the output 128 is a feature vector (e.g., containing values from the input feature vectors transformed/encoded by the table recognition model 126) that is useable by the one or more task models 112 to perform various task associated with the document 120 and/or table 122…such as table question answering, table fact verification, table formatting, table layout, table identification, and table captioning" [Morariu 0025].
Regarding claim 20, Ferrell teaches the medium of claim 14. Morariu, in the same field of endeavor of image embedding, teaches divide a first image frame of the first image data into a set of image patches corresponding to the first image frame ([0028] In various embodiments, the vision encoder 206 processes the input image as a sequence of patches. In one example, the input image I∈R3×H×W is decomposed into a batch of N=HW/P2 patches of size P×P, where H and W are a height and widths after resizing such that the values are divisible by the patch size P. In an embodiment, the patches are flattened (e.g., converted) into vectors which are linearly projected to generate the embeddings);
determine a set of linear embeddings corresponding to a content of each image patch of the set of image patches ([0024] the input generator 124 includes an encoder of a transformer model that processes an image of the table 122 as a sequence of patches which are flattened into vectors and linearly projected to patch embeddings);
determine a set of positional embeddings corresponding to a relative position of each of the image patches of the set of image patches ([0029] In an embodiment, the N embeddings include learned positional encodings that are added to the input of the attention layers of the decoder),
wherein the first feature vector is determined based on the set of linear embedding and the set of positional embeddings ([0029] In one example, the N embeddings are transformed into an output embedding by the split decoder 208. In addition, in an embodiment, the decoders (e.g., the split decoder 208 and OCR decoder 210) utilize seeds to generate candidate predictions (e.g., set of row and/or column split predictions 214). In an example, the seeds include a fixed length feature vector).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Morariu to determine linear and positional embeddings to determine "the output 128 is a feature vector (e.g., containing values from the input feature vectors transformed/encoded by the table recognition model 126) that is useable by the one or more task models 112 to perform various task associated with the document 120 and/or table 122…such as table question answering, table fact verification, table formatting, table layout, table identification, and table captioning" [Morariu 0025].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights