DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-8 and 14-20 are pending for examination in the application filed 06/24/2026. Claims 1, 5-8, 14, and 18-20 are currently amended. Claims 9-13 were withdrawn.
Response to Arguments and Amendments
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument, as facilitated by the newly added amendments.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 1 and 14, as amended, recite the limitations “wherein the first feature vector is determined based on: a first set of linear embeddings corresponding to a first set of image patches corresponding to a first image frame of the one or more image frames; and a first set of positional embeddings corresponding to a first relative position of a first content of each image patch of the set of image patches”. Claims 8 and 20, which depend from claims 1 and 14, respectively, recite the limitations “dividing, by the processing device, the first image frame of the first image data into a set of image patches corresponding to the first image frame; determining, by the processing device, a set of linear embeddings corresponding to a content of each image patch of the set of image patches; and determining, by the processing device, a set of positional embeddings corresponding to a relative position of each of the image patches of the set of image patches”. It is unclear if “a set of image patches” “a set of linear embeddings” and “a set of positional embeddings” in claims 8 and 20 are the same “a first set of linear embeddings”, “a first set of image patches”, and “a first set of positional embeddings” in amended independent claims 1 and 14. Please clarify.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 6, 8, 14-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell (US6751343B1) in view of Morariu (US20240104951A1) and Zhang (US20240004924A1).
Regarding claim 1, Ferrell teaches a method, comprising: storing, in a data storage device, a plurality of feature vectors representative of previously processed image frames that correspond to various substrate processing defects ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic);
receiving, by a processing device, first image data (query image) comprising one or more image frames indicative of a first substrate processing defect; determining, by the processing device, a first feature vector corresponding to the first image data ([col. 3 ln. 34-41] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. More particularly, the retrieving step includes several steps. First, the query image is converted into at least one query vector corresponding to a particular characteristic of the manufacturing image. [col. 5 ln. 44-47] Significantly, the image feature extraction module 2, using the query image as an input, can generate the associated vector of numerical descriptors. [col. 6 ln. 18-23] the image feature extraction module 2 can define three unique and approximately independent feature vectors of manufacturing-specific digital imagery. The three feature vectors include an anomaly/defect characteristic, a substrate/background characteristic, and an image modality and global characteristic);
determining, by the processing device, a selection of the plurality of feature vectors based on a proximity ([col. 5 ln. 47-50] Using the vector of numerical descriptors as a guideline, a very rapid traversal of indexing tree 6 in the first-level data reduction routine can produce a preliminary selection of matching images from the image database 5. [col. 9 ln. 27-47] Having created a series of feature vector lists 7 for images 8 stored in image database 5, the manufacturing-specific CBIR system 1 can initiate an indexing process which organizes the data into a hierarchical tree structure 6 allowing for rapid retrieval of imagery during the query process. Specifically, the manufacturing-specific CBIR system 1 employes feature indexing--a process by which the manufacturing-specific CBIR system 1 organizes feature vectors contained in each feature vector list 7 to facilitate rapid access and retrieval of similar imagery. The indexing technique is critical to the efficient retrieval of similar data from the software system. Unlike existing CBIR systems, the manufacturing-specific CBIR system 1 utilizes a sequential, agglomerative, hierarchical, non-overlapping [SAHN] algorithm for sorting the features contained in the feature vector list 7. Historically, the SAHN algorithm has been used as an investigative tool for unsupervised clustering in pattern recognition problems. For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation);
determining, by the processing device, second image data (manufacturing-specific digital images) different from the first image data, the second image data comprising one or more previously processed image frames from the data storage device corresponding to embedding data associated with the selection of the plurality of feature vectors, wherein the second image data indicates one or more additional substrate processing defects matching the first substrate processing defect ([col. 3 ln. 35-38] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. [col. 5 ln 6-18] Specifically, the manufacturing image can be characterized in terms of image modality and overall characteristics, substrate-background characteristics, and anomoly-defect charcteristics. Moreover, the characteristics used to describe the modality, background, and defect are based on the texture, color, and shape of the image. In the preferred embodiment, the image feature extraction module 2 pre-processes every image in the image database 5 to generate a series of vectors having these descriptive set of features, each vector weighted to a particular characteristic of the stored image. Subsequently, the image feature extraction module 2 can store each of the series of vectors in a corresponding feature vector list 7, contained as part of the image database 5);
and performing, by the processing device, an action based on determining the second image data, wherein the action comprises identifying the first substrate processing defect based on the second image data ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Ferrell does not explicitly teach wherein the first feature vector is determined based on: a first set of linear embeddings corresponding to a first set of image patches corresponding to a first image frame of the one or more image frames; and a first set of positional embeddings corresponding to a first relative position of a first content of each image patch of the set of image patches; determining, by the processing device, a selection of the plurality of feature vectors based on a proximity between the first set of linear embeddings and the first set of positional embeddings associated with the first feature vector and one or more second sets of linear embeddings and one or more second sets of positional embeddings associated with the selection of the plurality of feature vectors.
Morariu, in the same field of endeavor of image embedding, teaches wherein the first feature vector is determined based on: a first set of linear embeddings corresponding to a first set of image patches corresponding to a first image frame of the one or more image frames; and a first set of positional embeddings corresponding to a first relative position of a first content of each image patch of the set of image patches ([0024] the input generator 124 includes an encoder of a transformer model that processes an image of the table 122 as a sequence of patches which are flattened into vectors and linearly projected to patch embeddings. [0029] In an embodiment, the N embeddings include learned positional encodings that are added to the input of the attention layers of the decoder. [0029] In one example, the N embeddings are transformed into an output embedding by the split decoder 208. In addition, in an embodiment, the decoders (e.g., the split decoder 208 and OCR decoder 210) utilize seeds to generate candidate predictions (e.g., set of row and/or column split predictions 214). In an example, the seeds include a fixed length feature vector).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Morariu to determine linear and positional embeddings to determine "the output 128 is a feature vector (e.g., containing values from the input feature vectors transformed/encoded by the table recognition model 126) that is useable by the one or more task models 112 to perform various task associated with the document 120 and/or table 122…such as table question answering, table fact verification, table formatting, table layout, table identification, and table captioning" [Morariu 0025].
Zhang, in the same field of endeavor of image embedding retrieval, teaches wherein the first feature vector is determined based on: a first set of linear embeddings corresponding to a first set of image patches corresponding to a first image frame of the one or more image frames ([0232] To provide another illustration, in one or more embodiments, the search-based editing system 106 receives a text query, an image query, and a combination weight for conducting an image search; generates, utilizing the text embedding model, a text embedding for the text query within a text-image embedding space; generates, utilizing the image embedding model, an image embedding for the image query within the text-image embedding space; determines a linear combination of the text embedding and the image embedding utilizing the combination weight; and retrieves at least one digital image from the plurality of digital images based on a proximity of an embedding for the at least one digital image to the linear combination of the text embedding and the image embedding within the text-image embedding space. [0166] More particularly, the search-based editing system 106 separately encodes semantic information and layout information for distinct portions (e.g., patches or semantic segments) of the multi-modal search input 1220);
and a first set of positional embeddings corresponding to a first relative position of a first content of each image patch of the set of image patches ([0142] As further shown, the search-based editing system 106 utilizes an image embedding model 906 to generate an image embedding 908 from the image query 902. The image embedding 908 represents one or more visual features from the image query 902 (e.g., image characteristics or other patent or latent features of the image query 902, such as semantic information and/or layout information). [0165] Additionally, as shown in FIG. 12A, the search-based editing system 106 utilizes the encoder 1222 to generate regional semantic embeddings 1224 and regional layout embeddings 1226 from the multi-modal search input 1220. In one or more embodiments, a regional embedding includes an embedding generated from a particular region of an input, such as a multi-modal search input. For instance, in some cases, the search-based editing system 106 utilizes the encoder 1222 to generate the regional embeddings (i.e., the regional semantic embeddings 1224 and the regional layout embeddings 1226) from patches of the multi-modal search input 1220. In some cases, the search-based editing system 106 utilizes the encoder 1222 to generate the regional embeddings from semantic segments of the multi-modal search input 1220 (as will be discussed in more detail below with reference to FIG. 12B). [0166] By generating the regional semantic embeddings 1224 and the regional layout embeddings 1226, the search-based editing system 106 separately encodes semantic information and layout information from the multi-modal search input 1220. More particularly, the search-based editing system 106 separately encodes semantic information and layout information for distinct portions (e.g., patches or semantic segments) of the multi-modal search input 1220. [0167] In some implementations, however, the positional information 1230 is encoded within the regional layout embeddings 1226 generated from the multi-modal search input 1220);
determining, by the processing device, a selection of the plurality of feature vectors based on a proximity between the first set of linear embeddings and the first set of positional embeddings associated with the first feature vector and one or more second sets of linear embeddings and one or more second sets of positional embeddings associated with the selection of the plurality of feature vectors ([0231] In some cases, retrieving the one or more digital images utilizing the weighted combination of the text embedding and the image embedding comprises retrieving the one or more digital images using proximities of embeddings corresponding to the one or more digital images to the weighted combination. Further, in some instances, generating the text embedding for the text query and the image embedding for the image query comprises: generating, utilizing a text embedding model, the text embedding from the text query; and generating, utilizing an image embedding model, the image embedding from the image query. [0232] retrieves at least one digital image from the plurality of digital images based on a proximity of an embedding for the at least one digital image to the linear combination of the text embedding and the image embedding within the text-image embedding space. [0142] The image embedding 908 represents one or more visual features from the image query 902 (e.g., image characteristics or other patent or latent features of the image query 902, such as semantic information and/or layout information). [0167] In some implementations, however, the positional information 1230 is encoded within the regional layout embeddings 1226 generated from the multi-modal search input 1220).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Zhang to determine a selection of the plurality of feature vectors based on a proximity between first feature vector linear and positional embeddings and a plurality of feature vector linear and positional embeddings because "By considering the positional information 1230, the search-based editing system 106 can ensure that the retrieved digital images include objects portrayed in the multi-modal search input 1220 in positionings portrayed within the multi-modal search input 1220" [0167] and "By using the unified embedding 1232, the search-based editing system 106 represents the multi-modal search input 1220 within a common embedding space (e.g., an embedding space that encompasses representations for the various search modals)…the search-based editing system 106 utilizes the unified embedding 1232 to search for and retrieve digital images in response to the multi-modal search input 1220" [0168-0169].
Regarding claim 2, Ferrell, Morariu, and Zhang teach the method of claim 1. Ferrell further teaches identifying, by the processing device, the first substrate processing defect based on the second image data, wherein the action is further based on an identity of the first substrate processing defect ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Regarding claim 6, Ferrell, Morariu, and Zhang teach the method of claim 1. Ferrell further teaches receiving, by the processing device, a first selection of the first image frame of the first image data ([col. 5 ln. 37-40] The third module forming the manufacturing-specific CBIR system 1, a querying module 4, can accept a query image from a user and can return to the user, a collection of similar images stored in the image database 5);
generating, by the processing device, a second image frame by masking a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame ([col. 6 ln. 23-31] The defect region can undergo additional masking of morphological features which include an inner mask, which is a solid white-on-black mask identifying the defect; an annular mask, consisting of one pixel inside the inner mask border, one pixel on the border, and one pixel outside the border; and, an outer mask, surrounding the outside of the annular mask by a specified number of pixels. Generally, only the inner mask is required for the feature extraction. [col. 6 ln. 38-44] The characteristics forming each feature vector as extracted by the image feature extraction module 2 can be grouped into descriptive categories: color, texture, and shape. Each feature contributing to the vector description can be generated by masking the image 8 and plane with an appropriate mask and measuring the feature according to appropriate equations).
Regarding claim 8, Ferrell, Morariu, and Zhang teach the method of claim 1. Morariu, in the same field of endeavor of image embedding, teaches dividing, by the processing device, the first image frame of the first image data into a set of image patches corresponding to the first image frame ([0028] In various embodiments, the vision encoder 206 processes the input image as a sequence of patches. In one example, the input image I∈R3×H×W is decomposed into a batch of N=HW/P2 patches of size P×P, where H and W are a height and widths after resizing such that the values are divisible by the patch size P. In an embodiment, the patches are flattened (e.g., converted) into vectors which are linearly projected to generate the embeddings);
determining, by the processing device, a set of linear embeddings corresponding to a content of each image patch of the set of image patches ([0024] the input generator 124 includes an encoder of a transformer model that processes an image of the table 122 as a sequence of patches which are flattened into vectors and linearly projected to patch embeddings);
and determining, by the processing device, a set of positional embeddings corresponding to a relative position of each of the image patches of the set of image patches ([0029] In an embodiment, the N embeddings include learned positional encodings that are added to the input of the attention layers of the decoder).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Morariu to determine linear and positional embeddings to determine "the output 128 is a feature vector (e.g., containing values from the input feature vectors transformed/encoded by the table recognition model 126) that is useable by the one or more task models 112 to perform various task associated with the document 120 and/or table 122…such as table question answering, table fact verification, table formatting, table layout, table identification, and table captioning" [Morariu 0025].
Regarding claim 14, Ferrell teaches a non-transitory machine-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising ([col. 15 ln. 63-64] A computer apparatus programmed with a routine set of instructions stored in a fixed medium):
store, in a data storage device, a plurality of feature vectors representative of previously processed image frames that correspond to various substrate processing defects ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic);
receive first image data (query image) comprising one or more image frames indicative of a first substrate processing defect; determine a first feature vector corresponding to the first image data ([col. 3 ln. 34-41] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. More particularly, the retrieving step includes several steps. First, the query image is converted into at least one query vector corresponding to a particular characteristic of the manufacturing image. [col. 5 ln. 44-47] Significantly, the image feature extraction module 2, using the query image as an input, can generate the associated vector of numerical descriptors. [col. 6 ln. 18-23] the image feature extraction module 2 can define three unique and approximately independent feature vectors of manufacturing-specific digital imagery. The three feature vectors include an anomaly/defect characteristic, a substrate/background characteristic, and an image modality and global characteristic);
determine a selection of the plurality of feature vectors based on a proximity ([col. 5 ln. 47-50] Using the vector of numerical descriptors as a guideline, a very rapid traversal of indexing tree 6 in the first-level data reduction routine can produce a preliminary selection of matching images from the image database 5. [col. 9 ln. 27-47] Having created a series of feature vector lists 7 for images 8 stored in image database 5, the manufacturing-specific CBIR system 1 can initiate an indexing process which organizes the data into a hierarchical tree structure 6 allowing for rapid retrieval of imagery during the query process. Specifically, the manufacturing-specific CBIR system 1 employes feature indexing--a process by which the manufacturing-specific CBIR system 1 organizes feature vectors contained in each feature vector list 7 to facilitate rapid access and retrieval of similar imagery. The indexing technique is critical to the efficient retrieval of similar data from the software system. Unlike existing CBIR systems, the manufacturing-specific CBIR system 1 utilizes a sequential, agglomerative, hierarchical, non-overlapping [SAHN] algorithm for sorting the features contained in the feature vector list 7. Historically, the SAHN algorithm has been used as an investigative tool for unsupervised clustering in pattern recognition problems. For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation);
determine second image data (manufacturing-specific digital images) different from the first image data, the second image data comprising one or more previously processed image frames from the data storage device corresponding to embedding data associated with the selection of the plurality of feature vectors, wherein the second image data indicates one or more additional substrate processing defects matching the first substrate processing defect ([col. 3 ln. 35-38] Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. [col. 5 ln 6-18] Specifically, the manufacturing image can be characterized in terms of image modality and overall characteristics, substrate-background characteristics, and anomoly-defect charcteristics. Moreover, the characteristics used to describe the modality, background, and defect are based on the texture, color, and shape of the image. In the preferred embodiment, the image feature extraction module 2 pre-processes every image in the image database 5 to generate a series of vectors having these descriptive set of features, each vector weighted to a particular characteristic of the stored image. Subsequently, the image feature extraction module 2 can store each of the series of vectors in a corresponding feature vector list 7, contained as part of the image database 5);
and perform an action based on determining the second image data, wherein the action comprises identifying the first substrate processing defect based on the second image data ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Ferrell does not explicitly teach wherein the first feature vector is determined based on: a first set of linear embeddings corresponding to a first set of image patches corresponding to a first image frame of the one or more image frames; and a first set of positional embeddings corresponding to a first relative position of a first content of each image patch of the set of image patches; determine a selection of the plurality of feature vectors based on a proximity between the first set of linear embeddings and the first set of positional embeddings associated with the first feature vector and one or more second sets of linear embeddings and one or more second sets of positional embeddings associated with the selection of the plurality of feature vectors.
Morariu, in the same field of endeavor of image embedding, teaches wherein the first feature vector is determined based on: a first set of linear embeddings corresponding to a first set of image patches corresponding to a first image frame of the one or more image frames; and a first set of positional embeddings corresponding to a first relative position of a first content of each image patch of the set of image patches ([0024] the input generator 124 includes an encoder of a transformer model that processes an image of the table 122 as a sequence of patches which are flattened into vectors and linearly projected to patch embeddings. [0029] In an embodiment, the N embeddings include learned positional encodings that are added to the input of the attention layers of the decoder. [0029] In one example, the N embeddings are transformed into an output embedding by the split decoder 208. In addition, in an embodiment, the decoders (e.g., the split decoder 208 and OCR decoder 210) utilize seeds to generate candidate predictions (e.g., set of row and/or column split predictions 214). In an example, the seeds include a fixed length feature vector).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Morariu to determine linear and positional embeddings to determine "the output 128 is a feature vector (e.g., containing values from the input feature vectors transformed/encoded by the table recognition model 126) that is useable by the one or more task models 112 to perform various task associated with the document 120 and/or table 122…such as table question answering, table fact verification, table formatting, table layout, table identification, and table captioning" [Morariu 0025].
Zhang, in the same field of endeavor of image embedding retrieval, teaches wherein the first feature vector is determined based on: a first set of linear embeddings corresponding to a first set of image patches corresponding to a first image frame of the one or more image frames ([0232] To provide another illustration, in one or more embodiments, the search-based editing system 106 receives a text query, an image query, and a combination weight for conducting an image search; generates, utilizing the text embedding model, a text embedding for the text query within a text-image embedding space; generates, utilizing the image embedding model, an image embedding for the image query within the text-image embedding space; determines a linear combination of the text embedding and the image embedding utilizing the combination weight; and retrieves at least one digital image from the plurality of digital images based on a proximity of an embedding for the at least one digital image to the linear combination of the text embedding and the image embedding within the text-image embedding space. [0166] More particularly, the search-based editing system 106 separately encodes semantic information and layout information for distinct portions (e.g., patches or semantic segments) of the multi-modal search input 1220);
and a first set of positional embeddings corresponding to a first relative position of a first content of each image patch of the set of image patches ([0142] As further shown, the search-based editing system 106 utilizes an image embedding model 906 to generate an image embedding 908 from the image query 902. The image embedding 908 represents one or more visual features from the image query 902 (e.g., image characteristics or other patent or latent features of the image query 902, such as semantic information and/or layout information). [0165] Additionally, as shown in FIG. 12A, the search-based editing system 106 utilizes the encoder 1222 to generate regional semantic embeddings 1224 and regional layout embeddings 1226 from the multi-modal search input 1220. In one or more embodiments, a regional embedding includes an embedding generated from a particular region of an input, such as a multi-modal search input. For instance, in some cases, the search-based editing system 106 utilizes the encoder 1222 to generate the regional embeddings (i.e., the regional semantic embeddings 1224 and the regional layout embeddings 1226) from patches of the multi-modal search input 1220. In some cases, the search-based editing system 106 utilizes the encoder 1222 to generate the regional embeddings from semantic segments of the multi-modal search input 1220 (as will be discussed in more detail below with reference to FIG. 12B). [0166] By generating the regional semantic embeddings 1224 and the regional layout embeddings 1226, the search-based editing system 106 separately encodes semantic information and layout information from the multi-modal search input 1220. More particularly, the search-based editing system 106 separately encodes semantic information and layout information for distinct portions (e.g., patches or semantic segments) of the multi-modal search input 1220. [0167] In some implementations, however, the positional information 1230 is encoded within the regional layout embeddings 1226 generated from the multi-modal search input 1220);
determine a selection of the plurality of feature vectors based on a proximity between the first set of linear embeddings and the first set of positional embeddings associated with the first feature vector and one or more second sets of linear embeddings and one or more second sets of positional embeddings associated with the selection of the plurality of feature vectors ([0231] In some cases, retrieving the one or more digital images utilizing the weighted combination of the text embedding and the image embedding comprises retrieving the one or more digital images using proximities of embeddings corresponding to the one or more digital images to the weighted combination. Further, in some instances, generating the text embedding for the text query and the image embedding for the image query comprises: generating, utilizing a text embedding model, the text embedding from the text query; and generating, utilizing an image embedding model, the image embedding from the image query. [0232] retrieves at least one digital image from the plurality of digital images based on a proximity of an embedding for the at least one digital image to the linear combination of the text embedding and the image embedding within the text-image embedding space. [0142] The image embedding 908 represents one or more visual features from the image query 902 (e.g., image characteristics or other patent or latent features of the image query 902, such as semantic information and/or layout information). [0167] In some implementations, however, the positional information 1230 is encoded within the regional layout embeddings 1226 generated from the multi-modal search input 1220).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Zhang to determine a selection of the plurality of feature vectors based on a proximity between first feature vector linear and positional embeddings and a plurality of feature vector linear and positional embeddings because "By considering the positional information 1230, the search-based editing system 106 can ensure that the retrieved digital images include objects portrayed in the multi-modal search input 1220 in positionings portrayed within the multi-modal search input 1220" [0167] and "By using the unified embedding 1232, the search-based editing system 106 represents the multi-modal search input 1220 within a common embedding space (e.g., an embedding space that encompasses representations for the various search modals)…the search-based editing system 106 utilizes the unified embedding 1232 to search for and retrieve digital images in response to the multi-modal search input 1220" [0168-0169].
Regarding claim 15, Ferrell, Morariu, and Zhang teach the medium of claim 14. Ferrell further teaches identify the first substrate processing defect based on the second image data, wherein the action is further based on an identity of the first substrate processing defect ([col. 9 ln. 44-47] For the CBIR application, SAHN algorithms are used to quickly reduce the number of feature vectors, v.sub.i, that must be compared to the query vector, Q.sub.v during a retrieval operation. [col. 13 ln. 3-11] Specifically, the inventive method can assist with off-line review and analysis of unclassifiable defects, provide assisted defect library generation for supervised automatic defect classification systems, provide unsupervised classification of defects during early yield learning, and, assist in training yield management personnel. The three unique and approximately independent feature vectors allow the user to perform a broad variety of focused, detailed searches based on specific image queries).
Regarding claim 20, Ferrell, Morariu, and Zhang teach the medium of claim 14. Morariu, in the same field of endeavor of image embedding, teaches divide the first image frame of the first image data into a set of image patches corresponding to the first image frame ([0028] In various embodiments, the vision encoder 206 processes the input image as a sequence of patches. In one example, the input image I∈R3×H×W is decomposed into a batch of N=HW/P2 patches of size P×P, where H and W are a height and widths after resizing such that the values are divisible by the patch size P. In an embodiment, the patches are flattened (e.g., converted) into vectors which are linearly projected to generate the embeddings);
determine a set of linear embeddings corresponding to a content of each image patch of the set of image patches ([0024] the input generator 124 includes an encoder of a transformer model that processes an image of the table 122 as a sequence of patches which are flattened into vectors and linearly projected to patch embeddings);
and determine a set of positional embeddings corresponding to a relative position of each of the image patches of the set of image patches ([0029] In an embodiment, the N embeddings include learned positional encodings that are added to the input of the attention layers of the decoder).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Morariu to determine linear and positional embeddings to determine "the output 128 is a feature vector (e.g., containing values from the input feature vectors transformed/encoded by the table recognition model 126) that is useable by the one or more task models 112 to perform various task associated with the document 120 and/or table 122…such as table question answering, table fact verification, table formatting, table layout, table identification, and table captioning" [Morariu 0025].
Claims 3-4 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell in view of Morariu, Zhang, and Sawlani (WO2020146708A1).
Regarding claim 3, Ferrell, Morariu, and Zhang teach the method of claim 2. Sawlani, in the same field of endeavor of substrate defect identification, teaches identifying, by the processing device, an instance of abnormality of a fabrication process associated with the first substrate processing defect ([0005] a defect analysis computational system is provided, which system may be characterized by the following features: (a) one or more processors; and (b) program instructions for executing on the one or more processors…receive manufacturing information, determine, using the first stage defect classification and the manufacturing information, one or more sources of the defects on the substrate , and output a likelihood of the defects being caused by a first source associated with the manufacturing equipment and/or fabrication process); and
causing, by the processing device, performance of a corrective action associated with fabrication process equipment based on the instance of abnormality ([0009] In various embodiments, the second stage defect classification engine is additionally configured to further classify the defects on the substrate and/or provide suggested corrective actions to reduce generation of defects on the substrate and/or reduce occurrences of defects on substrates processed in the future).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Sawlani to identify the fabrication process associated with the defect so that "a bridging defect could be corrected by etching the substrate to remove the bridged material before proceeding with the next fabrication process" [Sawlani 0081].
Regarding claim 4, Ferrell, Morariu, and Zhang teach the method of claim 1. Sawlani teaches preparing, by the processing device, one or more image frames of the second image data ([0069] Morphology data may include images of specific defects and can be used to classify the size and/or shape of defects. [0076] Figure 5 shows morphology data for small particle defects 501 and for large particle defects 502, I an example embodiment, a first stage defect classification engine processes all ten images to produce one or more first stage defect classifications. In general, a first stage defect classification engine may process every image taken of a defect, which could be a single image or multiple images) for presentation on a graphical user interface (GUI) ([0118] As shown, computer system 800 includes an input/output subsystem 802, which may implement an interface for interacting with human users and/or other computer systems depending upon the application. Embodiments of the invention may be implemented in program code on system 800 with I/O subsystem 802 used to receive input program statements and/or data from a human user (e.g., via a GUI or keyboard) and to display them back to the user. The I/O subsystem 802 may include, e.g., a keyboard, mouse, graphical user interface, touchscreen, or other interfaces for input, and, e.g.. an LED or other flat screen display, or other interfaces for output).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Sawlani to present the image "to receive input program statements and/or data from a human user" [Sawlani 0118].
Regarding claim 16, Ferrell, Morariu, and Zhang teach the medium of claim 15. Sawlani, in the same field of endeavor of substrate defect identification, teaches identify an instance of abnormality of a fabrication process associated with the first substrate processing defect ([0005] a defect analysis computational system is provided, which system may be characterized by the following features: (a) one or more processors; and (b) program instructions for executing on the one or more processors…receive manufacturing information, determine, using the first stage defect classification and the manufacturing information, one or more sources of the defects on the substrate , and output a likelihood of the defects being caused by a first source associated with the manufacturing equipment and/or fabrication process); and
and cause performance of a corrective action associated with fabrication process equipment based on the instance of abnormality ([0009] In various embodiments, the second stage defect classification engine is additionally configured to further classify the defects on the substrate and/or provide suggested corrective actions to reduce generation of defects on the substrate and/or reduce occurrences of defects on substrates processed in the future).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Sawlani to identify the fabrication process associated with the defect so that "a bridging defect could be corrected by etching the substrate to remove the bridged material before proceeding with the next fabrication process" [Sawlani 0081].
Regarding claim 17, Ferrell, Morariu, and Zhang teach the medium of claim 14. Sawlani teaches prepare one or more image frames of the second image data ([0069] Morphology data may include images of specific defects and can be used to classify the size and/or shape of defects. [0076] Figure 5 shows morphology data for small particle defects 501 and for large particle defects 502, I an example embodiment, a first stage defect classification engine processes all ten images to produce one or more first stage defect classifications. In general, a first stage defect classification engine may process every image taken of a defect, which could be a single image or multiple images) for presentation on a graphical user interface (GUI) ([0118] As shown, computer system 800 includes an input/output subsystem 802, which may implement an interface for interacting with human users and/or other computer systems depending upon the application. Embodiments of the invention may be implemented in program code on system 800 with I/O subsystem 802 used to receive input program statements and/or data from a human user (e.g., via a GUI or keyboard) and to display them back to the user. The I/O subsystem 802 may include, e.g., a keyboard, mouse, graphical user interface, touchscreen, or other interfaces for input, and, e.g.. an LED or other flat screen display, or other interfaces for output).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Sawlani to present the image "to receive input program statements and/or data from a human user" [Sawlani 0118].
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell in view of Morariu, Zhang, and Gu (US20070177794A1).
Regarding claim 5, Ferrell, Morariu, and Zhang teach the method of claim 1. Ferrell further teaches receiving, by the processing device, a first selection of the first image frame of the first image data ([col. 5 ln. 37-40] The third module forming the manufacturing-specific CBIR system 1, a querying module 4, can accept a query image from a user and can return to the user, a collection of similar images stored in the image database 5).
Ferrell does not teach generating, by the processing device, a second image frame by cropping a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame.
Gu, in the same field of endeavor of image feature detection, teaches generating, by the processing device ([0024] As shown in FIG. 2, the image processing unit 41), a second image frame by cropping a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame ([0031] In both eyeglasses detection learning stage and actual detection of eyeglasses stage, image data passes through edge enhancement unit 401, image size shrinking unit 403, nose ridge mask unit 124 and feature vector unit 415. Face detection and cropping unit 395 crops face images. Edge enhancement unit 401 enhances edges in face images and outputs edge maps. The sizes of the edge maps image are modified by the image size shrinking unit 403. The nose ridge mask unit 124 receives the edge maps from image size shrinking unit 403, and extracts image regions located in the nose ridge region of the faces shown in the edge maps. Feature vector unit 415 extracts feature vectors from the nose ridge image regions output from nose ridge mask unit 124).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Gu to crop the image prior to determining the feature vectors to "improve eyeglasses detection by performing face registration of the faces in the face images received from face image operations unit 122" [Gu 0025].
Regarding claim 18, Ferrell, Morariu, and Zhang teach the medium of claim 14. Ferrell further teaches receive the first selection of a first image frame of the first image data ([col. 5 ln. 37-40] The third module forming the manufacturing-specific CBIR system 1, a querying module 4, can accept a query image from a user and can return to the user, a collection of similar images stored in the image database 5).
Ferrell does not teach generate a second image frame by at least one of cropping or masking a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame.
Gu, in the same field of endeavor of image feature detection, teaches generate a second image frame by at least one of cropping or masking a region of the first image frame based on the first selection, wherein the first feature vector is determined using the second image frame ([0031] In both eyeglasses detection learning stage and actual detection of eyeglasses stage, image data passes through edge enhancement unit 401, image size shrinking unit 403, nose ridge mask unit 124 and feature vector unit 415. Face detection and cropping unit 395 crops face images. Edge enhancement unit 401 enhances edges in face images and outputs edge maps. The sizes of the edge maps image are modified by the image size shrinking unit 403. The nose ridge mask unit 124 receives the edge maps from image size shrinking unit 403, and extracts image regions located in the nose ridge region of the faces shown in the edge maps. Feature vector unit 415 extracts feature vectors from the nose ridge image regions output from nose ridge mask unit 124).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Gu to crop the image prior to determining the feature vectors to "improve eyeglasses detection by performing face registration of the faces in the face images received from face image operations unit 122" [Gu 0025].
Claims 7 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ferrell in view of Morariu, Zhang, and Nogami (US20200118263A1).
Regarding claim 7, Ferrell, Morariu, and Zhang teach the method of claim 1. Ferrell further teaches a substrate processing defect ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic).
Ferrell does not teach extracting, by the processing device, a selection of text from the first image frame of the first image data; determining, by the processing device, an image scaling factor associated with the first image frame; and determining, by the processing device based on the image scaling factor, a size associated with the first defect; wherein the selection of the plurality of feature vectors is determined further using the size.
Nogami, in the same field of endeavor of defect detection, teaches extracting, by the processing device, a selection of text from the first image frame of the first image data ([0082] the second extraction unit 115 may extract a histogram feature amount using a method known as BoW (Bag of Words) or BoF (Bag of Features). In this case, a plurality of visual words are generated from multiple ROIs in advance, and the frequency with which the visual words appear in the ROI image to be processed can then be converted to a histogram);
determining, by the processing device, an image scaling factor associated with the first image frame ([0048] Depending on the position or orientation of the image capturing device when the image of the structure is captured, it may not be possible for the image capturing device to capture an image from directly opposite the concrete wall surface, but such an image can be generated by carrying out a geometric conversion process on the image. It is also assumed that the input image 101 is subjected to processing for correcting lens distortion. Furthermore, it is assumed that the input image 101 is subjected to an enlargement or reduction process so that the image resolution is constant relative to the actual space. With these processes, for example, an image in which a single pixel corresponds to 1 mm of the concrete wall surface can be obtained. Such conversion and correction can be carried out by the information processing device 100); and
determining, by the processing device based on the image scaling factor, a size associated with the first defect ([0113] In step S206, the second extraction unit 115 first calculates the size of each defect);
wherein the selection of the plurality of feature vectors is determined further using the size ([0121] The feature amount at a given pixel in the feature map 906 is expressed by an n-dimensional vector 907. [0130] The size of the ROI changes depending on the size of the defect (the length of the crack). On the other hand, the attribute determination unit 116 determines the attributes (crack width) of the defect using a defect feature amount having a predetermined dimension. Thus in the present embodiment, the second extraction unit 115 generates a defect feature amount v.sub.930 for the ROI 930).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Ferrell with the teachings of Nogami to determine the size of a defect and then determine the selection of plurality of feature vectors because "confirming the width of the crack makes it possible to determine the severity or level of danger posed by the crack" [Nogami 0004].
Regarding claim 19, Ferrell, Morariu, and Zhang teach the medium of claim 14. Ferrell further teaches a substrate processing defect ([Abstract] A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image, for instance, a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic).
Ferrell does not teach extract a selection of text from the first image frame of the first image data; determine an image scaling factor associated with the first image frame; and determine, based on the image scaling factor, a size associated with the first defect; wherein the selection of the plurality of feature vectors is determined further using the size.
Nogami, in the same field of endeavor of defect detection, teaches extract a selection of text from the first image frame of the first image data ([0082] the second extraction unit 115 may extract a histogram feature amount using a method known as BoW (Bag of Words) or BoF (Bag of Features). In this case, a plurality of visual words are generated from multiple ROIs in advance, and the frequency with which the visual words appear in the ROI image to be processed can then be converted to a histogram);
determine an image scaling factor associated with the first image frame ([0048] Depending on the position or orientation of the image capturing device when the image of the structure is captured, it may not be possible for the image capturing device to capture an image from directly opposite the concrete wall surface, but such an image can be generated by carrying out a geometric conversion process on the image. It is also assumed that the input image 101 is subjected to processing for correcting lens distortion. Furthermore, it is assumed that the input image 101 is subjected to an enlargement or reduction process so that the image resolution is constant relative to the actual space. With these processes, for example, an image in which a single pixel corresponds to 1 mm of the concrete wall surface can be obtained. Such conversion and correction can be carried out by the information processing device 100); and
determine, based on the image scaling factor, a size associated with the first defect ([0113] In step S206, the second extraction unit 115 first calculates the size of each defect);
wherein the selection of the plurality of feature vectors is determined further using the size ([0121] The feature amount at a given pixel in the feature map 906 is expressed by an n-dimensional vector 907. [0130] The size of the ROI changes depending on the size of the defect (the length of the crack). On the other hand, the attribute determination unit 116 determines the attributes (crack width) of the defect using a defect feature amount having a predetermined dimension. Thus in the present embodiment, the second extraction unit 115 generates a defect feature amount v.sub.930 for the ROI 930).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Ferrell with the teachings of Nogami to determine the size of a defect and then determine the selection of plurality of feature vectors because "confirming the width of the crack makes it possible to determine the severity or level of danger posed by the crack" [Nogami 0004].
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights