DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 9 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 (CN113190706A) in view of D2 (U.S. PG-PUB NO. 2025/0029206) and further in view of D3 (U.S. PG-PUB NO. 2015/0186374).
-Regarding claim 1, D1 discloses an information processing comprising: a first neural network configured to extract a feature of a query image and features of search object images (step 3, inputting the query image and the training image processed in the step 1 into a second-order attention convolution neural network respectively for feature extraction to obtain a query image feature and a training image feature); a processing: detect a degree of similarity between each of the search object images and a query image based on the feature of the query image and the features of the search object images (step 5, carrying out similarity measurement on the query image descriptors and the training image descriptors, and sequencing the training image descriptors according to the similarity to obtain a sequencing result); and the query image and feature transformation information relating to the degree of similarity between each of the search object images and the query image (step 6, several training images ranked at the top in the ranking result are selected, the average vector of the feature vectors of the training images is calculated, the result is rearranged according to the average vector, and the training image most similar to the query image is obtained through retrieval).
D1 is silent to teaching that a processor; calculate a score each of the search object images based on the degree of similarity between each of the search object images; a second neural network configured to output the feature transformation information based on the feature of the query image and the features of the search object images; a user interface configured to determine by a user the feature transformation information of each of the search object images based on at least one of the degree of similarity between each of the search object images and the query image or the score of each of the search object images, wherein the processing circuitry is configured to provide, as feedback, the feature transformation information of the search object images determined by the user to the second neural network. However, the claimed limitation is well known in the art as evidenced by D2 and D3.
In the same field of endeavor, D2 teaches a processor (processor 106, FIG. 1); calculate a score each of the search object images based on the degree of similarity between each of the search object images (processor 106 that uses one or more neural networks to perform a multi-stage inference process to generate information about one or more inputs (e.g., images) based, at least in part, on one or more confidence scores associated with said information, paragraph 57); a second neural network configured to output the feature transformation information based on the feature of the query image and the features of the search object images (first neural network 108 is a LLM and second neural network 114 is a vision transformer, paragraph 66).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of D1 with the teaching of D2 in order to transform the extracted query/search-image features into task-relevant output information and attach a reliability score to each candidate, thereby producing more accurate, confidence-weighted ranking of the search results.
In the same field of endeavor, D3 teaches a user interface configured to determine by a user the feature transformation information of each of the search object images based on at least one of the degree of similarity between each of the search object images and the query image or the score of each of the search object images (When the mouse slides over a node at either of the two stages, a selection label for user's satisfaction on the retrieval results is displayed, paragraph 134), wherein the processing circuitry is configured to provide, as feedback, the feature transformation information of the search object images determined by the user to the second neural network (The retrieving system regulates model parameters according to the evaluation result feedback, and triggers an update process of the fluoroscopic image representation database, paragraph 98).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of D1 and D2 with the teaching of D3. D1's twin-CNN retrieval and D2's first/second-NN (ViT) architecture lack a user relevance-feedback loop. Incorporating D3's interacting + correlation-feedback modules into D1+D2 — so the user's per-result satisfaction labels update D2's second NN — applies a known relevance-feedback technique to improve a similar retrieval apparatus, with predictable refinement (KSR).
-Regarding claim 2, the combination further discloses the user interface presents to the user a list of the search object images arranged in an order of the detected degree of similarity before the provided feedback (D2, The retrieving system regulates model parameters according to the evaluation result feedback, and triggers an update process of the fluoroscopic image representation database, paragraph 98), and presents to the user a list of the search object images arranged in an order of a magnitude of the score after the provided feedback (D3, A first stage is displaying an output result from the diversified filtering module, and a second stage is displaying other similar images that are in the same article category as a previous stage, paragraph 134).
-Regarding claim 3, the combination further discloses the first neural network is pre-trained, and wherein a weight of the second neural network is updated by repeated operations of the processing circuitry and the user interface (D2, trained neural network 708 can then be deployed to implement any number of machine learning operations, paragraph 111; D3, The retrieving system regulates model parameters according to the evaluation result feedback, and triggers an update process of the fluoroscopic image representation database, paragraph 98).
-Regarding claim 4, the combination further discloses the feature transformation information outputted from the second neural network has a binary value or a Sigmoid function value (D2, a confidence score is usually a value between zero and one, paragraph 64).\
-Regarding claim 9, the combination further discloses the processing circuitry stores the feature of the query image and the features of the search object images extracted by the first neural network, and wherein the processing circuitry stores the feature transformation information of each of the search object images with respect to the query image outputted from the second neural network (D2, extracting and/or storing data, paragraph 570).
-Regarding claim 20, D1 discloses an information processing method comprising: extracting, by a first neural network, a feature of a query image and features of search object images (step 3, inputting the query image and the training image processed in the step 1 into a second-order attention convolution neural network respectively for feature extraction to obtain a query image feature and a training image feature); detecting a degree of similarity between each of the search object images and the query image based on the feature of the query image and the features of the search object images (step 5, carrying out similarity measurement on the query image descriptors and the training image descriptors, and sequencing the training image descriptors according to the similarity to obtain a sequencing result); and the query image based on the feature of the query image and the features of the search object images (step 6, several training images ranked at the top in the ranking result are selected, the average vector of the feature vectors of the training images is calculated, the result is rearranged according to the average vector, and the training image most similar to the query image is obtained through retrieval).
D1 is silent to teaching that outputting, from a second neural network, feature transformation information relating to the degree of similarity between each of the search object images; calculating a score of each of the search object images based on the feature transformation information and the degree of similarity between each of the search object images and the query image; determining by a user the feature transformation information of each of the search object images based on at least one of the degree of similarity between each of the search object images and the query image or the score of each of the search object images; and providing as feedback the feature transformation information of the search object images determined by the user to the second neural network. However, the claimed limitation is well known in the art as evidenced by D2 and D3.
In the same field of endeavor, D2 teaches outputting, from a second neural network, feature transformation information relating to the degree of similarity between each of the search object images (first neural network 108 is a LLM and second neural network 114 is a vision transformer, paragraph 66); calculating a score of each of the search object images based on the feature transformation information and the degree of similarity between each of the search object images and the query image (processor 106 that uses one or more neural networks to perform a multi-stage inference process to generate information about one or more inputs (e.g., images) based, at least in part, on one or more confidence scores associated with said information, paragraph 57).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of D1 with the teaching of D2 in order to transform the extracted query/search-image features into task-relevant output information and attach a reliability score to each candidate, thereby producing more accurate, confidence-weighted ranking of the search results.
In the same field of endeavor, D3 teaches determining by a user the feature transformation information of each of the search object images based on at least one of the degree of similarity between each of the search object images and the query image or the score of each of the search object images (When the mouse slides over a node at either of the two stages, a selection label for user's satisfaction on the retrieval results is displayed, paragraph 134), providing as feedback the feature transformation information of the search object images determined by the user to the second neural network (The retrieving system regulates model parameters according to the evaluation result feedback, and triggers an update process of the fluoroscopic image representation database, paragraph 98).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of D1 and D2 with the teaching of D3. D1's twin-CNN retrieval and D2's first/second-NN (ViT) architecture lack a user relevance-feedback loop. Incorporating D3's interacting + correlation-feedback modules into D1+D2 — so the user's per-result satisfaction labels update D2's second NN — applies a known relevance-feedback technique to improve a similar retrieval apparatus, with predictable refinement (KSR).
Claim(s) 11-13, 15 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over D4 ("VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers", Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, Vasudev Lal; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 21406-21415) in view of D1 (CN113190706A).
-Regarding claim 11, D4 discloses an information processing apparatus (VL-InterpreT, section 3.1 Workflow) comprising: a neural network including a first part neural network extending from an input layer to an intermediate layer, and a second part neural network extending from the intermediate layer to an output layer, the neural network being configured to output, and second feature transformation information obtained by transforming each of features of search object images (Each input token is embedded as a d-dimension vector after processed by each layer, section 3.1 Workflow); a processing circuitry configured to: generate intermediate images based on the query image and intermediate features outputted from the first part neural network when the query image is inputted to the neural network (A user can select any attention head and image patch in the interface of our tool, and the corresponding L2V attention weights are displayed as a heatmap overlaid on the input text, section 3.2.1 Attention heads components); perform a feedback operation to select one or more of the intermediate features to be inputted to the second part neural network among the intermediate features outputted from the first part neural network based on the one or more of the intermediate images selected by the user (user can select any attention head and image patch in the interface of our tool, section 3.2.1 Attention heads components); and the query image that is inferred by inputting the query image to the neural network after the user selects the one or more of the intermediate images and the second feature transformation information of the search object images that is inferred by inputting the search object images to the neural network, and outputs a calculation result of the degree of similarity as a search result; and a user interface configured to have a user select one or more of the intermediate images (user can select any attention head and image patch in the interface of our tool, section 3.2.1 Attention heads components).
D4 is silent to teaching that first feature transformation information obtained by transforming a feature of a query image; calculate a degree of similarity between the first feature transformation information of the query image. However, the claimed limitation is well known in the art as evidenced by D1.
In the same field of endeavor, D1 teaches first feature transformation information obtained by transforming a feature of a query image (step 6, several training images ranked at the top in the ranking result are selected, the average vector of the feature vectors of the training images is calculated, the result is rearranged according to the average vector, and the training image most similar to the query image is obtained through retrieval); calculate a degree of similarity between the first feature transformation information of the query image (step 5, carrying out similarity measurement on the query image descriptors and the training image descriptors, and sequencing the training image descriptors according to the similarity to obtain a sequencing result).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of D4 with the teaching of D1 so user-selected heads determine which intermediate features feed the second-part network producing the transformation information for similarity ranking — applies a known interactive-interpretability technique to improve a similar retrieval apparatus (KSR).
-Regarding claim 12, the combination further discloses the user interface presents to the user the calculation result of the outputted degree of similarity before the user selects the one or more of the intermediate images (D1, step 5, carrying out similarity measurement on the query image descriptors and the training image descriptors, and sequencing the training image descriptors according to the similarity to obtain a sequencing result).
-Regarding claim 13, the combination further discloses the first part neural network includes a plurality of heads each outputting the intermediate feature, wherein each of the head is capable of being activated or deactivated (D4, a user can select a specific attention head and a text token, and the corresponding attention scores will be overlaid onto the image as a heatmap, section 3.2.1 Attention heads components), wherein the intermediate feature outputted from an activated one of the heads is inputted to the second part neural network, and wherein the first feature transformation information or the second feature transformation information is outputted from the second part neural network (D4, Nlayers and Nheads correspond to the number of layers and heads, respectively, section 3.2.1 Attention heads components).
-Regarding claim 15, the combination further discloses the heads are activated or deactivated based on the one or more of the intermediate images selected by the user, wherein the neural network inputs the first feature transformation information to the processing circuitry, the first feature transformation information being generated by inputting to the second part neural network the intermediate features outputted from the heads corresponding to the one or more of the intermediate images selected by the user, and wherein the processing circuitry outputs the calculation result corresponding to the intermediate feature, and sends the calculation result to the user interface as feedback (D4, a user can select a specific attention head and a text token, and the corresponding attention scores will be overlaid onto the image as a heatmap, section 3.2.1 Attention heads components).
-Regarding claim 16, the combination further discloses after the heads are activated or deactivated, the query image and the search object images are inputted to the neural network for inference (D4, a user can select a specific attention head and a text token, and the corresponding attention scores will be overlaid onto the image as a heatmap, section 3.2.1 Attention heads components), wherein the processing circuitry calculates the degree of similarity between the query image and each of the search object images based on the second feature transformation information of each of the search object images outputted from the neural network and the first feature transformation information of the query image (D1, step 5, carrying out similarity measurement on the query image descriptors and the training image descriptors, and sequencing the training image descriptors according to the similarity to obtain a sequencing result).
Claim(s) 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over D4 ("VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers", Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, Vasudev Lal; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 21406-21415) in view of D1 (CN113190706A) and further in view of D2 (U.S. PG-PUB NO. 2025/0029206).
-Regarding claim 17, the combination is silent to teaching that a memory configured to store the intermediate features outputted from the first part neural network after the search object images are inputted to the neural network before inference is performed and before the user selects the one or more of the intermediate images, wherein the processing circuitry calculates the degree of similarity between the first feature transformation information outputted from the neural network and the second feature transformation information outputted from the neural network, the first feature transformation information being obtained by inputting the query image to the neural network and inputting the intermediate features outputted from the heads and corresponding to the one or more of the intermediate images selected by the user to the second part neural network, and the second feature transformation information being obtained by inputting the intermediate features stored in the memory to the second part neural network for performing partial inference. However, the claimed limitation is well known in the art as evidenced by D2.
In the same field of endeavor, D2 teaches a memory configured to store the intermediate features outputted from the first part neural network after the search object images are inputted to the neural network before inference is performed and before the user selects the one or more of the intermediate images, wherein the processing circuitry calculates the degree of similarity between the first feature transformation information outputted from the neural network and the second feature transformation information outputted from the neural network, the first feature transformation information being obtained by inputting the query image to the neural network and inputting the intermediate features outputted from the heads and corresponding to the one or more of the intermediate images selected by the user to the second part neural network, and the second feature transformation information being obtained by inputting the intermediate features stored in the memory to the second part neural network for performing partial inference (extracting and/or storing data, paragraph 570).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of the combination with the teaching of D2 in order to transform the extracted query/search-image features into task-relevant output information and attach a reliability score to each candidate, thereby producing more accurate, confidence-weighted ranking of the search results.
-Regarding claim 18, the combination further discloses a memory configured to store the second feature transformation information outputted as a result of inference performed by inputting the search object images to the neural network with respect to all combinations of activation and deactivation of the heads before the user selects the one or more of the intermediate images, wherein the processing circuitry reads from the memory the second feature transformation information corresponding to the heads relating to the one or more of the intermediate images selected by the user to calculate the degree of similarity (D2, extracting and/or storing data, paragraph 570).
-Regarding claim 19, the combination further discloses the neural network has a vision transformer configuration in which the intermediate layer includes a plurality of heads, wherein each of the heads has a multi-head attention mechanism (D2, first neural network 108 is a LLM and second neural network 114 is a vision transformer, paragraph 66).
Allowable Subject Matter
Claims 5-8, 10, and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING Y HSIEH whose telephone number is (571)270-3011. The examiner can normally be reached Monday-Friday, 9am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING Y HSIEH/ Primary Examiner, Art Unit 2664
Prosecution Insights