DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on July 29th, 2024 and November 8th, 2024 has been considered and the listed references were noted.
Drawings
The drawings are objected to because of the following informality:
In Figure 10, the label "candidates" should be either "input images" to match specification for better clarity or be better explained in the specification
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
In Paragraph [0047], "Therefore if a correlation…" should read "Therefore, if a correlation…"
Appropriate correction is required.
Claim Objections
Claim 16 objected to because of the following informalities:
"…as claimed claim 1…" should read "…as claimed in claim 1…"
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 19 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter as follows.
Claim 19 recites “A computer program which, when run on a computer…”. Computer programs,
per se, are not in one of the statutory categories of invention because a computer program is
merely a set of instructions capable of being executed by a computer - the computer program
itself is not a process. MPEP § 2106.
A computer program, at best, is a functional descriptive material per se. Descriptive material can
be characterized as either "functional descriptive material" or "nonfunctional descriptive
material." Both types of "descriptive material" are nonstatutory when claimed as descriptive
material per se, 33 F.3d at 1360, 31 USPQ2d at 1759. When functional descriptive material is
recorded on some computer-readable medium, it becomes structurally and functionally
interrelated to the medium and will be statutory in most cases since use of technology permits the
function of the descriptive material to be realized. Compare In re Lowry, 32 F.3d 1579, 1583-
84, 32 USPQ2d 1031, 1035 (Fed. Cir. 1994) )(discussing patentable weight of data structure
limitations in the context of a statutory claim to a data structure stored on a computer readable
medium that increases computer efficiency) and >In re Warmerdam, 33 F.3d *>1354, 1360-
61,31 USPQ2d *>1754, 1759 (claim to computer having a specific data structure stored in
memory held statutory product-by-process claim) with Warmerdam, 33 F.3d at 1361,31 USPQ2d
at 1760 (claim to a data structure per se held nonstatutory). See MPEP 2106.01.
The rejection of claim 19 above may be overcome by amending the claim to recite, for example,
“A non-transitory computer-readable medium storing a computer program, when is executed by
a computer, causing the computer to …”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 10-11, 15, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zou (US 2022/0058429) in view of Mo (CN 111462282 A – IDS, w/ Publication Date of 07/28/2020, whose machine translation to English is provided for mapping purposes).
Regarding Claim 1, Zou discloses “A computer-implemented method comprising:” (Zou, Paragraph [0067], discloses “FIG. 1 illustrates a system architecture of an image retrieval system 100 in accordance with an example embodiment. A computer module 111 includes trained models 101, which can be used to implement aspects of the image retrieval method 300 (FIG. 4) according to an example embodiment. The input to the trained models 101 can be a scene sketch image or a scene image in an image collection. A scene sketch image is a sketch of a scene that is stored in image format and received from the user equipment 140. The image collection can be found in an image repository which stores image files of images, typically photographs of objects and scenes, such as data storage system 150. The trained models 101 can include target/model rules.”); “based on an input scene graph, generating a plurality of graph vectors; encoding an input image to generate a plurality of image vectors” (Zou, Paragraph [0068] and Figure 1, discloses: “As shown in FIG. 1, the trained model 101 can include a trained segmentation model 101A which includes a convolution neural network configured to generate a segmentation image of the scene sketch image or a scene image in the image collection, which includes object instances from the image, in which each monochromatic color represents one object instance. For example, the trained segmentation model 101A can be used to generate, for each object instance identified in the scene image, one or more attributes including: i) category label, ii) size and location identification (alternatively bounding box identification), iii) visual attributes, and iv) other information. The trained segmentation model 101A can generate a fine-grained feature vector of the visual attributes for each of the object instances. The term fine-grained feature vector is used because fine-grained attributes such as the visual attributes are contained, and which can be used for comparison purposes. The computation module 111 is configured to generate an attribute graph which integrates the fine-grained features for each semantic object instance detected from the scene sketch image. As shown in FIG. 1, the trained model 101 can include a graph encoder model 101B which includes a graph neural network (GNN) configured to generate a feature graph that encodes attributes of the scene sketch image or a scene image in the image collection. The graph encoder model 101B can generate the feature graph from the attribute graph. The feature graph of the scene sketch image can be compared to a respective feature graph of each scene image in the image collection, and the target scene image(s) that have the highest graph similarity are retrieved by the image retrieval method 300.”; Here, a graph vector is defined by the feature vector comprised of the various attributes of an object within the scene image, which is then incorporated to create a graph based on the feature vector for each object instance, thus making a plurality of graph vectors for future processing)
PNG
media_image1.png
679
809
media_image1.png
Greyscale
;
PNG
media_image2.png
732
344
media_image2.png
Greyscale
vectors.” (Zou, Paragraphs [0098]-[0102] and Figure 6A discloses the following:
PNG
media_image3.png
63
349
media_image3.png
Greyscale
PNG
media_image4.png
511
648
media_image4.png
Greyscale
). It is important to note that the feature graph (derived from feature vectors, thus being analogous to graph vectors) are being compared to the feature graphs and feature vectors of other images to ensure that the feature graph is similar to what was given in the scene graph and the feature vector (which is interpreted as an object query vector since it contains attributes about the object as a whole), and that similarity, as seen in Paragraph [0102], is computed through a matching score between the image and the scene graph. Zou does not explicitly disclose “performing an update process to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, wherein the update process comprises: updating the at least one object query vector based on the plurality of graph vectors; updating the at least one object query vector based on the plurality of image vectors; and updating the plurality of graph vectors based on the at least one object query vector; extracting from the at least one updated object query vector information indicating a region of at least one object and a category of the at least one object”. However, in an analogous field of endeavor, Mo discloses the following in Paragraphs [0057] - [0061]:
PNG
media_image5.png
338
842
media_image5.png
Greyscale
. For these paragraphs, it is important to note that the object node (which would be synonymous to the object query vector as it is a target object formed into a vector from being multiplied by the feature vector) is being updated accordingly based off of another node that is incorporated in a graph (which, as described before, is synonymous to being a graph vector). This means that the graphs that are found from the images are playing a direct influence to the updating of the object node. Mo also discloses the following in [0067]-[0078]:
PNG
media_image6.png
101
922
media_image6.png
Greyscale
For Paragraphs [0067]-[0073], we can clearly see that the node (which, as stated above, correlates with being an object query vector, as can be seen with the feature vectors being incorporated into the update formula) is being updated accordingly based on the plurality of image vectors (as seen in Paragraph [0068]);
PNG
media_image6.png
101
922
media_image6.png
Greyscale
PNG
media_image7.png
352
916
media_image7.png
Greyscale
In Paragraphs [0074]-[0084], the plurality of graph vectors are the feature vectors in this instance, as the weights are being determined to update them and will further, as we have seen previously, be incorporated into graphs. Additionally, the nodes are considered object query vectors as they contain the feature vectors from before while being updated to be characteristic vectors, which takes advantage of the image area descriptions to indicate the region (i.e., image area) where the target object would be in the image. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the technique of generating a plurality of graph vectors, image vectors, and matching scores seen in Zou with the method of updating the object query vector and plurality of graph vectors seen in Mo to achieve a complete computer-implemented method for detecting objects within an image. By implementing both of these techniques, one of ordinary skill in the art could effectively update the query that a user may think of based on the generated vectors to adequately detect specific objects within an image or video frames in a video. Therefore, it would have been obvious for one of ordinary skill in the art to combine both the Zou and Mo references to achieve a complete method as described in Claim 1.
Regarding Claims 2, 3, and 4, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1, wherein the update process comprises” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); and the following limitations: “iteratively updating the at least one object query vector based on the plurality of graph vectors.”, “iteratively updating the at least one object query vector based on the plurality of image vectors”, and “iteratively updating the plurality of graph vectors based on the at least one object query vector.”. (Zou, Paragraph [0014], discloses: “s3: the object, the visual relation and the image area description are regarded as different semantic levels understood by a scene image, the relation among different levels of semantics is established according to different semantic spaces and semantic relations, nodes in different semantic levels are connected through an information transfer graph, information can be transferred among different semantic features through edges in the graph so as to carry out feature combined iterative update on the semantic information of different levels, and three semantic tasks of different levels respectively correspond to three feature information iterative updates: updating object characteristic information, updating visual relation characteristic information and updating image area description characteristic information, continuously iterating the characteristic updating process until the characteristics of a semantic layer are converged, wherein the visual tasks at three different levels correspond to three parallel network branches, the respective corresponding characteristics of the visual tasks are sent to the corresponding pooling layer, and the output of each branch network is classified by using two full connection layers, so that the different network branches learn the characteristics of the corresponding visual tasks”). Here, we can see that the object characteristic information (analogous to object query vector) is being updated iteratively based on the graph (which can also apply to other images, making it a plurality of graph vectors) and the image area description (which is analogous to image vectors from what was disclosed above). The graph (which can lead to a plurality of graphs) is in turn updated iteratively through the iterative update of the object characteristic information. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to use the technique of iteratively updating the object vector and plurality of graph vectors seen in the combination of Zou and Mo to be able to improve the computer-implemented method in the same way for accuracy. By using the technique of iterative updates of the object and graph vectors seen in the combination of Zou and Mo, one of ordinary skill in the art could ensure the object detection is accurate as the similarities between them become more apparent. Therefore, it would have been obvious for one of ordinary skill in the art to use the combination of Zou and Mo to achieve the same methods discloses in Claims 2, 3, and 4.
Regarding Claim 10, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); “wherein the input scene graph comprises a mask node for which a label is to be predicted” (Zou, Paragraph [0024], discloses “In an example embodiment, the generating the respective graph similarity includes generating a category mask of the scene sketch image and a category mask of each respective image in the image collection, and computing an intersection of the category mask of the scene sketch image and the category mask of at least one respective scene image in the image collection.”), and “wherein the computer-implemented method comprises extracting information about the mask node from the plurality of updated graph vectors and/or the at least one updated object query vector to predict the label of the mask node based on the input image” (Zou, Paragraph [0141] and Figures 10A-10C, disclose: “FIGS. 10A, 10B and 10C illustrate another example graph matching method performed by the graph matching module 618, for use in the image retrieval method 300, in accordance with another example embodiment. FIG. 10A shows the scene sketch image 402 that is input to the image retrieval system 400. FIG. 10C illustrates an image generated by the example graph matching method. In an example, the image retrieval method 300 performed by the image retrieval system 400 generates a mask image of the object categories detected from the scene sketch image. For example, FIG. 10B show the target scene image 404. FIG. 10C shows the mask image of the scene sketch image 402 overlaid on the target scene image 404. In FIG. 10C, the mask of the first giraffe 1002 and the mask of the second giraffe 1004 in the scene sketch image 402 are a first color and the mask of the car 1006 in the scene sketch image 402 is a second color. In FIG. 10C, the mask of the first giraffe 1002 and the mask of the second giraffe 1004 in the scene sketch image 402 are grey color and the mask of the car 1006 in the scene sketch image 402 is black color.”
PNG
media_image8.png
640
223
media_image8.png
Greyscale
; Paragraphs [0057]-[0061] disclose the following (As previously described in Claim 1, object query vectors are represented by object nodes):
PNG
media_image5.png
338
842
media_image5.png
Greyscale
). For Figures 10A-10C, objects are defined as nodes, so this correlates to each of the objects (i.e. giraffes) masked in each of the images. Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to use the technique of using mask nodes to predict the label of an object seen in the combination of Zou and Mo to achieve the method described in Claim 10.
Regarding Claim 11, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1, wherein updating the plurality of graph vectors based on the at least one object query vector comprises” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); “adding and/or updating information in the graph vectors indicating a mask node based on information of at least one object in the input image.” (Zou, Paragraphs [0024]-[0027], disclose the following:
PNG
media_image9.png
531
571
media_image9.png
Greyscale
; Paragraphs [0100]-[0101], disclose the following:
PNG
media_image10.png
373
408
media_image10.png
Greyscale
). From the above paragraphs, it can be seen that the graph vectors (created by the feature vectors being inputted into a graph, as described in Claim 1) are being updated in terms of the applying weights on objects (defined as nodes) calculated from the graph similarities based on the category mask (which is analogous to a mask node, since they both serve as masks to a specific object within the input image). Therefore, it would have been obvious for one of ordinary skill in the art before the effective to use the technique of updating the graph vectors based on the object query vector and/or graph vectors indicating a mask node seen in the combination of Zou and Mo to achieve an improved computer-implemented method. By doing this, the connections between different graph vectors within the input scene graph can be strengthened, leading to an improvement in classifying various objects within the image. Therefore, it would have been obvious for one of ordinary skill in the art to use the combination of Zou and Mo to achieve the same method described in Claim 11.
Regarding Claim 15, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); “wherein the input image and the input scene graph are a training image and a training scene graph, respectively, and are associated with at least one of a training region, training category, a training mask node label, and training matching score” (Zou, Paragraph [0067], discloses: “FIG. 1 illustrates a system architecture of an image retrieval system 100 in accordance with an example embodiment. A computer module 111 includes trained models 101, which can be used to implement aspects of the image retrieval method 300 (FIG. 4) according to an example embodiment. The input to the trained models 101 can be a scene sketch image or a scene image in an image collection. A scene sketch image is a sketch of a scene that is stored in image format and received from the user equipment 140. The image collection can be found in an image repository which stores image files of images, typically photographs of objects and scenes, such as data storage system 150. The trained models 101 can include target/model rules.”); and “wherein the computer-implemented method further comprises comparing at least one of the region, category, mask node label, and matching score with at least one of the training region, training category, training mask node label, and training matching score, respectively, and updating at least one network weight based on the comparison” (Zou, Paragraphs [0100]-[0102] & [0145] discloses the following:
PNG
media_image10.png
373
408
media_image10.png
Greyscale
PNG
media_image11.png
79
406
media_image11.png
Greyscale
PNG
media_image12.png
832
575
media_image12.png
Greyscale
As seen in the following paragraphs, the input scene images from the trained model is being compared with the matching score to determine which score has the highest graph similarity of the image. Paragraph [0145] expands on the comparison between the two graphs to compute the score as well by giving a computation on adjusting the weighted distance (synonymous to network weight) based on the distance between the pairs of graphs. Therefore, it would have been obvious for one of ordinary skill in the art to use the techniques for comparing the input image and input scene graph that are training images and training scene graphs with one another to achieve the same method described in Claim 15.
Regarding Claim 18, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); “wherein updating the at least one object query vector based on the plurality of image vectors may comprise using an attention-based network or attention network.” (Mo, Paragraphs [0074]-[0084], discloses the following:
PNG
media_image6.png
101
922
media_image6.png
Greyscale
PNG
media_image7.png
352
916
media_image7.png
Greyscale
). Therefore, it would have been obvious for one of ordinary skill in the art to use the technique for updating the object query vector from using an attention -based network seen in the combination of Zou and Mo to achieve the same method disclosed in Claim 18.
Claim 19 recites a computer program with instructions corresponding to the steps recited in Claim 1. Therefore, the recited programming instructions of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Zou and Mo references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of Zou and Mo references discloses a program stored on a computer readable storage medium (for example, see Zou, Paragraph [0030]).
Claim 20 recites an apparatus (information processing apparatus) with features corresponding to the steps of the method recited in Claim 1. Therefore, the recited features of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Zou and Mo references, presented in rejection of Claim 1, apply to this claim. Finally, the combination of Zou and Mo references discloses an apparatus containing a processor and memory (for example, see Zou, Paragraph [0029]).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Zou in view of Mo, and further in view of Kottur (US 2025/0148784, w/ an effective filing date of September 6th, 2020) and Cherian (WO 2023106007 A1 w/ Publication Date of June 15th, 2023).
Regarding Claim 5, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1, wherein the update process comprises” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); understanding unit 520 (e.g., an action predictor), and a dialog state tracker 337 (e.g., a graph manipulator). In particular embodiments, the fact encoder 210 may receive a “fact” about a piece of sensor data, such as an image. As examples and not by way of limitation, such facts may include a caption in a pre-analyzed image in a training phase, text converted from user speech at the ASR 208, or a response to a user query about some property of an object (such as the color or price of an object of interest). The fact F.sub.t may be encoded via an RNN to produce a fact embedding f.sub.t used in subsequent stages of the scene generator 600. This fact, which provides the assistant system 140 with initial information about an image or other piece of sensor data, may be input to the fact encoder 210, which may generate an output feature vector representing the input fact” (Kottur, Paragraph [0116] and Figure 6 (see below)).
PNG
media_image13.png
371
569
media_image13.png
Greyscale
Here, the “fact” mentioned is an example of an input matching query vector, as it tells the specific query to the system performing the method, thus encoding that fact into a vector. Kottur also discloses that “In particular embodiments, the fact embedding output from the fact encoder may then be fed into the action predictor 520, which may generate instructions for how a scene graph should be created or modified based on this fact. As an example and not by way of limitation, if a user indicates an a new object of interest (such as a dress in a shopping mall), the action predictor 520 may determine that the resulting action should be to add a new node to the scene graph representing this dress. As another example and not by way of limitation, if the user subsequently asks about attributes of the dress (e.g., price, fabric, etc.), the responses to these queries (e.g., $50, cotton, etc.) may be facts input into the fact encoder 210, with the resulting output feature vectors/fact embeddings input into the action predictor 520, which may in turn determine that the appropriate action to take is to store these attributes in association with the “dress” node. As yet another example and not by way of limitation, if the user query concerns relational information between the dress and another object visible in the image, this relational information may be determined and then input as the fact, and the action predictor 520 may determine that the appropriate actions are to generate a new node in the scene graph for this other object, and then to generate an edge between these two nodes indicating the relational information. Thus, the action predictor 520 may consume the fact embedding and decode a series of actions along with an action vector for each action. These actions and their vectors together may constitute scene graph updates for the current fact, and the output may be used to dynamically update the scene graph as rounds of dialog progress. As an example and not by way of limitation, Table 1 below illustrates several actions that may be performed.” (Kottur, Paragraph [0117]). As seen in Paragraph [0117], the vector described in Paragraph [0116] is then fed into the action predictor, which examines the information it has within the image and the query, and tries to update the input fact (or input matching query vector) based on the relational information it has by adding new node and edges to then create an action vector denoting steps correlating with the query, which is a process analogous to updating the input matching query vector. Therefore, it would have been obvious for one ordinary skill in the art before the effective filing date of the claimed invention to combine the computer implemented method disclosed in the combination of Zou and Mo with the technique of updating the input matching query vector seen in Kottur to achieve the aforementioned limitations in Claim 5.
The combination of Zou, Mo, and Kottur does not explicitly disclose “wherein computing the matching score comprises computing the matching score based on the updated matching query vector”. However, in an analogous field of endeavor, Cherian discloses the following in Paragraphs [0065]-[0071] and Figure 6B:
PNG
media_image14.png
87
483
media_image14.png
Greyscale
PNG
media_image15.png
122
478
media_image15.png
Greyscale
PNG
media_image15.png
122
478
media_image15.png
Greyscale
PNG
media_image16.png
683
480
media_image16.png
Greyscale
PNG
media_image17.png
471
475
media_image17.png
Greyscale
PNG
media_image18.png
544
840
media_image18.png
Greyscale
In the paragraphs above, Cherian describes a thorough process of computing a matching score based on an updated matching query vector, with the important details being the fact that the input vector is updated accordingly based on the input query vector, and the updated matching query vector is in turn used to compute the attention score for specific words within the vector. Although it does not describe the exact matching score, the attention score is analogous to the process of determining the matching score based on the updated matching query vector. Therefore, it would have been obvious for one of ordinary skill in the art to combine the computer implemented method described in the combination of Zou, Mo, and Kottur with the technique of computing the matching score based on the updated query vector seen in Cherian to achieve the same method disclosed in Claim 5.
Claims 6, 8, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Zou in view of Mo, and further in view of Miao (CN 114764455 A).
Regarding Claim 6, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1, wherein updating the at least one object query vector based on the plurality of graph vectors comprises” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); media object, in which a relationship diagram may be constructed according to a feature vector sequence of the streaming media object and a user behavior sequence of a user, and then a prediction model is iteratively trained based on the relationship diagram, during training, the feature vector sequences of a plurality of streaming media objects are updated based on the relationship diagram, so as to obtain a plurality of prediction representation vectors, and an association relationship between any two streaming media objects is predicted according to the prediction representation vectors, and a parameter of the prediction model is updated according to a difference between the predicted association relationship and an actual association relationship, so that, since the relationship diagram is constructed according to the user behavior sequence, and the representation vectors of the streaming media objects are updated according to the relationship diagram, and when the prediction model converges, the obtained representation vectors of each streaming media object may be integrated with behavior information of the user, the comprehensiveness of information contained in the expression vector of the streaming media object is increased, so that the accuracy of vector expression of the streaming media object is improved, and a foundation is laid for subsequent matching and recommendation of the streaming media object” (Miao, Paragraph [0060]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the computer-implemented method seen in the combination of Zou and Mo with the technique for updating the one object query vector based on the correlation between the object vector and plurality of graph vectors seen in Miao to achieve the same method described in Claim 6.
Regarding Claim 8, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1 wherein updating the plurality of graph vectors based on the at least one object query vector comprises” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); (Miao, Paragraph [0060]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the computer-implemented method seen in the combination of Zou and Mo with the technique of updating one of the graph vectors based on the correlation between the object query vector and plurality of graph vectors seen in Mao to improve the method described in Claim 8.
Regarding Claim 9, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); initial expression vector and an attention weight value corresponding to each initial expression vector, respectively” (Miao, Paragraph [0159]). Miao also discloses “For a streaming media object, in order to retain information of the streaming media object itself during aggregation, the above aggregation process may be performed with the streaming media information itself as a first-degree node. (Miao, Paragraph [0160])”. Finally, Miao discloses “In the embodiment of the application, a relational graph can be constructed according to a feature vector sequence of a streaming media object and a user behavior sequence of a user, a prediction model is iteratively trained based on the relational graph, during training, the feature vector sequences of a plurality of streaming media objects are respectively updated based on the relational graph, a plurality of prediction expression vectors are correspondingly obtained, an association relationship between any two streaming media objects is predicted according to the prediction expression vectors, and parameters of the prediction model are updated according to the difference between the predicted association relationship and an actual association relationship, so that the obtained expression vectors of the streaming media objects can be integrated with behavior information of the user when the prediction model converges, and the comprehensiveness of information contained in the expression vectors of the streaming media objects is increased, and the accuracy of vector representation of the streaming media object is further improved, and a foundation is laid for subsequent matching and recommendation of the streaming media object.)” (Miao, Paragraph [0060]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the computer-implemented method seen in the combination of Zou and Mo with the technique of updating the input matching query vector based on the similarity between the one object query vector and the plurality of graph vectors seen in Miao to achieve the same method of Claim 9.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Zou in view of Mo, and further in view of Turkelson (US 2021/0004589).
Regarding Claim 7, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1, wherein updating the at least one object query vector based on the plurality of image vectors comprises” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); exclusive attributes. For example, exclusive attributes may indicate whether an image depicts a scene that is indoors or outdoors, while non-exclusive attributes may classify the scene as a living room decorated for a holiday. The output from the scene classification model may be passed to an object recognition model along with the image for which the scene was classified.” (Turkelson, Paragraph [0046]). Here, the scene classification vector is analogous to the object query vector, as it shows specific objects that one may inquire about when the scene classification vector is fed into an object recognition model. Turkelson also discloses “In some embodiments, the captured image (or set of images, such as those in a video preceding or following (or both) a frame in which the event occurred) and the coordinate location (or other parameters of the UI event) may be provided to a computer-vision object recognition system (which in some cases, may be an object detection and localization system). The object recognition system, which may include or use an object recognition model, may output a score for an object in an ontology of objects indicative of a confidence level that the object was recognized (e.g., the object was detected in the image, the object was selected, or both, where selection indicates that the selected object accords with the user's intent). Some embodiments may output such scores for each of a plurality of objects in an object ontology (e.g., in an object detection vector) and, in some cases, bounding polygons (with vertices expressed in pixel coordinates) of each object. For example, a feature vector may be generated from an input image, where dimensions correspond to features (like edges, blobs, corners, colors, and the like) in the input image. The feature vector may be input into a discriminative computer vision object recognition model, which may match the feature vector to a closest feature vector of an object in a labeled training set of images. Some embodiments may select an object having a highest score based on such a distance (e.g., upon determining the distance is greater than a threshold) as the object in the image. In some embodiments, the score may be used to select an object to be searched for (e.g., against a product catalog or object database). In some embodiments, data associated with the detected event may be used as training data for training an object recognition model to perform object recognition.” (Turkelson, Paragraph [0059]). In this paragraph, we can clearly see that the image can be represented as a feature vector (synonymous to an image vector), and the object feature vector is correlated to the set of input images to see where the object is found in the training set of images (each containing their own respective image vectors) to then update the object feature vector to determine where the object is located in the image (based off of the object detection vector). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the computer-implemented method seen in the combination of Zou and Mo with the technique for updating the object vector based on the correlation between one object query vector and a plurality of image vectors seen in Turkelson to achieve the method seen in Claim 7.
Claims 12-14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zou in view of Mo, and further in view of Roy (US 20220147743 A1).
Regarding Claim 12, the combination of Zou and Mo discloses “A computer-implemented method comprising, based on an input scene graph, performing the computer-implemented method as claimed in claim 1” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); (Zou, Abstract, discloses: “A sketch-based image retrieval method, device and system, to improve accuracy of image searching from a scene sketch image. For example, the image retrieval method, device and system can be used to retrieve a target scene image from a collection of stored images in a storage (i.e., an image collection). The image retrieval method includes: segmenting the scene sketch image using an image segmentation module into semantic object-level instances, and fine-grained features are obtained for each object instance, generating an attribute graph which integrates the fine-grained features for each semantic object instance detected from the query scene sketch image, generating a feature graph by using a graph encoder module from the attribute graph, and computing a similarity or distance between the feature graphs of the query scene sketch image and the scene images in the image collection by a graph matching module and the most similar scene images are returned”). The combination of Zou and Mo does not explicitly disclose “a plurality of times with different input images, respectively”. However, in analogous field of endeavor, Roy discloses “In at least one embodiment, a user may select pre-trained model 1406 that is to be updated, retrained, and/or fine-tuned, and pre-trained model 1406 may be referred to as initial model 1504 for training system 1304 within process 1500. In at least one embodiment, customer dataset 1506 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices at a facility) may be used to perform model training 1314 (which may include, without limitation, transfer learning) on initial model 1504 to generate refined model 1512. In at least one embodiment, ground truth data corresponding to customer dataset 1506 may be generated by training system 1304. In at least one embodiment, ground truth data may be generated, at least in part, by clinicians, scientists, doctors, practitioners, at a facility (e.g., as labeled clinic data 1312 of FIG. 13)” (Roy, Paragraph [0151]). Roy also discloses “In at least one embodiment, once customer dataset 1506 has associated ground truth data, ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used by during model training 1314 to generate refined model 1512. In at least one embodiment, customer dataset 1506 may be applied to initial model 1504 any number of times, and ground truth data may be used to update parameters of initial model 1504 until an acceptable level of accuracy is attained for refined model 1512. In at least one embodiment, once refined model 1512 is generated, refined model 1512 may be deployed within one or more deployment pipelines 1410 at a facility for performing one or more processing tasks with respect to medical imaging data.” (Roy, Paragraph [0154]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the computer implemented method that includes finding the best input image corresponding to the input scene graph with the technique for implementing such a method a plurality of times seen in Roy to achieve the complete method disclosed in Claim 12.
Regarding Claim 13, the combination of Zou, Mo, and Roy discloses “The computer-implemented method as claimed in claim 12 (Zou, Abstract, Paragraphs [0067]-[0068] & [0098]-[0102], as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 12) and “comprising ranking the input images based on their matching scores, and selecting at least one of the input images with the highest matching score.” (Zou, Paragraph [0101], discloses “Each of the images in the image collection 406 are also represented by a respective feature graph 624. The feature graph 616 of the scene sketch image 402 can be compared with the other feature graphs 624 to find the most similar scene image. The graph matching module 618 performs graph matching between the feature graph 616 of the scene sketch image 402 and each of the other feature graphs 624 of the scene images in the image collection 406, and outputs the respective graph similarity 620 for all of the scene images in the image collection 406. A ranking/cutoff module 622 ranks the graph similarities from the graph matching module 618 and cuts off scene images that are dissimilar. For example, the ranking/cutoff module 622 outputs a specified number of scene images that have the highest similarity to the feature graph 616 of the scene sketch image 402.”). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to use the method for ranking the input imaged and selecting at least one of the input images with the highest matching score seen in the combination of Zou, Mo, and Roy to achieve the same method described in Claim 13.
Regarding Claim 14, the combination of Zou, Mo, and Roy discloses “The computer-implemented method as claimed in claim 12” (Zou, Abstract, Paragraphs [0067]-[0068] & [0098]-[0102], as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 12), “wherein the input images are a series of images from a video.” (Zou, Paragraph [0066], discloses “Example embodiments that relate to images can be similarly applied to video. For example, a video can be considered a sequence of images (generally referred to as video frames). An image retrieval method in accordance with an example embodiment can be used to retrieve a particular target video frame, or the entire video from a video collection (i.e. a collection of stored videos in a storage).”). Therefore, it would have been obvious for one of ordinary skill in the art to incorporate input images that are a series of image from a video seen in the combination of Zou, Mo, and Roy to achieve the same computer-implemented method of Claim 14.
Regarding Claim 16, the combination of Zou and Mo discloses “A computer-implemented method comprising, based on an input scene graph, performing the computer-implemented method as claimed in claim 1” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); (Zou, Paragraph [0066], discloses “Example embodiments that relate to images can be similarly applied to video. For example, a video can be considered a sequence of images (generally referred to as video frames). An image retrieval method in accordance with an example embodiment can be used to retrieve a particular target video frame, or the entire video from a video collection (i.e. a collection of stored videos in a storage).”); and “wherein the method comprises selecting at least one object detected in a plurality of the input images as a target node.” (Zou, Paragraph [0012], discloses: “In an example embodiment, the image retrieval method of the present disclosure uses a feature graph generating method. The feature graph generating method can be performed on the scene sketch image and each of the scene images in the image collection. Nodes of each feature graph each represent a detected object instance in the respective scene image and an attribute feature vector which contains attributes of that object instance. Example attributes of the respective attribute feature vector for each object instance include: i) category label, ii) size and location identification (alternatively bounding box identification), iii) visual attributes, and iv) other information. Edges of the nodes represent a weight between one of the nodes and at least one of the other nodes. The feature graph of the scene image retrieved in the image collection that has the highest similarity to the feature graph of the scene sketch image is considered to belong to the target scene image, and the target scene image is then retrieved and output as the target image by the image retrieval method of the present disclosure.”). The combination of Zou and Mo does not explicitly disclose “a plurality of times with different input images, respectively”. However, in analogous field of endeavor, Roy discloses “In at least one embodiment, a user may select pre-trained model 1406 that is to be updated, retrained, and/or fine-tuned, and pre-trained model 1406 may be referred to as initial model 1504 for training system 1304 within process 1500. In at least one embodiment, customer dataset 1506 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices at a facility) may be used to perform model training 1314 (which may include, without limitation, transfer learning) on initial model 1504 to generate refined model 1512. In at least one embodiment, ground truth data corresponding to customer dataset 1506 may be generated by training system 1304. In at least one embodiment, ground truth data may be generated, at least in part, by clinicians, scientists, doctors, practitioners, at a facility (e.g., as labeled clinic data 1312 of FIG. 13)” (Roy, Paragraph [0151]). Roy also discloses “In at least one embodiment, once customer dataset 1506 has associated ground truth data, ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used by during model training 1314 to generate refined model 1512. In at least one embodiment, customer dataset 1506 may be applied to initial model 1504 any number of times, and ground truth data may be used to update parameters of initial model 1504 until an acceptable level of accuracy is attained for refined model 1512. In at least one embodiment, once refined model 1512 is generated, refined model 1512 may be deployed within one or more deployment pipelines 1410 at a facility for performing one or more processing tasks with respect to medical imaging data.” (Roy, Paragraph [0154]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the computer implemented method seen in Claim 1 including input images being a series of images in a video and selecting one object in the input images as a target node seen in the combination of Zou and Mo with the technique for implementing such a method a plurality of times seen in Roy to achieve the complete method disclosed in Claim 16.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Zou in view of Mo, and further in view of Cherian (WO 2023106007 A1).
Regarding Claim 17, the combination of Zou and Mo discloses “The computer-implemented method as claimed in claim 1” (Zou, Paragraphs [0067]-[0068] & [0098]-[0102] as well as Figures 1 and 6A; Mo, Paragraphs [0057]-[0061] & [0067]-[0084], please refer to the above-described analysis for Claim 1); properties of multiple nodes of the spatio-temporal scene graph describing an appearance, a location, and a motion of each of the dynamic objects at different instances of time. The method includes encoding the nodes of the spatio-temporal scene graph into a latent space using a spatiotemporal transformer encoding different combinations of different nodes of the spatio-temporal scene graph corresponding to different spatio-temporal volumes of the scene, wherein encoding of each node of the different nodes in each of the combinations is weighted with an attention score determined as a function of similarities of spatio-temporal locations of the different nodes in the combination. The method further includes outputting the encoded nodes of the spatio-temporal scene graph” (Cherian, Paragraph [0015]). Cherian also discloses “Further, in some embodiments, the spatio-temporal transformer 206 encodes different combinations of different nodes of the spatio-temporal scene graph 214 corresponding to different spatio-temporal volumes of the scene 102 into a latent space. The encoding of each node of the different nodes in each of the combinations is weighted with an attention score determined as a function of similarities of spatio-temporal locations of the different nodes in the combination, details of which are described later with reference to FIG. 6B.” (Cherian, Paragraph [0038]). Therefore, it would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the computer implemented method seen in the combination of Zou and Mo with the technique of updating the object query vector based on the plurality of graph vectors using an attention network seen in Cherian to achieve the same method described in Claim 17.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Maheswari et al. (US 2022/0391433) teaches systems and methods for image processing, including identifying an image including a plurality of objects, generating scene graph of an image, generating a node vector for the node and an edge vector for the edge, generating a scene graph embedding based on both the node and edge vectors using a GCN (graph convolutional network), and assign metadata to the image based on the scene graph embedding.
Chen (CN 112044082 A) teaches an information detection method, a device, and a computer readable storage medium that obtains target correlation matrix formed by the characteristic matrix and interaction behavior of the object set to be detected.
Zepeda et al. (US 20170262478 A1) teaches a method for retrieving at least one search image matching a query image commences by first extracting a set of search images, where the query image is encoded into an image query vector and the search images are encoded into search image feature vectors
Chang et al. (A Comprehensive Survey of Scene Graphs: Generation and Application) teaches a comprehensive investigation of current scene graph research, which relates to many aspects of this application.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SORIE I KOROMA JR whose telephone number is (571)272-9259. The examiner can normally be reached Monday - Friday 8AM-6:00PM; Alternate Fridays Off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SORIE I KOROMA JR/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights