DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-9, and 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over MA (US 20210312211 A1), and in view of Zhao (US 20230245435 A1).
Re Claim 1, MA discloses an image-language alignment model training method (see MA: e.g., --a training method of an image-text matching model, a bi-directional search method, and a relevant apparatus. The training method includes extracting a global feature and a local feature of an image sample; extracting a global feature and a local feature of a text sample; training a matching model according to the extracted global feature and local feature of the image sample and the extracted global feature and local feature of the text sample, to determine model parameters of the matching model; and determining, by the matching model, according to a global feature and a local feature of an inputted image and a global feature and a local feature of an inputted text, a matching degree between the image and the text.--, in abstract) comprising:
a first generation step of generating, by the image-language alignment model, an object representation vector for each object of an image in the inputted image (see MA: e.g., --extracting a global feature and a local feature of an image sample--, in [0008]; and, --The local feature of the image sample is also a vector, and information of each dimension of the vector represents that a local area of the image includes information--, in [0038]-[0039]);
a second generation step of generating, by the image-language alignment model, an object representation vector for each object of a text in the inputted text (see MA: e.g., --he global feature of the text sample is used to represent overall semantic information of the text sample. The local feature of the text sample is used to represent integration of local semantic information of the text. The local feature of the text sample focuses on local information, such as a word or a phrase, of the text.--, in [0040]-[0041]);
a step of training the image-language alignment model through target {loss} function by using the object representation vector generated at the first generation step and the object representation vector generated at the second generation step (see MA: e.g., --[0107] The training module 903 is configured to: determine a similarity of a semantically associated image-text pair represented by the parameters of the at least two fully connected layers, a similarity of a non-semantically associated image-text pair represented by the parameters of the at least two fully connected layers, a similarity between text samples associated with a same image sample represented by the parameters of the at least two fully connected layers, and a similarity between text samples associated with different image samples represented by the parameters of the at least two fully connected layers according to the global feature and the local feature of the image sample represented by the parameters of the at least two fully connected layers, and the global feature and the local feature of the text sample represented by the parameters of the at least two fully connected layers; and input the similarity of the semantically associated image-text pair represented by the parameters of the at least two fully connected layers, the similarity of the non-semantically associated image-text pair represented by the parameters of the at least two fully connected layers, the similarity between text samples associated with the same image sample represented by the parameters of the at least two fully connected layers, and the similarity between text samples associated with different image samples represented by the parameters of the at least two fully connected layers, to determine the parameters of the at least two fully connected layers.
[0108] In some embodiments, the preset target function is determined through one of the following:
LY=L(I.sub.n.sup.+,S.sub.n.sup.+,I.sub.n.sup.−,S.sub.n.sup.−)+L(S.sub.i,S.sub.l,S.sub.j) (3); and
LY=L(I.sub.n.sup.+,S.sub.n.sup.+,I.sub.n.sup.−,S.sub.n.sup.−)*L(S.sub.i,S.sub.l,S.sub.j) (4).
[0109] where L(I.sub.n.sup.+,S.sub.n.sup.+,I.sub.n.sup.−,S.sub.n.sup.−)=λ.sub.1 max(0, u.sub.1+d(I.sub.n.sup.+,S.sub.n.sup.+)−d(I.sub.n.sup.+,S.sub.n.sup.−))+λ.sub.2 max(0, u.sub.2+d(I.sub.n.sup.+,S.sub.n.sup.+)−d(I.sub.n.sup.−,S.sub.n.sup.+)), where LY is a preset target function, L(I.sub.n.sup.+,S.sub.n.sup.+,I.sub.n.sup.−,S.sub.n.sup.−) represents a relationship between a similarity of a semantically associated image-text pair and a similarity of a non-semantically associated image-text pair, and L(S.sub.i, S.sub.l, S.sub.j) represents a relationship between a similarity between text samples associated with a same image sample and a similarity between text samples associated with different image samples; I.sub.n.sup.+, I.sub.n.sup.− represent a text sample; S.sub.n.sup.+, S.sub.n.sup.− represent an image sample; in a case that a similarity between global features is determined, d( ) represents a similarity between global features of the text sample and the image sample; in a case that a similarity between local features is determined, d( ) represents similarity between local features of the text sample and the image sample;--, in 0107]-[0109]; also see:
--the preset target function needs to also require that a similarity between text samples associated with a same image sample is higher than a similarity between text samples associated with different image samples. To reach the objective, a formula for making a similarity between text samples associated with a same image sample higher than a similarity between text samples associated with different image samples is shown as formula (2):
L(S.sub.i,S.sub.l,S.sub.j)=max(0,u.sub.3+d(S.sub.i,S.sub.l)−d(S.sub.i,S.sub.j)) (2)
where L(S.sub.i, S.sub.l, S.sub.j) represents a preset target function that requires that a similarity between text samples associated with a same image sample is higher than a similarity between text samples associated with different image samples;….
--[0058] According to an effect of the preset target function, a depth matching model as shown in FIG. 4 may be trained. The depth matching model is used to determine a matching degree between an image and a text. In FIG. 4, a feature vector (image embedding) of an image (the feature vector is a feature vector of a global feature or a local feature) is inputted on a left side, and a feature vector (sentence embedding) of a text (the feature vector is a feature vector of a global feature or a feature vector of a local feature) is inputted on a right side. After the feature vectors of the image and the text are mapped to a same specified semantic space, a distance between the feature vectors is calculated. For example, in FIG. 4, S.sub.l.sup.1, S.sub.l.sup.2, S.sub.l.sup.3, S.sub.l.sup.4, and S.sub.n represent feature vectors of a text, and I.sub.1 and I.sub.n represent feature vectors of an image. In the specified semantic space, a distance between the image and the text is calculated. If the feature vectors used are feature vectors for representing global features, a calculation result is a similarity between the global features. If the feature vectors used are feature vectors for representing local features, a calculation result is a similarity between the local features.--, in [0055]-[0058]);
MA however does not explicitly disclose training the image-language alignment model through a contrastive loss function by using the object representation vector generated at the first generation step and the object representation vector generated at the second generation step;
Zhao discloses training the image-language alignment model through a contrastive loss function by using the object representation vector generated at the first generation step and the object representation vector generated at the second generation step (see Zhao: e.g., --raining a machine learning model using the training data based on an optimization of a combination of losses. The losses include, for each paired image-text pair, (i) a first multi-modal representation loss based on the paired image-text pair and (ii) a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs includes either the image or the text of the paired image-text pair.--, in abstract; and, --[0002] Contrastive learning may be used in self-supervised learning applications with visual data. For large-scale datasets having image-text pairs, contrastive learning may be used for pre-training to improve transfer learning on downstream tasks, which may improve performance in accuracy and robustness compared to other image-text matching methods when considering fine-tuning of downstream tasks, such as visual question answering (VQA), graph question answering (GQA), referring expressions, caption generation, and image retrieval. As mentioned above, these contrastive-learning approaches have focused on the pre-training stage with multi-modal datasets having image and text pairs to boost the performance of downstream tasks.--, in [0002], and, --[0052] In general, multi-modal architectures for downstream tasks typically involve learning the multi-modal representation vector z∈custom-character.sup.d, which can be represented as follows.
z=ƒ(x, w) (1)
In Equation (1), the function ƒ( ) represents the multi-modal network (such as the transformer network 340), and x and w represent the image-text pair. With learned representations, a linear projection layer with a softmax function can be used for a downstream task 225 with a cross-entropy loss or bounding box prediction with object detection loss, such as a smooth L1 loss.
[0053] In the process 300, the transformer network 340 can expect visual representations from the detected bounding boxes in the images 305 and word representations from the text 308. In this case, the multi-modal representation vector (z) 350 can be represented as follows.
z=ƒ(q,v|x,w;θ.sub.d,θ.sub.w) (2)
Here, θ.sub.d and θ.sub.w, represent the parameters of the object detector 315 and the word embedding model 320, respectively. Also, q and v represent the word embeddings 330 and the visual feature vectors 325, respectively.
[0054] Once the electronic device 101 determines the multi-modal representation vector 350, the electronic device 101 inputs the multi-modal representation vector 350 to a classifier 360, which outputs a predicted class label 370 for the paired image-text pair 310. The classifier 360 represents any suitable classification process, technique, or algorithm for classifying and predicting class labels for image-text pairs, such as a binary classifier. In the first iteration of the process 300 as shown in FIG. 3A, the contrastive learning objective for the paired image-text pair (x, w) 310 is represented as follows.
L.sub.c(x,w)=−custom-character.sub.(x,w)˜D log p(y=0|ƒ.sub.c(z;θ.sub.c|x,w;θ.sub.ƒ)) (3)
Here, L.sub.c(x, w) represents the contrastive learning objective, custom-character represents an expectation function, y=0 represents the category label for a positive image-text pair, θ.sub.c represents the parameters of the classifier 360, and θ.sub.ƒ represents the parameters of the transformer network 340.--, in [0052]-[0054]);
MA and Zhao are combinable as they are in the same field of endeavor: image-language alignment training. Therefore it would have been obvious to one of ordinary skill in the art at the time the invention was made to modify MA’s method using Zhao’s teachings by including training the image-language alignment model through a contrastive loss function to MA’s target function in training image-text alignment in order to improve performance in accuracy and robustness (see Zhao: e.g., in abstract, [0052]-[0054], and [0107]-[0109]).
Re Claim 2, MA and Zhao further disclose wherein the object representation vector is a vector that represents an attribute on an object (see Zhao: e.g., -- The object detector 315 detects one or more objects or regions within the image 305 (such as a dog, a sofa, a table, and the like) and represents each visual feature of the object or region as a feature vector (v) 325. The object detector 315 represents any suitable process, technique, or algorithm for generating feature vectors from an image. In this case, the visual feature vectors 325 can be represented as v.sub.i, i={0 . . . K−1}, where v.sub.i∈custom-character.sup.d, K represents the number of detected features, and d represents the dimension of the feature vectors 325.--, in [0049]).
Re Claim 3, MA and Zhao further disclose wherein a plurality of attributes are included for one object (see Zhao: e.g., -- The object detector 315 detects one or more objects or regions within the image 305 (such as a dog, a sofa, a table, and the like) and represents each visual feature of the object or region as a feature vector (v) 325. The object detector 315 represents any suitable process, technique, or algorithm for generating feature vectors from an image. In this case, the visual feature vectors 325 can be represented as v.sub.i, i={0 . . . K−1}, where v.sub.i∈custom-character.sup.d, K represents the number of detected features, and d represents the dimension of the feature vectors 325.--, in [0049]).
Re Claim 4, MA and Zhao further disclose wherein, at the second generation step, the plurality of attributes are generated by one object representation vector by using mean pooling or attentive pooling (see MA: e.g., --[0065] In one embodiment, to better extract a global representation in a text sample, the following operations are included: performing word segmentation on the text sample; for each word segment, determining a vector of the word segment, different word segments having a same vector length; and inputting vectors of word segments of a same text sample into a convolutional neural network used to extract the global feature of the text sample, the neural network used to extract the global feature of the text sample including a plurality of convolutional layers and a pooling layer connected after the plurality of convolutional layers, and a field of view of a specified size of a previous convolutional layer being used as input of a current convolutional layer, the field of view of a specified size including features of at least two word segment vectors extracted by the previous convolutional layer.
[0066] For example, during a specific implementation, a convolutional operation of a convolutional layer may be represented by using the following formula (5):
V.sub.(l,f).sup.i=σ(W.sub.l,f{tilde over (v)}.sub.l−1.sup.i+b.sub.(l,f)) (5) [0067] where {tilde over (v)}.sub.l−1.sup.i=v.sub.l−1.sup.i∥v.sub.l−1.sup.i+1 . . . ∥v.sub.l−1.sup.i+r−1.
[0068] In the formula (5), r represents a field of view of a specified size; l−1 represents the (1-1).sup.th convolutional layer (that is, a previous convolutional layer); l represents the l.sup.th convolutional layer (that is, a current convolutional layer); W.sub.l,f represents a product matrix; f represents the f.sup.th feature; i represents the i.sup.th location information; V.sub.(l,f).sup.i and represents an f.sup.th feature that corresponds to an ith location and that is extracted from the l.sup.th convolutional layer.
[0069] An operation of the pooling layer may be represented by the following formula (6):
v.sub.(l+1,f).sup.i=max(v.sub.(l,f).sup.M×i,v.sub.(l,f).sup.M×i+1, . . . v.sub.(l,f).sup.M×i+M−1) (6)
where v.sub.(l+1,f).sup.i represents output of the pooling layer; l+1 represents the pooling layer; v.sub.(l,f).sup.M×i represents the (M×i).sup.th feature extracted from the l.sup.th layer; and M represents a constant used to control a size of a pooling operation.--, in [0065]-[0069]).
Re Claim 5, MA and Zhao further disclose a step of classifying object attributes from the object representation vector generated at the first generation step (see Zhao: e.g., --[0054] Once the electronic device 101 determines the multi-modal representation vector 350, the electronic device 101 inputs the multi-modal representation vector 350 to a classifier 360, which outputs a predicted class label 370 for the paired image-text pair 310. The classifier 360 represents any suitable classification process, technique, or algorithm for classifying and predicting class labels for image-text pairs, such as a binary classifier. In the first iteration of the process 300 as shown in FIG. 3A, the contrastive learning objective for the paired image-text pair (x, w) 310 is represented as follows.
L.sub.c(x,w)=−custom-character.sub.(x,w)˜D log p(y=0|ƒ.sub.c(z;θ.sub.c|x,w;θ.sub.ƒ)) (3)
Here, L.sub.c(x, w) represents the contrastive learning objective, custom-character represents an expectation function, y=0 represents the category label for a positive image-text pair, θ.sub.c represents the parameters of the classifier 360, and θ.sub.ƒ represents the parameters of the transformer network 340.--, in [0054]),
wherein the step of training comprises training the image-language alignment model through a cross entropy loss function by using the classified attributes (see Zhao: e.g., --[0052] In general, multi-modal architectures for downstream tasks typically involve learning the multi-modal representation vector z∈custom-character.sup.d, which can be represented as follows.
z=ƒ(x, w) (1)
In Equation (1), the function ƒ( ) represents the multi-modal network (such as the transformer network 340), and x and w represent the image-text pair. With learned representations, a linear projection layer with a softmax function can be used for a downstream task 225 with a cross-entropy loss or bounding box prediction with object detection loss, such as a smooth L1 loss.
[0053] In the process 300, the transformer network 340 can expect visual representations from the detected bounding boxes in the images 305 and word representations from the text 308. In this case, the multi-modal representation vector (z) 350 can be represented as follows.
z=ƒ(q,v|x,w;θ.sub.d,θ.sub.w) (2)
Here, θ.sub.d and θ.sub.w, represent the parameters of the object detector 315 and the word embedding model 320, respectively. Also, q and v represent the word embeddings 330 and the visual feature vectors 325, respectively.
[0054] Once the electronic device 101 determines the multi-modal representation vector 350, the electronic device 101 inputs the multi-modal representation vector 350 to a classifier 360, which outputs a predicted class label 370 for the paired image-text pair 310. The classifier 360 represents any suitable classification process, technique, or algorithm for classifying and predicting class labels for image-text pairs, such as a binary classifier. In the first iteration of the process 300 as shown in FIG. 3A, the contrastive learning objective for the paired image-text pair (x, w) 310 is represented as follows.
L.sub.c(x,w)=−custom-character.sub.(x,w)˜D log p(y=0|ƒ.sub.c(z;θ.sub.c|x,w;θ.sub.ƒ)) (3)
Here, L.sub.c(x, w) represents the contrastive learning objective, custom-character represents an expectation function, y=0 represents the category label for a positive image-text pair, θ.sub.c represents the parameters of the classifier 360, and θ.sub.ƒ represents the parameters of the transformer network 340.--, in [0052]-[0054]).
Re Claim 6, MA and Zhao further disclose a third generation step of generating, by the image-language alignment model, a global representation vector of the image in the inputted image (see MA: e.g., --extracting a global feature and a local feature of an image sample--, in [0008]; and, --The local feature of the image sample is also a vector, and information of each dimension of the vector represents that a local area of the image includes information--, in [0038]-[0039]); and
a fourth generation step of generating, by the image-language alignment model, a global representation vector of the text in the inputted text (see MA: e.g., --he global feature of the text sample is used to represent overall semantic information of the text sample. The local feature of the text sample is used to represent integration of local semantic information of the text. The local feature of the text sample focuses on local information, such as a word or a phrase, of the text.--, in [0040]-[0041]),
wherein the step of training comprises training the image-language alignment model through a contrastive loss function by using the global representation vector generated at the third generation step and the global representation vector generated at the fourth generation step (see Zhao: e.g., --raining a machine learning model using the training data based on an optimization of a combination of losses. The losses include, for each paired image-text pair, (i) a first multi-modal representation loss based on the paired image-text pair and (ii) a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs includes either the image or the text of the paired image-text pair.--, in abstract; and, --[0002] Contrastive learning may be used in self-supervised learning applications with visual data. For large-scale datasets having image-text pairs, contrastive learning may be used for pre-training to improve transfer learning on downstream tasks, which may improve performance in accuracy and robustness compared to other image-text matching methods when considering fine-tuning of downstream tasks, such as visual question answering (VQA), graph question answering (GQA), referring expressions, caption generation, and image retrieval. As mentioned above, these contrastive-learning approaches have focused on the pre-training stage with multi-modal datasets having image and text pairs to boost the performance of downstream tasks.--, in [0002], and, --[0052] In general, multi-modal architectures for downstream tasks typically involve learning the multi-modal representation vector z∈custom-character.sup.d, which can be represented as follows.
z=ƒ(x, w) (1)
In Equation (1), the function ƒ( ) represents the multi-modal network (such as the transformer network 340), and x and w represent the image-text pair. With learned representations, a linear projection layer with a softmax function can be used for a downstream task 225 with a cross-entropy loss or bounding box prediction with object detection loss, such as a smooth L1 loss.
[0053] In the process 300, the transformer network 340 can expect visual representations from the detected bounding boxes in the images 305 and word representations from the text 308. In this case, the multi-modal representation vector (z) 350 can be represented as follows.
z=ƒ(q,v|x,w;θ.sub.d,θ.sub.w) (2)
Here, θ.sub.d and θ.sub.w, represent the parameters of the object detector 315 and the word embedding model 320, respectively. Also, q and v represent the word embeddings 330 and the visual feature vectors 325, respectively.
[0054] Once the electronic device 101 determines the multi-modal representation vector 350, the electronic device 101 inputs the multi-modal representation vector 350 to a classifier 360, which outputs a predicted class label 370 for the paired image-text pair 310. The classifier 360 represents any suitable classification process, technique, or algorithm for classifying and predicting class labels for image-text pairs, such as a binary classifier. In the first iteration of the process 300 as shown in FIG. 3A, the contrastive learning objective for the paired image-text pair (x, w) 310 is represented as follows.
L.sub.c(x,w)=−custom-character.sub.(x,w)˜D log p(y=0|ƒ.sub.c(z;θ.sub.c|x,w;θ.sub.ƒ)) (3)
Here, L.sub.c(x, w) represents the contrastive learning objective, custom-character represents an expectation function, y=0 represents the category label for a positive image-text pair, θ.sub.c represents the parameters of the classifier 360, and θ.sub.ƒ represents the parameters of the transformer network 340.--, in [0052]-[0054]; and, --[0058] According to an effect of the preset target function, a depth matching model as shown in FIG. 4 may be trained. The depth matching model is used to determine a matching degree between an image and a text. In FIG. 4, a feature vector (image embedding) of an image (the feature vector is a feature vector of a global feature or a local feature) is inputted on a left side, and a feature vector (sentence embedding) of a text (the feature vector is a feature vector of a global feature or a feature vector of a local feature) is inputted on a right side. After the feature vectors of the image and the text are mapped to a same specified semantic space, a distance between the feature vectors is calculated. For example, in FIG. 4, S.sub.l.sup.1, S.sub.l.sup.2, S.sub.l.sup.3, S.sub.l.sup.4, and S.sub.n represent feature vectors of a text, and I.sub.1 and I.sub.n represent feature vectors of an image. In the specified semantic space, a distance between the image and the text is calculated. If the feature vectors used are feature vectors for representing global features, a calculation result is a similarity between the global features. If the feature vectors used are feature vectors for representing local features, a calculation result is a similarity between the local features.--, in [0055]-[0058]).
Re Claim 7, MA and Zhao further disclose wherein the object is an object that is detected from the image by an artificial intelligence (AI) model which is trained to detect objects (see Zhao: e.g., -- The object detector 315 detects one or more objects or regions within the image 305 (such as a dog, a sofa, a table, and the like) and represents each visual feature of the object or region as a feature vector (v) 325. The object detector 315 represents any suitable process, technique, or algorithm for generating feature vectors from an image. In this case, the visual feature vectors 325 can be represented as v.sub.i, i={0 . . . K−1}, where v.sub.i∈custom-character.sup.d, K represents the number of detected features, and d represents the dimension of the feature vectors 325.--, in [0049]).
Re Claim 8, MA and Zhao further disclose further comprising a step of searching an image based on a text by using the trained image-language alignment model (see MA: e.g., --[0009] The embodiments of this application provide a computing device, including a memory and a processor, the memory being configured to store program instructions, and the processor being configured to invoke the program instructions stored in the memory, and perform the image-text bi-directional search method according to the embodiments of this application according to the obtained program instructions. The embodiments of this application provide a computer storage medium, the computer storage medium storing computer-executable instructions, the computer-executable instructions being used to enable the computer to perform the image-text bi-directional search method according to the embodiments of this application according to the computer-executable instructions.--, in abstract, [0009]; and, --the client of the terminal device 11 transmits the image and/or text to the server 12, and the server 12 extracts global features and local features of or a global feature and a local feature of the image and/or the text. Then, the server 12 searches for a text matching the image according to the global feature and the local feature of the image based on the matching model, or the server 12 searches for an image matching the text according to the global feature and the local feature of the text based on the matching model. Subsequently, the server 12 feeds back a search result to the terminal device 11.
[0030] The terminal device 11 and the server 12 may be communicably connected through a communications network. The network may be a local area network, a wide area network, or the like. The terminal device 11 may be a mobile phone, a tablet computer, a notebook computer, a personal computer, or the like. The server 12 may be any server device capable of supporting training of a corresponding image-text matching model and a bi-directional search.
[0031] The solution provided by this embodiment of this application is applicable to any scenario that needs to search for an image according to a text and/or needs to search for a text according to an image. For example, the solution may be applicable to searching for an image and/or a text in cloud album management (for example, searching for a matching image according to a given text), WeChat Moments, QQ Space, QQ Mass Innovation Space, and chat environments of WeChat and QQ.--, in [0029]-[0031]).
Re Claim 9, MA and Zhao further disclose a step of searching a text based on an image by using the trained image-language alignment model (see MA: e.g., --[0009] The embodiments of this application provide a computing device, including a memory and a processor, the memory being configured to store program instructions, and the processor being configured to invoke the program instructions stored in the memory, and perform the image-text bi-directional search method according to the embodiments of this application according to the obtained program instructions. The embodiments of this application provide a computer storage medium, the computer storage medium storing computer-executable instructions, the computer-executable instructions being used to enable the computer to perform the image-text bi-directional search method according to the embodiments of this application according to the computer-executable instructions.--, in abstract, [0009]; and, --the client of the terminal device 11 transmits the image and/or text to the server 12, and the server 12 extracts global features and local features of or a global feature and a local feature of the image and/or the text. Then, the server 12 searches for a text matching the image according to the global feature and the local feature of the image based on the matching model, or the server 12 searches for an image matching the text according to the global feature and the local feature of the text based on the matching model. Subsequently, the server 12 feeds back a search result to the terminal device 11.
[0030] The terminal device 11 and the server 12 may be communicably connected through a communications network. The network may be a local area network, a wide area network, or the like. The terminal device 11 may be a mobile phone, a tablet computer, a notebook computer, a personal computer, or the like. The server 12 may be any server device capable of supporting training of a corresponding image-text matching model and a bi-directional search.
[0031] The solution provided by this embodiment of this application is applicable to any scenario that needs to search for an image according to a text and/or needs to search for a text according to an image. For example, the solution may be applicable to searching for an image and/or a text in cloud album management (for example, searching for a matching image according to a given text), WeChat Moments, QQ Space, QQ Mass Innovation Space, and chat environments of WeChat and QQ.--, in [0029]-[0031]).
Re Claim 11, claim 11 is a corresponding method claim to claim 1, respectively. Claim 11 thus is rejected for the similar reasons for claim 1. See above discussions with regard to claim 1 respectively, Furthermore, MA and Zhao further disclose image-language alignment model computation method comprising:
a step of generating an image-language alignment model (see MA: e.g., --a training method of an image-text matching model, a bi-directional search method, and a relevant apparatus. The training method includes extracting a global feature and a local feature of an image sample; extracting a global feature and a local feature of a text sample; training a matching model according to the extracted global feature and local feature of the image sample and the extracted global feature and local feature of the text sample, to determine model parameters of the matching model; and determining, by the matching model, according to a global feature and a local feature of an inputted image and a global feature and a local feature of an inputted text, a matching degree between the image and the text.--, in abstract); and
a step of searching an image based on a text by using the generated image- language alignment model (see MA: e.g., --[0009] The embodiments of this application provide a computing device, including a memory and a processor, the memory being configured to store program instructions, and the processor being configured to invoke the program instructions stored in the memory, and perform the image-text bi-directional search method according to the embodiments of this application according to the obtained program instructions. The embodiments of this application provide a computer storage medium, the computer storage medium storing computer-executable instructions, the computer-executable instructions being used to enable the computer to perform the image-text bi-directional search method according to the embodiments of this application according to the computer-executable instructions.--, in abstract, [0009]; and, --the client of the terminal device 11 transmits the image and/or text to the server 12, and the server 12 extracts global features and local features of or a global feature and a local feature of the image and/or the text. Then, the server 12 searches for a text matching the image according to the global feature and the local feature of the image based on the matching model, or the server 12 searches for an image matching the text according to the global feature and the local feature of the text based on the matching model. Subsequently, the server 12 feeds back a search result to the terminal device 11.
[0030] The terminal device 11 and the server 12 may be communicably connected through a communications network. The network may be a local area network, a wide area network, or the like. The terminal device 11 may be a mobile phone, a tablet computer, a notebook computer, a personal computer, or the like. The server 12 may be any server device capable of supporting training of a corresponding image-text matching model and a bi-directional search.
[0031] The solution provided by this embodiment of this application is applicable to any scenario that needs to search for an image according to a text and/or needs to search for a text according to an image. For example, the solution may be applicable to searching for an image and/or a text in cloud album management (for example, searching for a matching image according to a given text), WeChat Moments, QQ Space, QQ Mass Innovation Space, and chat environments of WeChat and QQ.--, in [0029]-[0031]).
Re Claim 12, claim 12 is a corresponding system claim to claim 1, respectively. Claim 12 thus is rejected for the similar reasons for claim 1. See above discussions with regard to claim 1 respectively. Furthermore, MA and Zhao further disclose An image-language alignment model computation system comprising:
a processor configured to generate an image-language alignment model, and to search an image based on a text by using the generated image-language alignment model; and a storage unit configured to provide a storage space necessary for the processor (see MA: e.g., --[0009] The embodiments of this application provide a computing device, including a memory and a processor, the memory being configured to store program instructions, and the processor being configured to invoke the program instructions stored in the memory, and perform the image-text bi-directional search method according to the embodiments of this application according to the obtained program instructions. The embodiments of this application provide a computer storage medium, the computer storage medium storing computer-executable instructions, the computer-executable instructions being used to enable the computer to perform the image-text bi-directional search method according to the embodiments of this application according to the computer-executable instructions.--, in abstract, [0009]; and, --the client of the terminal device 11 transmits the image and/or text to the server 12, and the server 12 extracts global features and local features of or a global feature and a local feature of the image and/or the text. Then, the server 12 searches for a text matching the image according to the global feature and the local feature of the image based on the matching model, or the server 12 searches for an image matching the text according to the global feature and the local feature of the text based on the matching model. Subsequently, the server 12 feeds back a search result to the terminal device 11.
[0030] The terminal device 11 and the server 12 may be communicably connected through a communications network. The network may be a local area network, a wide area network, or the like. The terminal device 11 may be a mobile phone, a tablet computer, a notebook computer, a personal computer, or the like. The server 12 may be any server device capable of supporting training of a corresponding image-text matching model and a bi-directional search.
[0031] The solution provided by this embodiment of this application is applicable to any scenario that needs to search for an image according to a text and/or needs to search for a text according to an image. For example, the solution may be applicable to searching for an image and/or a text in cloud album management (for example, searching for a matching image according to a given text), WeChat Moments, QQ Space, QQ Mass Innovation Space, and chat environments of WeChat and QQ.--, in [0029]-[0031]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIWEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on Monday-Friday 8:30am-4:30pm east.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights