Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The amendments filed on 05/05/2026 has been entered. The status of the claims is as follows:
Claims 1-19, 21 remain pending in the application.
Claims 1-3, 9-11, 17-18 are amended.
Claim 20 is cancelled.
Claim 21 is new.
Response to Arguments
In reference to the Claim Rejections under 35 U.S.C 103:
Applicant’s arguments, see Remarks pg. 10-13, filed on 07/22/2025, with respect to the rejection(s) of claim(s) under 35 U.S.C 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Xu et al. (“Hierarchical Semantic Aggregation for Contrastive Representation Learning”) and further in view of Khosla et al. (“Supervised Contrastive Learning”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-7, 9-15 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Ge et al. (“Deep Metric Learning with Hierarchical Triplet Loss”) (hereafter referred to as “Ge”) in view of Yuan et al (“Hard-Aware Deeply Cascaded Embedding”), Goyal & Ghosh (“Hierarchical Class-Based Curriculum Loss”) (hereafter referred to as “Goyal”), Maksym Bekuzarov (“Losses explained: Contrastive Loss”) (hereafter referred to as “Bekuzarov”) and further in view of Xu et al. (“Hierarchical Semantic Aggregation for Contrastive Representation Learning”)
As per claim 1, Ge explicitly discloses:
A computer-implemented method for training a machine learning model for computer vision, the method comprising: receiving a set of image samples including a first image sample that is associated with a set of hierarchical labels at a plurality of levels of image categories, (Ge, Page 6, Section 4, Figure 2: “(a) A toy example of the hierarchical tree H. Different colors represent different image classes in CUB-200-2011 [31]. The leaves are the image classes in the training set. Then they are merged recursively until to the root node. (b) The training data distribution of 100 classes visualized by using t-SNE [16] to reduce the dimension of triplet embedding from 512 to 2.”
PNG
media_image1.png
300
802
media_image1.png
Greyscale
, Page 4, Figure 1: “(a) Caltech-UCSD Bird Species Dataset [31]. Images in each row are from the same class. There are four classes in different colors - red, green, blue and yellow.”, and Page 4, Section 3.1, ¶[1]: “”) [Examiner’s note: the training dataset i.e., the Caltech-UCSD Bird Species Dataset, a set of hierarchical labels at a plurality of levels i.e., level L0, L1, L2 and L3 of the hierarchical tree H]
selecting, for the first image sample as an anchor image, a plurality of positive image samples corresponding to the plurality of levels in the set of hierarchical labels, (Ge, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch… Finally, t images for each class are randomly collected, resulting in n (n = l’ mt) images in a mini-batch M.”, Page 7, Section 4.2, ¶[3]: “
A
l
'
m
2
indicates randomly selecting two classes - a positive class and a negative class, from all l’m classes in the mini-batch.
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
p
z
), from the positive class, and
C
t
1
means randomly selecting a negative sample (
x
n
z
) from the negative class.”, Pg. 4, Section 3.1: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa; xp; xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn. The labels of the triplet Tz satisfy ya = yp # yn. Triplet loss aims to pull samples belonging to the same class into nearby points on a manifold surface, and push samples with different labels apart from each other.”) [Examiner’s note: plurality of levels in the set of hierarchical labels i.e., the plurality levels of hierarchical tree H, labels yp # yn means that positive images have similar labels negative images have different labels]
generating a [[first]] training dataset including the first image sample, [[a]] the
plurality of positive image samples. [[pairs]], and the at least one negative image sample (Ge, Page 2, ¶[2]: “These loss functions are calculated on correlated samples, with a common goal of encouraging samples from the same class to be closer, and pushing samples of different classes apart from each other, in a projected feature space. The correlated samples are grouped into contrastive pairs, triplets or quadruplets, which form the training samples for these loss functions on deep metric learning.”, Page 4, Section 3.1, ¶[1]: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa, xp, xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn.”) [Examiner’s note: the contrastive outputs i.e., the contrastive triplet T = (xa, xp, xn)]
Ge fails to disclose:
computing a contrastive loss based on aggregating a pair loss representing a
similarity between the first image sample and each positive image sample of the plurality of positive images and a similarity between the first image sample and each negative image sample of the at least one negative image sample, at each level of the plurality of levels;
training [[a]] the machine learning model using the [[first]] contrastive loss such that
a first maximum contrastive loss computed in a training epoch using the first image sample and first positive image samples corresponding to a second level of the plurality of levels of hierarchical labels is no greater than a second maximum contrastive loss computed in the same training epoch using the first image sample and second positive image samples at a first level of the plurality of levels of hierarchical labels when the second level is lower than the first level;
distances between the first image sample and the plurality of positive image samples are closer in a label space than a distance between the first image sample[[ s ]] [[ of]] and the at least one negative image sample [[pair]]
in response to an input image at inference of the trained machine learning model, generating, by the trained machine learning model, an image category classification output classifying the input image to an image category of the image categories or an image retrieval output identifying an image from an image gallery that has a same image identifier as the input image
However, Goyal explicitly discloses:
a first maximum contrastive loss computed in a training epoch using the first image sample and first positive image samples corresponding to a second level of the plurality of levels of hierarchical labels is no greater than a second maximum contrastive loss computed in the same training epoch using the first image sample and second positive image samples at a first level of the plurality of levels of hierarchical labels when the second level is lower than the first level; (Goyal, Pg. 4, Section 3.1: “Consider the learning framework with training set T… with N training examples and input image features… We represent the labels as yi ϵ {-1, 1}C where C is the number of classes and yi, j= 1 means that the ith example belongs to jth class… Let the set of classes C be arranged in a hierarchy H defined by a hierarchy mapping function h : C → 2C which maps a category c ∈ C to its children categories. We use the function m: C → M to denote the mapping from a category c to its level in the hierarchy. We now define the following hierarchical constraint on a generic loss function l, the satisfaction of which would yield the loss function… The constraint implies that the loss increases monotonically with the level of the hierarchy i.e. loss of higher (i.e. closer to the root) levels in the hierarchy is lesser than that of the lower levels (i.e. closer to the leaves). The intuition is that identifying categories in higher level is easier than categories in lower level as they are coarser.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Goyal. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Goyal teaches Hierarchical multi-label classification (HMC) methods, which utilize the hierarchy of class labels to train a machine learning model in real world scenarios. One of ordinary skill would have motivation to combine Ge and Goyal because incorporate hierarchical relationships into the training objective results in more semantically meaningful and robust predictions.
However, Yuan explicitly discloses:
distances between the first image sample and the plurality of positive image samples are closer in a label space than a distance between the first image sample[[ s ]] [[ of]] and the at least one negative image sample [[pair]] (Yuan, Pg. 1, Section 1: “Although deep metric embedding is modified into different forms for various tasks, it shares the same objective to learn an embedding space that pulls similar images closer and pushes dissimilar images far away. Typically, the target embedding space is learned with a convolutional neural network equipped with contrastive/triplet loss.”, Pg. 4, Col. 2, Section 3.2: “Figure 2 to schematically the process of hard example mining. Cascade Model-1 will forward all pairs in P0 and N0, and try to push all positive points towards the anchor point while pushing all negative points away from the anchor point, and form P1, N1 (points in the 2nd and 3rd tier) by selecting hard samples according to its loss. Similarly, P2 and N2 (points in the 3rd tier) are formed by Cascade Model-2.”)”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Yuan. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Yuan teaches training a cascade of embedding models with increasing depth, where each level handles easy pairs of image and passes only the harder positive pairs to the next deeper model. One of ordinary skill would have motivation to combine Ge and Yuan to ensure that at a lower level, same-label pairs with small distances (low contrastive loss) are resolved, and only more difficult pairs (those with larger loss) are forwarded – effectively enforcing that the hardest positive pair at a shallow level is no harder than those handled at deeper levels.
However, Bekuzarov explicitly discloses:
in response to an input image at inference of the trained machine learning model, generating, by the trained machine learning model, an image category classification output classifying the input image to an image category of the image categories or an image retrieval output identifying an image from an image gallery that has a same image identifier as the input image (Bekuzarov, Pg. 5, ¶[4]: “Indeed, this is how Face Verification can be implemented — a CNN (convolutional neural network) is trained to map input train images of different people to vectors of real numbers (also called “feature-vectors” or “embeddings”) — for example, 128-d vectors, in such a way, that these embeddings of photos of the same person are very close to each other”, Pg. 16: “The input to the entire system is a pair of images (X1, X2) and a label Y . The images are passed through the functions, yielding two outputs G(X1) and G(X2).”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Bekuzarov. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Bekuzarov teaches using contrastive loss to train a machine learning model in face verification and face recognition tasks. One of ordinary skill would have motivation to combine Ge and Bekuzarov because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Xu explicitly discloses:
computing a contrastive loss based on aggregating a pair loss representing a
similarity between the first image sample and each positive image sample of the plurality
of positive images and a similarity between the first image sample and each negative image
sample of the at least one negative image sample, at each level of the plurality of levels; (Xu, Pg. 4, Col. 1, Section: Loss Function: “We simply adjust the contrastive loss in Eq. 1 to allow for multiple positives per anchor. Given an anchor sample xa and its nearest neighborhood set , we randomly select a positive sample xp ∈ Ω, and the loss function can be reformulated as:
PNG
media_image2.png
175
579
media_image2.png
Greyscale
. where each anchor sample qa encoded with fq, is pulled with two samples ka and kq encoded with fk, and pushed away with all other samples in the key encoder fk.” ,Pg. 4, Col. 2, Section 3.3: “we extend the proposed contrastive loss in Eq. 6 to the intermediate hidden layers, which targets at explicitly modeling the similarities among image/patches for better discrimination.”, Pg. 5, Col. 1: “So we get the total loss at stage l:
PNG
media_image3.png
53
220
media_image3.png
Greyscale
. When there are L losses corresponding to L intermediate
stages, the final losses of the whole network can be computed as:
PNG
media_image4.png
56
198
media_image4.png
Greyscale
”) [Examiner’s note: Equation (2) computes the loss for an anchor image, against a positive sample, this is the pairwise loss. Xu extends the contrastive loss to intermediate hidden layers and computes a level-specific loss Ll for each stage, then Ltotal aggregates the losses across all levels]
training [[a]] the machine learning model using the [[first]] contrastive loss such that (Xu, Pg. 3, Col. 1, Section 3.1: “the contrastive loss can be defined as:
PNG
media_image5.png
70
375
media_image5.png
Greyscale
”)
The combination of Ge and Xu are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings of Ge and Xu before them, to modify the teachings of Ge to include the teachings of Xu to improve conventional contrastive learning by incorporating multiple positive samples representing semantic similarity, rather than relying only on augmented views of the same image.
As per claim 2, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of claim 1 (as shown in the rejection above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
wherein the set of hierarchical labels takes a form of a tree structure according to the plurality of levels, and wherein the tree structure has [[a]] the root corresponding to a broadest label of the set of hierarchical labels. (Ge, Page 6, Section 4, Figure 2: “A toy example of the hierarchical tree H. Different colors represent different image classes in CUB-200-2011 [31]. The leaves are the image classes in the training set. Then they are merged recursively until to the root node
PNG
media_image1.png
300
802
media_image1.png
Greyscale
”) [Examiner’s note: the set of hierarchical labels takes a form of a tree structure according to the plurality of levels i.e., L0, L1, L2 and L3, the root node is L3]
As per claim 3, the combination of Ge, Yuan, Goyal,, Xu and Bekuzarov discloses all the limitations of claim 2 (as shown in the rejection above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
randomly selecting the first image sample an anchor image sample from the set of image samples; (Ge, Page 9, Algorithm 1, Line 3: “Sample anchors randomly and their neighborhoods according to H;”)
determining an anchor set of hierarchical labels in the tree structure associated with anchor image sample; (Ge, Page 8, Figure 3: “(a)Sampling strategy of each mini-batch. The images in red stand for anchors and the images in blue stand for the nearest neighbors. (b) Train CNNs with the hierarchical triplet loss. (c) Online update of the hierarchical tree.”
PNG
media_image6.png
317
855
media_image6.png
Greyscale
) [Examiner’s note: In part (a) of figure 3, it shows the sampling strategy where images marked in red are selected as anchor samples, and those marked in blue are selected as the nearest neighbors. The hierarchical tree structure depicted in part (c) is associated with updating the tree based on the loss from the loss function. This process involves determining the anchor set (the red-marked images) and associating these with hierarchical labels in the tree structure.]
randomly selecting, for the anchor image sample at a third level from the plurality of levels, a third positive image sample, from the plurality of positive image samples, that shares common label ancestry from the root up to the third level with the anchor image sample; and (Ge, Page 8, ¶[1]: “
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
n
z
), from the positive class,”, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch, which is important for training deep networks with batch normalization [9]. Then m - 1 nearest classes at the 0-th level are selected for each of the l’ nodes, based on the distance between classes computed in the feature space.”) [Examiner’s note: a positive sample is selected from the same positive class with an anchor sample, wherein the positive class is from level 0 (i.e., shares common label L0) of the hierarchical tree]
forming a first positive pair of the plurality of positive sample pairs from the anchor image sample and the third positive image sample (Ge, Page 4, Section 3.1: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa, xp, xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn…
PNG
media_image7.png
38
190
media_image7.png
Greyscale
denotes the hinge loss function, and is the violate margin that requires the distance of
PNG
media_image8.png
45
138
media_image8.png
Greyscale
negative pairs to be larger than the distance of
PNG
media_image9.png
43
137
media_image9.png
Greyscale
positive pairs.”) [Examiner’s note: the positive pair i.e.,
PNG
media_image9.png
43
137
media_image9.png
Greyscale
]
As per claim 4, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of claim 3 (as shown in the rejection above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
randomly selecting, for the anchor image sample at another level from the plurality of levels, another positive image sample, from the plurality of positive image samples, until positive image samples according to the plurality of levels have been sampled. (Ge, Page 7, Section 4.2, ¶[3]: “
A
l
'
m
2
indicates randomly selecting two classes - a positive class and a negative class, from all l’m classes in the mini-batch.
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
n
z
), from the positive class,”, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch, which is important for training deep networks with batch normalization [9]. Then m - 1 nearest classes at the 0-th level are selected for each of the l’ nodes, based on the distance between classes computed in the feature space.”, and Page 6, Figure 2: “The leaves are the image classes in the training set. Then they are merged recursively until to the root node.
PNG
media_image6.png
317
855
media_image6.png
Greyscale
”) [Examiner’s note: classes are selected randomly from all l’m classes (i.e., all classes of all nodes l’)]
As per claim 5, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of claim 4 (as shown in the rejection above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
randomly selecting, from the set of image samples, another anchor image until each image sample of the set of image samples [[have]] has been sampled in a training epoch. (Ge, Page 9, Algorithm 1, Lines 1-3: “while not converge do … Sample anchors randomly and their neighborhoods according to H ;”) [Examiner’s note: the while do loop discloses the training epoch, and the anchors are sampled randomly according to the hierarchical tree in the loop]
As per claim 9, Ge explicitly discloses:
A computer-implemented method for training a machine learning model for computer vision, the method comprising: receiving a set of image samples including a first image sample that is associated with a set of hierarchical labels at a plurality of levels of image categories, (Ge, Page 6, Section 4, Figure 2: “(a) A toy example of the hierarchical tree H. Different colors represent different image classes in CUB-200-2011 [31]. The leaves are the image classes in the training set. Then they are merged recursively until to the root node. (b) The training data distribution of 100 classes visualized by using t-SNE [16] to reduce the dimension of triplet embedding from 512 to 2.”
PNG
media_image1.png
300
802
media_image1.png
Greyscale
, Page 4, Figure 1: “(a) Caltech-UCSD Bird Species Dataset [31]. Images in each row are from the same class. There are four classes in different colors - red, green, blue and yellow.”, and Page 4, Section 3.1, ¶[1]: “”) [Examiner’s note: the training dataset i.e., the Caltech-UCSD Bird Species Dataset, a set of hierarchical labels at a plurality of levels i.e., level L0, L1, L2 and L3 of the hierarchical tree H]
selecting, for the first image sample as an anchor image, a plurality of positive image samples corresponding to the plurality of levels in the set of hierarchical labels, (Ge, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch… Finally, t images for each class are randomly collected, resulting in n (n = l’ mt) images in a mini-batch M.”, Page 7, Section 4.2, ¶[3]: “
A
l
'
m
2
indicates randomly selecting two classes - a positive class and a negative class, from all l’m classes in the mini-batch.
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
p
z
), from the positive class, and
C
t
1
means randomly selecting a negative sample (
x
n
z
) from the negative class.”, Pg. 4, Section 3.1: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa; xp; xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn. The labels of the triplet Tz satisfy ya = yp # yn. Triplet loss aims to pull samples belonging to the same class into nearby points on a manifold surface, and push samples with different labels apart from each other.”) [Examiner’s note: plurality of levels in the set of hierarchical labels i.e., the plurality levels of hierarchical tree H, labels yp # yn means that positive images have similar labels negative images have different labels]
generating a [[first]] training dataset including the first image sample, [[a]] the
plurality of positive image samples. [[pairs]], and the at least one negative image sample (Ge, Page 2, ¶[2]: “These loss functions are calculated on correlated samples, with a common goal of encouraging samples from the same class to be closer, and pushing samples of different classes apart from each other, in a projected feature space. The correlated samples are grouped into contrastive pairs, triplets or quadruplets, which form the training samples for these loss functions on deep metric learning.”, Page 4, Section 3.1, ¶[1]: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa, xp, xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn.”) [Examiner’s note: the contrastive outputs i.e., the contrastive triplet T = (xa, xp, xn)]
Ge fails to disclose:
computing a contrastive loss based on aggregating a pair loss representing a
similarity between the first image sample and each positive image sample of the plurality of positive images and a similarity between the first image sample and each negative image sample of the at least one negative image sample, at each level of the plurality of levels;
training [[a]] the machine learning model using the [[first]] contrastive loss such that
a first maximum contrastive loss computed in a training epoch using the first image sample and first positive image samples corresponding to a second level of the plurality of levels of hierarchical labels is no greater than a second maximum contrastive loss computed in the same training epoch using the first image sample and second positive image samples at a first level of the plurality of levels of hierarchical labels when the second level is lower than the first level;
distances between the first image sample and the plurality of positive image samples are closer in a label space than a distance between the first image sample[[ s ]] [[ of]] and the at least one negative image sample [[pair]]
in response to an input image at inference of the trained machine learning model, generating, by the trained machine learning model, an image category classification output classifying the input image to an image category of the image categories or an image retrieval output identifying an image from an image gallery that has a same image identifier as the input image
However, Goyal explicitly discloses:
a first maximum contrastive loss computed in a training epoch using the first image sample and first positive image samples corresponding to a second level of the plurality of levels of hierarchical labels is no greater than a second maximum contrastive loss computed in the same training epoch using the first image sample and second positive image samples at a first level of the plurality of levels of hierarchical labels when the second level is lower than the first level; (Goyal, Pg. 4, Section 3.1: “Consider the learning framework with training set T… with N training examples and input image features… We represent the labels as yi ϵ {-1, 1}C where C is the number of classes and yi, j= 1 means that the ith example belongs to jth class… Let the set of classes C be arranged in a hierarchy H defined by a hierarchy mapping function h : C → 2C which maps a category c ∈ C to its children categories. We use the function m: C → M to denote the mapping from a category c to its level in the hierarchy. We now define the following hierarchical constraint on a generic loss function l, the satisfaction of which would yield the loss function… The constraint implies that the loss increases monotonically with the level of the hierarchy i.e. loss of higher (i.e. closer to the root) levels in the hierarchy is lesser than that of the lower levels (i.e. closer to the leaves). The intuition is that identifying categories in higher level is easier than categories in lower level as they are coarser.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Goyal. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Goyal teaches Hierarchical multi-label classification (HMC) methods, which utilize the hierarchy of class labels to train a machine learning model in real world scenarios. One of ordinary skill would have motivation to combine Ge and Goyal because incorporate hierarchical relationships into the training objective results in more semantically meaningful and robust predictions.
However, Yuan explicitly discloses:
distances between the first image sample and the plurality of positive image samples are closer in a label space than a distance between the first image sample[[ s ]] [[ of]] and the at least one negative image sample [[pair]] (Yuan, Pg. 1, Section 1: “Although deep metric embedding is modified into different forms for various tasks, it shares the same objective to learn an embedding space that pulls similar images closer and pushes dissimilar images far away. Typically, the target embedding space is learned with a convolutional neural network equipped with contrastive/triplet loss.”, Pg. 4, Col. 2, Section 3.2: “Figure 2 to schematically the process of hard example mining. Cascade Model-1 will forward all pairs in P0 and N0, and try to push all positive points towards the anchor point while pushing all negative points away from the anchor point, and form P1, N1 (points in the 2nd and 3rd tier) by selecting hard samples according to its loss. Similarly, P2 and N2 (points in the 3rd tier) are formed by Cascade Model-2.”)”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Yuan. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Yuan teaches training a cascade of embedding models with increasing depth, where each level handles easy pairs of image and passes only the harder positive pairs to the next deeper model. One of ordinary skill would have motivation to combine Ge and Yuan to ensure that at a lower level, same-label pairs with small distances (low contrastive loss) are resolved, and only more difficult pairs (those with larger loss) are forwarded – effectively enforcing that the hardest positive pair at a shallow level is no harder than those handled at deeper levels.
However, Bekuzarov explicitly discloses:
in response to an input image at inference of the trained machine learning model, generating, by the trained machine learning model, an image category classification output classifying the input image to an image category of the image categories or an image retrieval output identifying an image from an image gallery that has a same image identifier as the input image (Bekuzarov, Pg. 5, ¶[4]: “Indeed, this is how Face Verification can be implemented — a CNN (convolutional neural network) is trained to map input train images of different people to vectors of real numbers (also called “feature-vectors” or “embeddings”) — for example, 128-d vectors, in such a way, that these embeddings of photos of the same person are very close to each other”, Pg. 16: “The input to the entire system is a pair of images (X1, X2) and a label Y . The images are passed through the functions, yielding two outputs G(X1) and G(X2).”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Bekuzarov. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Bekuzarov teaches using contrastive loss to train a machine learning model in face verification and face recognition tasks. One of ordinary skill would have motivation to combine Ge and Bekuzarov because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Xu explicitly discloses:
computing a contrastive loss based on aggregating a pair loss representing a similarity between the first image sample and each positive image sample of the plurality of positive images and a similarity between the first image sample and each negative image sample of the at least one negative image sample, at each level of the plurality of levels; (Xu, Pg. 4, Col. 1, Section: Loss Function: “We simply adjust the contrastive loss in Eq. 1 to allow for multiple positives per anchor. Given an anchor sample xa and its nearest neighborhood set , we randomly select a positive sample xp ∈ Ω, and the loss function can be reformulated as:
PNG
media_image2.png
175
579
media_image2.png
Greyscale
. where each anchor sample qa encoded with fq, is pulled with two samples ka and kq encoded with fk, and pushed away with all other samples in the key encoder fk.” ,Pg. 4, Col. 2, Section 3.3: “we extend the proposed contrastive loss in Eq. 6 to the intermediate hidden layers, which targets at explicitly modeling the similarities among image/patches for better discrimination.”, Pg. 5, Col. 1: “So we get the total loss at stage l:
PNG
media_image3.png
53
220
media_image3.png
Greyscale
. When there are L losses corresponding to L intermediate
stages, the final losses of the whole network can be computed as:
PNG
media_image4.png
56
198
media_image4.png
Greyscale
”) [Examiner’s note: Equation (2) computes the loss for an anchor image, against a positive sample, this is the pairwise loss. Xu extends the contrastive loss to intermediate hidden layers and computes a level-specific loss Ll for each stage, then Ltotal aggregates the losses across all levels]
training [[a]] the machine learning model using the [[first]] contrastive loss such that (Xu, Pg. 3, Col. 1, Section 3.1: “the contrastive loss can be defined as:
PNG
media_image5.png
70
375
media_image5.png
Greyscale
”)
The combination of Ge and Xu are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings of Ge and Xu before them, to modify the teachings of Ge to include the teachings of Xu to improve conventional contrastive learning by incorporating multiple positive samples representing semantic similarity, rather than relying only on augmented views of the same image.
As per claim 10, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of Claim 9 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
wherein the set of hierarchical labels takes a form of a tree structure according to the plurality of levels, and wherein the tree structure has [[a]] the root corresponding to a broadest label of the set of hierarchical labels. (Ge, Page 6, Section 4, Figure 2: “A toy example of the hierarchical tree H. Different colors represent different image classes in CUB-200-2011 [31]. The leaves are the image classes in the training set. Then they are merged recursively until to the root node
PNG
media_image1.png
300
802
media_image1.png
Greyscale
”) [Examiner’s note: the set of hierarchical labels takes a form of a tree structure according to the plurality of levels i.e., L0, L1, L2 and L3, the root node is L3]
As per claim 11, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of Claim 10 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform: randomly selecting the first image sample as an anchor image sample from the set of image samples; (Ge, Page 9, Algorithm 1, Line 3: “Sample anchors randomly and their neighborhoods according to H;”)
determining an anchor set of hierarchical labels in the tree structure associated with the anchor image sample; (Ge, Page 8, Figure 3: “(a)Sampling strategy of each mini-batch. The images in red stand for anchors and the images in blue stand for the nearest neighbors. (b) Train CNNs with the hierarchical triplet loss. (c) Online update of the hierarchical tree.”
PNG
media_image6.png
317
855
media_image6.png
Greyscale
) [Examiner’s note: In part (a) of figure 3, it shows the sampling strategy where images marked in red are selected as anchor samples, and those marked in blue are selected as the nearest neighbors. The hierarchical tree structure depicted in part (c) is associated with updating the tree based on the loss from the loss function. This process involves determining the anchor set (the red-marked images) and associating these with hierarchical labels in the tree structure.]
randomly selecting, for the anchor image sample at a third level from the plurality of levels, a third positive image sample, from the plurality of positive image samples, that shares common label ancestry from the root up to the third level with the anchor image sample; and (Ge, Page 8, ¶[1]: “
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
n
z
), from the positive class,”, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch, which is important for training deep networks with batch normalization [9]. Then m - 1 nearest classes at the 0-th level are selected for each of the l’ nodes, based on the distance between classes computed in the feature space.”) [Examiner’s note: a positive sample is selected from the same positive class with an anchor sample, wherein the positive class is from level 0 (i.e., shares common label L0) of the hierarchical tree]
forming a first positive pair of the plurality of positive sample pairs from the anchor image sample and the third positive image sample (Ge, Page 4, Section 3.1: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa, xp, xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn…
PNG
media_image7.png
38
190
media_image7.png
Greyscale
denotes the hinge loss function, and is the violate margin that requires the distance of
PNG
media_image8.png
45
138
media_image8.png
Greyscale
negative pairs to be larger than the distance of
PNG
media_image9.png
43
137
media_image9.png
Greyscale
positive pairs.”) [Examiner’s note: the positive pair i.e.,
PNG
media_image9.png
43
137
media_image9.png
Greyscale
]
As per claim 12, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of Claim 11 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform: randomly selecting, for the anchor image sample at another level from the plurality of levels, another positive image sample, from the plurality of positive image samples, until positive image samples according to the plurality of levels have been sampled. (Ge, Page 7, Section 4.2, ¶[3]: “
A
l
'
m
2
indicates randomly selecting two classes - a positive class and a negative class, from all l’m classes in the mini-batch.
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
n
z
), from the positive class,”, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch, which is important for training deep networks with batch normalization [9]. Then m - 1 nearest classes at the 0-th level are selected for each of the l’ nodes, based on the distance between classes computed in the feature space.”, and Page 6, Figure 2: “The leaves are the image classes in the training set. Then they are merged recursively until to the root node.
PNG
media_image6.png
317
855
media_image6.png
Greyscale
”) [Examiner’s note: classes are selected randomly from all l’m classes (i.e., all classes of all nodes l’)]
As per claim 13, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of Claim 12 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform: randomly selecting, from the set of image samples, another anchor image until each image sample of the set of image samples [[have]] has been sampled in a training epoch. (Ge, Page 9, Algorithm 1, Lines 1-3: “while not converge do … Sample anchors randomly and their neighborhoods according to H ;”) [Examiner’s note: the while do loop discloses the training epoch, and the anchors are sampled randomly according to the hierarchical tree in the loop]
As per claim 17, Ge explicitly discloses:
As per claim 1, Ge explicitly discloses:
A computer-implemented method for training a machine learning model for computer vision, the method comprising: receiving a set of image samples including a first image sample that is associated with a set of hierarchical labels at a plurality of levels of image categories, (Ge, Page 6, Section 4, Figure 2: “(a) A toy example of the hierarchical tree H. Different colors represent different image classes in CUB-200-2011 [31]. The leaves are the image classes in the training set. Then they are merged recursively until to the root node. (b) The training data distribution of 100 classes visualized by using t-SNE [16] to reduce the dimension of triplet embedding from 512 to 2.”
PNG
media_image1.png
300
802
media_image1.png
Greyscale
, Page 4, Figure 1: “(a) Caltech-UCSD Bird Species Dataset [31]. Images in each row are from the same class. There are four classes in different colors - red, green, blue and yellow.”, and Page 4, Section 3.1, ¶[1]: “”) [Examiner’s note: the training dataset i.e., the Caltech-UCSD Bird Species Dataset, a set of hierarchical labels at a plurality of levels i.e., level L0, L1, L2 and L3 of the hierarchical tree H]
selecting, for the first image sample as an anchor image, a plurality of positive image samples corresponding to the plurality of levels in the set of hierarchical labels, (Ge, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch… Finally, t images for each class are randomly collected, resulting in n (n = l’ mt) images in a mini-batch M.”, Page 7, Section 4.2, ¶[3]: “
A
l
'
m
2
indicates randomly selecting two classes - a positive class and a negative class, from all l’m classes in the mini-batch.
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
p
z
), from the positive class, and
C
t
1
means randomly selecting a negative sample (
x
n
z
) from the negative class.”, Pg. 4, Section 3.1: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa; xp; xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn. The labels of the triplet Tz satisfy ya = yp # yn. Triplet loss aims to pull samples belonging to the same class into nearby points on a manifold surface, and push samples with different labels apart from each other.”) [Examiner’s note: plurality of levels in the set of hierarchical labels i.e., the plurality levels of hierarchical tree H, labels yp # yn means that positive images have similar labels negative images have different labels]
generating a [[first]] training dataset including the first image sample, [[a]] the
plurality of positive image samples. [[pairs]], and the at least one negative image sample (Ge, Page 2, ¶[2]: “These loss functions are calculated on correlated samples, with a common goal of encouraging samples from the same class to be closer, and pushing samples of different classes apart from each other, in a projected feature space. The correlated samples are grouped into contrastive pairs, triplets or quadruplets, which form the training samples for these loss functions on deep metric learning.”, Page 4, Section 3.1, ¶[1]: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa, xp, xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn.”) [Examiner’s note: the contrastive outputs i.e., the contrastive triplet T = (xa, xp, xn)]
Ge fails to disclose:
computing a contrastive loss based on aggregating a pair loss representing a
similarity between the first image sample and each positive image sample of the plurality of positive images and a similarity between the first image sample and each negative image sample of the at least one negative image sample, at each level of the plurality of levels;
training [[a]] the machine learning model using the [[first]] contrastive loss such that
a first maximum contrastive loss computed in a training epoch using the first image sample and first positive image samples corresponding to a second level of the plurality of levels of hierarchical labels is no greater than a second maximum contrastive loss computed in the same training epoch using the first image sample and second positive image samples at a first level of the plurality of levels of hierarchical labels when the second level is lower than the first level;
distances between the first image sample and the plurality of positive image samples are closer in a label space than a distance between the first image sample[[ s ]] [[ of]] and the at least one negative image sample [[pair]]
in response to an input image at inference of the trained machine learning model, generating, by the trained machine learning model, an image category classification output classifying the input image to an image category of the image categories or an image retrieval output identifying an image from an image gallery that has a same image identifier as the input image
However, Goyal explicitly discloses:
a first maximum contrastive loss computed in a training epoch using the first image sample and first positive image samples corresponding to a second level of the plurality of levels of hierarchical labels is no greater than a second maximum contrastive loss computed in the same training epoch using the first image sample and second positive image samples at a first level of the plurality of levels of hierarchical labels when the second level is lower than the first level; (Goyal, Pg. 4, Section 3.1: “Consider the learning framework with training set T… with N training examples and input image features… We represent the labels as yi ϵ {-1, 1}C where C is the number of classes and yi, j= 1 means that the ith example belongs to jth class… Let the set of classes C be arranged in a hierarchy H defined by a hierarchy mapping function h : C → 2C which maps a category c ∈ C to its children categories. We use the function m: C → M to denote the mapping from a category c to its level in the hierarchy. We now define the following hierarchical constraint on a generic loss function l, the satisfaction of which would yield the loss function… The constraint implies that the loss increases monotonically with the level of the hierarchy i.e. loss of higher (i.e. closer to the root) levels in the hierarchy is lesser than that of the lower levels (i.e. closer to the leaves). The intuition is that identifying categories in higher level is easier than categories in lower level as they are coarser.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Goyal. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Goyal teaches Hierarchical multi-label classification (HMC) methods, which utilize the hierarchy of class labels to train a machine learning model in real world scenarios. One of ordinary skill would have motivation to combine Ge and Goyal because incorporate hierarchical relationships into the training objective results in more semantically meaningful and robust predictions.
However, Yuan explicitly discloses:
distances between the first image sample and the plurality of positive image samples are closer in a label space than a distance between the first image sample[[ s ]] [[ of]] and the at least one negative image sample [[pair]] (Yuan, Pg. 1, Section 1: “Although deep metric embedding is modified into different forms for various tasks, it shares the same objective to learn an embedding space that pulls similar images closer and pushes dissimilar images far away. Typically, the target embedding space is learned with a convolutional neural network equipped with contrastive/triplet loss.”, Pg. 4, Col. 2, Section 3.2: “Figure 2 to schematically the process of hard example mining. Cascade Model-1 will forward all pairs in P0 and N0, and try to push all positive points towards the anchor point while pushing all negative points away from the anchor point, and form P1, N1 (points in the 2nd and 3rd tier) by selecting hard samples according to its loss. Similarly, P2 and N2 (points in the 3rd tier) are formed by Cascade Model-2.”)”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Yuan. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Yuan teaches training a cascade of embedding models with increasing depth, where each level handles easy pairs of image and passes only the harder positive pairs to the next deeper model. One of ordinary skill would have motivation to combine Ge and Yuan to ensure that at a lower level, same-label pairs with small distances (low contrastive loss) are resolved, and only more difficult pairs (those with larger loss) are forwarded – effectively enforcing that the hardest positive pair at a shallow level is no harder than those handled at deeper levels.
However, Bekuzarov explicitly discloses:
in response to an input image at inference of the trained machine learning model, generating, by the trained machine learning model, an image category classification output classifying the input image to an image category of the image categories or an image retrieval output identifying an image from an image gallery that has a same image identifier as the input image (Bekuzarov, Pg. 5, ¶[4]: “Indeed, this is how Face Verification can be implemented — a CNN (convolutional neural network) is trained to map input train images of different people to vectors of real numbers (also called “feature-vectors” or “embeddings”) — for example, 128-d vectors, in such a way, that these embeddings of photos of the same person are very close to each other”, Pg. 16: “The input to the entire system is a pair of images (X1, X2) and a label Y . The images are passed through the functions, yielding two outputs G(X1) and G(X2).”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge and Bekuzarov. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Bekuzarov teaches using contrastive loss to train a machine learning model in face verification and face recognition tasks. One of ordinary skill would have motivation to combine Ge and Bekuzarov because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E): “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of the ordinary skill in the art.
However, Xu explicitly discloses:
computing a contrastive loss based on aggregating a pair loss representing a
similarity between the first image sample and each positive image sample of the plurality
of positive images and a similarity between the first image sample and each negative image
sample of the at least one negative image sample, at each level of the plurality of levels; (Xu, Pg. 4, Col. 1, Section: Loss Function: “We simply adjust the contrastive loss in Eq. 1 to allow for multiple positives per anchor. Given an anchor sample xa and its nearest neighborhood set , we randomly select a positive sample xp ∈ Ω, and the loss function can be reformulated as:
PNG
media_image2.png
175
579
media_image2.png
Greyscale
. where each anchor sample qa encoded with fq, is pulled with two samples ka and kq encoded with fk, and pushed away with all other samples in the key encoder fk.” ,Pg. 4, Col. 2, Section 3.3: “we extend the proposed contrastive loss in Eq. 6 to the intermediate hidden layers, which targets at explicitly modeling the similarities among image/patches for better discrimination.”, Pg. 5, Col. 1: “So we get the total loss at stage l:
PNG
media_image3.png
53
220
media_image3.png
Greyscale
. When there are L losses corresponding to L intermediate
stages, the final losses of the whole network can be computed as:
PNG
media_image4.png
56
198
media_image4.png
Greyscale
”) [Examiner’s note: Equation (2) computes the loss for an anchor image, against a positive sample, this is the pairwise loss. Xu extends the contrastive loss to intermediate hidden layers and computes a level-specific loss Ll for each stage, then Ltotal aggregates the losses across all levels]
training [[a]] the machine learning model using the [[first]] contrastive loss such that (Xu, Pg. 3, Col. 1, Section 3.1: “the contrastive loss can be defined as:
PNG
media_image5.png
70
375
media_image5.png
Greyscale
”)
The combination of Ge and Xu are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings of Ge and Xu before them, to modify the teachings of Ge to include the teachings of Xu to improve conventional contrastive learning by incorporating multiple positive samples representing semantic similarity, rather than relying only on augmented views of the same image.
As per claim 18, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of Claim 17 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
wherein the operations comprise: randomly selecting the first image sample as [[an]] the anchor image sample from the set of image samples; (Ge, Page 9, Algorithm 1, Line 3: “Sample anchors randomly and their neighborhoods according to H;”)
determining an anchor set of hierarchical labels in the tree structure associated with the anchor image sample; (Ge, Page 8, Figure 3: “(a)Sampling strategy of each mini-batch. The images in red stand for anchors and the images in blue stand for the nearest neighbors. (b) Train CNNs with the hierarchical triplet loss. (c) Online update of the hierarchical tree.”
PNG
media_image6.png
317
855
media_image6.png
Greyscale
) [Examiner’s note: In part (a) of figure 3, it shows the sampling strategy where images marked in red are selected as anchor samples, and those marked in blue are selected as the nearest neighbors. The hierarchical tree structure depicted in part (c) is associated with updating the tree based on the loss from the loss function. This process involves determining the anchor set (the red-marked images) and associating these with hierarchical labels in the tree structure.]
randomly selecting, for the anchor image sample at a third level from the plurality of levels, a third positive image sample, from the plurality of positive image samples, that shares common label ancestry from the root up to the third level with the anchor image sample; (Ge, Page 8, ¶[1]: “
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
n
z
), from the positive class,”, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch, which is important for training deep networks with batch normalization [9]. Then m - 1 nearest classes at the 0-th level are selected for each of the l’ nodes, based on the distance between classes computed in the feature space.”) [Examiner’s note: a positive sample is selected from the same positive class with an anchor sample, wherein the positive class is from level 0 (i.e., shares common label L0) of the hierarchical tree]
forming a first positive pair of the plurality of positive sample pairs from the anchor image sample and the third positive image sample; (Ge, Page 4, Section 3.1: “During the neural network training, training samples are selected and formed into triplets, each of which Tz = (xa, xp, xn) are consisted of an anchor sample xa, a positive sample xp and a negative sample xn…
PNG
media_image7.png
38
190
media_image7.png
Greyscale
denotes the hinge loss function, and is the violate margin that requires the distance of
PNG
media_image8.png
45
138
media_image8.png
Greyscale
negative pairs to be larger than the distance of
PNG
media_image9.png
43
137
media_image9.png
Greyscale
positive pairs.”) [Examiner’s note: the positive pair i.e.,
PNG
media_image9.png
43
137
media_image9.png
Greyscale
]
randomly selecting, for the anchor image sample at another level from the plurality of levels, another positive image sample, from the plurality of positive image samples, until positive image samples according to the plurality of levels have been sampled; and (Ge, Page 7, Section 4.2, ¶[3]: “
A
l
'
m
2
indicates randomly selecting two classes - a positive class and a negative class, from all l’m classes in the mini-batch.
A
t
2
means selecting two samples - a anchor sample (
x
a
z
) and a positive sample (
x
n
z
), from the positive class,”, Page 7, Section 4.2, ¶[2]: “We randomly select l’ nodes at the 0-th level of the constructed hierarchical tree H. Each node represents an original class, and collecting classes at the 0-th level aims to preserve the diversity of training samples in a mini-batch, which is important for training deep networks with batch normalization [9]. Then m - 1 nearest classes at the 0-th level are selected for each of the l’ nodes, based on the distance between classes computed in the feature space.”, and Page 6, Figure 2: “The leaves are the image classes in the training set. Then they are merged recursively until to the root node.
PNG
media_image6.png
317
855
media_image6.png
Greyscale
”) [Examiner’s note: classes are selected randomly from all l’m classes (i.e., all classes of all nodes l’)]
randomly selecting, from the set of image samples, another anchor image until each image sample of the set of image samples have been sampled in a training epoch. (Ge, Page 9, Algorithm 1, Lines 1-3: “while not converge do … Sample anchors randomly and their neighborhoods according to H ;”) [Examiner’s note: the while do loop discloses the training epoch, and the anchors are sampled randomly according to the hierarchical tree in the loop]
As per claim 6, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of claim 3 (as shown in the rejection above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
computing a first pair loss corresponding to the first positive pair based on a distance between the anchor representation and the first positive representation in a feature space. (Ge, Page 4, Col. 1: “Symmetrically, for positive sample xp, we also have:
PNG
media_image10.png
197
571
media_image10.png
Greyscale
The overall loss is the combination of the two losses, which is equipped with two positive samples in the query encoder fq, and the corresponding two positive samples in the key encoder fk. Each sample is accompanied with a random data augmentation as described in [5], and is pulled together with all positive samples (also undergo a random data augmentation) from the other encoder.”)
generating, by an encoder, an anchor representation and a first positive representation from the anchor image sample and the third positive image sample, respectively; and (Xu, Page 3, Col. 2, Section 3.2: “For positive samples, we simply make use of k nearest neighbors to search semantically similar images in the embedding space. Specially, given unlabeled training set X = {x1, x2, …, xn} and a query encoder fq, we obtain the corresponding embedding representation V = {v1, v2,…, vn} where vi = fq (xi)… Given an anchor sample xa, we compute the cosine similarity with all other images, and select the top k samples with the highest similarity as positives Ω = {x1, x2,…, xk}”, Page 4, Col. 1, ¶[1]: “We simply adjust the contrastive loss in Eq. 1 to allow for multiple positives per anchor. Given an anchor sample xa and its nearest neighborhood set Ω, we randomly select a positive sample xp ϵ Ω, and the loss function can be reformulated as:… where each anchor sample qa encoded with fq, is pulled with two samples ka and kq encoded with fk, and pushed away with all other samples in the key encoder fk.”, and Page 4, Col. 1, ¶[2]: “The overall loss is the combination of the two losses, which is equipped with two positive samples in the query encoder fq, and the corresponding two positive samples in the key encoder fk. Each sample is accompanied with a random data augmentation as described in [5], and is pulled together with all positive samples (also undergo a random data augmentation) from the other encoder.”) [Examiner’s note: an anchor representation i.e., each anchor sample qa encoded with fq, the positive representation i.e., positive samples in the query encoder fq]
As per claim 7, the combination of Ge, Yuan, Goyal, Bakuzarov and Xu discloses all the limitations of claim 6 (as shown in the rejection above).
Xu further discloses:
computing a loss objective based at least in part on summing pair losses over positive image samples at each level and over the plurality of levels. (Xu, Page 5, Col. 1: “When there are L losses corresponding to L intermediate stages, the final losses of the whole network can be computed as:
PNG
media_image11.png
84
397
media_image11.png
Greyscale
” ) [Examiner’s note: the training objective is being interpreted as Ltotal as this is the total losses corresponding to L levels (i.e., stages)]
As per claim 14, the combination of Ge, Yuan, Goyal, Xu and Bakuzarov discloses all the limitations of Claim 11 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
computing a first pair loss corresponding to the first positive pair based on a distance between the anchor representation and the first positive representation in a feature space. (Ge, Page 4, Col. 1: “Symmetrically, for positive sample xp, we also have:
PNG
media_image10.png
197
571
media_image10.png
Greyscale
The overall loss is the combination of the two losses, which is equipped with two positive samples in the query encoder fq, and the corresponding two positive samples in the key encoder fk. Each sample is accompanied with a random data augmentation as described in [5], and is pulled together with all positive samples (also undergo a random data augmentation) from the other encoder.”)
generating, by an encoder, an anchor representation and a first positive representation from the anchor image sample and the third positive image sample, respectively; and (Xu, Page 3, Col. 2, Section 3.2: “For positive samples, we simply make use of k nearest neighbors to search semantically similar images in the embedding space. Specially, given unlabeled training set X = {x1, x2, …, xn} and a query encoder fq, we obtain the corresponding embedding representation V = {v1, v2,…, vn} where vi = fq (xi)… Given an anchor sample xa, we compute the cosine similarity with all other images, and select the top k samples with the highest similarity as positives Ω = {x1, x2,…, xk}”, Page 4, Col. 1, ¶[1]: “We simply adjust the contrastive loss in Eq. 1 to allow for multiple positives per anchor. Given an anchor sample xa and its nearest neighborhood set Ω, we randomly select a positive sample xp ϵ Ω, and the loss function can be reformulated as:… where each anchor sample qa encoded with fq, is pulled with two samples ka and kq encoded with fk, and pushed away with all other samples in the key encoder fk.”, and Page 4, Col. 1, ¶[2]: “The overall loss is the combination of the two losses, which is equipped with two positive samples in the query encoder fq, and the corresponding two positive samples in the key encoder fk. Each sample is accompanied with a random data augmentation as described in [5], and is pulled together with all positive samples (also undergo a random data augmentation) from the other encoder.”) [Examiner’s note: an anchor representation i.e., each anchor sample qa encoded with fq, the positive representation i.e., positive samples in the query encoder fq]
As per claim 15, the combination of Ge, Yuan, Goyal, Bakuzarov and Xu discloses all the limitations of Claim 14 (as shown in the rejections above).
Xu further discloses:
wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform: computing a loss objective based at least in part on summing pair losses over positive image samples at each level and over the plurality of levels. (Xu, Page 5, Col. 1: “When there are L losses corresponding to L intermediate stages, the final losses of the whole network can be computed as:
PNG
media_image11.png
84
397
media_image11.png
Greyscale
” ) [Examiner’s note: the training objective is being interpreted as Ltotal as this is the total losses corresponding to L levels (i.e., stages)]
As per claim 19, the combination of Ge, Yuan, Goyal, Xu and Bekuzarov discloses all the limitations of claim 17 (as shown in the rejection above).
Ge in view of Yuan, Goyal, Xu and Bekuzarov further discloses:
computing a first pair loss corresponding to the first positive pair based on a distance between the anchor representation and the first positive representation in a feature space. (Ge, Page 4, Col. 1: “Symmetrically, for positive sample xp, we also have:
PNG
media_image10.png
197
571
media_image10.png
Greyscale
The overall loss is the combination of the two losses, which is equipped with two positive samples in the query encoder fq, and the corresponding two positive samples in the key encoder fk. Each sample is accompanied with a random data augmentation as described in [5], and is pulled together with all positive samples (also undergo a random data augmentation) from the other encoder.”)
generating, by an encoder, an anchor representation and a first positive representation from the anchor image sample and the third positive image sample, respectively; and (Xu, Page 3, Col. 2, Section 3.2: “For positive samples, we simply make use of k nearest neighbors to search semantically similar images in the embedding space. Specially, given unlabeled training set X = {x1, x2, …, xn} and a query encoder fq, we obtain the corresponding embedding representation V = {v1, v2,…, vn} where vi = fq (xi)… Given an anchor sample xa, we compute the cosine similarity with all other images, and select the top k samples with the highest similarity as positives Ω = {x1, x2,…, xk}”, Page 4, Col. 1, ¶[1]: “We simply adjust the contrastive loss in Eq. 1 to allow for multiple positives per anchor. Given an anchor sample xa and its nearest neighborhood set Ω, we randomly select a positive sample xp ϵ Ω, and the loss function can be reformulated as:… where each anchor sample qa encoded with fq, is pulled with two samples ka and kq encoded with fk, and pushed away with all other samples in the key encoder fk.”, and Page 4, Col. 1, ¶[2]: “The overall loss is the combination of the two losses, which is equipped with two positive samples in the query encoder fq, and the corresponding two positive samples in the key encoder fk. Each sample is accompanied with a random data augmentation as described in [5], and is pulled together with all positive samples (also undergo a random data augmentation) from the other encoder.”) [Examiner’s note: an anchor representation i.e., each anchor sample qa encoded with fq, the positive representation i.e., positive samples in the query encoder fq]
Claim(s) 8, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ge et al. (“Deep Metric Learning with Hierarchical Triplet Loss”) (hereafter referred to as “Ge”) in view of Yuan et al (“Hard-Aware Deeply Cascaded Embedding”) (hereafter referred to as “Yuan”), Goyal & Ghosh (“Hierarchical Class-Based Curriculum Loss”) (hereafter referred to as “Goyal”)and Maksym Bekuzarov (“Losses explained: Contrastive Loss”) (hereafter referred to as “Bekuzarov”), Xu et al. (“Hierarchical Semantic Aggregation for Contrastive Representation Learning”) (hereafter referred to as “Xu”) and further in view of Ramadiansyah & Rahadianti (“Proxy-based Losses and Pair-based Losses for Face Image Retrieval”)
As per claim 8, the combination of Ge, Yuan, Goyal, Bakuzarov and Xu discloses all the limitations of claim 6 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Bakuzarov and Xu further discloses:
at the respective level subject to a condition that the respective maximum pair loss is no less than another maximum pair loss corresponding to a lower level; and (Ge, Page 2, ¶[2]: “These loss functions are calculated on correlated samples, with a common goal of encouraging samples from the same class to be closer, and pushing samples of different classes apart from each other, in a projected feature space.”, Page 8, ¶[2]: “In our hierarchical triplet loss, a sample xa is encouraged to push the nearby points with different semantic meanings apart from itself.”, Page 8, Figure 3:
PNG
media_image6.png
317
855
media_image6.png
Greyscale
) [Examiner’s note: The concept of the pair loss of lower level being greater than the pair loss of upper level is disclosed by Ge when Ge illustrates that the loss function encourages samples from the same class (lower loss value) to be closer (upper level) and pushing samples of different classes (higher loss value) apart (lower level) ]
computing a loss objective based at least in part on summing maximum pair losses over positive image samples at each level and among the plurality of levels. (Xu, Page 5, Col. 1: “When there are L losses corresponding to L intermediate stages, the final losses of the whole network can be computed as:
PNG
media_image11.png
84
397
media_image11.png
Greyscale
” ) [Examiner’s note: the training objective is being interpreted as Ltotal as this is the total losses corresponding to L levels (i.e., stages)]
Ge in view of Yuan, Goyal, Bakuzarov and Xu fails to disclose:
determining, at each level from the plurality of levels, a respective maximum pair loss among positive pairs
However, Ramadiansyah & Rahadianti explicitly discloses:
determining, at each level from the plurality of levels, a respective maximum pair loss among positive pairs (Ramadiansyah & Rahadianti, Page 178, Col. 1, Section 1: “The two input images (anchor and pair) are passed through the ConvNet to generate a fixed-length feature vector and are calculated by using Contrastive loss as illustrated in Eq. 1 as follows:
PNG
media_image12.png
84
650
media_image12.png
Greyscale
where a is the anchor image or test image, p is positive image that belongs to same class with anchor image and n is negative image that has different class with anchor image.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge, Yuan, Goyal, Bekuzarov, Xu and Ramadiansyah. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Goyal teaches Hierarchical multi-label classification (HMC) methods, which utilize the hierarchy of class labels to train a machine learning model in real world scenarios. Bekuzarov teaches using contrastive loss to train a machine learning model in face verification and face recognition tasks. Xu tackles the representation inefficiency of contrastive learning and propose a hierarchical training strategy. Yuan teaches training a cascade of embedding models with increasing depth, where each level handles easy pairs of image and passes only the harder positive pairs to the next deeper model. Ramadiansyah teaches proxy-based losses and pair-based losses for face image retrieval. One of ordinary skill would have motivation to combine Ge, Yuan, Goyal, Bekuzarov, Xu and Ramadiansyah because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E) “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if variations are predictable to one of ordinary skill in the art.
As per claim 16, the combination of Ge, Yuan, Goyal, Bakuzarov and Xu discloses all the limitations of claim 14 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Bakuzarov and Xu further discloses:
at the respective level subject to a condition that the respective maximum pair loss is no less than another maximum pair loss corresponding to a lower label level; and (Ge, Page 2, ¶[2]: “These loss functions are calculated on correlated samples, with a common goal of encouraging samples from the same class to be closer, and pushing samples of different classes apart from each other, in a projected feature space.”, Page 8, ¶[2]: “In our hierarchical triplet loss, a sample xa is encouraged to push the nearby points with different semantic meanings apart from itself.”, Page 8, Figure 3:
PNG
media_image6.png
317
855
media_image6.png
Greyscale
) [Examiner’s note: The concept of the pair loss of lower level being greater than the pair loss of upper level is disclosed by Ge when Ge illustrates that the loss function encourages samples from the same class (lower loss value) to be closer (upper level) and pushing samples of different classes (higher loss value) apart (lower level) ]
computing a loss objective based at least in part on summing maximum pair losses over positive image samples at each level and among the plurality of levels. (Xu, Page 5, Col. 1: “When there are L losses corresponding to L intermediate stages, the final losses of the whole network can be computed as:
PNG
media_image11.png
84
397
media_image11.png
Greyscale
” ) [Examiner’s note: the training objective is being interpreted as Ltotal as this is the total losses corresponding to L levels (i.e., stages)]
Ge in view of Yuan, Goyal, Bakuzarov and Xu fails to disclose:
determining, at each level from the plurality of levels, a respective maximum pair loss among positive pairs
However, Ramadiansyah & Rahadianti explicitly discloses:
determining, at each level from the plurality of levels, a respective maximum pair loss among positive pairs (Ramadiansyah & Rahadianti, Page 178, Col. 1, Section 1: “The two input images (anchor and pair) are passed through the ConvNet to generate a fixed-length feature vector and are calculated by using Contrastive loss as illustrated in Eq. 1 as follows:
PNG
media_image12.png
84
650
media_image12.png
Greyscale
where a is the anchor image or test image, p is positive image that belongs to same class with anchor image and n is negative image that has different class with anchor image.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Ge, Yuan, Goyal, Bekuzarov, Xu and Ramadiansyah. Ge teaches a novel hierarchical triplet loss capable of automatically collecting informative training samples via a defined hierarchical tree. Goyal teaches Hierarchical multi-label classification (HMC) methods, which utilize the hierarchy of class labels to train a machine learning model in real world scenarios. Bekuzarov teaches using contrastive loss to train a machine learning model in face verification and face recognition tasks. Xu tackles the representation inefficiency of contrastive learning and propose a hierarchical training strategy. Yuan teaches training a cascade of embedding models with increasing depth, where each level handles easy pairs of image and passes only the harder positive pairs to the next deeper model. Ramadiansyah teaches proxy-based losses and pair-based losses for face image retrieval. One of ordinary skill would have motivation to combine Ge, Yuan, Goyal, Bekuzarov, Xu and Ramadiansyah because MPEP 2143 sets forth the Supreme Court rationales for obviousness including: (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results; (E) “Obvious to try” choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success; (F) Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if variations are predictable to one of ordinary skill in the art.
Claim(s) 21 is rejected under 35 U.S.C. 103 as being unpatentable over Ge et al. (“Deep Metric Learning with Hierarchical Triplet Loss”) (hereafter referred to as “Ge”) in view of Yuan et al (“Hard-Aware Deeply Cascaded Embedding”) (hereafter referred to as “Yuan”), Goyal & Ghosh (“Hierarchical Class-Based Curriculum Loss”) (hereafter referred to as “Goyal”)and Maksym Bekuzarov (“Losses explained: Contrastive Loss”) (hereafter referred to as “Bekuzarov”), Xu et al. (“Hierarchical Semantic Aggregation for Contrastive Representation Learning”) (hereafter referred to as “Xu”) and further in view of Khosla et al. (“Supervised Contrastive Learning”)
Regarding claim 21, the combination of Ge, Yuan, Goyal, Bakuzarov and Xu discloses all the limitations of claim 1 (as shown in the rejections above).
Ge in view of Yuan, Goyal, Bakuzarov and Xu fails to disclose:
wherein the plurality of positive image samples further include a randomly selected image sample different from the augmented version of the first image sample but depicting a same product as the first image sample
However, Khosla explicitly discloses:
wherein the plurality of positive image samples further include a randomly selected image sample different from the augmented version of the first image sample but depicting a same product as the first image sample (Khosla, Pg. 4, Section 3.1: “Data Augmentation module, Aug(·). For each input sample, x, we generate two random augmentations, x˜ = Aug(x), each of which represents a different view of the data and contains some subset of the information in the original sample. Sec. 4 gives details of the augmentations.”, Pg. 2, ¶[2]: “Our technical novelty in this work is to consider many positives per anchor in addition to many negatives (as opposed to self-supervised contrastive learning which uses only a single positive). These positives are drawn from samples of the same class as the anchor, rather than being data augmentations of the anchor, as done in self-supervised learning.”)
The combination of Ge, Yuan, Goyal, Bakuzarov and Xu are analogous art because they are in the same field of training time series data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention, having the teachings of Ge, Yuan, Goyal, Bakuzarov and Xu before them, to modify the teachings of Yuan, Goyal, Bakuzarov and Xu to include the teachings of Khosla to achieve state of the art performance without the need for hard negative mining, which can be difficult to tune properly.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY TRAN whose telephone number is (571)270-0693. The examiner can normally be reached Monday - Friday 7:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY NMN TRAN/ Examiner, Art Unit 2126
/DAVID YI/ Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights