Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see pg. 7, filed 12-22-2025, with respect to drawing objection have been fully considered and are persuasive. The objection has been withdrawn.
Applicant's arguments with respect to 103 rejection have been fully considered but they are not persuasive.
In re pg. 8, applicant argues
PNG
media_image1.png
149
832
media_image1.png
Greyscale
In response, the Examiner respectfully disagrees.
Please see the updated rejection that amended limitation is still disclosed by Almazan.
In re pgs. 8-9, applicant argues
PNG
media_image2.png
273
871
media_image2.png
Greyscale
PNG
media_image3.png
265
875
media_image3.png
Greyscale
In response, the Examiner respectfully disagrees.
Applicant spec disclose ([0004] Before its use, the neural network is trained on a data set which has the objective of optimizing a cost function that can be double, that is to say a function taking into account a double error, namely a "triplet loss", or an "identification loss".25 The cost functions generally the sum of the errors obtained for all the images in the training set, and the training aims to minimize said cost function. Once the neural network has been trained, an image is given as input to the neural network, which provides a digital "signature" as output.).
Thus the claimed digital “signature”, under broadest reasonable interpretation in light of the specification, is any output by the neural network.
Almazan disclose ([0025] It is desirable to design a method for training a convolutional neural network for person re-identification that includes: taking a plurality of triplet of images, wherein where each triplet contains a query image Iq, a positive image I+ corresponding to an image of a same subject as in the query image, and a negative image I− corresponding to an image of a different subject as in the query image; computing the triplet loss for each triplet; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the convolutional neural network on each of the triplets of the subset of triplets to determine trained values of a plurality of parameters of the neural network). Examiner Note: the person identification/re-identification is the output of the neural network; thus, the person identification/re-identification is the digital signature, which is based on mathematical function (e.g. triplet loss).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-4, 6-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Almazan et al (US 20200226421 A1) in view of Nießner (US 20200160502 A1)
1. (currently amended) Almazan disclose A computer-implemented training method for training a neural network configure to output
a first training phase of said neural network based on a set of training images and a training algorithm aiming to minimize a first cost function ([0038] Initially, a training method, implemented by the data processing means 11a of the first server 1a, is executed. The method trains a convolutional neural network for person re-identification in a set of images. [0039] As illustrated in FIGS. 2 and 3, the training method comprises a pre-training classification strategy in which a model is trained to learn the task of person identification classification (which requires the model to first recognize individuals within a closed set of possible identities) before training it for the more challenging task of re-identifying persons unseen during training. [0064] The training method adjusts the current values of the parameters of the neural network using the triplet loss (S25 of FIG. 3) to minimize the triplet loss. The training method can adjust the current values of the parameters of the neural network using conventional neural network training techniques, e.g., stochastic gradient descent with backpropagation. Examiner Note: the person identification classification is the output), and
a second training phase comprising at least one iteration of ([0025] It is desirable to design a method for training a convolutional neural network for person re-identification that includes: taking a plurality of triplet of images, wherein where each triplet contains a query image Iq, a positive image I+ corresponding to an image of a same subject as in the query image, and a negative image I− corresponding to an image of a different subject as in the query image; computing the triplet loss for each triplet; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the convolutional neural network on each of the triplets of the subset of triplets to determine trained values of a plurality of parameters of the neural network. Examiner Note: retraining indicates at least one iteration of training)
providing an image from said set of training images to said neural network, an output of the neural network forming a taking a plurality of triplet of images, wherein where each triplet contains a query image Iq, a positive image I+ corresponding to an image of a same subject as in the query image, and a negative image I− corresponding to an image of a different subject as in the query image; computing the triplet loss for each triplet; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the convolutional neural network on each of the triplets of the subset of triplets to determine trained values of a plurality of parameters of the neural network. Examiner Note: the person identification classification is the output of the neural network),
generating, for said image, at least one corresponding taking a plurality of triplet of images, wherein where each triplet contains a query image Iq, a positive image I+ corresponding to an image of a same subject as in the query image, and a negative image I− corresponding to an image of a different subject as in the query image; computing the triplet loss for each triplet; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the convolutional neural network on each of the triplets of the subset of triplets to determine trained values of a plurality of parameters of the neural network. Examiner Note: the triplet loss is a mathematical function)
calculating an error based upon said real signature and said at least one computing the triplet loss for each triplet; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the convolutional neural network on each of the triplets of the subset of triplets to determine trained values of a plurality of parameters of the neural network. Examiner Note: the triplet loss is a mathematical function indicating error) and
updating at least one layer of said neural network, based upon said error, in order to minimize a second cost function ([0025] It is desirable to design a method for training a convolutional neural network for person re-identification that includes: taking a plurality of triplet of images, wherein where each triplet contains a query image Iq, a positive image I+ corresponding to an image of a same subject as in the query image, and a negative image I− corresponding to an image of a different subject as in the query image; computing the triplet loss for each triplet; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the convolutional neural network on each of the triplets of the subset of triplets to determine trained values of a plurality of parameters of the neural network).
While Almazan disclose image representation ([0104] As described previously, combining Hard Triplet Mining with the other strategies mentioned above is effective for training of the image representation for re-identification.), Almazan fails to disclose digital signature, real signature, artificial signature of the image.
However, Nießner disclose image classification (thereby in same field of endeavor) using image signature ([0006] A computer that identifies a fake image is described. During operation, the computer receives an image. Then, the computer performs analysis on the image to determine a signature that includes multiple features. Based at least in part in the determined signature, the computer classifies the image as having a first signature associated with the fake image or as having a second signature associated with a real image, where the first signature corresponds to a finite resolution of a neural network that generated the fake image, a finite number of parameters in the neural network that generated the fake image, or both. For example, the finite resolution may correspond to floating point operations in the neural network. Alternatively or additionally, the first signature may correspond to differences between the image and, given locations of one or more light sources and one or more objects in the image, predictions of a physics-based rendering technique.)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify image representation of Almazan to incorporate image signature of Nießner with predictable result.
Given the fact that image signature could be used by NN ([0070] In some embodiments, one or more computer vision filters such as SIFT, a Sobel filter, HOG, etc. can be used to obtain a signature for an image. This signature could be used in a neural network and/or another machine-learning approach, such as: a support vector machine. The signature may be used to discriminate between manipulated or fake images and real images under the hypothesis that the features (or signature) detected from the one or more computer vision filters may have a different distribution for a manipulated or fake image versus a real image. Note that the one or more computer vision filters may be signal processing-based and, thus, may not be learned from training data.), one having ordinary skill in the art would have been motivated to make this obvious modification.
3. Almazan disclose the computer-implemented training method according to claim1, wherein the updating is performed using the training algorithm of the first training phase ([0046] In an embodiment, the re-identification neural network (20 of FIGS. 2 and 3) has a three-stream Siamese architecture at training time. [0047] A Siamese neural network is a class of neural network architectures that contain two or more identical subnetworks—identical here means the networks have the same configuration with the same parameters and with shared weights. [0048] Siamese neural networks are particularly adapted for tasks that involve finding similarities or a relationship between two comparable things. [0049] To train the representation end-to-end, a three-stream Siamese architecture in which the weights are shared between all streams is used).
4. Almazan disclose the computer-implemented training method according to claim 1, wherein the training algorithm uses gradient backpropagation method ([0064] The training method adjusts the current values of the parameters of the neural network using the triplet loss (S25 of FIG. 3) to minimize the triplet loss. The training method can adjust the current values of the parameters of the neural network using conventional neural network training techniques, e.g., stochastic gradient descent with backpropagation).
6. Almazan disclose the computer-implemented training method according to claim 1, wherein, for said real signature, number of artificial signatures generated is greater than 1 (0025] It is desirable to design a method for training a convolutional neural network for person re-identification that includes: taking a plurality of triplet of images, wherein where each triplet contains a query image Iq, a positive image I+ corresponding to an image of a same subject as in the query image, and a negative image I− corresponding to an image of a different subject as in the query image.)
While Almazan use 2 artificial image/signatures, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to use any number of artificial image/signatures. It is “Obvious to try” – choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success. See MPEP 2141 III E.
7. Almazan disclose the computer-implemented training method according to claim 1, wherein the first cost function is a double cost function based on a double error, said double error comprising a triplet loss or a contrastive loss, and loss identification error or a classification error, comprising a cross-entropy ([0025] It is desirable to design a method for training a convolutional neural network for person re-identification that includes: taking a plurality of triplet of images, wherein where each triplet contains a query image Iq, a positive image I+ corresponding to an image of a same subject as in the query image, and a negative image I− corresponding to an image of a different subject as in the query image; computing the triplet loss for each triplet; ranking the triplets by the triplet loss computed; selecting a subset of triplets among the plurality of triplets, the subset of triplets having the largest computed loss among the plurality of triplets; and retraining the convolutional neural network on each of the triplets of the subset of triplets to determine trained values of a plurality of parameters of the neural network. [0014] Some recent works have addressed re-identification by applying models that were trained for a classification task using e.g. a cross-entropy loss. Others treat person re-identification, not as a recognition, but rather as a ranking problem, and use losses that are more appropriate for ranking).
8. Almazan disclose the computer-implemented training method according to claim 1, wherein the second cost function calculates an aggregate error based on a real error, calculated based on the real signature; and an artificial error, calculated based on each artificial signature of said at least one corresponding artificial signature by averaging all artificial errors obtained for all artificial signatures of said at least one corresponding artificial signature ([0087] This table reports mean average precision (mAP) on Market and Duke, using ResNet-101 as backbone architecture.).
Claims 9-10 are claims having similar limitation as of claim 1 and are rejected under the same rationale. Note for CNN and object re-identification (See [0002] Embodiments are generally related to the field of machine learning, and more precisely to a method for training a convolutional neural network for person re-identification, and to a method for using such trained convolutional neural network for person re-identification).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Almazan et al (US 20200226421 A1) in view of Nießner (US 20200160502 A1), and further in view of Dalli et al (US 20210256377 A1).
2. Dalli disclose neural network (thereby in same field of endeavor) and further disclose locking NN layers ([0054] Human knowledge can improve XNNs by refining the rule-based knowledge bases in the XNNs via gradient descent techniques. In an exemplary embodiment, special configurations may be applied to lock specific neurons or layers, thereby preventing the network or system from updating those neurons or layers.).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the NN of Almazan to incorporate XNN of Dalli with predictable result of The computer-implemented method according to claim 1, wherein the second training phase further comprises, prior to the updating, locking said at least one layer of the neural network so that said at least one layer that is locked is not updated during updating.
Given the advantage of locking a layer ([0054] Human knowledge can improve XNNs by refining the rule-based knowledge bases in the XNNs via gradient descent techniques. In an exemplary embodiment, special configurations may be applied to lock specific neurons or layers, thereby preventing the network or system from updating those neurons or layers), one having ordinary skill in the art would have been motivated to make this obvious modification.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Almazan et al (US 20200226421 A1) in view of Nießner (US 20200160502 A1), and further in view of Kar et al (US 20200160178 A1).
5. Kar disclose training for neural network (thereby in same field of endeavor) and further disclose generating synthetic data from a normal distribution in which a mean matches the real data; and a variance is a predetermined value of 0.1 ([0037] In order to generate the distribution 214 and/or the distribution 216, the real scenes 212 and/or the generated synthetic scenes 210 (e.g., images representative thereof) may be applied to a feature extractor. The feature extractor may include a computer vision algorithm, a deep neural network trained for feature extraction, or another type of feature extractor algorithm. For example, and without limitation, the feature extractor may be a machine learning model(s) using linear regression, logistic regression, decision trees, support vector machines (SVM), Naïve Bayes, k-nearest neighbor (Knn), K means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, Long/Short Term Memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), and/or other types of machine learning models. The features computed by the feature extractor may be used to determine the distributions 214 and/or 216 (e.g., the information may be analyzed to determine, in a driving scene, locations, poses, colors, sizes, etc. of vehicles). In some embodiments, with respect to the distributions 214, the distributions 214 may be determined from the transformed scene graph 110 without requiring a feature extractor (e.g., the transformed scene graph may include semantic and/or other information indicating attributes of objects in the scenes 210). In other examples, a combination of a feature extractor, the transformed scene graphs 110, and/or another attribute determination method may be used to determine the distributions 214 corresponding to the generated synthetic scenes 210 and/or the real scenes 212. [0045] Equation (4), above, may require being able to sample and/or measure a likelihood from the distribution transformer 108. For continuous attributes, the distribution transformer 108 may be interpreted to be predicting the mean of a normal distribution per attribute, with a pre-defined variance. A re-parametrization trick to sample from the normal distribution may be used in some embodiments. For categorical attributes, it may be possible to sample from a multinomial distribution from the predicted log probabilities per category. As described herein, in some non-limiting embodiments, categorical attributes may be immutable.).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the training data of Almazan to incorporate the training for NN of Kar with predictable result of the computer-implemented method according to claim 1, wherein said at least one corresponding artificial signature is based on a normal distribution having a mean equal to the real signature; and a variance is equal to a predetermined value.
Given the advantage of using a generative model to generate synthetic data ([0003] Data collection and labeling is a laborious, costly, and time consuming task that requires countless human and compute resources. However, machine learning models—such as neural networks—require large amounts of data and corresponding ground truth information for effective training prior to deployment. As a result, the data collection and labeling portion of model training presents a significant bottleneck in most machine learning pipelines. [0004] To combat this issue, synthetic data generation has emerged as a solution to generate ground truth information in greater volumes—e.g., using a graphics engine. Some conventional methods for synthetic data creation require qualified experts to create virtual worlds from which synthetic data is sampled. However, the process of creating virtual worlds manually is an equally laborious task as manual labeling of real-world data. In lieu of creating virtual worlds in this way, some conventional approaches use domain randomization (DR) as a cheaper alternative to photo-realistic environment simulation. DR techniques generate a large amount of diverse scenes by inserting objects into environments in random locations and poses—however, this randomness often results in creating an environment that is very different from—and not optimal as—a proxy for real-world scenes. [0006] Embodiments of the present disclosure relate to generating synthetic datasets for training neural networks. Systems and methods are disclosed that use a generative model—such as a graph convolutional network (GCN)—to transform initial scene graphs sampled from a scene grammar into transformed scene graphs having updated attributes with respect to attributes of the initial scene graphs. [0007] The generative model may be trained to compute the transformed scene graphs such that distributions of corresponding attributes more closely reflect distributions of real-world environments or scenes. In addition, synthetic datasets and corresponding ground truth generated using the transformed scene graphs may be used to train a downstream task network, and the performance of the downstream task network on real-world validation datasets may be leveraged to further train and fine-tune the generative model. As a result, the generative model may not only predict transformed scene graphs that may be used to render more synthetic datasets, but the synthetic datasets may also be tuned for more effectively training a downstream task network for its corresponding task.), one having ordinary skill in the art would have been motivated to make this obvious modification.
It would also have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to use any predetermined variance value. It is “Obvious to try” – choosing from a finite number of identified, predictable solutions, with a reasonable expectation of success. See MPEP 2141 III E.
Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
WANG et al (US 20210019541 A1) disclose transferring visual attributes to image with normal distribution with zero mean and unit variance [0062], Variational Autoencoder-Generative Adversarial Networks (VAE-GANs) [0013].
HAM et al (US 20210064853 A1) disclose Person re-identification using generative adversarial network (GAN). See [0040].
Jansen et al (US 20200372295 A1) disclose machine-learned embedding model can be trained in multiple stages. For example, in a first training stage…, In a second training stage,…See [0033].
Zhang et al (“Auxiliary Training: Towards Accurate and Robust Models” 2020) disclose 2 stages training method. See Fig. 1.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LUT WONG whose telephone number is (571)270-1123. The examiner can normally be reached M-F 10am-6pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at 5712703169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LUT WONG/Primary Examiner, Art Unit 2127
Prosecution Insights