DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Foreign priority is not claimed.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/06/2024 and 09/19/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. (JP 2021077377 A) in view of Saito et al. (JP-2020119154-A).
Regarding Claim 1,
Zhao discloses A learning apparatus that performs machine learning of a learning model capable of outputting a feature quantity of a person when a person image including the person is inputted, the learning apparatus comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: (Zhao, Description, Note that the units may be logical modules divided according to the particular functionality they implement and are not used to limit a particular implementation. For example, they can be implemented in software, hardware, or a combination of software and hardware. In the actual implementation, the above-mentioned units may be implemented as independent physical entities, or may be implemented by a single entity (eg, processor (such as CPU or DSP), integrated circuit, etc.). Also, the individual units described above are shown by dashed lines in the figure to show that they may not actually exist, and the operations / functions they implement may be implemented by the processing circuit itself; In addition to including a plurality of units, the above-mentioned learning device may be realized in various other forms such as a general-purpose processor and a dedicated processing circuit such as an ASIC. For example, the learning device can be configured by a circuit (hardware) or a central processing unit such as a CPU (central processing unit). Further, the learning device may be equipped with a program (software) for operating a circuit (hardware) or a central processing unit. The program can be stored in a storage device connected from the outside (located in a storage device, etc.) or an external storage medium, and downloaded via a network (Internet, etc.); processor, CPU, memory, program disclosed)
extract a first sample feature quantity that is a feature quantity of a first sample person and a second sample feature quantity that is a feature quantity of a second sample person, by inputting, as the person image, a sample image to the learning model; (Zhao, In the learning stage, the face image learning set is first input to the deep face model to obtain the feature vector of the face image, and then the classification probability from the feature vector using the existing loss function such as Softmax loss function and its transformation. P1, P2, P3 ,. .. .. , Pc (c indicates the number of categories in the learning set such as the face ID corresponding to the c category), and the actual value of the classification probability indicating the probability that the image belongs to each c category and the acquired classification probability. Compared with 0, 1, 0, ..., 0 (1 indicates a true value), determine the difference between the two as loss data, such as cross entropy, and based on the difference to update the deep face model. The above-mentioned operation is restarted using the updated face model until a specific condition is satisfied, and the learned deep face model is acquired; In the testing phase, the face image to be identified or the face image to be authenticated may be input to the trained deep face model in order to extract features for identification or authentication. Specifically, in a real application system there can be two specific applications: face / object recognition and face / object verification. The input to face / object recognition is generally a single face / object image. A trained convolutional neural network is used to identify whether a face / object in the current image is a recognized object, and the input to face / object validation is typically a human face / object image pair. And the trained convolutional neural network is used to extract the feature pairs of the image input pairs, and finally, whether the image input pairs correspond to the same object is based on the similarity of the feature pairs. It is judged; An example of the face recognition operation is shown in FIG. During operation, the two face images to be authenticated are input to the trained deep face model to authenticate whether the two face images are the same person's face image. Specifically, in order for the deep face model to form a feature vector pair, the feature vectors of the two face images can be obtained individually, and then the similarity between the two feature vectors can be determined, for example, similarity. The degree can be determined by the cosine function. If the similarity is equal to or higher than a specific threshold, the two facial images are regarded as the same person's facial image, and if the similarity is less than the specific threshold, the two facial images are not the same person's facial image. You may consider it; training phase, sample image of first person face and sample image of second person face image is input to the training or learning unit and its features are extracted as first and second feature quantities) and
perform the machine learning by using a first loss function regarding accuracy of verification processing of determining, on the basis of the first sample feature quantity, whether or not the first sample person captured in the sample image is the same as the verification subject, and by using a second loss function regarding a distance between the first and second sample feature quantities. (Zhao, In the learning stage, the face image learning set is first input to the deep face model to obtain the feature vector of the face image, and then the classification probability from the feature vector using the existing loss function such as Softmax loss function and its transformation. P1, P2, P3 ,. .. .. , Pc (c indicates the number of categories in the learning set such as the face ID corresponding to the c category), and the actual value of the classification probability indicating the probability that the image belongs to each c category and the acquired classification probability. Compared with 0, 1, 0, ..., 0 (1 indicates a true value), determine the difference between the two as loss data, such as cross entropy, and based on the difference to update the deep face model. The above-mentioned operation is restarted using the updated face model until a specific condition is satisfied, and the learned deep face model is acquired; In the testing phase, the face image to be identified or the face image to be authenticated may be input to the trained deep face model in order to extract features for identification or authentication. Specifically, in a real application system there can be two specific applications: face / object recognition and face / object verification. The input to face / object recognition is generally a single face / object image. A trained convolutional neural network is used to identify whether a face / object in the current image is a recognized object, and the input to face / object validation is typically a human face / object image pair. And the trained convolutional neural network is used to extract the feature pairs of the image input pairs, and finally, whether the image input pairs correspond to the same object is based on the similarity of the feature pairs. It is judged; An example of the face recognition operation is shown in FIG. During operation, the two face images to be authenticated are input to the trained deep face model to authenticate whether the two face images are the same person's face image. Specifically, in order for the deep face model to form a feature vector pair, the feature vectors of the two face images can be obtained individually, and then the similarity between the two feature vectors can be determined, for example, similarity. The degree can be determined by the cosine function. If the similarity is equal to or higher than a specific threshold, the two facial images are regarded as the same person's facial image, and if the similarity is less than the specific threshold, the two facial images are not the same person's facial image; An example of the face recognition operation is shown in FIG. During operation, the two face images to be authenticated are input to the trained deep face model to authenticate whether the two face images are the same person's face image. Specifically, in order for the deep face model to form a feature vector pair, the feature vectors of the two face images can be obtained individually, and then the similarity between the two feature vectors can be determined, for example, similarity. The degree can be determined by the cosine function. If the similarity is equal to or higher than a specific threshold, the two facial images are regarded as the same person's facial image, and if the similarity is less than the specific threshold, the two facial images are not the same person's facial image. As can be seen from the above explanation, the performance of the deep face model directly affects the accuracy of object recognition, and the prior art learns a deep face model such as a deep convolutional neural network model to learn a more complete depth convolutional neural network. Various methods have been used to obtain the model. With reference to FIG. 2A, the learning process for the deep convolutional neural network model in the prior art will be described below; a training data set is input, and the training data set can include a large number of object images such as face images, for example, tens of thousands, hundreds of thousands, or millions of object images; images in the input learning dataset can then be preprocessed, and the preprocessing operations can include, for example, object detection, object alignment, normalization, and the like. In particular, object detection can refer to, for example, detecting a human face from an image containing a human face and acquiring an image mainly containing the human face to be identified, and object alignment can be an image. It can refer to aligning the included object images in different poses to the same or appropriate poses, thereby performing object detection / recognition / tracking based on the aligned object images. Face recognition is a general object recognition operation, and various preprocessing including face detection, face alignment, and the like can be performed on the face recognition learning image set. It should be noted that the pre-processing operation can also include any other type of pre-processing operation known in the art and is not described in detail here; training phase, sample image of first person face and sample image of second person face image is input to the training or learning unit and its features are extracted as first and second feature quantities and compared to determine loss functions for verifications of distance between first person and second person feature quantities and first person and verification person features where first and verification person are same and first and second person are different)
Zhao does not explicitly disclose including the first sample person who is the same as a verification subject and the second sample person who is different from the first sample person,
Saito discloses including the first sample person who is the same as a verification subject and the second sample person who is different from the first sample person,(Saito, Description, discloses the expression on the upper right side matches Expression (4), and the lower expression on the right side shows the loss function in the case of a negative example (in the case of different image pairs). In the case of the negative example, some of the signs are inverted so that the target feature amount and the representative point obtained from different recognizers are learned so as to be further apart. Also, in this example, it was assumed that the data to be recognized and the other data in the data in the Minibatch are always the same person, but this is not the case. You can also do it. In that case, representative points with different labels may be generated for representative points representing different persons and representative points representing the same person. In the above, the case of learning is exemplified, but it is possible to perform the distance-based recognition processing even at the time of detection. Specifically, for example, the distance can be calculated in consideration of the representative points generated by the following formula. It is possible to determine whether or not they are the same person based on the distance obtained by the equation (6) and a predetermined threshold value. It should be noted that it is not a value that comprehensively considers the target feature amount and the representative point as in Expression (6), but when the target feature amount and the representative point output by different recognizers are compared, May be adopted. For example, the distance between the target feature amount output by the recognizer A and the representative point output by the recognizer B is calculated, and the smallest distance value is used. Note that the example of the case of performing one-to-one matching with respect to matching of person images has been described above, but when a plurality of recognition candidates are given and the most matched data is selected from a plurality of search candidates, Possible as a use case. For example, in monitoring in a spectator seat of a stadium, it is desired to identify a person who behaves differently from other persons. As a specific example, for example, in Reference 6, “Figure 3 Shooting Example in Spectator Seat of Sports Stadium: Among people watching the competition, only the people indicated by red circles are different places such as watches and smartphones. is classified as suspicious behavior in this study. The size of the face in this video is about 25 to 30 pixels in the vertical direction.” Thus, acting differently from other persons is considered to be important evidence for the observer in finding suspicious persons. Therefore, as an example, it is considered to determine whether or not a certain person is acting differently from other persons on the screen. In such a case, it is possible to regard it as a problem setting in which the most matched data is selected from a plurality of search candidates, given a plurality of recognition target candidates. In the following, for simplification of description, a case will be described in which the recognition target candidate and the search candidate overlap, and the correspondence between the plurality of search candidates is checked. The case where there is no overlap and the case where there is only a partial overlap will be described later. Here, FIG. 10 shows an example of the configuration in the case of performing matching with a plurality of search candidates using a plurality of recognition targets (during learning). In the example of FIG. 10, a part of the configuration has a function similar to the function of FIG. 6, and in that case, the same reference numerals as those in FIG. 6 are given and duplicate description is omitted. The transformation processing 1001 of FIG. 10 is the transformation processing already described in the present embodiment, and processing for bringing all Minibatch data to the second dimension (feature amount dimension) is performed to create a feature amount of size 1002. Here, N may be variable. In order to convert this feature amount into the feature amount of size 635, the normalized variable representative point generation processing 1003 is used; different positive and negative samples of person images as input as sample images where positive being same person and negative being different person to train and test the classifier that detects person for verification)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Zhao in view of Saito having a method of detecting person with use of loss function between feature quantities of sample first and second person, with the teachings of Saito having, by the training module, performing training and testing of person classifier with use of positive and negative samples of same person to be verified and different person as sample input images for training a classifier in order to accurately validate individuals for applications including authentication.
Regarding Claim 2,
The combination of Zhao and Saito further discloses wherein the first loss function is a loss function that becomes smaller as a probability increases that the first sample person is determined, by the verification processing, to be the same as the verification subject, the second loss function is a loss function that becomes smaller as the distance is increased, and the at least one processor configured to execute the instructions to perform the machine learning such that an integrated loss function obtained by integrating the first and second loss functions becomes smaller. (Saito, Description of embodiments, Further, at the time of detection, a distance calculation process is introduced in place of the loss calculation process 615 in the configuration of FIG. 10, and the distance between the N pieces of recognition target data obtained at the time of detection and the M representative points generated from them. The abnormality determination may be performed based on the result of calculating. Normally, when there are N recognition targets, it is expected to increase the calculation time in the order of N squared in order to calculate the distance between them. According to the method of FIG. 10, it can be compressed in the calculation time of the order of N.Math.M. When M is larger than N, for example, N pieces of recognition target data may be compared with each other one-to-one, or the representative points generated based on the respective recognition target data may be converted by the method described in the present embodiment. The distance may be calculated by comparing. As described above, when the recognition target and the search candidate do not overlap, or when there is only a partial overlap, only the distance between the search candidate and the representative point is calculated when calculating the distance. The calculation process may be shortened by performing the calculation; As a further development, a plurality of search candidates may be gradually narrowed down, for example. In the above-described example of a plurality of recognition targets, the number N of recognition targets may be variable, and thus a method of gradually narrowing down search candidates may be used at the time of learning or detection, and other methods may be used. A process for narrowing down search candidates may be performed using a known method. When using an example of a plurality of recognition targets, it is possible to perform processing for narrowing down the search candidates, for example, by excluding search candidates that are close to the generated representative point. Then, based on the narrowed-down search candidates, the process of generating a representative point is performed again; distance between target and search candidate is determined and based on distance of feature quantities loss function verification of being same person is determined). Additionally, the rational and motivation to combine the references Zhao and Saito as applied in rejection of claim 1 apply to this claim.
Regarding Claim 3,
The combination of Zhao and Saito further discloses wherein the learning model to which the sample image is inputted, outputs a feature map indicating a feature of the sample image, and area information about a first map area corresponding to the first sample person of the feature map, and a second map area corresponding to the second sample person of the feature map, and the at least one processor configured to execute the instructions to extractIn the following description, the data regarding the recognition target for the object and its state will be referred to as recognition target data, and the feature amount extracted from the recognition target data will be referred to as the target feature amount. Note that even when the recognition target data is mapped onto the feature space, it is generally considered that it is often referred to as data on the feature space. Therefore, when simply referred to as recognition target data in the following description, for example, the original image Not only the data on the space but also the data on the feature space are included. The recognition target may be specified by a human, may be specified by a machine as a processing target, and may be singular or plural. In each of the following cases, the nature of the recognition target may be described as a specific example, but the present invention is not intended to be limited to such a case in principle. In the present embodiment, firstly, an example of performing recognition processing of a recognition target in a constant time that does not depend on the number of learning data will be shown. In addition, since an example of the abnormality detection system is given in the present embodiment, an example of an algorithm for learning for the purpose of abnormality detection is given. Examples of the learning algorithm include a Two-Class learning algorithm based on a normal class/abnormal class, and a One-Class learning using only normal data for learning. Of course, the present invention can be used for other purposes, and various configuration examples will be described below in order; first and second feature quantities are mapped in space after extraction for comparing and validating). Additionally, the rational and motivation to combine the references Zhao and Saito as applied in rejection of claim 1 apply to this claim.
Regarding Claim 4,
The combination of Zhao and Saito further discloses wherein position information indicating a position of the first map area and a position of the second map area in the feature map, is given to the sample image as a correct answer label, and the at least one processor configured to execute the instructions to perform the machine learning by using a third loss function regarding respective errors between the positions of the first and second maps area outputted by the learning model and the positions of the first and second maps area given as the correct answer label. (Saito, Description of embodiments, Further, at the time of detection, a distance calculation process is introduced in place of the loss calculation process 615 in the configuration of FIG. 10, and the distance between the N pieces of recognition target data obtained at the time of detection and the M representative points generated from them. The abnormality determination may be performed based on the result of calculating. Normally, when there are N recognition targets, it is expected to increase the calculation time in the order of N squared in order to calculate the distance between them. According to the method of FIG. 10, it can be compressed in the calculation time of the order of N.Math.M. When M is larger than N, for example, N pieces of recognition target data may be compared with each other one-to-one, or the representative points generated based on the respective recognition target data may be converted by the method described in the present embodiment. The distance may be calculated by comparing. As described above, when the recognition target and the search candidate do not overlap, or when there is only a partial overlap, only the distance between the search candidate and the representative point is calculated when calculating the distance. The calculation process may be shortened by performing the calculation; As a further development, a plurality of search candidates may be gradually narrowed down, for example. In the above-described example of a plurality of recognition targets, the number N of recognition targets may be variable, and thus a method of gradually narrowing down search candidates may be used at the time of learning or detection, and other methods may be used. A process for narrowing down search candidates may be performed using a known method. When using an example of a plurality of recognition targets, it is possible to perform processing for narrowing down the search candidates, for example, by excluding search candidates that are close to the generated representative point. Then, based on the narrowed-down search candidates, the process of generating a representative point is performed again; distance between target and search candidate is determined and based on distance of feature quantities loss function verification of being same person is determined). Additionally, the rational and motivation to combine the references Zhao and Saito as applied in rejection of claim 1 apply to this claim.
Claims 5-6 recite apparatus with elements corresponding to the apparatus elements recited in Claims 1-2. Therefore, the recited elements of the apparatus claims 5-6 are mapped to the proposed combination in the same manner as the corresponding elements of Claims 1-2. Additionally, the rationale and motivation to combine the Zhao and Saito references presented in rejection of Claim 1, apply to these claims.
Furthermore, the combination of Zhao and Saito further discloses extract a target feature quantity that is a feature quantity of a target person, by inputting, as a person image, a target image including the target image, to a learning model capable of outputting a feature quantity of a person when the person image including the person is inputted; anddiscloses the expression on the upper right side matches Expression (4), and the lower expression on the right side shows the loss function in the case of a negative example (in the case of different image pairs). In the case of the negative example, some of the signs are inverted so that the target feature amount and the representative point obtained from different recognizers are learned so as to be further apart. Also, in this example, it was assumed that the data to be recognized and the other data in the data in the Minibatch are always the same person, but this is not the case. You can also do it. In that case, representative points with different labels may be generated for representative points representing different persons and representative points representing the same person. In the above, the case of learning is exemplified, but it is possible to perform the distance-based recognition processing even at the time of detection. Specifically, for example, the distance can be calculated in consideration of the representative points generated by the following formula. It is possible to determine whether or not they are the same person based on the distance obtained by the equation (6) and a predetermined threshold value. It should be noted that it is not a value that comprehensively considers the target feature amount and the representative point as in Expression (6), but when the target feature amount and the representative point output by different recognizers are compared, May be adopted. For example, the distance between the target feature amount output by the recognizer A and the representative point output by the recognizer B is calculated, and the smallest distance value is used. Note that the example of the case of performing one-to-one matching with respect to matching of person images has been described above, but when a plurality of recognition candidates are given and the most matched data is selected from a plurality of search candidates, Possible as a use case. For example, in monitoring in a spectator seat of a stadium, it is desired to identify a person who behaves differently from other persons. As a specific example, for example, in Reference 6, “Figure 3 Shooting Example in Spectator Seat of Sports Stadium: Among people watching the competition, only the people indicated by red circles are different places such as watches and smartphones. is classified as suspicious behavior in this study. The size of the face in this video is about 25 to 30 pixels in the vertical direction.” Thus, acting differently from other persons is considered to be important evidence for the observer in finding suspicious persons. Therefore, as an example, it is considered to determine whether or not a certain person is acting differently from other persons on the screen. In such a case, it is possible to regard it as a problem setting in which the most matched data is selected from a plurality of search candidates, given a plurality of recognition target candidates. In the following, for simplification of description, a case will be described in which the recognition target candidate and the search candidate overlap, and the correspondence between the plurality of search candidates is checked. The case where there is no overlap and the case where there is only a partial overlap will be described later. Here, FIG. 10 shows an example of the configuration in the case of performing matching with a plurality of search candidates using a plurality of recognition targets (during learning). In the example of FIG. 10, a part of the configuration has a function similar to the function of FIG. 6, and in that case, the same reference numerals as those in FIG. 6 are given and duplicate description is omitted. The transformation processing 1001 of FIG. 10 is the transformation processing already described in the present embodiment, and processing for bringing all Minibatch data to the second dimension (feature amount dimension) is performed to create a feature amount of size 1002. Here, N may be variable. In order to convert this feature amount into the feature amount of size 635, the normalized variable representative point generation processing 1003 is used; different positive and negative samples of person images as input as sample images where positive being same person and negative being different person to train and test the classifier that detects person for verification)
Claim 7 recite method with steps corresponding to the apparatus elements recited in Claim 1. Therefore, the recited steps of the method claim 7 is mapped to the proposed combination in the same manner as the corresponding elements of Claim 1. Additionally, the rationale and motivation to combine the Zhao and Saito references presented in rejection of Claim 1, apply to this claim.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Suzuki et al. (US-20200117991-A1, A feature model, which calculates a feature value of an input image, is trained on a plurality of first images. First feature values corresponding one-to-one with the first images are calculated using the feature model, and feature distribution information representing a relationship between a plurality of classes and the first feature values is generated. When a detection model which determines, in an input image, each region with an object and a class to which the object belongs is trained on a plurality of second images, second feature values corresponding to regions determined within the second images by the detection model are calculated using the feature model, an evaluation value, which indicates class determination accuracy of the detection model, is modified using the feature distribution information and the second feature values, and the detection model is updated based on the modified evaluation value, Abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PINALBEN V PATEL whose telephone number is (571)270-5872. The examiner can normally be reached M-F: 10am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at 571-272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Pinalben Patel/Examiner, Art Unit 2673
Prosecution Insights