Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The claimed invention is directed to non-statutory subject matter in the form of “computer storage medium “. Claims 17-20 are rejected under 35 U.S.C. § 101 because the claim is directed to non-statutory subject matter in the form of a “computer-readable storage medium.” Applicant’s specification is silent with respect to the term “computer readable storage medium”; therefore, the ordinary and customary meaning of the term is being used. The ordinary and customary meaning of the term “computer readable storage medium” covers “forms of non-transitory tangible media and transitory propagating signals per se…, particularly when the specification is silent.” Subject Matter Eligibility of Computer Readable Media, 1351 Off. Gaz. Pat. Office 212 (Feb. 23, 2010).
“A claim drawn to such a computer readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation ‘non-transitory’ to the claim.” Id.
An amendment that would overcome the instant ‘101 rejection, follows:
Examiner suggests amending to recite: “a non-transitory computer storage medium.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s)1-13 and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over EP 2672396 A1 to Leistner et al., hereinafter “Leistner” in view of US 2021/0232862 A1 to Inoshita.
Claim 1. Leistner teaches A method comprising: [Abstract] A method for annotating a set of images by automatically identifying and visually marking objects
[0056] the set of images is acquired, e.g., by retrieving them from a computer memory or data storage unit 96.
training an object recognition model to recognize an object using a set of training images; [0015] an incremental learning approach which continually updates an object detector and detection thresholds as a user interactively corrects annotations proposed by the system…
determining a performance score indicating an accuracy associated with inferencing performed by the object recognition model using a validation image; [0018] given image data representing a patch of an image, …and the patch score value represents a confidence that the patch is part of an instance of a particular object class (where in the images in which the objects have been detected and considered the validated images)
[0009-0025] teaches that the hypothesis (inference), the annotation costs and a threshold are used to determine whether the detect object (characteristic).
selecting an image characteristic that would cause a performance improvement in the object recognition model as a result of a new image depicting the image characteristic being added to a new set of training images used to retrain the object recognition model; [0016] the hypothesis generating unit comprises a computer-based implementation of a classification method or device which is designed to classify image data and to be incrementally updated, given annotated image data. This allows for the incremental improvement of the hypothesis generating unit after each object detection and its subsequent correction in the annotation step. Examiner interprets annotated image data to be image characteristics.
[0025] selecting one of the images of the subset on the basis of the annotation cost comprises the steps of displaying, on a user interface, a representation (e.g. of a metric that is indicative) of the annotation cost for each of the images of the subset, or for a subset of the images of the subset; Examiner understands selecting one of the images of the subset on the basis of the annotation cost to be selecting an image characteristic
[0025] the step of selecting one of the images of the subset on the basis of the annotation cost comprises the steps of for each image of the subset, or for a subset of the images of the subset, evaluating the annotation cost by an evaluation function; and selecting one of the images according to the result of the evaluation function.
[0009-0025] teaches a threshold are used to determine whether the detect object and thus characteristic would be a false positive (worsen) or improve the object recognition and thus to determine whether a new image would improve the training
[0051-0054] the user selects images with high value of the annotation cost, which are the images that cause the most performance improvement in the object recognition model. The images that are the most useful to improving performance in the prediction model or object recognition model are the high annotation cost images.
[0056] automatic selection based on the ranking of the highest annotation cost (which are the most useful to improve performance of the object recognition model or prediction model).
[0041] a threshold on the maximal allowed annotation cost per image can be used, i.e., the images with the highest expected annotation cost below the threshold are selected. Examiner understands this to be improving performance.- the image that improve performance are selected.
Leistner fails to explicitly teach upon selecting the image characteristic, outputting for display to a user an interface, a prompt asking the user to select an area of a new training image associated with a label that depicts the image characteristic on the object, where the image characteristic associated with the label. Inoshita, in the field of labeling an object in image data, teaches upon selecting the image characteristic, outputting for display to a user an interface, a prompt asking the user to select an area of a new training image associated with a label that depicts the image characteristic on the object, where the image characteristic associated with the label; [0073] As described above, the identification unit 106 calculates the reliabilities of the “automobile”, the “motorcycle”, the “bus”, and the “background” by applying the image to the model, and determines, as the identification result, the item having the highest reliability.
[0074] When the reliability is equal to or less than the predetermined threshold value, the transmission target data determination unit 111 determines that the image as the target used in the identification processing is the transmission target data, and the label reception unit 110 displays the screen on which the operator inputs the label or designates the rectangular region surrounding the object in the image …. FIG. 8 is displayed, the operator inputs the correct label indicating the object appearing in the image 301, designates the rectangular region surrounding the object (in this example, automobile) in the image 301, and then clicks the confirmation button 308. … and determines that the group of the image, the label, and the coordinate is data to be transmitted to the collection device 20 (adding to the collection)
and retraining the object recognition model using the new set of training images. [0107] These images are images (understood to be the new images) for which correct identification results are difficult to be obtained, and a model with high identification accuracy can be obtained by performing deep learning by using these images as the training data. (understood to be retraining)
[0069] …these images are images for which correct identification results and high reliabilities are not obtained, and a model having higher identification accuracy can be generated by using, as the training data, such images, labels indicating objects appearing in the images, and rectangular regions surrounding the objects in the images. (the image with its image characteristic that is selected as a target is the one that will improve the performance of the object recognition model)
Leistner is in the field of annotating images. Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Leistner with the teachings of Inoshita [0002, 0011-0014] to identify the object appearing in the image by applying an image newly captured by the camera to the model and easily collecting data that can contribute to generation of a model with high identification accuracy.
Claim 2. Leistner teaches wherein the method further comprises: generating the image characteristic for a subset of images of the set of training images; [0009] The method serves for annotating images from a set of still images or frames from a video stream (the frames henceforth also called images) by automatically identifying objects in the images and by visually marking the objects in the images. Examiner interprets the identifying the objects is an image characteristics
generating a set characteristic for the set of training images based on the image characteristic, the set characteristic defining a feature of the set of training images; [0009] Given such a set of images showing objects to be identified, the method comprises performing for at least a subset of the set of images, the following steps ■ for each image of the subset, a hypothesis generating unit generating one or more object hypotheses, wherein an object hypothesis defines a region within the image and is associated with an object class (in other words, the region, too, is associated with an object class); ■ for each image of the subset, computing an estimated annotation cost, wherein the annotation cost is a measure of the effort for correcting the object hypotheses in the image, for example, by a human user
associating the performance score with the set characteristic to generate an improvement model training set; [0019] for each object hypothesis, given an object score which represents a confidence that the region of the image associated with the object hypothesis represents an instance of a particular object class
using the improvement model training set to train an improvement model; [0009-0020] incremental learning using the improved training set.
using the improvement model to select the image characteristic. [0009-0025] the annotation costs and a threshold are used to determine whether the detect object and thus characteristic would cause a false positive or improve the object recognition
Claim 3. Leistner teaches wherein the set characteristic comprises a coefficient of variance for the image characteristic of the subset of images. [0043]
Claim 4. Leistner teaches wherein the improvement model is a random decision forest model. [0017] the hypothesis generating unit comprises a computer-based implementation of a single decision tree or of a set of decision trees, that is, a decision forest, in particular of a Hough forest.
Claim 5. Leistner teaches wherein the method further comprises generating a new set of characteristics for the new set of training images, and wherein the improvement model uses the new set of characteristics to select the image characteristic. [0009-0025] the method perform iteratively which implies that the method is creating new characteristics for new training images thus that the improvement model is performed for the new characteristics as well.
Claim 6. Leistner teaches wherein the method further comprises calculating a performance measure of the object recognition model using the new set of characteristics as input to the improvement model. [0009-0025] the method perform iteratively which implies that the method is creating new characteristics for new training images thus that the improvement model is performed for the new characteristics as well.
Claim 7. Leistner teaches wherein the performance measure includes pairs of set characteristic values. [0009-0025] the object score
Claim 8. Reviewed and analyzed in the same way as claim 1. See the above analysis and rationale.
Claim 9. Leistner teaches wherein the processor further performs the operations comprising generating a set characteristic associated with the set of training images, [0009] The method serves for annotating images from a set of still images or frames from a video stream (the frames henceforth also called images) by automatically identifying objects in the images and by visually marking the objects in the images. Given such a set of images showing objects to be identified, the method comprises performing for at least a subset of the set of images, the following steps…
wherein the set characteristic describes a characteristic of the set of training images as a whole. [0009] for each image of the subset, a hypothesis generating unit generating one or more object hypotheses, wherein an object hypothesis defines a region within the image and is associated with an object class (in other words, the region, too, is associated with an object class); ■ for each image of the subset, computing an estimated annotation cost, wherein the annotation cost is a measure of the effort for correcting the object hypotheses in the image, for example, by a human user. Examiner understands the characteristic for each image to be collectively the characteristic of the entire subset (whole).
Claim 10. Leistner teaches wherein the new image depicts a second person different from the first person. Fig. 1 shows multiple people.
Claim 11. Inoshita teaches the system of claim 10, wherein the processor further performs the operations comprising: training an improvement model based on the performance score and the set characteristic; [0073] As described above, the identification unit 106 calculates the reliabilities of the “automobile”, the “motorcycle”, the “bus”, and the “background” by applying the image to the model, and determines, as the identification result, the item having the highest reliability.
[0074] When the reliability is equal to or less than the predetermined threshold value, the transmission target data determination unit 111 determines that the image as the target used in the identification processing is the transmission target data, and the label reception unit 110 displays the screen on which the operator inputs the label or designates the rectangular region surrounding the object in the image …. FIG. 8 is displayed, the operator inputs the correct label indicating the object appearing in the image 301, designates the rectangular region surrounding the object (in this example, automobile) in the image 301, and then clicks the confirmation button 308. … and determines that the group of the image, the label, and the coordinate is data to be transmitted to the collection device 20 (adding to the collection)
and causing the improvement model to determine a predicted performance measure of the object recognition model based on retraining the object recognition model based on the new image. [0107] These images are images (understood to be the new images) for which correct identification results are difficult to be obtained, and a model with high identification accuracy can be obtained by performing deep learning by using these images as the training data. (understood to be retraining)
[0069] …these images are images for which correct identification results and high reliabilities are not obtained, and a model having higher identification accuracy can be generated by using, as the training data, such images, labels indicating objects appearing in the images, and rectangular regions surrounding the objects in the images. (the image with its image characteristic that is selected as a target is the one that will improve the performance of the object recognition model)
Claim 12. Leistner teaches wherein the processor further performs the operations comprising training the object recognition model to recognize the second person using the new image. Active Learning - Returning to Fig. 1, a GUI for annotating a sequence of images or frames is shown, with a central portion displaying the image… the objects are all of the same class, i.e. human figures…a user can select one of these frames for the next training step… Fig. 1 and 4 disclose person detection
Claim 13. Leistner teaches wherein the set characteristic comprises a coefficient of variance for smile intensity of images in the set of training images. [0043]
Claim 16. Leistner teaches wherein the image characteristic are identifiable to a user looking at an image. [0034] Each binary test selects an image feature and compares, for example, the feature values at two pixel locations in a patch. Features are, for example, brightness, colour, gradients, other values determined by filtering the image, etc.
Claim 17. Leistner teaches A computer-storage media having computer-executable instructions embodied thereon that when executed by a computer processor cause a computing device to perform a method comprising: receiving a first set of training images depicting at least a face of a first person; Fig. 1 and 4 discloses person detection
determining an image characteristic that, as a result of retraining a facial recognition model, causes a performance improvement in the facial recognition model based on a new image containing the image characteristic being added to the first set of training images to retrain the facial recognition model; [0009-0025] discloses that the hypothesis, the annotation costs and a threshold are used to determine whether the detect object and thus characteristic would give a false positive or improve the object recognition and thus to determine whether a new image would improve the training or not. a threshold are used to determine whether the detect object and thus characteristic would be a false positive (worsen) or improve the object recognition and thus to determine whether a new image would improve the training
[0051-0054] the user selects images with high value of the annotation cost, which are the images that cause the most performance improvement in the object recognition model… The images that are the most useful to improving performance in the prediction model or object recognition model are the high annotation cost images
[0056] automatic selection based on the ranking of the highest annotation cost (which are the most useful to improve performance of the object recognition model or prediction model).
[0056] automatic selection based on the ranking of the highest annotation cost (which are the most useful to improve performance of the object recognition model or prediction model).
[0041] a threshold on the maximal allowed annotation cost per image can be used, i.e., the images with the highest expected annotation cost below the threshold are selected. Examiner understands this to be improving performance.- the image that improve performance are selected.
Leistner fails to explicitly teach causing a user interface to display a prompt asking a user to provide a label for the new image, the label associated with an area of the new image that depicts the image characteristic. Inoshita, in the field of labeling an object in image data, teaches causing a user interface to display a prompt asking a user to provide a label for the new image, the label associated with an area of the new image that depicts the image characteristic; [0073] As described above, the identification unit 106 calculates the reliabilities of the “automobile”, the “motorcycle”, the “bus”, and the “background” by applying the image to the model, and determines, as the identification result, the item having the highest reliability.
[0074] When the reliability is equal to or less than the predetermined threshold value, the transmission target data determination unit 111 determines that the image as the target used in the identification processing is the transmission target data, and the label reception unit 110 displays the screen on which the operator inputs the label or designates the rectangular region surrounding the object in the image …. FIG. 8 is displayed, the operator inputs the correct label indicating the object appearing in the image 301, designates the rectangular region surrounding the object (in this example, automobile) in the image 301, and then clicks the confirmation button 308. … and determines that the group of the image, the label, and the coordinate is data to be transmitted to the collection device 20 (adding to the collection)
generating a second set of training images including the new image; [0107] These images are images (understood to be the new images) for which correct identification results are difficult to be obtained, and a model with high identification accuracy can be obtained by performing deep learning by using these images as the training data.
and retraining the facial recognition model using the second set of training images. . [0107] These images are images (understood to be the new images) for which correct identification results are difficult to be obtained, and a model with high identification accuracy can be obtained by performing deep learning by using these images as the training data. (understood to be retraining)
Leistner is in the field of annotating images. Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Leistner with the teachings of Inoshita [0002, 0011-0014] to identify the object appearing in the image by applying an image newly captured by the camera to the model and easily collecting data that can contribute to generation of a model with high identification accuracy.
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over EP 2672396 A1 to Leistner et al., hereinafter “Leistner” in view of US 2021/0232862 A1 to Inoshita and in further view of US 2010/0214430 A1 to De Boer et al., hereinafter, “De Boer”.
Claim 14. Leistner fails to explicitly teach the set characteristic comprises a coefficient of variance for exposure of images in the set of training images. De Boer, in the field of acquiring image samples, teaches wherein the set characteristic comprises a coefficient of variance for exposure of images in the set of training images. [0070]
Leistner is in the field of annotating images. Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Leistner with the teachings of De Boer [0007] for improved Coefficient of Variance (CV).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over EP 2672396 A1 to Leistner et al., hereinafter “Leistner” in view of US 2021/0232862 A1 to Inoshita and in further view of US 2014/0146640 A1 to Matsuoka et al., hereinafter, “Matsuoka”.
Claim 15. Leistner fails to explicitly teach the set characteristic comprises a coefficient of variance for exposure of images in the set of training images. Matsuoka, in the field of object identification, teaches wherein the set characteristic comprises a coefficient of variance for a facial landmark in images in the set of training images. [0035] there may be variance in the reflection coefficient of the surface (such as between a white person and a black person)
Leistner is in the field of annotating images. Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Leistner with the teachings of Matsuoka [0003-0004] for improved object detection.
Claim(s) 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over EP 2672396 A1 to Leistner et al., hereinafter “Leistner” in view of US 2021/0232862 A1 to Inoshita and in further view of US 9754190 B1 to Guttmann.
Claim 18. Leistner teaches wherein the method further comprises: generating the image characteristic for a subset of images of the first set of training images; [0009] the creation of image characteristics for each image, wherein the image set is used for training purposes and is thus a training set
generating a set characteristic for the first set of training images based on the image characteristic, wherein the set characteristic indicates a feature of the first set of training images; [0009] for each image of the subset, a hypothesis generating unit generating one or more object hypotheses, wherein an object hypothesis defines a region within the image and is associated with an object class (in other words, the region, too, is associated with an object class); ■ for each image of the subset, computing an estimated annotation cost, wherein the annotation cost is a measure of the effort for correcting the object hypotheses in the image, for example, by a human user. Examiner understands the characteristic for each image to be collectively the characteristic of the entire subset (whole).
training the facial recognition model to based on the first set of training images determining a performance score for the first set of training images, the performance score measuring performance of inferencing performed by the facial recognition model based on a validation image; [0018] teaches an object score which is representing a confidence in the object detection that a characteristic in the image represents an instance of a particular object wherein the images in which the objects have been detected are regarded as validation images
generating an improvement model training set based on the performance score and the set characteristic; [0009-0020] disclose that the hypothesis generating unit is incrementally updated by using the improvement model training set
Leistner fails to explicitly teach training a random decision forest model based on the improvement model training set. Guttmann, in the field of image classification, teaches training a random decision forest model based on the improvement model training set; [col. 40, lines 35-40] the first inference model and the second inference model may comprise random decision forests, and the random decision forest of the first inference model may comprise a smaller number of decision trees and/or smaller decision trees than the random decision forest of the second inference model.
and wherein determining the image characteristic is performed by the random decision forest model. [col. 40, lines 35-40] the first inference model and the second inference model may comprise random decision forests, and the random decision forest of the first inference model may comprise a smaller number of decision trees and/or smaller decision trees than the random decision forest of the second inference model.
Leistner is in the field of annotating images. Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Leistner with the teachings of Guttmann [0003-0004] for inference model for object detection.
Claim 19. Leistner teaches wherein the performance score includes a pair of values. [0009-0025]
Claim 20. Guttmann teaches wherein the method further comprises: determining a second characteristic for the second set of training images; and wherein the random decision forest model uses the second characteristic to determine the image characteristic. [col. 40, lines 35-40] the first inference model and the second inference model may comprise random decision forests, and the random decision forest of the first inference model may comprise a smaller number of decision trees and/or smaller decision trees than the random decision forest of the second inference model.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DELOMIA L GILLIARD whose telephone number is (571)272-1681. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DELOMIA L GILLIARD/Primary Examiner, Art Unit 2661
Prosecution Insights