DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114 was filed in this application after a decision by the Patent Trial and Appeal Board, but before the filing of a Notice of Appeal to the Court of Appeals for the Federal Circuit or the commencement of a civil action. Since this application is eligible for continued examination under 37 CFR 1.114 and the fee set forth in 37 CFR 1.17(e) has been timely paid, the appeal has been withdrawn pursuant to 37 CFR 1.114 and prosecution in this application has been reopened pursuant to 37 CFR 1.114. Applicant’s submission filed on September 1, 2026 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 9-20 are rejected under 35 U.S.C. 103 as being unpatentable over Puszkiewicz et al. (US PGPUB 2020/0159647; hereinafter “Puszkiewicz”) in view of Grechanik et al. (US PGPUB 2011/0307864; hereinafter “Grechanik”), Kumar et al. (US PGPUB 2021/0049234; hereinafter “Kumar”) and Gisslen et al. (US PGPUB 2021/0366183; hereinafter “Gisslen”).
Claim 1: (Currently Amended)
Puszkiewicz teaches method for performing functional testing on an Application Under Test (AUT), the method comprising:
building, by a test system, a model defining each of a plurality of object classifications, the building comprising training the model on data that graphically or visually identifies technological objects ([0024] “test code may be executed that is tightly coupled to an in-memory representation of the GUI of the application to determine whether the in-memory representation is consistent with the developer's intent.” [0036] “As shown in FIG. 3, interface validation system 108 also comprises … a model generator 324 configured to generate a model 326.” [0059] “because model 326 is configured to analyze elements at an object level, as opposed to a specific or exacting combination of pixels in application GUI 312, objects in a captured image may be identified and/or classified irrespective of the precise makeup of pixels and the location of such objects within the captured image.” [0090] “model 326 may be trained at the object level, model 326 may still effectively classify graphical objects using the training data even if the graphical object in an image appears in a different location, comprises a different size or color, and/or contains other noise (e.g., different alphanumeric characters, such as in a completion list that may include different selectable options to complete a phrase or string).”),
at least one of which is different from objects of a user interface of the AUT ([0073] “model generator 324 may train model 326, in part, based on a repository or catalog of training data. For instance, model generator 324 may train model 326 using a repository of associations of generic or commonly found graphical elements (e.g., save icons, close icons, menu bars, etc.) that may be present across a plurality of applications such that model 326 may more accurately classify graphical objects.”);
executing, by the test system, a test script defining one or more functional tests on the AUT ([0025] “A test script launcher may be provided to execute the application comprising the GUI for which validation is desired, as well as a test script that is configured to automatically interact with the GUI of the application.”);
identifying, by the test system, an object type of an object in an image of the user interface of the AUT ([0056] “In step 210, a model is applied that classifies one or more graphical objects in the image. For instance, with reference to FIG. 3, GUI validator 112 may be configured to obtain 336 a captured image and apply the captured image to model 326 to classify one or more graphical objects in the image.” [0058] “This captured image may be applied to model 326, which may analyze the image to identify and classify each object that is present in the image (i.e., the menu, completion list, save button, close button, and/or other elements present).”); and
storing, by the test system, a result of the successful identification of the object ([0034] “In implementations, the storage devices may be configured to store images in hundreds, thousands, millions, and even greater numbers. In implementations, the storage devices may also be configured to store images for a plurality of applications being validated, such as applications validated simultaneously, or a history of images for applications that were previously validated by interface validation system 108.”).
With further regard to Claim 1, Puszkiewicz does not teach the following, however, Grechanik teaches:
navigating, by the test system, the user interface of the AUT to the identified object in the image of the user interface of the AUT ([0059] “Each statement 108 in test scripts (148, 152), which accesses and manipulates GUI objects 118 may include the following operations: (1) navigate to some destination GUI object 142, using GUI object properties 122, and (2) invoke methods 110 to perform actions 140 on the GUI objects 118, including getting and setting values 140.” [0063] “The TIGOR architecture 200 includes script navigate logic 230 (see Table 1) that is executed on the script side to obtain the object v which is contained in the GUI object 118 that is referenced in the test script (108, 202, and 204) using the object oT. The properties of the returned object v are defined in the object repositories 214 under the name p. The script navigate 230 breaks an operation into multiple operations, locating and navigating to the GUI object 118 in the corresponding applications, determine the state of the application when the GUI object is active, locates the GUI object by identifying by matching the recorded properties in the test script in a object repository. Next the GUI object type is matched to the operation to be performed as identified by a composed test script.”), and
wherein navigating the user interface of the AUT performs the one or more functional tests and mimics a user interaction with the user interface of the AUT ([0008] “In order to automate testing of GAPs, test engineers write programs using scripting languages (e.g., JavaScript and VBScript), and these testing scripts drive GAPs through different states by mimicking users who interact with these GAPs by performing actions on their GUI objects. Often test scripts simulate users of GAPs, and their statements access and manipulate GUI objects of these GAPs.” [0056] “The test scripts 152 and composed test script statements 148 may be synchronized in order to facilitate GAPs 154 to exchange data. In a way, composing test scripts 148 mimic how users interact with integrated systems 154.”)
due to the object being successfully identified and classified based on the graphical appearance of the object in the image of the user interface of the AUT ([0050] “For example, the test script statement 108 may include VbWindow (State).Select 3. The API call 112 VbWindow is exported by a GUI testing framework 146. Executing the API call 112 identifies a list box GUI object 118 whose property ‘name’ (e.g., GUI objects properties 122) has the value ‘State’ (e.g., action results 140). By calling the method 110 `Select` with the value 3, the third item in its value list is selected. However, if the referenced GUI object 118 is not of the GUI object type `list box` (e.g., a special form of a `text box` type), the API call 112 will result in a runtime exception.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz with the performing of tests based on the identified objects as taught by Grechanik in order to “provide error free control of GUI objects referenced in composed test scripts to test composed applications” (Grechanik [0016]).
With further regard to Claim 1, Puszkiewicz in view of Grechanik does not teach the following, however, Kumar teaches:
storing, by the test system, a hash of an image of the object ([0085] “At step 212, the process 28 calculates the difference distance between the perceptual hash value of each image saved to the crawler knowledge datastore 100, for the target web page under examination, and the original perceptual hash value of the original image corresponding to the sought element, such as the image provided in field 80, stored in the web element knowledge datastore 48.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik with the storing of hashed object images as taught by Kumar since “One benefit of hash based comparison of multimedia objects is that comparing hash values of the objects is faster than comparing the multimedia objects themselves” (Kumar [0084]).
With further regard to Claim 1, Puszkiewicz in view of Grechanik and Kumar does not teach the following, however, Gisslen teaches:
identifying, an object type of an object based on a scored match between a graphical appearance of the object and an object classification of the plurality of object classifications defined in the model ([0085] “At block 740, the game testing system can execute the testing process on one or more computing systems. The game application can be executed in a specific test mode that is used by the game engine for analyzing the virtual environment during runtime.” [0004] “wherein the analysis of the machine learning model comprises: for individual virtual objects of the plurality of virtual objects, identifying a virtual object in a plurality of frames; determining a classification of the virtual object based on an analysis of a rendered appearance of the virtual object captured within the plurality of frames.” [0071] “The model can analyze the image data included in each frame. Each frame can include a plurality of virtual objects. The model can classify each virtual objects within the frame. The classification of each virtual object can be based on a plurality of images acquired for the object at different angles… The state data can be used to identify the objects within the frames.”),
the identifying comprising:
determining a confidence score based on a degree of matching between the graphical appearance of the object in the image of the user interface of the AUT and the object classification ([0005] “the method includes determining a confidence score associated with the classification of the virtual object.” [0072] “Based on the analysis, the model can generate outputs 210, which can include classification data, confidence data, and identification data.” [0095] “At block 830, the machine learning model can determine a confidence score for the virtual object… The confidence score indicates a level of confidence that the asset has been correctly classified.”);
determining whether the scored match indicates a successful identification of the object based on whether the confidence score exceeds a predefined threshold ([0096] “At decision block 840, the confidence score can be compared to a confidence threshold to determine whether the confidence score satisfies the threshold. If the confidence score satisfies the threshold. Then the classification associated with a virtual object is maintained and the process proceeds to block 870.”); and
responsive to the successful identification of the object, assigning the object type of the object based on the object classification defined in the model ([0087] “At block 760, the game testing system receives the analysis output from the glitch detection process. The output data can include classification data, confidence data, and location data.” [0099] “At block 870, the glitch detection system can output the analysis data. The analysis data may be compiled for the entire glitch analysis session prior to being output by the glitch detection system. For individual glitches, the analysis data can include classification data, confidence data, and location data associated with the virtual objects.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik and Kumar with the identifying objects based on a match score during execution of the test script as taught by Gisslen in order to “provide a more robust analysis of virtual environments within a game application… [and] analyze more portions of virtual environments in less time than when performing manual testing” (Gisslen [0020]).
Claim 2: (Currently Amended)
Puszkiewicz in view of Grechanik, Kumar and Gisslen teaches the method of claim 1. Puszkiewicz in view of Grechanik and Gisslen does not teach the following, however, Kumar teaches:
wherein executing the test script further comprises determining, based on the stored hash of the image of the object and an image of an other object on which one of the one or more functional tests is to be performed, whether the other object on which one of the one or more functional tests is to be performed was previously identified and wherein identifying the other object on which one of the one or more functional tests is to be performed comprises using a result from the stored result of the successful identification of the object ([0084] “FIG. 12 shows an embodiment of the image based element rediscovery function 28. At step 210, the process calculates the perceptual hash value for each image saved to the crawler knowledge datastore 100, for the target web page under examination, which comprised the error at step 14. At step 212, the process 28 calculates the difference distance between the perceptual hash value of each image saved to the crawler knowledge datastore 100, for the target web page under examination, and the original perceptual hash value of the original image corresponding to the sought element, such as the image provided in field 80, stored in the web element knowledge datastore 48.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik and Gisslen with the comparing of hashed object images as taught by Kumar since “One benefit of hash based comparison of multimedia objects is that comparing hash values of the objects is faster than comparing the multimedia objects themselves” (Kumar [0084]).
Claim 3: (Currently Amended)
Puszkiewicz in view of Grechanik, Kumar and Gisslen teaches the method of claim 1, and Puszkiewicz further teaches wherein building the model comprises:
receiving a set of images, each image of the set of images comprising an image of a user interface of a plurality of user interfaces and representing one or more objects of the user interface ([0073] “model 326 may also be trained based on one or more elements unique to a particular application GUI.” [0079] “Each image captured by image capturer 306 at various points in time may comprise any number of bounded regions identifying locations of the expected objects in the image. For instance, a particular image may comprise zero bounded regions, a single bounded region, or a plurality (e.g., dozens or more) of bounded regions.”); and
tagging each object in each image of the set of images ([0078] “with reference to FIG. 3, image tagger 308 may be configured to bound a region of one or more captured images representing application GUI 312 during execution of test script 302. In some implementations, image tagger 308 may be configured to bound a region of one or more such captured images during a training phase.” [0080] “In step 504, the bounded region of the image is tagged with a region identifier. For instance, with reference to FIG. 3, image tagger 308 may be configured to tag each bounded region with a region identifier. The region identifier may comprise, for example, an identifier that identifies the expected object bounded by the region.”).
With further regard to Claim 3, Puszkiewicz in view of Grechanik and Kumar does not teach the following, however, Gisslen teaches:
assigning each image of the set of images to either a training data set of the model or a validation data set of the model, wherein assigning each image to either the training data set or the validation data set further comprises balancing the training data set and the validation data set; training the model based on the training data set; and validating the model based on the validation data set ([0062] “all the images correspondent to one object are present in either the training or validation set but not in both at the same time … For example, 20% of the objects can be placed in the validation set while the remaining 80% can be used for training,” wherein the “balancing” in Gisslen is the stated 20-80 ratio of validation and training images. [0067] “At block 430, the training data can be used to generate the machine learning model.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik and Kumar with the assignment of images to a training set and a validation set as taught by Gisslen as this “gives more validity to the performance observed in the validation set” (Gisslen [0062]).
Claim 9: (Currently Amended)
Puszkiewicz in view of Grechanik, Kumar and Gisslen teaches the method of claim 1. Puszkiewicz in view of Grechanik and Kumar does not teach the following, however, Gisslen teaches wherein identifying the object type of the object in the image of the user interface of the AUT comprises:
identifying a respective object type for each of objects in the image of the user interface of the AUT based on matching a respective graphical appearance of the respective object to one of the plurality of object classifications defined in the model; determining whether the scored match between the respective graphical appearance of the respective object and the one of the plurality of object classifications defined in the model indicates a successful identification of the respective object; and in response to determining the scored match between the respective graphical appearance of the respective object and the one of the plurality of object classifications defined in the model indicates a successful identification of the respective object, classifying the respective object based on the one of the plurality of object classifications defined in the model ([0085] “At block 740, the game testing system can execute the testing process on one or more computing systems. The game application can be executed in a specific test mode that is used by the game engine for analyzing the virtual environment during runtime.” [0004] “wherein the analysis of the machine learning model comprises: for individual virtual objects of the plurality of virtual objects, identifying a virtual object in a plurality of frames; determining a classification of the virtual object based on an analysis of a rendered appearance of the virtual object captured within the plurality of frames.” [0071] “The model can analyze the image data included in each frame. Each frame can include a plurality of virtual objects. The model can classify each virtual objects within the frame. The classification of each virtual object can be based on a plurality of images acquired for the object at different angles… The state data can be used to identify the objects within the frames.” [0095] “At block 830, the machine learning model can determine a confidence score for the virtual object… The confidence score indicates a level of confidence that the asset has been correctly classified.”),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik and Kumar with the identifying objects based on a match score during execution of the test script as taught by Gisslen in order to “provide a more robust analysis of virtual environments within a game application… [and] analyze more portions of virtual environments in less time than when performing manual testing” (Gisslen [0020]).
Claim 10: (Currently Amended)
Puszkiewicz in view of Grechanik, Kumar and Gisslen teaches the method of claim 9 and Puszkiewicz further teaches
wherein identifying the object type of the object in the image of the user interface of the AUT further comprises:
in response to determining the scored match between the respective graphical appearance of the respective object and the one of the plurality of object classifications defined in the model does not indicate a successful identification of the respective object ([0063] “If the measure of confidence of one or more graphical objects is below the threshold, UI image validator 322 may indicate that the validation of the image is unsuccessful.”):
evaluating one or more properties of the respective object ([0064] “validator UI 104 may be configured to obtain, from GUI validator 112, additional information relating to the successful or failed validation, including but limited to results from one or more individual image validation results and any associated information (e.g., an identification of the associated tags indicating the expected objects in the image, the classified graphical objects and/or measures of confidence, etc.).” [0065] “For example, where a particular validation for application GUI 312 has failed, validator UI 104 may enable a user, such as a developer of application 310, to view the results of the validation, analyze one or more images validated by GUI validator 112, including but not limited to graphical objects classified by applying model 326, or any other information associated with validation of application GUI 312.”);
determining whether the one or more properties of the respective object confirm identification of the object type for the respective object based on one or more corresponding properties for the one of the plurality of object classifications defined in the model ([0065] “Where a developer determines that a validation failure is erroneous (i.e., application GUI 312 should have been successfully validated by GUI validator 112)”); and
in response to determining the one or more properties of the respective object confirm identification of the object type for the respective object based on one or more corresponding properties for the one of the plurality of object classifications defined in the model, increasing a confidence score of the scored match between the respective graphical appearance of the respective object and the model and classifying the respective object based on the one of the plurality of object classifications defined in the model ([0065] “Where a developer determines that a validation failure is erroneous … validator UI 104 may enable the developer to modify or correct the information that resulted in the erroneous validation … In this manner, model generator 324 may be configured to continuously retrain and/or refine model 326 based on user input (or lack thereof), thereby further improving the accuracy of the model and the automated validation of an application GUI.”).
Claims 11 and 13-15:
With regard to Claims 11 and 13-15, these claims are equivalent in scope to Claims 1, 3, 9 and 10 rejected above, merely having a different independent claim type, and as such Claims 11 and 13-15 are rejected under the same grounds and for the same reasons as discussed above with regard to Claims 1, 3, 9 and 10.
With further regard to Claim 11, the claim recites additional elements not specifically addressed in the rejection of Claim 1. The Puszkiewicz reference also anticipates these additional elements of Claim 11, for example, wherein the system comprises:
a processor (Fig. 8: Processor Circuit 802); and
a memory coupled with and readable by the processor and storing therein a set of instructions which, when executed by the processor, causes the processor to perform functional testing on an Application Under Test (AUT) (Fig. 8: Hard Disk Drive 814. [0104] “Processor circuit 802 may execute program code stored in a computer readable medium, such as … application programs 832.” [0106] “A number of program modules may be stored on the hard disk… These programs include… application programs 832… Application programs 832 … may include, for example, computer program logic (e.g., computer program code or instructions) for implementing computing device 102, validator UI 104, Server 106, interface validation system 108, test script launcher 110, GUI validator 112, image storage 316, model generator 324, flowchart 200, flowchart 400, flowchart 500, and/or flowchart 700 (including any suitable step of flowcharts 200, 400, 500, or 700) and/or further example embodiments described herein.”).
Claim 12: (Currently Amended)
Puszkiewicz in view of Grechanik, Kumar and Gisslen teaches the system of claim 11 and Puszkiewicz further teaches
wherein the instruction further cause the processor to retrain the model based on the result of the successful identification of the object in the image of the user interface of the AUT ([0065] “UI image validator 322 may also enable model 326 to be continuously refined and/or retrained 340 based on the outcome of the validation of application GUI 312.” [0093] “validator UI 104 may enable the developer to validate the new images in a similar manner as described earlier to generate an updated or replacement catalog of training data that may be used by model generator 324 to retrain model 326.”).
Claims 16-20:
With regard to Claims 16-20, these claims are equivalent in scope to Claims 11-15 rejected above, merely having a different independent claim type, and as such Claims 16-20 are rejected under the same grounds and for the same reasons as discussed above with regard to Claims 11-15.
With further regard to Claim 16, the claim recites additional elements not specifically addressed in the rejection of Claim 11. The Puszkiewicz reference also anticipates these additional elements of Claim 16, for example, wherein Puszkiewicz teaches:
a non-transitory, computer-readable medium comprising a set of instructions stored therein which, when executed by a processor, causes the processor to perform functional testing on an Application Under Test (AUT) ([0104] “Processor circuit 802 may execute program code stored in a computer readable medium, such as … application programs 832.” [0106] “A number of program modules may be stored on the hard disk… These programs include… application programs 832… Application programs 832 … may include, for example, computer program logic (e.g., computer program code or instructions) for implementing computing device 102, validator UI 104, Server 106, interface validation system 108, test script launcher 110, GUI validator 112, image storage 316, model generator 324, flowchart 200, flowchart 400, flowchart 500, and/or flowchart 700 (including any suitable step of flowcharts 200, 400, 500, or 700) and/or further example embodiments described herein.”).
Claims 4-5 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Puszkiewicz in view of Grechanik, Kumar and Gisslen as applied to Claim 3 above, and further in view of Hato (US PGPUB 2017/0169595; hereinafter “Hato”).
Claim 4:
Puszkiewicz in view of Grechanik, Kumar and Gisslen teaches the method of claim 1, and Puszkiewicz teaches wherein tagging each object in each image of the set of images further comprises:
assigning a tag to each object in each image of the set of images ([0078] “with reference to FIG. 3, image tagger 308 may be configured to bound a region of one or more captured images representing application GUI 312 during execution of test script 302. In some implementations, image tagger 308 may be configured to bound a region of one or more such captured images during a training phase.” [0080] “In step 504, the bounded region of the image is tagged with a region identifier. For instance, with reference to FIG. 3, image tagger 308 may be configured to tag each bounded region with a region identifier. The region identifier may comprise, for example, an identifier that identifies the expected object bounded by the region.”).
With further regard to Claim 4, Puszkiewicz in view of Grechanik, Kumar and Gisslen does not teach the following, however, Hato teaches wherein tagging each object in each image of the set of images further comprises:
removing from the objects of the set of images any object having a size less than a predefined object size ([0230] “In S1322, the unusable area determination unit 133 deletes an object area 391 smaller than the size threshold, from the plurality of the object areas 391.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik, Kumar and Gisslen with the removal of objects smaller than a predefined size as taught by Hato since “The object area 391 to be deleted is assumed to be a noise area which is not actually an object area 391 but was selected erroneously” (Hato [0230]).
Claim 5:
Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato teaches the method of claim 4. Puszkiewicz in view of Grechanik, Kumar and Gisslen does not teach the following, however, Hato teaches wherein tagging each object in each image of the set of images further comprises
evaluating graphical characteristics of each image of the set of images and removing objects from the set of images based on the evaluating of the graphical characteristics of the images ([0230] “In S1322, the unusable area determination unit 133 deletes an object area 391 smaller than the size threshold, from the plurality of the object areas 391,” wherein the size of the “object areas” is an evaluated graphical characteristic of the images.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik, Kumar and Gisslen with the removal of objects as taught by Hato since “The object area 391 to be deleted is assumed to be a noise area which is not actually an object area 391 but was selected erroneously” (Hato [0230]).
Claim 8:
Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato teaches the method of claim 4, and Puszkiewicz further teaches wherein tagging each object in each image of the set of images further comprises truncating a portion of each image outside of a bounding box for the image ([0079] “As described herein, bounding of an expected object in a captured image may include bounding a region with a box, circle, or any other shape.” [0070] “In some examples, object classifier 320 may be configured to perform a cropping operation on an image for each identified graphical object such that a cropped image (e.g., a portion of the overall image) representing each graphical object may be applied to model 326. Based on applying an image, or a portion of an image, to model 326, a classification of each graphical object may be determined.”).
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato as applied to Claim 4 above, and further in view of Desai et al. (US PGPUB 2018/0114334; hereinafter “Desai”).
Claim 6:
Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato teaches all the limitations of claim 4 as described above. Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato does not teach the following, however, Desai teaches wherein tagging each object in each image of the set of images further comprises
determining whether more than one tag is defined for an object and, in response to determining more than one tag is defined for the object, removing all tags for the object other than a first tag ([0108] “At block 706, a set of suggested labels is generated for each of the unlabeled data samples of the set of unlabeled data. Once the set of suggested labels for each of the samples is generated and presented to the user, the user selects one label for each sample. The selected label is verified for truthfulness, and the selected label, along with features automatically extracted from the sample, are stored as a training instance. The data is stored along with its label and other information such as visual domain, unique identifier, features, etc,” wherein the “label” is the “tag”. [0109] “The suggested labels can be generated by application of a machine learning model.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato with the removal of all but one tag as taught by Desai in order “to combine the power of automatic image labeling and human feedback” (Desai [0114]).
Claim 7:
Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato teaches all the limitations of claim 4 as described above. Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato does not teach the following, however, Desai teaches wherein tagging each object in each image of the set of images further comprises
determining whether an object within a bounding box for the image is tagged more than once ([0099] “the embodiments of the present invention run the object proposal technique as follows: (1) Based on user satisfaction, only top k proposed bounding boxes (with the highest confidence) are reserved… (2) Among the k bounding boxes, the bounding boxes with large overlap to other bounding boxes are removed. The remaining bounding boxes are cropped out from the original image as the unlabeled training images.”) and,
in response to determining the image within the bounding box is tagged more than once, removing all tags for the object other than a first tag ([0108] “At block 706, a set of suggested labels is generated for each of the unlabeled data samples of the set of unlabeled data. Once the set of suggested labels for each of the samples is generated and presented to the user, the user selects one label for each sample. The selected label is verified for truthfulness, and the selected label, along with features automatically extracted from the sample, are stored as a training instance. The data is stored along with its label and other information such as visual domain, unique identifier, features, etc,” wherein the “label” is the “tag”. [0109] “The suggested labels can be generated by application of a machine learning model.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as disclosed by Puszkiewicz in view of Grechanik, Kumar, Gisslen and Hato with the bounding box determination and the removal of all but one tag as taught by Desai in order “to combine the power of automatic image labeling and human feedback” (Desai [0114]).
Response to Arguments
Applicant's arguments, see Pages 11-19 of the Remarks filed September 1, 2026, with respect to the rejections under 35 U.S.C. 103 of Claims 1-20 have been fully considered but are moot in view of new grounds of rejection.
Applicant's arguments, see Pages 11-19 of the Remarks filed September 1, 2026, with respect to the rejections under 35 U.S.C. 103 of Claims 1-20 have been fully considered but they are not persuasive. With respect to the Applicant’s arguments that the newly amended language of Claims 1, 11 and 16 is not taught by the previously cited prior art, these arguments have been fully considered but are moot in view of the Gisslen et al. (US PGPUB 2021/0366183) reference as discussed above in the respective rejections.
The Office notes that the Gisslen reference was previously cited in the rejections of Claims 3, 13 and 18 in the Final Rejection mailed October 11, 20224. As such the current use of Gisslen in the rejections of independent claims 1, 11 and 16 constitutes a new grounds of rejection, as necessitated by the Applicant’s amendment.
With respect to the Applicant’s further arguments, Page 18 Paragraph 5 of the Remarks, that the features of the remaining claims are not taught by the cited prior art, the Office respectfully disagrees. These arguments rely upon the arguments as presented in relation to claims discussed above, and as such the Office directs the Applicant to the responses above regarding these arguments.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure is as follows:
Madduri et al. (US PGPUB 2021/0192540) discloses a system and method for compliance auditing using cloud based computer vision, including enabling a user to train one or more object recognition models visually by identifying regions of an audit image.
Hu et al. (“AppFlow: Using Machine Learning to Synthesize Robust, Reusable UI Tests,” 2018) discusses a UI testing system called AppFlow which leverages machine learning to automatically recognize common screens and widgets, including the classification of UI elements using a confidence score.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Joanne G. Macasiano whose telephone number is (571)270-7749. The examiner can normally be reached Monday to Thursday, 10:30 AM to 6:00 PM Eastern Standard Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bradley Teets can be reached at (571) 272-3338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOANNE G MACASIANO/Examiner, Art Unit 2197
Prosecution Insights