DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/07/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to the claim(s) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Prior art Baranwal et al., US 2022/0374930 A1 (Baranwal) has been newly added to assist in teaching the newly added claim amendments. Prior arts Ahn (Ahn et al., “Labeling Images with a Computer Game”), Taoka (Taoka et al., US 2018/0260983 A1), Gonzalez (Gonzalez et al., US 2019/0228262 A1), and Besen (Besen et al., US 2019/0213908 A1) are no longer used within the current Office Action.
Applicant's arguments filed 01/07/2026, with regards to the claim limitation “to maintain or increase operator engagement with viewing subsequent portion of the video”, have been fully considered but they are not persuasive. Applicant argues that none of prior arts Zhang (Zhang et al., US 2023/0410487 A1), Ahn (Ahn et al., “Labeling Images with a Computer Game”) nor Taoka (Taoka et al., US 2018/0260983 A1) teaches this limitation. However, regardless of whether the previously used prior art teaches this limitation is moot, since the limitation itself is an intended use statement. In response to applicant's argument that the prior art doesn’t teach “to maintain or increase operator engagement with viewing subsequent portion of the video”, a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim. The Examiner would also like to point out that to “increase operator engagement” is subjective.
Claims 1-7 and 9-14 are pending; claims 8 and 15 were previously canceled; claims 1, 3, 5, 6, and 9-14 have been currently amended.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-7 and 9-14 are rejected under 35 U.S.C. 103 as being unpatentable over Baranwal et al., US 2022/0374930 A1 (Baranwal) and further in view of Zhang et al., US 2023/0410487 A1 (Zhang).
Regarding claim 1, Baranwal teaches method of gamifying annotation (gamification of data labeling) ([0013]) of gestures (a label summarizing the visual content of the video data) ([0038]) and self-perception gestures (labeling based on facial expressions) ([0038]) in a video (wherein labeling of the data sources by data labelers can include video data) ([0032]), the method comprising:
viewing, by an operator (viewing by a labeler; wherein the labeler can include human data labelers) ([0035]), a first portion of a video (a clip of a video) ([0032]) of a subject performing a first gesture (identifying visual content) ([0038]) and a first self-perception gesture (and annotating based on facial expressions) ([0038]);
receiving, by a computational device (user device) ([0035]), a first input from the operator (first input from the labeler) ([0036] and [0038]) that identifies the first gesture and the first self-perception gesture of the subject (wherein the labeler can label the video data with multiple labels including the visual content of the video as well as a mood of the subject based on the facial expression of the person) ([0036] and [0038]);
storing, by the computational device (user device) ([0035]), an annotation (wherein the labeling management system can store the labeled data) ([0031]) corresponding to the first gesture and the first self-perception gesture in the video based on the first input (wherein the labeler can label the video data with multiple labels including the visual content of the video as well as a mood of the subject based on the facial expression of the person) ([0036] and [0038]);
determining, by the computational device (user device) ([0035]), a gamified feedback based on the first input (the interface may be an interface that applies gamification to the labeling performed by the data labelers; such as showing a score for the labeler) ([0044]); and
providing, by the computational device (user device) ([0035]), the gamified feedback to the operator to maintain or increase operator engagement (wherein the gamified feedback is to maintain or increase the operator engagement by incentivizing the labeler; such as to get a high score to be on the top of a scoreboard) ([0044-0048]) with viewing a subsequent portion of the video (use of labeled data facilitating further labeling of additional data of the data source, such as the video, by the labeler) ([0031] and [0047-0048]) and providing a second input that corresponds with a second gesture and a second self-perception gesture (wherein further data can be acquired and the labeler can label the visual content and facial expression of further data) ([0036-0038] and [0047-0048]) in the subsequent portion of the video (wherein the labeler that has been doing a good job is incentivized with a reward/score which includes getting more data for labeling) ([0047-0048]).
However, Baranwal does not explicitly state that the video is of a subject performing a gesture “for product testing”.
Zhang teaches a method of annotation (semantic labels) ([0029]) of gestures in a video of a subject performing a gesture (video of a person performing a gesture, such as “applying eye makeup”) ([0029]), the method comprising: viewing, by an operator (viewing by a human analyst) ([0029]), a first portion of a video (a portion of a video stream) ([0029]) of a subject performing a first gesture for product testing (of a subject performing a gesture such as “applying eye makeup”; wherein the eye makeup is the product) ([0029]); receiving, by a computational device (computing device 700) (Fig. 7; [0052-0054]), a first input from the operator that identifies the first gesture (an input, such as an annotation, from the human analyst viewing the video of the subject corresponding to the gesture, such as “applying eye makeup”) ([0029]); and storing (wherein the semantic labels are a semantic description of action categories, which can be stored in a database) ([0027] and [0037]), by the computational device (computing device 700) (Fig. 7; [0052-0054]), an annotation corresponding to the first gesture in the video based on the first input (wherein the annotation corresponds to the gesture in the video, such as “applying eye makeup”) ([0029]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Baranwal to include being able to annotate/label a gesture of a subject for product testing since more annotations/labels for systems increase the accuracy of the systems using the annotated/labeled data (Zhang; [0021]); while also increasing the modality of Baranwal’s labeling system.
Regarding claim 2, Baranwal teaches wherein the self-perception gesture comprises a facial expression, a facial contortion, a smile, a frown, a facial movement, a remark, or a vocalization (wherein the self-perception gesture can include a facial expression) ([0038]).
However, Baranwal does not explicitly teach “wherein the gesture comprises applying a cosmetic product, or removing a cosmetic product”.
Zhang teaches wherein the gesture comprises applying a cosmetic product or removing a cosmetic product (wherein the gesture in the video is “applying eye makeup”) ([0029]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Baranwal to include being able to annotate/label a gesture of a subject for product testing since more annotations/labels for systems increase the accuracy of the systems using the annotated/labeled data (Zhang; [0021]); while also increasing the modality of Baranwal’s labeling system.
Regarding claim 3, Baranwal teaches wherein storing the annotation (wherein the labeling management system can store the labeled data) ([0031]) comprises associating, with the computational device (user device) ([0035]), the input with a portion of the video at which the gesture and the self-perception gesture is performed (storing the video data that included the visual content and facial expression label) ([0031] and [0036-0038]). Zhang teaches wherein storing the annotation (wherein the semantic labels are a semantic description of action categories, which can be stored in a database) ([0027] and [0037]) comprises associating, with the computational device (computing device 700) (Fig. 7; [0052-0054]), the input with a portion of the video at which the gesture is performed (wherein the annotation corresponds to the specific action occurring at that time in the video, such as “applying eye makeup”, by detecting different actions within the video) ([0020] and [0029]).
Regarding claim 4, Baranwal teaches further comprising: depicting an amount of time left in the video; depicting a high score (incentivized to reach the top of the scoreboard) ([0047]) for the operator and/or a plurality of operators (displaying a scoreboard for the labelers) ([0046-0048]); or depicting a level up or a level down for the operator (wherein the scoreboard is updated in real-time; thus showing leveling up or down of the labeler) ([0046-0048]).
Regarding claim 5, Baranwal teaches further comprising: comparing the first input with other inputs for a comparison and depicting a positive or negative feedback for the operator based on the comparison (wherein the label input from the labeler can be compared to a consensus (other inputted labels) and depict a positive (moving up the scoreboard) or negative (moving down the scoreboard) as feedback) ([0046]); comparing the first input with an input received from a second operator for a comparison and depicting a positive or negative feedback for the operator based on the comparison (wherein the label input from the labeler can be compared to a consensus (other inputted labels) and depict a positive (moving up the scoreboard) or negative (moving down the scoreboard) as feedback) ([0046]).
Regarding claim 6, Baranwal teaches a method of gamifying annotation (gamification of data labeling) ([0013]) of self-perception gestures (labeling based on facial expressions) ([0038]) in a video (wherein labeling of the data sources by data labelers can include video data) ([0032]), the method comprising:
depicting (displaying the video data) ([0078]), by a computational device (user device) ([0035]), a video of a subject that comprises an image of a first self-perception gesture by the subject (receiving a video that can include a video of a person having a facial expression) ([0036] and [0038]);
storing, by the computational device (user device) ([0035]), an annotation (wherein the labeling management system can store the labeled data) ([0031]) corresponding to the first self-perception gesture in the video (wherein the labeler can label the video data with multiple labels including a mood of the subject based on the facial expression of the person) ([0036] and [0038]);
determining, with the computational device (user device) ([0035]), a gamified feedback based on the first self-perception gesture or the annotation (the interface may be an interface that applies gamification to the labeling performed by the data labelers; such as showing a score for the labeler) ([0044]); and
providing, with the computational device (user device) ([0035]), the gamified feedback to the subject to maintain or increase subject engagement (wherein the gamified feedback is to maintain or increase the operator engagement by incentivizing the labeler; such as to get a high score to be on the top of a scoreboard) ([0044-0048]) with providing a second self-perception gesture (wherein further data can be acquired and the labeler can label the facial expression of further data) ([0036-0038] and [0047-0048]) in a subsequent portion of the video (wherein the labeler that has been doing a good job is incentivized with a reward/score which includes getting more data for labeling) ([0047-0048]), wherein the first and second self-perception gestures comprise a facial expression, a facial contortion, a smile, a frown, a facial movement, a remark, or a vocalization (wherein the self-perception gestures can include facial expressions) ([0036] and [0038]). The Examiner would like to point out that it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that since the data for labeling in Baranwal can be a plurality of different types of video data ([0032] and [0034]) that the video can include video of the actual labeler (i.e. a video of a homemade movie).
However, Baranwal does not explicitly state that the video is of a subject “for product testing”.
Zhang teaches a method of annotation (semantic labels) ([0029]) of gestures in a video of a subject for product testing (of a subject performing a gesture such as “applying eye makeup”; wherein the eye makeup is the product) ([0029]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Baranwal to include being able to annotate/label a gesture of a subject for product testing since more annotations/labels for systems increase the accuracy of the systems using the annotated/labeled data (Zhang; [0021]); while also increasing the modality of Baranwal’s labeling system.
Regarding claim 7, Zhang teaches wherein the subject is applying a cosmetic product to a body portion of the subject, removing the cosmetic product from the body portion of the subject, or both, in the video of the subject (wherein the gesture in the video is “applying eye makeup”) ([0029]).
Regarding claim 9, Baranwal teaches further comprising: receiving, by the computational device (user device) ([0035]), a first input from the subject (first input from the labeler) ([0036] and [0038]) that corresponds with the first self-perception gesture (wherein the labeler can label the video data such as a mood of the subject based on the facial expression of the person) ([0036] and [0038]); and generating, by the computational device (user device) ([0035]), the annotation based on the first input (wherein the labeler can generate a label for the video data such as a mood of the subject based on the facial expression of the person) ([0036] and [0038]), wherein the annotation identifies the first self-perception gesture in the video (wherein the label identifies the facial expression in the video) ([0036] and [0038]).
Regarding claim 10, Baranwal teaches wherein generating the annotation identifying the first self-perception gesture in the video (wherein the label identifies the facial expression in the video) ([0036] and [0038]) based on the first input (wherein the labeler can generate a label for the video data such as a mood of the subject based on the facial expression of the person) ([0036] and [0038]) comprises associating, with the computational device (user device) ([0035]), the first input with a portion of the video at which the first self-perception gesture is performed (storing the video data that includes the facial expression label) ([0031] and [0036-0038]).
Regarding claim 11, Baranwal teaches a computational device (user device) ([0035]) for gamifying video annotation (gamification of data labeling) ([0013]), the computational device (user device) ([0035]) comprising circuitry configured to perform a method, the method comprising:
depicting (displaying the video data) ([0078]) a video and/or a first image (a clip of a video and/or image data) ([0032]) of a subject performing a first gesture (identifying visual content) ([0038]) and a first self-perception gesture (and facial expressions) ([0038]);
receiving, from a user, a first input (first input from the labeler) ([0036] and [0038]) that corresponds with the first gesture and the first self-perception gesture (wherein the labeler can label the video data with multiple labels including the visual content of the video as well as a mood of the subject based on the facial expression of the person) ([0036] and [0038]) and annotating the first gesture and the first self-perception gesture in the video and/or the first image based on the first input (wherein the labeler can label the video data with multiple labels including the visual content of the video as well as a mood of the subject based on the facial expression of the person) ([0036] and [0038]);
determining, with the computational device (user device) ([0035]), a gamified feedback based on the first input (the interface may be an interface that applies gamification to the labeling performed by the data labelers; such as showing a score for the labeler) ([0044]); and
providing, with the computational device (user device) ([0035]), the gamified feedback to the user to maintain or increase engagement (wherein the gamified feedback is to maintain or increase the operator engagement by incentivizing the labeler; such as to get a high score to be on the top of a scoreboard) ([0044-0048]) with providing a second input that corresponds with a second gesture and a second self-perception gesture (wherein further data can be acquired and the labeler can label the visual content and facial expression of further data) ([0036-0038] and [0047-0048]) in a subsequent portion of the video or a second image (wherein the labeler that has been doing a good job is incentivized with a reward/score which includes getting more data for labeling) ([0047-0048]);
wherein the first self-perception gesture comprises a facial expression, a facial contortion, a smile, a frown, a facial movement, a remark, or a vocalization (wherein the self-perception gesture can include a facial expression) ([0038]).
However, Baranwal does not explicitly state that the video is of a subject performing a gesture “for product testing”.
Zhang teaches a method of annotation (semantic labels) ([0029]) of gestures in a video of a subject performing a gesture (video of a person performing a gesture, such as “applying eye makeup”) ([0029]), the method comprising: viewing, by an operator (viewing by a human analyst) ([0029]), a first portion of a video (a portion of a video stream) ([0029]) of a subject performing a first gesture for product testing (of a subject performing a gesture such as “applying eye makeup”; wherein the eye makeup is the product) ([0029]); receiving, by a computational device (computing device 700) (Fig. 7; [0052-0054]), a first input from the operator that identifies the first gesture (an input, such as an annotation, from the human analyst viewing the video of the subject corresponding to the gesture, such as “applying eye makeup”) ([0029]); and storing (wherein the semantic labels are a semantic description of action categories, which can be stored in a database) ([0027] and [0037]), by the computational device (computing device 700) (Fig. 7; [0052-0054]), an annotation corresponding to the first gesture in the video based on the first input (wherein the annotation corresponds to the gesture in the video, such as “applying eye makeup”) ([0029]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Baranwal to include being able to annotate/label a gesture of a subject for product testing since more annotations/labels for systems increase the accuracy of the systems using the annotated/labeled data (Zhang; [0021]); while also increasing the modality of Baranwal’s labeling system.
Regarding claim 12, Zhang teaches wherein the first gesture(wherein the gesture in the video is “applying eye makeup”) ([0029]).
Regarding claim 13, Baranwal teaches wherein the first input is received from the subject or an operator (first input from the labeler) ([0036] and [0038]), and wherein the gamified feedback is provided to the subject or the operator (wherein the gamified feedback is to maintain or increase the operator engagement by incentivizing the labeler; such as to get a high score to be on the top of a scoreboard) ([0044-0048]).
Regarding claim 14, Baranwal teaches wherein the method further comprises:
depicting a high score (incentivized to reach the top of the scoreboard) ([0047]) for the operator and/or a plurality of operators (displaying a scoreboard for the labelers) ([0046-0048]); depicting a level up or a level down for the operator (wherein the scoreboard is updated in real-time; thus showing leveling up or down of the labeler) ([0046-0048]); comparing the first input, or a lack of the input, with an input received from a second operator for a comparison and depicting a positive or negative feedback for the operator based on the comparison (wherein the label input from the labeler can be compared to a consensus (other inputted labels) and depict a positive (moving up the scoreboard) or negative (moving down the scoreboard) as feedback) ([0046]).
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J VANCHY JR whose telephone number is (571)270-1193. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J VANCHY JR/Primary Examiner, Art Unit 2666 Michael.Vanchy@uspto.gov
Prosecution Insights