Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments filed June 3rd 2026 have been entered. Applicant’s amendments to the claims have overcome the 35 U.S.C. § 103 rejections previously set forth in the non-final office action mailed March 4th 2026. However, new 35 U.S.C. § 103 rejections have been entered as necessitated by amendment and in light of further search. Applicants amendments have overcome the previously set forth claim objections and said objections are accordingly withdrawn.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
Claim(s) 1-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Amer (US 2019/0303404 A1) in view of Deng (US 2015/0365627 A1), Edward Yu-Te Shen, & Bing-Yu Chen. (2005). Toward gesture-based behavior authoring. International 2005 Computer Graphics, 59–65. https://doi.org/10.1109/cgi.2005.1500374 (hereinafter “Shen”), Konečný, J. and Hagara, M., “One-Shot-Learning Gesture Recognition using HOG-HOF Features”, arXiv e-prints, Art. no. arXiv:1312.4190, 2013. doi:10.48550/arXiv.1312.4190. (hereinafter “Konecny”) and Lee (US 2021/0295578 Al).
Regarding claim 1,
A method comprising:
obtaining a script and an animation associated with a text segment of the script (Figure 1C #112, 113, & 124, paragraphs [0014], [0072] & [0074]) and a first vector generated during script authoring (paragraph [0066], [0152]) based at least in part on a selection of the text segment (paragraphs [0012], [0151], [0152]) and the animation in a user interface corresponding to the script authoring (Fig. 1A #108, Fig. 4, 7B, 7C – selection of events and videos correspond to selection of animation) and captured, via a camera, based on a performance of a gesture by a user during the script authoring, (paragraph [0031], [0070]- “Sensor […] may be a camera and/or a behavior analytics system capable of detecting movements, gestures, poses, and other information.”) the first vector including a first gesture parameter (paragraph [0036], [0119], [0135], [0137], [0152], Amer describes the projection of “three-dimensional deictic vectors from the deixis and gaze onto a display illustrating an animation” [0119], these deixis and gaze are describe earlier as “gestures arising from the subject pointing at props, an object, or other items arising from the subject’s gaze on screen.”, additionally [0152] describes encoding one or more data store into a vector, this data store includes source data which “may include commands, parsed speech, textual information, non-verbal information (e.g., movement, gesture, and gaze-tracking data)” [0135]. Therefore, the vectorization of the data store includes the vectorization of gesture data, and while only the text data is specifically mentioned as being vectorized during what is analogous to the script authoring, it would be obvious to one of ordinary skill in the art to do the same with the gesture information.);
determining a portion of an audio stream of a video stream corresponds to the text segment (paragraph [0138]);
determining a live performance of the gesture performed by the user matches the performance of the gesture (paragraph [0137]) based at least in part on a similarity score determined by at least comparing a second vector, including a second gesture parameter generated based on the performance of the gesture (paragraphs [0036], [0073], [0119], [0135], [0137], [0152], [0070] – “Sensor 171 may detect or sense images, motion, and/or speech and may be a camera and/or a behavior analytics system capable of detecting movements, gestures, poses, and other information.”, as described before Amer teaches a vector generated based on performance of a gesture, paragraph [0070] further describes “gestures” in plural form, implying multiple gestures), and the first vector including the first gesture parameter (paragraphs [0150]-[0152], A ranker encodes one or more from the data store, which can include or at least suggests the gesture vectors as described above, and ranks them based on their similarity scores to a video clip. While this process does not directly compare the two gesture vectors, it provides a mechanism to allow for comparison between the gesture vectors based on their relation to the video clip and because of the broad language “at least in part” can be considered to teach the comparison limitation.); and
responsive to determining vector matches the stored gesture parameter (paragraph [0051], [0152] – describe in detail) and the portion of the audio stream of the video stream corresponds to the text segment (paragraph [0138]), causing the animation to be displayed in the video stream (paragraphs [0105]-[107]).
Amer fails to explicitly teach a captured performance of a gesture during script authoring.
determining a portion of an audio stream of a live video stream corresponds to the text segment;
determining a live performance of the gesture performed by the user in the live video stream matches performance of the gesture by the user during script authoring based at least in part on a similarity score determined by at least comparing a second gesture to first vector;
responsive to determining the similarity score is above a threshold, causing the animation to be displayed in the live video stream.
However, Deng teaches a live video stream; determining a live performance of a gesture performed by a user in the live video stream matches the stored gesture parameter (Figure 2, paragraph [0013]); and responsive to determining the gesture matches the gesture parameter causing the animation to be displayed in the live video stream (paragraphs [0019] & [0036]).
Deng is considered analogous to the claimed invention as it is in the same field of video content, specifically animated video content, generation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date to modify the audio, text and gesture-based animation generation of Amer to include Deng’s live video implementation because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, the audio, text and gesture-based animation generation of Amer and gesture based live video stream animation generation of Deng perform the same general and predictable function, the predictable function being animation generation. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of Amer’s video stream by replacing it with Deng’s live video stream. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Amer in view of Deng fails to explicitly teach a captured performance of a gesture during script authoring.
determining a performance of the gesture performed by the user matches performance of the gesture by the user during script authoring based at least in part on a similarity score determined by at least comparing a second gesture to the first vector;
responsive to determining the similarity score is above a threshold, causing the animation to be displayed.
However, Shen teaches a captured performance of a gesture during script authoring (Shen – intro paragraph 2, section 3.3)
Shen describes a script authoring system which allows users to capture performances of mouse gestures during script authoring which align with a gesture of an animated character. While this is not directly analogous to a human performed gesture, it is still a performed gesture captured during script authoring. Shen is considered analogous to the claimed invention as it is in the same field of animation and script authoring. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Amer in view of Deng’s gesture detection to include Shen’s gesture recognition during script authoring because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, Amer in view of Deng’s gesture recognition as modified by Shen’s script authoring system can yield a predictable result of improving the user experience. Thus, a person of ordinary skill would have appreciated including in Amer in view of Deng’s gesture recognition, the ability to do so in script authoring, as is done in Shen, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Amer in view of Deng and Shen fail to teach determining a performance of the gesture performed by the user matches performance of the gesture by the user during script authoring based at least in part on a similarity score determined by at least comparing a second gesture to the first vector;
responsive to determining the similarity score is above a threshold, causing the animation to be displayed.
However, Konecny teaches determining a performance of the gesture performed by the user matches performance of the gesture by the user during script authoring (abstract, sections 1 & 3 – the problem of one shot gesture recognition is analogous to determining a gesture during script authoring matches a gesture performed in presentation because both involve utilizing one representation of a first gesture to recognize a subsequent gesture) based at least in part on a similarity score determined by at least comparing a second vector, including a second gesture parameter to the first vector (sections 5.2, 5.3, section 6.1, 6.2 – The distance of histograms can be considered analogous to a similarity score as it is representative of the difference between two gesture representations, which are based on optical flow vectors); Konecny is considered analogous to the claimed invention as it is in the same field of gesture recognition. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Konecny with Amer in view of Deng and Shen because such a modification is based on the use of known techniques to improve similar devices in the same way. More specifically, Konecny’s one-shot gesture recognition is comparable to Amer and Deng’s gesture recognition. Therefore, it is within the capabilities of one of ordinary skill in the art to modify Amer in view of Deng and Shen’s gesture recognition to include Konecny’s one-shot gesture recognition with the predictable result of improved gesture recognition to “narrow the gap between human and machine performance” (section 8).
Amer in view of Deng, Shen and Konecny fail to teach responsive to determining the similarity score is above a threshold, causing the animation to be displayed.
However, Lee teaches responsive to determining the similarity score is above a threshold, causing the animation to be displayed (paragraphs [0082], [0094], [0101], [0102]). Lee describes the generation of avatar movement (gestures) based upon text detected from inputted audio. Lee matches the animation and text using a similarity score, and displaying the animation when the score is greater than or equal to a threshold. Lee is considered analogous to the claimed invention as it is in the same field of audio-based animation generation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Lee with Amer in view of Deng, Shen and Konecny because such a modification would have been obvious to try. More specifically, Lee’s similarity threshold is one of a predictable and ascertainable group of similar features. This group addresses the design need and/or other recognized problem of determining a correct animation to display with a reasonable level of success. Therefore, it would have been obvious to try to modify Amer in view of Deng, Shen and Konecny to include Lee’s similarity threshold since there are a finite number of identified, predictable potential solutions to the recognized need (as discussed above) and one of ordinary skill in the art could have pursued the known potential solutions with a reasonable expectation of success.
Regarding claim 2, Amer in view of Deng, Shen, Konecny and Lee teach the method of claim 1. Amer
further teaches the method further comprises:
obtaining a selection of words in the script from the user (paragraph [0012]); and
generating the text segment based on the selection of words and at least one word in the script preceding the selection of words (paragraphs [0010],[0062],[0063],[0101],[0102]).
Amer describes generating animations for a text input phrase, which is analogous to a selection of words. Amer suggests the use of a human collaborator to modify scene description, which would include modifying or selecting the text input phrase, analogous to obtaining a selection of words in the script from the user. Amer further describes the use of a language parser to process text input and perform event extraction and glean context from descriptions, analogous to generation based on at least one word in the script preceding the selection of words.
Regarding claim 3, Amer in view of Deng, Shen, Konecny and Lee teach the method of claim 2. Deng further teaches wherein determining the live performance of the gesture performed by the user in the live video stream matches the performance of the gesture by the user during script authoring further comprises, causing generation of gesture parameters corresponding to the gesture performed by the user in the live video stream(paragraphs [0036], [0037]). Deng describes association of some example gestures performed by a user and their corresponding gesture parameters.
Lee further teaches in response to detecting the at least one word in the portion of the audio stream causing a gesture model to generate (paragraph [0013]). Lee describes the generation of avatar movement (gestures) based upon text detected from inputted audio. Lee uses a mapping table and a decision-making process to generate a relationship between text and avatar gestures; this is analogous to an audio stream causing a gesture model to generate.
Regarding claim 4, Amer in view of Deng, Shen, Konecny and Lee teach the method of claim 1. Amer further teaches wherein causing the animation to be displayed in the live video stream further comprises initiating an adaptation interval during which the animation is displayed (paragraphs [0011], [0187]) based on at least one of: a second portion of the audio stream corresponding to a word in the text segment or a cadence of the user determined based on the second portion of the audio stream (paragraphs [0019], [0062], [0138], [0187]). Amer describes the use of “event frames” which specify a certain input event to be animated. Amer also describes the use of loudness, pitch, speaking rate, and prosody of speech in an audio stream to generate event frames related to different text portions parsed from the audio stream. This is analogous to basing the adaptation interval on both words and speech cadence of a second portion of the audio stream
Regarding claim 5, Amer in view of Deng, Shen, Konecny and Lee teach the method of claim 1. Amer further teaches causing the animation to be displayed in the live video stream further comprises initiating an adaptation interval during which the animation is displayed based on a similarity score indicating an amount that a second gesture performed by the user in the live video stream matching a second gesture parameter (paragraph [0014], [0037]). Amer describes the use of event frames for the animations which are analogous to the claimed invention’s adaptation interval. Amer also describes the assignment of similarity scores for different animations to be displayed. These animations being compared are generated from gesture recognition in various embodiments, therefore the similarity scores are based on gesture similarity.
Regarding claim 6, Amer in view of Deng, Shen, Konecny and Lee teach the method of claim 5. Amer further teaches wherein the animation includes a plurality of graphical states, where a graphical state of the plurality of graphical states defines a set of graphic parameters for an object in the animation (paragraphs [0055] & [0056]). Amer describes various methods of evaluating every frame or time resolution (graphical states) of an animation, these evaluations result in scores or parameters for validating the graphical states. Amer also describes object detection, meaning that the scores of parameters described previously may be representative of an object in the animation.
Regarding claim 7, Amer in view of Deng, Shen, Konecny and Lee teach the method of claim 6. Amer further teaches method further comprises determining to advance the animation to a second graphical state of the plurality of graphical states based on a weight value applied to the similarity score (paragraphs [0014]-[0016], [0056]). Amer describes producing scores of every frame or time resolution using discriminators which produce said scores using weights. The scores generated are also described as informing which animation to display.
Claim(s) 8, 9, and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Amer in view of Liao, J., Karim, A., Jadon, S. S., Kazi, R. H., & Suzuki, R. (2022). RealityTalk: Real-time speech-driven augmented presentation for AR Live Storytelling. Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, 1–12. https://doi.org/10.1145/3526113.3545702 (hereinafter “Liao”), Shen and Konecny.
Regarding claim 8, Amer teaches
A non-transitory computer-readable medium storing executable instructions (paragraph [0016]) embodied thereon, which, when executed by a processing device, cause the processing device to perform operations comprising:
obtaining a script and an animation to be applied to a video stream in response to a text segment included in the script and a first vector representing a gesture (paragraph [0036], [0119], [0135], [0137], [0152], Amer describes the projection of “three-dimensional deictic vectors from the deixis and gaze onto a display illustrating an animation” [0119], these deixis and gaze are described earlier as “gestures arising from the subject pointing at props, an object, or other items arising from the subject’s gaze on screen.”, additionally [0152] describes encoding one or more data store into a vector, this data store includes source data which “may include commands, parsed speech, textual information, non-verbal information (e.g., movement, gesture, and gaze-tracking data)” [0135]. Therefore, the vectorization of the data store includes the vectorization of gesture data.) to be performed in the video stream (Figure 1c #124, #112, #113, paragraphs [0063], [0072], [0074]), the animation including a plurality of states of an object; and
causing the animation to be displayed in the video stream (paragraph [0048], [0056]) in response to:
detecting a first portion of the text segment based on text converted from an audio stream corresponding to the video stream (paragraph [0012], [0138]); and
detecting, by a gesture model, the gesture performed by the user in the video stream (paragraph [0137]) by at least determining a similarity score (paragraphs [0160], [0192]) based on a second vector generated based on the gesture in the video stream and the first vector (paragraphs [0036], [0070], [0073], [0119], [0135], [0137], [0150]-[0152] - as described before Amer teaches a vector generated based on performance of a gesture, paragraph [0070] further describes “gestures” in plural form, implying multiple gestures. Additionally, a ranker encodes one or more from the data store, which can include, or at least suggests, the gesture vectors as described above, and ranks them based on their similarity scores to a video clip. While this process does not directly compare the two gesture vectors, it provides a mechanism to allow for comparison between the gesture vectors based on their relation to the video and enables implementation of a more specific comparison mechanism for gesture recognition.)
Amer describes the generation of video content, such as an animation, with text, audio, or non-text related input such as gestures. More specifically, Amer describes a non-transitory computer-readable medium storing executable instructions, a method for obtaining audio, text and gesture information, both input and output of video (analogous to video stream), animation which includes a plurality of frames (each frame can be considered a graphical state), speech recognition of audio input, and gesture recognition. Amer is analogous to the claimed invention as it is in the same field of multi-input animation generation. Therefore, it would have been obvious to one of ordinary skill in the art to combine the various teachings of input methods, animation generation, and video editing ([0132]) together to apply the methods to an existing video stream rather than the generated stream of Amer.
Amer fails to teach a first gesture to be performed live during the video stream, wherein the text segment is selected by a user during script authoring prior to the video stream and is associated with a second gesture performed during script authoring and a similarity score between a first vector generated based on the first gesture performed live during the video stream and the first vector generated based on the gesture performed during script authoring.
However, Liao teaches a gesture to be performed live during the video stream (section 5.3, Figs 6, 8 & 9), wherein the text segment is selected by a user during script authoring prior to the video stream (Sections 5.1, 5.2) and is associated with the gesture (section 5.3) and a similarity score between a second vector generated based on the gesture performed live during the video stream and the gesture defined during script authoring (section 6.4). Liao describes a live storytelling system that inserts visuals associated with textual elements to a video, which can be modified and manipulated using gestures. Liao includes a script authoring phase that allows users to associated visual elements with text and speech. Liao describes recognizing gestures based on calculations using position and finger and pinching states, because these states are indicative of direction (i.e. upward or inward for finger state) the information can be considered analogous to a vector. Liao is considered analogous to the claimed invention as it is in the same field of image processing, and live graphics processing. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Liao with Amer to improve the system of augmenting live video or presentations.
Amer in view of Liao fails to teach a gesture performed during script authoring and a similarity score between a gesture performed live during the video stream and the gesture performed during script authoring.
However, Shen teaches a gesture performed during script authoring (section 3). Shen describes capturing gestures during script authoring. While Shen does not generate a vector based on this gesture and calculate a similarity score, it would have been obvious to one of ordinary skill in the art to integrate the capturing of a gesture during script authoring from Shen with the animation generation system of Amer in view of Liao, which generates representative data vectors, in order to allow non-programmers to engage in creating animations.
Amer in view of Liao and Shen fail to teach a similarity score between a gesture performed live during the video stream and the gesture performed during script authoring.
However, Konecny teaches a similarity score between a gesture performed live during the video stream and the gesture performed during script authoring (sections 5.2, 5.3, section 6.1, 6.2). Konecny is considered analogous to the claimed invention as it is in the same field of gesture recognition. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Konecny with Amer in view of Liao and Shen because such a modification is based on the use of known techniques to improve similar devices in the same way. More specifically, Konecny’s one-shot gesture recognition is comparable to Amer in view of Liao and Shen’s gesture recognition. Therefore, it is within the capabilities of one of ordinary skill in the art to modify Amer in view of Liao and Shen’s gesture recognition to include Konecny’s one-shot gesture recognition with the predictable result of improved gesture recognition to “narrow the gap between human and machine performance” (section 8).
Regarding claim 9, Amer in view of Liao, Shen and Konecny teaches the medium of claim 8. Amer further teaches wherein the first portion of the text segment includes a plurality of words preceding a selection of words in the script provided by a user through a script authoring interface (paragraphs [0010], [0012], [0062], [0063], [0101], [0102], [105]). Amer describes generating animations for a text input phrase, which is analogous to a selection of words. Amer suggests the use of a human collaborator to modify scene description, which would include modifying or selecting the text input phrase, analogous to obtaining a selection of words in the script from the user. Amer further describes the use of a language parser to process text input and perform event extraction and gleaning context from descriptions, analogous to generation based on a plurality of words in the script preceding the selection of words.
Regarding claim 15, Amer in view of Liao, Shen and Konecny teaches The medium of claim 8. Amer further teaches the second vector generated on a second image of the user captured during the video stream (paragraphs [0051], [0056], [0070]). Amer describes encoding animation frames and input information as vectors.
Konecny further teaches wherein the first vector is generated based on a first image of the user performing the gesture captured during script authoring (section 3 paragraph 1, section 5.2 ). While Konecny doesn’t describe it as “script authoring”, the datasets used to conduct training/experiments contain videos depicting a user performing a gesture to be later recognized, in the same manner as the script authoring phase.
Claim(s) 10-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Amer in view of Liao, Shen, Konecny as applied to claim 8 and in further view of Lee.
Regarding claim 10, the combination of Amer in view of Liao, Shen and Konecny teach the medium of claim 9. Amer in view of Liao and Shen fail to teach wherein the script authoring interface allows the user to associate the plurality of states of the object included in the animation with a plurality of gestures to be performed in the video stream. However, Lee teaches wherein the script authoring interface allows the user to associate the plurality of states of the object included in the animation with a plurality of gestures to be performed in the video stream (paragraph [0059]). Lee describes a user terminal recommending a plurality of avatar motions (gestures) to be displayed which are associated with the plurality of states necessary to display the animated avatar (object). Lee is considered analogous to the claimed invention as it is in the same field of audio-based animation generation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Liao’s gesture and presentational object association to include Lee’s animation object and gesture association because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, Liao’s gesture and presentational object association and Lee’s animation object and gesture association perform the same general and predictable function, the predictable function being mapping control of content to a gesture. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of Liao’s gesture and presentational object association by replacing it with Lee’s animation object and gesture association. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Regarding claim 11, Amer in view of Liao, Shen, Konecny and in further view of Lee teaches the medium of claim 10. Amer further teaches wherein causing the animation to be displayed in the video stream further comprises advancing the animation to a first state of the object of the plurality of states of the object based on detecting the gesture in the video stream (paragraphs [0070], [0117], [0119]). Amer describes the generation of a composition graph based on a variety of input, including gestures detected via the behavior analytics system. Amer further describes animating based on the composition graph and the detected gestures. These animations necessitate a plurality of states which is reinforced by the mention of draws paths or trajectories for objects or actors in order to generate the animations.
Regarding claim 12, Amer in view of Liao, Shen, Konecny and Lee teaches the medium of claim 11. Lee further teaches wherein causing the animation to be displayed in the video stream further comprises advancing the animation to a second state of the object of the plurality of states of the object based on a first amount of time elapsed from displaying the animation and a second amount of time corresponding to the user speaking the text segment (paragraphs [0013], [0101], [0105]). Lee describes determining the time of a text string (text segment) in order to determine when to display an animation. Lee goes on to describes the changing of animation states based on whether or not the two animations are overlapping, in other words, based on the amount of time elapsed from the time point of the text string.
Regarding claim 13, Amer in view of Liao, Shen, Konecny and Lee teaches the medium of claim 11. Lee further teaches wherein causing the animation to be displayed in the video stream further comprises advancing the animation to a second state of the object of the plurality of states of the object based on detecting a second gesture of the plurality of gestures in the video stream (Amer, paragraphs [0070], [0117], [0119]; Lee paragraphs [0013], [0101], [0105])..
Regarding claim 14, Amer in view of Liao, Shen and Konecny teaches the medium of claim 8. Amer in view of Liao and Shen fail to teach wherein the plurality of states of the object are associated with a plurality of gestures.
However, Lee teaches wherein the plurality of states of the object are associated with a plurality of gestures (paragraphs [0013], [0101], [0105]). Lee describes a plurality of states associated with detected text strings which are associated with a plurality of avatar gestures. The motivation to combine Lee with Amer in view of Liao, Shen and Konecny would have been the same as that of claim 10.
Claim(s) 16-18, 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Amer in view of Lee, Liao and Konecny.
Regarding claim 16, Amer teaches a system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
obtaining a text segment include in a script and a first set of gesture parameters associated with a gesture to be performed by a user during a video stream, the text segment and the gesture associated with an animation (Figure 1c #124, #112, #113, paragraphs [0063], [0072]-[0074]);
obtaining the video stream and an audio stream corresponding to the video stream (paragraphs [0012], [0138]);
detecting the gesture performed by the user in the video stream (paragraph ) and determining the user speaking a portion of the text segment included in the script (paragraphs [0012] [0035] [0138]) and determining a similarity based on the first set of gesture parameters included in the first vector (paragraphs [0150]-[0152], [0157]) and a second set of gesture parameters included in a second vector generated based on the gesture performed by the user in the video stream (paragraphs [0036], [0073], [0119], [0135], [0137], [0152], [0070]); and
as a result of detecting the gesture, applying the animation to the video stream (paragraphs [0106]). As described before with regard to claim 1, Amer teaches a vector generated based on performance of a gesture, paragraph [0070] further describes “gestures” in plural form, implying multiple gestures or a set of gesture parameters. Further, a ranker encodes one or more from the data store, which can include or at least suggests the gesture vectors as described above, and ranks them based on their similarity scores to a video clip. While this process does not directly compare the two gesture vectors, it provides a mechanism to allow for comparison of the gesture vectors and the similarity can be considered to be at least based on the two vectors.
Amer fails to explicitly teach a gesture to be performed live by a user; detecting the gesture performed by the user in the video stream based on determining the user speaking a portion of the text segment included in the script; and where the set of gesture parameters are generated based on a second gestured included in a video captured during script authoring; and
determining a similarity between the first set of gesture parameters included in the first vector and a second set of gesture parameters included in a second vector generated based on the gesture performed by the user in the video stream; and as a result of the similarity being above a threshold, applying the animation to the video stream;
However, Lee teaches detecting the gesture in the video stream based on determining the user speaking a portion of the text segment included in the script (paragraph [0013]) and
and as a result of the similarity being above a threshold, applying the animation to the video stream (paragraphs [0082], [0094], [0101], [0102]).
Lee is considered analogous to the claimed invention as it is in the same field of applying video effects. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to implement the association of speech and gesture movement of Lee with the system of Amer to allow for association of multiple different input types.
Amer in view of Lee fails to teach a gesture to be performed live by a user; where the first set of gesture parameters are included in a first vector generated based on the gesture performed by a user included in a video captured during script authoring; determining a similarity between the first set of gesture parameters included in the first vector and a second set of gesture parameters included in a second vector generated based on the gesture performed by the user in the video stream.
However, Liao teaches a gesture to be performed live by a user (section 5.3, Figs 6, 8 & 9); where the first set of gesture parameters are generated based on the gesture performed by a user included in a video captured during script authoring (section 5, section 6.4). Liao describes obtaining visual elements associated with text segments during script authoring. These visual elements can be considered gestured parameters as they are the elements to be modified by the performed gestures. Liao is considered analogous to the claimed invention as it is in the same field of image processing, and live graphics processing. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Liao with Amer to improve the system of augmenting live video or presentations.
Amer in view of Lee and Liao fail to teach determining a similarity between the first set of gesture parameters included in the first vector and a second set of gesture parameters included in a second vector generated based on the gesture performed by the user in the video stream.
However, Konecny teaches determining a similarity between the first set of gesture parameters included in the first vector and a second set of gesture parameters included in a second vector generated based on the gesture performed by the user in the video stream. (abstract, sections 1 & 3 – the problem of one shot gesture recognition is analogous to determining a gesture during script authoring matches a gesture performed in presentation because both involve utilizing one representation of a first gesture to recognize a subsequent gesture, sections 5.2, 5.3, section 6.1, 6.2 – The distance of histograms can be considered analogous to a similarity score as it is representative of the difference between two gesture representations, which are based on optical flow vectors and contain gesture feature information, which can be considered analogous to gesture parameters). Konecny is considered analogous to the claimed invention as it is in the same field of gesture recognition. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the teachings of Konecny with Amer in view of Lee and Liao because such a modification is based on the use of known techniques to improve similar devices in the same way. More specifically, Konecny’s one-shot gesture recognition is comparable to Amer’s gesture recognition. Therefore, it is within the capabilities of one of ordinary skill in the art to modify Amer in view of Lee and Liao’s gesture recognition to include Konecny’s one-shot gesture recognition with the predictable result of improved gesture recognition which can “narrow the gap between human and machine performance” (section 8).
Regarding claim 17, Amer in view of Lee, Liao and Konecny teaches the system of claim 16. Lee further teaches wherein the operations further comprise determining a location in the script based on a portion of the audio stream and a script index indicating locations within the script and corresponding words in the script (paragraphs [101] & [105]). Lee describes time points of specific text strings in the input audio, this is analogous to a script index indicating locations within the script corresponding to words in the script.
Regarding claim 18 Amer in view of Lee, Liao and Konecny teach the system of claim 17. Lee further teaches wherein determining the user speaking the portion of the text segment included in the script further comprises determining the location in the script corresponds to the portion of the text segment (paragraph [105]). Lee describes utilizing time points associated with specific phrases to indicate avatar gestures.
Regarding claim 21, Amer in view of Lee, Liao and Konecny teach the system of claim 16. Liao further teaches where applying the animation to the video stream further comprises determining the user is speaking the portion of the text segment included in the script (Liao sections 2.3, 4.4 5.2.1, figs. 4 & 6). Liao describes keywords associated with visual elements as defined by authoring phase. These visual elements (animations) are displayed when the keyword (text segment included in script) is spoken by a user.
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Amer in view of Lee, Liao and Konecny as applied to claim 16 and in further view of Novikoff (US 2015/0058733 A1).
Regarding claim 19, Amer in view of Lee, Liao and Konecny teach the system of claim 16.
Konecny further teaches where detecting the gesture performed by the user in the video stream further comprises determining an action performed by the user based on a hand of the user being static (section 3 – “In every sequence, different gestures are separated by the user returning to a resting position.”, fig. 6 & desc. – “We add a new node – RP, or Resting Position – representing state where the user is not performing any particular gesture.”). The resting position is analogous to a hand of the user being static.
Amer in view of Lee, Liao and Konecny fails to teach an intentionality of an action
However, Novikoff teaches the system of claim 16, where detecting the gesture performed by the user in the video stream further comprises determining an intentionality of an action performed by the user based on a hand of the user being static (paragraph [0036], [0039]). Novikoff describes gestures including those that are performed by one or both of the user’s hands. Novikoff further describes the determination of intent, including basing the determination based on the speed and time of the gesture, which is analogous to determining intent based on a user’s hand being static (speed of 0). Novikoff is considered analogous to the claimed invention as it is in the same field of image processing and manipulation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to implement the intent detection of Novikoff with Amer in view of Lee, Liao and Konecny.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aidan W McCoy whose telephone number is (571)272-5935. The examiner can normally be reached 8:00 AM-5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AIDAN W MCCOY/ Examiner, Art Unit 2611
/TAMMY GODDARD/ Supervisory Patent Examiner, Art Unit 2611
