DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicants Amendments filed on April 30, 2026 has been entered and made of record.
Currently pending Claim(s) 1-22
Independent Claim(s) 1, 10, 17
Amended Claim(s) 1, 10, 17
Canceled Claim(s) 14, 16
New Claim(s) 22
Response to Arguments
This office action is responsive to the Applicant’s Arguments/Remarks Made in an Amendment received on April 30, 2026.
In view of amendments filed on April 30, 2026, the Applicant has amended independent Claim 1 to recite the additional imitation of “determining, by the processing device using the machine learning model, a size for the animated content relative to the object based on the type of the animated content”. The applicant has further amended Claim 1 by including the limitation of “determining, by the processing device using the machine learning model, a gesture to trigger display of the animated content”. Originally, (in the claim set dated February 25, 2026) Claim 1 recited “determining, by the processing device using the machine learning model, a time for displaying the animated content relative to a detected movement of the object”, and was rejected using Banica (US Pub No 2014/0359656) and Kundu (US Pub No 2023/0236660).
As discussed in the following paragraphs, the combination of Banica and Kundu does not render obvious the newly amended claims. However, the Applicant’s amendment necessitated the new grounds of rejection presented in this Office action. Upon conducting a new search, the Examiner argues that the newly amended Claim(s) 1 and 10 are unpatentable over Banica, Kundu, and Sun (US Pub No 2016/0048995), since Sun teaches determining the size of animated content relative to the type of animated content. Claim 17 remains unpatentable over Banica, Kundu, Nguyen (US Pub No 2021/0272363), and Sun.
In view of the Applicant’s Arguments/Remarks Made filed on April 30, 2026, the Applicant explained (on Remarks, page 10) that both Banica and Kundu fail, “to teach or suggest determining, by the processing device using the machine learning model, a size for the animated content relative to the object based on the type of the animated content; determining, by the processing device using the machine learning model”.
The Applicant explained (on Remarks, page 11) that Banica fails to teach determining a size of an animation. The Examiner agrees. Banica teaches displaying rendering the animated content within the frame of the digital video at a location, (see paragraph [0025], “Depending on the duration of an overlay, it can appear in frames spanning multiple clips or scenes of the video content”) and displaying the rendered animated content within the frame of the digital video in a user interface (see paragraph [0008], “The system inserts the overlay into the selected location and renders the video content with the inserted overlay on the display device”), but fails to explicitly teach determining a size of the overlay.
The Applicant further argued (on Remarks page 10) that Kundu, (who teaches displaying animations in response to the movements of a virtual actor), fails to teach the new limitation because, “Kundu describes inserting a representation into a scene, but not determining a size for the representation”.
The Examiner respectfully disagrees. Kundu teaches that a size of a representation can be determined relative to a detected object (see paragraph [0130], “In some examples, when combining more than one actor into a shared space, the distance between the imaging unit (e.g., a camera) of the actor can be used to modulate the scale of that actor when added into a local space of another actor. More specifically, the actor is made larger when coming closer to the camera and made smaller when moving further from the camera”, where the user is the detected object, and the size of the ‘actor representations’ are modulated with respect to the size of the user in the frame).
Neither Banica nor Kundu explicitly teaches determining a size for animated content relative to an object based on the type of animated content. However, Sun teaches a method of generating animations (see paragraph [0006], “The method includes: acquiring weather information and orientation information of a terminal device; generating a weather animation according to the weather information”), where the size of animated content may be determined based on the type of animated content (see paragraphs [0039-0040], “The above Step S21 includes determining the type of the weather element, the number of objects of each weather element, and the size of each object according to the weather type and the weather grade. For example, when the weather type is rainy and the weather grade is heavy rain, it is determined that the type of the weather element in the weather animation is rain drops, and the number of rain drops in the weather animation is 300. The size of each rain drop can be set”).
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the determination of animation size by type taught by Sun with the teachings of Banica and Kundu. The motivation for doing so would be to enhance the quality of the displayed animations (see Sun, paragraph [0009], “In this way, a plurality of weather conditions are comprehensively and dynamically presented via integrated motion pictures, and the weather conditions are displayed more accurately, realistically, and intuitively”). Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the newly claimed limitation.
The Applicant then argued (on Remarks, pages 11-12) that neither Banica nor Kundu teach “determining by the processing device using the machine learning model, a gesture to trigger display of the animated content, and displaying, by the processing device, the rendered animated content within the frame of the digital video in a user interface in response to detecting the gesture being performed by the object.” The Applicant explained that Banica fails to teach determining a gesture. The Examiner agrees. Banica teaches that user faces may be recognized (see paragraph [0117]), but fails to teach recognizing user movements or gestures. The Applicant then explained that Kundu “generally describes controlling a representation” and that Kundu teaches “interaction with an existing virtual object”, and not triggering a display of an animation.
The Examiner respectfully disagrees. Kundu teaches determining by the processing device using the machine learning model (see paragraph [0140], “The recognition of the motion or movement of the actor can be done using video recognition approaches, such as You Only Look Once (YOLO)”, where the user is the object, and YOLO is a well-known machine learning model),
a gesture to trigger display of the animated content (see paragraph [0099], “The system captures a frame of the video feed of the actual user in his physical room from the imaging unit… The system recognizes the pose or gesture of the user's hand as being an open hand indicative of the beginning of a “grab” gesture”, where the ‘grab’ gesture is the determined gesture to display animated content),
and displaying, by the processing device, the rendered animated content in response to detecting the gesture being performed by the object (see paragraph [0063], “see paragraph [0101], “the actual user proceeds to move his first to the right. The process proceeds as before. The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right”, where the grabbing and dragging gestures are interpreted as the gestures to trigger display of the movement of the animated object).
Kundu is combinable with Banica since both are from the analogous field of image analysis and animation. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine determining a time for displaying animations taught by Kundu with the animation generation method taught by Banica. The motivation to do so would be to provide the user with real-time animations of objects and scenes (see paragraph [0005], “The present disclosure provides system and method that enable the user(s) to interact in real-time with other objects or items in the scene or even with each other in the case of multiple users”).
Claim 10 has been amended to recite “determining, using a machine learning model, a gesture to trigger execution of the behavior”. The Applicant has further amended Claim 10 by including the limitation of “displaying the rendered animated content within the frame of the digital video in a user interface, in response to detecting the gesture being performed”. Originally, Claim 10 was rejected by Banica in view of Kundu.
Upon further consideration, the Examiner argues that the newly amended Claim 10 remains unpatentable over Banica, Kundu, and Sun. Kundu teaches determining, using a machine learning model, gesture to execution to trigger the behavior (behavior (see paragraph [0101], “The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right. The system displays the scene to the user, before the process loop proceeds to begin the next iteration", where the ‘object’ is the user, the ‘trigger’ is the ‘dragging’ gesture, and the behavior is the interactive object moving to the right).
Claim 10 has also been amended to include the size limitation recited in newly amended Claim 1. As discussed earlier, both Kundu and Sun teach determining the size of animated content. Thus, Claim 10 is rejected under Banica, Kundu, and Sun.
Claim 17 has been amended to recite “determining, using a machine learning model, an updated behavior of the animated content, including a trigger related to the object for execution of the behavior”. The applicant has further amended Claim 17 by including the limitation of “displaying the rendered animated content within the frame of the digital video in a user interface, including the updated behavior in response to detecting the trigger related to the object”. Originally, Claim 17 was rejected by Banica in view of Kundu and Nguyen (US Pub No 2021/0272363). Nguyen discusses an animation path that moves according a motion path sketch drawn by the user.
Upon further consideration, the Examiner argues that the newly amended Claim 17 remains unpatentable over Banica, Nguyen, Kundu, and Sun. Kundu teaches determining an updated behavior of the animated content, including a trigger related to the object of execution of the behavior (see paragraph [0101], “The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right. The system displays the scene to the user, before the process loop proceeds to begin the next iteration", where the ‘object’ is the user, and the user moves, the ‘trigger’ is the drag gesture, and the updated behavior is the interactive object moving to the right). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kundu with the teachings of Banica and Nguyen. The motivation for doing so would be to produce real-time animation effects, as taught by Kundu in paragraph [0005].
Claim 17 has also been amended to include the size limitation recited in newly amended Claim 1. As discussed earlier, both Kundu and Sun teach determining the size of animated content. Thus, Claim 17 is rejected under Banica, Nguyen, Kundu and Sun.
As to newly added Claim 22, the Applicant argued that Kundu fails to teach the new limitation because “interactions with a virtual object by a user is not the same as determining an updated location to move animated content”.
The Examiner respectfully disagrees. Kundu teaches obtaining a video frame with of an object (see paragraph [0099], “The system captures a frame of the video feed of the actual user in his physical room from the imaging unit… The system recognizes the pose or gesture of the user's hand as being an open hand indicative of the beginning of a “grab” gesture”),
and determining updated location to move the animated content based on detecting a change in location of the object within the frame of the digital video (see paragraph [0100], “ The system captures the video frame of the actual user in this pose. The system isolates the most recent user body image (i.e., with the closed fist)… the system recognizes the closed first (followed by the open hand previously) as indicative of a “grab” gesture”, and see paragraph [0101], “the actual user proceeds to move his first to the right. The process proceeds as before. The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right”, where the movement of the user to the right is recognized and then reflected in the animation). Thus, it would have been obvious to combine the teachings of Kundu with Banica and Sun in order to obtain the invention as claimed in Claim 22.
Thus, the Applicant’s amendment necessitated the new grounds of rejection presented in Office Action, and the independent Claims 1 and 10 are rejected over Banica, Kundu, and Sun (US Pub No 2016/0048995), since Sun teaches determining the size of animated effects relative to their type. Claim 17 is unpatentable over Banica, Kundu, Nguyen, and Sun (US Pub No 2016/0048995), Therefore, the rejection to the dependent claims 2-9, 11-13, 15, and 18-21 are maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-6, 8, 10-11, 13, 15, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Banica et al. (US Pub No 2014/0359656), hereinafter Banica in view of Kundu (US Pub No 2023/0236660), hereinafter Kundu, and further in view of Sun et al. (US Pub No US 20160048995), hereinafter Sun.
As to Claim 1, Banica claims a method (see paragraph [0007]) comprising:
receiving, by a processing device, a digital video and data executable to generate animated
content (see paragraph [0007], “method receiving, at a computing device, video content and an
indication of an overlay to be placed in the video content”),
detecting, by the processing device, an object depicted in a frame of the digital video (see paragraph [0118], “Certain of these embodiments use an attention model that detects the Attention Objects (AOs). Each AO is described by the following values: ROI (region of interest), AV (attention value), and MPS (minimal perceivable size). ROI indicates the area where the object is (and it is represented as a shape such as a rectangle)... The AV of each face takes into account the face size and position (larger faces and faces detected in the center of a frame have a higher AV));
determining, by the processing device using a machine learning model, (see paragraph [0008],
“the system performs attention modeling on frames of the video content to identify zones in the video content likely to be of interest to a viewer of the video content”, where attention modeling is an
example of a machine learning model),
a location within a frame of the digital video to place the animated content, (see paragraph
[0007], “Based at least in part on properties of the overlay and properties of the video content, the
method determines locations where the overlay can be placed within the video content”, where an
overlay can be multimedia content including animation),
rendering, by the processing device, the animated content within the frame of the digital video at the location, (see paragraph [0025], “Depending on the duration of an overlay, it can appear in frames spanning multiple clips or scenes of the video content”)
and displaying, by the processing device, the rendered animated content within the frame of the digital video in a user interface (see paragraph [0008], “The system inserts the overlay into the selected location and renders the video content with the inserted overlay on the display device”).
Banica fails to teach that the location indicates layering for the animated content based on a type of the animated content. Banica further fails to teach determining a size for the animated content relative to the object based on type of animated content, determining a gesture to trigger display of the animated content, and displaying the animated content in response to detecting the gesture being performed by the object.
However, in an analogous art Kundu teaches determining a location, including a layering assignment for the animated content relative to the object, based on a type of the animated content (see paragraph [0120], “In some examples, particular types of content may be prioritized to be assigned to the subject layer which is further in front and therefore less likely to be obstructed. By way of an example, text content can be prioritized to be assigned as the subject layer so that it is positioned in front of the actor layer, thereby not obstructed by the actor layer”),
and using a machine learning model (see paragraph [0140], “The recognition of the motion or movement of the actor can be done using video recognition approaches, such as You Only Look Once (YOLO)”),
to determine a gesture to trigger display of the animated content (see paragraph [0099], “The system captures a frame of the video feed of the actual user in his physical room from the imaging unit… The system recognizes the pose or gesture of the user's hand as being an open hand indicative of the beginning of a “grab” gesture”, wherein the ‘grab’ gesture is the gesture to trigger display),
and displaying, by the processing device, the rendered animated content in response to detecting the gesture being performed by the object (see paragraph [0101], “the actual user proceeds to move his first to the right. The process proceeds as before. The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right”, where the grabbing and dragging gestures are interpreted as the gestures to trigger display of the movement of the animated object).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to the layering and triggering of animations taught by Kundu with the teachings of Banica. The motivation to do so would be to provide the user with real-time animations of objects and scenes (see paragraph [0005], “The present disclosure provides system and method that enable the user(s) to interact in real-time with other objects or items in the scene or even with each other in the case of multiple users”). Thus, it would have been obvious to combine the teachings of Kundu with the teachings of Banica.
Banica fails to teach determining a size for the animated content relative to the object based on a type of the animated content, and rendering the content at the determined size. However, Kundu teaches determining a size for animated content relative to an object, and then rendering the content at the determined size see paragraph [0130], “In some examples, when combining more than one actor into a shared space, the distance between the imaging unit (e.g., a camera) of the actor can be used to modulate the scale of that actor when added into a local space of another actor. More specifically, the actor is made larger when coming closer to the camera and made smaller when moving further from the camera”, where the user is the detected object, and the size of the ‘actor representations’ are modulated with respect to the size of the user in the frame).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the size determination taught by Kundu with the teachings of Banica. The motivation for doing so would be to ensure that animations are proportional to objects in the frame. Kundu teaches in paragraph [0129], “By way of an example, when displaying more than one actors in the shared space, scaling and normalization of the image capturing the actors provides proportioning of the actors' image size captured in the video stream or the data feed such that they are similar in size on the display”).
Neither Banica nor Kundu teaches determining a size for animated content relative to an object based on the type of animated content. However, Sun teaches a method of generating animations (see paragraph [0006], “The method includes: acquiring weather information and orientation information of a terminal device; generating a weather animation according to the weather information”), where the size of animated content may be determined based on the type of animated content (see paragraphs [0039-0040], “The above Step S21 includes determining the type of the weather element, the number of objects of each weather element, and the size of each object according to the weather type and the weather grade. For example, when the weather type is rainy and the weather grade is heavy rain, it is determined that the type of the weather element in the weather animation is rain drops, and the number of rain drops in the weather animation is 300. The size of each rain drop can be set”, where the size of the animated weather element is determine based on the type of animated weather content).
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the determination of animation size by type taught by Sun with the teachings of Banica and Kundu. The motivation for doing so would be to enhance the quality of the animations (see Sun, paragraph [0009], “In this way, a plurality of weather conditions are comprehensively and dynamically presented via integrated motion pictures, and the weather conditions are displayed more accurately, realistically, and intuitively”). Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 1.
As to Claim 2, Banica in view of Kundu and Sun teaches a machine learning model (see Banica, “attention modeling” in paragraph [0008]) that determines the location based on a location of an object (see Banica, paragraph [0025], “The method and system determine non-obtrusive locations within the video that the provided overlays can be placed in. Non-obtrusiveness can be based on properties of the overlays and properties of frames of the video content. Depending on the duration of an overlay, it can appear in frames spanning multiple clips or scenes of the video content. Locations are determined to be non-obtrusive if an overlay having certain size, dimension, color, and/or translucency properties will not obscure or overlap with important objects in the frames”).
As to Claim 3, Banica in view of Sun fails to teach that rendering the animated content involves attaching a portion of the animated content to the object detected in the frame of the digital video. However, Kundu teaches that animated content can be ‘attached’ to an object in a video (see paragraph [0072], “Once the user's image is isolated, some embodiments of the system may use all or some of this image as the user representation (also referred to as “virtual user” or “virtual user representation”). In some embodiments, the virtual user representation may be entirely the “in real life” image of the physical user. In other embodiments, the virtual user representation may be in part the “in real life” image (for example, only the face) while other parts of the virtual user may be virtual (such as the body of an avatar) or augmented (such as wearing virtual clothing or holding virtual objects)”, where the clothes and objects are ‘attached’ to the actor in the frame of the digital video). Kundu and Banica are combinable because both are from analogous fields of image analysis and display. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kundu. The motivation for doing so would be to allow for the user to easily change the appearance of themselves in the digital video by attaching animations to the user. Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 3.
As to Claim 4, Banica in view of Sun teaches a machine learning model (see Banica, “attention modeling” in paragraph [0008]) , that determines the location based on tracking a person in the frame (see Banica, paragraph [0118], “These are deemed to attract more attention (although they may not be salient from a bottom-up perspective). The AV of each face takes into account the face size and position (larger faces and faces detected in the center of a frame have a higher AV)”, where the face of the detected person is tracked and deemed ‘salient’, and animated content is located away from ‘salient’ regions).
Banica fails to teach that the machine learning model determines the location based on tracking a pose of a detected person depicted in frame of the digital video. However, Kundu teaches that poses can be detected (see paragraph [0140], “In some examples, recognition of the movements or motions of the actor is used to cause interaction with the interactive object. The recognition of the motion or movement of the actor can be done using video recognition approaches, such as You Only Look Once (YOLO), a real-time object detection method, human pose estimation (including based on skeletal based, contour-based, and/or volume-based models), and the like. Similarly, the actor's body part (hand, foot, head, etc.) can be tracked by object recognition approaches such as YOLO, human pose estimation (including based on skeletal-based, contour-based, and/or volume-based models), etc.”),
and the locations of animations can be determined from the detected pose (see paragraph [0063], “if the physical actor is determined to be making a pinching gesture with their fingers as if holding a pen, then in step 1313 the system maps the physical location of the physical actor to corresponding virtual location of the virtual actor within the virtual three-dimensional space where the recognition of the pinching gesture (i.e., the pen gesture) causes the system in step 1313 to update the virtual three-dimensional space to show writing or drawing in the virtual location of the hand (or virtual pen being held by the hand) of the virtual actor in the virtual three-dimensional space”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kundu. The motivation for doing so would be to allow the user to interact with animated content without accidentally obscuring the content. Kundu teaches in paragraph [0004], “The present disclosure provides systems and methods that are useful for many situations where the actor needs to be on screen in realtime with other content, but in a way that does not overlap or occlude some objects in the virtual world. This is accomplished by inserting a representation of the actor into a “scene”. Such a scene is composed of content at a multitude of different levels or layers of depth where, from the point of view of the viewer, some of the content is behind the actor and some content is in front of the actor and therefore not occluded by the actor.” Thus, by recognizing poses and gestures, the animated content can be displayed in a way such that gestures done by a person in a video do not obscure the animated content. Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 4.
As to Claim 5, Banica in view of Kundu and Sun teaches a method wherein rendering the animated content further comprises auto-framing a face depicted in the frame of the digital video (see paragraph [0118] of Banica, “Besides saliency, the attention modeling also incorporates top-down, semantic information by identifying faces and text. These are deemed to attract more attention (although they may not be salient from a bottom up perspective)”). It is recognized that the citations and evidence provided above are derived from potentially different embodiments of a single reference. Nevertheless, it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to employ combinations and sub-combinations of these complementary embodiments, because Banica explicitly motivates doing so at least in paragraphs [0161] including “Embodiments of the methods disclosed herein may be performed in the operation of such computing devices. The order of the steps presented in the examples above can be varied—for example, steps can be re-ordered, combined, and/or broken into sub-steps. Certain steps or processes can be performed in parallel” and otherwise motivating combing the method for recognizing faces with the method of placing unobtrusive displays so that displays are not placed over the faces of actors in a digital video.
As to Claim 6, Banica fails to teach a portion of the animated content is layered behind or in front of the object depicted in the frame of the digital video based on determining whether the object depicted in the frame is a salient feature of the digital video. Banica teaches identifying salient features in videos (see paragraph [0118], “These are deemed to attract more attention (although they may not be salient from a bottom-up perspective). The AV of each face takes into account the face size and position (larger faces and faces detected in the center of a frame have a higher AV)”, where the face of the detected person is tracked and deemed ‘salient’, and animated content is located away from ‘salient’ regions), but fails to teach layering. However, Kundu teaches that animated content may be layered to prevent an object detected in the video from being obstructed (see paragraph [0123], “The position of the layers (i.e., the depth position within the layers, for example, a subject layer and a background layer) in a scene can be determined based on the location of the content within or on the layer. By way of an example, the content of the data toward the top of the scene is deprioritized and assigned to the background layer while content toward the middle or bottom of the scene is prioritized to be in the subject layer. This enables the actor to be able to stand and present content which is occluding the actor's lower and possibly middle section without the actor's head being obstructed by the content located higher up in the scene”, where the actor is a salient feature). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the salient object recognition taught by Banica with the layering taught by Kundu and Sun. The motivation for doing so would be to prevent important objects in the video from being obstructed, as taught in Kundu in paragraph [0123]. Thus, it would have been obvious to combine the layering taught by Kundu with the system taught by Banica and Sun in order to obtain the invention as claimed in Claim 6.
As to Claim 8, Banica fails to teach that a behavior of the animated content is triggered by a word or phrase detected in audio of the digital video. However, Kundu teaches animations can be triggered by a word or phrase (see paragraph [0140], “In some examples, the actor uses speech or sound to trigger an action in the scene. Specifically, the actor can say “next slide” to trigger the action of changing the scene or triggering an action from one slide to the next in a presentation and, similarly, “previous slide” to trigger the action of changing the scene from one slide to the previous slide in a presentation”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kundu. The motivation for doing so would be to allow the users to trigger actions and point out animated content without needing to use a computer mouse. Kundu teaches in paragraph [0007], “This is particularly problematic for remote presentations. Today, the actor's video stream is typically displayed in a window completely separate from the content, thereby making it more difficult for the actor to point out content (with anything other than a mouse pointer) and more effectively communicate with the audience.” Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 8.
As to Claim 10, Banica teaches a system comprising: a memory component and a processing device coupled to the memory component, (see paragraph [0008], “In another embodiment, a system has an input device, a display device, a processor, and a memory”),
receiving, by a processing device, a digital video and data executable to generate animated content (see paragraph [0007], “method receiving, at a computing device, video content and an indication of an overlay to be placed in the video content”),
using a machine learning model (see paragraph [0057], “ automated process implements overlay placement and transformation algorithms”)
to determine a behavior for a portion of the animated content (see paragraph [0140], “As described above with reference to FIGS. 1A and 1B, the transformations can include spatial (i.e., reshaping), color, translucency, transparency, and/or resizing transformations. In embodiments, step 912 can comprise applying two main types of effects so that an overlay fits better into video content received in step 902. The first type includes spatial transformations that move the corners of an overlay”)
and displaying, by the processing device, the rendered animated content within the frame of the digital video in a user interface (see paragraph [0008], “The system inserts the overlay into the selected location and renders the video content with the inserted overlay on the display device”).
Banica fails to teach determining a gesture to trigger execution of the behavior, and displaying the . However, Kundu teaches that a machine learning model, (see paragraph [0057], “In some examples, the extraction or isolation or segmentation of the actor can use chroma key green screen, virtual green screen technology, skeletal/human pose recognition/estimation, neural networks trained to isolate the actor within an image or video stream, and the like”)
that can recognize a gesture a trigger to execute a behavior including a specified movement for a portion of the animated content relative to the object, based on the digital video and the object (see paragraph [0099], “The system captures a frame of the video feed of the actual user in his physical room from the imaging unit… The system recognizes the pose or gesture of the user's hand as being an open hand indicative of the beginning of a “grab” gesture”, wherein the ‘grab gesture’ is the gesture to trigger movement of the animated content),
and then display the specified behavior of the animated content in response to the gesture (see paragraph [0101], “the actual user proceeds to move his first to the right. The process proceeds as before. The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right”, where the grabbing and dragging gestures are interpreted as the gestures to trigger display of the movement or ‘behavior’ of the animated object).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the timing determination taught by Kundu with the teachings of Banica. The motivation for doing so would be to produce real-time animation effects, as taught by Kundu in paragraph [0005]. Thus, it would have been obvious to combine the teachings of Banica and Kundu in order to obtain the invention as claimed in Claim 10.
Banica fails to teach determining a size for the animated content relative to the object based on a type of the animated content, and rendering the content at the determined size.
However, Kundu teaches determining a size for animated content relative to an object, and then rendering the content at the determined size see paragraph [0130], “In some examples, when combining more than one actor into a shared space, the distance between the imaging unit (e.g., a camera) of the actor can be used to modulate the scale of that actor when added into a local space of another actor. More specifically, the actor is made larger when coming closer to the camera and made smaller when moving further from the camera”, where the user is the detected object, and the size of the ‘actor representations’ are modulated with respect to the size of the user in the frame).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the size determination taught by Kundu with the teachings of Banica. The motivation for doing so would be to ensure that animations are proportional to objects in the frame. Kundu teaches in paragraph [0129], “By way of an example, when displaying more than one actors in the shared space, scaling and normalization of the image capturing the actors provides proportioning of the actors' image size captured in the video stream or the data feed such that they are similar in size on the display”).
Neither Banica nor Kundu teaches determining a size for animated content relative to an object based on the type of animated content. However, Sun teaches a method of generating animations (see paragraph [0006], “The method includes: acquiring weather information and orientation information of a terminal device; generating a weather animation according to the weather information”), where the size of animated content may be determined based on the type of animated content (see paragraphs [0039-0040], “The above Step S21 includes determining the type of the weather element, the number of objects of each weather element, and the size of each object according to the weather type and the weather grade. For example, when the weather type is rainy and the weather grade is heavy rain, it is determined that the type of the weather element in the weather animation is rain drops, and the number of rain drops in the weather animation is 300. The size of each rain drop can be set”, where the size of the animated weather element is determine based on the type of animated weather content).
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the determination of animation size by type taught by Sun with the teachings of Banica and Kundu. The motivation for doing so would be to enhance the quality of the animations (see Sun, paragraph [0009], “In this way, a plurality of weather conditions are comprehensively and dynamically presented via integrated motion pictures, and the weather conditions are displayed more accurately, realistically, and intuitively”). Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 10.
As to Claim 11, Banica in view of Sun fails to teach that determining the behavior is based on detected speech in the digital video. However, Kundu teaches that the behavior of an animation can be triggered by a spoken word or phrase (see paragraph [0140], “In some examples, the actor uses speech or sound to trigger an action in the scene. Specifically, the actor can say “next slide” to trigger the action of changing the scene or triggering an action from one slide to the next in a presentation and, similarly, “previous slide” to trigger the action of changing the scene from one slide to the previous slide in a presentation”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica and Sun with the teachings of Kundu. The motivation for doing so would be to allow users to interact with virtual educational content, and thus enhance the student’s learning experience as taught by Kundu in paragraph [0141].
As to Claim 13, Banica in view of Sun fails to teach that rendering the animated content further comprises removing a background from the frame of the digital video. However, Kundu teaches that the background can be removed (see paragraph [0071], “In step 1604 the system isolates a user representation from the video data received in Step 1602. In some embodiments, the isolation is done by means of human body segmentation or “selfie segmentation” as is commonly used by video conferencing software to isolate or segment the user's image from the image of their surrounding actual environment so as remove and replace it with a more desirable background”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kundu. The motivation for doing so would be to allow the user to replace the background with a more desirable background. Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 13.
As to Claim 15, Banica fails to teach a machine learning model determines the behavior-based tracking a pose of a detected person depicted in the frame of the digital video. However, Kundu teaches a machine learning model (see paragraph [0140], “cognition of the motion or movement of the actor can be done using video recognition approaches, such as You Only Look Once (YOLO)”, where YOLO is a well-known machine learning model)
that determines the behavior of an object based on a pose of the user (see paragraph [0102], “Regarding snapshot 2304, with the visual feedback that his virtual self has successfully dragged the virtual piece of candy 1208 to its desired final position, the actual user opens his fist. The image 2304 is displayed showing the user's virtual user representation 1312 having released the piece of candy (an interaction object 1208) in its final desired location. This is an example of a user's gesture/pose triggering a programmed action of an interactive object 1208”, where the movement of the animated candy is an example of a behavior based on tracking the pose of the user).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kundu. The motivation for doing so would be to allow users to easily interact with virtual objects in a scene. Kundu teaches in paragraph [0007], “This is particularly problematic for remote presentations. Today, the actor's video stream is typically displayed in a window completely separate from the content, thereby making it more difficult for the actor to point out content (with anything other than a mouse pointer) and more effectively communicate with the audience. If an actor is placed in front of content (such as by use of a chroma key green screen), the actor occludes some of the very content they are trying to present.” Thus, by tracking the location of an actor, behaviors of virtual objects can be programmed so that they are seen and not accidentally occluded. Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 15.
As to Claim 21, Banica in view of Sun fails to teach identifying portion of the object and determining, using the machine learning model, the location within the frame of the digital video to place the animated content that avoids covering the portion of the object. However Kundu teaches that a portion of the object can be identified(see paragraph [0128], “Once trained, the neural network is used as a classifier by which it can tag, in a binary matter, which regions of the image are most likely part of a human face or body parts”, where the object is the body of a user, and the face and body parts are portions of the user), and that animated content can be placed to avoid covering the portion of the object (see paragraph [0123], “The position of the layers (i.e., the depth position within the layers, for example, a subject layer and a background layer) in a scene can be determined based on the location of the content within or on the layer. By way of an example, the content of the data toward the top of the scene is deprioritized and assigned to the background layer while content toward the middle or bottom of the scene is prioritized to be in the subject layer. This enables the actor to be able to stand and present content which is occluding the actor's lower and possibly middle section without the actor's head being obstructed by the content located higher up in the scene”, where the actor’s head is the portion of the object that is not obstructed). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the timing determination taught by Kundu with the teachings of Banica and Sun. The motivation for doing would be to prevent the user’s face being obstructed by the animations, as taught Kundu in paragraph [0123].
As to Claim 22, Banica in view of Sun fails to teach determining an updated location to move the animated content based on detecting a change in location of the object within the frame of the digital video.
However Kundu teaches obtaining a frame of an object, (see paragraph [0099], “The system captures a frame of the video feed of the actual user in his physical room from the imaging unit… The system recognizes the pose or gesture of the user's hand as being an open hand indicative of the beginning of a “grab” gesture”, where the user is the ‘object’ in frame, and his gestures/movements are captured),
and determining updated location to move the animated content based on detecting a change in location of the object within the frame of the digital video (see paragraph [0100], “ The system captures the video frame of the actual user in this pose. The system isolates the most recent user body image (i.e., with the closed fist)… the system recognizes the closed first (followed by the open hand previously) as indicative of a “grab” gesture”, and see paragraph [0101], “the actual user proceeds to move his first to the right. The process proceeds as before. The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right”, where the movement of the user to the right is recognized and then reflected in the animation). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the change in location taught by Kundu with the teachings of Banica and Sun. The motivation for doing so would be to produce real-time animation effects, as taught by Kundu in paragraph [0005]. Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 22.
Claim(s) 9 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Banica et al. (US Pub No 2014/0359656), hereinafter Banica, in view of Kundu (US Pub No 2023/0236660), hereinafter Kundu, in view of Sun et al. (US Pub No US 20160048995), hereinafter Sun, and further in view of Nguyen (US Pub No US 2021/0272363) hereinafter Nguyen.
As to Claim 9, Banica fails to teach that the animated content comprises a behavior including a specified movement for a portion of the animated content selected by a user. However, Nguyen teaches that animated content can comprise a movement selected by a user (see paragraph [0007], “The video prototyping module can receive an input of a motion path sketch on the additional spatial layer, and the augmented reality feature of the spatial layer is assigned to the motion path sketch on the additional spatial layer. The augmented reality feature then displays as an animation that moves according to the motion path sketch during playback of the captured video”). Banica, Kundu, Sun and Nguyen are combinable because both are from analogous fields of image capture, analysis and display. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica, Kundu, and Sun with the teachings of Nguyen. The motivation for doing so would be to allow the user to control the movement of animated objects in an intuitive manner. Nguyen teaches in paragraph [0026], “The transformation of the movement in 3D space (as six-degrees-of-freedom movement) can be recorded as a motion path, which allows the designer to express complex motion trajectory in a more intuitive manner than with a traditional, spline-based interface.” Thus, it would have been obvious to combine the teachings of Banica and Nguyen in order to obtain the invention as claimed in Claim 9.
As to Claim 17, Banica teaches a non-transitory computer-readable storage medium storing executable instructions (see paragraph [0145], “For example, some functionality performed by client devices 134 a-n and server 104 shown in FIGS. 1A and 1B, can be implemented in the computer system 1000 using hardware, software, firmware, non-transitory computer readable media having instructions stored thereon”),
which when executed by a processing device, cause the processing device to perform operations comprising:
receiving, by a processing device, a digital video and data executable to generate animated content (see paragraph [0007], “method receiving, at a computing device, video content and an indication of an overlay to be placed in the video content”),
detecting an object depicted in a frame of the digital video (see paragraph [0118], “Certain of these embodiments use an attention model that detects the Attention Objects (AOs). Each AO is described by the following values: ROI (region of interest), AV (attention value), and MPS (minimal perceivable size). ROI indicates the area where the object is (and it is represented as a shape such as a rectangle)... The AV of each face takes into account the face size and position (larger faces and faces detected in the center of a frame have a higher AV)),
using a machine learning model (see paragraph [0057], “automated process implements overlay placement and transformation algorithms”),
to determine a behavior for a portion of the animated content (see paragraph [0140], “As described above with reference to FIGS. 1A and 1B, the transformations can include spatial (i.e., reshaping), color, translucency, transparency, and/or resizing transformations. In embodiments, step 912 can comprise applying two main types of effects so that an overlay fits better into video content received in step 902. The first type includes spatial transformations that move the corners of an overlay”)
and displaying, by the processing device, the rendered animated content within the frame of the digital video in a user interface (see paragraph [0008], “The system inserts the overlay into the selected location and renders the video content with the inserted overlay on the display device”).
However, Banica fails to teach receiving a behavior including a specified movement for a portion of the animated content, and determining an updated behavior based on the digital video and based on the object, and rendering the animated content within a frame of the digital video including the updated behavior determined by the machine learning mode. However, Nguyen teaches that specified behaviors can be received, and updated behaviors can be displayed (see paragraph [0007], “The video prototyping module can receive an input of a motion path sketch on the additional spatial layer, and the augmented reality feature of the spatial layer is assigned to the motion path sketch on the additional spatial layer. The augmented reality feature then displays as an animation that moves according to the motion path sketch during playback of the captured video”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Nguyen. The motivation for doing so would be to allow the user to control the movement of animated objects in an intuitive manner. Nguyen teaches in paragraph [0026], “The transformation of the movement in 3D space (as six-degrees-of-freedom movement) can be recorded as a motion path, which allows the designer to express complex motion trajectory in a more intuitive manner than with a traditional, spline-based interface.”
Banica in view of Nguyen fails to teach determining an updated behavior including a trigger related to the object for execution of the behavior. However, Kundu teaches that a machine learning model, (see paragraph [0057], “In some examples, the extraction or isolation or segmentation of the actor can use chroma key green screen, virtual green screen technology, skeletal/human pose recognition/estimation, neural networks trained to isolate the actor within an image or video stream, and the like”)
that can recognize a gesture a trigger to execute a behavior including a specified movement for a portion of the animated content relative to the object, based on the digital video and the object (see paragraph [0101], “The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right”, where the ‘drag gesture’ is the trigger, and ‘interactive object’ is the animated content performing a specified behavior of moving),
and then determine and display an updated behavior of the animated content in response to the gesture (see paragraph [0101], “the actual user proceeds to move his first to the right. The process proceeds as before. The system identifies the moving first as a “drag” gesture. The system therefore updates the scene this time to show interactive object 1208 being dragged to the right”, wherein the movement of the interactive object to the right is the ‘updated’ behavior).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the gesture and trigger determination taught by Kundu with the teachings of Banica. The motivation for doing so would be to produce real-time animation effects, as taught by Kundu in paragraph [0005]. Thus, it would have been obvious to combine the teachings of Banica and Kundu in order to obtain the invention as claimed in Claim 10.
Banica fails to teach determining a size for the animated content relative to the object based on a type of the animated content, and rendering the content at the determined size.
However, Kundu teaches determining a size for animated content relative to an object, and then rendering the content at the determined size see paragraph [0130], “In some examples, when combining more than one actor into a shared space, the distance between the imaging unit (e.g., a camera) of the actor can be used to modulate the scale of that actor when added into a local space of another actor. More specifically, the actor is made larger when coming closer to the camera and made smaller when moving further from the camera”, where the user is the detected object, and the size of the ‘actor representations’ are modulated with respect to the size of the user in the frame).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the size determination taught by Kundu with the teachings of Banica. The motivation for doing so would be to ensure that animations are proportional to objects in the frame. Kundu teaches in paragraph [0129], “By way of an example, when displaying more than one actors in the shared space, scaling and normalization of the image capturing the actors provides proportioning of the actors' image size captured in the video stream or the data feed such that they are similar in size on the display”).
Neither Banica nor Kundu teaches determining a size for animated content relative to an object based on the type of animated content. However, Sun teaches a method of generating animations (see paragraph [0006], “The method includes: acquiring weather information and orientation information of a terminal device; generating a weather animation according to the weather information”), where the size of animated content may be determined based on the type of animated content (see paragraphs [0039-0040], “The above Step S21 includes determining the type of the weather element, the number of objects of each weather element, and the size of each object according to the weather type and the weather grade. For example, when the weather type is rainy and the weather grade is heavy rain, it is determined that the type of the weather element in the weather animation is rain drops, and the number of rain drops in the weather animation is 300. The size of each rain drop can be set”, where the size of the animated weather element is determine based on the type of animated weather content).
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the determination of animation size by type taught by Sun with the teachings of Banica and Kundu. The motivation for doing so would be to enhance the quality of the animations (see Sun, paragraph [0009], “In this way, a plurality of weather conditions are comprehensively and dynamically presented via integrated motion pictures, and the weather conditions are displayed more accurately, realistically, and intuitively”). Thus, it would have been obvious to combine the teachings of Banica, Kundu, and Sun in order to obtain the invention as claimed in Claim 17.
As to Claim 18, Banica in view of Nguyen and Sun fails to teach herein the machine learning model determines the updated behavior based on a location of an object detected in the frame of the digital video. However, Kundu teaches that behavior of an animated object can be updated based on the location of an object (see paragraph [0063], “in step 1313, the system may also take as input and utilize the additional actor information so as to update the scene 1302 based on location, pose, or gestures of the actor 1301. For example, by utilizing location/depth information 1309 and/or pose & gesture information 1311, the positioning & updating step 1313 involves determining if and how the virtual actor makes contact with interactive objects within the three-dimensional scene so as to trigger corresponding actions. Upon actions being triggered, in step 1313, the system updates the three-dimensional scene accordingly to reflect the action (such as visual state change, etc.”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kundu. The motivation for doing so would be to allow users to easily interact with virtual objects in a scene. Kundu teaches in paragraph [0007], “This is particularly problematic for remote presentations. Today, the actor's video stream is typically displayed in a window completely separate from the content, thereby making it more difficult for the actor to point out content (with anything other than a mouse pointer) and more effectively communicate with the audience. If an actor is placed in front of content (such as by use of a chroma key green screen), the actor occludes some of the very content they are trying to present.” Thus, by tracking the location of an actor, behaviors of virtual objects can be programmed so that they are seen and not accidentally occluded. Thus, it would have been obvious to combine the teachings of Banica, Nguyen, Kundu, and Sun in order to obtain the invention as claimed in Claim 18.
As to Claim 19, Banica in view of Nguyen fails to teach that the machine learning model determines the updated behavior based on a location of an object detected in the frame of the digital video. However, Kundu teaches that the pose of an actor can be detected, (see paragraph [0127], “the system extracts an image of the actor from the video stream or the data feed using various techniques including, but not limited to, use of chroma key green screen, virtual green screen technology, skeletal/human pose recognition/estimation, neural networks trained to isolate the actor within the image”) and that behavior can be updated based on the detected pose (see paragraph [0063], “in step 1313, the system may also take as input and utilize the additional actor information so as to update the scene 1302 based on location, pose, or gestures of the actor 1301. For example, by utilizing location/depth information 1309 and/or pose & gesture information 1311, the positioning updating step 1313 involves determining if and how the virtual actor makes contact with interactive objects within the three-dimensional scene so as to trigger corresponding actions. Upon actions being triggered, in step 1313, the system updates the three-dimensional scene accordingly to reflect the action (such as visual state change, etc.”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kundu. The motivation for doing so would be to allow users to easily interact with virtual objects in a scene. Kundu teaches in paragraph [0007], “This is particularly problematic for remote presentations. Today, the actor's video stream is typically displayed in a window completely separate from the content, thereby making it more difficult for the actor to point out content (with anything other than a mouse pointer) and more effectively communicate with the audience. If an actor is placed in front of content (such as by use of a chroma key green screen), the actor occludes some of the very content they are trying to present.” Thus, by tracking the location of an actor, behaviors of virtual objects can be programmed so that they are seen and not accidentally occluded. Thus, it would have been obvious to combine the teachings of Banica, Nguyen, Kundu, and Sun in order to obtain the invention as claimed in Claim 19.
Claim(s) 7 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Banica et al. (US Pub No 2014/0359656), hereinafter Banica, in view of Kundu (US Pub No 2023/0236660), in view of Sun et al. (US Pub No US 20160048995), hereinafter Sun, and further in view of Kamemota et al. (US Pub No US 2024/0428484), hereinafter Kamemota.
As to Claim 7, Banica in view of Kundu and Sun fails to teach that the location is based on detected audio of the digital video. However, Kamemota teaches that the location of an animation caption can be based on the detected audio. (see paragraph [0062], “As for a character display position, an algorithm for determining a character display position having a predetermined positional relationship with a speaker is stored in the storage 9. The predetermined positional relationship is, for example, such a relationship that a distance to the speaker is within a certain range. As the predetermined positional relationship, a distance to the speaker is preferably close. By reducing the distance to the speaker, when utterance content is displayed as a caption, a viewer can easily specify the speaker of the caption”). Banica and Kamemota are combinable as both are from analogous fields of display of videos and animations. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica with the teachings of Kamemota. The motivation for doing so would be to align the caption to the speaker, and thus make it easier for the viewer to determine who is speaking. Thus, it would have been obvious to combine the teachings of Banica, Kundu, Sun, and Kamemota in order to obtain the invention as claimed in Claim 7.
As to Claim 12, Banica in view of Kundu and Sun fails to teach further comprising converting the detected speech into text for incorporation into the animated content. However, Kamemota teaches that speech can be converted into text and displayed (see paragraph [0135] and [0136], “It is assumed that the sound indicated by the sound signal includes a narration (speech) “Beautiful autumn leaves”. In this case, the sound data extraction processor 43 extracts the sound “Beautiful autumn leaves” using a sound recognition technique. The conversion processor 44 converts the sound extracted by the sound data extraction processor 43 into a caption signal. In this example, the conversion processor 44 converts data on the sound “Beautiful autumn leaves” obtained from the sound data extraction processor 43 into a caption signal. The conversion processor 44 inputs the converted caption signal to a caption feature extractor 6. A caption is displayed on the display 16 in the display form determined by the display form determiner 10.”). Kamemota further teaches that captions can appear as animations, (see paragraph [0107], “For example, when a category of content is drama, a caption is displayed at a position close to a person who speeches words of the caption as animation. As a result, the person who speeches is clearly identified and realistic sensations may become high. Furthermore, for example, when a category of content is variety show, a caption is moved, enlarged, or miniaturized as animation to enhance fun of a program”). Kamemota is combinable with Banica, Kundu, and Sun because all are from analogous fields of video display. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the method taught by Banica and Kundu with the teachings of Kamemota. The motivation for doing so would be to enhance the user’s experience by using animations to convey the atmosphere of the content. Kamemota teaches in paragraph [0107], “In addition to the example described here, an atmosphere of the content can be more easily given to the user who is the viewer by adding an appropriate animation to the caption depending on a category of the content.” Thus, it would have been obvious to combine the teachings of Banica, Kundu, Sun, and Kamemota in order to obtain the invention as claimed in Claim 12.
Claim(s) 20 is rejected under 35 U.S.C. 103 as being unpatentable over Banica et al. (US Pub No 2014/0359656), in view of Kundu (US Pub No 2023/0236660), in view of Nguyen (US Pub No 2021/0272363), in view of Sun et al. (US Pub No US 20160048995), hereinafter Sun, and further in view of Kamemota et al. (US Pub No US 2024/0428484).
As to Claim 20, Banica in view of Nguyen, Kundu, and Sun fails to teach that wherein the machine learning model determines the updated behavior based on audio of the digital video. However, Kamemota teaches that the audio in the scene can cause an updated behavior of animated captions (see paragraph [0115], “It is assumed that the caption feature extractor 6 extracts a caption of explanation "sound of raining". In this case, the display form determiner 10 replaces the caption "sound of raining" by a caption of characters "pouring" indicating sound with reference to the table illustrated in FIG. 8. for example… When "animation" is added to the caption of "pouring", the caption of "pouring" is moved on the display screen, for example”, where the updated movement would be moving the “pouring” caption onto the display, and where the caption feature extractor extracts the caption from the audio of rain in the digital video). Kamemota is combinable with Banica, Kundu, Nguyen, and Sun because all five are from analogous fields of image analysis and displaying animations. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Kamemota with the teachings of Banica, Kundu, Nguyen, and Sun. The motivation for doing so would be to provide a more realistic sensation. Kamemota teaches in paragraph [0117], “By this, since the explanation about sound is replaced by the characters representing the sound in display, realistic sensations may be given to the user.” Thus, it would have been obvious to combine the teachings of Kamemota with the teachings of Banica, Kundu, Nguyen, and Sun in order to obtain the invention as claimed in Claim 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tsurumi et al. (US Pub No 2014/0292809) teaches a method where an animation is added to an object detected within a frame of an image. The size of the animation is adjusted with respect to the size of the object.
Ruppert et al. (US Pub No 10754418) teaches a method where an animation is added to an object (the user’s hand) within the image. The size of the animations with respect to the user’s hand can be adjusted based on the type of animations displayed.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOUMYA THOMAS whose telephone number is (571)272-8639. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.T./Examiner, Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights