DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-30 are pending.
Information Disclosure Statement
The IDS filed 11/18/24 has been considered.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4-5, 10-17, 21, 26-27, and 29-30 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Danker et al. (US 20180184171 A1 Hereinafter “Danker”).
Regarding claim 1, Danker teaches a sender device comprising:
a memory configured to store an image frame of a video (Fig. 12, [0241]: “At least one computer-readable storage medium 1236 may include instructions that, when executed, cause a system to perform any of the computer-implemented methods described herein”. This computer-readable storage medium acts as a memory is used to store the video data which includes the image frame of the video. This is in the sender device due to this architecture being in a mobile device “The device 1200 may be, for example, an ultra-mobile device, a mobile device, a fixed device, a machine-to-machine (M2M) device, a personal digital assistant (PDA), a mobile computing device, a smart phone, a telephone” [0237]); and
one or more processors coupled to the memory (Fig. 12, [0236]: “Other platform components 1234 include common computing elements, such as one or more processors, multi-core processors, co-processors, memory units, chipsets, controllers,”. The processing unit can be seen coupled to the memory) and configured to:
obtain encoded image data representing the image frame (Fig. 6, [0111]: “At block 602, a system (such as a media effect server, a client device, or a communication server facilitating a video call or broadcast) may access a video stream. For example, the system may receive one or more frames associated with a video call or broadcast, such as when a video call application operating on a client mobile device receives one or more video frames from a camera of the mobile device or the frames are forwarded to a media effect server. The frames may include audio data received on an audio channel and/or video data received on a video channel”. Client device acts as the sending device (Fig. 4, #402-1)); and
send, to a recipient device during a video communication between the sender device and the recipient device (Fig. 4, [0087-0088]: “A sending client 402-1 may be a device associated with a first participant in a communication. The sending client 402-1 may be, for example a mobile device (although the present invention is not limited to application by mobile devices) executing a communications application 404-1 for participating in a video communication with one or more other participants. [0088] The communications application 404-1 may cause information associated with the video communication to be transmitted to one or more servers that facilitate the communication. For example, the information may include video data 408 containing video frames associated with the communication, audio data 412 containing sound information to be synchronized with the graphical frames, and control data 416. The control data 416 may include various instructions, identifiers, metadata, etc. used to apply media effect that are associated with (e.g., synchronized to) the video data 408 and the audio data 412”), a bitstream including the encoded image data as part of the video communication (Fig. 4, [0088]: “The communications application 404-1 may cause information associated with the video communication to be transmitted to one or more servers that facilitate the communication. For example, the information may include video data 408 containing video frames associated with the communication, audio data 412 containing sound information to be synchronized with the graphical frames, and control data 416. The control data 416 may include various instructions, identifiers, metadata, etc. used to apply media effect that are associated with (e.g., synchronized to) the video data 408 and the audio data 412”) and a modification indicator that indicates whether image modification is sender permitted or sender prohibited at the recipient device (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. The user can send instructions with the effects of if they are permitted or prohibited).
Regarding claim 2, Danker teaches the sender device of claim 1, wherein the bitstream further includes one or more feature indicators that indicate whether one or more modification features associated with the image frame are sender permitted or sender prohibited at the recipient device (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. The user can send instructions with the effects of if they are permitted or prohibited. An example of the effects can be seen in Fig. 1D with the eyes and mouth (120-1, 120-2, and 120-3). For determining which media effects may be applied, how and when they are applied, and by whom, modification features associated with the image frame must be permitted or prohibited by the sender and therefore part of the bitstream).
Regarding claim 4, Danker teaches the sender device of claim 1, wherein the one or more processors are configured to, based at least in part on determining that a modification feature is source permitted, selectively send, in the bitstream, a feature indicator indicating that the modification feature is sender permitted (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. The user can send instruction with the effects of if they are permitted or prohibited, if permitted, those feature indicators can be selectively sent allowing them to be used by the recipient).
Regarding claim 5, Danker teaches the sender device of claim 4, wherein the one or more processors are configured to:
analyze the image frame to determine whether a modification feature is performable on the image frame (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. If the circumstances for applying the media effects are defined, then the ability to be performable must have been determined); and
based on determining that the modification feature is performable on the image frame, selectively send, in the bitstream, the feature indicator indicating that the modification feature is sender permitted (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. If the conditions for performing the media effects are determined, the ability of the media effects to be performable has to have been determined, the feature indicator indicating the modification feature must be sent in the bitstream to allow use of the media effect).
Regarding claim 10, Danker teaches the sender device of claim 1, further comprising a camera configured to generate the image frame (Fig. 6, [0111]: “For example, the system may receive one or more frames associated with a video call or broadcast, such as when a video call application operating on a client mobile device receives one or more video frames from a camera of the mobile device or the frames are forwarded to a media effect server”).
Regarding claim 11, Danker teaches the sender device of claim 1, further comprising a display device configured to display a graphical user interface (GUI), wherein a value of the modification indicator is based on a user-selectable option of the GUI ([0112]: “At block 604, the system may receive an instruction to apply a media effect. The instruction may be an explicit instruction, such as when an instruction is received by the media effect server on the control channel. When performed locally at a client device, the instruction may be received as an input from a user (e.g., by interacting with a touch display or issuing a voice instruction)”. Touch display would include a GUI).
Regarding claim 12, Danker teaches the sender device of claim 1, wherein the memory and the one or more processors are integrated in a portable device ([0087]: “A sending client 402-1 may be a device associated with a first participant in a communication. The sending client 402-1 may be, for example a mobile device (although the present invention is not limited to application by mobile devices) executing a communications application 404-1 for participating in a video communication with one or more other participants”. Memory and processors are found in the mobile device to be able to perform all the operations).
Regarding claim 13, the content of claim 13 is similar to the content of claim 1, therefore it is rejected for the same reasons of anticipation as claim 1.
Regarding claim 14, the content of claim 14 is similar to the content of claim 2, therefore it is rejected for the same reasons of anticipation as claim 2.
Regarding claim 15, Danker teaches the recipient device comprising:
a memory configured to store encoded image data that represents an image frame of a video (Fig. 10, [0214]: “As shown in FIG. 10, the computing architecture 1000 comprises a processing unit 1002, a system memory 1004”. This memory is used to store video data which includes the image frame of the video); and
one or more processors coupled to the memory (Fig. 10, [0214]: “As shown in FIG. 10, the computing architecture 1000 comprises a processing unit 1002, a system memory 1004”. The processing unit can be seen coupled to the memory) and configured to:
obtain a modification indicator that indicates whether image modification is sender permitted or sender prohibited, the modification indicator obtained from a bitstream received from a sender device during a video communication between the sender device and the recipient device, wherein the bitstream also includes the encoded image data as part of the video communication (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. The user can send instructions with the effects of if they are permitted or prohibited. This is received by the media effect server which determines the context for the image and whether the user permits the media effect to apply); and
selectively generate a modified image frame at least partially based on whether the image modification is sender permitted or sender prohibited, the modified image frame at least partially based on the encoded image data (Fig. 6, [0135]: “At block 620, the system may add the media effect to the audiovisual data of the video stream. For example, the system may access video data on the video channel and audio data on the audio channel, incorporate the media effect into the audio and/or video data, and synchronize the data for transmission to the broadcasting server. At block 622, the system may transmit the resulting modified audiovisual data towards its destination (e.g., to receiving client devices or a broadcast server)”. The media effect is then used on the audio/visual data based on the sender permissions).
Regarding claim 16, Danker teaches the recipient device of claim 15, wherein the one or more processors are configured to, based on a determination that the modification indicator indicates that image modification is sender permitted, that a feature indicator indicates that a modification feature associated with the image frame is sender permitted, that the modification feature is recipient permitted, and that the modification feature is performable on a decoded image frame, apply the modification feature to the decoded image frame to generate a modified image frame, the decoded image frame based on the encoded image data, wherein the feature indicator is obtained from the received bitstream (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. The user can send instructions with the effects of if they are permitted or prohibited. An example of the effects can be seen in Fig. 1D with the eyes and mouth (120-1, 120-2, and 120-3). For determining which media effects may be applied, how and when they are applied, and by who, modification features associated with the image frame must be permitted or prohibited by the sender and therefore part of the bitstream. If the conditions for performing the media effects are determined, the ability of the media effects to be performable has to have been determined, the feature indicator indicating the modification feature must be sent in the bitstream to allow use of the media effect. Once the permission and performability are determined, the image data can be directly affected by the media effects “At block 620, the system may add the media effect to the audiovisual data of the video stream. For example, the system may access video data on the video channel and audio data on the audio channel, incorporate the media effect into the audio and/or video data, and synchronize the data for transmission to the broadcasting server. At block 622, the system may transmit the resulting modified audiovisual data towards its destination (e.g., to receiving client devices or a broadcast server)” [0135]).
Regarding claim 17, Danker teaches the recipient device of claim 15, wherein the one or more processors are configured to decode the encoded image data to generate a decoded image frame, the modified image frame based on the decoded image frame (Fig. 6, [0135]: “At block 620, the system may add the media effect to the audiovisual data of the video stream. For example, the system may access video data on the video channel and audio data on the audio channel, incorporate the media effect into the audio and/or video data, and synchronize the data for transmission to the broadcasting server. At block 622, the system may transmit the resulting modified audiovisual data towards its destination (e.g., to receiving client devices or a broadcast server)”. The audio/visual data must be decoded since it is received in an encoded state, to use media effects on the data).)
Regarding claim 21, Danker teaches the recipient device of claim 17, wherein the one or more processors are configured to:
analyze the decoded image frame to determine whether a modification feature is performable on the decoded image frame ([0092]: “The media effect server 418 may be configured to analyze the video data 408 and/or audio data 412 using, e.g., emotion detection logic 420 for detecting an emotional state associated with the video data 408 and/or audio data 412, object recognition logic 432 for identifying objects in the video, facial recognition logic 434 for identifying faces in the video, and other types of logic for analyzing the content of the video data 408 and/or the audio data 412” The media effect server analyzes the decoded image frame, and to apply the media effect to the video that was sent from the sender, there must be a determination of it being performable or else it would not be performed); and
based on determining that the modification feature is sender permitted and that the modification feature is performable on the decoded image frame, apply the modification feature to the decoded image frame to generate the modified image frame (Fig. 6, [0135]: “At block 620, the system may add the media effect to the audiovisual data of the video stream. For example, the system may access video data on the video channel and audio data on the audio channel, incorporate the media effect into the audio and/or video data, and synchronize the data for transmission to the broadcasting server. At block 622, the system may transmit the resulting modified audiovisual data towards its destination (e.g., to receiving client devices or a broadcast server)”. The media effect is then used on the audio/visual data based on the sender’s permissions and performability).
Regarding claim 26, Danker teaches the recipient device of claim 15, further comprising a modem configured to receive the bitstream (Fig. 10, [0224]: “When used in a WAN networking environment, the computer 1001 can include a modem 1054, or is connected to a communications server on the WAN 1050, or has other means for establishing communications over the WAN 1050, such as by way of the Internet”).
Regarding claim 27, Danker teaches the recipient device of claim 15, further comprising a display device configured to display the modified image frame (Fig. 10, [0221]: “A monitor 1042 or other type of display device is also connected to the system bus 1006 via an interface, such as a video adaptor 1044”).
Regarding claim 29, the content of claim 29 is similar to the content of claim 15, therefore it is rejected for the same reasons of anticipation as claim 15.
Regarding claim 30, the content of claim 30 is similar to the content of claim 16, therefore it is rejected for the same reasons of anticipation as claim 16.
Claim 9 is rejected under 35 U.S.C. 102(a)(1) as being unpatentable over Danker et al. (US 20180184171 A1 Hereinafter “Danker”) as evidence by Mobile (“Understanding Cellular Modem Specifications (Mobile Routers, Hotspots, Phones)” Hereinafter “Mobile”).
Regarding claim 9, Danker teaches the sender device of claim 1, (Fig. 12, [0237]: “The device 1200 may be, for example, an ultra-mobile device, a mobile device, a fixed device, a machine-to-machine (M2M) device, a personal digital assistant (PDA), a mobile computing device, a smart phone, a telephone, a digital telephone, a cellular telephone, user equipment”).
The cellular telephone the sender device can be as listed, contains a cellular modem as evidence by Mobile (Page 2, last paragraph: “Inside every cellular device is a component known as a cellular modem, and the specifications and capabilities of that modem can make an absolutely huge difference in your mobile internet experience”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 6, 7, 18, 19, 28 are rejected under 35 U.S.C. 103 as being unpatentable over Danker et al. (US 20180184171 A1 Hereinafter “Danker”) in view of GARRIDO et al. (WO 2019216988 A1 Hereinafter “GARRIDO”).
Regarding claim 3, Danker teaches the sender device of claim 1, wherein the one or more processors are configured to perform facial landmark detection on the image frame to generate facial landmark data ([0058]: “Optionally, one or more axes of the bounding shape 116 may be defined. In the depicted example, the facial detection technique involves identifying features of the face, such as the ears, eyes, nose, and mouth. A vertical axis extends vertically through the approximate center of the face, splitting the eyes, nose, and mouth in half. A horizontal axis extends horizontally through the approximate center of the face, approximately through the ears and eyes”),
While it is presumed that the bitstream data would include the facial landmark data since the media effects are determined by the emotional state, and the emotional state is determined by the facial features detected ([0059]: “The face may also be analyzed to identify especially expressive features, which may be particularly relevant to emotion detection”), the bitstream should contain facial landmark data used to determine the emotional state), Danker does not expressly disclose wherein the bitstream includes the facial landmark data.
However, GARRIDO teaches including the facial landmark data in bitstream data for video conferencing ([0057]: “Augmented-reality (AR) metadata may include information that describes physical objects such as people, facial features, animals, trees, bushes, flowers, streets, street signs, furniture or other physical objects that are represented in the video data image frames. For example the AR metadata may include depth information, size information, pixel location information, motion information, orientation information or other information about the physical objects. Content merging engine 406 may use the AR metadata to position, orient, size, and/or adapt, over time, the supplemental content that is added to the video stream from the camera”. This data is merged for encoding and sending to another device to decode).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects permits additional data which helps applying video effects to the transmitted video ([0057]: “the AR metadata may indicate the boundaries, location, size, depth, and features (e.g., the locations of the eyes, ears, mouth, etc.) of the user’s face, which can be used to size, locate, orient, and determine the facial expression of the animated animal face, as well as to animate the animated animal face to correspond to changes in the depth at which the user’s face is imaged, the orientation or location of the imaged face, and/or the facial expression of the user”). This known benefit in GARRIDO is applicable to Danker’s video stream system as they both share characteristics and capabilities, namely, they are directed to altering video streams to add effects to the stream. Therefore, it would have been recognized that modifying Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects in altering video streams to add effects to the stream and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 6, Danker teaches the sender device of claim 5, wherein the one or more processors are configured to, based at least in part on determining that the modification feature is performable on the image frame (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. If the conditions for performing the media effects are determined, the ability of the media effects to be performable has to have been determined, the feature indicator indicating the modification feature must be sent in the bitstream to allow use of the media effect),
While it is presumed that the bitstream data would include the facial landmark data since the media effects are determined by the emotional state, and the emotional state is determined by the facial features detected ([0059]: “The face may also be analyzed to identify especially expressive features, which may be particularly relevant to emotion detection”), the bitstream should contain facial landmark data used to determine the emotional state), Danker does not expressly disclose wherein the bitstream includes the facial landmark data.
However, GARRIDO teaches including the facial landmark data in bitstream data for video conferencing ([0057]: “Augmented-reality (AR) metadata may include information that describes physical objects such as people, facial features, animals, trees, bushes, flowers, streets, street signs, furniture or other physical objects that are represented in the video data image frames. For example the AR metadata may include depth information, size information, pixel location information, motion information, orientation information or other information about the physical objects. Content merging engine 406 may use the AR metadata to position, orient, size, and/or adapt, over time, the supplemental content that is added to the video stream from the camera”. This data is merged for encoding and sending to another device to decode).
The rationale for combination is similar to the rationale mentioned in the claim 3 due combination to similar methods of combination (including facial feature data in bitstream) and benefits (assisting with applying media effects).
Regarding claim 7, Danker teaches the sender device of claim 1, wherein the one or more processors are configured to:
While it is presumed that the bitstream data would include the facial landmark data since the media effects are determined by the emotional state, and the emotional state is determined by the facial features detected ([0059]: “The face may also be analyzed to identify especially expressive features, which may be particularly relevant to emotion detection”), the bitstream should contain facial landmark data used to determine the emotional state), Danker does not expressly disclose wherein the bitstream includes the facial landmark data.
However, GARRIDO teaches including the facial landmark data in bitstream data for video conferencing ([0057]: “Augmented-reality (AR) metadata may include information that describes physical objects such as people, facial features, animals, trees, bushes, flowers, streets, street signs, furniture or other physical objects that are represented in the video data image frames. For example the AR metadata may include depth information, size information, pixel location information, motion information, orientation information or other information about the physical objects. Content merging engine 406 may use the AR metadata to position, orient, size, and/or adapt, over time, the supplemental content that is added to the video stream from the camera”. This data is merged for encoding and sending to another device to decode) and second facial landmark data based on detection of a deviation of the second facial landmark data from the first facial landmark data that is greater than threshold deviation ([0057]: “the AR metadata may indicate the boundaries, location, size, depth, and features (e.g., the locations of the eyes, ears, mouth, etc.) of the user’s face, which can be used to size, locate, orient, and determine the facial expression of the animated animal face, as well as to animate the animated animal face to correspond to changes in the depth at which the user’s face is imaged, the orientation or location of the imaged face, and/or the facial expression of the user”. To determine the change in depth at which the user face is imaged a second set of features must be captured and the difference must be bigger than a threshold to determine that there was a change in depth, this would be so the system could operate in a live format (updates consistently) “At block 906, the receiving device displays the modified live incoming video stream with the supplemental content (e.g., in the video conferencing application). In some scenarios, the receiving device may also provide, for display, a supplemental content option together with the modified live incoming video stream with the supplemental content” [0088]. Additionally video conferencing is described as real time (updates consistently) by the authors “In this way, users in remote locations can talk, in real time, while viewing incoming video streams of the other user or users’ faces, to mimic an in- person conversation or conference” [0024]).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data permits additional data which helps applying video effects to the transmitted video and updating the features when changes are detected ([0057]: “the AR metadata may indicate the boundaries, location, size, depth, and features (e.g., the locations of the eyes, ears, mouth, etc.) of the user’s face, which can be used to size, locate, orient, and determine the facial expression of the animated animal face, as well as to animate the animated animal face to correspond to changes in the depth at which the user’s face is imaged, the orientation or location of the imaged face, and/or the facial expression of the user”). This known benefit in GARRIDO is applicable to Danker’s video stream system as they both share characteristics and capabilities, namely, they are directed to altering video streams to add effects to the stream. Therefore, it would have been recognized that modifying Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data in altering video streams to add effects to the stream and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 18, Danker teaches the recipient device of claim 17, wherein the one or more processors are configured to
While it is presumed that the bitstream data would include the facial landmark data since the media effects are determined by the emotional state, and the emotional state is determined by the facial features detected ([0059]: “The face may also be analyzed to identify especially expressive features, which may be particularly relevant to emotion detection”), the bitstream should contain facial landmark data used to determine the emotional state), Danker does not expressly disclose wherein the bitstream includes the facial landmark data
However, GARRIDO teaches including the facial landmark data in bitstream data for video conferencing and using the facial landmark data to apply media effects ([0057]: “Augmented-reality (AR) metadata may include information that describes physical objects such as people, facial features, animals, trees, bushes, flowers, streets, street signs, furniture or other physical objects that are represented in the video data image frames. For example the AR metadata may include depth information, size information, pixel location information, motion information, orientation information or other information about the physical objects. Content merging engine 406 may use the AR metadata to position, orient, size, and/or adapt, over time, the supplemental content that is added to the video stream from the camera”. This data is merged for encoding and sending to another device to decode).
The rationale for combination is similar to the rationale mentioned in claim 3 due combination due to similar methods of combination (including facial feature data in bitstream for use in applying media effects) and benefits (assisting with applying media effects).
Regarding claim 19, the combination of Danker and GARRIDO teaches the recipient device of claim 18, in addition, GARRIDO further teaches wherein the facial landmark data is received in the bitstream ([0057]: “Augmented-reality (AR) metadata may include information that describes physical objects such as people, facial features, animals, trees, bushes, flowers, streets, street signs, furniture or other physical objects that are represented in the video data image frames. For example the AR metadata may include depth information, size information, pixel location information, motion information, orientation information or other information about the physical objects. Content merging engine 406 may use the AR metadata to position, orient, size, and/or adapt, over time, the supplemental content that is added to the video stream from the camera”. This data is merged for encoding and sending to another device to decode. The decoding of the data means the facial landmark data (features) were received by the bitstream).
The rationale for combination is similar to the rationale mentioned in the claim 3 due combination due to similar methods of combination (including facial feature data in bitstream for use in applying media effects) and benefits (assisting with applying media effects).
Regarding claim 28, Danker teaches the recipient device of claim 15,
Danker does not expressly disclose the recipient device being a portable device.
However, GARRIDO teaches the recipient device being a portable device (Fig. 4, [0032]: “The electronic device 115 may include a touchscreen and may be, for example, a portable computing device such as a laptop computer that includes a touchscreen, a smartphone that includes a touchscreen, a peripheral device that includes a touchscreen (e.g., a digital camera, headphones)”).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Danker’s video stream system to include GARRIDO’s use of a portable device as a recipient device because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, GARRIDO’s use of a portable device as a recipient device permits improved mobility of the user by allowing the integrated system to be portable. This known benefit in GARRIDO is applicable to Danker’s video stream system as they both share characteristics and capabilities, namely, they are directed to altering video streams to add effects to the stream. Therefore, it would have been recognized that modifying Danker’s video stream system to include GARRIDO’s use of a portable device as a recipient device would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate GARRIDO’s use of a portable device as a recipient device in altering video streams to add effects to the stream and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Danker et al. (US 20180184171 A1 Hereinafter “Danker”) in view of GARRIDO et al. (WO 2019216988 A1 Hereinafter “GARRIDO”) in further view of Gordiichuk et al. (US 11302156 B1 Hereinafter “Gordiichuk”).
Regarding claim 8, Danker teaches the sender device of claim 1, wherein the one or more processors are configured to selectively send, in the bitstream, (Fig. 4, [0088]: “The communications application 404-1 may cause information associated with the video communication to be transmitted to one or more servers that facilitate the communication. For example, the information may include video data 408 containing video frames associated with the communication, audio data 412 containing sound information to be synchronized with the graphical frames, and control data 416. The control data 416 may include various instructions, identifiers, metadata, etc. used to apply media effect that are associated with (e.g., synchronized to) the video data 408 and the audio data 412”).
While it is presumed that the bitstream data would include the facial landmark data since the media effects are determined by the emotional state, and the emotional state is determined by the facial features detected ([0059]: “The face may also be analyzed to identify especially expressive features, which may be particularly relevant to emotion detection”), the bitstream should contain facial landmark data used to determine the emotional state), Danker does not expressly disclose wherein the bitstream includes the facial landmark data.
However, GARRIDO teaches including the facial landmark data in bitstream data for video conferencing ([0057]: “Augmented-reality (AR) metadata may include information that describes physical objects such as people, facial features, animals, trees, bushes, flowers, streets, street signs, furniture or other physical objects that are represented in the video data image frames. For example the AR metadata may include depth information, size information, pixel location information, motion information, orientation information or other information about the physical objects. Content merging engine 406 may use the AR metadata to position, orient, size, and/or adapt, over time, the supplemental content that is added to the video stream from the camera”. This data is merged for encoding and sending to another device to decode) and second facial landmark data based on detection of a deviation of the second facial landmark data from the first facial landmark data that is greater than threshold deviation ([0057]: “the AR metadata may indicate the boundaries, location, size, depth, and features (e.g., the locations of the eyes, ears, mouth, etc.) of the user’s face, which can be used to size, locate, orient, and determine the facial expression of the animated animal face, as well as to animate the animated animal face to correspond to changes in the depth at which the user’s face is imaged, the orientation or location of the imaged face, and/or the facial expression of the user”. To determine the change in depth at which the user face is imaged a second set of features must be captured and the difference must be bigger than a threshold to determine that there was a change in depth, this would be so the system could operate in a live format (updates consistently) “At block 906, the receiving device displays the modified live incoming video stream with the supplemental content (e.g., in the video conferencing application). In some scenarios, the receiving device may also provide, for display, a supplemental content option together with the modified live incoming video stream with the supplemental content” [0088]. Additionally video conferencing is described as real time (updates consistently) by the authors “In this way, users in remote locations can talk, in real time, while viewing incoming video streams of the other user or users’ faces, to mimic an in- person conversation or conference” [0024]).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data permits additional data which helps applying video effects to the transmitted video and updating the features when changes are detected ([0057]: “the AR metadata may indicate the boundaries, location, size, depth, and features (e.g., the locations of the eyes, ears, mouth, etc.) of the user’s face, which can be used to size, locate, orient, and determine the facial expression of the animated animal face, as well as to animate the animated animal face to correspond to changes in the depth at which the user’s face is imaged, the orientation or location of the imaged face, and/or the facial expression of the user”). This known benefit in GARRIDO is applicable to Danker’s video stream system as they both share characteristics and capabilities, namely, they are directed to altering video streams to add effects to the stream. Therefore, it would have been recognized that modifying Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects alongside providing updates facial feature data in altering video streams to add effects to the stream and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
The combination of Danker and GARRIDO does not expressly disclose determining the video frame is an intraframe.
However, Gordiichuk teaches the determination of the image frame being an intraframe for data conversion (Col. 38, lines 10-25: “In some examples, the server may determine if the I-Frame is capable of being converted to the data format necessary for transmitting to the image to the client device 214, 216. For example, the server(s) 224 may process the I-Frame to determine if the I-Frame can be converted to the data format. If the I-Frame cannot be converted (e.g., because data packet(s) are lost, data packet(s) are incomplete or broken, etc.), the server(s) 224 may perform similar functions on the next I-Frame in the image data received from the A/V device 210. This process may be performed until a suitable I-Frame is determined that can be converted the data format”. By determining if the I-frame is suitable of being converted, there must have been a determination that the frame is an I-frame)
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify the combination of Danker and GARRIDO’s video stream system to include Gordiichuk’s determination of the image frame being an intraframe for data conversion because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically Gordiichuk’s determination of the image frame being an intraframe for data conversion permits determining of an I-frame that is suitable for data conversion (such as encoding in a similar manner to the combination of Danker and GARRIDO’s video stream system). This known benefit in Gordiichuk is applicable to the combination of Danker and GARRIDO’s video stream system as they both share characteristics and capabilities, namely, they are directed to transmitting video data between devices. Therefore, it would have been recognized that modifying the combination of Danker and GARRIDO’s video stream system to include Gordiichuk’s determination of the image frame being an intraframe for data conversion would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Gordiichuk’s determination of the image frame being an intraframe for data conversion would have yielded predictable results in transmitting video data between devices and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Danker et al. (US 20180184171 A1 Hereinafter “Danker”) in view of GARRIDO et al. (WO 2019216988 A1 Hereinafter “GARRIDO”) in further view of Li et al. (US 20160005206 A1 Hereinafter “Li”).
Regarding claim 20, the combination of Danker and GARRIDO teaches the recipient device of claim 18, in addition, GARRIDO further teaches wherein the one or more processors are configured to, based on a determination that a modification feature is sender permitted and that corresponding facial landmark data is
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects permits additional data which helps applying video effects to the transmitted video ([0057]: “the AR metadata may indicate the boundaries, location, size, depth, and features (e.g., the locations of the eyes, ears, mouth, etc.) of the user’s face, which can be used to size, locate, orient, and determine the facial expression of the animated animal face, as well as to animate the animated animal face to correspond to changes in the depth at which the user’s face is imaged, the orientation or location of the imaged face, and/or the facial expression of the user”). This known benefit in GARRIDO is applicable to Danker’s video stream system as they both share characteristics and capabilities, namely, they are directed to altering video streams to add effects to the stream. Therefore, it would have been recognized that modifying Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects in altering video streams to add effects to the stream and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
The combination of Danker and GARRIDO does not expressly disclose generating facial feature data when it is determined there is no facial feature data present.
However, Li teaches generating facial feature data when it is determined there is no facial feature data present ([0026]: “Generating the avatar animation locally on the client device may involve applying a smoothing process to the frames indicated as lacking facial motion data to obtain replacement facial motion data”. This facial motion data includes facial features “The facial motion data may include various parameters that represent facial features and/or expressions such as, for example, the position of an individual's eyes, mouth, cheeks, teeth, eyebrows, etc., head turns, nods, blinks, frowns, smiles, winks, and so forth” [0014])
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify the combination of Danker and GARRIDO’s video stream system to include Li’s determination of no facial features and obtaining of new facial features because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically Li’s determination of no facial features and obtaining of new facial features permits obtaining facial feature data when none was obtained in the previously transmitted frame. This known benefit in Li is applicable to the combination of Danker and GARRIDO’s video stream system as they both share characteristics and capabilities, namely, they are directed to using facial features for video altering. Therefore, it would have been recognized that modifying the combination of Danker and GARRIDO’s video stream system to include Li’s determination of no facial features and obtaining of new facial features would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Li’s determination of no facial features and obtaining of new facial features in using facial features for video altering and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Claims 22-24 are rejected under 35 U.S.C. 103 as being unpatentable over Danker et al. (US 20180184171 A1 Hereinafter “Danker”) in view of Chalom et al. (US 20170308734 A1 Hereinafter “Chalom”).
Regarding claim 22, Danker teaches the recipient device of claim 17, wherein the one or more processors are configured to:
([0092]: “The media effect server 418 may be configured to analyze the video data 408 and/or audio data 412 using, e.g., emotion detection logic 420 for detecting an emotional state associated with the video data 408 and/or audio data 412, object recognition logic 432 for identifying objects in the video, facial recognition logic 434 for identifying faces in the video, and other types of logic for analyzing the content of the video data 408 and/or the audio data 412” The media effect server uses a decoded image frame); and
([0092]: “The media effect server 418 may be configured to analyze the video data 408 and/or audio data 412 using, e.g., emotion detection logic 420 for detecting an emotional state associated with the video data 408 and/or audio data 412, object recognition logic 432 for identifying objects in the video, facial recognition logic 434 for identifying faces in the video, and other types of logic for analyzing the content of the video data 408 and/or the audio data 412” The media effect server uses a decoded image frame).
Danker does not expressly disclose obtain additional image data corresponding to a modification of a portion of the decoded image frame and generate the modified image frame based on a combination of the additional image data and the original image.
However, Chalom teaches obtain additional image data corresponding to a modification of a portion of the decoded image frame ([0049]: “Returning again to FIG. 3, as shown, crop and resize module 302 may receive facial landmarks 214 and input image 212 and crop and resize module 302 may generate eye regions 311 using any suitable technique or techniques. For example, input image 212 may be cropped to generate one or more eye regions 311 such that eye regions 311 include all or most eye landmarks corresponding to a detected eye and, optionally, a buffer region around the outer most eye landmarks in the horizontal and vertical directions” The cropped regions are used to generate additional data) and generate the modified image frame based on a combination of the additional image data and the original image ([0057]: “Motion vector fields 313 and input image 212 may be received by warp and integrate module 307, which may warp eye regions 311 based on motion vector fields 313 and integrate the warped eye region or regions into input image 212”. The generated eye regions from the original image are warped back to fill the original image).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Danker’s video stream system to include Chalom’s use of additional image data and merging of additional and original image data because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, Chalom’s use of additional image data and merging of additional and original image data permits the use of altered data to correct data in the image that is undesirable. This known benefit in Chalom is applicable to Danker’s video stream system as they both share characteristics and capabilities, namely, they are directed to altering video streams for video communication. Therefore, it would have been recognized that modifying Danker’s video stream system to include Chalom’s use of additional image data and merging of additional and original image data would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Chalom’s use of additional image data and merging of additional and original image data in altering video streams for video communication and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regarding claim 23, the combination of Danker and Chalom teaches recipient device of claim 22, in addition, Danker further teaches wherein the one or more processors are configured to, based on determining that a modification feature of the modification is sender permitted, (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. The user can send instructions with the effects of if they are permitted or prohibited).
However, Chalom further teaches generating additional image data corresponding to a modification of a portion of the decoded image frame ([0049]: “Returning again to FIG. 3, as shown, crop and resize module 302 may receive facial landmarks 214 and input image 212 and crop and resize module 302 may generate eye regions 311 using any suitable technique or techniques. For example, input image 212 may be cropped to generate one or more eye regions 311 such that eye regions 311 include all or most eye landmarks corresponding to a detected eye and, optionally, a buffer region around the outer most eye landmarks in the horizontal and vertical directions” The cropped regions are used to generate additional data).
The combination of Danker and Chalom teaches selectively generating additional image data for the decoded image using Chalom’s method, when the modification is permitted using Danker’s method. It is based on the permission because the decoded image has a permission that is sent with it, if there is no permission for the use of media effect then the image should not be altered, but if there is permission to alter the image, the additional generated image data would take place.
The rationale for the combination is similar to the claim 22 combination due to similar methods of combination (generation of additional data) and benefits (improvement of image by correcting undesirable data).
Regarding claim 24, the combination of Danker and Chalom teaches recipient device of claim 22, in addition, Danker further teaches wherein the (Fig. 6, [0119]: “In embodiments where alternate-user effect application is enabled, each user may be permitted to define how and when media effects may be applied to them. For example, a user may identify which media effects may be applied to them, under what circumstances the media effects may be applied, who may apply the media effects (e.g., only friends, only a specified group of friends, etc.)”. The user can send instructions with the effects of if they are permitted or prohibited in the bitstream. The method of Chalom could be performed before the transmission of the image data, in that case, the additional image data would be present in the bitstream, hence the combination of Danker and Chalom teaches the claimed limitation).
The rationale for the combination is similar to the claim 22 combination due to similar methods of combination (generation of additional data) and benefits (improvement of image by correcting undesirable data).
Claim 25 rejected under 35 U.S.C. 103 as being unpatentable over Danker et al. (US 20180184171 A1 Hereinafter “Danker”) in view of GARRIDO et al. (WO 2019216988 A1 Hereinafter “GARRIDO”) in further view of Ciereszko et al. (US 11210862 B1 Hereinafter “Ciereszko”).
Regarding claim 25, Danker teaches the recipient device of claim 15, wherein the one or more processors are configured to:
obtain, from the bitstream, second encoded image data that represents a second image frame of the video ([0064]: “The media effect may be added to the user's video stream and broadcast to other users, so that other participants in the communication may experience the media effect”. Video stream means multiple image frames are part of the content, a second encoded image data would be present in the bitstream); and
(Fig. 6, [0135]: “At block 620, the system may add the media effect to the audiovisual data of the video stream. For example, the system may access video data on the video channel and audio data on the audio channel, incorporate the media effect into the audio and/or video data, and synchronize the data for transmission to the broadcasting server. At block 622, the system may transmit the resulting modified audiovisual data towards its destination (e.g., to receiving client devices or a broadcast server)”. The media effect is then used on the audio/visual data based on the sender permissions).
While it is presumed that the bitstream data would include the facial landmark data since the media effects are determined by the emotional state, and the emotional state is determined by the facial features detected ([0059]: “The face may also be analyzed to identify especially expressive features, which may be particularly relevant to emotion detection”), the bitstream should contain facial landmark data used to determine the emotional state), Danker does not expressly disclose wherein the bitstream includes the facial landmark data
However, GARRIDO teaches including the facial landmark data in bitstream data for video conferencing ([0057]: “Augmented-reality (AR) metadata may include information that describes physical objects such as people, facial features, animals, trees, bushes, flowers, streets, street signs, furniture or other physical objects that are represented in the video data image frames. For example the AR metadata may include depth information, size information, pixel location information, motion information, orientation information or other information about the physical objects. Content merging engine 406 may use the AR metadata to position, orient, size, and/or adapt, over time, the supplemental content that is added to the video stream from the camera”. This data is merged for encoding and sending to another device to decode).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects permits additional data which helps applying video effects to the transmitted video ([0057]: “the AR metadata may indicate the boundaries, location, size, depth, and features (e.g., the locations of the eyes, ears, mouth, etc.) of the user’s face, which can be used to size, locate, orient, and determine the facial expression of the animated animal face, as well as to animate the animated animal face to correspond to changes in the depth at which the user’s face is imaged, the orientation or location of the imaged face, and/or the facial expression of the user”). This known benefit in GARRIDO is applicable to Danker’s video stream system as they both share characteristics and capabilities, namely, they are directed to altering video streams to add effects to the stream. Therefore, it would have been recognized that modifying Danker’s video stream system to include GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate GARRIDO’s input of facial feature data in the transmitted bitstream data and use of data to assist application of the media effects in altering video streams to add effects to the stream and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
The combination of Danker and GARRIDO does not expressly disclose using a first landmark data for image generation when a first landmark data is absent.
However, Ciereszko teaches using previous feature data when current feature data is not available (Fig. 4, lines 28-36: “In this manner, the virtual element manager can select one or more feature points from previous frames to supplement or replace detected or missing feature points of the current frame when defining feature surfaces. As such, even though the feature points 112 and 116 are no longer available in the current video frame (of the field of view 102), the feature points 112 from a previous frame (of the field of view 100) can still be used”).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify the combination of Danker and GARRIDO’s video stream system to include Ciereszko’s use of prior feature data when current feature data is absent because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically Ciereszko’s use of prior feature data when current feature data is absent permits using previously obtained data to perform operations that current feature data would in a case where the current data is not available. This known benefit in Ciereszko is applicable to the combination of Danker and GARRIDO’s video stream system as they both share characteristics and capabilities, namely, they are directed to using image features for image processing. Therefore, it would have been recognized that modifying the combination of Danker and GARRIDO’s video stream system to Ciereszko’s use of prior feature data when current feature data is absent because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Ciereszko’s use of prior feature data when current feature data is absent in using image features for image processing and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Peters et al. (US 20210185276 A1) teaches video conferencing platform
Bathiche et al. (US 20140376785 A1) teaches enhancement of facial expressions of users in images
Troutman et al. (US 11321882 B1) teaches an engine for digital makeup application
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/ Examiner, Art Unit 2663
/SEAN M CONNER/ Primary Examiner, Art Unit 2663
