Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Claims 1,4, 7, and 13 are amended.
Claims 1-16 are presently pending.
Applicant’s arguments with respect to claim 1 have been considered but are moot in view of the new ground(s) of rejection.
Furthermore, with respect to the limitation of ‘selection of a location on the specific object to provide the overlay’, ‘initiating a poll’, and ‘overlay comprising animations’ it is noted that the Examiner took Official Notice that these features are widely understood to be known in the art. As Applicant has failed to traverse the Official Notice in a timely manner, the Official Notice is taken to be admitted prior art, and the Official Notice is made final. See MPEP 2144.03(C).
REQUIREMENT FOR INFORMATION
Applicant and the assignee of this application are required under 37 CFR 1.105 to provide the following information that the examiner has determined is reasonably necessary to the examination of this application.
The information is required to enter in the record the art suggested by the applicant as relevant to this examination in the Specification [0059] which recites: “In the example of FIG. 12 , users can also overlay images on top of other character's faces depending on the desired implementation. For example, by using known AI models and techniques such as deep fake, the face of one character can be swapped with a different face (e.g., another character, an animated icon, another person, etc.)”. The Examiner requests Applicant provide any art known regarding the known AI models stated for overlaying images on top of other character’s faces.
This requirement is an attachment of the enclosed Office action. A complete reply to the enclosed Office action must include a complete reply to this requirement. The time period for reply to this requirement coincides with the time period for reply to the enclosed Office action.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 2, 10, 15 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 2, 10, and 15 are respectively directed toward specific embodiments of the invention that specify overlaying information associated with objects/people identified in a video as shown in [Figs. 7-11 and 13-15] and their respective descriptions. [0005-7], [0023], [0029-32] and [0074] of the Specification specifically notes that the AI SoC of the claimed invention may apply neural network models to process received video and audio content for the purposes of detection and identification of people and objects within the received televised content. [0029-30] specifically notes that APU 206 processes the content to identify objects/people in the content, and Internet interface 208 is subsequently utilized to search for and retrieve relevant supplemental information which is subsequently utilized by OPU 212 to provide/output the received supplemental information. As such, the amended claim recitation in Claim 1 of “generating, based on execution of the loaded ML model, a control instruction and an overlay associated with the specific object” appears to be unsupported by the Specification as originally presented as there is insufficient written support to suggest that the recited ML model is utilized for generating the instructions or overlays associated with specific objects. Rather, the Specification only discloses that the AI SoC utilizes the loaded ML models for the object detection aspects of the invention. At best, [0059] discloses that AI models may be used for the example of Fig. 12 for performing face-swapping (covered in at least Claim 11). However, there is insufficient teaching that the Embodiments of [Figs. 7-11 and 13-15] apply any of the AI models for generating the overlays providing relevant data associated with the detected objects. As such, it cannot be said that Applicant was in possession of the claimed invention with respect to Claims 2, 10, and 15 at the time of filing.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-9, and 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over Tang (US 2015/0248918 A1) (of record, hereinafter Tang), in view of Carter et al. (US 2019/0222776 A1) (hereinafter Carter), further in view of Baca et al. (US 2019/0080170 A1) (hereinafter Baca), further in view of Liu et al. (US 2021/0329306 A1) (of record, hereinafter Liu), and further in view of Walter et al. (US 2019/0370072 A1) (of record, hereinafter Walter),
Regarding Claim 1, Tang discloses a method, comprising:
receiving televised content; [0027-29: content may be received and displayed, including television programming]
displaying, through a mobile application interface, identified objects for selection; [Fig. 6: 0038-39: guidance application may use various learning systems (e.g., neural networks) to perform content recognition on some media asset to detect objects, people, etc. within the media; 0097-100: screen 600 provides selectable list of objects/people in detected in the media presented in screen 500]
receiving, from the mobile application interface, a selection of a specific object; [Fig. 6: 0097-100: screen 600 provides selectable list of objects/people in detected in the media presented in screen 500]
generating, based on execution of the loaded ML model, a control instruction and an overlay associated with the specific object; [Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected object detected in the media utilizing various overlays to provide various information about the selected object, such as player statistics, etc.] and
transmitting the control instruction to cause modification of a display of the received televised content to render the overlay in association with the specific object within image frames of the received televised content. [Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected object detected in the media utilizing various overlays to provide various information about the selected object, such as player statistics, etc.]
Tang further discloses wherein the system may be implemented utilizing any type of processing circuitry comprising one or more processors or ASICs. [0057]
Tang fails to explicitly disclose generating, based on execution of the loaded ML model, a control instruction and an overlay associated with the specific object. (Emphasis on the particular element of the limitations not explicitly disclosed by Tang)
Carter, in analogous art, teaches generating, based on execution of the loaded ML model, a control instruction and an overlay associated with the specific object. [Figs. 2A-B; 0022-25, 0034-39, 0043-47: augmented overlays over video data (such as the overlays/information, etc. of Tang) may employ machine learning models to analyze video data to determine candidate regions to apply overlays and other data on the received video data]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the method of Tang with the teachings of Carter to utilize ML models for generating control instructions and overlays in order to provide automated overlaid video augmentations that selects appropriate portions/regions of the image for such overlays without human involvement. [Carter – ABST; 0022-25]
Tang and Carter fail to explicitly disclose downloading the ML model from a cloud platform and loading the downloaded ML model into a memory associated with a system on chip (SoC).
Baca, in analogous art, teaches downloading the ML model from a cloud platform and loading the downloaded ML model into a memory associated with a system on chip (SoC).. [Fig. 3; 0020, 0025: computer vision systems that perform object recognition (such as the system of Tang and Carter above), may download pre-created ML/NN models containing appropriate data/tests from cloud resources; 0027, 0062: wherein systems may be applicable to any Integrated circuits, such as SoCs, executing modules, etc., stored on any computer-readable media]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the method of Tang and Carter with the teachings of Baca to specify downloading ML models from cloud platforms as it is readily understood that a variety of different pre-created ML/NN models may exist in online systems that may be downloaded to perform various computer-vision applications [Baca – 0020,0025], and specifying use of an SoC as they are a known type of IC. [Baca – 0062]
Tang, Carter, and Baca fail to explicitly disclose selecting a machine learning (ML) model based on an input from the mobile application interface.
Liu, in analogous art, teaches selecting a machine learning (ML) model based on an input from the mobile application interface. [0596-0599: a plurality of pre-trained AI models may exist, where a user may select a machine learning model to be used in specific applications (such as the AI system and mobile application of Tang, Carter, and Baca above)]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the method of Tang, Carter, and Baca with the teachings of Liu to allow selection of the machine learning model to execute as it is understood that users may select specific machine learning models from a plurality of models to be used in specific applications. [Liu – 0599]
Tang, Carter, Baca, and Liu fail to explicitly disclose using an artificial intelligence System on Chip (AI SoC).
Walter, in analogous art, teaches an artificial intelligence System on Chip (AI SoC). [0010: platforms, such as a learning machine or STB, may be implemented as System on Chip (SOC)]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the method of Tang, Carter, Baca, and Liu with the teachings of Walter to specify use of an AI SoC as it is understood that SoCs have the benefits of lower power consumption due to a higher level of integration. [Walter – 0010]
Regarding Claim 2, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses retrieving information associated with the specific object; and generating the overlay from the retrieved information. [Tang – Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected objected detected in the media utilizing various overlays to retrieve and provide various information about the selected object, such as player statistics, etc.]
Regarding Claim 3, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses the modifying the display of the received televised content to display the overlay comprises displaying the visual overlay on the specific object. [Tang – Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected objected detected in the media utilizing various overlays to retrieve and provide various information about the selected object, such as player statistics, etc.]
Regarding Claim 4, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 3, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses wherein the modifying the display comprises: identifying a selection of a person and an object, displaying the overlay on the object when the object is associated with the person. [Tang – Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected objected detected in the media utilizing various overlays to retrieve and provide various information about the selected object, such as player statistics, etc.]
Regarding Claim 5, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses receiving a selection of one or more users through the mobile application interface, modifying the display of the received televised content of the selected one or more users to display the overlay. [Tang – Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected objected detected in the media utilizing various overlays to retrieve and provide various information about the selected object, such as player statistics, etc.]
Regarding Claim 6, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses retrieving information for display on the mobile application interface for the specific object. [Tang – Figs. 6-7; 0069: second screen device may be used to select information to be provided, where both the television equipment 402 and wireless user device 406 may show the same information and programming]
Regarding Claim 7, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang and Walter disclose wherein the AI SoC is disposed on one of a television, a set top box, or an edge device, [Tang – Figs. 3-4; 0030: user device may be a STB or television, or any other receiver or computing equipment; Walter – 0010] wherein the method further comprises receiving, through the mobile application interface, a channel to obtain the received televised content. [Tang – 0030, 0069, 0091-93: second screen device may be used to receive user inputs to select programming to be displayed on the user television equipment 402]
Regarding Claim 8, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang and Liu discloses teaches receiving, through the mobile application interface, a selection of the machine learning model; wherein the AI SoC is configured to execute the selected machine learning model in response to the selection. [Tang – 0052, 0068-70: second screen may provide an interface for adjusting settings and preferences of information on the first device; Liu – 0596-0599: a plurality of pre-trained AI models may exist, where a user may select a machine learning model to be used in specific applications]
Regarding Claim 9, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses receiving, through the mobile application interface preferences and customizations of display is presented. [Tang – 0052, 0068-70: second screen may provide an interface for adjusting settings and display preferences of information on the first device; 0122, 0132: preferences of user may affect how the selected object is to be marked/highlighted]
Tang and Walter fail to explicitly disclose receiving, through the mobile application interface, a selection of a location on the specific object to provide the overlay; wherein the modifying the display of the received televised content to display the overlay comprises providing the overlay on the specific object.
However, the Examiner takes Official Notice that selection of overlay location is a well-known and utilized presentation customization that users may manually set in order to ensure that any overlay information does not overly interfere with some underlying content or object in a video.
Regarding Claim 13, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Tang and Walter fail to explicitly disclose receiving, through the mobile application interface, instructions to initiate a poll; wherein the poll is provided to mobile application interfaces of one or more users viewing the received televised content.
However, the Examiner takes Official Notice that creating and providing polls to users viewing received television are a well-known and common means of user engagement and social interaction between viewers who are experiencing the same media at the same time.
Regarding Claim 14, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses that the overlay may comprise a variety of different manners of marking or highlighting an object, including changing coloring, bordering, shape, rhythm, or any other manner of visually marking an object. [0102, 0106-108, 0119-120
Tang and Walter fail to explicitly disclose wherein the overlay comprises animations.
However, the Examiner takes Official Notice that overlaying animations is a well-known and common means of visually marking or annotating video data.
Regarding Claim 15, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses wherein the overlay comprises statistics associated with the specific object. . [Tang – Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected objected detected in the media utilizing various overlays to retrieve and provide various information about the selected object, such as player statistics, etc.]
Regarding Claim 16, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses determining, for the specific identified object, spatial position information within the received televised content, and rendering the overlay based on the spatial position information. [Tang – Figs. 6-7; 0102, 0121-123: once selected, system will highlight or otherwise mark the selected objected detected in the media utilizing various overlays to retrieve and provide various information about the selected object, such as player statistics, etc.; 0007, 0105, 0117: position of the object will be determined to render 704 as the object moves]
Claim(s) 10 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang, Carter, Baca, Liu, and Walter as applied to claim 1 above, and further in view of Ortiz et al. (US 2012/0167001 A1) (of record, hereinafter Ortiz)
Regarding Claim 10, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses wherein the system may provide chat sessions between users. [0027, 0043]
Tang, Carter, Baca, Liu, and Walter fail to explicitly disclose wherein the overlay comprises text messages; wherein the modifying the display of the received televised content to display the overlay comprises modifying the display of a plurality of users to display the text messages.
Ortiz, in analogous art, teaches wherein the overlay comprises text messages; wherein the modifying the display of the received televised content to display the overlay comprises modifying the display of a plurality of users to display the text messages. [Fig. 5; 0082-83: facial recognition application (such as the AI SoC of Tang, Carter, Baca, Liu, and Walter above) may identify persons within a video content, wherein selecting one of the identified persons in the image (such as selection of objects in Tang, Carter, Baca, Liu, and Walter above) may open a communication window on the display device, wherein communications may be video chat, text messages, etc.]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the method of Tang, Carter, Baca, Liu, and Walter with the teachings of Ortiz to specify overlaying text messages as it is understood that video systems may allow users to communicate with other users through overlaid text messaging. [Ortiz – 0082-83]
Regarding Claim 12, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Furthermore, Tang discloses wherein the system may provide chat sessions between users. [0027, 0043]
Tang, Carter, Baca, Liu, and Walter fail to explicitly disclose receiving a selection of a person, and generating a chat application in the mobile application interface to facilitate chat with the person.
Ortiz, in analogous art, teaches receiving a selection of a person, and generating a chat application in the mobile application interface to facilitate chat with the person. [Fig. 5; 0082-83: facial recognition application (such as the AI SoC of Tang and Walter above) may identify persons within a video content, wherein selecting one of the identified persons in the image (such as selection of objects in Tang, Carter, Baca, Liu, and Walter above) may open a communication window on the display device, wherein communications may be video chat, text messages, etc.]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the method of Tang, Carter, Baca, Liu, and Walter with the teachings of Ortiz to specify overlaying text messages as it is understood that video systems may allow users to communicate with other users by selecting a detected person in a group video. [Ortiz – 0082-83]
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang and Walter as applied to claim 1 above, and further in view of Morrison (US 2019/0288972 A1) (of record, hereinafter Morrison).
Regarding Claim 11, Tang, Carter, Baca, Liu, and Walter disclose all of the limitations of Claim 1, which are analyzed as previously discussed with respect to that claim.
Tang, Carter, Baca, Liu, and Walter fail to explicitly disclose receiving a selection of a first person having a first face and a second person having a second face, and the generating comprises generating an overlay of the second face on the first person and an overlay of the first face on the second person.
Morrison, in analogous art, teaches receiving a selection of a first person having a first face and a second person having a second face, and the generating comprises generating an overlay of the second face on the first person and an overlay of the first face on the second person. [0020: video content items may allow graphic overlay options that may create a modified version of an image, such as AR software for swapping peoples’ heads or faces (which implicitly suggests selection of the faces to swap)]
It would have been obvious to one of ordinary skill in the art prior to the filing date of the invention to modify the method of Tang, Carter, Baca, Liu, and Walter with the teachings of Morrison to provide features to allow overlay of two peoples’ faces on the other’s respective bodies as it is a known form of entertainment to generate a distorted version an image. [Morrison – 0020]
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Briggs et al. (US 2019/0287024 A1); Lee et al. (US 2019/0361140 A1); and Gajendran et al. (US 2022/0050695 A1) which all teach downloading of pre-trained ML models from cloud systems.
Bogan, III, et al. (US 10,552,667 B1) which teaches use of AI to perform face-swapping processing.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM J KIM whose telephone number is (571)272-2767. The examiner can normally be reached 9:30am - 5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hadi Armouche can be reached at (571) 270-3618. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM J KIM/Primary Examiner, Art Unit 2409
