Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1 objected to because of the following informalities: First content is misspelled as "first contest" in line 13 of claim . First content is misspelled as "first contest" in line . Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 9, 11 and 18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, it recites “generate a prompt for input to a generative AI model… and generate second content, associated with the first contest using a generative AI model having an input of the generated prompt.” From the claim language it is not clear if there are two generative AI models or if the second recitation of “a generative AI model” is referring to the first recitation of “a generative AI model.” The specification states “…the generative AI system may include a user interface 10100, an AI framework 10200, a generative AI model 10300, an application and service,” which does not provide sufficient clarity (spec, para [0054]). It is unclear what is meant by the claim language and the claim takes on an unreasonable degree of uncertainty and is thus indefinite. For the purpose of compact prosecution and art rejection, the examiner will interpret the limitations in question to mean one AI model where the second recitation of “a generative AI model” is referring to the first recitation of “a generative AI model.”
Regarding claim 11, it recites “generating a prompt for input to a generative artificial intelligence (AI) model… and generating second content, associated with the first contest using generative AI having an input of the generated prompt.” From the claim language it is not clear if the “generative AI” is referring to the “generative artificial intelligence (AI) model” in the previous limitation or if it is referring to a model at all.
The specification does not provide guidance regarding the scope of the term “generative AI” as it is used in claim 10 (spec, para [0222]). It is unclear what is meant by the claim language and the claim takes on an unreasonable degree of uncertainty and is thus indefinite. For the purpose of compact prosecution and art rejection, the examiner will interpret the limitations in question to mean one AI model where the “generative AI” is referring to the “generative artificial intelligence (AI) model” in the previous limitation.
Regarding claim 18, it is rejected using the same citations and rationales described in the rejection of claim 11.
Regarding claim 9, it recites the limitation "the result of learning" in line 4 of the claim. There is insufficient antecedent basis for this limitation in the claim.
Regarding claim 17, it is rejected using the same citations and rationales described in the rejection of claim 9.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 10-14, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Marzorati (US 20210225052 A1; hereinafter Marzorati) in view of Suzuki (US 20250336105 A1; hereinafter Suzuki).
Regarding claim 1, Marzorati teaches an electronic device comprising: a display (“FIG. 2 which schematically depicts a system diagram 100 of one illustrative embodiment of a personalized information overlay device 20 which includes a computing system 21 and visual output device 22 connected in a network environment 30 to a server computing system 101… In the depicted example, the visual output device 22 may be embodied as a display screen on a smart TV, tablet, or other motion video display, such as a wearable device (e.g., watch or smart glasses),” (page 4, para [0026]; Fig 2). A display includes a visual output device.);
at least one processor, comprising processing circuitry (“the computing system 21 which is illustrated as including one or more processing devices 23 and one or more memory devices 24 which are operatively connected together with other computing device elements generally known in the art, including buses, storage devices, communication interfaces, and the like,” (page 4, para [0027]).); and
a memory configured to store instructions executable by the at least one processor, wherein at least one processor, individually and/or collectively, is configured to execute the instructions and to cause the electronic device to (“the processing device(s) 23 may be used to process visual image data received at the computing system 21 for storage in the memory device(s) 24 which stores data 25 and instructions 26. In accordance with selected embodiments of the present disclosure, the stored data 25 and instructions 26 may embody a smart display overlay engine 27 which is configured to process video/image data and provide overlay content images or augmentations on the user's video screen,” (page 4, para [0027]).):
based on user input“the smart display overlay engine 27 may include a first classifier module 28A for dynamically identifying and classifying different objects or entities Ei in the captured visual information (e.g., video image) on the display screen 22 as the user or agent is viewing the video content shown on the display screen 22,” (page 4, para [0028]).
“At step 404, one or more objects in the received image/video are identified and/or classified, and corresponding content and/or attributes for each identified object is identified to form a history. In selected embodiments, a first computing system (e.g., computing system 21) may process captured image/video data from the display screen with a smart display overlay engine to classify or identify objects or entities in the image/video data. For example, an image classification algorithm, such as a binary classifier, may be executed on the personalized information overlay device to identify objects as products (e.g., food products, drinks, sandwiches, snacks, fruit, cereal, vegetables, milk containers, soda bottles and brands, cereal box brands, etc.) in the image/video data while the viewer is watching the display screen,” (page 7, para [0043]). First content includes a received / captured image / video.
“FIG. 4 processing commences at 401, such as when a personalized information overlay device is connected or attached to a video or image playback device, such as a display screen on a smart TV, tablet, or other motion video display, such as a wearable device (e.g., watch or smart glasses) and the user activates a video or image playback,” (page 7, para [0040] – [0041]; Fig 4).
“At step 403, viewer criteria and/or profile information is set up with one or more user configuration or specialization data files. In selected embodiments, the viewer criteria/profile may specify the health/medical/dietary criteria for the viewer…” (page 7, para [0042]). The user input includes user activation of playback and / or the user configuration. It is clear from Figure 4 that based on the user input one or more objects is identified.);
obtain first information associated with at least one of the object or the first content (“the smart display overlay engine 27 may also include a second content annotator module 28B for generating and/or accessing a product content list (e.g., ingredients and/or allergens) for each identified entity/product appearing on the display screen 22… the second content annotator module 28B may retrieve a product content list from a medical health corpus 109 that is ingested from multiple public domain sources, such as food ingredient labels, FDA corpus, medical journals, and the like,” (pages 4-5, para [0029]).
“the first computing system (e.g., computing system 21) may process captured image/video data using any suitable classifier model to identify corresponding content and/or attributes for each identified object, such as by retrieving product content information (e.g., ingredients, calories, nutrition, allergens, etc.) for each identified product/object,” (page 7, para [0043]).
“applying a binary classifier model to identify or match detected images on the display with entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.),” (page 3, para [0019]). First information includes any of a product content list, ingredients, allergens, corresponding content and / or attributes, and /or labels.), and second information associated with the electronic device or a user (“To personalize the predictive model 29 to the viewer of the displayed content, the model 29 may ingest user-specific health data about the health and/or dietary condition of the viewer, such as the viewer's age, food allergies, medical conditions, religious or dietary restrictions, amount of food or calories consumed, etc,” (page 5, para [0030] – [0031]).
“the viewer criteria/profile may specify the health/medical/dietary criteria for the viewer which will be used to specify the types of health-related image overlay content or augmentation based on the viewer's individual health/medical/dietary conditions. The viewer criteria may also specify the age, dietary goals, allergies, and/or eating disorders of the viewer, as well as applicable healthy eating priorities...” (page 7, para [0042]). Second information includes any of the user-specific health data and / or viewer criteria/profile.);
generate a model input based on at least one of the first content, the first information, and the second information (“the artificial intelligence (AI) machine learning analysis is applied by deploying a linear regression learning model to evaluate the product content information against viewer health criteria to generate a predicted health-related interaction for the viewer. Based on the predicted health-related interaction for the viewer, the information handling system generates a display overlay or augmentation for the display image which is provided as feedback to the viewer by displaying on the display screen the display overlay or augmentation with the display image,” (pages 8-9, para [0051]).
“The smart display overlay engine 27 may also include a third interaction identification module 28C for measuring or detecting the health interactions between the viewer and the content of each identified product/entity identified in the display screen 22. In selected embodiments, the third interaction identification module 28C may employ any suitable predictive model, including but not limited to a linear and logic regression model, to predict if an identified or matched product/entity is healthy for the viewer based on the corpus it was trained on. To personalize the predictive model 29 to the viewer of the displayed content, the model 29 may ingest user-specific health data about the health and/or dietary condition of the viewer, such as the viewer's age, food allergies, medical conditions, religious or dietary restrictions, amount of food or calories consumed, etc,” (page 5, para [0030]).); and
generate second content, associated with the first contest using a predictive model having an input of the generated model input (“The predictive model 29 will then predict what type of information should be included as an overlay or augmentation for the displayed product. For example, the third interaction identification module 28C may process the product contents for all products identified on the display screen 22 for negative health interactions (e.g., known allergens or other health or dietary counterindications for the viewer) and/or for positive health interactions (e.g., healthy food choices, recommended daily servings of food groups, etc.) that are tailored or specific to the viewer,” (page 5, para [0030]).
“…the smart display overlay engine 27 may also include a fourth display overlay module 28D to provide personalized feedback information about the health interactions in the form of personalized health-related information which augments and/or overlays the displayed video/image 22. In selected embodiments, the fourth display overlay module 28D may generate a visual overlay for the area of the screen containing any product that is detrimental to the viewer, where the visual overlay may be a colored square, circle or other polygon. In addition or in the alternative, the generated visual overlay may include iconography and/or coloring to indicate or identify the allergen or condition/affliction in an identified product which should not be used or consumed by the viewer. In addition or in the alternative, the generated visual overlay may identify the caloric content of an identified product, alone or as a percentage of the viewer's available caloric budget for the day,” (page 5, para [0031]; page 3, para [0021]).
“In addition or in the alternative, the generated visual overlay may include a visual augmentation to indicate that an identified product should not be used or consumed by the viewer. An example visual augmentation could include a facial overlay of decayed teeth on a person drinking a sugary soft drink to visually indicate the negative effects that sugar has on the human body over time. Other visual augmentations and/or overlay content for specific products or ingredients may be built up and/or crowd-sourced by third parties,” (page 8, para [0049]).
The second content includes the content included in visual overlay.).
Marzorati is not relied upon teaching but Suzuki teaches based on a request to generate artificial intelligence (AI) content, identify an object (“…the information acquisition unit accepts upload or selection of the element image, and acquires the position information based on layout of the element image in a frame of the result image,” (page 1, para [0024]; Fig 5).
From Figure 5, it is clear that there is a user interface with interactive buttons for a user to create an image and / or display generated image.
“The server apparatus 1 acquires generation information from the user (1001). The server apparatus 1 generates a prompt based on the acquired generation information (1002). The server apparatus 1 inputs the prompt into the generative model (1003). The server apparatus 1 acquires output information (result image) of the generated model (1004). The server apparatus 1 presents the output information to the user (1005),” (page 5, para [0060]).
“the generation information acquisition unit 111 may support the user to easily recognize the necessary generation information by providing the user with a guide such as ‘Please upload an image of the subject’ or ‘Please indicate reference websites’,” (pages 3-4, para [0046]-[0048]).
“The server apparatus 1 may perform, for example, preprocessing of a partial image acquired by the generation information acquisition unit 111. The server apparatus 1 may determine the subject of the partial image, for example, and remove the background except for the subject part. The generation information acquisition unit 111 may also highlight the subject from the partial image, for example,” (page 5, para [0062]).
“The generation information acquisition unit 111 acquires an element image that is the basis for the configuration of part of the result image. The generation information acquisition unit 111 may accept upload of the element image. Furthermore, as illustrated as an example in FIG. 5, for example, the generation information acquisition unit 111 may store material images (for example, 201 in FIG. 5) in the server apparatus 1 and present those to the user terminal 3, accept a selection operation of the material image from the user, and acquire the material image selected by the user as the generation information,” (page 3, para [0046]).)
Further, Marzorati does not explicitly disclose the predictive model is a generative AI model, and the model input is a prompt for input to a generative AI model.
Suzuki teaches the predictive model is a generative AI model (“The result image generation unit 112 inputs, for example, prompt information generated by the generation information acquisition unit 111 based on at least one of the style information, element image, and position information into the generative model, and acquires an image output from the generative model,” (page 4, para [0053]).
“the result image generation unit 112 may input the prompt into an image generative model in a case of image or design, while inputting the prompt into a text generative model (for example, a large language model such as ChatGPT) in a case of text information,” (page 5, para [0067]).
“The generation information may also include element images that are the basis for the configuration of part of the result image,” (page 2, para [0038]; page 1, para [0007]).), and
the model input is a prompt for input to a generative AI model (“The generation information acquisition unit 111 generates a prompt, a prerequisite condition, or the like (collectively referred to as prompt information herein) to be input into an image generative model based on the acquired generation information... The prompt generated by the generation information acquisition unit 111 includes at least text representing the style. The generation information acquisition unit 111 uses a feature extraction module and a language model, for example, to generate prompt information,” (page 4, para [0049]-[0050]).
“The result image generation unit 112 inputs, for example, prompt information generated by the generation information acquisition unit 111 based on at least one of the style information, element image, and position information into the generative model,” (page 4, para [0053]).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Suzuki to Marzorati. The motivation would have been to increase scalability. Additional motivation would have been to increase the amount and / or variety of content that can be produced. Additional motivation would have been “to allow users to easily generate target images,” (page 1, para [0006]).
Further motivation would have been to substitute Suzuki’s generative AI model to replace Marzorati’s predictive model. A generative AI model was known; a predictive model was known. The predictable result would have been creating content associated with the first content.
Regarding claims 11 and 18, they are rejected using the same citations and rationales described in the rejection of claim 1. Claim 18 additionally recites a non-transitory computer-readable recording medium having recorded thereon a computer program which, when executed by at least one processor, individually and/or collectively, of an electronic device, causes the electronic device to perform operations comprising:
Marzorati teaches a non-transitory computer-readable recording medium having recorded thereon a computer program which, when executed by at least one processor, individually and/or collectively, of an electronic device, causes the electronic device to perform operations comprising (“Furthermore, aspects of the present invention may take the form of computer program product embodied in a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention,” (Marzorati; pages 1-2, para [0011]-[0012]).):
Regarding claim 2, Marzorati in view of Suzuki teaches the electronic device of claim 1, wherein the second content comprises AI content, educational content, or learning content (Marzorati; “The techniques disclosed herein can be tailored for specific audiences and users, such as educating children, elderly individuals, or users who are new to their condition/ailment about how everyday foods/product can affect them. There are additional education benefits of explaining to individual viewers the health benefits/risks over time (e.g., sugar effects on tooth decay or obesity),” (page 8, para [0050]).
“…generate the overlay panel 4 which visually identifies a negative or problematic interaction with the health of the viewer. In this example, the problematic interaction is identified with a negative “no” sign with appropriate coloring and/or a written statement (e.g., ‘Soda Mouth—The sugar in soft drinks is the leading cause of tooth decay and obesity.’),” (Marzorati; page 3, para [0021]; Fig 1A).
“generate the overlay panel 8 which visually identifies a positive or non-problematic interaction with the health of the viewer. In this example, the non-problematic interaction is identified with a positive “happy face” sign with appropriate coloring and/or a written statement (e.g., ‘Drinking Milk—Great source of calcium which prevents the breaking of bones.’),” (Marzorati; page 3, para [0022]; Fig 1B). The second content includes a visual overlay. The visual overlays comprise educational content and / or learning content.).
Regarding claim 3, Marzorati in view of Suzuki teaches the electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: extract, as the first information, at least one of object identification information, a content-describing keyword obtained by analyzing the first content or the identified object, and additional information based on metadata of content (Marzorati; “…applying a binary classifier model to identify or match detected images on the display with entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.),” (page 3, para [0019]). Object identification information includes entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.
“the smart display overlay engine 27 may also include a second content annotator module 28B for generating and/or accessing a product content list (e.g., ingredients and/or allergens) for each identified entity/product appearing on the display screen 22… the second content annotator module 28B may employ any suitable classification algorithm or model 29, such as a binary classifier model, or the like, to detect and match each identified entity/product with a corresponding product content list,” (Marzorati; pages 4-5, para [0029]). A content-describing keyword obtained by analyzing the first content or the identified object includes any parts of a product content list, ingredients, and / or allergens.); and
extract, as the second information, at least one of user learning setting information, user app usage information, user profile information, and context information (Marzorati; “At step 403, viewer criteria and/or profile information is set up with one or more user configuration or specialization data files. In selected embodiments, the viewer criteria/profile may specify the health/medical/dietary criteria for the viewer which will be used to specify the types of health-related image overlay content or augmentation based on the viewer's individual health/medical/dietary conditions. The viewer criteria may also specify the age, dietary goals, allergies, and/or eating disorders of the viewer, as well as applicable healthy eating priorities, such as low cholesterol, lean protein, low starch vegetables, whole grains, healthy fats, and fruit,” (page 7, para [0042]).
“To personalize the predictive model 29 to the viewer of the displayed content, the model 29 may ingest user-specific health data about the health and/or dietary condition of the viewer, such as the viewer's age, food allergies, medical conditions, religious or dietary restrictions, amount of food or calories consumed, etc,” (Marzorati; page 5, para [0030] – [0031]). The user profile information includes viewer criteria and / or profile information.).
Regarding claim 12, Marzorati in view of Suzuki teaches the method of claim 11, wherein the second content comprises AI content, educational content, or learning content (Marzorati; “The techniques disclosed herein can be tailored for specific audiences and users, such as educating children, elderly individuals, or users who are new to their condition/ailment about how everyday foods/product can affect them. There are additional education benefits of explaining to individual viewers the health benefits/risks over time (e.g., sugar effects on tooth decay or obesity),” (page 8, para [0050]).
“…generate the overlay panel 4 which visually identifies a negative or problematic interaction with the health of the viewer. In this example, the problematic interaction is identified with a negative “no” sign with appropriate coloring and/or a written statement (e.g., ‘Soda Mouth—The sugar in soft drinks is the leading cause of tooth decay and obesity.’),” (Marzorati; page 3, para [0021]; Fig 1A).
“generate the overlay panel 8 which visually identifies a positive or non-problematic interaction with the health of the viewer. In this example, the non-problematic interaction is identified with a positive “happy face” sign with appropriate coloring and/or a written statement (e.g., ‘Drinking Milk—Great source of calcium which prevents the breaking of bones.’),” (Marzorati; page 3, para [0022]; Fig 1B). The second content includes a visual overlay. The visual overlays comprise educational content and / or learning content), wherein the first information comprises at least one of object identification information, a content-describing keyword used for analyzing the first content or the identified object, and additional information based on metadata of content (Marzorati; “…applying a binary classifier model to identify or match detected images on the display with entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.),” (page 3, para [0019]). Object identification information includes entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.); and wherein the second information comprises at least one of user learning setting information, user app usage information, user profile information, and context information (Marzorati; “At step 403, viewer criteria and/or profile information is set up with one or more user configuration or specialization data files. In selected embodiments, the viewer criteria/profile may specify the health/medical/dietary criteria for the viewer which will be used to specify the types of health-related image overlay content or augmentation based on the viewer's individual health/medical/dietary conditions. The viewer criteria may also specify the age, dietary goals, allergies, and/or eating disorders of the viewer, as well as applicable healthy eating priorities, such as low cholesterol, lean protein, low starch vegetables, whole grains, healthy fats, and fruit,” (page 7, para [0042]).
“To personalize the predictive model 29 to the viewer of the displayed content, the model 29 may ingest user-specific health data about the health and/or dietary condition of the viewer, such as the viewer's age, food allergies, medical conditions, religious or dietary restrictions, amount of food or calories consumed, etc,” (Marzorati; page 5, para [0030] – [0031]). The user profile information includes viewer criteria and / or profile information.).
Regarding claim 4, Marzorati in view of Suzuki teaches the electronic device of claim 3, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to select and extract the second information associated with the first information (Marzorati; “process the ingredients list for each product identified on the screen for specified health criteria (e.g., known allergens or counterindications within the product and/or the user's known afflictions, dietary restrictions, and/or medical conditions) and to generate personalized health-related information in response to the ingredients list for a product matching with or correlating to the specified health criteria,” (para [0045]).
“To personalize the predictive model 29 to the viewer of the displayed content, the model 29 may ingest user-specific health data about the health and/or dietary condition of the viewer, such as the viewer's age, food allergies, medical conditions, religious or dietary restrictions, amount of food or calories consumed, etc. The predictive model 29 will then predict what type of information should be included as an overlay or augmentation for the displayed product. For example, the third interaction identification module 28C may process the product contents for all products identified on the display screen 22 for negative health interactions (e.g., known allergens or other health or dietary counterindications for the viewer) and/or for positive health interactions (e.g., healthy food choices, recommended daily servings of food groups, etc.) that are tailored or specific to the viewer,” (Marzorati; page 5, para [0030]-[0031]).
“the health criteria of a second viewer of the displayed image 15 may be used to generate a second set of display panels 16-19 that is overlaid on top of the meal components to display personalized health-related information about the meal components based on the health criteria of the second viewer…In this example where the first viewer is sensitive to glutens contained in the chicken nuggets, the positive interactions are identified with green overlay boxes (e.g., a first green overlay box 11 around the milk drink, a second green overlay box 14 around the fries, and a third green overlay box 13 around the fruit), while negative interactions are identified with red overlay boxes (e.g., a first red overlay box 12 around the chicken nuggets),” (Marzorati; pages 3-4, para [0023]-[0025]; Fig 1C).
Generating the personalized health-related information / personalized overlay includes selecting and extracting second information associated with the first information.).
Regarding claim 13, Marzorati in view of Suzuki teaches the method of claim 11, wherein the obtaining of the first information associated with at least one of the object or the first content and the second information associated with the electronic device or the user comprises selecting and extracting the second information associated with the first information (Marzorati; “process the ingredients list for each product identified on the screen for specified health criteria (e.g., known allergens or counterindications within the product and/or the user's known afflictions, dietary restrictions, and/or medical conditions) and to generate personalized health-related information in response to the ingredients list for a product matching with or correlating to the specified health criteria,” (para [0045]).
“To personalize the predictive model 29 to the viewer of the displayed content, the model 29 may ingest user-specific health data about the health and/or dietary condition of the viewer, such as the viewer's age, food allergies, medical conditions, religious or dietary restrictions, amount of food or calories consumed, etc. The predictive model 29 will then predict what type of information should be included as an overlay or augmentation for the displayed product. For example, the third interaction identification module 28C may process the product contents for all products identified on the display screen 22 for negative health interactions (e.g., known allergens or other health or dietary counterindications for the viewer) and/or for positive health interactions (e.g., healthy food choices, recommended daily servings of food groups, etc.) that are tailored or specific to the viewer,” (Marzorati; page 5, para [0030]-[0031]).
“the health criteria of a second viewer of the displayed image 15 may be used to generate a second set of display panels 16-19 that is overlaid on top of the meal components to display personalized health-related information about the meal components based on the health criteria of the second viewer…In this example where the first viewer is sensitive to glutens contained in the chicken nuggets, the positive interactions are identified with green overlay boxes (e.g., a first green overlay box 11 around the milk drink, a second green overlay box 14 around the fries, and a third green overlay box 13 around the fruit), while negative interactions are identified with red overlay boxes (e.g., a first red overlay box 12 around the chicken nuggets),” (Marzorati; pages 3-4, para [0023]-[0025]; Fig 1C).
Generating the personalized health-related information / personalized overlay includes selecting and extracting second information associated with the first information.).
Regarding claim 5, Marzorati in view of Suzuki teaches the electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to, as an operation of identifying the object, identify the object through an input of selecting the object included in the first content or through recognition of the object in the first content (Marzorati; “…the smart display overlay engine 27 may include a first classifier module 28A for dynamically identifying and classifying different objects or entities Ei in the captured visual information (e.g., video image) on the display screen 22 … the first classifier module 28A may employ any suitable image classification service or object detection algorithm, such as Convolutional Neural Networks (CNN), binary classifier, or the like, to identify the objects or entities that are included in the captured visual information…,” (page 4, para [0028]).
“an image classification algorithm, such as a binary classifier, may be executed on the personalized information overlay device to identify objects as products (e.g., food products, drinks, sandwiches, snacks, fruit, cereal, vegetables, milk containers, soda bottles and brands, cereal box brands, etc.) in the image/video data while the viewer is watching the display screen,” (Marzorati; page 7, para [0043]).
“…perform visual recognition of objects displayed on a screen by applying a binary classifier model to identify or match detected images on the display with entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.),” (Marzorati; page 3, para [0019]). Marzorati teaches identifying the object through recognition of the object in the first content.).
Regarding claim 14, it is rejected using the same citations and rationales described in the rejection of claim 5.
Regarding claim 10, Marzorati in view of Suzuki teaches the electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to, based on selecting the first content displayed on the display, display a second content generation item and a data collection item on at least a part of the display (Marzorati; “the health criteria of a second viewer of the displayed image 15 may be used to generate a second set of display panels 16-19 that is overlaid on top of the meal components to display personalized health-related information about the meal components based on the health criteria of the second viewer…In this example where the first viewer is sensitive to glutens contained in the chicken nuggets, the positive interactions are identified with green overlay boxes (e.g., a first green overlay box 11 around the milk drink, a second green overlay box 14 around the fries, and a third green overlay box 13 around the fruit), while negative interactions are identified with red overlay boxes (e.g., a first red overlay box 12 around the chicken nuggets),” (Marzorati; pages 3-4, para [0023]-[0025]; Fig 1C).
“FIG. 1A provides an illustration 1 of example display screen 2 depicting an image 3 showing a boy consuming a soft drink from a bottle. In the displayed image 3, an example display panel 4 is overlaid to display personalized health-related information about the soft drink product displayed in the image 3… the overlay panel 4 which visually identifies a negative or problematic interaction with the health of the viewer. In this example, the problematic interaction is identified with a negative “no” sign with appropriate coloring and/or a written statement (e.g., ‘Soda Mouth—The sugar in soft drinks is the leading cause of tooth decay and obesity.’).” (Marzorati; page 3, para [0021]-[0022]; Fig 1A; Fig 1B).
“the feedback is provided to the viewer by generating a visual overlay for the area of the display screen containing any product that is detrimental to the viewer's health or medical condition, where the visual overlay may be a square, circle or other polygon that is colored a first color (e.g., red) to indicate a problematic or negative interaction, and is colored a second color (e.g., green) to indicate a non-problematic or positive interaction. In addition or in the alternative, the generated visual overlay may include iconography and/or text to indicate or identify the allergen or condition/affliction in an identified product which should not be used or consumed by the viewer, such as a text overlay stating that ‘The sugar in soft drinks is the leading cause of tooth decay’,” (Marzorati; page 8, para [0049]).
Selecting the first content displayed on the display includes identifying / selecting / recognizing a product that should not be consumed by the viewer. A second content generation item includes a colored polygon to indicate a problematic or negative interaction. A data collection item includes a display panel containing iconography and / or text. Based on identifying / selecting / recognizing a product that should not be consumed by the viewer (due to a problematic or negative interaction), a second content generation item is displayed indicating the problematic or negative interaction. Based on identifying / selecting / recognizing a product that should not be consumed by the viewer (due to a problematic or negative interaction), a data collection item is displayed indicating the problematic or negative interaction.) and,
based on selection of the second content generation item or recognition of the object included in the first content, receive the request to generate the AI content (Marzorati; “…the smart display overlay engine 27 may include a first classifier module 28A for dynamically identifying and classifying different objects or entities Ei in the captured visual information (e.g., video image) on the display screen 22 … the first classifier module 28A may employ any suitable image classification service or object detection algorithm, such as Convolutional Neural Networks (CNN), binary classifier, or the like, to identify the objects or entities that are included in the captured visual information…,” (page 4, para [0028]; Marzorati; page 7, para [0043]).
“…perform visual recognition of objects displayed on a screen by applying a binary classifier model to identify or match detected images on the display with entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.),” (Marzorati; page 3, para [0019]).
“At step 407, one or more machine learning, natural language processing (NLP), and/or artificial intelligence (AI) processing techniques are applied, alone or in combination, to the new image/video data to predict a new relevancy value with respect to problematic and/or nonproblematic interactions based on specified viewer criteria/profile data. In selected embodiments, a server computing system (e.g., computing system 101) and/or first computing system 21 may employ artificial intelligence processing techniques using one or more machine learning models (e.g., a binary classifier model and/or predictive model) which are trained with the specified viewer criteria/profile and the history of identified product/objects E (E1, E2, . . . En) and corresponding content/attributes to predict or determine if any products identified in the new image have corresponding product content (e.g., ingredients, allergens, caloric counts, etc.) that is allowed or not allowed for the viewer,” (Marzorati; pages 7-8, para [0047]).
“If the new image relevancy does not exceed the predetermined threshold (negative outcome to detection step 408), then no further action is taken, and the methodology returns to step 406 to await detection of a new image. However, if the new image relevancy does exceed the predetermined threshold (affirmative outcome to detection step 408), then the method proceeds to overlay information on the new image related to the predicted new image relevancy,” (Marzorati; pages 7-8, para [0047]-[0049]; Fig 4).
Receive the request to generate the AI content includes the affirmative outcome to the detection step. At least illustrated by Figure 4, based on recognition of the object included in the first content, the outcome to the detection step is determined and received.).
Regarding claim 17, Marzorati in view of Suzuki in teaches the method of claim 11, further comprising: based on the displaying of the second content on the display, obtaining edition information in which the AI content associated with the second content has been modified or changed, or feedback information regarding the result of learning of the second content; (Suzuki; “Upon presenting a plurality of generated result images to the user terminal 3, the result image generation unit 112 may accept a selection operation from the user on the user terminal 3 to select an image that is close to or deviated from the result image desired to be generated from those images, and further generate a result image based on the result image selected by the selection operation.”(pages 4-5, para [0056]-[0057]).
Feedback information regarding the result of learning of the second content includes a selection operation from the user. The result of learning includes an evaluation / judgment of the user upon viewing the second content. Based on the presenting a plurality of generated result images to the user terminal, a selection operation from the user on the user terminal is obtained.); and updating the second content, based on the obtained edition information or feedback information (Suzuki; “…, and further generate a result image based on the result image selected by the selection operation.” (pages 4-5, para [0056]-[0057]). Updating the second content includes further generating a result image based on the result image selected by the selection operation.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Suzuki to Marzorati. The motivation would have been to generate content that is closer to the user’s desired result. Additional motivation would have been to increase the relevance of generated content. Additional motivation would have been to improve the user’s experience.
Claims 6-7, 9 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Marzorati in view of Suzuki in further view of Bossard et al. (US 20220335242 A1; hereinafter Bossard).
Regarding claim 6, Marzorati in view of Suzuki teaches the electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to display the generated second content on the display (Marzorati; “To enable personalized user feedback of positive and/or negative health interactions between the viewer and the content of each identified product/entity identified in the display screen 22, the smart display overlay engine 27 may also include a fourth display overlay module 28D to provide personalized feedback information about the health interactions in the form of personalized health-related information which augments and/or overlays the displayed video/image 22,” (page 5, para [0030] - [0031]).
“FIG. 1A provides an illustration 1 of example display screen 2 depicting an image 3 showing a boy consuming a soft drink from a bottle. In the displayed image 3, an example display panel 4 is overlaid to display personalized health-related information about the soft drink product displayed in the image 3,” (Marzorati; page 3, para [0021]-[0022]; Fig 1A; Fig 1B).), wherein the second content comprises Marzorati; “…applying a binary classifier model to identify or match detected images on the display with entity labels (e.g., milk container, soda bottle and brand, cereal box brands, etc.),” (page 3, para [0019]).
“generate the overlay panel 4 which visually identifies a negative or problematic interaction with the health of the viewer. In this example, the problematic interaction is identified with a negative “no” sign with appropriate coloring and/or a written statement (e.g., ‘Soda Mouth—The sugar in soft drinks is the leading cause of tooth decay and obesity.’),” (Marzorati; page 3, para [0021]-[0022]; Fig 1A; Fig 1B).
“generate the overlay panel 8 which visually identifies a positive or non-problematic interaction with the health of the viewer. In this example, the non-problematic interaction is identified with a positive “happy face” sign with appropriate coloring and/or a written statement (e.g., ‘Drinking Milk—Great source of calcium which prevents the breaking of bones.’),” (Marzorati; page 3, para [0021]-[0022]; Fig 1A; Fig 1B).
From Fig 1A and 1B the overlay panel clearly contains object identification information.), a type of generated AI content (Marzorati; “In addition or in the alternative, the generated visual overlay may include a visual augmentation to indicate that an identified product should not be used or consumed by the viewer. An example visual augmentation could include a facial overlay of decayed teeth on a person drinking a sugary soft drink to visually indicate the negative effects that sugar has on the human body over time. Other visual augmentations and/or overlay content for specific products or ingredients may be built up and/or crowd-sourced by third parties,” (page 8, para [0049]). After combination, Marzorati’s visual augmentation and / or visual overlay is generated by Suzuki’s generative model.), and a content description (Marzorati; “…a written statement (e.g., ‘Soda Mouth—The sugar in soft drinks is the leading cause of tooth decay and obesity.’),” (Marzorati; page 3, para [0021]-[0022]; Fig 1A; Fig 1B).
“… a written statement (e.g., ‘Drinking Milk—Great source of calcium which prevents the breaking of bones.’),” (Marzorati; page 3, para [0021]-[0022]; Fig 1A; Fig 1B). The written statement in Fig 1A and Fig 1B include a content description.).
Marzorati in view of Suzuki is not relied upon teaching but Bossard teaches the second content comprises a content or object image (“The visual indicators may be selectable to present information to users about various elements of interest such as objects and scenes included in the image or video. The information may include, for example, what the element of interest of interest is, its type, and/or a synopsis or a description about the element of interest, a selectable element (e.g., a link to buy or download) associated with the element of interest,” (page 1, para [0017]; page 3, para [0037]).
“on selecting the visual indicator 606, the user may be presented with information 608 corresponding to the content of the image … the information 608 may include description of the element of interest (e.g., painting) and one or more selectable elements,” (page 6, para [0063]-[0065]; Fig 6; Fig 7).
From Fig 6 and Fig 7, it is clear that a content and / or object image is included in the second content.),
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Bossard to Marzorati in view of Suzuki. The motivation would have been to provide clarity to the user regarding which object the information / second content relates to. Additional motivation would have been to help the user quickly confirm the relevance of the information / second content. Additional motivation would have been to improve the user experience.
Regarding claim 15, it is rejected using the same citations and rationales described in the rejection of claim 6.
Regarding claim 7, Marzorati in view of Suzuki in further view of Bossard teaches the electronic device of claim 6, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to display the second content in the form of a pop-up window overlaid on the first content, or to switch the second content into the form of a foreground and then display the switched second content (Marzorati; “FIG. 1A provides an illustration 1 of example display screen 2 depicting an image 3 showing a boy consuming a soft drink from a bottle. In the displayed image 3, an example display panel 4 is overlaid to display personalized health-related information about the soft drink product displayed in the image 3,” (page 3, para [0021]-[0022]; Fig 1A; Fig 1B). A pop-up window overlaid on the first content includes a display panel.
“To enable personalized user feedback of positive and/or negative health interactions between the viewer and the content of each identified product/entity identified in the display screen 22, the smart display overlay engine 27 may also include a fourth display overlay module 28D to provide personalized feedback information about the health interactions in the form of personalized health-related information which augments and/or overlays the displayed video/image 22,” (Marzorati; page 5, para [0030] - [0031]).).
Regarding claim 9, Marzorati in view of Suzuki in further view of Bossard teaches the electronic device of claim 6, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: obtain edition information in which the AI content associated with the second content has been modified or changed, or feedback information regarding the result of learning of the second content (Suzuki; “Upon presenting a plurality of generated result images to the user terminal 3, the result image generation unit 112 may accept a selection operation from the user on the user terminal 3 to select an image that is close to or deviated from the result image desired to be generated from those images, and further generate a result image based on the result image selected by the selection operation.”(pages 4-5, para [0056]-[0057]).
Feedback information regarding the result of learning of the second content includes a selection operation from the user. The result of learning includes an evaluation / judgment of the user upon viewing the second content.); and update the second content, based on the obtained edition information or feedback information (Suzuki; “…, and further generate a result image based on the result image selected by the selection operation.” (pages 4-5, para [0056]-[0057]). Updating the second content includes further generating a result image based on the result image selected by the selection operation.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Suzuki to Marzorati in view of Bossard. The motivation would have been to generate content that is closer to the user’s desired result. Additional motivation would have been to increase the relevance of generated content. Additional motivation would have been to improve the user’s experience.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Marzorati in view of Suzuki in further view of Bossard in further view of Gray et al. (US 11769017 B1; hereinafter Gray).
Regarding claim 8, Marzorati in view of Suzuki in further view of Bossard is not relied upon teaching but Gray teaches the electronic device of claim 6, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to, based on source information of the content description being different, apply different visual effects to the text of the content description and display the text of the content description to be visually distinguished according to the source information (“FIG. 7A1 depicts an example client device 710 with a display 780 rendering, in response to a query 782, a graphical interface that includes an example NL based summary 784 and additional example search results 788 that are responsive to the query 782. In the NL based summary 784, there are three linkified portions, each indicated by underlining and a source identifier (S1, S2, S3) provided immediately following the linkified portions. Each linkified portion, and its source identifier, is selectable to cause navigation to a corresponding search result document that verifies the linkified portion,” (col 27, lines 59-67; col 28, lines 1- 5; Fig 7A1).
“the system can additionally or alternatively include, as part of the content, a source identifier of the SRD. For example, the source identifier can be a token included at the beginning and/or the end of the content. The token can be unique relative to other source identifier(s) for other SRD(s) of the set. The token can be descriptive of the underlying source document or can be non-descriptive thereof (e.g., it can be one of N default source identifiers such as S1, S2, S3, etc.). As described herein, in some implementations including the source identifier in the content can enable the LLM output, generated based on processing the content using the LLM, to reflect which portion(s) of the NL based summary are supported by which SRD(s),” (col 16, lines 37- 51).
“a portion, of a visually rendered NL based summary, that is supported by a first SRD can be selectable (and optionally underlined, highlighted, and/or otherwise annotated),” (col 18, lines 29-67).
“a portion with a high confidence measure can be annotated in a first color (e.g., green), a portion with a medium confidence measure can be annotated in a second color (e.g., orange), and a portion with a low confidence measure can be annotated in a third color (e.g., red),” (col 16 lines 52-67; col 17 lines 1 - 11).
“the confidence measure of a portion can be based on trustworthiness of the SRD(s)) that verify that portion and/or a quantity of the SRD(s) that verify that portion,” (col 19, lines 36-55).
The visual effects include underlining, highlighting, coloring, otherwise annotating and / or including a source identifier (such as S1, S2, S3). From at least Fig 7A1, it is clear the text of the content description is visually distinguished according to the source information. Based on the search result document (SRD) being different, the visual effects are applied.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Gray to Marzorati in view of Suzuki in further view of Bossard. The motivation would have been to “[enable] a user to quickly ascertain which portion(s) of the NL based summary are verifiable,” (Gray; col 18, lines 27-67). Additional motivation would have been to allow a user to quickly find more information about a portion of the content description. Additional motivation would have been to improve user experience.
Regarding claim 16, it is rejected using the same citations and rationales described in the rejection of claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERICA G THERKORN whose telephone number is (571)272-2939. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERICA G THERKORN/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights