Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant's response to the last office action, filed May 11, 2026 has been entered and made of record. Claims 1, 3, 5-10 are amended; claims 2, 4, are cancelled; and claims 11-20 are new. By this amendment, claims 1, 3, 5-20 are now pending for examination.
Response to Arguments
Applicant’s arguments with respect to claims 1, 3, 5-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, and 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al, (US-PGPUB 20210118447) in view of Wallace (US-Patent 10,480,990); and further in view of Babu et al, (US-PGPUB 20220268523)
In regards to claim 1, Kim et al discloses a description generation device, (see
at least: Fig. 1, Par. 0057, “AI device (or an AI apparatus) 100”), comprising:
at least one memory; and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to:
acquire, for a task including a plurality of steps, videos”, of each of the steps that capture the task, including a plurality of steps “implicit by the cooking video explaining a food recipe for the cooking process”]);
based on the video characteristic amounts of the respective videos of each of the steps, specify actions with respect to materials that are included in the videos of each of the steps,
based on learning processor 130 may provide the cooking content image 604, “i.e., video characteristic amounts”, to the text recognition model 605 as input data to generate the image description text 606 as output data, “i.e., generate sentences describing procedures of the task for each of the steps”; and from Par. 0174, if the text … is Trim oyster mushrooms,” the text recognition model may output the image description text 803 “Trim oyster mushrooms”, [i.e., based on the specified actions, “Trim oyster mushrooms”, and the video characteristic amounts, “based on the video characteristic amounts”, generate sentences describing procedures of the task for each of the steps, “generate the image description text 606 as output data”]); and
generate the sentences by using a second model that has been trained in advance so as to generate sentences describing procedures of the task for each of the steps, based on material characteristic amounts, actions and video characteristic amounts, (see at least: Abstract, and Par. 0013, implicit by using an artificial intelligence apparatus including a learning processor configured to generate recipe text including at least one of cooking ingredient information or description text of cooking from cooking content).
Kim et al does not expressly disclose acquiring material characteristic amounts expressing respective materials used in the task; updating the material characteristic amounts of specified materials in accordance with specified actions; and generating, sentences describing procedures of the task for each of the steps, based on the updated material characteristic amounts; use, as the material characteristic amounts that are targets of updating, material characteristic amounts that have been updated with respect to a video of a previous step, in chronological order of the steps in the task; specify actions from video characteristic amounts, and update the material characteristic amounts by using a first model that has been trained in advance so as to update the material characteristic amounts based on the specified actions.
Wallace discloses acquiring material characteristic amounts expressing respective materials used in the task, (see at least: col. 4, lines 9-11, implicit by displaying the recipe or one or more ingredients such that the desired amount of information is visible on a display field at the same time);
updating the material characteristic amounts of specified materials in accordance with specified actions, (see at least: col. 2, line 53 through col. 3, line 2, implicit by adjustment to the amount of the ingredient, “updating the material characteristic amounts of specified materials”, through preselecting an adjusted amount or adding less or more, “specified actions”, than the targeted amount of the ingredient as predetermined in the recipe);
generating, based on the updated material characteristic amounts, sentences describing procedures of the task for each of the steps, (see at least: col. 11, lines 56-64, performing adjustments to ingredients including highlighting each recipe ingredient block 112. Such highlighting can include textual, visual, audio, video or other notification of operations being performed regarding that recipe ingredient block, [i.e., generating sentences describing procedures of the task for each of the steps, “implicit by highlighting including textual notification”, based on the updated material characteristic amounts, “implicitly based on the adjustments to ingredients”]); and
use, as the material characteristic amounts that are targets of updating, material characteristic amounts that have been updated with respect to a video of a previous step, in chronological order of the steps in the task, (see at least: col. 2, line 53 through col. 3, line 2, implicit by adjustment to the amount of the ingredient. Furthermore, the chronological order of the steps in the task, is well known in the art); and
specify actions from video characteristic amounts, (see at least: col. 4, lines 7-9, implicit by presenting a recipe or one or more ingredients to a user via one or more formats that include text, video, graphics, audio, and text).
Kim and Wallace are combinable because they are both concerned with cooking recipe description. Therefore, it would have been obvious to a person of ordinary skill in the art, to modify Kim, to use textual notification, as though by Wallace, in order to highlight the adjustments to ingredients, (Wallace, col. 11, lines 56-64).
The combine teaching Kim and Wallace as whole does not expressly disclose updating the material characteristic amounts by using a first model that has been trained in advance so as to update the material characteristic amounts based on the specified actions.
However, Babu discloses updating the material characteristic amounts by using a first model that has been trained in advance so as to update the material characteristic amounts based on the specified actions, (see at least: Par. 0029, the inventory engine 215 may monitor the user's progress following the recipe using image processing techniques or machine learned models on sensor data captured within the kitchen 140, and may further adapt a recipe based on a quantity of food added by the user, as determined from the sensor data, [i.e., specifying actions from video characteristic amounts, “determining that pasta that the user is cooking is al dente based on image data, detecting that the user has added eggs to a pan, …”]. The inventory engine 215 may update the recipe in the recipe datastore 280 and send indications of the adaptations to the recipe to the user interface engine 210, [i.e., update the material characteristic amounts, “adapt a recipe based on a quantity of food added by the user”, based on the specified actions, “as determined from the sensor data”]).
Kim, Wallace, and Babu are combinable because they are all concerned with cooking recipe description. Therefore, it would have been obvious to a person of ordinary skill in the art, to modify the combine teaching Kim and Wallace, to use the inventory engine 215, as though by Babu, in order to adapt a recipe based on a quantity of food added by the user, as determined from the sensor data, and further update the recipe in the recipe datastore 280, (Babu, Par. 0029).
In regards to claim 3, the combine teaching Kim, Wallace, and Babu as whole discloses the limitations of claim 1.
Wallace further discloses wherein the at least one processor is configured to carry out at least one of addition, deletion or merging of material characteristic amounts with respect to the updated material characteristic amounts, (col. 3, lines 1-2, implicit by adding less or more, “at least one of addition”, than the targeted amount of the ingredient as predetermined in the recipe).
In regards to claim 5, the combine teaching Kim, Wallace, and Babu as whole discloses the limitations of claim 1.
Kim further discloses at least one processor configured to train the first model and the second model by using, as training data, a material list and videos for each of the steps, and sentences of correct answers that correspond to the material list and the videos for each of the steps, (see at least: Par. 0145-0148, implicit by providing cooking content including video and audio of cooking process, to a recipe text generation model and generate recipe text. Further, from Par. 0198, learning processor 130 may use different neural network models, in order to generate numerical information of the unclear word depending on whether the unclear word is about the amount of ingredient or a word about a cooking progress state, “i.e., implicitly using first and second neural network models for generating sentences of correct answers that correspond to the material list and the videos for each of the steps]).
Regarding claim 9, claim 9 recites substantially similar limitations as set forth in claim 1. As such, claim 9 is rejected for at least similar rational.
The Examiner further acknowledged the following additional limitation(s): “description generation method executed by a computer”. However, Kim discloses the “description generation method executed by a computer”, (see at least: Fig. 4, and Par. 0011, “method”).
Regarding claim 10, claim 10 recites substantially similar limitations as set forth in claim 1. As such, claim 10 is rejected for at least similar rational.
The Examiner further acknowledged the following additional limitation(s): “a non-transitory storage medium storing a description generation program that is executable by a computer”. However, discloses the “non-transitory storage medium storing a description generation program that is executable by a computer”, (see at least: Par. 0211, “computer-readable recording medium“).
Regarding claim 11, claim 11 recites substantially similar limitations as set forth in claim 3. As such, claim 11 is rejected for at least similar rational.
Regarding claim 12, claim 12 recites substantially similar limitations as set forth in claim 5. As such, claim 12 is rejected for at least similar rational.
Regarding claim 17, claim 17 recites substantially similar limitations as set forth in claim 5. As such, claim 17 is rejected for at least similar rational.
Regarding claim 18, claim 18 recites substantially similar limitations as set forth in claim 6. As such, claim 18 is rejected for at least similar rational.
Allowable Subject Matter
Claims 6-8, 14-15, and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
With respect to claim 6, the prior art of record, alone or in reasonable combination, does not teach or suggest, the following underlined limitation(s), (in consideration of the claim as a whole):
“wherein the training section is configured to train the first model and the second model so as to minimize a total loss that includes a first loss, which is based on comparison of sentences generated by the generating section and the sentences of the correct answers, and a second loss, which is based on comparison of the actions and the material characteristic amounts specified at the updating section, and actions and materials of correct answers included in the videos for each of the steps”.
Kim et al (US-PGPUB 20210118447) discloses a description generation device, (see at least: Fig. 1, Par. 0057, “AI device (or an AI apparatus) 100”), comprising:
an acquiring section configured to acquire, for a task including a plurality of steps, video characteristic amounts extracted from respective videos of each of the steps that capture the task, (see at least: Par. 0181-0183, the recipe information may include … description of steps of a cooking process, [i.e., a task, “cooking process”, including a plurality of steps, “implicit by description of steps of a cooking process”]. Further, from Par. 0162, If the cooking content is a cooking video explaining a food recipe, the image 701 in the cooking content may be a frame-by-frame image in the cooking image, [i.e., acquiring video characteristic amounts, “implicit by the cooking content”, extracted from respective videos, “cooking videos”, of each of the steps that capture the task, including a plurality of steps “implicit by the cooking video explaining a food recipe for the cooking process”]);
an updating section configured to, based on the video characteristic amounts of the respective videos of each of the steps, specify actions with respect to materials that are included in the videos of each of the steps, (see at least: Par. 0185, when the word “egg” about the cooking ingredient in the description text is included in cooking ingredient information, the processor 180 may determine the description text “Boil an egg” as recipe information of cooking, [i.e., based on the video characteristic amounts of the respective videos of each of the steps, “when word egg is included in the cooking content”, specify actions with respect to materials that are included in the videos of each of the steps, “determining that the text “Boil an egg” as recipe information of cooking”]); and
a generating section configured to, based on the specified actions and the video characteristic amounts, generate sentences describing procedures of the task for each of the steps, (see at least: Par. 0170-0173, the learning processor 130 may provide the cooking content image 604, “i.e., video characteristic amounts”, to the text recognition model 605 as input data to generate the image description text 606 as output data, “i.e., generate sentences describing procedures of the task for each of the steps”; and from Par. 0174, if the text … is Trim oyster mushrooms,” the text recognition model may output the image description text 803 “Trim oyster mushrooms”, [i.e., based on the specified actions, “Trim oyster mushrooms”, and the video characteristic amounts, “based on the video characteristic amounts”, generate sentences describing procedures of the task for each of the steps, “generate the image description text 606 as output data”]).
Kim et al further discloses the artificial neural network, which may be used to determine the model parameters that minimize a loss function, which the loss function may be used as an index to determine optimal model parameters in the learning process of the artificial neural network, (Par. 0039).
However, while disclosing using the artificial neural network to determine the model parameters that minimize a loss function; Kim et al fails to teach or suggest, either alone or in combination with the other cited references, training the first model and the second model so as to minimize a total loss that includes a first loss, which is based on comparison of sentences generated by the generating section and the sentences of the correct answers, and a second loss, which is based on comparison of the actions and the material characteristic amounts specified at the updating section, and actions and materials of correct answers included in the videos for each of the steps”
A further prior art of record, Peters et al, (US-PGPUB 20210086753) discloses training the first model and the second model so as to minimize a total loss, (see at least: Fig. 7, and Par. 0075, first model portion 404 and the second model portion 410 may be trained by reducing, for example minimizing, the total loss value 614); but fails to teach or suggest, either alone or in combination with the other cited references, that the total loss includes a first loss, which is based on comparison of sentences generated by the generating section and the sentences of the correct answers, and a second loss, which is based on comparison of the actions and the material characteristic amounts specified at the updating section, and actions and materials of correct answers included in the videos for each of the steps”.
Regarding claims 7-8, claims 7 and 8 are in condition for allowance based at least on their dependency from claim 6.
Regarding claim 13, claim 13 recites substantially similar limitations as set forth in claim 6. As such, claim 13 is in condition for allowance, for at least similar reasons, as stated above.
Regarding claims 14-15, claims 14 and 15 are in condition for allowance based at least on their dependency from claim 6.
Regarding claim 18, claim 18 recites substantially similar limitations as set forth in claim 6. As such, claim 18 is in condition for allowance, for at least similar reasons, as stated above.
Regarding claims 19-20, claims 7 and 8 are in condition for allowance based at least on their dependency from claim 6.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMARA ABDI whose telephone number is (571)272-0273. The examiner can normally be reached 9:00am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMARA ABDI/Primary Examiner, Art Unit 2668 06/26/2026
Prosecution Insights