DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “units” in claims 1-18.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-6 and 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 2, the limitation “workload” renders the claim indefinite. It is unclear and confusing what is referred to by the limitation “workload”. The applicant’s specification (p23:29-p24:3) recites that the “workload” is somehow quantified without providing any description on how to quantify. For example, is it the total amount of predicted force exerted by the user during cooking considered “workload”? Or are there predetermined difficulty levels representing workload for each cooking motion that is stored for retrieval upon recognition of the cooking motion? Similar reasons apply to claims 3-6 that recite the limitation “workload”. Please amend the claims for clarification.
Regarding claim 9, the limitation “cooking proficiency level for distinguishing between being good at cooking and being poor at cooking” renders the claim indefinite. It appears to recite subjectively relative terms (i.e., “good” and “poor”) that render the claim indefinite. The applicant’s specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Regarding claim 10, the limitation “from a side” renders the claim indefinite for the following reasons:
ii) The applicant’s specification (p9:4-7) indicates that the videos are recorded “from a side”, and not “from above”. However, it is unclear and confusing how to clearly distinguish videos taken “from a side” and videos taken “from above”. For example, is a video is taken at an angle of 89 degrees with respect to the horizontal work surface considered to have been taken “from a side” or “from above”?
iii) The common understanding of the phrase “from a side” includes “from above”, because the limitation “above” indicates a direction (i.e., “side”).
Please amend the claim for clarification.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 8, 10-12, and 14-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bansal et al. (“Kitchen activity recognition based on scene context”).
Regarding claim 1, Bansal discloses:
a cooking motion estimation device comprising a storage unit and a control unit (see section 1, a computer, which is a processor and a memory), wherein
the storage unit includes: a video storage unit that stores a cooking behavior video of each of users (see section 3, a video database including cooking videos of multiple users), and the control unit includes:
a hand estimating unit that identifies coordinates of a joint point and estimates a hand region that is a coordinate region of a hand, for each of video frames that constitute the cooking behavior video, based on posture recognition technology (see section 4.2 and fig 1, the computer identifying coordinates of a wrist (i.e., the dot in fig 1(b)) and estimating a segmented hand (i.e., “wrist to finger tips), for each frames of a cooking video, using hand posture recognition algorithm);
a cooking utensil identifying unit that identifies a cooking utensil region that is a coordinate region of a cooking utensil, for each of the video frames that constitute the cooking behavior video, based on object recognition technology (see section 4.3 and fig 2, the computer identifying coordinates of a kitchen tool (i.e., “knife, chopsticks”), for each of the frames of the cooking video, using object classification algorithm); and
a cooking motion estimating unit that estimates a cooking motion for each of the video frames from a type of the cooking utensil when the hand region and the cooking utensil region overlap (see sections 4.4 and 4.5, the computer classifying a cooking motion depicted in the frames of the cooking video based on what type of kitchen tool the segmented hand approaches and grasps).
Regarding claim 8, Bansal further discloses: wherein the storage unit further includes:
a model storage unit that stores an object recognition model in which cooking utensil video frames in which a plurality of the cooking utensils are recorded are training data (see sections 3 and 4.3, the computer storing the object classification algorithm that is trained using a plurality of training cooking videos),
the video frames that constitute the cooking behavior video are input, and the cooking utensil region is output (see fig 1(a), the computer inputting the video frames of the cooking video; and see fig 1(b), the computer outputting the segmented hand), and
the cooking utensil identifying unit identifies the cooking utensil region that is the coordinate region of the cooking utensil, for each of the video frames that constitute the cooking behavior video, using the object recognition model (see rejection of claim 1, the computer identifying coordinates of the kitchen tool, for each frames of the cooking video, using the object classification algorithm).
Regarding claim 10, Bansal further discloses: wherein the cooking behavior video is a video that records cooking of each of the users from a side in any kitchen including a home kitchen of the user (see fig 1-3, the cooking video depicts cooking of the multiple users in a kitchen).
Regarding claim 11, Bansal further discloses: wherein the storage unit further includes:
a model storage unit that stores a cooking motion estimation model that is a machine learning model in which a cooking video labeled with the hand region is training data, the hand region and the cooking utensil region are explanatory variables, and the cooking motion is a response variable (see sections 3 and 5.1, the computer storing a cooking motion classifying model that is a support vector machine (i.e., a well-known machine learning algorithm) model that is trained using a plurality of training cooking videos; while the segmented hand and the kitchen tool are each variables that induces the resulting cooking motion classification), and
when the hand region and the cooking utensil region overlap, the cooking motion estimating unit estimates the cooking motion for each of the video frames from a type of the cooking utensil, using the cooking motion estimation model (see sections 4.4 and 4.5, the computer classifying a cooking motion depicted in the frames of the cooking video based on what type of kitchen tool the segmented hand approaches and grasps).
Regarding claim 12, Bansal further discloses: wherein the control unit further includes: a food ingredient identifying unit that identifies a food ingredient region that is a coordinate region of a food ingredient, for each of the video frames that constitute the cooking behavior video, based on the object recognition technology or image segmentation technology for the video frames (see section 4.3 and fig 2, the computer identifying coordinates of an ingredient (i.e., “eggs”), for each frames of the cooking video, using the object classification algorithm).
Regarding claim 14, Bansal further discloses: wherein when the hand region and the food ingredient region overlap, or when the cooking utensil region and the food ingredient region overlap, the cooking motion estimating unit further estimates the cooking motion for each of the video frames from a type of the food ingredient (see sections 3 and 4.4, when the segmented hand approaches the eggs, the computer classifying the cooking motion depicted in the frames of the cooking video as “breaking” or “peeling”).
Regarding claim 15, Bansal further discloses: wherein the control unit further includes: a seasoning identifying unit that identifies a seasoning region that is a coordinate region of a seasoning, for each of the video frames that constitute the cooking behavior video, based on the object recognition technology (see section 4.3 and fig 2, the computer identifying coordinates of a seasoning (i.e., “salt box”), for each of the frames of the cooking video, using the object classification algorithm).
Regarding claim 16, Bansal further discloses: wherein when the hand region and the seasoning region overlap, the cooking motion estimating unit further estimates the cooking motion for each of the video frames from a type of the seasoning (see sections 3 and 4.4, when the segmented hand approaches the seasoning, the computer classifying the cooking motion depicted in the frames of the cooking video as “seasoning”).
Regarding claims 17 and 18, Bansal discloses everything claimed as applied above (see rejection of claim 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Bansal in view of Okamura et al. (“Cooking activities recognition in egocentric videos using hand shape feature with OpenPose”).
Regarding claim 7, Bansal further discloses: wherein the storage unit further includes:
a model storage unit that stores a posture recognition model (see section 4.2, the computer storing the hand posture recognition algorithm),
the video frames that constitute the cooking behavior video are input, and the hand region is output (see fig 1(a), the computer inputting the video frames of the cooking video; and see fig 1(b), the computer outputting the segmented hand), and
the hand estimating unit identifies the coordinates of a joint point and estimates the hand region that is the coordinate region of a hand, for each of the video frames that constitute the cooking behavior video, using the posture recognition model (see rejection of claim 1, the computer identifying coordinates of the wrist and estimating a segmented hand, for each frames of the cooking video, based on the hand posture recognition algorithm).
However, Bansal does not disclose: in which hand video frames in which a plurality of hand movements during cooking are recorded are training data (i.e., Bansal discloses that the hand posture recognition algorithm identifies the wrist and the segmented hand, however, does not disclose that the hand posture recognition algorithm requires any training; and Bansal also discloses the presence of training cooking videos, however, the training cooking videos are not specifically used for training the hand posture recognition algorithm).
In a similar field of endeavor of classifying a cooking motion depicted in a cooking video, Okumura discloses: in which hand video frames in which a plurality of hand movements during cooking are recorded are training data (see sections 3.1-3.2, a computer identifying coordinates of the wrist and estimating a segmented hand by a hand posture recognition algorithm that has been trained with videos depicting hand motions).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Bansal with Okamura, and provide a hand posture recognition algorithm and training cooking videos, as disclosed by Bansal, and further train such hand posture recognition algorithm with the provided training cooking videos, as disclosed by Okamura, for the purpose of increasing recognition accuracy (see Okamura section 3.1-3.2).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Bansal in view of Allen et al. (US 2017/0103676).
Regarding claim 13, Bansal discloses everything clamed as applied above (see rejection of claim 12), however, does not disclose: wherein the food ingredient identifying unit further estimates intake nutrients from the food ingredient (i.e., Bansal discloses recognizing an ingredient (i.e., “eggs”), however, does not estimate nutrients from such ingredient).
In a similar field of endeavor of recognizing ingredients used in a cooking video, Allen discloses: wherein the food ingredient identifying unit further estimates intake nutrients from the food ingredient (see para [29], recognizing an ingredient used in a cooking video and retrieving corresponding nutritional values).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective fling date of the claimed invention to combine Bansal with Allen, and provide an object classification algorithm that recognizes an ingredient, as disclosed by Bansal, and further retrieve corresponding nutritional values, as disclosed by Allen, for the purpose of informing a user of the nutritional values (see Allen para [33]-[34]).
Allowable Subject Matter
The prior art of record does not disclose the subject matter recited in claims 2-6, however, these claims are rejected under 35 U.S.C. 112(b) as stated above. These claims would be allowable if amended to overcome the 112(b) rejection and rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 2, Bansal further discloses: a time setting unit that sets the video frames in connection with an elapsed time (see fig 6, time series of the cooking video). However, Bansal does not disclose: a classification calculating unit that calculates a cooking time and a workload for each cooking motion classification that distinguishes a feature of the cooking motion, based on the cooking motion for each of the video frames.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Sako et al. (US 2016/0005329), Stipe et al. (US 2020/0367692), Tagawa et al. (US 2018/0232202), and Kamei et al. (US 2017/0299194) each discloses optically recognizing a cooking process. Singer et al. (US 2015/0148980) discloses a relationship of cooking time and workload.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SJ PARK whose telephone number is (571)270-3569. The examiner can normally be reached M-F 8:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, EMILY TERRELL can be reached at 571-270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SJ Park/Primary Examiner, Art Unit 2675
Prosecution Insights