Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments and amendment have persuasively overcome the antecedent basis rejection for claim 9.
The remaining issues are addressed below.
The examiner notes that if Applicant presents evidence of an actual reduction to practice, that could be persuasive.
112a
Applicant argues:
Addressing basis (1), a diagram of a particular structure is not required to show possession and there is no requirement in the MPEP or any other source that an Applicant supply a diagram of a neural network or any other type of machine-learning model (e.g., a random forest, a support vector machine, etc.) for a person of skill in the art to understand how to train a machine- learning model.
Examiner responds:
Applicant’s arguments are mere attorney argument. MPEP 2145(I). If Applicant timely submits evidence (such as a sworn declaration), it will be considered.
Applicant argues:
Applicant also notes that, counter to the presumptions of the Office Action, not all machine-learning models include layers (e.g., random forest models). Insofar as the method of
claim 1 is performable using a random forest model, a description of "layers" of a layered architecture, such as a neural network, would be unhelpful to a person of ordinary skill in the art.
Examiner responds:
Applicant has not identified support for how to implement a random forest that contains sufficient details (such as how the trees are chosen, or how overfitting is avoided)
Applicant argues:
The present claims do not recite a specific machine learning model architecture or layers thereof, and the lack of a corresponding architecture diagram is therefore irrelevant.
Examiner responds:
Applicant is claiming a genus of many, many types of machine learning. Earlier, Applicant cited MPEP § 2163(II)(A)(3)(a), but omitted the beginning of the sentence that states “The written description requirement for a claimed genus may be satisfied through sufficient description of a representative number of species … .” Applicant has not satisfied the written description requirement for any specie, much less a sufficient number to be able to claim machine learning generally.
Applicant argues:
Applicant further notes that the as-filed application provides description of the functionality of the trained machine-learning model at, e.g., [0131], including the specific type of data used to train the machine-learning model
Examiner responds:
Specification [0131] states “The computer-implemented machine learning model can be trained on a sufficiently large data set of mental states of various portrayed individuals and/or portrayed groups before and after a catalog of actions are performed by one or more interacting individuals, where the actions are performed by interacting with the portrayed individuals and/or portrayed group.”
First, it is not clear what this is describing. For example, how would one measure an individual’s mental state? How would one measure the mental states of a group of people portrayed in a video? What data would go into the model? (E.g., would you use a 1024x1024 jpg image?) How large is sufficiently large?
Applicant argues:
There is no requirement in the MPEP or in any other source that the Applicant disclose code or pseudocode of a simulator or identify an off-the-shelf part suitable for performing the functionality of the simulator.
Examiner responds:
MPEP 2164.06(a)(I) explains “A disclosure of an electrical circuit apparatus, depicted in the drawings by block diagrams with functional labels, was held to be nonenabling in In re Gunn, 537 F.2d 1123, 1129, 190 USPQ 402, 406 (CCPA 1976), where there was no indication in the specification as to whether the parts represented by boxes were “off the shelf” or must be specifically constructed or modified for applicant’s system.”
The same reasoning applies here. If Applicant asserts that they have a “simulator,” that is unlike technology that the examiner has come across and there is no evidence that the simulator exists, Applicant needs to disclose enough about the simulator to demonstrate possession. However, if Applicant shows that this is an “off-the-shelf” product, or produces code or pseudocode, the examiner will consider it and those may be persuasive in overcoming this rejection.
Applicant argues:
Difficulty is not a valid basis for rejecting a claim as lacking written description under 35 U.S.C. § 112(a).
Examiner responds:
This task is difficult because more inventing needs to be done, and thus requires more disclosure.
Applicant argues:
Moreover, the collection and labeling (manually or otherwise) of large volumes of training data for model training are common practices in the art, and cannot be reasonably characterized as "infeasible."
Examiner responds:
Applicant’s arguments are mere attorney argument. MPEP 2145(I). If Applicant timely submits evidence (such as on an IDS), it will be considered.
While collecting large amounts of certain types of data are known (e.g., the Stanford Cars Dataset), the particular data at issue here appears unknown.
Applicant argues:
The specification provides very specific details outlining how mental state can be predicted (e.g., the discussion of FIGS. 2-3 at [0056]-[0100]) and how actions that are predicted to adjust that mental state can be generated (e.g., the discussion of FIGS. 6-7 at [0126]-[0152]).
Examiner responds:
First, this does not address the point made in the office action. Second, the citations are too all encompassing to understand Applicant’s point. The question is not if Applicant has asserted the ability to do these various things, the question is if Applicant has met the 112a requirements. The examiner’s review of the specification has not demonstrated 112a support.
Applicant argues:
That the specification allegedly does not provide sufficient detail to show possession of a hypothetical (unclaimed) method of improved joke-telling is irrelevant to whether the specification supports the method of claim 1, as the method of claim 1 does not purport to relate to joke telling or to "dynamic" mental states.
Examiner responds:
The examiner guessed that telling a joke might be an output action at step 506 of Fig. 7. Applicant may wish to identify examples of output actions for step 506 of Fig. 7 from the specification to clarify the examiner’s understanding.
Applicant argues:
Paragraph [0072], in particular, enumerates a number of different approaches that can be adopted to extract features from video data that can be used to classify mental state.
Examiner responds:
Paragraph [0072] is a list of desired functionality, but there is insufficient explanation to demonstrate possession.
Applicant argues:
However, there is significant language describing all three steps identified in steps (6)-(8) at, e.g., [0056]-[0100] and [0126]-[0152].
Examiner responds:
It’s not the volume of the language that is at issue.
Applicant argues:
Insofar as the Office Action may have intended to reject the claims under § 112(b)
Examiner responds:
This is an (a) rejection.
Applicant argues:
Applicant notes that "unlimited functional claim limitations" are not per se invalid under 35 U.S.C. § 112(a) or 35 U.S.C. § 112(b) (MPEP § 2173.05(g))
Examiner responds:
The office action has made a prima facie case.
Applicant argues:
Rather, "unlimited" functional claim limitations are only improper under 35 U.S.C. § 112(b) where they are not commensurate in scope with the disclosure (Id.).
Examiner responds:
The MPEP explicitly directs a 112(a) rejection.
Applicant argues:
as well as how actions predicted to adjust mental state are generated
Examiner responds:
Applicant may wish to identify a specific example from the specification because the examiner has not found one.
Applicant argues:
The interpretation of the claims advanced by the Office Action as reading on "persuading people, via video, of something they would not otherwise do" stretches the claims beyond their broadest reasonable interpretation, in violation of MPEP @ 2111, both mischaracterizing the claims and disregarding the overwhelming majority of recited limitations.
Examiner responds:
The examiner believes “persuade” or “manipulate” are reasonable short hands for, as Applicant describes, “2) how actions to modify mental state are predicted via iterative simulation.”
Applicant argues:
is intended to match values to dimensions of the model
Examiner responds:
Then the claims should be amended to recite “match.”
Applicant argues:
The suggested amendment would cause the claims to recite that each mental state value "is" a dimension of the multidimensional mental state model
Examiner responds:
The examiner is looking for a specific one-to-one correspondence, such as matching according to a table.
Applicant argues:
The term "Euclidean distance" is commonly known by those of even remedial skill in the art and
Examiner responds:
The issue is not the Euclidean calculation but rather determining the location of the endpoints. Is the intent to draw line segments between the points in Fig. 6? Note that the examiner’s review of the specification (e.g., [0122]) suggests that placement of these points is arbitrary, which goes back to the challenge of computing distances.
Applicant argues:
Rather, the Office Action only addresses the foregoing inaccurate "summarization" of the recitations of the claims.
Examiner responds:
The examiner believes “persuade” or “manipulate” are reasonable short hands for, as Applicant describes, “2) how actions to modify mental state are predicted via iterative simulation.”
Applicant argues:
However, analysis under Step 2A, Prong Two and under Step 2B requires analysis of the claim as a whole.
Examiner responds:
Applicant has not identified any additional elements that are not part of the abstract idea.
As to the prior art, the wrote “legal precedent teaches third individual being a person MPEP 2144.04(II).” The MPEP citation is merely a courtesy, but the examiner acknowledges that it should have been a roman numeral three instead of a roman numeral two. Applicant has not addressed the thrust of the rejection (i.e., “legal precedent teaches third individual being a person” as opposed to El Kaliouby’s computer).
As to double patenting, Applicant’s offer to submit a terminal disclaimer upon resolution of all other issues is non-responsive. The examiner has exercised discretion to overlook this.
Information Disclosure Statement
The information disclosure statement filed August 13, 2025 fails to comply with the provisions of 37 CFR 1.98(a)(4) because it lacks the appropriate size fee assertion. It has been placed in the application file, but the information referred to therein has not been considered as to the merits.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 (all claims) are rejected on the ground of nonstatutory double patenting as being unpatentable over the claims of each of U.S. Patent No. US 12299490 B1, US 12367894 B2, and US 12367895 B2 in view of the prior art as applied below.
Claims 1-20 (all claims) are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over the claims of copending Application Nos. 18134989, 18134987, and 17951967. Although the claims at issue are not identical, they are not patentably distinct from each other.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Both the pending claims and the conflicting patents and applications are all directed to performing video analysis of a first person to inform a second person how to change the mental state of the first person. Therefore, all of the conflicting patents and applications are directed to the same problem as the present application. Further, any differences between the present claims and the claims in any of the conflicting patents are obvious in view of the prior art as applied below. It would have been obvious to one of ordinary skill in the art, before the effective filing date, to combine the below prior art with any of the conflicting patents for implementation details (especially as the patent claims lack implementation details). Based on the findings herein, this is an example of “(A) Combining prior art elements according to known methods to yield predictable results.” MPEP 2143.
As an example of how similar the claims are, below is a comparison between the instant claim 1 and claim 1 of the first conflicting patent (however, due to length of the present claims, they are not all reproduced). Note that instant claim 1 recites multiple instances of features from conflicting claim 1. While these features are obvious over the below applied prior art, they are also obvious as duplication of parts. MPEP 2144.04(VI)(B).
Instant claim 1
Conflicting claim 1
1. A method of adjusting mental state, the method comprising:
acquiring video data of a first individual and a second individual;
extracting first image data of the first individual from the video data;
extracting first audio data of the first individual from the video data;
extracting second image data of the second individual from the video data;
extracting second audio data of the second individual from the video data;
extracting first semantic text data from the first audio data;
extracting second semantic text data from the second audio data;
identifying a first set of features from at least one of the first image data, the first audio data, and the first semantic text data;
identifying a second set of features from at least one of the second image data, the second audio data, and the second semantic text data;
predicting a first baseline mental state of the first individual based on the first set of features, wherein:
the first baseline mental state comprises a first mental state value and a second mental state value;
the first mental state value corresponds to a first dimension of a multidimensional mental state model; and
the second mental state value corresponds to a second dimension of the multidimensional mental state model;
predicting a second baseline mental state of the second individual based on the second set of features, wherein:
the second baseline mental state comprises a third mental state value and a fourth mental state value;
the third mental state value corresponds to the first dimension of the multidimensional mental state model; and
the fourth mental state value corresponds to the second dimension of the multidimensional mental state model;
predicting an average baseline mental state, wherein:
the average baseline mental state comprises a fifth mental state value and a sixth mental state value;
the fifth mental state value corresponds to the first dimension of the multidimensional mental state model;
the sixth mental state value corresponds to the second dimension of the multidimensional mental state model;
the fifth mental state value is an average of the first mental state value and the third mental state value; and
the sixth mental state value is an average of the second mental state value and the fourth mental state value;
identifying a target mental state, wherein:
the target mental state comprises a seventh mental state value and an eighth mental state value;
the seventh mental state value corresponds to the first dimension of the multidimensional mental state model; and
the eighth mental state value corresponds to the second dimension of the multidimensional mental state model;
simulating, by a simulator, a predicted path from the average baseline mental state toward the target mental state using the multidimensional mental state model, a plurality of actions, and a first computer-implemented machine learning model, wherein:
the first computer-implemented machine learning model is configured to relate actions of the plurality of actions and changes in value in at least one of the first dimension and the second dimension of the multidimensional mental state model;
the predicted path comprises one or more actions of the plurality of actions and corresponding changes to at least one of the first dimension and the second dimension of the multidimensional mental state model; and
the one or more actions are performable by a third individual by to adjust the average baseline mental state toward the target mental state; and
outputting an indication of the one or more actions to the third individual.
1. A method comprising:
acquiring video data of a first individual;
extracting image data and …
audio data from the video data;
extracting semantic text data from the audio data;
analyzing at least one of the image data, the audio data, and the semantic text data to identify a first set of features;
predicting a baseline mental state of the first individual based on the first set of features, wherein:
the baseline mental state comprises a first mental state value and a second mental state value;
the first mental state value corresponds to a first dimension of a multidimensional mental state model; and
the second mental state value corresponds to a second dimension of the multidimensional mental state model;
identifying a target mental state for the first individual, wherein:
the target mental state comprises a third mental state value and a fourth mental state value;
the third mental state value corresponds to the first dimension of the multidimensional mental state model; and
the fourth mental state value corresponds to the second dimension of the multidimensional mental state model;
simulating, by a simulator, a predicted path from the baseline mental state toward the target mental state using the multidimensional mental state model, a plurality of actions, and a first computer-implemented machine learning model, wherein:
the first computer-implemented machine learning model is configured to relate actions of the plurality of actions and changes in value in at least one of the first dimension and the second dimension of the multidimensional mental state model;
the predicted path comprises one or more actions of the plurality of actions and corresponding changes to at least one of the first dimension and the second dimension of the multidimensional mental state model; and
the one or more actions are performable by a second individual to adjust the baseline mental state of the first individual toward the target mental state; and
outputting an indication of the one or more actions to the second individual.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 (all claims) are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1 and 20 are rejected because there is series of critical technology gaps in the written description.
First, while claims 1 and 20 recite a variety of functions performed by machine learning models, a review of the specification shows that not even a single neural network architecture class is identified, much less the implementation details needed to demonstrate possession. For example, where one of skill in the art would expect an architectural diagram of a machine learning model and a discussion of why various layers were chosen/how they interrelate, Applicant’s figures merely repeat claim language. See, e.g., Figs. 2-5. The examiner notes that specification, [0153] makes reference to use of a “generic” machine learning model, but this is omitting critical details.
Second, claims 1 and 20 recite a “simulator,” but the specification does not disclose what this is beyond a description of functionality and a statement that an (unidentified) machine learning model is used. Specification, [0131].
Third, the discussion of training data is insufficient. Specification, [0153]-[0154] discuss the training data, but provide no guidance on how to acquire the data beyond saying that it is “labeled.” This section is explicit that a multimodal machine learning model is used. A model capable of performing the scope of claims 1 and 20 requires an incredible amount of data, and it is infeasible for a person to gather and label this data manually. One of ordinary skill in the art would expect a discussion of how to train the (unidentified) model, and would find this lacking.
Fourth, human mental states are dynamic. For example, if I find a knock-knock joke funny the first time, I might find knock-knock jokes annoying the tenth time that I have heard them. However, the specification does not address how to update either the simulator or the machine learning model in light of dynamic targets (as opposed to a static target, such as identifying pictures of kittens, where once a model can identify a kitten, it can be reliably identify kittens).
Claims 1, 12 and 20 recite “extracting” data from certain individuals, but the claims do not recite how this is performed, specifically, how the data from the named individual is identified. Thus, these claim elements are reciting results rather than the steps taken to achieve these results. MPEP 2173.05(g).
Claims 1, 5, 9, 12, 15, and 20 recite “predicting” various data, but this is also unlimited functional claiming. MPEP 2173.05(g).
Claims 1, 12, 14, 16, and 20 recite “identifying” various data, but this is also unlimited functional claiming. MPEP 2173.05(g).
Claims 1, 17, 18, and 20 recite “simulating” mental states, but this is not limited to a particular technology, and thus reads on any means of accomplishing this. MPEP 2173.05(g).
Dependent claims are likewise rejected.
Claims 1-20 (all claims) are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, because the specification, while being enabling for trivial cases, does not reasonably provide enablement for the full range of mental states and manipulations thereof. The specification does not enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make or use the invention commensurate in scope with these claims.
Claims 1-20 read on persuading people, via video, of something that they would not otherwise do, but the disclosure does not address how the third person’s actions are chosen to persuade someone to do something that they would not otherwise do.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 (all claims) are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 12, 17, 18, and 20 recite “corresponds,” but this is subjective because different people can have different opinions as to whether two things “correspond.” One option to overcome this rejection is to recite an objective standard, such as “is.”
Claim 12 recites “the third set of features,” but this lacks sufficient antecedent basis because claim 12 recites “a third set of features” but also depends from claim 9. MPEP 2173.05(e).
Claims 17 and 18 recite “distances,” but the distances do not exist, either in the physical world or on a graph, and thus there is no mechanism to assess them.
Dependent claims are likewise rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 (all claims) are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process) without significantly more.
Step 1: Claim 1 (and its dependents) recite a method, and processes satisfy Step 1 of the eligibility test.
Claim 20 recites a system, and machines satisfy Step 1 of the eligibility test.
Step 2A, prong one: All of the elements of the claims are a mental process because a person can watch someone and decide how to persuade/manipulate the other person. Further, the various models are also mental processes, see example 47, claim 2, element (d) (from the July 2024 AI subject matter eligibility examples). MPEP 2106.04(a)(2)(III)(C) explains that use of a generic computer or in a computer environment is still a mental process. In particular, this section begins by citing Gottschalk v. Benson, 409 US 63 (1972). “The Supreme Court recognized this in Benson, determining that a mathematical algorithm for converting binary coded decimal to pure binary within a computer’s shift register was an abstract idea.” In Benson the Supreme Court did not separately analyze the computer hardware at issue; the specifics of what hardware was claimed is only included in an appendix to the decision.
Because there are no additional elements, no further analysis is required for Step 2A, prong two or Step 2B.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 (all claims) are rejected under 35 U.S.C. 103 as being unpatentable over US20170171614A1 (“El Kaliouby”) in view of legal precedent.
1. A method of adjusting mental state, the method comprising:
acquiring video data of a first individual and a second individual; (El Kaliouby, claim 1, “obtaining a plurality of images from a plurality of participants involved in an interactive digital environment;”)
extracting first image data of the first individual from the video data; (El Kaliouby, claim 1, “analyzing emotional content within the plurality of images for a first set of participants within the plurality of participants”)
extracting first audio data of the first individual from the video data; (El Kaliouby, claim 8, “wherein the audio information includes voice data.”)
extracting second image data of the second individual from the video data; (El Kaliouby, claim 1, “obtaining a plurality of images from a plurality of participants involved in an interactive digital environment;”)
extracting second audio data of the second individual from the video data; (El Kaliouby, claim 8, “wherein the audio information includes voice data.”)
extracting first semantic text data from the first audio data; (El Kaliouby, claim 7, “augmenting the analyzing of emotional content, within the plurality of images, with evaluation of audio information.”)
extracting second semantic text data from the second audio data; (El Kaliouby, claim 7, “augmenting the analyzing of emotional content, within the plurality of images, with evaluation of audio information.”)
identifying a first set of features from at least one of the first image data, the first audio data, and the first semantic text data; (El Kaliouby, [0027] “FIG. 13 illustrates feature extraction for multiple faces.”)
identifying a second set of features from at least one of the second image data, the second audio data, and the second semantic text data; (El Kaliouby, [0027] “FIG. 13 illustrates feature extraction for multiple faces.”)
predicting a first baseline mental state of the first individual based on the first set of features, wherein: (El Kaliouby, Fig. 7A. The emotion at time 0 teaches the claimed baseline.)
the first baseline mental state comprises a first mental state value and a second mental state value; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the first mental state value corresponds to a first dimension of a multidimensional mental state model; and (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the second mental state value corresponds to a second dimension of the multidimensional mental state model; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
predicting a second baseline mental state of the second individual based on the second set of features, wherein: (El Kaliouby, Fig. 7A. The emotion at time 0 teaches the claimed baseline.)
the second baseline mental state comprises a third mental state value and a fourth mental state value; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the third mental state value corresponds to the first dimension of the multidimensional mental state model; and (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the fourth mental state value corresponds to the second dimension of the multidimensional mental state model; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
predicting an average baseline mental state, wherein: (El Kaliouby, Fig. 7A. The emotion at time 0 teaches the claimed baseline.)
the average baseline mental state comprises a fifth mental state value and a sixth mental state value; (El Kaliouby, [0061] “In embodiments, the emotional state of each viewer may be averaged or otherwise combined to form an aggregate emotional state, representative of a collective emotional state of the viewers.”)
the fifth mental state value corresponds to the first dimension of the multidimensional mental state model; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the sixth mental state value corresponds to the second dimension of the multidimensional mental state model; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the fifth mental state value is an average of the first mental state value and the third mental state value; and (El Kaliouby, [0061] “In embodiments, the emotional state of each viewer may be averaged or otherwise combined to form an aggregate emotional state, representative of a collective emotional state of the viewers.”)
the sixth mental state value is an average of the second mental state value and the fourth mental state value; (El Kaliouby, [0061] “In embodiments, the emotional state of each viewer may be averaged or otherwise combined to form an aggregate emotional state, representative of a collective emotional state of the viewers.”)
identifying a target mental state, wherein: (El Kaliouby, [0049] “The emotional analysis can determine points in a stream, a live stream, a game, a competitive game, and so on, that most correspond to a likelihood that a participant will pay.” The user paying money teaches the claimed target mental state)
the target mental state comprises a seventh mental state value and an eighth mental state value; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the seventh mental state value corresponds to the first dimension of the multidimensional mental state model; and (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
the eighth mental state value corresponds to the second dimension of the multidimensional mental state model; (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” That different emotions are simultaneously detected teaches the claimed multidimensional mental state model.)
simulating, by a simulator, a predicted path from the average baseline mental state toward the target mental state using the multidimensional mental state model, a plurality of actions, and a first computer-implemented machine learning model, wherein: (El Kaliouby, [0049] “The emotional analysis can determine points in a stream, a live stream, a game, a competitive game, and so on, that most correspond to a likelihood that a participant will pay.”)
the first computer-implemented machine learning model is configured to relate actions of the plurality of actions and changes in value in at least one of the first dimension and the second dimension of the multidimensional mental state model; (El Kaliouby, [0049] “The emotional analysis can determine points in a stream, a live stream, a game, a competitive game, and so on, that most correspond to a likelihood that a participant will pay.”)
the predicted path comprises one or more actions of the plurality of actions and corresponding changes to at least one of the first dimension and the second dimension of the multidimensional mental state model; and (El Kaliouby, [0050] “Similarly, emotional data can simplify discovering what current users already like and can focus marketing on the content that provides the biggest emotional impact, thus increasing the chance of adoption.”)
the one or more actions are performable by a third individual by to adjust the average baseline mental state toward the target mental state; and (El Kaliouby, Fig. 1, Modify Environment 170. See also, [0050] “Embodiments can include modifying the shared digital environment based on the spending habits of the third set of participants and the results of emotional analysis for the third set of participants.”)
outputting an indication of the one or more actions to the third individual. (El Kaliouby, Fig. 1, Modify Environment 170. See also, [0050] “Embodiments can include modifying the shared digital environment based on the spending habits of the third set of participants and the results of emotional analysis for the third set of participants.”)
El Kaliouby is not relied on for the claimed third individual being a person because El Kaliouby teaches that this is done via computer.
However, legal precedent teaches third individual being a person. (MPEP 2144.04)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply legal precedent to the teachings of El Kaliouby such that a player makes the offer rather than the computer because the player and the offeree may be friends. El Kaliouby, [0050].
Based on the above, this is an example of “combining prior art elements according to known methods to yield predictable results.” MPEP 2143.
Alternatively, legal precedent serves as its own modification rationale.
2. The method of claim 1, wherein: the target mental state is identified based on a task performed by the first individual; and (El Kaliouby, [0049] “if it is detected that a user tends to make a purchase when in a particular emotional state.” Making the purchase teaches the claimed task.)
the video data depicts the first individual performing the task. (El Kaliouby, claim 1, “obtaining a plurality of images from a plurality of participants involved in an interactive digital environment;”)
3. The method of claim 1, wherein the one or more actions output to the third individual are for performing a second task different than the first task. (El Kaliouby, [0048] “For example, the purchases can be “in game” purchases that can include purchases of additional weapons for a shooter game, car upgrades for a racing game, healing powers, or virtual currencies (e.g. purchasing gems or gold bars) for purchasing items within the game.” The various possible purchases each teach a different task.)
4. The method of claim 3, wherein performance of the second task causes the third individual to interact with the first individual and the second individual. (El Kaliouby, [0048] “For example, the purchases can be “in game” purchases that can include purchases of additional weapons for a shooter game, car upgrades for a racing game, healing powers, or virtual currencies (e.g. purchasing gems or gold bars) for purchasing items within the game.”)
5. The method of claim 1, wherein:
predicting the first baseline mental state comprises:
generating, by a second computer-implemented machine learning model, the first mental state value based on the first set of features; and (El Kaliouby, [0027] “FIG. 13 illustrates feature extraction for multiple faces.”)
generating, by a third computer-implemented machine learning model, the second mental state value based on the first set of features; and (El Kaliouby, [0027] “FIG. 13 illustrates feature extraction for multiple faces.”)
predicting the second baseline mental state comprises:
generating, by the second computer-implemented machine learning model, the third mental state value based on the second set of features; and (El Kaliouby, [0027] “FIG. 13 illustrates feature extraction for multiple faces.”)
generating, by the third computer-implemented machine learning model, the fourth mental state value based on the second set of features. (El Kaliouby, [0027] “FIG. 13 illustrates feature extraction for multiple faces.”)
6. The method of claim 5, wherein the first dimension describes an intensity of a first mental state and the second dimension describes a pleasantness of the first mental state. (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” Happiness teaches the claimed pleasantness. See also, [0067] “The vertical axis of each emotional state graph represents an emotion intensity.”)
7. The method of claim 5, wherein the first dimension describes an intensity of the first mental state, a pleasantness of the first mental state, an importance of information conveyed by the first individual, a positivity of the conveyed information, or a subject of the conveyed information. (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .” See also, [0067] “The vertical axis of each emotional state graph represents an emotion intensity.”)
Claims 8 and 9 are rejected as per claim 5.
Claim 10 is rejected as per claims 6 and 9.
11. The method of claim 10, wherein:
the second computer-implemented machine learning model is configured to relate intensity of the first mental state with features of the first set of features and the second set of features; (El Kaliouby, Figs. 7 and 13)
the third computer-implemented machine learning model is configured to relate pleasantness of the first mental state with features of the first set of features and the second set of features; and (El Kaliouby, [0038] “The results of the analysis can include detection of one or more of sadness, stress, happiness … .”)
the fourth computer-implemented machine learning model is configured to relate an importance of information conveyed by the first individual with features of the third set of features. (El Kaliouby, Fig. 13, bounding box 1320. Fig. 13 shows feature extraction, and the bounding box shows which information of the user’s face is more most relevant for feature extraction.)
Claims 12-15 are rejected as per claim 9.
16. The method of claim 14, wherein analyzing the audio data to identify the second set of features comprises: converting the audio data to a spectrogram; and analyzing the spectrogram with a fourth computer-implemented machine learning model. (El Kaliouby, [0042] “For example, detecting shifting in chairs or shuffling of feet can indicate a lack of engagement. Additionally, tones, or voice can indicate a sentiment such as excitement, anger, or surprise. Emotions such as stress can be detected by audio information such as tone and volume.” To the extent that tone and volume do not teach a spectrogram, spectrograms are known substitutes. MPEP 2144.06.)
17. The method of claim 1, wherein the simulating the predicted path comprises: generating a first plurality of intermediate points based on the changes in value in at least one of the first dimension and the second dimension and the average baseline mental state, wherein each intermediate point of the first plurality of intermediate points corresponds to an action of the one or more actions; (El Kaliouby, [0049] “The emotional analysis can determine points in a stream, a live stream, a game, a competitive game, and so on, that most correspond to a likelihood that a participant will pay.”)
measuring a first plurality of Euclidean distances between the first plurality of intermediate points and the target mental state; (El Kaliouby, [0049] “For example, if a participant has previously purchased virtual “weapons” for a game when in an angry emotional state, … ” The claim does not specify what the distance is of (e.g., there is not an objective coordinate system). It appears that the intent is most likely next step.)
determining a first preferred intermediate point of the first plurality of intermediate points, the first preferred intermediate point having a shortest Euclidean distance of the first plurality of Euclidean distances to the target mental state; and (El Kaliouby, [0049] “For example, if a participant has previously purchased virtual “weapons” for a game when in an angry emotional state, … ” That the user buys weapons when angry teaches that anger to buying weapons to a shorter distance than other emotions to buying weapons.)
storing, as a first step of the predicted path, the change in value in at least one of the first dimension and the second dimension used to generate the first preferred intermediate point and the corresponding action of the one or more actions. (El Kaliouby, [0049] “… then the participant may be prompted to purchase some virtual weapons for the game at a future time when an emotional state of anger is detected on the participant.”)
Claim 18 is rejected as per claim 17.
19. The method of claim 1, wherein outputting the indication of the one or more actions to the third individual comprises: cross-referencing, with a table of actions and instructions, the one or more actions to determine one or more instructions for performing the one or more actions; and (El Kaliouby, [0050] “This can include … adding additional game content such as weapons, vehicle upgrade, additional lives, additional virtual currency, or other suitable add-ons … .” Adding the various add-ons teach the claimed actions, the offers from the game teach the claimed instructions)
outputting the one or more instructions to the third individual. (El Kaliouby, [0050] “Similarly, emotional data can simplify discovering what current users already like and can focus marketing on the content that provides the biggest emotional impact, thus increasing the chance of adoption.”)
Claim 20 is rejected as per claim 1. See also, El Kaliouby, claim 32, teaching the claimed hardware.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220273907 A1 – “Method and apparatus for neuroenhancement to enhance emotional response”
US 20200187841 A1 – “System And Method For Measuring Perceptual Experiences”
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID ORANGE whose telephone number is (571)270-1799. The examiner can normally be reached Mon-Fri, 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ORANGE/Primary Examiner, Art Unit 2663
Prosecution Insights