DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on February 26, 2026, has been entered.
Response to Amendment
Claims 1 and 26-28 have been amended. Claim 13 has been canceled. Claims 30 and 31 are new. Claims 1-12 and 14-31 are pending and are provided to be examined upon their merits.
Response to Arguments
Applicant’s arguments with respect to claims 1-12 and 14-31 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. A response is provided below in bold where appropriate.
Applicant notes claim amendments, starting pg. 12 of Remarks:
Examiner thanks Applicant for providing support. However, the Examiner had problems of finding certain claimed features. The Examiner provides claim interpretation in some of the cases and had to provide 35 USC 112 rejection in one case.
Regarding 35 USC 112 (b) rejection:
Applicant did not address this in their Remarks. However, as Applicant has canceled Claim 13, this rejection is withdrawn.
Applicant notes Double Patenting, pg. 14 of Remarks:
Double Patenting
The Applicant will submit a power of attorney. Once the power of attorney has been approved by the Office, the Applicant will file a terminal disclaimer, thereby obviating the double patenting rejection.
Noted, however the claims are provisionally rejected as there are no patented claims. Applicant should probably wait on the Terminal Disclaimer pending claim resolution. The claims will be monitored over time for double patenting.
Applicant argues 35 USC §103 Rejection, starting pg. 15 of Remarks:
Nonobviousness
The Examiner contended that the invention of claims 1-29 would have been obvious over Pub. No. US 2018/0315247 to Van Andel in view of Pub. No. US 2023/0021339 to Bosnak et al.
Advantages of the invention of amended claim 1 over the cited prior art are as follows:
Amended claim 1 defines extracting activity measurements comprising at least data pertaining to the execution of the session of the experience itself. This is opposed to the prior art focusing on raw physical limb measurements which may correspond, at best, to the user measurements of claim 1 only.
From Van Andel…
Example of completes a scheduled session of therapy (experience itself)…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
Examples according to claim 1 include: tracking how many times a user paused the game or counting the collection of non-goal items (e.g., red rocks instead of blue) to assess concentration skill (see paragraphs [0012], [0146]).
Applicant’s arguments are not commensurate with the scope of their claims. There is no game, counting non-goal item, or rocks claimed.
The Applicant emphasizes its position that no description is provided in Van Andel, in particular in paragraph [0029] relied upon by the Examiner, that corresponds to a plurality of activities within a single interactive experience, in the sense of claim 1.
The Examiner respectfully disagrees. Also, Applicant’s arguments are not commensurate with the scope of their claims. There is no “plurality of activities within a single interactive experience” claimed.
Previously amended Claim 1 already recited:
"... providing on a user device an interactive virtual experience in which the use can participate, the experience comprising one or more activities;...
extracting one or more activity measurements from the provided experience;
The Examiner contended in the Office action that:
Performing movement is an activity. Bending a knee is an activity. Exercise would be an activity. Applicant's claims use broad terms such as "activity. "Limitations from the specification are not read into the claim.
However, physical movements of the user like bending a knee of Van Andel cannot be read into "activities from an interactive virtual experience" and certainly cannot read into "extracting one or more activity measurements from the provided [interactive virtual] experience" as in claim 1.
Just about anything can be an experience. Breathing is an experience. Bending a knee is an experience. Also, Applicant’s specification does not define experience.
However, to expedite allowance, claim 1 is amended as outlined above to emphasize that the system extracts activity measurements from the experience which comprises at least data pertaining to the execution of the experience itself. This is not taught by Van Andel which extracts only measurements that relate to the user himself, or the equipment that he carries.
Applicant needs to provide for the record where their specification gives a definition of “experience.” Otherwise, this is broadly interpreted and any activity could be considered an “experience.”
Van Andel teaches:
“The physical object 340 may include a sensor 355. The sensor 355 may be used to track patient progress, such as a duration or number of repetitions completed by the user 305. In an example, the sensor 355 may be used to identify the physical object 340 to the AR device 310. Once identified, the AR device 310 may select the virtual object 345 that corresponds to the physical object 340. For example, if the user 305 has a ten pound weight and a five pound weight available, these may be identified via sensors in the weights, and the AR device 310 may determine the five pound weight is to be used in an exercise, and the virtual object 345 may resemble the five pound weight. The virtual object 345 may change to resemble the ten pound weight in another exercise or when increasing weight to increase difficulty for the exercise. In an example, the AR device 310 may display a virtual movement of a limb.” [0040]
Therefore, track patient progress such as duration or number (activity measurements) of repetitions (experience itself).
2. Amended claim 1 defines determining combined measurements based on both the user measurements and the activity measurements. The purpose is to evaluate technical and non-technical "human skills" such as cognitive, social, and behavioral more accurately, based on both the user's movements, but in combination with how the session of the activity run and what happened during the session.
Respectfully, the above is not claimed.
From Claim 1…
“determining one or more combined measurements based on the analyzed measurements, wherein the combined measurements are aggregated over the duration of the session of the experience…”
Applicant’s arguments are not commensurate with the scope of their claims. There is no evaluate technical and non-technical “human skills” such as cognitive, social, and behavioral more accurately, based on both the user's movements,” in their independent claims.
Examples are detailed in paragraphs [0185]-[0190], such as:
correlating a user's heart rate (user measurement) with game speed (activity measurement) to objectively calculate a "stress tolerance" level.
Measuring a lack of focus derived from constantly moving the eye gaze to
the side (a user measurement) when combined with a focused target the
user 110 has to shoot at (an activity measurement).
A positive emotion, when combining the physical emotion of rising the
corners of the mouth with reaching an end of a level in the experience.
Respectfully, the above is not claimed.
Also from Applicant’s specification…
“In some examples, the later analysis including determining the combined measurements is performed with respect to the following parameters. Some of these parameters are also referred to during the analysis performed while the user participated in the experience: … Aggregated: over multiple experiences (e.g., improvement, total exercise time)” [0173], [0177]
Therefore, measurements combined over an exercise period.
As previously explained, the step of "determining one or more combined measurements based on the analyzed measurements" is not necessarily performed in real time, and hence, it is not required in the claim to limit it to real-time usage only, as the Examiner commented.
The Applicant explained that the repeatedly loop comprising the steps of:
obtaining camera output;
extracting user measurements;
adapting at least one of the activities...;
Then, separately, the claim includes the step of:
determining one or more combined measurements based on the analyzed measurements.
Amended claim 1 further clarifies that the combined measurements are aggregated over the duration of a session of the experience.
In the context of the claim and as further described in the specification, e.g., at paragraph [0172], the system performs real-time analysis during the experience to enable adaptation of activities. In addition, a later analysis, performed after the experience is completed, combines the results of prior experience analyses. The purpose of this higher-level performance analysis after the experience is completed is to obtain parameters relating to human skills in order to determine a level of performance of the skills over the entire experience.
From Applicant’s specification…
[0172] In some cases, after the running of a session of experience is complicated, analysis module 224 can determine one or more combined measurements based on the analyzed measurements (block 360). In some examples, analyzing the measurements performed while the user participating the game is performed for adapting the experience during its run, such that the experience that is running, is reacting to the current performance of the user 110. In some examples, an additional later analysis is performed after the experience is completed. One purpose of the later analysis is to obtain parameters relating to human skills, and to derive performance metrics (that cover therapeutically relevant aspects (like movement fluency), so, later, the level of performance of the skills can be determined.
From Claim 1…
“adapting at least one of the activities in the experience, dynamically and in real-time, based on at least the analyzed measurements;”
From Van Andel…
Feedback and perform immediately (dynamically and in real-time)…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc. The system may capture movement of a target body part, such as the limb. Data collected from the captured movement may then be compared to a reference set, for example including range of motion capabilities of the limb (e.g., based on patient specific data, such as height, gender, etc.). The reference set may be gathered by capturing similar movements performed by the patient's mirroring target body part (e.g., when the patient is having an issue with a right leg, then a reference set of data may be collected from capturing movement of the patient's left leg). The reference set may include compiled data collected from other people who share traits similar to the patient, such as gender, weight, or height. In another example, the reference set may include data created using machine learning techniques, based on clinical data, or based on a machine-determined range of motion. A machine learning system may generate ideal or optimized data for the reference set based on patient specific parameters, such as hip to knee measurement, knee to ankle measurement, etc. The comparison between the patient's data and the reference set may determine a difference set. An assessment for the patient may then be determined based on the difference set. The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
Another example of “task may be altered in response to determining that the data indicates proper form is decreasing…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like. In an example, a user may be tasked with performing twenty repetitions of a leg lift. Utilizing the sensor 445, such as an accelerometer, a gyroscope, or a nine-axis sensor, the movement of the leg lift may be tracked. The sensor 445 may count the number of leg lift repetitions performed by the user or monitor the form of a leg lift. The sensor 445 may be used to determine if the user is experiencing fatigue or having difficulty completing a repetition or a task. For example, data from the sensor 445 may indicate that a adherence to form is decreasing for subsequently performed leg lifts. Tasks or aspects of a task (e.g., intensity or duration) may be altered in response to determining that the data indicates the proper form is decreasing. In an example, the data may be analyzed to determine the user's level of fatigue from performing repetitions.” [0046]
`Thus, the "combined measurements," relied upon both the user measurements and the activity measurement, are not merely a capture of raw sensor data or joint motion information, as in Van Andel in "bending knees," as referred to by the Examiner, but instead reflect an analysis of multiple types of analyzed measurements, collected across different iterations of the experience's session. Paragraph [0183] and following paragraphs illustrate examples of the combined measurements.
The above argument is not commensurate with the scope of Claim 1 (see above).
Analysis of both the user measurements and the activity measurements is an important and advantageous step that involves combining and analyzing data from two distinct domains: the user's physical movement, as extracted from the captured video, such as physical motions, facial expressions, or internal functions) and the dynamics and execution of the virtual experience- what happened in the activity itself (such as game triggers, pauses, or collision events).
The above argument is not commensurate with the scope of Claim 1 (see above). There is no claimed analyzing data from two distinct domains.
This joint analysis, which is not taught by Van Andel's singular focus on raw physical motion data (user's kick) for assessing limb mobility, allows the system to derive higher-level performance metrics and interpret outcomes. For instance, combining the occurrence of a stressful game event ("activity measurement") with a user's lack of movement or anxious facial expression ("user measurement") allows the system to analyze the user's stress tolerance or fear index in a much higher depth than the physical success of the Van Andel's user to kick.
The above argument is not commensurate with the scope of Claim 1 (see above). There is no claimed joint analysis and combining stressful game event as argued above.
The Examiner relied upon Van Andel's sensor 355, which identifies a
sensor attached to a physical equipment (weights) that counts repetitions. The sensor 355 merely tracks physical equipment or counts quantitative body repetitions, which constitutes a user-centric metric of limb mobility rather than an extraction of experience measurements including data pertaining to the to the execution of the session of the interactive virtual experience itself.
Session is not defined. Experience is not defined. These are given their broadest reasonable interpretation. A time period could be a session. A number or duration of an activity could be a session. Some kind of activity could be experiences.
3. Real-time dynamic feedback loop and adaptation of the activity: The system utilizes a "repeatedly executing" loop to adapt activities while the user participates. The experience itself forms a dynamically running computerized environment. Based on the user's and the activity's measurements, the system adapts the activities in the experience, in a dynamic manner and in real-time.
Loop is not taught in Applicant’s specification.
“Dynamically running computerized environment” is not taught in Applicant’s specification.
Example according to claim 1: automatically replacing an intense virtual
background with a calmer one if the system detects user stress in real-time (see paragraph [0112]).
Neither the "feedback" or the "adaptation" of the prior art cannot correspond to the dynamic adaptation of activities in the experience of claim 1, since the feedback and the adaptation (removing a virtual circle) are limited and in response to the user's performance only. Van Andel utilizes a "movement capture apparatus" primarily to capture "movement information from a patient." This is explicitly mentioned in the paragraph relied upon by the Examiner. As such, the adaptation is dependent on the user's movement only. The prior art lacks the extraction of activity measurements, data pertaining to the execution of the experience itself, and hence, cannot adapt the experience as defined by claim 1 ("based on at least the analyzed measurements"). Amended claim 1 defines that adapting the activities in the experience, is done in a dynamic manner and in real-time, based on at least the analyzed measurements. Recall, the analyzed measurements include both the user activity measurements as well as the user's measurements.
Van Andel teaches:
Lifting the right leg (user activity) and knee bent at 90 degree angle (user measurement)…
“FIG. 1A and FIG. 1B illustrate real and virtual aspects of an augmented reality diagnosis system in accordance with some embodiments. In an example, the system 100 includes a video capture device 105, a display device 107, and AR glasses 110 worn by a user 115. The display device 107 may display an avatar 120 to demonstrate the movements the user 115 should perform. In an example, the avatar 120 is demonstrating the movement of lifting the right leg with the knee bent at a 90 degree angle. The demonstrated bent knee movement is also being performed by the user 115. The user 115 may perform a movement, as instructed, which is captured by the video capture device 105. In an example, a movement may be performed by the user 115 with one or more target body parts, and the movements may be captured by the video capture device 105. In an example, the video capture device 105 may be a camera, a depth camera, or an infrared detection device, such as the Kinect from Microsoft of Redmond, Wash.” [0030]
Sensors to track movement (activity measurements) of the user…
“In the example, the user 115 in system 100 may wear an apparatus to capture the motion of the target body part. In an example, a leg sleeve 125 is worn on the right leg in the area of the knee for user 115. The worn apparatus may be any type of apparatus that may be secured to a user's body and allow movement (e.g., a sleeve, glove, sock, or wristband). The worn apparatus may contain a sensor for tracking or capturing the movement of the user 115 or the target body part. The sensor utilized may include one or more of an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like.” [0033]
The technical advantageous over the cited prior art lies in enabling the system to respond immediately to indications derived from the combined analysis of internal activity metrics and user state (e.g., replacing an intense virtual background with a calmer one if the analyzed measurements indicate stress). While Van Andel discloses an interaction where a virtual target is "removed" (e.g., fades out or explodes) in response to a user interaction (e.g., kicks the circle), this response a consequence of task completion by the user. It is not a continuous, dynamic modification of the activity or its parameters in real-time based on the analysis of combined data streams (based on both the user and the activity).
The above argument is not commensurate with the scope of Claim 1 (see above). There is no claimed respond immediately with internal activity metrics and user state such as replace intense virtual background with calmer one.
Van Andel teaches:
Raised leg (user activity) with bent knee (user state)…
“In system 100, the user 115 may wear the AR glasses 110. In an example, view 150 may represent a view the user 115 sees while wearing the AR glasses 110. The view 150 may include virtual and real aspects in an environment, such as the user's leg (real) augmented by a guide (virtual). In an example, the user 115 may look downward to observe the view 150 and the user may see the raised leg 135 with bent knee. The view 150 using the AR glasses 110 may include a virtual object or goal for the target body part to reach or extend to, such as a virtual line 140. The virtual line 140 may represent an extension distance for the target body part to achieve during a movement. The AR glasses 110 may display virtual aspects in a real environment, such as in view 150.” [0034]
Moreover, Van Andel describes a "therapy program" where assessment data is typically produced after the user completes a scheduled session, which leads to "post therapy adjustment" of a clinical plan by a clinician, not activity adaptation during the experience. This real-time adaptation of the running activity, based on comprehensive combined measurements, distinguishes the subject matter of claim 1 from Van Andel. Applicant notes that the Examiner contended on page 11 that Van Andel's disclosure of tracking repetitions and increasing weights teaches real-time analysis and the "dynamic modification of activities." However, the Applicant emphasizes that this does not teach "adapting" the activity of the experience as defined in Claim 1. Van Andel merely describes a scenario where a user or clinician manually changes the physical exercise parameters, by adding physical weight, and the system simply receives and identifies this modified activity state determined by the user. This manual adjustment of physical equipment represents a change to a clinical protocol rather than the system autonomously adapting the execution of the experience or virtual environment of a running activity based on analyzed measurements.
From Van Andel and perform immediately…
“… The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
Dependent claim 3:
Claim 3 specifically requires that activity measurements are collected,
where at least one such activity measurement is not related to success in completing the goals of the experience, but is nevertheless used to determine the user's level of performance. As illustrated in paragraph [0146] of the application as published, the system extracts data independent of and unrelated to mission success, to assess the level of human skills. For instance, if a user is tasked with collecting blue rocks (the goal), the system may extract the number of red rocks collected as activity measurement. While a standard game would record this merely as a failure, the claimed subject matter utilizes this interaction with the virtual elements in the experience to evaluate cognitive performance, such as color discrimination or concentration.
Van Andel does not describe extracting any such activity measurements. Moreover, the cited paragraph [0019] relates solely to capturing patient movement data (i.e., user measurements) and comparing those movements to a reference set in order to assess limb mobility. This reference set of movements was gathered by capturing similar movements performed by the patient's mirroring target body part, such as when the patient uses their other leg.
Applicant is arguing things such as extracting, which is not claimed. Van Andel teaches reference data for determining level of performance. This is not part of data used for success of a user completing goals.
The Examiner's reliance on Van Andel paragraph is unclear. This analysis is limited to evaluating whether a movement was performed correctly and does not involve extracting data from an activity or experience itself. Even if, arguendo, the Examiner interprets this analysis as involving data beyond the desired physical movement, it still does not involve extracting or analyzing information from the activity or virtual experience itself, as defined in Claims 1 and 3. Therefore, Van Andel does not disclose the subject matter of claim 3.
There is no claimed “the activity” but just “at least one activity” which is any activity. Therefore reference data is any activity measurement.
Dependent claims 30 and 31:
New dependent claim 30 is added, reciting that the one or more human skills comprises at least one skill selected from physical skills, cognitive skills, behavioral skills, social skills, and mental abilities, and new claim 31 depending from claim 30 is added, reciting that the level of performance is determined for at least two different human skills. The analysis and assessment of these skills distinguish the claimed subject matter from Van Andel, which is narrowly confined to assessing limb mobility and physical range of motion (physical skill only).
A rejection is provided for the new claims.
Applicant argues 35 USC §101Rejection, starting pg. 23 of Remarks:
Subject Matter Eligibility
Step 2A -Prong 1
As presently amended without prejudice or disclaimer, the independent claims are not directed toward an abstract idea, as can be seen by analogy to eligible claim 4 of USPTO Subject Matter Example 46, which is directed toward collection of animal-specific information within a herd including body position data, feeding behavior data, and movement pattern data:
4. A system for monitoring health and activity in a herd of dairy livestock animals comprising:
a memory;
a processor coupled to the memory programmed with executable instructions, the instructions including a livestock interface for obtaining animal-specific information for a plurality of animals in the herd, wherein the animal-specific information comprises animal identification data and at least one of body position data, body temperature data, feeding behavior data, and movement pattern data; and
a herd monitor including
(a) a radio frequency reader for collecting the animal-specific information from a plurality of animal sensors attached to the animals in the herd when the animal sensors are within proximity to the radio frequency reader, each animal sensor having a radio frequency transponder, and
(b) a transmitter for transmitting the collected animal-specific information to the livestock interface.
Just as claim 4 of Subject Matter Example 46 recites collecting, in a specific way, animal-specific information such as body position data, body temperature data, feeding behavior data, and movement pattern data, the Applicant's invention is directed toward using a computerized environment to collect, in a specific way, user-specific information including user measurements and activity measurements (in specific embodiments, the user measurement including, for example, body motion, facial muscles and expressions, or internal functions as recited in dependent claims).
The above is directed to animal livestock. The claims were abstract as directed to certain methods of organizing human activity.
In particular, the Applicant has amended the independent claims without prejudice or disclaimer in a manner that defines the claimed invention as not being directed to an abstract idea, but, rather, as being directed to collection of user information in a very specific way, inter alia, by:
providing an interactive experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience;
repeatedly obtaining a video of the user performing user actions; extracting user measurements from the user actions (such as, for example, body motion, facial muscles and expressions, or internal functions as recited in dependent claims); and
extracting activity measurements including data pertaining to the execution of the experience itself and user interaction within the experience.
More specifically, the present amendments to the independent claims define the independent claims, inter alia, as being directed to providing an interactive experience "forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience," repeatedly obtaining a video of the user performing user actions, extracting user measurements from the user actions, the activity measurements extracted from the provided interactive experience being defined by the present amendment as "comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience."
In the Examiner's response to the arguments in the Office action, the Examiner maintained that the Applicant's claimed invention was directed to an abstract idea, contending:
Respectfully, the claims require a user to participate to determine level of performance. Further, a person can watch a video and mentally extract measurements by estimating in their mind distance, etc., analyze measurements, etc. from user's actions. There is no claimed computer actually performing the steps.
There are similar remarks by the Examiner in the response to arguments section of the Office action to the effect that various other steps do not require a computer.
But, as is discussed above, the present amendments to the claims define the specific, computerized, way that user information is collected as including "forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience," repeatedly obtaining a video of the user performing user actions, extracting user measurements from the user actions, the activity measurements extracted from the provided interactive experience "comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience."
Thus, as presently amended, the Applicant's independent claims do recite steps that can only be performed by computerized equipment. Although a computer in itself is generic, as was noted by the Examiner in the response to arguments section of the Office action, claims reciting use of a generic computer to perform sequences of steps that do not amount to an abstract idea and to which the claims are directed are subject- matter eligible as is demonstrated by claim 4 of Subject Matter Example 46 discussed above.
From Example 46, Claim 4 (pg. 41) and the reason why the Claim 4 was not abstract:
“There is no exception recited in the claim. The claim does not recite any abstract ideas, such as a mathematical concept, mental process, or a method of organizing human activity such as a fundamental economic concept or managing interactions between people. The system’s operation, like all computers, is based on mathematical theory, but that underlying operation does not trigger an eligibility analysis because it is not set forth or described in the claim. Similarly, while the claim involves the observation of natural phenomena or laws of nature (the behavior of the livestock animals), such limited involvement does not rise to the level of this claim actually reciting a natural phenomenon or law of nature. See MPEP 2106.04(II) and October Update at Section I(A) for more information about what “recite” means.”
Claim 4 of Example 46 is “[a] system for monitoring health and activity in a herd of dairy livestock animals…” (preamble). From above, there were not abstract concepts claimed. For example, the claims were not directed to certain methods of organizing human activity as dairy livestock animals are not humans. Applicant’s claims are not monitoring a herd of dairy animals’ skills but rather are “[s]ystem and method for assessing and facilitating improvement of human skills” (title of the instant application). This requires interaction with a human, presumably to improve their skills.
Thus, it cannot be said that the Applicant's independent claims as amended are "directed to" an abstract idea. The heart of the Applicant's inventive concept as defined by the Applicant's claims as amended lies in the specific way that user-specific information is collected (including, in specific embodiments, user measurements such as, for example, body motion, facial muscles and expressions, or internal functions as recited in dependent claims), by forming a dynamically running computerized environment comprising virtual elements configured to generate particular kinds of data and extracting user measurements and activity measurements.
Specific embodiments are not the requirement for an “additional element.” Further, even if some of the steps as claimed were not abstract, they are at a high level.
Moreover, the formation of a dynamically running computerized environment comprising virtual elements configured to generate particular kinds of data and extracting user measurements and activity measurements, the user measurements including data pertaining to the execution of the experience itself and the user interaction within the experience, is much more than merely "managing personal behavior," as is recited by the Examiner in the response to arguments section of the Office action in characterizing certain claim limitations prior to the present claim amendments. Rather, the claimed sequence of steps of the Applicant's independent claims is directed toward a system for collecting data about activity of users in a specific way, just as claim 4 of Subject Matter Example is directed toward a system for monitoring health and activity in a herd of dairy livestock animals by collecting data in a specific way.
Using a generic computer at a high level of generality to perform a judicial exception as claimed is not enough. The “dynamically running computerized environment” is very high level.
Respectfully, collecting data to manage a herd of animals is not the same as organizing human activity by managing personal behavior.
Nor are the Applicant's claims as amended directed, as the Examiner contends, toward a mental process, since steps such as "forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience" cannot be practically performed in the human mind. The August 4, 2025 Office memorandum entitled "Reminders on evaluating subject matter eligibility of claims under 35 U.S.C. 101" warns:
The mental process grouping is not without limits. Examiners are reminded not to expand this grouping in a manner that encompasses claim limitations that cannot practically be performed in the human mind.
Furthermore, analyzing the executing of a digital experience (activity measurements) with or without the combination with user measurements is not something a human can do with his or her mind. These operations require digital analysis, an extraction of digital data from the execution of the experience, analyzing the measurements together in a computational manner, which is not practical for the human mind.
Moreover, extracting experience measurements is computer-specific operations (MPEP §2106.04(a)(2)(III)(A) - claims do not recite mental processes where "the human mind is not equipped" to perform the limitations (citing SRI, SiRF, Research Corp.).
The activity measurements relate to internal virtual game parameters, such as errors, retries, object selection, or timing, which are not visible or accessible to a human observer. These data must be extracted and processed by the computing system in real time - it cannot be done by a human with pen and paper.
Also, the claims require structured, real-time analysis of user and activity data during an interactive virtual experience. Unlike mere data gathering or display, the system dynamically and in real time adapts the e based on the analyzed measurements, enabling a feedback loop that would not be possible without specialized computation.
The above steps though are at a high level of generality. However, based on Applicant’s arguments regarding mental processes and further consideration, this mental process argument is withdrawn. The claims remain abstract as managing personal behavior under certain methods of organizing human activity.
Step 2A -Prong 2
Although the Applicant's independent claims recite certain steps involving analyzing the measurements, adapting activities of the experience dynamically and in real time based on the analysis, and determining a level of performance of one or more human skills based on the one or more combined measurements, these steps do not transform the Applicant's claimed invention into being "directed to" an abstract idea because even assuming arguendo that abstract ideas are recited in these additional steps, such ideas would be, under Step 2A, Prong 2, integrated into the above-described practical application of collecting, in a specific way, user-specific information by providing an interactive experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data.
Considering MPEP §2106.04(d):"a claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception, "the amended claim is integrated into a real-world interactive system for evaluating and improving human skills.
As is discussed above, the independent claims recite specific technological components and steps, such as a camera for obtaining video of a user in a virtual experience, and automated extraction of user measurements and activity measurements. These operations are not generic data step, but, rather, a practical integration.
The Applicant refers once again to the MEMORANDUM of August 4, 2025 which emphasizes that "the additional limitations should not be evaluated in a vacuum, completely separate from the recited judicial exception. Instead, the analysis should take into consideration all the claim limitations and how these limitations interact and impact each other when evaluating whether the exception is integrated into a practical application. " These limitations of claim 1 impose meaningful constraints on any alleged abstract concept. The claim does not monopolize or broadly cover the idea of human skill evaluation. Instead, it is limited to a specific computer-implemented method involving a virtual experience, real-time data capture, and adaptive interaction. The limitations work together to form a closed technical loop that dynamically analyzes user and activity measurements to determine skill progression. This integration of the alleged abstract idea into a responsive computing environment reflects a practical application, as emphasized in the August 4, 2025 Memorandum.
Using generic technology for a judicial exception as claimed is not enough to make abstract claims statutory.
Step 2B -Claims Amount to "Significantly More"
The independent claims are also patent-eligible under Step 2B because the claims recite a non-conventional and non-generic arrangement of computing operations. The claims define a specific ordered combination: providing an interactive virtual experience, capturing real-time user actions via video, extracting user and activity measurements, analyzing those data, dynamically adapting the experience, and determining level of performance based on combined measurements. These are not generic data manipulations or mere automation of human judgment. Instead, these actions reflect a technical solution that requires integrated sensor input, system adaptation, and structured data analysis. As in BASCOM, the inventive concept lies not in each individual component but in the claimed interaction between them to support real-time, personalized evaluation of human skills in a dynamic and adapted experience.
BASCOM improved computer technology. Applicant is using existing technology for managing personal behavior. Based on the above response, the rejection is respectfully modified but maintained.
Claim Interpretation
Applicant claims “dynamically running computerized environment” where there is no specific teaching of this in the claim.
Applicant’s specificaoin also teaches:
“The game may be provided to the user as part of a treatment plan formed for the user based on a set of one or more skills that were identified as requiring improvement. Based on the user's level of performance, and as the user makes progress in the game, the treatment plan, the experience (e.g., the game) or activities in the experience, can be updated and adapted to the user's level and progress, resulting in enhancing the overall improvement of the skills of the user.” [0016]
“Human skills may refer to skills pertaining to motor, cognitive, social, emotional, language, and adaptive skills. Some examples of human skills may include communication, self awareness, creative-thinking, resilience, understanding nonverbal cues, growth mindset, gross motor, fine motor, visual perception and processing, motor planning and control, executive functioning, and other skills. Consider an example of an individual struggling with the skill of resilience. The individual may struggle to recover quickly and effectively after experiencing failures. Enhancing the resilience skill may involve the ability of the individual to effectively manage stress and pressure. An example of an experience to assist the individual in enhancing his resilience skill may involve a game featuring a virtual environment inspired by nature or the wilderness, where the lower level of difficulties of the game features a calm nature, but, as the user makes progress, the nature may be featured with a more intimidating elements, gradually forcing the user to confront them. Monitoring measurements pertaining to the user and the game during the user's participation may assist in obtaining data on the level of performance of the user of the skill of resilience, if the user makes progress.” [0095]
“Extraction module 223 can also extract data pertaining to at least one activity measurement from the provided experience (block 340). For example, extraction module 223 can receive data on execution of the game, and analyze the execution to extract activity measurements pertaining to measuring one or more of performance, goals or events within the executed game, or a combination thereof. For example, the following data can be extracted: one or more activity measurements relating to continuous running of the experience, e.g. how many times the game was paused or how many times a different level was selected by the user while another level is running, data pertaining to the trajectory of game character, such as location, speed, and movement direction of an avatar, or a game character over time, collision or collection of interactive elements, other triggers that affect the game (cut scenes, pauses, etc.), status of the game elements that can affect the user, for example, status of elements that induce stress, distraction, etc. For illustration, consider that the experience includes a scripted or randomly activated game element that triggers stress for the user 110. This could be e.g., time pressure, by stating “your oxygen level is critically low, replenish withing 10s”. Another example in the experience is a “sand storm” which reduces the visibility for the user 110 and makes navigation more challenging. Activity measurements can include parameters pertaining to these elements, such as the time that they were in certain status, duration in each status, etc.” [0145]
Therefore, a game or activity is interpreted as an example of a dynamically running computerized environment.
Applicant claims “determining one or more combined measurements based on the analyzed measurements wherein the combined measurements are aggregated over the duration of the session of the experience;” where no teaching of aggregated over the duration of the session of the experience is taught.
Applicant also teaches:
“In some examples, the later analysis including determining the combined measurements is performed with respect to the following parameters. Some of these parameters are also referred to during the analysis performed while the user participated in the experience: … Aggregated: over multiple experiences (e.g., improvement, total exercise time)” [0173], [0177]
Therefore, this is interpreted as data can be aggregated over an exercise time, as an example.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-12 and 14-31 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-12, 14-26, 28, and 30-35 of copending Application No. 18/610856 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other because the claims are broader in the instant claims over their copending application and/or recite similar limitations.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 and 14-31 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1-12 and 14-31 are directed to a method, system or product, which are statutory categories of invention. (Step 1: YES).
The Examiner has identified method Claim 1 as the claim that represents the claimed invention for analysis and is similar to system claims 26 and 28, and product claim 27.
Claim 1 recites the limitations of:
A computer-implemented method for determining performance of human skills of a user, the method comprising:
providing on a user device an interactive virtual experience in which the user can participate, the experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience;
while the user participates in the session of the experience, repeatedly executing:
obtaining camera output comprising a video of the user performing one or more user actions;
extracting one or more user measurements from the one or more user actions;
extracting one or more activity measurements from the provided experience comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience;
analyzing the one or more user measurements and the one or more activity measurements to obtain analyzed measurements;
adapting at least one of the activities in the experience, dynamically and in real-time, based on at least the analyzed measurements; and
determining one or more combined measurements based on the analyzed measurements wherein the combined measurements are aggregated over the duration of the session of the experience; and
determining level of performance of one or more human skills based on the one or more combined measurements.
These above limitations, under their broadest reasonable interpretation, cover performance of the limitation as certain methods of organizing human activity. The claim recites elements, in non-bold above, which covers performance of the limitation as managing personal behavior, including teaching, following rules or instructions. Providing a virtual experience in which a user can participate with continuous user participation of a session (following rules and instructions) and while the user participates in the session of the experience (user participation, therefore following rules and instructions), obtaining a video of the user performing user actions, extracting user measurements of user actions, extracting activity measurements form a provided experience pertaining to the execution of the experience itself and the user interaction within the experience, adapting activities in the experience based on analyzed measurements, and determining level of performance of one or more human skills based on measurements (teaching). If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation as managing personal behavior, then it falls within the “Certain Methods of Organizing Human Activity” grouping of abstract ideas. Accordingly, the claim recites an abstract idea. Claims 1 and 19 are also abstract for similar reasons. (Step 2A-Prong 1: YES. The claims are abstract)
This judicial exception is not integrated into a practical application. In particular, the claims only recite: computer, user device, camera (Claim 1); computer, user device, camera (claim 26); non-transitory memory, computer, user device, camera (Claims 27); user device, computer, camera (Claim 28). The computer hardware is recited at a high-level of generality (i.e., as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. The camera can be part of the computing device, such as a smartphone (see specification, pg. 14, lines 22-31 to pg. 15, lines 1-9). The video can be just digital data or information. Accordingly, these additional elements, when considered separately and as an ordered combination, do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Therefore claims 26-28 are directed to an abstract idea without a practical application. (Step 2A-Prong 2: NO. The additional claimed elements are not integrated into a practical application)
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered separately and as an ordered combination, they do not add significantly more (also known as an “inventive concept”) to the exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a computer hardware amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Accordingly, these additional elements, when considered separately and as an ordered combination, do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Steps such as obtaining (receiving) are steps that are considered insignificant extra solution activity and mere instructions to apply the exception using general computer components (see MPEP 2106.05(d), II). Thus claims 26-28 are not patent eligible. (Step 2B: NO. The claims do not provide significantly more)
Dependent claims 2-12, 14-25, and 29-31 further define the abstract idea that is present in their respective independent claims 1 and 28 and thus correspond to Certain Methods of Organizing Human Activity and hence are abstract for the reasons presented above. The dependent claims do not include any additional elements that integrate the abstract idea into a practical application or are sufficient to amount to significantly more than the judicial exception when considered both individually and as an ordered combination. Claim 22 recites a generic camera that is part of a mobile device. The other claims 2-12, 13-21, 23-25, and 29 are themselves abstract under certain methods of organizing human activity and/or mental processes. Claim 4 adjusts audio output or changes visual appearance of elements, which are recited at a high level of using computer devices to do what computer device do. Claims 6 and 16 recite avatar mimicking at least one user action or illustrate a more accurate version of body motion, which are recited at a high level of generality, and would be providing feedback to a user for teaching/training purposes, itself abstract. Therefore, claims 2-25 and 29 are directed to an abstract idea. Thus, the claims 1-12 and 14-31 are not patent-eligible.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-12 and 14-31 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites “providing on a user device an interactive virtual experience in which the user can participate, the experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience” where no teaching of virtual elements generate quantitative data can be found in the specification.
From Applicant’s specification…
[0145] Extraction module 223 can also extract data pertaining to at least one activity measurement from the provided experience (block 340). For example, extraction module 223 can receive data on execution of the game, and analyze the execution to extract activity measurements pertaining to measuring one or more of performance, goals or events within the executed game, or a combination thereof. For example, the following data can be extracted: one or more activity measurements relating to continuous running of the experience, e.g. how many times the game was paused or how many times a different level was selected by the user while another level is running, data pertaining to the trajectory of game character, such as location, speed, and movement direction of an avatar, or a game character over time, collision or collection of interactive elements, other triggers that affect the game (cut scenes, pauses, etc.), status of the game elements that can affect the user, for example, status of elements that induce stress, distraction, etc. For illustration, consider that the experience includes a scripted or randomly activated game element that triggers stress for the user 110. This could be e.g., time pressure, by stating “your oxygen level is critically low, replenish withing 10s”. Another example in the experience is a “sand storm” which reduces the visibility for the user 110 and makes navigation more challenging. Activity measurements can include parameters pertaining to these elements, such as the time that they were in certain status, duration in each status, etc.
Therefore, from above data pertaining to an avatar can be extracted. Claims 26-28 have a similar problem.
Claims 2-12, 14-25, and 29-31 are further rejected as they depend from their respective independent claims.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-12 and 14-31 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “providing on a user device an interactive virtual experience in which the user can participate, the experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience” where it is indefinite as to a virtual element configured to generate quantitative data. A virtual element such as an avatar, falling rocks, etc. can be represented by pixels on a display device, where the pixels are controlled by data addressees. It’s indefinite as to how virtual elements can generate data. For examination purposes, this is interpreted as computer or processor generating data. Claims 26-28 have a similar problem.
Claims 2-12, 14-25, and 29-31 are further rejected as they depend from their respective independent claims.
Examiner Request
The Applicant is requested to indicate where in the specification there is support for amendments to claims should Applicant amend. The purpose of this is to reduce potential 35 U.S.C. §112(a) or §112 1st paragraph issues that can arise when claims are amended without support in the specification. The Examiner thanks the Applicant in advance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-12 and 14-31 are rejected under 35 U.S.C. 103 as being unpatentable over Pub. No. US 2018/0315247 to Van Andel in view of Pub. No. US 2011/0098928 to Hoffman et al. and in further view of Pub. No. US 2023/0021339 to Bosnak et al.
Regarding claims 1 and 26-28
(claim 1) A computer-implemented method for determining performance of human skills of a user, the method comprising:
Van Andel teaches:
Smartphone as a user device with a camera…
“The patient may be presented with an option of performing the therapy program at home, for example by utilizing a personal AR device, a smartphone with camera, a video capture device such as a Kinect from Microsoft of Redmond, Wash., or the like.” [0029]
Patient to perform movements to accurately portray (determine) the movement capabilities of the target body part (performance of human skills)…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022]
providing on a user device an interactive virtual experience in which the user can participate, the experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience;
Van Andel continues teaching:
Personal AR (user augmented reality, therefore, virtual) device for performing therapy program (activities), where program provides interactive experience comprising activities…
“…The patient may be presented with an option of performing the therapy program at home, for example by utilizing a personal AR device, a smartphone with camera, a video capture device such as a Kinect from Microsoft of Redmond, Wash., or the like.” [0029]
Example of user kicks virtual target (interactive virtual experience)…
“The AR display 810 may display the AR environment overlaid on the real environment. The display AR 810 may show the virtual target, using the AR device 801, in the fixed position in the AR environment. In an example, the AR device 801 may remove the virtual target from display in the AR environment in response to detecting a user interaction with the virtual target. For example, when the virtual target is a line or circle, the virtual target may be removed (e.g., fade out, popped, explode, etc.) when a user interacts with the virtual target (e.g., kicks the circle, moves a body part past the line, etc.).” [0059]
Sports activity with before sports activity (sessions of the experience) with avatar in a dynamically running computerized environment and produce movement set (generate quantitative data)…
“The movements of the user 605 may be captured through a camera or sensor attached to the user 605 or to a sports instrument, such as a golf club or baseball bat. The user 605 may be instructed, such as using an avatar 615 on a display or within an AR field of view in the first direction 610, to perform a movement related to the predetermined sports movement. In an example, three-dimensional data points of the user's movements are captured as a measurement set. The three-dimensional data points of the measurement set may be classified to produce a movement set which includes the range of motion for the target body parts related to the sports movement. A reference set corresponding to the sports movement may be obtained. In an example, the reference set may include three-dimensional data points captured from multiple professional athletes of the corresponding sport. A different set may be computed wherein a member of the difference set defines a three-dimensional relationship between a member of the movement set and a corresponding member of the reference set. An assessment for how the user 605 may improve performance may be determined based on the difference set. The assessment may include a physical therapy program, a stretch the user performs before the sports activity, or a movement or strengthening technique the user 605 is guided through by the avatar 615.” [0053]
Augmented reality device with processor…
“FIG. 8 illustrates an augmented reality diagnosis system 800 in accordance with some embodiments. The augmented reality diagnosis system 800 includes an augmented reality device 801, and may include a display device 812. The augmented reality device 801 includes a processor 802, memory 804, a camera 806, and an AR display 810. The augmented reality device 801 may include a feedback controller 808 or a sensor 816. The augmented reality device may be connected to the display device 812, which may display video or other aspects. In an example, the augmented reality device 801 may be a HoloLens manufactured by Microsoft of Redmond, Wash. The processor 802 of the augmented reality device 801 includes an augmented reality modeler 818. The augmented reality modeler 818 may be used by the processor 802 to create the augmented reality environment. For example, the augmented reality modeler 818 may receive dimensions of a room, such as from the camera 806 or sensor 816, and create the augmented reality environment to fit within the physical structure of the room. In another example, physical objects may be present in the room and the augmented reality modeler 818 may use the physical objects to present virtual objects in the augmented reality environment. For example, the augmented reality modeler 818 may use or detect a table present in the room and present a virtual object as resting on the table. In an example, a plurality of physical items may be used by a user to interact with the virtual target. The virtual object may be used by a user to complete a physical therapy movement.” [0057]
Example of machine (computer) with created data….
“… In another example, the reference set may include data created using machine learning techniques, based on clinical data, or based on a machine-determined range of motion. A machine learning system may generate ideal or optimized data for the reference set based on patient specific parameters, such as hip to knee measurement, knee to ankle measurement, etc. The comparison between the patient's data and the reference set may determine a difference set. An assessment for the patient may then be determined based on the difference set. The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
See Dynamic below.
while the user participates in the session of the experience, repeatedly executing:
obtaining camera output comprising a video of the user performing one or more user actions;
Fig. 1A, ref. 107 teaches example of camera (ref. 105) output of movement (video) of a user performing an action…
PNG
media_image1.png
340
484
media_image1.png
Greyscale
Video capture (camera) device…
“FIG. 1A and FIG. 1B illustrate real and virtual aspects of an augmented reality diagnosis system in accordance with some embodiments. In an example, the system 100 includes a video capture device 105, a display device 107, and AR glasses 110 worn by a user 115. The display device 107 may display an avatar 120 to demonstrate the movements the user 115 should perform. In an example, the avatar 120 is demonstrating the movement of lifting the right leg with the knee bent at a 90 degree angle. The demonstrated bent knee movement is also being performed by the user 115. The user 115 may perform a movement, as instructed, which is captured by the video capture device 105. In an example, a movement may be performed by the user 115 with one or more target body parts, and the movements may be captured by the video capture device 105. In an example, the video capture device 105 may be a camera, a depth camera, or an infrared detection device, such as the Kinect from Microsoft of Redmond, Wash.” [0030]
Another example…
“…In an example, a display screen 205 presents captured video (e.g., video captured of the movement performed by the user 225 or of the user 225). A virtual guide avatar 215 may be presented on the display screen 205. The virtual guide avatar 215 may be a computer generated figure, an animated figure, or a recording of a human instructor. The virtual guide avatar 215 may instruct the user 225 through a movement to perform (e.g., a speed of movement, where to extend a limb, how far to bend a joint, etc.)…” [0036]
Fig. 2, ref. 200…
PNG
media_image2.png
354
446
media_image2.png
Greyscale
extracting one or more user measurements from the one or more user actions;
Capture (extracting) movement of body part with data (measurements) of limb (user measurements) from movement (actions)…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc. The system may capture movement of a target body part, such as the limb. Data collected from the captured movement may then be compared to a reference set, for example including range of motion capabilities of the limb (e.g., based on patient specific data, such as height, gender, etc.). The reference set may be gathered by capturing similar movements performed by the patient's mirroring target body part (e.g., when the patient is having an issue with a right leg, then a reference set of data may be collected from capturing movement of the patient's left leg)….” [0019]
Movement creates measurement set…
“The captured movement of the target body part creates the measurement set. The measurement set includes of three-dimensional data points for the movement performed by the target body part. The three-dimensional data points may be captured with video using a movement capture apparatus such as a camera or a Kinect from Microsoft of Redmond, Wash. The patient may have one or more sensors attached to the target body part, which may supply information about the movement of the target body part or its position in three-dimensional space. The one or more sensors may be attached directly to the patient or may be part of a worn apparatus such as a sleeve the patient places around an arm or leg. The one or more sensors may include one or more of an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a multi-axis sensor, or a contact pressure sensor. In an example, a patent may wear a sleeve that extends from an upper thigh to a lower calf. The sleeve may have one or more sensors located above the knee or below the knee such that the one or more sensors may collect data about the movement of the patient's knee joint.” [0021]
extracting one or more activity measurements from the provided experience comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience;
Perform more movements (plural), such as bending knee and extending an arm, and measurement set (therefore includes activity measurements)…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022]
Example 1 and series of motions…
“Example 1 is a method comprising: capturing three-dimensional data points of a target body part of a subject during a series of motions, the target body part being mobile relative to a reference point of the subject, wherein the data points are members of a measurement set; classifying members of the measurement set to produce a movement set, members of the movement set including range of motion information for the target body part; obtaining a reference set corresponding to the target body part, the reference set defining range of motion capabilities for the target body part; computing a difference set, a member of the difference set defining a three-dimensional relationship between a member of the movement set and a corresponding member of the reference set; determining an assessment for the target body part based on the difference set; and outputting the assessment.” [0077]
Example 34 and include movement data normative to physical activities of the subject (therefore, both motion/movement and activities)…
“In Example 34, the subject matter of any one or more of Examples 1-33 optionally include wherein the reference set includes collected movement data normative to the physical activities of the subject.” [0110]
Sports activity with before sports activity (sessions of the experience) with avatar in a dynamically running computerized environment and produce movement set (generate quantitative data)…
“The movements of the user 605 may be captured through a camera or sensor attached to the user 605 or to a sports instrument, such as a golf club or baseball bat. The user 605 may be instructed, such as using an avatar 615 on a display or within an AR field of view in the first direction 610, to perform a movement related to the predetermined sports movement. In an example, three-dimensional data points of the user's movements are captured as a measurement set. The three-dimensional data points of the measurement set may be classified to produce a movement set which includes the range of motion for the target body parts related to the sports movement. A reference set corresponding to the sports movement may be obtained. In an example, the reference set may include three-dimensional data points captured from multiple professional athletes of the corresponding sport. A different set may be computed wherein a member of the difference set defines a three-dimensional relationship between a member of the movement set and a corresponding member of the reference set. An assessment for how the user 605 may improve performance may be determined based on the difference set. The assessment may include a physical therapy program, a stretch the user performs before the sports activity, or a movement or strengthening technique the user 605 is guided through by the avatar 615.” [0053]
analyzing at least one of the one or more user measurements and the one or more activity measurements to obtain analyzed measurements; and
Example of assessment (analyzing) the data (user measurement) and reference data set (activity measurements)…
“… A machine learning system may generate ideal or optimized data for the reference set based on patient specific parameters, such as hip to knee measurement, knee to ankle measurement, etc. The comparison between the patient's data and the reference set may determine a difference set. An assessment for the patient may then be determined based on the difference set. The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
Movements Measurement set…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022]
Example of evaluation system and determine (analyze) movement set (user and activity measurements)…
“FIG. 4A illustrates a patient evaluation system 400 in accordance with some embodiments. The systems and techniques described herein may use a reference set and a movement set to determine a difference set, which is then used to determine an assessment for a target body part. In an example, a patient may identify a target body part for therapy (e.g., the target body part is causing pain to the patient, the target body part is causing the patient to have difficulty walking, the patient has stiffness with the target body part, etc.)…” [0042]
adapting at least one of the activities in the experience, dynamically and in real-time, based on at least the analyzed measurements; and
Example of adjust therapy…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
Feedback and perform immediately (dynamically and in real-time)…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc. The system may capture movement of a target body part, such as the limb. Data collected from the captured movement may then be compared to a reference set, for example including range of motion capabilities of the limb (e.g., based on patient specific data, such as height, gender, etc.). The reference set may be gathered by capturing similar movements performed by the patient's mirroring target body part (e.g., when the patient is having an issue with a right leg, then a reference set of data may be collected from capturing movement of the patient's left leg). The reference set may include compiled data collected from other people who share traits similar to the patient, such as gender, weight, or height. In another example, the reference set may include data created using machine learning techniques, based on clinical data, or based on a machine-determined range of motion. A machine learning system may generate ideal or optimized data for the reference set based on patient specific parameters, such as hip to knee measurement, knee to ankle measurement, etc. The comparison between the patient's data and the reference set may determine a difference set. An assessment for the patient may then be determined based on the difference set. The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
See Dynamic below.
determining one or more combined measurements based on the analyzed measurements wherein the combined measurements are aggregated over the duration of the session of the experience; and
Measurement set (combined measurements)…
“The captured movement of the target body part creates the measurement set. The measurement set includes of three-dimensional data points for the movement performed by the target body part. The three-dimensional data points may be captured with video using a movement capture apparatus such as a camera or a Kinect from Microsoft of Redmond, Wash. The patient may have one or more sensors attached to the target body part, which may supply information about the movement of the target body part or its position in three-dimensional space. The one or more sensors may be attached directly to the patient or may be part of a worn apparatus such as a sleeve the patient places around an arm or leg. The one or more sensors may include one or more of an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a multi-axis sensor, or a contact pressure sensor. In an example, a patent may wear a sleeve that extends from an upper thigh to a lower calf. The sleeve may have one or more sensors located above the knee or below the knee such that the one or more sensors may collect data about the movement of the patient's knee joint.” [0021]
Example of camera to capture three-dimensional movement…
“The system 100 may be used to capture three-dimensional movement. For example, the video capture device 105 may include a movement capture apparatus. The movement capture apparatus may include two or more infrared sensors or cameras to detect or capture three-dimensional movement. The two or more cameras can triangulate positions of the user 115 or identified portions of the user 115 moving within a three-dimensional space. The video capture device 105 may include a camera to capture video in conjunction with infrared captured movement.” [0032]
Number of repetitions (aggregated data) completed (therefore over duration) by the user…
“A user 405 may wear an apparatus such as a sleeve 440 on the target body part. The sleeve 440 may include a sensor 445. The sensor 445 may be used to track patient progress, such as a duration or number of repetitions completed by the user 405. In an example, the sensor 445 may be embedded in a user worn apparatus, such as the sleeve 440. This tracked movement may be used to augment movement captured by the camera 425 or a user-facing movement capture apparatus (e.g., a sensor, a sensor array, a camera, an infrared camera, two or more cameras, a depth camera, etc.). Movement of the user 405 may be tracked using a user-facing movement capture apparatus, the camera 425, the sensor, or any of the movement capture apparatus, or the like.” [0045]
Another example of leg lifts (aggregated data) and duration…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like. In an example, a user may be tasked with performing twenty repetitions of a leg lift. Utilizing the sensor 445, such as an accelerometer, a gyroscope, or a nine-axis sensor, the movement of the leg lift may be tracked. The sensor 445 may count the number of leg lift repetitions performed by the user or monitor the form of a leg lift. The sensor 445 may be used to determine if the user is experiencing fatigue or having difficulty completing a repetition or a task. For example, data from the sensor 445 may indicate that a adherence to form is decreasing for subsequently performed leg lifts. Tasks or aspects of a task (e.g., intensity or duration) may be altered in response to determining that the data indicates the proper form is decreasing. In an example, the data may be analyzed to determine the user's level of fatigue from performing repetitions.” [0046]
See Aggregate and Session below.
determining level of performance of one or more human skills based on the one or more combined measurements.
Example of different levels of performance, such as greater range (measurement) of motion…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
Dynamic
Van Andel teaches feedback and real-time. They do not literally teach dynamic.
Bosnak et al. also in the business of feedback and real-time teaches:
Example of continuous real-time feedback and attunement….
“In some embodiments, an exercise called Repetition may be incorporated into the interactive attuned discrete avatar training. Through this activity the interactive attuned discrete avatar may receive continuous real-time feedback on its attunement. In some embodiments, Repetition may include a back-and-forth between the avatar and the user such that the avatar may be programming to ask and/or assert emotional characteristics of the user based on imagery and audio captured by input devices. The user may provide feedback to the emotional characteristics stated by the avatar to provide training data to update the models of the avatar and improve recognition of emotions and attunement to the emotions. In some embodiments, the user may also provide feedback to the rendered expressions of the avatar as a technique for training the models of the avatar for improved attunement to the user.” [0061]
Real-time feedback to motivate user to get going…
“In some embodiments, an avatar may be tuned to emote a more active emotive state to help push the user through a workout. In some embodiments, the avatar can work with the user's fitness level, physical abilities and limitations to craft a tailored routine for the individual and then give real-time feedback on reps, form and activities. In some embodiments, the avatar may motivate the user to get going when their resolve or desires are flagging and champion them through the process.” [0136]
Real-time…
“It is understood that at least one aspect/functionality of various embodiments described herein can be performed in real-time and/or dynamically. As used herein, the term “real-time” is directed to an event/action that can occur instantaneously or almost instantaneously in time when another event/action has occurred. For example, the “real-time processing,” “real-time computation,” and “real-time execution” all pertain to the performance of a computation during the actual time that the related physical process (e.g., a user interacting with an application on a mobile device) occurs, in order that results of the computation can be used in guiding the physical process.” [0148]
Dynamically…
“As used herein, the term “dynamically” and term “automatically,” and their logical and/or linguistic relatives and/or derivatives, mean that certain events and/or actions can be triggered and/or occur without any human intervention. In some embodiments, events and/or actions in accordance with the present disclosure can be in real-time and/or based on a predetermined periodicity of at least one of: nanosecond, several nanoseconds, millisecond, several milliseconds, second, several seconds, minute, several minutes, hourly, several hours, daily, several days, weekly, monthly, etc.” [0149]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of Van Andel the ability to use dynamic and real-time for interaction as taught by Bosnak et al. since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Van Andel who teaches improving real-time and feedback. Van Andel benefits by further detail of using dynamic with real-time and feedback as they teach using machine learning for processing information.
Aggregate and Session
The combined references teach sport activity. They do not literally teach aggregate.
Hoffman et al. also in the business of sport activity teaches:
Example of aggregation of data values obtained from time period (duration) for sessions…
“Similarly, if the user selects the "Month" button 817, the athletic data display configuration module 605 will modify the user interface 801 to display icons representing the aggregations of data values from athletic data sets obtained over each monthly time period. For example, if the user has selected the "Distance" button 809 as well, the user interface 801 may display an icon 835 representing the aggregation of total distance values from data sets obtained for athletic activity sessions performed during each calendar month, as illustrated in FIG. 8E. The user interface 801 also may include a calendar month field 837 specifying the calendar month to which each icon 835 is associated. As shown in this figure, the user interface 801 thus includes an icon 835A representing the aggregation of total distance values from data sets obtained for athletic activity sessions performed during the month of August, and another icon 835B representing the aggregation of total distance values from data sets obtained for athletic activity sessions performed during the month of January. The height of the icon 835A represents the sum of the total distance values for each athletic data set obtained for athletic activity sessions performed in August (i.e., 18.84 miles), while the height of the icon 835B correspond to the sum of each of the total distance data values for each athletic data set obtained for athletic activity sessions performed in January (i.e., 58.84 miles).” [0120]
“Multiple activities performed on the same day may be tracked and recorded separately from one another to provide a distinct breakdown 2617 of a user's day. In one example, workouts may be separated based on type of activity, whether the workout is a session workout or a daily workout and the like. In one or more configurations, step activity data may be monitored and tracked separately from running data. For example, a distance or pace a user walked might not be counted towards running distances and levels and vice versa. Instead, a separate distance and/or pace tracker may be provided for each of running activities and walking activities. Accordingly, multiple workouts may be recorded and displayed for a single day (e.g., July 15 includes 4 different workouts).” [0171]
Example of avatar…
“Additionally, various goals, landmarks or tasks such as goals 7805a, 7805b and 7805c may be displayed along path 7803. Further, the user's progress along path 7803 may be animated in one or more configurations (e.g., the user's avatar or icon may be moved along path 7803 or a color of path 7803 may change gradually representing the user's progress). A goal might not be revealed until an immediately preceding goal or a previous number of goals have been completed. Interface 7800 may further display images, video, text and/or audio in area 7807 of a current location of the user. Accordingly, if the user is running by or along the Brooklyn Bridge, pictures of the Brooklyn Bridge may be displayed in snapshots area 7807. The location may be a virtual location of a user in the progress map or an actual physical location of the user. Other users progressing through the same location may be identified along path 7803 using icons 7809. This allows the user to track their progress relative to his or her own progress. Interacting with icons 7809 may allow a user to contact and/or view details about the other users.” [0215]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to aggregate data with a session as taught by Hoffman et al. since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Hoffman et al. who teaches the advantages of aggregating data for a session.
Level
The combined references teach performance. They do not literally teach level.
Bosnak et al. also in the business of performance teaches:
Fitness level and reps, form and activities *combined measurements)…
“In some embodiments, an avatar may be tuned to emote a more active emotive state to help push the user through a workout. In some embodiments, the avatar can work with the user's fitness level, physical abilities and limitations to craft a tailored routine for the individual and then give real-time feedback on reps, form and activities. In some embodiments, the avatar may motivate the user to get going when their resolve or desires are flagging and champion them through the process.” [0136]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to determine a fitness level as taught by Bosnak et al. since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by the combined references that teach improving physical movement. The combined references benefit by knowing and improving fitness level.
Regarding claim 2
The method of claim 1, the method further comprising:
based on the level of performance, generating feedback; and
Van Andel teaches:
Feedback based on of movement (therefore, generating feedback)…
“The systems and methods herein describe using a movement capture apparatus, such as a camera or an infrared (IR) detection device, to capture movement information from a patient. The systems and methods may be used for remote diagnosis or feedback. In an example, a system may provide a user interface, a video display, an augmented reality (AR) display, a virtual reality (VR) display, audible alerts, haptic feedback, a non-contact alert, or the like to present feedback.” [0018]
providing the feedback;
Present feedback…
“The systems and methods herein describe using a movement capture apparatus, such as a camera or an infrared (IR) detection device, to capture movement information from a patient. The systems and methods may be used for remote diagnosis or feedback. In an example, a system may provide a user interface, a video display, an augmented reality (AR) display, a virtual reality (VR) display, audible alerts, haptic feedback, a non-contact alert, or the like to present feedback.” [0018]
whereby the provided feedback facilitates the user to improve the level of performance of the human skills by improving performance of at least some of the user measurements.
Example of challenges (improving performance) to provoke thresholds (feedback of measured mobility and stability (measurements)…
“In general, this disclosure describes techniques that may allow a computer system equipped with cameras and sensors to assess a patient's mobility and stability three-dimensionally. The system may capture three-dimensional data about the of a target body part on a person, such as an arm or leg and including the movement of joints such as the elbow or knee. The gathered data may then be analyzed by a remote clinician or an artificial intelligence system to determine the patient's range of motion and motor control. The three-dimensional data may then be analyzed by a clinician or artificial intelligence system against norms to determine a patient assessment. The assessment may then be a program to immerse the patient in three-dimensional tasks and challenges aimed at provoking the thresholds of the patient's measured mobility and stability.” [0017]
Regarding claim 3
The method of claim 1, wherein at least one activity measurement is not related to success of the user in completing goals of the experience but is used in the determining the level of performance.
Van Andel teaches:
Example of capturing reference data (not related to success of completing goals)…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc. The system may capture movement of a target body part, such as the limb. Data collected from the captured movement may then be compared to a reference set, for example including range of motion capabilities of the limb (e.g., based on patient specific data, such as height, gender, etc.). The reference set may be gathered by capturing similar movements performed by the patient's mirroring target body part (e.g., when the patient is having an issue with a right leg, then a reference set of data may be collected from capturing movement of the patient's left leg)…” [0019]
Regarding claim 4
The method of claim 1, wherein adapting at least one of the activities comprises performing at least one of the following:
changing a level of difficulty of at least one of the activities,
replacing virtual activities included in the experience,
activating and/or deactivating specific activities,
adjusting at least one of the activities to assist the user in participating in the experience,
providing guidance on experience in the activity,
Van Andel teaches:
Guidance…
“The assessment provided to the patient may include a physical therapy program including various movements or stretches the patient is to perform with the target body part. The system used to capture the movements of the target body part for diagnosis may be used to assist in guiding the patient through the therapy routines.” [0020]
adjusting or triggering audio output, or
Example of audible feedback (output)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
changing visual appearance of elements included in the experience.
Example of arrow pointing, etc. (changing visual appearance)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
Regarding claim 5
The method of claim 1, wherein at least one of the activities involves interaction with a plurality of virtual interactive elements, and wherein adapting at least one of the activities comprises changing the interactive elements in response to a user action.
Van Andel teaches:
Example of user kicks virtual target of a line or circle (interactive virtual experience, where line circle is an element), and fade out (changing the element in response to the user action)…
“The AR display 810 may display the AR environment overlaid on the real environment. The display AR 810 may show the virtual target, using the AR device 801, in the fixed position in the AR environment. In an example, the AR device 801 may remove the virtual target from display in the AR environment in response to detecting a user interaction with the virtual target. For example, when the virtual target is a line or circle, the virtual target may be removed (e.g., fade out, popped, explode, etc.) when a user interacts with the virtual target (e.g., kicks the circle, moves a body part past the line, etc.).” [0059]
Regarding claim 6
The method of claim 5, wherein at least one element includes an avatar configured for mimicking at least one of the user actions.
Van Andel teaches:
User may control avatar (avatar configured for mimicking) based on user movement (actions)…
In an example, the user 605 may control movement of the avatar 615. For example, the user 605 may hold an apparatus used for a sport such as a golf club or baseball bat. The sport apparatus may have a motion sensitive sensor such as an accelerometer, a gyroscope, an inertial measurement unit, or a nine-axis sensor. The motion sensor may track the movement of the sport apparatus. The tracked movement may be used by the sports training AR system 600 to display the movement using a representation of the movement via the avatar 615 performing the movement virtually. In another example, the avatar 615 may be used to display a training movement. For example, the training movement may represent an idealized or professional movement of the sport movement attempted by the user 605. The user 605 may control the avatar 615 using the sport apparatus 635 such that the avatar 615 moves a corresponding virtual sport apparatus 640 similarly to the sport apparatus 635. The avatar 615 may move the virtual sport apparatus 640 according to the training movement. Thus the movement of the avatar 615 may generally follow the movement of the user 605, but may differ by displaying an improved or training movement for the user to attempt to match. In this way, the user 605 may move and control the sport apparatus 635 while viewing similar movements by the virtual sport apparatus 640, the similar movements representing an ideal or improved movement.” [0052]
Joint or location of avatar represents (mimics) user pain…
“The view 300 may allow the user 305 or a remote user, such as a clinician, to select a joint or location on an AR avatar. By selecting a joint or location on the AR avatar, the user 305 or remote user may indicate what hurts or needs attention on the user 305, select a side of the body, be prompted for a pain level, or the like. In another example, the user 305 may select options from a user interface within the view 300, such as a scale bar to select pain. The view 300 may include a questionnaire for the patient to answer so that a therapist may evaluate progress or determine exercises for the patient.” [0041]
Regarding claim 7
The method of claim 1, wherein the one or more user measurements comprise a body motion.
Van Andel teaches:
Fig .6 and swing…
PNG
media_image3.png
272
406
media_image3.png
Greyscale
“The movements of the user 605 may be captured through a camera or sensor attached to the user 605 or to a sports instrument, such as a golf club or baseball bat. The user 605 may be instructed, such as using an avatar 615 on a display or within an AR field of view in the first direction 610, to perform a movement related to the predetermined sports movement. In an example, three-dimensional data points of the user's movements are captured as a measurement set…” [0053]
Regarding claim 8
The method of claim 1, wherein the one or more user measurements comprise facial muscles and expressions.
Van Andel teaches:
Evaluate head of the user…
“In an example, the system 100 may be calibrated using the video capture device 105. The video capture device 105 may use infrared light to detect the user 115 in a field of view. The system 100 may evaluate the detection (e.g., captured images) of the user 115 to identify joints, limbs, appendages, a head, etc., of the user 115. These identified body parts may be used with later captured video of an exercise to label specific body parts.” [0031]
The combined references teach evaluate head. They do not teach facial muscles or expressions.
Bosnak et al. also in the business of evaluating head teaches:
“The present disclosure generally relates to computer-based platforms and systems configured for automated real-time generation of an interactive attuned discrete avatar, including responsive artificially generated speech and facial features in response to user interaction.” [0002]
Measure of non-speech expression of face with muscle activity…
“Herein, the term valence may refer to a measure of the pleasantness and/or unpleasantness indicated by non-speech expression, e.g., as measured using the Facial Action Coding System and micro-expressions, muscle activity detected through facial electromyography, functional brain imaging, or by any other suitable valence measurement technique or any combination thereof.” [0052]
Face with musculature…
“In some embodiments, the system may utilize the above principles to create interactive attuned discrete avatars maybe photorealistic human replicas, e.g., seen from mid-torso to the top of their head, and capable of torso, face and head movements. The face maybe constructed on a platform similar to human anatomy with a musculature on top of a fixed frame, under a cutaneous and subcutaneous layer allowing for the display of both “normal” and more specifically, micro expressions which the user may sense and observe on a subliminal level. A shadow image of the user may be silhouetted in the interactive attuned discrete avatar pupil creating a sensation of intimate presence.” [0057]
“In some embodiments, an exercise called Repetition may be incorporated into the interactive attuned discrete avatar training. Through this activity the interactive attuned discrete avatar may receive continuous real-time feedback on its attunement. In some embodiments, Repetition may include a back-and-forth between the avatar and the user such that the avatar may be programming to ask and/or assert emotional characteristics of the user based on imagery and audio captured by input devices. The user may provide feedback to the emotional characteristics stated by the avatar to provide training data to update the models of the avatar and improve recognition of emotions and attunement to the emotions. In some embodiments, the user may also provide feedback to the rendered expressions of the avatar as a technique for training the models of the avatar for improved attunement to the user.” [0061]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to measure facial muscles as taught by Bosnak et al. since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Bosnak et al. who teaches the benefits of monitoring and providing feedback to users based on facial movements.
Regarding claim 9
The method of claim 1, wherein the one or more user measurements comprise internal functions.
Van Andel teaches:
Heart rate, oxygen, blood pressure (internal measurements)…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like. In an example, a user may be tasked with performing twenty repetitions of a leg lift. Utilizing the sensor 445, such as an accelerometer, a gyroscope, or a nine-axis sensor, the movement of the leg lift may be tracked. The sensor 445 may count the number of leg lift repetitions performed by the user or monitor the form of a leg lift. The sensor 445 may be used to determine if the user is experiencing fatigue or having difficulty completing a repetition or a task. For example, data from the sensor 445 may indicate that a adherence to form is decreasing for subsequently performed leg lifts. Tasks or aspects of a task (e.g., intensity or duration) may be altered in response to determining that the data indicates the proper form is decreasing. In an example, the data may be analyzed to determine the user's level of fatigue from performing repetitions.” [0046]
Regarding claim 10
The method of claim 1, wherein the one or more activity measurements relate to continuous running of the experience.
Van Andel teaches:
Example of completes a scheduled session or therapy, therefore, continuous activity…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
Regarding claim 11
The method of claim 1, wherein analyzing the one or more user measurements comprise comparing at least one of the extracted user measurements to a corresponding reference user measurement.
Van Andel teaches:
Example of data captured from movement of user and compare to reference, where reference can be patients movement…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc. The system may capture movement of a target body part, such as the limb. Data collected from the captured movement may then be compared to a reference set, for example including range of motion capabilities of the limb (e.g., based on patient specific data, such as height, gender, etc.). The reference set may be gathered by capturing similar movements performed by the patient's mirroring target body part (e.g., when the patient is having an issue with a right leg, then a reference set of data may be collected from capturing movement of the patient's left leg)…” [0019]
Regarding claim 12
The method of claim 1, wherein extracting the one or more activity measurements comprise measuring one or more of performance, goals, or events within the experience, or a combination thereof.
Van Andel teaches:
Goal…
“In system 100, the user 115 may wear the AR glasses 110. In an example, view 150 may represent a view the user 115 sees while wearing the AR glasses 110. The view 150 may include virtual and real aspects in an environment, such as the user's leg (real) augmented by a guide (virtual). In an example, the user 115 may look downward to observe the view 150 and the user may see the raised leg 135 with bent knee. The view 150 using the AR glasses 110 may include a virtual object or goal for the target body part to reach or extend to, such as a virtual line 140. The virtual line 140 may represent an extension distance for the target body part to achieve during a movement. The AR glasses 110 may display virtual aspects in a real environment, such as in view 150.” [0034]
Regarding claim 14
The method of claim 2, further comprising: providing the feedback to a third party related to the user.
Van Andel teaches:
Remote clinician [to receive] limb mobility problem (feedback)…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc…” [0019]
Regarding claim 15
The method of claim 2, wherein at least one of the user actions comprises a body motion, and wherein the feedback comprises data relating to a more accurate version of the body motion.
Van Andel teaches:
Movements performed in certain manner to accurately portray movement and guide (feedback) patient…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022]
Regarding claim 16
The method of claim 15, wherein the feedback comprises an avatar configured to illustrate a more accurate version of the body motion.
Van Andel teaches:
Avatar demonstrate movements that user should perform (more accurate version)…
“FIG. 1A and FIG. 1B illustrate real and virtual aspects of an augmented reality diagnosis system in accordance with some embodiments. In an example, the system 100 includes a video capture device 105, a display device 107, and AR glasses 110 worn by a user 115. The display device 107 may display an avatar 120 to demonstrate the movements the user 115 should perform. In an example, the avatar 120 is demonstrating the movement of lifting the right leg with the knee bent at a 90 degree angle. The demonstrated bent knee movement is also being performed by the user 115. The user 115 may perform a movement, as instructed, which is captured by the video capture device 105. In an example, a movement may be performed by the user 115 with one or more target body parts, and the movements may be captured by the video capture device 105. In an example, the video capture device 105 may be a camera, a depth camera, or an infrared detection device, such as the Kinect from Microsoft of Redmond, Wash.” [0030]
Regarding claim 17
The method of claim 2, wherein the experience is selected based on a personalized treatment plan generated to facilitate the user to improve one or more human skills by improving performance of at least some of the user measurements, wherein at least some of the activities in the selected experience correspond to the one or more human skills.
Van Andel teaches:
Physical therapy program (treatment plan) for the patient (personalized)…
“The assessment provided to the patient may include a physical therapy program including various movements or stretches the patient is to perform with the target body part. The system used to capture the movements of the target body part for diagnosis may be used to assist in guiding the patient through the therapy routines.” [0020]
“…An assessment for how the user 605 may improve performance may be determined based on the difference set. The assessment may include a physical therapy program, a stretch the user performs before the sports activity, or a movement or strengthening technique the user 605 is guided through by the avatar 615.” [0053]
Regarding claim 18
The method of claim 17, wherein the treatment plan is automatically generated, based on data pertaining to the user.
Van Andel teaches:
Using machine-learning (automatically generate) to determine motion range capabilities (treatment plan)…
“The technique 1100 may include determining performed range of motion information for the target body part based on the movement set. The technique 1100 may include obtaining range of motion capabilities for the target body part. In an example, obtaining the range of motion capabilities includes obtaining the range of motion capabilities from a database. The technique 1100 may include training range of motion capabilities data using a machine-learning technique. The range of motion capabilities to be retrieved from the database may be queried from the database from the learned range of motion capabilities (e.g., output from the machine learning system). In another example, obtaining the range of motion capabilities includes obtaining the range of motion capabilities from the camera by capturing a reference body part of the subject that mirrors the target body part, the reference body part performing the three-dimensional movement.” [0075]
Regarding claim 19
The method of claim 17, the method further comprising: providing the feedback to a third party related to the user,
wherein the third party is a therapist involved in generating treatment plans;
Van Andel teaches:
Instruct and guide (treatment plans) and therapist…
“Systems and methods for using augmented reality (AR) to remotely instruct and guide a patient through movements of a target body part, as well as guide the patient with therapy routines, are described herein. The systems and methods herein describe using AR to display a virtual representation of a patient's limb to illustrate to the patient how to complete a clinical movement. In an example, the AR may be used to display directions for a patient to perform a clinical movement. The real therapist may, for example, be a live therapist appearing with a patient, or may be a video recording of a therapist. The real therapist may be defined as any therapist that is not virtual.” [0028]
in response to providing the feedback, receiving a modification input from the therapist; and
Therapist determine exercises (receiving input), where determine exercise based on progress, therefore, changes (modified) based on changing progress…
“The view 300 may allow the user 305 or a remote user, such as a clinician, to select a joint or location on an AR avatar. By selecting a joint or location on the AR avatar, the user 305 or remote user may indicate what hurts or needs attention on the user 305, select a side of the body, be prompted for a pain level, or the like. In another example, the user 305 may select options from a user interface within the view 300, such as a scale bar to select pain. The view 300 may include a questionnaire for the patient to answer so that a therapist may evaluate progress or determine exercises for the patient.” [0041]
adapting at least one of the activities and/or the experience in accordance with the modification input.
Changes (modified) based on changing progress…
“The view 300 may allow the user 305 or a remote user, such as a clinician, to select a joint or location on an AR avatar. By selecting a joint or location on the AR avatar, the user 305 or remote user may indicate what hurts or needs attention on the user 305, select a side of the body, be prompted for a pain level, or the like. In another example, the user 305 may select options from a user interface within the view 300, such as a scale bar to select pain. The view 300 may include a questionnaire for the patient to answer so that a therapist may evaluate progress or determine exercises for the patient.” [0041]
Regarding claim 20
The method of claim 19, wherein the modification input includes data pertaining to modification in the personalized treatment plan, and prior to the modifying, the method further comprising:
based on the modification input, updating the treatment plan; and
Van Andel teaches:
Adjust (updating) the prescribed therapy (treatment plan)…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
adapting at least one of the activities and/or the experience, based on the updated treatment plan.
Prescribed movement…
“The display 415 may be used to animate the virtual representation 430 of the user 405 to perform a movement along a path. The path may be used to show a patient how to perform a prescribed movement. The path may be used to determine whether the movement by the user was successful. In an example, the path may be used to determine whether a user has successfully completed an exercise or movement. For example, if the user completes the exercise within the path, the exercise may be determined to be completed successfully. If the user moves outside the path while attempting to complete the exercise, the exercise may be determined to not be completed successfully.” [0048]
Regarding claim 21
The method of claim 20, wherein adapting at least one of the activities and/or the experience comprises performing at least one of the following:
changing a level of difficulty of at least one of the activities,
replacing the virtual activities,
activating and/or deactivating specific activities,
adjusting at least one of the activities to assist the user in participating the experience,
providing guidance on the experience on the activity,
Van Andel teaches:
Guidance…
“The assessment provided to the patient may include a physical therapy program including various movements or stretches the patient is to perform with the target body part. The system used to capture the movements of the target body part for diagnosis may be used to assist in guiding the patient through the therapy routines.” [0020]
adjusting or triggering audio output, or
Example of audible feedback (output)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
changing visual appearance of elements included in the experience.
Example of arrow pointing, etc. (changing visual appearance)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
Regarding claim 22
The method of claim 1, wherein the user device is a mobile user device comprising a camera.
Van Andel teaches:
Smartphone with camera…
“…The patient may be presented with an option of performing the therapy program at home, for example by utilizing a personal AR device, a smartphone with camera, a video capture device such as a Kinect from Microsoft of Redmond, Wash., or the like.” [0029]
Regarding claim 23
The method of claim 1, wherein determining the one or more combined measurements comprises processing the analyzed user measurements and the activity measurements in order to assess one or more of the following aspects pertaining to human skills:
motion patterns,
Van Andel teaches:
Golf swing (motion patterns)…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
reaction, performance related to abilities,
Example of assessment (analyzing) the data (user measurement) and reference data set (activity measurements)…
“… A machine learning system may generate ideal or optimized data for the reference set based on patient specific parameters, such as hip to knee measurement, knee to ankle measurement, etc. The comparison between the patient's data and the reference set may determine a difference set. An assessment for the patient may then be determined based on the difference set. The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
aggregated measurements over a plurality of experiences,
comparison with goals,
Goal to reach to (comparison)….
“In system 100, the user 115 may wear the AR glasses 110. In an example, view 150 may represent a view the user 115 sees while wearing the AR glasses 110. The view 150 may include virtual and real aspects in an environment, such as the user's leg (real) augmented by a guide (virtual). In an example, the user 115 may look downward to observe the view 150 and the user may see the raised leg 135 with bent knee. The view 150 using the AR glasses 110 may include a virtual object or goal for the target body part to reach or extend to, such as a virtual line 140. The virtual line 140 may represent an extension distance for the target body part to achieve during a movement. The AR glasses 110 may display virtual aspects in a real environment, such as in view 150.” [0034]
cognitive and physical abilities,
social skills, and
behavioral skills.
Regarding claim 24
The method of claim 1, wherein the analyzing comprises analyzing one or more user measurements using at least one of image analysis and analysis of extracted human body.
Van Andel teaches:
Example of image and processor determine (analyze) range of motion (measurements) of a subject (extracted from a human body)…
In an example, the camera 806 or other motion capture device such as an infrared capture device, may capture a series of images of three-dimensional movement of a target body part of a subject, the target body part moving relative to a reference point of the subject. The processor 802 may determine range of motion information for the target body part based on the captured series of images. In an example, the processor 802 may obtain a reference range of motion capabilities for the target body part. The processor 802 may determine that the reference range of motion capabilities exceed the range of motion for the target body part. In an example, the display device 812 or the AR display 810 may display, in response to the determination, a virtual animation of the target body part, the virtual animation created using the reference range of motion capabilities and the range of motion information for the target body part. The virtual animation may be displayed using the AR display 810 as a virtual aspect represented in a real environment, such as in a room or augmented over the target body part of the subject
Regarding claim 25
The method of claim 1, wherein determining the level of performance comprises mapping the combined measurements to the one or more human skills.
Van Andel teaches:
Mapping reference set (performance) to movement (human skills)…
“In Example 55, the subject matter of any one or more of Examples 1-54 optionally include wherein to member of the difference set is a mapping from the movement set to the reference set.” [0131]
Regarding claim 29
The method of claim 28, wherein providing data comprises: based on one or more parameters relating to human skills, generating feedback; and providing the feedback; whereby the provided feedback facilitates the user to improve the level of performance of the human skills by improving performance of parameters indicative of human skills.
Van Andel teaches:
Example of challenges (improving performance) to provoke thresholds (feedback of measured mobility and stability (measurements)…
“In general, this disclosure describes techniques that may allow a computer system equipped with cameras and sensors to assess a patient's mobility and stability three-dimensionally. The system may capture three-dimensional data about the of a target body part on a person, such as an arm or leg and including the movement of joints such as the elbow or knee. The gathered data may then be analyzed by a remote clinician or an artificial intelligence system to determine the patient's range of motion and motor control. The three-dimensional data may then be analyzed by a clinician or artificial intelligence system against norms to determine a patient assessment. The assessment may then be a program to immerse the patient in three-dimensional tasks and challenges aimed at provoking the thresholds of the patient's measured mobility and stability.” [0017]
Regarding claim 30
The method of claim 1, wherein the one or more human skills comprises at least one skill selected from physical skills, cognitive skills, behavioral skills, social skills, and mental abilities.
Van Andel teaches:
Example of strengthening or improving physical movement (physical skills)….
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
Regarding claim 31
The method of claim 30, wherein the level of performance is determined for at least two different human skills.
Van Andel teaches:
Example of different levels of performance, such as greater range (measurement) of motion…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
Level
Van Andel teaches performance. They do not literally teach level or two skills.
Bosnak et al. also in the business of performance teaches:
Emotional cues and behavioral skills…
“In some embodiments, to fill the need for the trusted intimate relationship, an interactive attuned discrete avatar may be created with photorealistic empathic AI driven autonomous virtual companions. Through biometric, physical and vocal markers the interactive attuned discrete avatar may be able to discern emotional cues from body language, vocal qualities (including but not limited to acoustic tone, pitch cadence, etc), and natural language processing (NLP). In some embodiments, the interactive attuned discrete avatar may be able to portray similar responses in their appearance, facial musculature and be programmed to ask questions to lead a conversation based on inferences about the emotions they observed in the user's behavior. Through this process of connection, synchronizing and attunement between the user and the interactive attuned discrete avatar an attachment may form. From this initial attachment the user may move into transference, the psychological act of observing one's own unconscious moods and images as if they were coming from someone else in close affiliation. This creates a sense of closeness which serves as the beginning of an intimate relationship.” [0005]
Level of emotional resistance….
“Herein, the term resistance may refer to the level of emotional resistance the user exhibits in a given circumstance or state, measured on a scale from obstructive to conducive.” [0055]
Example of interpersonal (social) skills and emotional (behavioral) intelligences (skills)…
“Currently many medical schools employ actors to play the role of patient for medical students to try and make diagnoses. It is an expensive process. In some embodiments, an avatar can be programmed with hundreds of different maladies which can present both physically and through manufactured test data to give med students a much more real experience in patient interaction. The students may also be able to improve their interpersonal patient skills and can be evaluated on their Emotional Intelligences along with their medical knowledge.” [0134]
Fitness level and reps, form and activities *combined measurements)…
“In some embodiments, an avatar may be tuned to emote a more active emotive state to help push the user through a workout. In some embodiments, the avatar can work with the user's fitness level, physical abilities and limitations to craft a tailored routine for the individual and then give real-time feedback on reps, form and activities. In some embodiments, the avatar may motivate the user to get going when their resolve or desires are flagging and champion them through the process.” [0136]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to determine a skill levels as taught by Bosnak et al. since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by the combined references that teach improving physical movement. The combined references benefit by knowing and improving various types of skill levels.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
The following prior art teaches at least activity and avatar:
US-20160086500-A1; US-20180350144-A1; US-20100240458-A1; US-20130063432-A1; US-9997082-B2; US-11990233-B2; US-12150791-B2
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENNETH BARTLEY whose telephone number is (571)272-5230. The examiner can normally be reached Mon-Fri: 7:30 - 4:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SHAHID MERCHANT can be reached at (571) 270-1360. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KENNETH BARTLEY/Primary Examiner, Art Unit 3684
Prosecution Insights