DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on February 26, 2026, has been entered.
Response to Amendment
Claims 1, 28, and 30-32 have been amended. Claims 13, 27, and 29 have been canceled. Claims 34 and 35 are new. Claims 1-12, 14-26, 28, 30-35 are pending and are provided to be examined upon their merits.
Response to Arguments
Applicant's arguments filed February 26, 2026, have been fully considered but they are not persuasive. A response is provided below in bold where appropriate.
Applicant notes Double Patenting, pg. 18 of Remarks:
Double Patenting
The Applicant will submit a power of attorney. Once the power of attorney has been approved by the Office, the Applicant will file a terminal disclaimer, thereby obviating the double patenting rejection.
Noted. The instant application provides limitations not found in the 18/447491 claims (e.g., progress markers and skill level achieved when all markers are satisfied), therefore the double patenting is withdrawn but may be reinstated at a later date pending claim amendments. The double patterning on 18/447491 would be maintained as it is broader than the instant claims.
Applicant argues 35 USC §103, starting pg. 18 of Remarks:
Nonobviousness
The Examiner contended that the invention of claims 1-33 would have been obvious over Pub. No. US 2018/0315247 to Van Andel in view of Pub. No. US 2023/0021339 to Bosnak et al.
In response, the Applicant submits arguments closely tracking arguments that are being presented in a response filed in the corresponding application number US 18/447,491 on the same date as the present response, followed, on page 25 of this response, by supplemental arguments specific to the claims of the present application. Arguments Tracking Those Presented in Corresponding Application US 18/447,491
Noted.
Advantages of the invention of amended claim 1 over the cited prior art are as follows:,
1. Amended claim 1 defines extracting activity measurements comprising at least data pertaining to the execution of the session of the experience itself. This is opposed to the prior art focusing on raw physical limb measurements which may correspond, at best, to the user measurements of claim 1 only.
From Van Andel…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like. In an example, a user may be tasked with performing twenty repetitions of a leg lift. Utilizing the sensor 445, such as an accelerometer, a gyroscope, or a nine-axis sensor, the movement of the leg lift may be tracked. The sensor 445 may count the number of leg lift repetitions performed by the user or monitor the form of a leg lift. The sensor 445 may be used to determine if the user is experiencing fatigue or having difficulty completing a repetition or a task. For example, data from the sensor 445 may indicate that a adherence to form is decreasing for subsequently performed leg lifts. Tasks or aspects of a task (e.g., intensity or duration) may be altered in response to determining that the data indicates the proper form is decreasing. In an example, the data may be analyzed to determine the user's level of fatigue from performing repetitions.” [0046]
Therefore, count repetitions and measure fatigue, etc. This is more than just physical limb measurements.
Examples according to claim 1 include: tracking how many times a user paused the game or counting the collection of non-goal items (e.g., red rocks instead of blue) to assess concentration skill (see paragraph [0164]).
The Applicant emphasizes its position that no description is provided in Van Andel, in particular in paragraph [0029] relied upon by the Examiner, that corresponds to a plurality of activities within a single interactive experience, in the sense of claim 1.
Previously amended Claim 1 already recited:
"... providing on a user device an interactive virtual experience in which the user can participate, the experience comprising one or more activities; ...
extracting one or more activity measurements from the provided experience;
The Examiner contended in the Office action that:
Performing movement is an activity. Bending a knee is an activity. Exercise would be an activity. Applicant's claims use broad terms such as "activity. "Limitations from the specification are not read into the claim.
However, physical movements of the user like bending a knee of Van Andel cannot be read into "activities from an interactive virtual experience" and certainly cannot read into "extracting one or more activity measurements from the provided [interactive virtual] experience" as in claim 1.
With all due respect, activities can be anything including bending a knee. Bending a knee requires physical movement. This is an action. Applicant appears to be arguing bending a knee is not an activity. The Examiner respectfully disagrees.
However, to expedite allowance, claim 1 is amended as outlined above to emphasize that the system extracts activity measurements from the experience which comprises at least data pertaining to the execution of the experience itself. This is not taught by Van Andel which extracts only measurements that relate to the user himself, or the equipment that he carries.
Counting sit-ups is an experience itself and not measurements that relate to the user himself or carried equipment.
Respectfully, Applicant cannot define their claim language as activities, except for activities related to measurement of a user himself or equipment the user carries, unless this has been defined in their specification.
From Applicant’s specification…
“A computer-implemented method for assessing and determining the level of performance of human skills of a user and facilitating the user to improve these skills, by improving level of performance of these skills, is provided. The method enables a user using a user device, such as a mobile device, to participate in an interactive virtual experience including various activities, for example, a first activity of climbing a mountain, and a second activity of avoiding falling rocks. While the user is participating in the game, a camera, such as the device's camera, can capture the user when participating in the game. Data pertaining to the user can be gathered, such as physical motions of climbing, expressions, and internal functions pertaining to how the user reacted to the rocks that he had to climb, and to the climbing itself. Data on inactivity of the user, such as no physical motion of the user at a certain point in time, can also be gathered as user actions.” [0012]
Therefore, a user’s physical motion is an activity, where the motion is not limited to exclude knee bending or the equipment user is carrying.
2. Amended claim 1 defines determining combined measurements based on both the user measurements and the activity measurements. The purpose is to evaluate technical and non-technical "human skills" such as cognitive, social, and behavioral more accurately, based on both the user's movements, but in combination with how the session of the activity run and what happened during the session.
Examples are detailed in paragraphs [0203]-[0208], such as:
" correlating a user's heart rate (user measurement) with game speed (activity measurement) to objectively calculate a "stress tolerance" level.
" Measuring a lack of focus derived from constantly moving the eye gaze to the side (a user measurement) when combined with a focused target the
user 110 has to shoot at (an activity measurement).
" A positive emotion, when combining the physical emotion of rising the corners of the mouth with reaching an end of a level in the experience.
The above is not commensurate with the scope of the claim. There is no claimed game speed, lack of focus, or positive emotion.
As previously explained, the step of "determining one or more combined measurements based on the analyzed measurements" is not necessarily performed in real time, and hence, it is not required in the claim to limit it to real-time usage only, as the Examiner commented.
The Applicant explained that the repeatedly loop comprising the steps of:
obtaining camera output;
extracting user measurements;
adapting at least one of the activities...;
Then, separately, the claim includes the step of:
determining one or more combined measurements based on the analyzed
measurements.
Amended claim 1 further clarifies that the combined measurements are aggregated over the duration of a session of the experience.
Noted. However, Van Andel teaches user performs movement and completes a scheduled session…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
In the context of the claim and as further described in the specification, e.g., at paragraph [0184], the system performs real-time analysis during the experience to enable adaptation of activities. In addition, a later analysis, performed after the experience is completed, combines the results of prior experience analyses. The purpose of this higher-level performance analysis after the experience is completed is to obtain parameters relating to human skills in order to determine a skill progression over the entire experience.
Thus, the "combined measurements," relied upon both the user
measurements and the activity measurement, are not merely a capture of raw sensor data or joint motion information, as in Van Andel in "bending knees," as referred to by the Examiner, but instead reflect an analysis of multiple types of analyzed measurements, collected across different iterations of the experience's session. Paragraph [0191] and following paragraphs illustrate examples of the combined measurements.
Respectfully, multiple types of measurements are not being claimed. A person performing multiple knee bends would read on the claim. Measuring a person’s physical motion of climbing would read on the claim.
From Van Andel…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like…”
Therefore, Van Andel teaches multiple measurements can be made.
Analysis of both the user measurements and the activity measurements is
an important and advantageous step that involves combining and analyzing data from two distinct domains: the user's physical movement, as extracted from the captured video, such as physical motions, facial expressions, or internal functions) and the dynamics and execution of the virtual experience- what happened in the activity itself (such as game triggers, pauses, or collision events).
This joint analysis, which is not taught by Van Andel's singular focus on raw physical motion data (user's kick) for assessing limb mobility, allows the system to derive higher-level performance metrics and interpret outcomes. For instance, combining the occurrence of a stressful game event ("activity measurement") with a user's lack of movement or anxious facial expression ("user measurement") allows the system to analyze the user's stress tolerance or fear index in a much higher depth than the physical success of the Van Andel's user to kick.
Respectfully, the above argument is not commensurate with the scope of the claims.
From Van Andel…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like…”
The Examiner relied upon Van Andel's sensor 355, which identifies a
sensor attached to a physical equipment (weights) that counts repetitions. The sensor 355 merely tracks physical equipment or counts quantitative body repetitions, which constitutes a user-centric metric of limb mobility rather than an extraction of experience measurements including data pertaining to the to the execution of the session of the interactive virtual experience itself.
3. Real-time dynamic feedback loop and adaptation of the activity:
The system utilizes a "repeatedly executing" loop to adapt activities while the user participates. The experience itself forms a dynamically running computerized environment. Based on the user's and the activity's measurements, the system adapts the activities in the experience, in a dynamic manner and in real-time.
Loop is not claimed. The prior art teaches session. A person is not going to perform one exercise (e.g. one knee bend) as an exercise and then stop.
Example according to claim 1: automatically replacing an intense virtual
background with a calmer one if the system detects user stress in real-time (see paragraph [0130]).
The above is not commensurate with the scope of the claims.
Neither the "feedback" or the "adaptation" of the prior art cannot correspond to the dynamic adaptation of activities in the experience of claim 1, since the feedback and the adaptation (removing a virtual circle) are limited and in response to the user's performance only. Van Andel utilizes a "movement capture apparatus" primarily to capture "movement information from a patient." This is explicitly mentioned in the paragraph relied upon by the Examiner. As such, the adaptation is dependent on the user's movement only. The prior art lacks the extraction of activity measurements, data pertaining to the execution of the experience itself, and hence, cannot adapt the experience as defined by claim 1 ("based on at least the analyzed measurements"). Amended claim 1 defines that adapting the activities in the experience, is done in a dynamic manner and in real-time, based on at least the analyzed measurements. Recall, the analyzed measurements include both the user activity measurements as well as the user's measurements.
Bending a knee is a dynamic activity.
From Van Andel…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like…”
The technical advantageous over the cited prior art lies in enabling the system to respond immediately to indications derived from the combined analysis of internal activity metrics and user state (e.g., replacing an intense virtual background with a calmer one if the analyzed measurements indicate stress). While Van Andel discloses an interaction where a virtual target is "removed" (e.g., fades out or explodes) in response to a user interaction (e.g., kicks the circle), this response a consequence of task completion by the user. It is not a continuous, dynamic modification of the activity or its parameters in real-time based on the analysis of combined data streams (based on both the user and the activity).
Moreover, Van Andel describes a "therapy program" where assessment data is typically produced after the user completes a scheduled session, which leads to "post therapy adjustment" of a clinical plan by a clinician, not activity adaptation during the experience. This real-time adaptation of the running activity, based on comprehensive combined measurements, distinguishes the subject matter of claim 1 from Van Andel. Applicant notes that the Examiner contended on page 11 that Van Andel's disclosure of tracking repetitions and increasing weights teaches real-time analysis and the "dynamic modification of activities." However, the Applicant emphasizes that this does not teach "adapting" the activity of the experience as defined in Claim 1. Van Andel merely describes a scenario where a user or clinician manually changes the physical exercise parameters, by adding physical weight, and the system simply receives and identifies this modified activity state determined by the user. This manual adjustment of physical equipment represents a change to a clinical protocol rather than the system autonomously adapting the execution of the experience or virtual environment of a running activity based on analyzed measurements.
From Van Andel and example of feedback based on patient’s movements…
“The systems and techniques described herein, may be utilized in a pharmacy, an urgent care center, or a walk-in clinic. In an example, the system may be presented as a kiosk with a screen that prompts a patient for information and then guides the patient through prescribed motions for a diagnosis. The kiosk may include a camera, AR glasses, a sleeve, glove, or the like, as described above, such as for capturing or providing feedback based on the patient's movements.” [0028]
Dependent claim 3:
Claim 3 specifically requires that activity measurements are collected,
where at least one such activity measurement is not related to success in completing the goals of the experience, but is nevertheless used to determine the user's skill progression. As illustrated in paragraph [0164] of the application as published, the system extracts data independent of and unrelated to mission success, to assess the level of human skills. For instance, if a user is tasked with collecting blue rocks (the goal), the system may extract the number of red rocks collected as activity measurement. While a standard game would record this merely as a failure, the claimed subject matter utilizes this interaction with the virtual elements in the experience to evaluate cognitive performance, such as color discrimination or concentration.
From Claim 3…
“The method of claim 1, wherein at least one activity measurement is not related to success of the user in completing goals of the experience but is used in the determining the skill progression.”
From Van Andel…
Example of improving a technique or range of motion for sports movement such as a swing (skill)…
“A person may also wish to increase or improve technique or range of motion for a sports-related movement, such as a when swinging a golf club. Similar issues arise for receiving instructions related to sports movements as those described above for the patient. Further, finding a qualified trainer may be difficult.” [0004]
Capture movements and motion sensitive sensor (measurement)…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
Van Andel does not describe extracting any such activity measurements. Moreover, the cited paragraph [0019] relates solely to capturing patient movement data (i.e., user measurements) and comparing those movements to a reference set in order to assess limb mobility. This reference set of movements was gathered by capturing similar movements performed by the patient's mirroring target body part, such as when the patient uses their other leg.
The Examiner's reliance on Van Andel paragraph is unclear. This analysis is limited to evaluating whether a movement was performed correctly and does not involve extracting data from an activity or experience itself. Even if, arguendo, the Examiner interprets this analysis as involving data beyond the desired physical movement, it still does not involve extracting or analyzing information from the activity or virtual experience itself, as defined in Claims 1 and 3. Therefore, Van Andel does not disclose the subject matter of claim 3.
The sections of Van Andel above are further provided to teach improving a skill.
Supplemental Arguments Specific to the Claims of the Present Application
Below are additional arguments regarding skill progression, which is the subject of the claims of the present application.
The Applicant notes that claim 1 specifies that both user measurements and activity measurements are extracted and analyzed to obtain analyzed measurements. Notably, the activity measurement comprising "data pertaining to the execution of the session of the experience itself and the user interaction within the experience." Meaning, the interaction of the user with the experience, as also extracted from the activity, rather than solely from the user measurement, indicates the type of data that is extracted. That is to say, the contribution of the data from the experience itself, is more than a simple application metadata of providing an exercise to the user (as argued based on Van Andel- this is discussed further below, which Applicant submits that cannot correspond to a user interaction within the virtual experience itself, as extracted from the experience as in claim 1).
As detailed in the specification and the first response, these measurements include, for example, technical events such as how many times the experience was paused, or the collision and collection of virtual elements. Specific examples illustrate that these measurements can include interactions with "wrong" virtual elements to evaluate non-motor skills like concentration.
Respectfully, the above is not claimed.
As defined by claim 1, these analyzed measurements are used to adapt activities while the user participates in the experience. Additionally, after the session is over, these analyzed measurements are used to determine skill progression. Hence, the determination of skill progression is likewise based on these activity measurements, which are extracted from the execution of the session and the user's interaction within the virtual experience.
Sensor 355 and reference set of Van Andel
On pages 5-6, the Examiner contends that sensor 355 of Van Andel teaches the "skill progression" of claim 1, as well as the markers of dependent claims 27-30. The Examiner relies on sensor 355 to demonstrate that "[d]uration or number of repetitions completed are progress markers, the skill is performing an exercise". Additionally, the Examiner relies on the "reference set" that may be provided to the user to enable them to mimic the movement.
The Applicant respectfully disagrees. These examples do not teach
extracting activity measurements from a virtual activity, nor do they teach relying on activity measurements to determine a skill progression or its markers.
The problem is that progress markers could be just about anything.
The sensor 355 of Van Andel merely tracks physical equipment or counts quantitative body repetitions, which constitutes user measurements of limb mobility. Even if, arguendo, the Examiner refers to some digital counting that is obtained from that sensor, then this does not constitutes the activity measurements from the provided experience of claim 1 where: "the experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience ", and the "activity measurements ... comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience ".
The Examiner respectfully disagrees with the above. Fig. 1A of Van Andel alone teaches a dynamic activity (notice leg movement, ref. 125 to 130) in a computerized environment (para. [0017]).
The same argument applies to the reference set of Van Andel. This reference set of movements was gathered by capturing similar movements performed by the patient's mirroring target body part, such as when the patient uses their other good leg. This analysis is likewise limited to evaluating whether a movement was performed correctly and does not involve extracting data from an activity or experience itself. Even if, arguendo, the Examiner interprets this analysis as involving data beyond the desired physical movement, it still does not involve extracting or analyzing data from and on the execution of the activity, or the execution of the virtual experience itself, as defined in Claim 1.
Providing an exercise to the user
In addition, on page 9 of the Office action, the Examiner contends that
"Exercise is an experience as it is in activity." The Examiner also relies upon Van Andel that describes that upon completion of the motion, the collected data may be transmitted for determining an assessment. The assessment may include an exercise provided to the user.
Here again, the Applicant respectfully disagrees with this interpretation because it conflates the two technically distinct categories of user measurements and activity measurements, and confuses the monitoring of a physical exercise performed by the user, or provided to the user, with the extraction of data pertaining to the execution of the session of the virtual experience itself.
Van Andel teaches user movement measurements and counting repetitions.
Van Andel describes monitoring a subject performing physical movements (e.g., knee bends) to collect 3D data points regarding limb mobility. While the Examiner views the exercise as an "activity," the data gathered by Van Andel pertains exclusively to how the user moves their body against a clinical benchmark. This information defines the user's physical capabilities and thus constitutes user measurements, at best. Van Andel fails to gather data on what happened within the activity's execution environment itself, in the sense explained above.
Since Van Andel does not disclose any activity measurements, the prior art also does not disclose, at least: "evaluating whether each progress marker associated with each skill level is satisfied based on one or more performance metrics derived from at least one of the user measurements, the analyzed measurements, and the combined measurements," as recited in claim 1 as amended
Based on the above response, the Examiner respectfully disagrees with the above that Van Andel does not teach activity measurements.
Dependent claims 34 and 35
New dependent claim 34 recites that "the one or more human skills
comprises at least one skill selected from physical skills, cognitive skills, behavioral skills, social skills, and mental abilities," and new claim 35 depends from claim 33 and recites that "a level of performance is determined for at least two different human skills." The analysis and assessment of these skills distinguish the claimed subject matter from Van Andel, which is narrowly confined to assessing limb mobility and physical range of motion (physical skill only).
The rejection is modified but respectfully maintained based on the above response and the claim amendments.
Applicant argues 35 USC §101, starting pg. 30 of Remarks:
Subject Matter Eligibility
Step 2A - Prong 1
As presently amended without prejudice or disclaimer, the independent claims are not directed toward an abstract idea, as can be seen by analogy to eligible claim 4 of USPTO Subject Matter Example 46, which is directed toward collection of animal-specific information within a herd including body position data, feeding behavior data, and movement pattern data:
4. A system for monitoring health and activity in a herd of dairy livestock
animals comprising:
- a memory;
- a processor coupled to the memory programmed with executable instructions, the instructions including a livestock interface for obtaining animal-specific information for a plurality of animals in the herd, wherein the animal-specific information comprises animal identification data and at least one of body position data, body temperature data, feeding behavior data, and movement pattern data; and
- a herd monitor including
(a) a radio frequency reader for collecting the animal-specific information from a plurality of animal sensors attached to the animals in the herd when the animal sensors are within proximity to the radio frequency reader, each animal sensor having a radio frequency transponder, and
(b) a transmitter for transmitting the collected animal-specific information to the livestock interface.
Just as claim 4 of Subject Matter Example 46 recites collecting, in a specific way, animal-specific information such as body position data, body temperature data, feeding behavior data, and movement pattern data, the Applicant's invention is directed toward using a computerized environment to collect, in a specific way, user-specific information including user measurements and activity measurements (in specific embodiments, the user measurement including, for example, body motion, facial muscles and expressions, or internal functions as recited in dependent claims).
The above Example 46 is about managing dairy livestock, not methods of organizing human activity.
In particular, the Applicant has amended the independent claims without prejudice or disclaimer in a manner that defines the claimed invention as not being directed to an abstract idea, but, rather, as being directed to collection of user information in a very specific way, inter alia, by:
providing an interactive experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience;
repeatedly obtaining a video of the user performing user actions;
extracting user measurements from the user actions (such as, for example, body motion, facial muscles and expressions, or internal functions as recited in dependent claims); and
extracting activity measurements including data pertaining to the execution of the experience itself and user interaction within the experience.
The measurements are analyzed, and activities of the experience are adapted dynamically and in real time based on the analysis, and skill progression of one or more human skills is determined based on combined measurements aggregated over a session of the experience.
More specifically, the present amendments to the independent claims define the independent claims, inter alia, as being directed to providing an interactive experience "forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience," repeatedly obtaining a video of the user performing user actions, extracting user measurements from the user actions, the activity measurements extracted from the provided interactive experience being defined by the present amendment as "comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience."
In the Examiner's response to the arguments in the Office action, the Examiner maintained that the Applicant's claimed invention was directed to an abstract idea, contending:
A person can obtain images, extract user and activity measurements, analyze the measurements, adapt activities based on the measurements, and determine skill progression.
Respectfully, if the above computer vision and sensor data is required, it is not being claimed.
There are similar remarks by the Examiner in the response to arguments section of the Office action to the effect that various other steps do not require a computer.
But, as is discussed above, the present amendments to the claims define the specific, computerized, way that user information is collected as including "forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience," repeatedly obtaining a video of the user performing user actions, extracting user measurements from the user actions, the activity measurements extracted from the provided interactive experience "comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience."
Respectfully, even if the claims are not part of an abstract process of organizing human activity, the limitations are too high level. Further, using a computer for an abstract idea has been shown not to be enough to make abstract claims statutory.
Thus, as presently amended, the Applicant's independent claims do recite steps that can only be performed by computerized equipment. Although a computer in itself is generic, as was noted by the Examiner in the response to arguments section of the Office action, claims reciting use of a generic computer to perform sequences of steps that do not amount to an abstract idea and to which the claims are directed are subject- matter eligible as is demonstrated by claim 4 of Subject Matter Example 46 discussed above.
Example 46 is about managing dairy livestock and is unrelated to the instant claims which are facilitating improvement of human skills (Title).
Thus, it cannot be said that the Applicant's independent claims as amended are "directed to" an abstract idea. The heart of the Applicant's inventive concept as defined by the Applicant's claims as amended lies in the specific way that user-specific information is collected (including, in specific embodiments, user measurements such as, for example, body motion, facial muscles and expressions, or internal functions as recited in dependent claims), by forming a dynamically running computerized environment comprising virtual elements configured to generate particular kinds of data and extracting user measurements and activity measurements.
Moreover, the formation of a dynamically running computerized environment comprising virtual elements configured to generate particular kinds of data and extracting user measurements and activity measurements, the user measurements including data pertaining to the execution of the experience itself and the user interaction within the experience, is much more than merely "managing personal behavior," as is recited by the Examiner in the response to arguments section of the Office action in characterizing certain claim limitations prior to the present claim amendments. Rather, the claimed sequence of steps of the Applicant's independent claims is directed toward a system for collecting data about activity of users in a specific way, just as claim 4 of Subject Matter Example is directed toward a system for monitoring health and activity in a herd of dairy livestock animals by collecting data in a specific way.
Collecting data in a specific way is not the same as providing additional elements or significantly more. Novel abstract specific steps are still abstract.
Nor are the Applicant's claims as amended directed, as the Examiner contends, toward a mental process, since steps such as "forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience" cannot be practically performed in the human mind. The August 4, 2025 Office memorandum entitled "Reminders on evaluating subject matter eligibility of claims under 35 U.S.C. 101" warns:
The mental process grouping is not without limits. Examiners are reminded not to expand this grouping in a manner that encompasses claim limitations that cannot practically be performed in the human mind.
Furthermore, analyzing the executing of a digital experience (activity measurements) with or without the combination with user measurements is not something a human can do with his or her mind. These operations require digital analysis, an extraction of digital data from the execution of the experience, analyzing the measurements together in a computational manner, which is not practical for the human mind.
The above does not require a computer and even if it did a person can perform measurements. However, based on Applicant’s arguments, the claim amendments, and further consideration, the mental process reasoning is withdrawn.
Moreover, extracting experience measurements is computer-specific operations (MPEP §2106.04(a)(2)(III)(A) - claims do not recite mental processes where "the human mind is not equipped" to perform the limitations (citing SRI, SiRF, Research Corp.).
The activity measurements relate to internal virtual game parameters, such as errors, retries, object selection, or timing, which are not visible or accessible to a human observer. These data must be extracted and processed by the computing system in real time - it cannot be done by a human with pen and paper.
Also, the claims require structured, real-time analysis of user and activity data during an interactive virtual experience. Unlike mere data gathering or display, the system dynamically and in real time adapts the experience based on the analyzed measurements, enabling a feedback loop that would not be possible without specialized computation.
There is no claimed feedback in the independent claims or specialized computation.
Step 2A - Prong 2
Although the Applicant's independent claims recite certain steps involving analyzing the measurements, adapting activities of the experience dynamically and in real time based on the analysis, and determining skill progression for one or more human skills based on the one or more combined measurements, these steps do not transform the Applicant's claimed invention into being "directed to" an abstract idea because even assuming arguendo that abstract ideas are recited in these additional steps, such ideas would be, under Step 2A, Prong 2, integrated into the above-described practical application of collecting, in a specific way, user-specific information by providing an interactive experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data.
Considering MPEP §2106.04(d): "a claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception,,"the amended claim is integrated into a real-world interactive system for evaluating and improving human skills.
As is discussed above, the independent claims recite specific technological components and steps, such as a camera for obtaining video of a user in a virtual experience, and automated extraction of user measurements and activity measurements. These operations are not generic data step, but, rather, a practical integration.
Using generic computer components or camera at a high level is not enough to make abstract claims statutory.
The Applicant refers once again to the MEMORANDUM of August 4, 2025 which emphasizes that "the additional limitations should not be evaluated in a vacuum, completely separate from the recited judicial exception. Instead, the analysis should take into consideration all the claim limitations and how these limitations interact and impact each other when evaluating whether the exception is integrated into a practical application." These limitations of claim 1 impose meaningful constraints on any alleged abstract concept. The claim does not monopolize or broadly cover the idea of human skill evaluation. Instead, it is limited to a specific computer-implemented method involving a virtual experience, real-time data capture, and adaptive interaction. The limitations work together to form a closed technical loop that dynamically analyzes user and activity measurements to determine skill progression. This integration of the alleged abstract idea into a responsive computing environment reflects a practical application, as emphasized in the August 4, 2025 Memorandum.
Preemption is not a two-way test for eligibility. Lack of preemption does not make claims statutory. Closed technical loop is not claimed.
Step 2B - Claims Amount to "Significantly More"
The independent claims are also patent-eligible under Step 2B because the claims recite a non-conventional and non-generic arrangement of computing operations. The claims define a specific ordered combination: providing an interactive virtual experience, capturing real-time user actions via video, extracting user and activity measurements, analyzing those data, dynamically adapting the experience, and determining skill progression based on combined measurements. These are not generic data manipulations or mere automation of human judgment. Instead, these actions reflect a technical solution that requires integrated sensor input, system adaptation, and structured data analysis. As in BASCOM, the inventive concept lies not in each individual component but in the claimed interaction between them to support real-time, personalized evaluation of human skills in a dynamic and adapted experience.
BASCOM improved Internet filtering, which improved computer technology itself. Based on the above response and claim amendments, the rejection is respectfully maintained but modified.
Double Patenting
No double patenting rejection is made at this time. While application 18/447491 is similar, it is broader and does not contain all of the elements of the instant claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12, 14-26, 28, 30-35 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1-12, 14-26, 28, 30-35 are directed to a method, system or product, which are statutory categories of invention. (Step 1: YES).
The Examiner has identified method Claim 1 as the claim that represents the claimed invention for analysis and is similar to system claims 30 and 32, and product claim 31.
Claim 1 recites the limitations of:
A computer-implemented method for determining performance of human skills of a user, the method comprising:
providing on a user device an interactive virtual experience in which the user can participate, the experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience;
while the user participates in the session of the experience, repeatedly executing:
obtaining camera output comprising a video of the user performing one or more user actions;
extracting one or more user measurements from the one or more user actions;
extracting one or more activity measurements from the provided experience comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience;
analyzing the one or more user measurements and the one or more activity measurements to obtain analyzed measurements;
adapting at least one of the activities in the experience dynamically and in real-time, based on at least the analyzed measurements; and
determining one or more combined measurements based on the analyzed measurements, wherein the combined measurements are aggregated over the duration of the session of the experience; and
determining skill progression for of one or more human skills based on the one or more combined measurements;
wherein the skill progression is determined based on predefined data structures stored in memory, the predefined data structures representing a plurality of skill levels associated with a respective human skill, each skill level being associated with a respective set of one or more progress markers;
wherein determining skill progression further comprises:
evaluating whether each progress marker associated with each skill level is satisfied based on one or more performance metrics derived from at least one of the user measurements, the analyzed measurements, and the combined measurements; and
determining that a given skill level is achieved when all progress markers associated with the given skill level are satisfied, thereby enabling advancement of the user to a subsequent skill level.
These above limitations, under their broadest reasonable interpretation, cover performance of the limitation as certain methods of organizing human activity. The claim recites elements, in non-bold above, which covers performance of the limitation as managing personal behavior, including teaching, following rules or instructions (e.g., providing interactive experience in which a user can participate, while user participates, repeatedly executing obtaining video of the user performing user actions, adapting activities based on measurements, determining skill progression for one or more human skills). Evaluating a progress marker is satisfied based on performance metrics (following rules and instructions) and determining that a skill level is achieved (teaching) are also abstract. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation as managing personal behavior, then it falls within the “Certain Methods of Organizing Human Activity” grouping of abstract ideas. Accordingly, the claim recites an abstract idea. Claims 1 and 19 are also abstract for similar reasons. (Step 2A-Prong 1: YES. The claims are abstract)
This judicial exception is not integrated into a practical application. In particular, the claims only recite: computer, user device, camera, memory (Claim 1); computer, user device, camera, memory (Claim 30); non-transitory computer storage medium, computer, user device, camera, memory (Claim 31); user device, computer, camera, memory (Claim 32). The computer hardware is recited at a high-level of generality (i.e., as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. The computerized environment comprising virtual elements is recited at a high level of generality. The camera can be part of the computing device, such as a smartphone (see specification, pg. 15, lines 8-26) and the video can be just digital data or information and would be similar to a mirror providing an image of a user. The extracting and analyzing measurements are recited at a high level of generality and are part of the abstract process of managing personal behavior as being used for determining skill progression which is teaching. Accordingly, these additional elements, when considered separately and as an ordered combination, do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Therefore claims 26-28 are directed to an abstract idea without a practical application. (Step 2A-Prong 2: NO. The additional claimed elements are not integrated into a practical application)
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered separately and as an ordered combination, they do not add significantly more (also known as an “inventive concept”) to the exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a computer hardware amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Accordingly, these additional elements, when considered separately and as an ordered combination, do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Steps such as obtaining (receiving) are steps that are considered insignificant extra solution activity and mere instructions to apply the exception using general computer components (see MPEP 2106.05(d), II). Thus claims 26-28 are not patent eligible. (Step 2B: NO. The claims do not provide significantly more)
Dependent claims 2-12, 14-26, 28, and 33-35 further define the abstract idea that is present in their respective independent claims 1 and 32 and thus correspond to Certain Methods of Organizing Human Activity and hence are abstract for the reasons presented above. The dependent claims do not include any additional elements that integrate the abstract idea into a practical application or are sufficient to amount to significantly more than the judicial exception when considered both individually and as an ordered combination. Claim 22 recites a generic camera that is part of a mobile device and is applied at a high level of generality. The other claims 2-21, and 23-29 are themselves abstract under certain methods of organizing human activity and/or mental processes. Claim 4 adjusts audio output or changes visual appearance of elements, which is recited at a high level of using computer devices to do what computer device do. Claims 6 and 16 recite avatar mimicking at least one user action or illustrate a more accurate version of body motion, which are recited at a high level of generality, and would be providing feedback to a user for teaching/training purposes, itself abstract. Therefore, claims 2-12, 14-26, 28, and 33-35 are directed to an abstract idea. Thus, the claims 1-12, 14-26, 28, 30-35 are not patent-eligible.
Examiner Request
The Applicant is requested to indicate where in the specification there is support for amendments to claims should Applicant amend. The purpose of this is to reduce potential 35 U.S.C. §112(a) or §112 1st paragraph issues that can arise when claims are amended without support in the specification. The Examiner thanks the Applicant in advance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7, 9-12, 14-26, 28, 30-35 are rejected under 35 U.S.C. 103 as being unpatentable over Pub. No. US 2018/0315247 to Van Andel in view of Pub. No. US 2017/0061817 to Mettler May.
Regarding claims 1 and 30-32
(claim 1) A computer-implemented method for determining performance of human skills of a user, the method comprising:
providing on a user device an interactive virtual experience in which the user can participate, the experience forming a dynamically running computerized environment comprising virtual elements configured to generate quantitative data relating to the continuous running of a session of the experience and the user participation in one or more activities in the experience;
Van Andel teaches:
Personal AR (user) device for performing therapy program (activities), where program provides interactive experience comprising activities…
“…The patient may be presented with an option of performing the therapy program at home, for example by utilizing a personal AR device, a smartphone with camera, a video capture device such as a Kinect from Microsoft of Redmond, Wash., or the like.” [0029]
User performs movement and completes a scheduled session…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
Example of user kicks virtual target (interactive virtual experience)…
“The AR display 810 may display the AR environment overlaid on the real environment. The display AR 810 may show the virtual target, using the AR device 801, in the fixed position in the AR environment. In an example, the AR device 801 may remove the virtual target from display in the AR environment in response to detecting a user interaction with the virtual target. For example, when the virtual target is a line or circle, the virtual target may be removed (e.g., fade out, popped, explode, etc.) when a user interacts with the virtual target (e.g., kicks the circle, moves a body part past the line, etc.).” [0059]
Fig. 1A, ref. 120 is virtual elements…
PNG
media_image1.png
218
252
media_image1.png
Greyscale
Avatar demonstrating (generate) 90 degree angle (quantitative data)…
“FIG. 1A and FIG. 1B illustrate real and virtual aspects of an augmented reality diagnosis system in accordance with some embodiments. In an example, the system 100 includes a video capture device 105, a display device 107, and AR glasses 110 worn by a user 115. The display device 107 may display an avatar 120 to demonstrate the movements the user 115 should perform. In an example, the avatar 120 is demonstrating the movement of lifting the right leg with the knee bent at a 90 degree angle. The demonstrated bent knee movement is also being performed by the user 115. The user 115 may perform a movement, as instructed, which is captured by the video capture device 105. In an example, a movement may be performed by the user 115 with one or more target body parts, and the movements may be captured by the video capture device 105. In an example, the video capture device 105 may be a camera, a depth camera, or an infrared detection device, such as the Kinect from Microsoft of Redmond, Wash.” [0030]
Fig.1A, ref. 125 to ref. 130 shows leg movement (dynamic)…
PNG
media_image2.png
386
448
media_image2.png
Greyscale
Example of computer system…
“In general, this disclosure describes techniques that may allow a computer system equipped with cameras and sensors to assess a patient's mobility and stability three-dimensionally. The system may capture three-dimensional data about the of a target body part on a person, such as an arm or leg and including the movement of joints such as the elbow or knee. The gathered data may then be analyzed by a remote clinician or an artificial intelligence system to determine the patient's range of motion and motor control. The three-dimensional data may then be analyzed by a clinician or artificial intelligence system against norms to determine a patient assessment. The assessment may then be a program to immerse the patient in three-dimensional tasks and challenges aimed at provoking the thresholds of the patient's measured mobility and stability.” [0017]
See Dynamic below.
while the user participates in the session of the experience, repeatedly executing:
obtaining camera output comprising a video of the user performing one or more user actions;
Fig. 1A, ref. 107 teaches example of camera (ref. 105) output of movement (video) of a user performing an action…
PNG
media_image3.png
340
484
media_image3.png
Greyscale
Video capture (camera) device…
“FIG. 1A and FIG. 1B illustrate real and virtual aspects of an augmented reality diagnosis system in accordance with some embodiments. In an example, the system 100 includes a video capture device 105, a display device 107, and AR glasses 110 worn by a user 115. The display device 107 may display an avatar 120 to demonstrate the movements the user 115 should perform. In an example, the avatar 120 is demonstrating the movement of lifting the right leg with the knee bent at a 90 degree angle. The demonstrated bent knee movement is also being performed by the user 115. The user 115 may perform a movement, as instructed, which is captured by the video capture device 105. In an example, a movement may be performed by the user 115 with one or more target body parts, and the movements may be captured by the video capture device 105. In an example, the video capture device 105 may be a camera, a depth camera, or an infrared detection device, such as the Kinect from Microsoft of Redmond, Wash.” [0030]
Another example…
“…In an example, a display screen 205 presents captured video (e.g., video captured of the movement performed by the user 225 or of the user 225). A virtual guide avatar 215 may be presented on the display screen 205. The virtual guide avatar 215 may be a computer generated figure, an animated figure, or a recording of a human instructor. The virtual guide avatar 215 may instruct the user 225 through a movement to perform (e.g., a speed of movement, where to extend a limb, how far to bend a joint, etc.)…” [0036]
Fig. 2, ref. 200…
PNG
media_image4.png
354
446
media_image4.png
Greyscale
User performs movement and completes a scheduled session…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
extracting one or more user measurements from the one or more user actions;
Capture (extracting) movement of body part with data (measurements) of limb (user measurements) from movement (actions)…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc. The system may capture movement of a target body part, such as the limb. Data collected from the captured movement may then be compared to a reference set, for example including range of motion capabilities of the limb (e.g., based on patient specific data, such as height, gender, etc.). The reference set may be gathered by capturing similar movements performed by the patient's mirroring target body part (e.g., when the patient is having an issue with a right leg, then a reference set of data may be collected from capturing movement of the patient's left leg)….” [0019]
Movement creates measurement set…
“The captured movement of the target body part creates the measurement set. The measurement set includes of three-dimensional data points for the movement performed by the target body part. The three-dimensional data points may be captured with video using a movement capture apparatus such as a camera or a Kinect from Microsoft of Redmond, Wash. The patient may have one or more sensors attached to the target body part, which may supply information about the movement of the target body part or its position in three-dimensional space. The one or more sensors may be attached directly to the patient or may be part of a worn apparatus such as a sleeve the patient places around an arm or leg. The one or more sensors may include one or more of an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a multi-axis sensor, or a contact pressure sensor. In an example, a patent may wear a sleeve that extends from an upper thigh to a lower calf. The sleeve may have one or more sensors located above the knee or below the knee such that the one or more sensors may collect data about the movement of the patient's knee joint.” [0021]
extracting one or more activity measurements from the provided experience comprising at least data pertaining to the execution of the experience itself and the user interaction within the experience;
Perform more movements (plural), such as bending knee and extending an arm, and measurement set (therefore includes activity measurements)…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022]
Example 1 and series of motions…
“Example 1 is a method comprising: capturing three-dimensional data points of a target body part of a subject during a series of motions, the target body part being mobile relative to a reference point of the subject, wherein the data points are members of a measurement set; classifying members of the measurement set to produce a movement set, members of the movement set including range of motion information for the target body part; obtaining a reference set corresponding to the target body part, the reference set defining range of motion capabilities for the target body part; computing a difference set, a member of the difference set defining a three-dimensional relationship between a member of the movement set and a corresponding member of the reference set; determining an assessment for the target body part based on the difference set; and outputting the assessment.” [0077]
Example 34 and include movement data normative to physical activities of the subject (therefore, both motion/movement and activities)…
“In Example 34, the subject matter of any one or more of Examples 1-33 optionally include wherein the reference set includes collected movement data normative to the physical activities of the subject.” [0110]
Avatar demonstrating 90 degree angle (activity measurement) and user performing the bent knee movement (user interaction with the experience)…
“FIG. 1A and FIG. 1B illustrate real and virtual aspects of an augmented reality diagnosis system in accordance with some embodiments. In an example, the system 100 includes a video capture device 105, a display device 107, and AR glasses 110 worn by a user 115. The display device 107 may display an avatar 120 to demonstrate the movements the user 115 should perform. In an example, the avatar 120 is demonstrating the movement of lifting the right leg with the knee bent at a 90 degree angle. The demonstrated bent knee movement is also being performed by the user 115. The user 115 may perform a movement, as instructed, which is captured by the video capture device 105. In an example, a movement may be performed by the user 115 with one or more target body parts, and the movements may be captured by the video capture device 105. In an example, the video capture device 105 may be a camera, a depth camera, or an infrared detection device, such as the Kinect from Microsoft of Redmond, Wash.” [0030]
analyzing at least one of the one or more user measurements and the one or more activity measurements to obtain analyzed measurements; and
Example of assessment (analyzing) the data (user measurement) and reference data set (activity measurements)…
“… A machine learning system may generate ideal or optimized data for the reference set based on patient specific parameters, such as hip to knee measurement, knee to ankle measurement, etc. The comparison between the patient's data and the reference set may determine a difference set. An assessment for the patient may then be determined based on the difference set. The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
Movements Measurement set…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022]
Example of evaluation system and determine (analyze) movement set (user and activity measurements)…
“FIG. 4A illustrates a patient evaluation system 400 in accordance with some embodiments. The systems and techniques described herein may use a reference set and a movement set to determine a difference set, which is then used to determine an assessment for a target body part. In an example, a patient may identify a target body part for therapy (e.g., the target body part is causing pain to the patient, the target body part is causing the patient to have difficulty walking, the patient has stiffness with the target body part, etc.)…” [0042]
adapting at least one of the activities in the experience dynamically and in real-time, based on at least the analyzed measurements; and
Example of adjust therapy…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
Example of guide patient for proper motion utilizing augmented reality, therefore dynamically and in real time…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022] Inherent with guide patient for proper motion by utilizing augmented reality is dynamically and in real time.
determining one or more combined measurements based on the analyzed measurements, wherein the combined measurements are aggregated over the duration of the session of the experience; and
Measurement set (combined measurements)…
“The captured movement of the target body part creates the measurement set. The measurement set includes of three-dimensional data points for the movement performed by the target body part. The three-dimensional data points may be captured with video using a movement capture apparatus such as a camera or a Kinect from Microsoft of Redmond, Wash. The patient may have one or more sensors attached to the target body part, which may supply information about the movement of the target body part or its position in three-dimensional space. The one or more sensors may be attached directly to the patient or may be part of a worn apparatus such as a sleeve the patient places around an arm or leg. The one or more sensors may include one or more of an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a multi-axis sensor, or a contact pressure sensor. In an example, a patent may wear a sleeve that extends from an upper thigh to a lower calf. The sleeve may have one or more sensors located above the knee or below the knee such that the one or more sensors may collect data about the movement of the patient's knee joint.” [0021]
Example of camera to capture three-dimensional movement…
“The system 100 may be used to capture three-dimensional movement. For example, the video capture device 105 may include a movement capture apparatus. The movement capture apparatus may include two or more infrared sensors or cameras to detect or capture three-dimensional movement. The two or more cameras can triangulate positions of the user 115 or identified portions of the user 115 moving within a three-dimensional space. The video capture device 105 may include a camera to capture video in conjunction with infrared captured movement.” [0032]
See Aggregate below.
determining skill progression for of one or more human skills based on the one or more combined measurements;
{
From Applicant’s specification on skills…
“The memory 230 can also store Skills 234, including one or more non-technical human skills 235a-23Sn. Each of the skills may refer to human skills that enable humans to function in in every day life. A skill 235a of human skills 235a-23Sn may refer to motor, cognitive, social, emotional, language, and adaptive skill as described above. The skill 235a may consist of a name and a description, and a predefined number of levels a1-an. The skill level a1 may have a description. A certain level may be determined as a target level for a particular user 110, e.g. as a part of the treatment plan, such that achieving that target level is considered as an achieved goal in the treatment plan. Each of the skill levels a1-an may also be associated with a respective set of one or more progress markers a1. The progress markers may indicate measures to evaluate the progress of a patient in a skill level. A single progress marker may be used to determine when a skill level is reached, so the user 110 may move to the next level. Different types of progress markers exist. Progress markers can be defined in tabular form. A level can be reached once all progress markers are met, either all at once or over time.” (pg. 17, lines 3-16)
Therefore, skill is anything that enables a human to function. Walking, running, and exercising would be a motor skills.
}
Example of patient progress of number of repetitions (skill progression) and increasing weight to increase difficulty for the exercise (skill progression based on measurements)…
“The physical object 340 may include a sensor 355. The sensor 355 may be used to track patient progress, such as a duration or number of repetitions completed by the user 305. In an example, the sensor 355 may be used to identify the physical object 340 to the AR device 310. Once identified, the AR device 310 may select the virtual object 345 that corresponds to the physical object 340. For example, if the user 305 has a ten pound weight and a five pound weight available, these may be identified via sensors in the weights, and the AR device 310 may determine the five pound weight is to be used in an exercise, and the virtual object 345 may resemble the five pound weight. The virtual object 345 may change to resemble the ten pound weight in another exercise or when increasing weight to increase difficulty for the exercise. In an example, the AR device 310 may display a virtual movement of a limb.” [0040]
Example of improving physical movement (motor skill) and different levels of performance, such as greater range (measurement) of motion…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
wherein the skill progression is determined based on predefined data structures stored in memory, the predefined data structures representing a plurality of skill levels associated with a respective human skill, each skill level being associated with a respective set of one or more progress markers;
{
From Applicant’s specification on skill progression and progress markers…
“The memory 230 can also store Skills 234, including one or more non-technical human skills 235a-235n. Each of the skills may refer to human skills that enable humans to function in in every day life. A skill 235a of human skills 235a-235n may refer to motor, cognitive, social, emotional, language, and adaptive skill as described above. The skill 235a may consist of a name and a description, and a predefined number of levels a.sub.1-a.sub.n. The skill level a.sub.1 may have a description. A certain level may be determined as a target level for a particular user 110, e.g. as a part of the treatment plan, such that achieving that target level is considered as a.sub.n achieved goal in the treatment plan. Each of the skill levels a.sub.1-a.sub.n may also be associated with a respective set of one or more progress markers a.sub.1. The progress markers may indicate measures to evaluate the progress of a patient in a skill level. A single progress marker may be used to determine when a skill level is reached, so the user 110 may move to the next level. Different types of progress markers exist. Progress markers can be defined in tabular form. A level can be reached once all progress markers are met, either all at once or over time.” [0113]
Therefore, memory storing progress markers, where the marker itself could be anything, including a level of progress, a milestone, an achievement toward a skill level.
}
Storage device with data structures…
“The storage device 916 may include a machine readable medium 922 on which is stored one or more sets of data structures or instructions 924 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 924 may also reside, completely or at least partially, within the main memory 904, within static memory 906, or within the hardware processor 902 during execution thereof by the machine 900. In an example, one or any combination of the hardware processor 902, the main memory 904, the static memory 906, or the storage device 916 may constitute machine readable media.” [0064]
wherein determining skill progression further comprises:
evaluating whether each progress marker associated with each skill level is satisfied based on one or more performance metrics derived from at least one of the user measurements, the analyzed measurements, and the combined measurements; and
See Storage and Progress Marker below.
determining that a given skill level is achieved when all progress markers associated with the given skill level are satisfied, thereby enabling advancement of the user to a subsequent skill level.
See Storage and Progress Marker below.
Dynamic
Van Andel teaches motion sensors with an interactive virtual environment between a person and a virtual image. They do not literally teach dynamic.
Mettler May also in the business of an interactive environment teaches:
User’s environment with movement dynamics…
“One or more motion sensors, either embedded, or deployed in the user's environment, can be used with the system to provide measurements of movement dynamics encompassing one or more users, actors and their associated equipment (if any). As will be appreciated by those skilled in the art, given the depth of hierarchical levels of the movement system, the scope of motion analysis can be conducted at multiple levels. For example, it could focus on neuro-motor aspects, movement technique and structure, and outcomes, all the way up to tactical and strategic levels.” [0071]
Dynamic environments….
“Complex movements frequently involve a sequence of distinct movement phases. Therefore, motor programs encompass mechanisms in order to time and sequence these elements. The movement phases are usually formed to support various functional characteristics, such as biomechanical constraints, task structure and various sensory interactions with the environment. Movement segments can be conceptualized as a movement directed towards a subgoal, each with its particular biomechanical and sensory-motor constraints. This structure allows the break down complex movements into simpler movement elements. It can also help acquisition of complex movement skills and support the flexibility and adaptability needed to operate in dynamic and uncertain environments.” [0077]
Real-time feedback an session…
“…The feedback provided to the user can be one or more of each of a phase transition cue, a phase profile cue, an alert, and an outcome validation cue. Additionally, the feedback can be provided real-time, near real-time, or at a time remote to an actual movement session. Alerts can be computed from the map of movement feature to biomechanical features (e.g. to protect from injury or wear). One or more movement units can be further segmented into two or more movement phases. Two or more movement phases can be decomposed into two or more synergies wherein the two or more synergies are one or more of each of biomechanical relationships and neuro-muscular relationships. A local host device, such as a local computing device, can be configurable to communicate one or more of each of a training information and a cuing information to the user. One or more parsed motion data for change can be prioritized based on whether a feature of the movement unit is changeable and will impact a quality or outcome of a repeated movement. Additionally, the collected motion data can be one or more of a user motion data and a user controlled device motion data.” [0027]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of Van Andel the ability to determine dynamic and real-time movement in a session as taught by Mettler May since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Van Andel who teaches feedback, session, and a virtual environment. It would be obvious that this would be a dynamic environment.
Aggregate
The combined references teach measurement. They do not teach aggregate.
Mettler May also in the business of measurement teaches:
Data aggregated for a session…
“The outputs of these assessments are typically provided after a training or play session. The data is presented as summaries of session performance, as well as time. The data is also aggregated to provide statistical trends. The main shortcoming of these products is that the analysis is based on outcome variables (referred to as knowledge of results in the human skill literature) and thus does not provide actionable information that can be leveraged directly for training.” [0012]
Aggregates user’s movement performance data…
“The typical user scenario considers a user who regularly performs and trains skills for an activity over a period of months or years, and who wants to track progress and receive instructions for developing, maintaining or rehabilitating movement skill from session to session. More specifically, a user may perform an activity two to three times a week, or even more frequently. The platform aggregates and assesses the user's movement performance data within the assessment loop (see FIG. 1). This process generates a movement repertoire map and skill model which describes the user's current and historical skill states. At any time during the activity, the user can choose to activate feedback augmentation, in the form of instructions and/or feedback cueing (FIG. 12). Even if the user does not enable augmentations, the assessment loop can operate in a passive mode, continuing to collect data whenever the user is engaged in an activity and continually updating the skill model.” [0271]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to aggregate measurements as taught by Mettler May since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Mettler May who teaches aggregating data and the combined references benefit as data is collected over time for analysis, therefore it would be obvious to aggregate data
Skill Progression
The combined references teach rehabilitation. They also teach exercise (skill) and progression. They do not literally teach skill with progression.
Mettler May also in the business of rehabilitation teaches:
Motion skills and progress of movement skills (skill progression)…
“Humans rely on motion skills to perform daily tasks ranging from actions essential to our autonomy to more specialized domains requiring highly refined motion skills. Professional athletes, musicians, surgeons, and even elite amateurs require thousands of hours of systematical and focused training, as well as continued training to maintain high skill levels. Even simple daily acts involve complex coordination of a range of processes, from sensing and motor-control to perception and cognition. Learning, maintaining, and rehabilitating movement skills are valuable, but at the same time, complex and challenging tasks. Acquiring and maintaining specialized movement skills takes time. Progress of movement skills does not develop linearly with training time. Rather, skills progress exponentially with what is left to learn in order to be proficient at the movement. Movement performance relies on a broad range of functions (e.g., sensory, perceptual, planning, cognition). Many movement skills within the category of complex movement are unnatural and therefore require adaptation of innate functions to the specific task requirement and familiarization with the external task elements. Complex movement requires coordinating large numbers of muscles and body segments. The computational requirements for movement need to be initiated within short time-frames and often need to be adapted during performance. Movement is typically learned by trial and error, focusing on the outcome. The specific details regarding how movement is organized are only known implicitly in procedural memory. Explicit knowledge surrounding movement details are typically not used during practice and execution. Movements often unfold too quickly to even be perceived. Complex movements involve too many dimensions. For example, the path of a piece of equipment, such as a tennis racket, involves combining three translational and rotational variables (e.g., six degrees of freedom).” [0005]
Cameras for extracting motion data…
“Measurements y can be obtained from different components of the human actors, equipment, or system. Typically, instrumentation is designed to obtain measurements that encompass relevant variables for the particular level of analysis. For example, in the analysis of human tennis stroke performance, the states, or a subset of the racket motion may be sufficient. Additional measurements about the body segments (e.g. arm, legs, feet, etc.) are necessary to enable an analysis of the movement on the court, the footwork, or the body motion such as the kinematic chain or other movement units. These measurements can be obtained using a variety of technologies, including inertial motion unit (IMU), visual or optical tracking systems, etc. Examples include the use of video cameras that capture the broader agent behavior and the task environment. Vision processing can also be used to extract information about the motion of individual body segments. An important class of measurements are those that capture physiological quantities. For example, a gaze tracking system to measure the visual attention. Thus, as shown in FIG. 2, a user 10 (or player) holding a tennis racket 20 which impacts a ball 30 during the swing of the racket has a gaze 12 which follows a trajectory which changes during the motion. One or more motion tracking cameras 210, 210′ can be provided which capture data related to the user 10, the tennis racket 20, the ball 30, the motion and the environment. Inertial measurement units can be embedded or affixed to the equipment; worn by the agent to measure the movement of body segment; or even placed on the skin or implanted in the body to measure muscle activity or neural signals involved in the control of muscles. See FIG. 5, which illustrates an arm 14 of a user with a surface electromyogram (EMG) sensor 510, an intramuscular electrode 520 and an IMU 530 for illustration purposes. The IMU measures velocity, orientation and gravitational forces using a combination of accelerometers and gyroscopes, and sometimes also magnetometers.” [0157]
Skills and track progress…
“The typical user scenario considers a user who regularly performs and trains skills for an activity over a period of months or years, and who wants to track progress and receive instructions for developing, maintaining or rehabilitating movement skill from session to session. More specifically, a user may perform an activity two to three times a week, or even more frequently. The platform aggregates and assesses the user's movement performance data within the assessment loop (see FIG. 1). This process generates a movement repertoire map and skill model which describes the user's current and historical skill states. At any time during the activity, the user can choose to activate feedback augmentation, in the form of instructions and/or feedback cueing (FIG. 12). Even if the user does not enable augmentations, the assessment loop can operate in a passive mode, continuing to collect data whenever the user is engaged in an activity and continually updating the skill model.” [0271]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to determine skill progression as taught by Mettler May since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Van Andel who teaches improving physical movement. Van Andel benefits by determining skill progression as they determine exercise progression.
Storage and Progress Marker
The combined references teach skill progression. They do not teach storage and progress marker.
Mettler May also in the business of skill level teaches:
Data storage for skill levels (markers)…
“… The platform combines components used for communication, user interface, computing and data storage. In particular, it combines cloud computing, which makes it possible to store, manage and process large amounts of data, with embedded sensing and computing, which makes it possible to build the necessary wearable hardware for data collection and real time feedback. Such a system opens opportunities for capabilities that go well beyond what can be achieved with data from a single individual. Data from a population of players opens the possibility to analyze skills from different skill levels and differences in biomechanics, age, health, equipment, etc. Therefore, an important aspect of the platform is the organization and management of the collective motion data, motion models, skill models, cueing laws, etc. A capability enabled by population data is the availability of information about movement technique at different skill levels. This can be used to cross-reference the skill analysis of similarly skilled individuals provides opportunities to borrow technical advantages from one individual to enhance training of a second individual. As will be appreciated by those skilled in the art, the use of a cloud computing environment is just one configuration and other computing environments can be used without departing from the scope of the disclosure.” [0180]
Tracking skill development (progress) and stored…
“The movement segmentation and finite-state models for the evolving movement architecture shown in FIGS. 15A-E can be extracted from individual users in a data-driven fashion. These models can be stored in a library and enable tracking of the skill development process as shown in FIG. 17. With data from a sufficiently broad skill population (both in terms of style and technique), knowledge about the skill acquisition process in terms of pattern formation, associated with movement architecture, etc., it is possible to model long-term skill acquisition. This information can then be use to design long-term training protocols that can account for broad individual characteristics such as morphological and physical characteristics. The understanding of the development of the movement architecture can be applied to refine an individual's motion structure. Capturing the development of the movement architecture through a formal hierarchical model structure makes it possible to design training schedules that allow systematic skill development paths based on the understanding of the refinement in movement architecture.” [0266]
Example of track progress and user’s movement performance data (progress marker, user measurement)…
“The typical user scenario considers a user who regularly performs and trains skills for an activity over a period of months or years, and who wants to track progress and receive instructions for developing, maintaining or rehabilitating movement skill from session to session. More specifically, a user may perform an activity two to three times a week, or even more frequently. The platform aggregates and assesses the user's movement performance data within the assessment loop (see FIG. 1). This process generates a movement repertoire map and skill model which describes the user's current and historical skill states. At any time during the activity, the user can choose to activate feedback augmentation, in the form of instructions and/or feedback cueing (FIG. 12). Even if the user does not enable augmentations, the assessment loop can operate in a passive mode, continuing to collect data whenever the user is engaged in an activity and continually updating the skill model.” [0271]
Skill states (skill level achieved) based on movement repertoire map (all progress markers)…
“The typical user scenario considers a user who regularly performs and trains skills for an activity over a period of months or years, and who wants to track progress and receive instructions for developing, maintaining or rehabilitating movement skill from session to session. More specifically, a user may perform an activity two to three times a week, or even more frequently. The platform aggregates and assesses the user's movement performance data within the assessment loop (see FIG. 1). This process generates a movement repertoire map and skill model which describes the user's current and historical skill states. At any time during the activity, the user can choose to activate feedback augmentation, in the form of instructions and/or feedback cueing (FIG. 12). Even if the user does not enable augmentations, the assessment loop can operate in a passive mode, continuing to collect data whenever the user is engaged in an activity and continually updating the skill model.” [0271]
Specifies movement patterns that must be formed to improve skills (skill level)…
“The movement model and augmentation mechanisms provide the components for computer assisted training (CAT). CAT handles various aspects of the management and operation of the augmented training process. The CAT system has two primary subsystems: a Training Agent (TA), which serves as virtual coach that manages the training process, and a Cueing Agent (CA), which controls the Cueing System and manages the various cueing functions during play or performance. The Training Agent analyzes the diagnostic results to formulate a Training Schedule (TS). At the repertoire level, the TA determines which movement patterns must be optimized or refined, and also specifies what new movement patterns must be formed. The former improves the skills of specific movement patterns and the latter is directed at broadening the repertoire either through the introduction of a new pattern or by helping the differentiation of movement within an existing pattern.” [0023]
Movement technique at different skill levels…
“…A capability enabled by population data is the availability of information about movement technique at different skill levels. This can be used to cross-reference the skill analysis of similarly skilled individuals provides opportunities to borrow technical advantages from one individual to enhance training of a second individual. As will be appreciated by those skilled in the art, the use of a cloud computing environment is just one configuration and other computing environments can be used without departing from the scope of the disclosure.” [0180]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to have storage and progress markers for skills as taught by Mettler May since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Mettler May who teaches the benefits of tracking progress and it would be beneficial to store such information. The combined references benefit as they also teach skill progression as they determine exercise progression.
Regarding claim 2
The method of claim 1, the method further comprising:
based on skill progression, generating feedback
comprising at least one of: one or more skills, current user’s level in a scale of levels associated with the experience, and progress over the levels; and
Van Andel teaches:
Feedback based on of movement (therefore, generating feedback)…
“The systems and methods herein describe using a movement capture apparatus, such as a camera or an infrared (IR) detection device, to capture movement information from a patient. The systems and methods may be used for remote diagnosis or feedback. In an example, a system may provide a user interface, a video display, an augmented reality (AR) display, a virtual reality (VR) display, audible alerts, haptic feedback, a non-contact alert, or the like to present feedback.” [0018]
Example of movement along a path and exercise (one or more skill) and determined successful or not (feedback)…
“The display 415 may be used to animate the virtual representation 430 of the user 405 to perform a movement along a path. The path may be used to show a patient how to perform a prescribed movement. The path may be used to determine whether the movement by the user was successful. In an example, the path may be used to determine whether a user has successfully completed an exercise or movement. For example, if the user completes the exercise within the path, the exercise may be determined to be completed successfully. If the user moves outside the path while attempting to complete the exercise, the exercise may be determined to not be completed successfully.” [0048]
providing the feedback;
Present feedback…
“The systems and methods herein describe using a movement capture apparatus, such as a camera or an infrared (IR) detection device, to capture movement information from a patient. The systems and methods may be used for remote diagnosis or feedback. In an example, a system may provide a user interface, a video display, an augmented reality (AR) display, a virtual reality (VR) display, audible alerts, haptic feedback, a non-contact alert, or the like to present feedback.” [0018]
whereby the provided feedback facilitates the user to improve the
skill progression by improving performance of at least some of the user measurements.
Example of challenges (improving performance) to provoke thresholds (feedback with progression) of measured mobility and stability (measurements)…
“In general, this disclosure describes techniques that may allow a computer system equipped with cameras and sensors to assess a patient's mobility and stability three-dimensionally. The system may capture three-dimensional data about the of a target body part on a person, such as an arm or leg and including the movement of joints such as the elbow or knee. The gathered data may then be analyzed by a remote clinician or an artificial intelligence system to determine the patient's range of motion and motor control. The three-dimensional data may then be analyzed by a clinician or artificial intelligence system against norms to determine a patient assessment. The assessment may then be a program to immerse the patient in three-dimensional tasks and challenges aimed at provoking the thresholds of the patient's measured mobility and stability.” [0017]
Regarding claim 3
The method of claim 1, wherein at least one activity measurement is not related to success of the user in completing goals of the experience but is used in the determining the skill progression.
Van Andel teaches:
Example of improving a technique or range of motion for sports movement such as a swing (skill)…
“A person may also wish to increase or improve technique or range of motion for a sports-related movement, such as a when swinging a golf club. Similar issues arise for receiving instructions related to sports movements as those described above for the patient. Further, finding a qualified trainer may be difficult.” [0004]
Capture movements and motion sensitive sensor (measurement)…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
Another example of difference set used to improve sports movement (skill progression)…
“The movements of the user 605 may be captured through a camera or sensor attached to the user 605 or to a sports instrument, such as a golf club or baseball bat. The user 605 may be instructed, such as using an avatar 615 on a display or within an AR field of view in the first direction 610, to perform a movement related to the predetermined sports movement. In an example, three-dimensional data points of the user's movements are captured as a measurement set. The three-dimensional data points of the measurement set may be classified to produce a movement set which includes the range of motion for the target body parts related to the sports movement. A reference set corresponding to the sports movement may be obtained. In an example, the reference set may include three-dimensional data points captured from multiple professional athletes of the corresponding sport. A different set may be computed wherein a member of the difference set defines a three-dimensional relationship between a member of the movement set and a corresponding member of the reference set. An assessment for how the user 605 may improve performance may be determined based on the difference set. The assessment may include a physical therapy program, a stretch the user performs before the sports activity, or a movement or strengthening technique the user 605 is guided through by the avatar 615.” [0053]
Regarding claim 4
The method of claim 1, wherein adapting at least one of the activities comprises performing at least one of the following:
changing a level of difficulty of at least one of the activities,
replacing virtual activities included in the experience,
activating and/or deactivating specific activities,
adjusting at least one of the activities to assist the user in participating in the experience,
providing guidance on experience in the activity,
Van Andel teaches:
Guidance…
“The assessment provided to the patient may include a physical therapy program including various movements or stretches the patient is to perform with the target body part. The system used to capture the movements of the target body part for diagnosis may be used to assist in guiding the patient through the therapy routines.” [0020]
adjusting or triggering audio output, or
Example of audible feedback (output)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
changing visual appearance of elements included in the experience.
Example of arrow pointing, etc. (changing visual appearance)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
Regarding claim 5
The method of claim 1, wherein at least one of the activities involves interaction with a plurality of virtual interactive elements, and wherein adapting at least one of the activities comprises changing the interactive elements in response to a user action.
Van Andel teaches:
Example of user kicks virtual target of a line or circle (interactive virtual experience, where line circle is an element), and fade out (changing the element in response to the user action)…
“The AR display 810 may display the AR environment overlaid on the real environment. The display AR 810 may show the virtual target, using the AR device 801, in the fixed position in the AR environment. In an example, the AR device 801 may remove the virtual target from display in the AR environment in response to detecting a user interaction with the virtual target. For example, when the virtual target is a line or circle, the virtual target may be removed (e.g., fade out, popped, explode, etc.) when a user interacts with the virtual target (e.g., kicks the circle, moves a body part past the line, etc.).” [0059]
Regarding claim 6
The method of claim 5, wherein at least one element includes an avatar configured for mimicking at least one of the user actions.
Van Andel teaches:
User may control avatar (avatar configured for mimicking) based on user movement (actions)…
In an example, the user 605 may control movement of the avatar 615. For example, the user 605 may hold an apparatus used for a sport such as a golf club or baseball bat. The sport apparatus may have a motion sensitive sensor such as an accelerometer, a gyroscope, an inertial measurement unit, or a nine-axis sensor. The motion sensor may track the movement of the sport apparatus. The tracked movement may be used by the sports training AR system 600 to display the movement using a representation of the movement via the avatar 615 performing the movement virtually. In another example, the avatar 615 may be used to display a training movement. For example, the training movement may represent an idealized or professional movement of the sport movement attempted by the user 605. The user 605 may control the avatar 615 using the sport apparatus 635 such that the avatar 615 moves a corresponding virtual sport apparatus 640 similarly to the sport apparatus 635. The avatar 615 may move the virtual sport apparatus 640 according to the training movement. Thus the movement of the avatar 615 may generally follow the movement of the user 605, but may differ by displaying an improved or training movement for the user to attempt to match. In this way, the user 605 may move and control the sport apparatus 635 while viewing similar movements by the virtual sport apparatus 640, the similar movements representing an ideal or improved movement.” [0052]
Joint or location of avatar represents (mimics) user pain…
“The view 300 may allow the user 305 or a remote user, such as a clinician, to select a joint or location on an AR avatar. By selecting a joint or location on the AR avatar, the user 305 or remote user may indicate what hurts or needs attention on the user 305, select a side of the body, be prompted for a pain level, or the like. In another example, the user 305 may select options from a user interface within the view 300, such as a scale bar to select pain. The view 300 may include a questionnaire for the patient to answer so that a therapist may evaluate progress or determine exercises for the patient.” [0041]
Regarding claim 7
The method of claim 1, wherein the one or more user measurements comprise a body motion.
Van Andel teaches:
Fig .6 and swing…
PNG
media_image5.png
272
406
media_image5.png
Greyscale
“The movements of the user 605 may be captured through a camera or sensor attached to the user 605 or to a sports instrument, such as a golf club or baseball bat. The user 605 may be instructed, such as using an avatar 615 on a display or within an AR field of view in the first direction 610, to perform a movement related to the predetermined sports movement. In an example, three-dimensional data points of the user's movements are captured as a measurement set…” [0053]
Regarding claim 9
The method of claim 1, wherein the one or more user measurements comprise internal functions.
Van Andel teaches:
Heart rate, oxygen, blood pressure (internal measurements)…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like. In an example, a user may be tasked with performing twenty repetitions of a leg lift. Utilizing the sensor 445, such as an accelerometer, a gyroscope, or a nine-axis sensor, the movement of the leg lift may be tracked. The sensor 445 may count the number of leg lift repetitions performed by the user or monitor the form of a leg lift. The sensor 445 may be used to determine if the user is experiencing fatigue or having difficulty completing a repetition or a task. For example, data from the sensor 445 may indicate that a adherence to form is decreasing for subsequently performed leg lifts. Tasks or aspects of a task (e.g., intensity or duration) may be altered in response to determining that the data indicates the proper form is decreasing. In an example, the data may be analyzed to determine the user's level of fatigue from performing repetitions.” [0046]
Regarding claim 10
The method of claim 1, wherein the one or more activity measurements relate to continuous running of the experience.
Van Andel teaches:
Example of completes a scheduled session or therapy, therefore, continuous activity…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
Regarding claim 11
The method of claim 1, wherein analyzing the one or more user measurements comprise comparing at least one of the extracted user measurements to a corresponding reference user measurement.
Van Andel teaches:
Example of data captured from movement of user and compare to reference, where reference can be patients movement…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc. The system may capture movement of a target body part, such as the limb. Data collected from the captured movement may then be compared to a reference set, for example including range of motion capabilities of the limb (e.g., based on patient specific data, such as height, gender, etc.). The reference set may be gathered by capturing similar movements performed by the patient's mirroring target body part (e.g., when the patient is having an issue with a right leg, then a reference set of data may be collected from capturing movement of the patient's left leg)…” [0019]
Regarding claim 12
The method of claim 1, wherein extracting the one or more activity measurements comprise measuring one or more of performance, goals, or events within the experience, or a combination thereof.
Van Andel teaches:
Goal…
“In system 100, the user 115 may wear the AR glasses 110. In an example, view 150 may represent a view the user 115 sees while wearing the AR glasses 110. The view 150 may include virtual and real aspects in an environment, such as the user's leg (real) augmented by a guide (virtual). In an example, the user 115 may look downward to observe the view 150 and the user may see the raised leg 135 with bent knee. The view 150 using the AR glasses 110 may include a virtual object or goal for the target body part to reach or extend to, such as a virtual line 140. The virtual line 140 may represent an extension distance for the target body part to achieve during a movement. The AR glasses 110 may display virtual aspects in a real environment, such as in view 150.” [0034]
Regarding claim 14
The method of claim 2, further comprising: providing the feedback to a third party related to the user.
Van Andel teaches:
Remote clinician [to receive] limb mobility problem (feedback)…
“In an example, a system may be used to remotely or locally diagnose a limb mobility problem for a patient. For example, a remote clinician or an artificial intelligence system (e.g., a machine learning system trained to detect limb mobility problems) may be used to diagnose an issue. The system may provide feedback to the patient, such as an assessment, a recommended physical therapy technique, surgery, a consultation, etc…” [0019]
Regarding claim 15
The method of claim 2, wherein at least one of the user actions comprises a body motion, and wherein the feedback comprises data relating to a more accurate version of the body motion.
Van Andel teaches:
Movements performed in certain manner to accurately portray movement and guide (feedback) patient…
“The patient may be directed to perform one or more movements with the target body part (e.g., bending of the knee, raising and lowering an extended leg, extending an arm in a certain direction, etc.). These movements are captured to collect the three-dimensional data points which comprise the measurement set. The movements of the target body part may be performed in a certain manner for the collected three-dimensional data points to accurately portray the movement capabilities of the target body part and be relatively comparable to the reference set data. The system may guide the patient through the steps for proper motion by utilizing augmented reality.” [0022]
Regarding claim 16
The method of claim 15, wherein the feedback comprises an avatar configured to illustrate a more accurate version of the body motion.
Van Andel teaches:
Avatar demonstrate movements that user should perform (more accurate version)…
“FIG. 1A and FIG. 1B illustrate real and virtual aspects of an augmented reality diagnosis system in accordance with some embodiments. In an example, the system 100 includes a video capture device 105, a display device 107, and AR glasses 110 worn by a user 115. The display device 107 may display an avatar 120 to demonstrate the movements the user 115 should perform. In an example, the avatar 120 is demonstrating the movement of lifting the right leg with the knee bent at a 90 degree angle. The demonstrated bent knee movement is also being performed by the user 115. The user 115 may perform a movement, as instructed, which is captured by the video capture device 105. In an example, a movement may be performed by the user 115 with one or more target body parts, and the movements may be captured by the video capture device 105. In an example, the video capture device 105 may be a camera, a depth camera, or an infrared detection device, such as the Kinect from Microsoft of Redmond, Wash.” [0030]
Regarding claim 17
The method of claim 2, wherein the experience, and one or more levels associated with the experience are selected based on a personalized treatment plan generated to facilitate the user to improve one or more human skills by improving performance of at least some of the user measurements.
Van Andel teaches:
Physical therapy program (treatment plan) for the patient (personalized)…
“The assessment provided to the patient may include a physical therapy program including various movements or stretches the patient is to perform with the target body part. The system used to capture the movements of the target body part for diagnosis may be used to assist in guiding the patient through the therapy routines.” [0020]
Example of tasks altered based on response (levels associated with the experience)…
“The sensor 445, which may be located in or on sleeve 440 may include an accelerometer, a magnetometer, a gyroscope, an optical sensor, an ultrasonic sensor, an inertial measurement unit, a nine-axis sensor, a contact pressure sensor, or the like. The sensor 445 may include a physiological sensor such as a heart rate monitor, a blood-oxygen monitor, a blood pressure monitor, or the like. The sensor 445 may be used to track movement, guide a movement, count repetitions, measure fatigue, detect muscle fatigue, detect exhaustion, or the like. In an example, a user may be tasked with performing twenty repetitions of a leg lift. Utilizing the sensor 445, such as an accelerometer, a gyroscope, or a nine-axis sensor, the movement of the leg lift may be tracked. The sensor 445 may count the number of leg lift repetitions performed by the user or monitor the form of a leg lift. The sensor 445 may be used to determine if the user is experiencing fatigue or having difficulty completing a repetition or a task. For example, data from the sensor 445 may indicate that a adherence to form is decreasing for subsequently performed leg lifts. Tasks or aspects of a task (e.g., intensity or duration) may be altered in response to determining that the data indicates the proper form is decreasing. In an example, the data may be analyzed to determine the user's level of fatigue from performing repetitions.” [0046]
“…An assessment for how the user 605 may improve performance may be determined based on the difference set. The assessment may include a physical therapy program, a stretch the user performs before the sports activity, or a movement or strengthening technique the user 605 is guided through by the avatar 615.” [0053]
Regarding claim 18
The method of claim 17, wherein the treatment plan is automatically generated, based on data pertaining to the user.
Van Andel teaches:
Using machine-learning (automatically generate) to determine motion range capabilities (treatment plan)…
“The technique 1100 may include determining performed range of motion information for the target body part based on the movement set. The technique 1100 may include obtaining range of motion capabilities for the target body part. In an example, obtaining the range of motion capabilities includes obtaining the range of motion capabilities from a database. The technique 1100 may include training range of motion capabilities data using a machine-learning technique. The range of motion capabilities to be retrieved from the database may be queried from the database from the learned range of motion capabilities (e.g., output from the machine learning system). In another example, obtaining the range of motion capabilities includes obtaining the range of motion capabilities from the camera by capturing a reference body part of the subject that mirrors the target body part, the reference body part performing the three-dimensional movement.” [0075]
Regarding claim 19
The method of claim 17, the method further comprising: providing the feedback to a third party related to the user,
wherein the third party is a therapist involved in generating treatment plans;
Van Andel teaches:
Instruct and guide (treatment plans) and therapist…
“Systems and methods for using augmented reality (AR) to remotely instruct and guide a patient through movements of a target body part, as well as guide the patient with therapy routines, are described herein. The systems and methods herein describe using AR to display a virtual representation of a patient's limb to illustrate to the patient how to complete a clinical movement. In an example, the AR may be used to display directions for a patient to perform a clinical movement. The real therapist may, for example, be a live therapist appearing with a patient, or may be a video recording of a therapist. The real therapist may be defined as any therapist that is not virtual.” [0028]
in response to providing the feedback, receiving a modification input from the therapist; and
Therapist determine exercises (receiving input), where determine exercise based on progress, therefore, changes (modified) based on changing progress…
“The view 300 may allow the user 305 or a remote user, such as a clinician, to select a joint or location on an AR avatar. By selecting a joint or location on the AR avatar, the user 305 or remote user may indicate what hurts or needs attention on the user 305, select a side of the body, be prompted for a pain level, or the like. In another example, the user 305 may select options from a user interface within the view 300, such as a scale bar to select pain. The view 300 may include a questionnaire for the patient to answer so that a therapist may evaluate progress or determine exercises for the patient.” [0041]
adapting at least one of the activities and/or the experience in accordance with the modification input.
Changes (modified) based on changing progress…
“The view 300 may allow the user 305 or a remote user, such as a clinician, to select a joint or location on an AR avatar. By selecting a joint or location on the AR avatar, the user 305 or remote user may indicate what hurts or needs attention on the user 305, select a side of the body, be prompted for a pain level, or the like. In another example, the user 305 may select options from a user interface within the view 300, such as a scale bar to select pain. The view 300 may include a questionnaire for the patient to answer so that a therapist may evaluate progress or determine exercises for the patient.” [0041]
Regarding claim 20
The method of claim 19, wherein the modification input includes data pertaining to modification in the personalized treatment plan, and prior to the modifying, the method further comprising:
based on the modification input, updating the treatment plan; and
Van Andel teaches:
Adjust (updating) the prescribed therapy (treatment plan)…
“The systems and techniques described herein may determine an assessment for a target body part. The assessment may include a physical therapy program to help strengthen the target body part. The worn apparatus sensor 445 may produce data (e.g., whether the user is successfully performing movements, a percentage of successful performance, metrics related to number of repetitions, weight used, etc., or the like) to be used, for example, by a clinician or other caregiver (e.g., a surgeon). The data may be produced after the user performs a movement, completes a scheduled session, or performs at least part of scheduled therapy. The data may be used to adjust prescribed therapy, movement, exercises, medication, surgery timetable, or the like. The user and the clinician may use an AR device or multiple AR devices to share an augmented experience. For example, the clinician may have an AR device, and the AR device of the clinician may display augmented and virtual aspects in a shared environment with an AR device used by the user, overlaid on the same real environment. The clinician may manipulate virtual or real aspects of the shared environment such that the user may see the manipulation.” [0047]
adapting at least one of the activities and/or the experience, based on the updated treatment plan.
Prescribed movement…
“The display 415 may be used to animate the virtual representation 430 of the user 405 to perform a movement along a path. The path may be used to show a patient how to perform a prescribed movement. The path may be used to determine whether the movement by the user was successful. In an example, the path may be used to determine whether a user has successfully completed an exercise or movement. For example, if the user completes the exercise within the path, the exercise may be determined to be completed successfully. If the user moves outside the path while attempting to complete the exercise, the exercise may be determined to not be completed successfully.” [0048]
Regarding claim 21
The method of claim 20, wherein adapting at least one of the activities and/or the experience comprises performing at least one of the following:
changing a level of difficulty of at least one of the activities,
replacing the virtual activities,
activating and/or deactivating specific activities,
adjusting at least one of the activities to assist the user in participating the experience,
providing guidance on the experience on the activity,
Van Andel teaches:
Guidance…
“The assessment provided to the patient may include a physical therapy program including various movements or stretches the patient is to perform with the target body part. The system used to capture the movements of the target body part for diagnosis may be used to assist in guiding the patient through the therapy routines.” [0020]
adjusting or triggering audio output, or
Example of audible feedback (output)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
changing visual appearance of elements included in the experience.
Example of arrow pointing, etc. (changing visual appearance)…
“As the user 605 turns and moves, the type of assistance provided through an AR device may change. In an example, the user 605 may view the avatar 615 when looking in a first direction 610 or at a first depth, and when the user 605 is looking in a second direction 620 or at a second depth and views an arm or a leg of the user, other virtual guidance information may be displayed. The other virtual guidance information may include an arrow pointing to a correct positioning, an overlay of a virtual arm or a virtual leg, a virtual movement indicator (e.g., a symbol indicating rotation to be applied), an alert, audible feedback, haptic feedback, or the like…” [0054]
Regarding claim 22
The method of claim 1, wherein the user device is a mobile user device comprising a camera.
Van Andel teaches:
Smartphone with camera…
“…The patient may be presented with an option of performing the therapy program at home, for example by utilizing a personal AR device, a smartphone with camera, a video capture device such as a Kinect from Microsoft of Redmond, Wash., or the like.” [0029]
Regarding claim 23
The method of claim 1, wherein determining the one or more combined measurements comprises processing the analyzed user measurements and the activity measurements in order to assess one or more of the following aspects pertaining to human skills:
motion patterns,
Van Andel teaches:
Golf swing (motion patterns)…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
reaction, performance related to abilities,
Example of assessment (analyzing) the data (user measurement) and reference data set (activity measurements)…
“… A machine learning system may generate ideal or optimized data for the reference set based on patient specific parameters, such as hip to knee measurement, knee to ankle measurement, etc. The comparison between the patient's data and the reference set may determine a difference set. An assessment for the patient may then be determined based on the difference set. The assessment may be determined by an analysis performed by a clinician or a process performed by an artificial intelligence system. The assessment is then output to the patient. The output may include a physical therapy program for the patient to perform immediately or over a prescribed period of time. The output may include a recommendation for the patient to visit a doctor as the target body part may require surgery or other medical procedure.” [0019]
aggregated measurements over a plurality of experiences,
comparison with goals,
Goal to reach to (comparison)….
“In system 100, the user 115 may wear the AR glasses 110. In an example, view 150 may represent a view the user 115 sees while wearing the AR glasses 110. The view 150 may include virtual and real aspects in an environment, such as the user's leg (real) augmented by a guide (virtual). In an example, the user 115 may look downward to observe the view 150 and the user may see the raised leg 135 with bent knee. The view 150 using the AR glasses 110 may include a virtual object or goal for the target body part to reach or extend to, such as a virtual line 140. The virtual line 140 may represent an extension distance for the target body part to achieve during a movement. The AR glasses 110 may display virtual aspects in a real environment, such as in view 150.” [0034]
cognitive and physical abilities,
social skills, and
behavioral skills.
Regarding claim 24
The method of claim 1, wherein the analyzing comprises analyzing one or more user measurements using at least one of image analysis and analysis of extracted human body.
Van Andel teaches:
Example of image and processor determine (analyze) range of motion (measurements) of a subject (extracted from a human body)…
In an example, the camera 806 or other motion capture device such as an infrared capture device, may capture a series of images of three-dimensional movement of a target body part of a subject, the target body part moving relative to a reference point of the subject. The processor 802 may determine range of motion information for the target body part based on the captured series of images. In an example, the processor 802 may obtain a reference range of motion capabilities for the target body part. The processor 802 may determine that the reference range of motion capabilities exceed the range of motion for the target body part. In an example, the display device 812 or the AR display 810 may display, in response to the determination, a virtual animation of the target body part, the virtual animation created using the reference range of motion capabilities and the range of motion information for the target body part. The virtual animation may be displayed using the AR display 810 as a virtual aspect represented in a real environment, such as in a room or augmented over the target body part of the subject
Regarding claim 25
The method of claim 1, wherein determining the skill progression comprises mapping the combined measurements to the one or more human skills.
Van Andel teaches:
Example of range of motion (progression) and sports movement (skill progression) and reference set corresponding to (mapping) …
“The movements of the user 605 may be captured through a camera or sensor attached to the user 605 or to a sports instrument, such as a golf club or baseball bat. The user 605 may be instructed, such as using an avatar 615 on a display or within an AR field of view in the first direction 610, to perform a movement related to the predetermined sports movement. In an example, three-dimensional data points of the user's movements are captured as a measurement set. The three-dimensional data points of the measurement set may be classified to produce a movement set which includes the range of motion for the target body parts related to the sports movement. A reference set corresponding to the sports movement may be obtained. In an example, the reference set may include three-dimensional data points captured from multiple professional athletes of the corresponding sport. A different set may be computed wherein a member of the difference set defines a three-dimensional relationship between a member of the movement set and a corresponding member of the reference set. An assessment for how the user 605 may improve performance may be determined based on the difference set. The assessment may include a physical therapy program, a stretch the user performs before the sports activity, or a movement or strengthening technique the user 605 is guided through by the avatar 615.” [0053]
“In an example, the camera 806 or other motion capture device such as an infrared capture device, may capture a series of images of three-dimensional movement of a target body part of a subject, the target body part moving relative to a reference point of the subject. The processor 802 may determine range of motion information for the target body part based on the captured series of images. In an example, the processor 802 may obtain a reference range of motion capabilities for the target body part. The processor 802 may determine that the reference range of motion capabilities exceed the range of motion for the target body part. In an example, the display device 812 or the AR display 810 may display, in response to the determination, a virtual animation of the target body part, the virtual animation created using the reference range of motion capabilities and the range of motion information for the target body part. The virtual animation may be displayed using the AR display 810 as a virtual aspect represented in a real environment, such as in a room or augmented over the target body part of the subject.”[0061]
Mapping reference set (performance) to movement (human skills)…
“In Example 55, the subject matter of any one or more of Examples 1-54 optionally include wherein to member of the difference set is a mapping from the movement set to the reference set.” [0131]
Regarding claim 26
The method of claim 1, further comprising: determining level of performance of one or more of the human skills.
{
From Applicant’s specification on level of performance…
“In some cases, after the execution of a session of experience is completed, combined measurements can be determined. The combination measurements can be based on the user activity in the game, and the experience itself. For example, the user and the experience measurements that were extracted and analyzed. A level of performance of human skills can then be determined based on the analyzed data. The level of performance of human skills can include monitoring and analyzing the skill progression. Alternatively, the skill progression can be determined independently of the level of performance. The level of performance of the user of some human skills and the skill progression can be used to indicate if the user is struggling with certain skills, or has made progress along the course of treatment, e.g., compared to previous performance of the user, and other insights that can be useful in order to assist the user to improve performance of these skills.” (pg. 4, lines 15-26)
The level of performance can be used to indicate progress with skills. Therefore, determining level of performance can be determining progress of skills.
}
Van Andel teaches:
Track (determining) patient progress of duration or number or repetitions completed (skill progression and level of performance)…
“The physical object 340 may include a sensor 355. The sensor 355 may be used to track patient progress, such as a duration or number of repetitions completed by the user 305. In an example, the sensor 355 may be used to identify the physical object 340 to the AR device 310. Once identified, the AR device 310 may select the virtual object 345 that corresponds to the physical object 340. For example, if the user 305 has a ten pound weight and a five pound weight available, these may be identified via sensors in the weights, and the AR device 310 may determine the five pound weight is to be used in an exercise, and the virtual object 345 may resemble the five pound weight. The virtual object 345 may change to resemble the ten pound weight in another exercise or when increasing weight to increase difficulty for the exercise. In an example, the AR device 310 may display a virtual movement of a limb.”[0040]
Example of improving physical movement…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
Range of motion (skill and performance) capability (progression)…
“In Example 60, the subject matter of Example 59 optionally includes training range of motion capabilities data using a machine-learning technique, and wherein obtaining from the database includes querying the database for the range of motion capability from learned range of motion capabilities.” [0136]
Regarding claim 28
The method of claim 27, wherein the progress levels are selected from a group comprising: completion rate and level success, skill level, effort, counts, metrics or a combination thereof.
Van Andel teaches:
Number of repetitions (counts) or weights (metric)…
“The physical object 340 may include a sensor 355. The sensor 355 may be used to track patient progress, such as a duration or number of repetitions completed by the user 305. In an example, the sensor 355 may be used to identify the physical object 340 to the AR device 310. Once identified, the AR device 310 may select the virtual object 345 that corresponds to the physical object 340. For example, if the user 305 has a ten pound weight and a five pound weight available, these may be identified via sensors in the weights, and the AR device 310 may determine the five pound weight is to be used in an exercise, and the virtual object 345 may resemble the five pound weight. The virtual object 345 may change to resemble the ten pound weight in another exercise or when increasing weight to increase difficulty for the exercise. In an example, the AR device 310 may display a virtual movement of a limb.”[0040]
Regarding claim 33
The method of claim 32, wherein providing data comprises: based on one or more parameters relating to human skills, generating feedback; and providing the feedback; whereby the provided feedback facilitates the user to improve the level of performance of the human skills by improving performance of parameters indicative of human skills.
Van Andel teaches:
Example of challenges (improving performance) to provoke thresholds (feedback of measured mobility and stability (measurements)…
“In general, this disclosure describes techniques that may allow a computer system equipped with cameras and sensors to assess a patient's mobility and stability three-dimensionally. The system may capture three-dimensional data about the of a target body part on a person, such as an arm or leg and including the movement of joints such as the elbow or knee. The gathered data may then be analyzed by a remote clinician or an artificial intelligence system to determine the patient's range of motion and motor control. The three-dimensional data may then be analyzed by a clinician or artificial intelligence system against norms to determine a patient assessment. The assessment may then be a program to immerse the patient in three-dimensional tasks and challenges aimed at provoking the thresholds of the patient's measured mobility and stability.” [0017]
Regarding claim 34
The method of claim 1, wherein the one or more human skills comprises at least one skill selected from physical skills, cognitive skills, behavioral skills, social skills, and mental abilities.
Van Andel teaches:
Increase extension of a golf swing (improve golf swing skill) or greater range when throwing a baseball (improve baseball throwing skill), therefore physical skills…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
Regarding claim 35
The method of claim 34, wherein a level of performance is determined for at least two different human skills.
Van Andel teaches:
Increase extension of a golf swing (improve golf swing skill) or greater range when throwing a baseball (improve baseball throwing skill), therefore physical skills…
“FIG. 6. illustrates real and virtual aspects of a sports training AR system 600 in accordance with some embodiments. The sports training AR system 600 may be used for strengthening or improving physical movement, such as for a sporting performance. For example, a user may wish to increase extension of a golf swing or gain a greater range of motion when throwing a baseball or football. The sports training AR system 600 may capture the movements of a user 605 with a camera or any of the other previously described devices such as an AR device or motion sensitive sensor. In an example, the user may wear an AR device, such as AR glasses, to augment what is viewed by the user 605 or a virtual reality (VR) device, such as VR glasses, to view virtual reality aspects. The user may see, by way of the AR glasses, an avatar 615 within an AR or VR field of view in a first direction 610. The avatar 615 may be used to demonstrate the proper motions for a user 605. In an example, when the user 605 wishes to improve a golf swing, the avatar 615 may perform the swing such the user 605 may watch and mimic the performance by the avatar 615.” [0051]
The combined references teach human physical skill. They do not teach two different human skills.
Mettler May also in the business of skills teaches:
Example of skill that includes game tactics (mental abilities)…
“The range of motion sensors, available either as embedded or deployed in the environment, provides measurements of broad aspects of the movement dynamics of users, actors and their equipment. Given the depth of hierarchical levels of the movement system, the scope of motion analysis can be conducted at multiple levels. For example, it could focus on neuro-motor aspects, movement technique and structure, outcomes, all the way up to tactical and strategic levels.” [0147]
“Detailed analysis of movement skill, in particular for open motor skills, quickly become complex. In tennis, for example, the stroke motion is part of a much larger system of coordination and interactions that include the ball trajectory, the footwork, going all the way to court motion, the game tactics, etc. Analysis of the stroke motion usually encompasses the racket trajectory (i.e. end effector or equipment), even though that trajectory is the result of a kinematic chain which involves the upper body and the driving motion that starts from the feet, legs, and hips. Therefore, many elements and body segments should be tracked to provide a complete description of movement performance.” [0148]
Cognitive capabilities and skill level…
“The repertoire represents a discretization of the behavioral space at the task performance level. It reflects how the entire range of behavior is broken up to achieve outcomes under different task conditions while best exploiting the biomechanical, neuro-motor, perceptual and cognitive capabilities. An individual's capacity to exploit the range of conditions, and in turn create distinct categories, is also a reflection of the skill level. The fact that the development of a repertoire requires making categories, which involves recognizing environmental and task conditions, etc. is also a result of the higher level cognitive skills. The different movement patterns that compose the repertoire represent the basic unit of organization associated with the concept of motor equivalences. The extracted classes measure the breadth of the motor repertoire associated with a particular activity as well as the quality of the synergies or movement classes. Given an activity, the goal is to span a large enough repertoire to support versatile performance, i.e., deal with different environmental and task constraints. The patterns that span the repertoire should provide adaptation to uncertainties and disturbances. It is expected that advanced players' repertoires are structured and diversified with a broad range of stroke classes that have distinct characteristics to achieve a range of outcomes. On the other hand, novice players' repertoires are expected to be much less structured with fewer patterns that are themselves less organized. Therefore, the amount of structure and the breadth of patterns and how well they achieve their outcome provide basic elements of systematic skill evaluation.” [0197]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to have two human skills as taught by Mettler May since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Mettler May who teaches movement skill is complex and represents both physical and cognitive abilities.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over the combined references in section (7) above in further view of Pub. No. US 2023/0021339 to Bosnak et al.
Regarding claim 8
The method of claim 1, wherein the one or more user measurements comprise facial muscles and expressions.
Van Andel teaches:
Evaluate head of the user…
“In an example, the system 100 may be calibrated using the video capture device 105. The video capture device 105 may use infrared light to detect the user 115 in a field of view. The system 100 may evaluate the detection (e.g., captured images) of the user 115 to identify joints, limbs, appendages, a head, etc., of the user 115. These identified body parts may be used with later captured video of an exercise to label specific body parts.” [0031]
The combined references teach evaluate head. They do not teach facial muscles or expressions.
Bosnak et al. also in the business of evaluating head teaches:
“The present disclosure generally relates to computer-based platforms and systems configured for automated real-time generation of an interactive attuned discrete avatar, including responsive artificially generated speech and facial features in response to user interaction.” [0002]
Measure of non-speech expression of face with muscle activity…
“Herein, the term valence may refer to a measure of the pleasantness and/or unpleasantness indicated by non-speech expression, e.g., as measured using the Facial Action Coding System and micro-expressions, muscle activity detected through facial electromyography, functional brain imaging, or by any other suitable valence measurement technique or any combination thereof.” [0052]
Face with musculature…
“In some embodiments, the system may utilize the above principles to create interactive attuned discrete avatars maybe photorealistic human replicas, e.g., seen from mid-torso to the top of their head, and capable of torso, face and head movements. The face maybe constructed on a platform similar to human anatomy with a musculature on top of a fixed frame, under a cutaneous and subcutaneous layer allowing for the display of both “normal” and more specifically, micro expressions which the user may sense and observe on a subliminal level. A shadow image of the user may be silhouetted in the interactive attuned discrete avatar pupil creating a sensation of intimate presence.” [0057]
“In some embodiments, an exercise called Repetition may be incorporated into the interactive attuned discrete avatar training. Through this activity the interactive attuned discrete avatar may receive continuous real-time feedback on its attunement. In some embodiments, Repetition may include a back-and-forth between the avatar and the user such that the avatar may be programming to ask and/or assert emotional characteristics of the user based on imagery and audio captured by input devices. The user may provide feedback to the emotional characteristics stated by the avatar to provide training data to update the models of the avatar and improve recognition of emotions and attunement to the emotions. In some embodiments, the user may also provide feedback to the rendered expressions of the avatar as a technique for training the models of the avatar for improved attunement to the user.” [0061]
It would have been obvious to one of ordinary skill in the art before the effective filing date to include in the method and system of the combined references the ability to measure facial muscles as taught by Bosnak et al. since the claimed invention is merely a combination of old elements and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. Further motivation is provided by Bosnak et al. who teaches the benefits of monitoring and providing feedback to users based on facial movements.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
The following prior art teaches dynamic tracking activity of users:
US-20230148135-A1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENNETH BARTLEY whose telephone number is (571)272-5230. The examiner can normally be reached Mon-Fri: 7:30 - 4:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SHAHID MERCHANT can be reached at (571) 270-1360. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KENNETH BARTLEY/Primary Examiner, Art Unit 3684
Prosecution Insights