DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office action is in response to amendment filed 9/17/2025. Accordingly, claims 1, 4-7, 13-15 and 17 were amended, claim 19 was cancelled; and claims 1-18 are pending for examination.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 6-12 and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Browy (US 11,221,814 B2) in view of Bridgeman et al. (US 2019/0304188 A1, hereinafter Bridgeman) and KR 20150038498 A, hereinafter 498).
Regarding claim 1, Browy discloses a system for training session replay as shown in figure 1), comprising: a plurality of wearable devices (figure 2, 110) for wearing by a trainee for measuring biometric data of the trainee and capturing a first-person perspective of a training session in the training environment (col. 9 lines 21-31, a head mounted component of a user may be configured to monitor eye-based and facial indicators of stress, as well as heart rate, heart rate variability, etc., and to report out this information to others; further, a head mounted component of a user may be configured to utilize computer vision resources, such as camera and local or remote/connected computing resources, to not only scan rooms around them for mapping and machine learning functionalities, but also to conduct facial recognition of personnel), the wearable devices including at least eye track glasses, a camera and a microphone (figure 103 and col. 43 line 49 through col. 44 line 2, the wearable component can include light field camera 2, inertial measurement unit 10, infrared eye tracking cameras 15, microphone array 38); a local device (figure 7, 6) configured to receive, store and relay the biometric data (col. 7 line 16-25 and col. 9 line 59 through col. 60 line 3, belt pack component (6) may have antenna and memory, wherein overall system may provide data not only pertinent to that operators biometrics, but also information pertaining to the environment around such that the belt pack component obviously can receive, store and relay the biometric data) and state estimation sensors for attachment to dynamic objects in the training environment to record movement of the dynamic objects during the training session (col. 28 lines 41-67, an inertial measurement unit mounted to a temple arm of the head device to output a set of coordinates of the wearable head device relative to the user's environment). Browy differs from the claimed invention in not specifically teaching a local device configured to relay the first-person perspective, and a replay server to receive the biometric data and the first-person perspective from the local device; automatically analyze the biometric data to extract psychophysiometric insights and indicators of the trainee's behavioral response to areas of interest or stimuli corresponding to one or more of the dynamic objects in the training environment; and generate a post-session replay of the training session showing the first-person perspective. However, Bridgeman teaches methods for multi-user virtual reality remote training having VR headset 110 fully immersed inside a VR environment which may replicate the AR environment as seen by the trainee ([0054]) to relay the first-person perspective, and a replay server (figure 1, 150) to receive the biometric data and the first-person perspective from the local device ([0054], trainer user may connect to the trainee's session via the backend server and may see a digital replica of the same objects that the trainee user sees in the physical environment); automatically analyze the biometric data to extract psychophysiometric insights and indicators of the trainee's behavioral response to areas of interest or stimuli corresponding to one or more of the dynamic objects in the training environment ([0030], the server 150 may register trainee user “U” actions and may generate responses within the simulated environment based on the trainee's actions, determine how parts of the simulated environment may change as a training procedure is carried out such that the replay server to receive the movement of the dynamic objects from the state estimation sensors and automatically analyze the biometric data response to areas of interest or stimuli corresponding to one or more of the dynamic objects in the training environment); and generate a post-session replay of the training session showing the first-person perspective ([0005] and [0030], records all actions for a session which may be archived for assessment and review, and a second user is enabled to switch between a third person perspective view of the VR simulated environment watching the first user to a first-person perspective view of the VR simulated environment, as viewable by the first user). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Browy in having a local device configured to relay the first-person perspective, and a replay server to receive the biometric data and the first-person perspective from the local device; automatically analyze the biometric data to extract psychophysiometric insights and indicators of the trainee's behavioral response to areas of interest or stimuli corresponding to one or more of the dynamic objects in the training environment; and generate a post-session replay of the training session showing the first-person perspective, as per teaching of Bridgeman, in order to provide more user flexibility and training aids. The combination of Browy and Bridgeman differs from the claimed invention in not specifically the replay server to generate a post-session replay of the training session showing the first-person perspective temporally synchronized with the psychophysiometric insights and indicators. However, 498 teaches a method to detect, store and/or transmit physical performance information to a processing system for overlaying an indication of the at least one physical activity measurement on the video replay (figure 15, abstract and claim 7) in order to make user friendly by allowing a user to select or deselect measurements that will be combined and stored in files corresponding to. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Browy and Bridgeman in having the replay server to generate a post-session replay of the training session showing the first-person perspective temporally synchronized with the psychophysiometric insights and indicators, as per teaching of 498, in order to make user friendly
Regarding claim 2, Browy teaches to host a session replay console application for presenting the post session replay to a user (col. 14 lines 31-46, training system services may be remotely hosted resources, and may include, for example: a relatively comprehensive database, which may be referred to as a “data lake”, for the storage of user account and training performance data; a file store for collecting and sharing training scenarios; available server resources earmarked for cloud hosting of TSS/S training servers as needed), as well as Bridgeman ([0036], the trainee user “U”'s progress may be automatically saved on the backend servers 150, and the training session can be resumed at any time on any supported devices that are connected to the backend servers).
Regarding claim 3, Browy discloses that the wearable devices as shown in figure 6 further include one or more of: a global positioning system tracker, a heart rate monitor, a galvanic skin response monitor, a transducer (respiration) belt, a body temperature sensor, an electromyograph electrode, an electroencephalograph electrode and a full-body motion capture system (col. 7 lines 3-56, a device may be configured to sense the temperature, blood pressure, location (such as by GPS and/or mobile connectivity triangulation and/or IP address mapping), partial pressure of blood oxygen, and other variables related to such operator and his position and condition).
Regarding claim 4, Browy discloses a scanner device for capturing a three-dimensional view of a training environment to generate a virtual mesh of the environment, wherein the virtual mesh comprises points in space, polygons or vectors representing objects and boundaries in the training environment (col. 6 lines 52-57, a connected collective may be utilized as a 3-D sensor of sorts to reconstruct a mesh or model of one or more objects which may be positioned in between or surrounded by one of more transmitting/receiving devices).
Regarding claim 6, Bridgeman discloses that the replay server is further configured to: receive the virtual mesh from the scanner device; and generate a virtual twin of the training environment from the virtual mesh, the virtual twin including static objects and the dynamic objects in the training environment; and receive user input to label one or more of the static objects and/or the dynamic objects as an area of interest (AOI) ([0054]-[0057], The trainee user may see digital notes coupled to and/or superimposed on top of the physical counterpart and may directly interact with the physical counterpart following the digital note provided by the trainer user as guidance for the trainee such that trainer user may see exactly what the trainee user sees through the AR device, a recomposed image with digital notes superimposed on top of the real-world environment, wherein the AR device may further include a tracking module, a user input module, and an overlay module configured to coordinate the overlay of virtual objects over physical or other virtual objects) to provide more user flexibility and training aids.
Regarding claim 7, Browy teaches to extract two-dimensional gaze tracking videos, gaze vector data, and visual behavior data from the biometric data measured by the eye track glasses (col. 16 lines 45-51, the headset may be configured to capture biometric data including heart rate, eye gaze, and perceived pupil dilation); estimate a head pose and a head position of the trainee from the two-dimensional gaze tracking videos, the gaze vector data, and the visual behavior data (col. 18 line 54 through col. 19 line 45, utilize the wearable system (2, 6) and localize his position in the driver's seat based on pattern recognition of the control gage, i.e., the device may be configured to track this based on the localized position and subsequent head pose of the device); estimate a body position of the trainee from the biometric data measured by the full-body motion capture system worn by the trainee (col. 28 lines 41-67, an inertial measurement unit mounted to a temple arm of the head device to output a set of coordinates of the wearable head device relative to the user's environment); and generate a bio twin of the trainee based on the head pose, the head position and the body position of the trainee (col. 19 lines 25-45, taking sensor data from the wearable computing system (2, 6) on the user and creating a dense mesh model of the environment, and also tracking the position/orientation of the user's head within this environment).
Regarding claim 8, Bridgman teaches to: generate a virtual scene of the bio twin of the trainee within the virtual twin of the training environment for each video frame captured by the camera during the training session ([0028], to generate a VR training session for a trainee user “U”, wherein the training session may simulate a real-world environment in which the trainee user “U” expects to perform a work-related task or series of tasks associated with a procedure) in order to provide more user flexibility and training aids.
Regarding claim 9, Bridgeman discloses to generate a post-session replay of the training session from a plurality of virtual scenes showing the first-person perspective of the training session ([0050], trainee module may include features which allow the trainee to request the trainer's help, switch to the trainer's first-person view) in order to provide more user flexibility and training aids.
Regarding claim 10, Browy teaches that the pose and the position of the static objects and the dynamic objects in the post-session replay are temporally synchronized with the head pose and the head position of the trainee (col. 19 lines 2-45, device may be configured to render the correct stimulus information at the correct depth so that the gunner can practice target acquisition and engagement while interacting with the rest of the crew, and device may be configured to track this based on the localized position and subsequent head pose of the device such that data may be time stamped and geo-tagged so that the transforms of where the spatial data resides to the user's eye can be constrained by the location and gaze of the user of the wearable computing system).
Regarding claim 11, Bridgeman teaches that the replay server is further configured to: receive user input to position a virtual camera within the virtual scene; and generate the post-session replay of the training session from a perspective of the virtual camera ([0027]-[0028], a “user interface” may display for example, different menus, controllable components, and virtual scenes based on user triggered selections and simulated environment may generate objects to be worked on or manipulated by the trainee user “U”) in order to provide more user flexibility and training aids.
Regarding claim 12, Bridgeman teaches that the body position and movements of the bio twin of the trainee in the post-session replay are temporally synchronized to movements made by the trainee during the training session ([0030], the server may register trainee user “U” actions and may generate responses within the simulated environment based on the trainee's actions, and generally records all actions for a session which may be archived for assessment and review) in order to provide more user flexibility and training aids.
Regarding claim 15, Browy teaches to generate a behavior report including the biometric and psychophysiometric insights and indicators (col. 9 lines 21-31, to monitor eye-based and facial indicators of stress, as well as heart rate, heart rate variability, etc., and to report out this information to others).
Regarding claim 16, Browy teaches to implement one or more algorithms for processing the biometric data to generate the psychophysiometric insights and indicators of the trainee's behavioral response as a benchmarked metric or score (col. 19 lines 21-30 and col. 20 lines 15-35, use of the machine learning model to dynamically change the data received and displayed in stressful situations, reducing the cognitive load on the user, and data may be collected and sent to the TMT where the crew and users performance is aggregated with previous training, combat, and orthogonal data to update their overall lethality score and training roadmap for future training scenarios).
Regarding claim 17, Browy teaches to receive the biometric data (col. 23 lines 7-31, the headset may be configured to capture biometric data including heart rate, eye gaze, and perceived pupil dilation). Browy differs from the claimed invention in not specifically teaching the live view server configured to: receive the first-person perspective from the local device over a wireless network. However, Bridgeman teaches a live view server configured to: receive the the first-person perspective from the local device over a wireless network (figure 2 and [0036], trainee user “U” is at one physical location, and a trainer user “T” is at another physical location remote from the trainee user “U” observing and instructing the trainee user “U”) such that the live view server configured to receive the the first-person perspective from the local device and to generate a live replay of the training session showing the first-person perspective of the training session. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Browy in having the live view server configured to: receive the first-person perspective from the local device over a wireless network and to generate a live replay of the training session showing the first-person perspective of the training session, as per teaching of Bridgeman, in order to provide more user flexibility and training aids. The combination of Browy and Bridgeman differs from the claimed invention in not specifically teaching to generate a live replay of the training session showing the first-person perspective of the training session temporally synchronized with the biometric data. However, 498 teaches to overlay, by the processing system, an indication of the at least one physical activity measurement on the video replay (claim 7). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Browy and Bridgeman in generating a live replay of the training session showing the first-person perspective of the training session temporally synchronized with the biometric data, as per teaching of 498, in order to make user friendly.
Regarding claim 18, Bridgeman discloses to host a live action console application for presenting the live replay of the training session in near real-time to a user ([0035], trainee user's view may be captured via a video camera module on the AR glasses and transmitted in real-time to the trainer user) in order to provide more user flexibility and training aids.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Browy (US 11,221,814 B2) in view of Bridgeman et al. (US 2019/0304188 A1, hereinafter Bridgeman) and KR 20150038498 A, hereinafter 498) as applied in claim 4 above, and further in view of Luo et al. (US 2022/0346888 hereinafter Luo).
Regarding claim 5, Browy teaches to receive the virtual mesh from the scanner device (col. 18 lines 34-39, mesh sensor data, in the form of a versioned raw data stream, from a sensor-laden wearable computing headset, may be sent to this service to be pre-processed before sending the LWM). The combination of Browy, Bridgeman and 498 differs from the claimed invention in not specifically teaching that the replay server is further configured to host a mesh editor application configured for editing, smoothing, segmenting and stitching the virtual mesh to eliminate gaps and/or incongruities. However, Luo teaches a volume and image data system to implement a volume editing process which allows the one or more editable volume objects and the one or more surface representations associated therewith to be modified in real-time or separated into the multiple independent segments, wherein the signed distance function (SDF) may include but is not limited to a plane, a geometric primitive which may be a cuboid or a sphere, or a manifold mesh ([0066]-[0067]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Browy, Bridgeman and 498 in having that the replay server is further configured to host a mesh editor application configured for editing, smoothing, segmenting and stitching the virtual mesh to eliminate gaps and/or incongruities, as per teaching of Luo, in order to help in the better visualization of the anatomical structures.
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Browy (US 11,221,814 B2) in view of Bridgeman et al. (US 2019/0304188 A1, hereinafter Bridgeman) and KR 20150038498 A, hereinafter 498) as applied in claim 8 above, and further in view of Verbeke et al. (US 2021/0398562 A1, herein after Verbeke).
Regarding claim 13, Bridgeman teaches the replay server is further configured to: receive user input to label one or more of the static objects and/or the dynamic objects as an area of interest (AOI) ([0043] and [0046], the trainer may trigger a function highlighting an object to direct the trainee's attention to the object, and control trainee's progress, as well as jumping forward/backward on the list of subtasks performed; wherein a panel may list detailed step by step procedures for the trainer to guide the trainee, and the bottom panel 145 may include a list of components referenced in each step for the trainer to highlight, with the ability to direct the trainee's view toward any specific component) and Browy teaches to determine where a gaze vector of the trainee intersects with the AOI (col. 21 lines 3-28, localized dynamic dimming relates to contrast adjustment specifically around a particular portion or portions of presented imagery or content, such as a region at which the operator's eye gaze is directed). The combination of Browy, Bridgeman and 498 differs from the claimed invention in not specifically teaching to record AOI observation data in a log, including an observation reaction time, a total number of fixations on the AOI, number of saccades on the AOI and gaze path(s) of visual attention. However, Verbeke teaches media controller application could feed the image data into a machine learning model that has been trained to classify a cognitive load, emotional load, and/or amount of mind wandering by analyzing eye saccades and fixations, blink rate duration, and/or other parameters such that the amount of mind wandering experienced by a user may be determined by comparing an amount that eye saccades and fixations of the user have changed, relative to a baseline for the user ([0024] and [0026]) in order to compute a cognitive load that a user is experiencing as a mental state metric. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Browy, Bridgeman and 498 to record AOI observation data in a log, including an observation reaction time, a total number of fixations on the AOI, number of saccades on the AOI and gaze path(s) of visual attention, as pe teaching of Verbeke, in order to compute a cognitive load.
Regarding claim 14, Bridgeman discloses that the replay server is further configured to: generate the post-session replay of the training session from a plurality of virtual scenes showing the first-person perspective of the training session ([0070], the trainer may switch to a first-person view which directly shows what the trainee sees from his/her perspective, as if seeing through trainee's eyes); and superimpose the AOI observation data superimposed on the first-person perspective ([0046], the bottom panel 145 may include a list of components referenced in each step for the trainer to highlight, with the ability to direct the trainee's view toward any specific component).
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments filed 9/17/2025 have been fully considered but they are not persuasive.
In response to applicant's argument that “none of the cited art discloses or suggests attaching state estimation sensors (e.g., inertial measurement units) to objects in a training environment to track movement of the objects during a training session”, examiner respectfully disagreed because Browy clearly discloses that inertial data may be captured via an inertial measurement unit of the head-wearable device and a position of the head-wearable device can be estimated based on the image and the inertial data via one or more processors of the head-wearable device such that an Inertial Measurement Unit (IMU) may use to provide improved accuracy and/or more timely information on rapid movements of the handheld controller 400B (abstract and col. 32 lines 26-29). Thus, Browy teaches the broad claimed limitations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GEORGE ENG whose telephone number is (571)272-7495. The examiner can normally be reached Flex M to F, 7 am to 3 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alford Kindred can be reached at 571-272-4037. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GEORGE ENG/Supervisory Patent Examiner, Art Unit 2699
Prosecution Insights