DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Election/Restrictions
2. Claims 2, 5-10, 12-13, 16 and 19-20 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected Species, there being no allowable generic or linking claim. Election was made without traverse of Species II directed to Claims 1, 14-15, 3- 4, 11 and 17-18 in the reply filed on September 04, 2026 is acknowledged.
Claim Rejections - 35 USC § 103
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
4. Claims 1, 3, 11, 14-15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Aimone et al., (“Aimone”) [US-2016/0077547-A1] in view of Beall et al. (“Beall”) [US-10,403,050-B1]
Regarding claim 1, Aimone discloses a human-machine interaction collection method for a wearable extended reality device (Aimone- Fig. 1 and ¶0053, at least disclose “a training system 100 with a wearable device 105 and other computing device 160”; ¶0057, at least discloses “the wearable device 105 is, for example, a wearable headset worn on a user's head. The computing device 150 of the wearable device 105 is configured to create a VR environment on the stereoscopic display 110 and sound generator 140 for presentation to a user”; ¶0028, at least discloses “A VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes.”), the method comprising:
obtaining data synchronously collected, the data comprising a virtual reality base signal collected by the extended reality device (Aimone- ¶0039, at least discloses “the processor is configured to identify a portion of the bio-signal data based on the event time data and process the portion of the bio-signal data to determine the user states during the VR event, the bio-signal time data synchronized to the event time data using a common timeline or clock or a synchronization or context operation”; ¶0057, at least discloses “the wearable device 105 is, for example, a wearable headset worn on a user's head. The computing device 150 of the wearable device 105 is configured to create a VR environment on the stereoscopic display 110 and sound generator 140 for presentation to a user”; ¶0066, at least discloses “integration of data from other devices and synchronization of events with brainwave data aid in context aware analysis (during the VR event to provide additional context, for example) as well as storage and future analysis”; ¶0028, at least discloses “A VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes.”), human body perception data collected by a human body parameter perception device (Aimone- ¶0057, at least discloses “receive bio-signal data of the user from the bio-signal sensors 120, at least one of the bio-signal sensors 120 comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the user”), and environmental perception data collected by an environmental parameter perception device (Aimone- ¶0028, at least discloses “A VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes”; ¶0055, at least discloses “The tracker 145 is an inertial sensor for measuring movement of the wearable device 105. It detects the 3-dimensional coordinates of the wearable device 105 and accordingly its user's location, orientation or movement in the VR environment including the user's gaze direction”), and the human body perception data comprising at least one of eye movement data, a physiological state parameter, and electroencephalographic data (Aimone- ¶0057, at least discloses “receive bio-signal data of the user from the bio-signal sensors 120, at least one of the bio-signal sensors 120 comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the user”);
providing an extended reality scene based on the virtual reality base signal collected by the extended reality device, obtaining human body state information of an individual based on the human body perception data collected by the human body parameter perception device, performing three-dimensional modeling based on the environmental perception data collected by the environmental parameter perception device, and obtaining a three-dimensional environment scene of a position of the individual (Aimone- ¶0057, at least discloses “The computing device 150 of the wearable device 105 is configured to create a VR environment on the stereoscopic display 110 and sound generator 140 for presentation to a user; receive bio-signal data of the user from the bio-signal sensors 120, at least one of the bio-signal sensors 120 comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the user; and determine brain state response elicited by the VR environment at least partly by determining a correspondence between the brainwave data and a predefined bio-signal measurement stored in a user profile, the predefined bio-signal measurement associated with predefined brain state response type. The brain state response may comprise an emotional response type”; wherein ¶0028, at least discloses “a VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes” in combination with ¶0074, at least discloses “The parameters for the prediction models are used by the user state estimator 325 to predict a brain state (e.g. drowsy, or agitated etc.) or other user states such as eye position, high muscle tension etc. during the VR event.” wherein ¶0080, at least discloses “The context logic 310 analyzes and determines what the user is doing using manual input 320 of the user and user state estimator 345 (example direction of eye gaze). Also the context of the VR environment that the user is in is supplied by the VR application 315. The context is used as an additional feature and or label of the biological features, for example a truck suddenly appears, used by the data mining improver 330 to update the prediction model in the user state estimator 325. The direction, location or orientation of the user 305 or other data from tracker 145 or sensors 120, 130 may also provide input to context logic 310 to determine context of the VR environment, such as, for example, the position of the truck relative to the user 305 within the VR environment or the orientation or eye gaze of the user 305 relative to the truck within the VR environment.”, for example, ¶0115, at least discloses “effects in the VR environment, feedback or a combination thereof, including brain state inputs and other inputs in combination, for example: 1) noticing something salient, 2) focus, 3) attention, 4) head orientation, and 5) eye gaze. In this example, these inputs are processed by the VR system, and the output may include 1) an environmental response, 2) the user's position in the simulated 3D space, and 3) the user's velocity. In this example, a participant may have a wearable device 1202 with bio-signal sensors and a display to provide a VR environment with a participant view 1208.”);
displaying the extended reality scene, the human body state information, and the three-dimensional environment scene on a display unit of the extended reality device (Aimone- ¶0043, at least discloses a wearable computing device with a bio-signal sensor, the wearable device in connection with a display to provide an interactive virtual reality (“VR”) environment for a user. The bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor. The computing device having or in communication with a processor configured to as part of the interactive VR environment, present content on the display where the content has a VR event, desired user states, and desired effects; Fig. 1 and ¶0053, at least disclose “The wearable device 105 has a stereoscopic display 110 […] The stereoscopic display 110 is a 3-dimensional (3D) (or dual 2-dimensional (2D) images giving “3D”) display, but alternatively may be a 2-dimensional (2D) display […] The VR environment includes an interactive VR environment where content presented on the display may update or modify in response to input from bio-signal sensors 130, other sensors, user manual inputs, or other inputs”; ¶0063, at least discloses “in response to a determination that the user has achieved a particular brain state, or maintained a particular brain state for a period of time, the wearable device may update the display to provide an indication of the determination (e.g. indicating to the user what brain state has been achieved, and, optionally for how long) and may further display an indication of a particular reward assigned to the user in response to the determination”; Fig. 2 and ¶0071, at least disclose “a perspective view of a training system 200 with a display 225 […] The display 225 having the functionality of the stereoscopic display 110”; ¶0082, at least discloses “The VR application 315 is presenting a stream of content to the user 305 thereby creating the VR environment. In the VR environment, there are VR events 430” in combination with Fig. 3 and ¶0075, at least disclose “the VR Application 315 uses the user's intention and user state, including brain state, to control the content of a multimedia VR environment presented to the user to provide an interactive environment” and with ¶0080, at least discloses “the direction, location or orientation of the user 305 or other data from tracker 145 or sensors 120, 130 may also provide input to context logic 310 to determine context of the VR environment, such as, for example, the position of the truck relative to the user 305 within the VR environment or the orientation or eye gaze of the user 305 relative to the truck within the VR environment”, for example, ¶0082, at least discloses “The trainee or user 305 is, for example, driving a car in a training VR environment. The user 305 enters manual input 410 such as those from steering wheel, gas, and brakes; while the user's 305 biological features 420, such as brain waves […], are sensed by the sensors 120 and 130. The biological features 420 and manual input 410 are used to determine user state 425. The user state 425 and the desired user state 435 optionally may only include brain states and not include the manual input 410. The manual input 410 is further used to determine its effect in VR environment 415. The effect in the VR environment 415 is used by the VR application 315 to change aspects of the VR environment accordingly. The VR environment is interactive as it changes depending on manual input 410 and user state 425” or ¶0100, or Fig. 9 and ¶0111, at least disclose “the device allows the opportunity to create a VR environment where users learn and practise sensitivity by engaging with characters in the VR environment where they wear their “emotions on their sleeve” (visually represented in real time via colour-coding, auras, icons, logos, avatars, etc.) based on determined user or brain states during VR events within the VR environment.”, wherein Fig. 12 and ¶0114, at least disclose "a user 902, 904 may see the same […] content as the other user 902, 904 in the VR environment", or "the navigation within the VR environment may be an example of feedback based on the user state score", with an "example feedback based on his scored brain state"); and
storing the human body state information of the individual and the three-dimensional environment scene to a cloud server (Aimone- ¶0027, at least discloses “The wearable device, client device, or both, may also connect to a cloud server system with a processor”; Fig. 1 and ¶0053-0054, at least disclose “The other computing device 160 and the computing device 150 may be, for example, a computer, embedded computer, server, laptop, tablet, or mobile phone […] The other computing device 160 may also be a server over the Internet or other network”; ¶0079, at least discloses “The data mining improver 330 further may use previous VR sessions to add new labelled data which is used to create a new prediction model or adapt an existing prediction model in the user state estimator 325. The parameters of the new prediction model are stored in the user profile 335 of that user.”; ¶0074, at least discloses “the user profile 335 may optionally store the history of all of the user's sessions, raw biological data, processed data, demographic data, etc.” and ¶0066, at least discloses “this user profile 335 may be stored in the cloud, allowing each user to maintain a single profile across multiple computing devices”).
Aimone does not explicitly disclose uploading the human body state information of the individual and the three-dimensional environment scene to a cloud server.
However, Beall discloses
uploading the human body state information to a cloud server (Beall- col 15, line 61 to col 16, line 6, at least disclose “The description herein may also refer to the terms audience and observers, which may also be used interchangeably, and include individuals which may view a VR presentation from a user's perspective or from a participant's perspective but are not active participants in the VR session, unless the context indicates otherwise”; col 16, lines 23-40, at least disclose “a VR slide presentation session comprises one or more scenes, in which a given scene may include fixed or dynamic objects and/or displayed representations (e.g., avatars) of session participants […] a given scene in a presentation includes certain visual and/or audio animations which may be controlled by a user. Optionally, a scene may be a three dimensional representation of a virtual environment in which a presenter and participant(s) are interacting together, for example, a room, an outdoor plaza, a park, etc. Optionally, a scene completely encompasses a presenter and participant(s), can be viewed by a user and/or participant in any direction (e.g., forward, behind, above, and below), and a perspective of a scene adjusts appropriately by the VR system (e.g. via head tracking technologies) in order to create a life-like user experience.”; col 24, lines 28-38, at least disclose “Once a user is finished (or partially finished) with the user's VR slide presentation, the Designer client/system optionally presents a set of controls (e.g., a file control, an upload control, etc.) enabling a user to upload their finished presentation to a network-based server 200 accessible by a user/presenter and one or more VR participants and/or observers (e.g., collaborators). For example, a File, “Upload Scene” control is provided which may then be selected by a user to upload the new (or edited) VR slide presentation to a network-based server 200.”).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Aimone to incorporate the teachings of Beall, and apply uploading the human body state information to a cloud server into storing the human body state information of the individual and the three-dimensional environment scene to a cloud server, as taught by Aimone’s teachings, for uploading the human body state information of the individual and the three-dimensional environment scene to a cloud server. One of ordinary skill in the art could have substituted one known element for another, and the results of the substitution would have been predictable.
Doing so would provide high fidelity tracking of user and object motion in a virtual and/or augmented reality experience.
Regarding claim 3, Aimone in view of Beall, discloses the method according to claim 1, and discloses the method further comprising:
generating relevant evaluation information based on the human body perception data and the environmental perception data (Aimone- ¶0058, at least discloses “The VR (stereoscopic) display 110 is positioned for viewing through an eye mask 125 wearable by a user. The eye mask 125 comprises an aperture for each eye, and a plurality of facial bio-signal sensors 130 positioned on the mask for contacting the user's face when the wearable device 105 is worn. One or more straps, which may optionally be adjustable, are attached to the eye mask 125 or display portion of the wearable device 105. Optionally, bio-signal sensors may be positioned along one or more of the straps to sense brainwave activity in the user through the user's head. Sensors positioned along the straps may be specifically configured to travel a distance from the strap, past the user's hair, if any, to the user's scalp. Accordingly, any such sensors may include an elongated contact area, which is optionally of a resilient construction. The facial bio-signal sensors 130 measure, include, electrical bio-signals such as EEG, EMG (Electromyograph) and EOG (Electrooculography), as well as FNIRS (functional near infrared spectroscopy)”; ¶0164, at least discloses “When the seminar begins, Jorge is invited to put on a pair of VR-enabled goggles that contain EEG-sensing equipment. The other members of the sensitivity training group are also instructed to do the same. The training program begins: Jorge sees himself—i.e., his avatar in the constructed VR environment—from an external position, so it is like Jorge is witnessing himself and his actions in the third person. As the sensitivity training seminar proceeds, Jorge views himself under changing situations, i.e.: he sees himself as part of a different racial/ethnic group; as part of a group with a different ability, status, etc. The other people in the group observe Jorge's avatar in his new states as they cycle through, as per the instructor's wishes. Both the EEG signatures of the group and of Jorge are recorded during the testing period. Jorge can watch as the EEG patterns of the other people in the therapy group are “played back” during the course of the therapy session. they are played back and interpreted visually much in the same way as a song is played back and visually interpreted as part of a “visualization program” for music tracks. In this way, Joe can see how his transformations from physical state to physical state are “interpreted” by others in his group as actual EEG brainwave data. Joe reports how this experience has changed/benefited him as part of a post-group post-mortem. The instructor can therefore gauge the effectiveness of the class based on how Jorge interprets the EEG patterns of the other participants”), the relevant evaluation information comprising three-dimensional environment scene evaluation information and/or individual behavior decision information (Aimone- ¶0118, at least discloses “a video game may be generated for a head-mounted display. To validate the world, users may be tested using a brain-sensor and HMD in combination. Motor areas, emotional areas, focus, attention, vigilance, drowsiness are all used to evaluate the user's engagement with the virtual world and their interest in the game. In addition other biological sensors such as: heart rate, muscle activity, respiration rate, temperature etc. can be used to determine user's level of arousal, stress and other physiological parameters that are correlated to emotional state […] The system also learns that certain types of motions/controls in their world generate emetic responses and feelings of nausea, or other general feelings of discomfort like neck pain, eyestrain, claustrophobia, headaches, ringing ears, etc.”).
Regarding claim 11, Aimone in view of Beall, discloses the method according to claim 3, and further discloses wherein said generating the relevant evaluation information based on the human body perception data and the environmental perception data (see Claim 3 rejection for detailed analysis) comprises:
generating a training data set based on human body perception data historically collected, the environmental perception data historically collected, and corresponding relevant evaluation information, to train a machine model (Aimone- ¶0074, at least discloses “The user profile 335 contains the signal processing parameters tuned to that user 305 and the parameters for the user's specific prediction models. The signal processing parameters are used by the Biological Signal Processing Pipeline 345 to interpret the user's bio-signal data received during a VR event. The parameters for the prediction models are used by the user state estimator 325 to predict a brain state (e.g. drowsy, or agitated etc.) or other user states such as eye position, high muscle tension etc. during the VR event. The user profile 335 may optionally store the history of all of the user's sessions, raw biological data, processed data, demographic data, etc.”; ¶0078, at least discloses “The data mining improver 330 may also initialize the prediction model for a new user that has no previous history with the system based on aggregate of other users' data.”; ¶0116, at least discloses “Janice wanders around Paris and enjoys photorealistic sights and amazing sound quality. As she gets a little less engaged with the sensory experience over time, information about the things she's seeing is overlaid on the screen (history and physical attributes of buildings, cultural references, etc.)”; ¶0164, at least discloses “When the seminar begins, Jorge is invited to put on a pair of VR-enabled goggles that contain EEG-sensing equipment. The other members of the sensitivity training group are also instructed to do the same. The training program begins: Jorge sees himself—i.e., his avatar in the constructed VR environment—from an external position, so it is like Jorge is witnessing himself and his actions in the third person. As the sensitivity training seminar proceeds, Jorge views himself under changing situations, i.e.: he sees himself as part of a different racial/ethnic group; as part of a group with a different ability, status, etc. The other people in the group observe Jorge's avatar in his new states as they cycle through, as per the instructor's wishes. Both the EEG signatures of the group and of Jorge are recorded during the testing period. Jorge can watch as the EEG patterns of the other people in the therapy group are “played back” during the course of the therapy session. they are played back and interpreted visually much in the same way as a song is played back and visually interpreted as part of a “visualization program” for music tracks. In this way, Joe can see how his transformations from physical state to physical state are “interpreted” by others in his group as actual EEG brainwave data. Joe reports how this experience has changed/benefited him as part of a post-group post-mortem. The instructor can therefore gauge the effectiveness of the class based on how Jorge interprets the EEG patterns of the other participants); and
inputting human body perception data and the environmental perception data that are currently collected into the trained machine model to output, by the trained machine model, relevant evaluation information corresponding to data currently collected (Aimone- ¶0024, at least discloses “A user may interact with the VR environment using various input devices including bio-signal sensors and in response the VR environment may update or modify to provide enhanced training applications”; ¶0028, at least discloses “training apparatus with an input device and a wearable computing device with a bio-signal sensor and a display to provide an interactive VR environment for a user, the bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor […] A user may interact in that world using input data such as gesture data, manual input, sensor data, and so on. The computing device having or in communication with a processor configured to as part of the interactive VR environment, present content on the display where the content has a VR event, desired user states, and desired effects”; ¶0058, at least discloses “The VR (stereoscopic) display 110 is positioned for viewing through an eye mask 125 wearable by a user. The eye mask 125 comprises an aperture for each eye, and a plurality of facial bio-signal sensors 130 positioned on the mask for contacting the user's face when the wearable device 105 is worn. One or more straps, which may optionally be adjustable, are attached to the eye mask 125 or display portion of the wearable device 105. Optionally, bio-signal sensors may be positioned along one or more of the straps to sense brainwave activity in the user through the user's head. Sensors positioned along the straps may be specifically configured to travel a distance from the strap, past the user's hair, if any, to the user's scalp. Accordingly, any such sensors may include an elongated contact area, which is optionally of a resilient construction. The facial bio-signal sensors 130 measure, include, electrical bio-signals such as EEG, EMG (Electromyograph) and EOG (Electrooculography), as well as FNIRS (functional near infrared spectroscopy)”; ¶0129, at least discloses “input data from EEG, heart rate, eye-tracking, and other sensor data and translated it into VR maps with responsive changes to colour, music, opacity, transparency, and other elements that help create a more immersive virtual world”; ¶0164, at least discloses “When the seminar begins, Jorge is invited to put on a pair of VR-enabled goggles that contain EEG-sensing equipment. The other members of the sensitivity training group are also instructed to do the same. The training program begins: Jorge sees himself—i.e., his avatar in the constructed VR environment—from an external position, so it is like Jorge is witnessing himself and his actions in the third person. As the sensitivity training seminar proceeds, Jorge views himself under changing situations, i.e.: he sees himself as part of a different racial/ethnic group; as part of a group with a different ability, status, etc. The other people in the group observe Jorge's avatar in his new states as they cycle through, as per the instructor's wishes. Both the EEG signatures of the group and of Jorge are recorded during the testing period. Jorge can watch as the EEG patterns of the other people in the therapy group are “played back” during the course of the therapy session. they are played back and interpreted visually much in the same way as a song is played back and visually interpreted as part of a “visualization program” for music tracks. In this way, Joe can see how his transformations from physical state to physical state are “interpreted” by others in his group as actual EEG brainwave data. Joe reports how this experience has changed/benefited him as part of a post-group post-mortem. The instructor can therefore gauge the effectiveness of the class based on how Jorge interprets the EEG patterns of the other participants”).
Regarding claim 14, Aimone in view of Beall, discloses a human-machine interaction collection apparatus for a wearable extended reality device, the human-machine interaction collection apparatus (see Claim 1 rejection for detailed analysis) comprising:
an extended reality device configured to collect a virtual reality base signal (Aimone- ¶0057, at least discloses “the wearable device 105 is, for example, a wearable headset worn on a user's head. The computing device 150 of the wearable device 105 is configured to create a VR environment on the stereoscopic display 110 and sound generator 140 for presentation to a user”; ¶0028, at least discloses “A VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes.”);
a human body parameter perception device configured to perceive and collect human body perception data (Aimone- ¶0057, at least discloses “receive bio-signal data of the user from the bio-signal sensors 120, at least one of the bio-signal sensors 120 comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the user”), the human body perception data comprising at least one of eye movement data, a physiological state parameter, and electroencephalographic data (Aimone- ¶0057, at least discloses “receive bio-signal data of the user from the bio-signal sensors 120, at least one of the bio-signal sensors 120 comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the user”);
an environmental parameter perception device configured to collect environmental perception data (Aimone- ¶0028, at least discloses “A VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes”; ¶0055, at least discloses “The tracker 145 is an inertial sensor for measuring movement of the wearable device 105. It detects the 3-dimensional coordinates of the wearable device 105 and accordingly its user's location, orientation or movement in the VR environment including the user's gaze direction”);
a data processing module configured to convert the virtual reality base signal into an extended reality scene corresponding to a real scene, convert the collected human body perception data into human body state information of an individual, and convert the collected environmental perception data into a three-dimensional environment scene of a position of the individual (Aimone- ¶0057, at least discloses “The computing device 150 of the wearable device 105 is configured to create a VR environment on the stereoscopic display 110 and sound generator 140 for presentation to a user; receive bio-signal data of the user from the bio-signal sensors 120, at least one of the bio-signal sensors 120 comprising a brainwave sensor, and the received bio-signal data comprising at least brainwave data of the user; and determine brain state response elicited by the VR environment at least partly by determining a correspondence between the brainwave data and a predefined bio-signal measurement stored in a user profile, the predefined bio-signal measurement associated with predefined brain state response type. The brain state response may comprise an emotional response type”; wherein ¶0028, at least discloses “a VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes” in combination with ¶0074, at least discloses “The parameters for the prediction models are used by the user state estimator 325 to predict a brain state (e.g. drowsy, or agitated etc.) or other user states such as eye position, high muscle tension etc. during the VR event.” wherein ¶0080, at least discloses “The context logic 310 analyzes and determines what the user is doing using manual input 320 of the user and user state estimator 345 (example direction of eye gaze). Also the context of the VR environment that the user is in is supplied by the VR application 315. The context is used as an additional feature and or label of the biological features, for example a truck suddenly appears, used by the data mining improver 330 to update the prediction model in the user state estimator 325. The direction, location or orientation of the user 305 or other data from tracker 145 or sensors 120, 130 may also provide input to context logic 310 to determine context of the VR environment, such as, for example, the position of the truck relative to the user 305 within the VR environment or the orientation or eye gaze of the user 305 relative to the truck within the VR environment.”, for example, ¶0115, at least discloses “effects in the VR environment, feedback or a combination thereof, including brain state inputs and other inputs in combination, for example: 1) noticing something salient, 2) focus, 3) attention, 4) head orientation, and 5) eye gaze. In this example, these inputs are processed by the VR system, and the output may include 1) an environmental response, 2) the user's position in the simulated 3D space, and 3) the user's velocity. In this example, a participant may have a wearable device 1202 with bio-signal sensors and a display to provide a VR environment with a participant view 1208.”);
a display module configured to display the extended reality scene, the human body state information, and the three-dimensional environment scene that are obtained from the data processing module (Aimone- ¶0043, at least discloses a wearable computing device with a bio-signal sensor, the wearable device in connection with a display to provide an interactive virtual reality (“VR”) environment for a user. The bio-signal sensor receives bio-signal data from the user, the bio-signal sensor comprising a brainwave sensor. The computing device having or in communication with a processor configured to as part of the interactive VR environment, present content on the display where the content has a VR event, desired user states, and desired effects; Fig. 1 and ¶0053, at least disclose “The wearable device 105 has a stereoscopic display 110 […] The stereoscopic display 110 is a 3-dimensional (3D) (or dual 2-dimensional (2D) images giving “3D”) display, but alternatively may be a 2-dimensional (2D) display […] The VR environment includes an interactive VR environment where content presented on the display may update or modify in response to input from bio-signal sensors 130, other sensors, user manual inputs, or other inputs”; ¶0063, at least discloses “in response to a determination that the user has achieved a particular brain state, or maintained a particular brain state for a period of time, the wearable device may update the display to provide an indication of the determination (e.g. indicating to the user what brain state has been achieved, and, optionally for how long) and may further display an indication of a particular reward assigned to the user in response to the determination”; Fig. 2 and ¶0071, at least disclose “a perspective view of a training system 200 with a display 225 […] The display 225 having the functionality of the stereoscopic display 110”; ¶0082, at least discloses “The VR application 315 is presenting a stream of content to the user 305 thereby creating the VR environment. In the VR environment, there are VR events 430” in combination with Fig. 3 and ¶0075, at least disclose “the VR Application 315 uses the user's intention and user state, including brain state, to control the content of a multimedia VR environment presented to the user to provide an interactive environment” and with ¶0080, at least discloses “the direction, location or orientation of the user 305 or other data from tracker 145 or sensors 120, 130 may also provide input to context logic 310 to determine context of the VR environment, such as, for example, the position of the truck relative to the user 305 within the VR environment or the orientation or eye gaze of the user 305 relative to the truck within the VR environment”, for example, ¶0082, at least discloses “The trainee or user 305 is, for example, driving a car in a training VR environment. The user 305 enters manual input 410 such as those from steering wheel, gas, and brakes; while the user's 305 biological features 420, such as brain waves […], are sensed by the sensors 120 and 130. The biological features 420 and manual input 410 are used to determine user state 425. The user state 425 and the desired user state 435 optionally may only include brain states and not include the manual input 410. The manual input 410 is further used to determine its effect in VR environment 415. The effect in the VR environment 415 is used by the VR application 315 to change aspects of the VR environment accordingly. The VR environment is interactive as it changes depending on manual input 410 and user state 425” or ¶0100, or Fig. 9 and ¶0111, at least disclose “the device allows the opportunity to create a VR environment where users learn and practise sensitivity by engaging with characters in the VR environment where they wear their “emotions on their sleeve” (visually represented in real time via colour-coding, auras, icons, logos, avatars, etc.) based on determined user or brain states during VR events within the VR environment.”, wherein Fig. 12 and ¶0114, at least disclose "a user 902, 904 may see the same […] content as the other user 902, 904 in the VR environment", or "the navigation within the VR environment may be an example of feedback based on the user state score", with an "example feedback based on his scored brain state"); and
a data transmission module configured to upload the human body state information of the individual and the three-dimensional environment scene of the position of the individual to a cloud server (Aimone- ¶0027, at least discloses “The wearable device, client device, or both, may also connect to a cloud server system with a processor”; Fig. 1 and ¶0053-0054, at least disclose “The other computing device 160 and the computing device 150 may be, for example, a computer, embedded computer, server, laptop, tablet, or mobile phone […] The other computing device 160 may also be a server over the Internet or other network”; Fig. 1 and ¶0061, at least disclose “the sensors may monitor the user, storing data in local storage mounted to the wearable device 105, and once moving into proximity with the other computing device 160, the sensors, or a transmitter of the wearable device 105 may transmit stored data to the other computing device 160 for processing.”; ¶0079, at least discloses “The data mining improver 330 further may use previous VR sessions to add new labelled data which is used to create a new prediction model or adapt an existing prediction model in the user state estimator 325. The parameters of the new prediction model are stored in the user profile 335 of that user.”; ¶0074, at least discloses “the user profile 335 may optionally store the history of all of the user's sessions, raw biological data, processed data, demographic data, etc.” and ¶0066, at least discloses “this user profile 335 may be stored in the cloud, allowing each user to maintain a single profile across multiple computing devices”).
Regarding claim 15, Aimone in view of Beall, discloses a human-machine interaction collection system for a wearable extended reality device (Aimone- Fig. 1 and ¶0053, at least disclose “a training system 100 with a wearable device 105 and other computing device 160”; ¶0057, at least discloses “the wearable device 105 is, for example, a wearable headset worn on a user's head. The computing device 150 of the wearable device 105 is configured to create a VR environment on the stereoscopic display 110 and sound generator 140 for presentation to a user”; ¶0028, at least discloses “A VR environment may also refer to an augmented reality environment which may be a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by computer processes.”), the human-machine interaction collection system comprising:
a processor (Aimone- ¶0027, at least discloses “a system uses a processor in a variety of contexts. However, it will be appreciated that a system can use multiple processors. As an illustrative example, a wearable device may include a processor that may in turn connect to a client device with another processor.”; Fig. 15 and ¶0181, at least disclose “a computer device 500 that may include a central processing unit (“CPU”) 502 connected to a storage unit 504 and to a random access memory 506); and
a memory having computer instructions stored thereon, wherein the processor is configured to execute the computer instructions stored in the memory, the processor, when executing the computer instructions, implementing a human-machine interaction collection method for the wearable extended reality device (Aimone- ¶0027-0028, at least disclose “a system uses a processor in a variety of contexts. However, it will be appreciated that a system can use multiple processors. As an illustrative example, a wearable device may include a processor that may in turn connect to a client device with another processor. The wearable device, client device, or both, may also connect to a cloud server system with a processor […] a processor may be connected to memory, but it will be appreciated that a variety of bridges and controllers may reside between the processor and memory […] the processor receives the bio-signal data of the user from the bio-signal sensor during the VR event and processes the bio-signal data to determine user states of the user, including brain states, during the VR event, where the user states are processed using a user profile stored in a data storage device accessible by the processor and the user states include brain states; Fig. 15 and ¶0181, at least disclose “a computer device 500 that may include a central processing unit (“CPU”) 502 connected to a storage unit 504 and to a random access memory 506. The CPU 502 may process an operating system 501, application program 503, and data 523. The operating system 501, application program 503, and data 523 may be stored in storage unit 504 and loaded into memory 506, as may be required. Computer device 500 may further include a graphics processing unit (GPU) 522 which is operatively connected to CPU 502 and to memory 506 to offload intensive image processing calculations from CPU 502 and run these calculations in parallel with CPU 502.”), the method comprising:
obtaining data synchronously collected, the data comprising a virtual reality base signal collected by the extended reality device, human body perception data collected by a human body parameter perception device, and environmental perception data collected by an environmental parameter perception device, and the human body perception data comprising at least one of eye movement data, a physiological state parameter, and electroencephalographic data (see Claim 1 rejection for detailed analysis);
providing an extended reality scene based on the virtual reality base signal collected by the extended reality device, obtaining human body state information of an individual based on the human body perception data collected by the human body parameter perception device, performing three-dimensional modeling based on the environmental perception data collected by the environmental parameter perception device, and obtaining a three-dimensional environment scene of a position of the individual (see Claim 1 rejection for detailed analysis);
displaying the extended reality scene, the human body state information, and the three-dimensional environment scene on a display unit of the extended reality device (see Claim 1 rejection for detailed analysis); and
uploading the human body state information of the individual and the three-dimensional environment scene to a cloud server (see Claim 1 rejection for detailed analysis).
The system of claim 17 is similar in scope to the functions performed by the method of claim 3 and therefore claim 17 is rejected under the same rationale.
5. Claims 4 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Aimone in view of Beall, further in view of Zhu (“Zhu”) [US-2018/0165745-A1]
Regarding claim 4, Aimone in view of Beall, discloses the method according to claim 3, and discloses the method further comprising:
receiving a team behavior recommendation generated by the cloud server based on team behavior decision information to generating, by the cloud server, the team behavior decision information based on physical state information of a plurality of individuals in a team and the environmental perception data (Aimone- ¶0090-0091, at least disclose “Some exemplary brain user states are ability of operator to learn; ability of operator to correctly predict accuracy of their decisions […] ability of operator to correctly predict accuracy of their decisions and 3) emotional state leading to impaired thinking.”; ¶0093, at least discloses “The ability of operator to correctly predict accuracy of their decisions is the difference in the operator's expectation of his/her error and the correct decision is known as prediction error. Prediction error can be measured from the operator's EEG. So if an operator expected that he/she made the correct decision but the system informed them that their decision was wrong would produce a large prediction error.”; ¶0131, at least discloses “For example, Liana is trying to exercise more. But she finds it difficult to stick with any one exercise for a prolonged period. At the same time, she doesn't enjoy aerobics or weightlifting classes. One of the trainers at the gym recommends that she use the system to maintain her attention while she is on machines like treadmills and bikes”; Beall- col 24, lines 28-38, at least disclose “Once a user is finished (or partially finished) with the user's VR slide presentation, the Designer client/system optionally presents a set of controls (e.g., a file control, an upload control, etc.) enabling a user to upload their finished presentation to a network-based server 200 accessible by a user/presenter and one or more VR participants and/or observers (e.g., collaborators). For example, a File, “Upload Scene” control is provided which may then be selected by a user to upload the new (or edited) VR slide presentation to a network-based server 200).
The prior art does not explicitly disclose receiving a team behavior recommendation subsequent to generating the team behavior decision information.
However, Zhu discloses
receiving a team behavior recommendation subsequent to generating the team behavior decision information (Zhu- ¶0035, at least discloses “the purpose of the present disclosure is to recommend products that are more valuable and more conform to the user's intention before the product is purchased and to accelerate the user's decision for making order, and to advise more strategies to the user after the product is purchased through reasonable and intelligent recommendation”; ¶0040-0041, at least disclose “The recommendation system needs to continually decide what to recommend to the user based on a series of behaviors of the user (e.g., products, stores, brands and events). Reinforcing learning is an example method to model intelligent decision-making. In a nutshell, intensive learning recursively models the changes in the short-term state of smart decision maker, ultimately progressively optimize their long-term goals […] a state of an intelligent decision maker (such as a recommendation system) is defined as the information that the recommendation system gathers prior to recommending to the user. For instance, the state includes the user's attribute information (such as gender, age, city and purchasing power) and the user's operation behavior sequence at the client terminal prior to the recommendation”).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Aimone/Beall to incorporate the teachings of Zhu, and apply the system needs to continually decide what to recommend to the user based on a series of behaviors of the user into Aimone/Beall’s teachings, for receiving a team behavior recommendation generated by the cloud server based on team behavior decision information subsequent to generating, by the cloud server, the team behavior decision information based on physical state information of a plurality of individuals in a team and the environmental perception data.
Doing so would achieve the purpose of guiding the user and to improve the shopping experience of the user.
The system of claim 18 is similar in scope to the functions performed by the method of claim 4 and therefore claim 18 is rejected under the same rationale.
Conclusion
6. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. They are recited in the attached PTO-892 form.
7. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL LE whose telephone number is (571)272-5330. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL LE/Primary Examiner, Art Unit 2614
