DETAILED ACTION
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-6, 11-13, and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Aguilar et al. (US 2010/0100001 A1).
As to claim 1, Aguilar teaches a method (i.e. the method of figures 1-3 and 10-11 which shows a heard mounted display device having augmented reality display application for a miliary personnel) (see Fig. 1, 10-11, [0030-0033]) comprising:
receiving one or more requested stimuli data from a user application on a smart device (i.e. the helmet device of the user element 12 as seen in figures 1-2) (see Fig. 1-2, [0030-0036]);
receiving at least one of sensor data and other context data, wherein the other context data includes data that is un-sensed (i.e. as seen in figure 11, the system of Aguilar uses the user stimuli 608 and a context a prior knowledge of stimuli 606 to combine together to form correlate CUI and Stimuli 612) (see Fig. 11, [0065-0068]);
transforming at least a portion of the requested stimuli data, into modified stimuli, based at least in part on at least one of the sensor data and the other context data (i.e. as seen in figure 10 and 11 the stimuli detected is further modified to create useable data for process temporal sequence 600 for the actual user interaction such as command and control application in a combat environment) (see Fig. 10-11, [0066-0068]);
presenting the modified stimuli and environmental stimuli to the user with a rendering device configured to mix the modified stimuli and the environmental stimuli, thereby resulting in rendered stimuli (i.e. as seen in figure 10a embodiment shows a digital display of the combat map simulation viewable in the computer monitor 510) (see Fig. 10A, [0061-0062]);
receiving biosignals from the user representing the user's focus (i.e. the system of Aguilar uses user eye-movement data to track user focus) (see Fig. 10, [0061-0062]), the biosignals generated in response to the rendered stimuli (i.e. the user viewing the data on the display screen as seen in figure 10A) (see Fig. 10A), on a wearable biosignal sensing device (i.e. the figure 1-2 HMD device 12) (see Fig. 1-2, [0030-0036]), wherein the wearable biosignal sensing device is configured to perform gaze tracking to detect the user's focus based in part on the user's fixation on at least one of the rendered stimuli (i.e. as seen in figure 10A the system of figure 10a shows a training system that uses gaze tracking to control the display system of figure 10A) (see Fig. 10A, [0061-0062]);
classifying the received biosignals using a classifier based on the modified stimuli, resulting in a classified selection (i.e. the selection on the display screen as seen in figure 10A) (see Fig. 10A, [0061-0062]); and
returning the classified selection to the user application (i.e. the display monitor 510 shows the updated display 508 after detection the user 502’s gaze command) (see Fig. 10A, [0061-0062]).
However, the Fig 1-2 and 10-11 embodiments Aguilar uses different display system to illustrate the design of the input output system, one of head mounted displays system used in a battle field condition as seen in figure 1-2, and the other seen as training system of figure 10a which shows an actual feedback mapping application in training exercises.
Since both display are disclosed as possible implementation of the same use gaze detection system, it would have been obvious for an artisan at the time of the filing date of the current application to have used the actual software implementation of figure 10A in the illustrated device of figure 1-2 to enable a combat application implementation of Aguilar’s display system as disclosed in figure 10 and 11 (see Aguilar Fig. 1-11, [0061-0067]).
As to claim 11, Aguilar teaches a system comprising:
a smart device (i.e. the Smart device 12 being worn by the user) (see Fig. 1-2, [0030-0034]);
a rendering device (i.e. the display system of figure 1-2 on the HMD that render’s the augmented display) (see Fig. 1-2, [0030-0034]);
a wearable biosignal sensing device on a user (i.e. the HMD device is an eye tracking device) (see Fig. 1-2, [0030-0034]);
a processor (i.e. as seen in figure 3 the signal processor 42) (see Fig. 3, [0033]); and
a memory storing instructions that, when executed by the processor (i.e. the system of figure 1-3 shows a computer system using electronic memory to enable the HMD to operate in a battle applications) (see Fig. 1-3, [0030-0033]), configure the system to:
receive one or more requested stimuli data from a user application on the smart device (i.e. the eye movement of the user as well as the EEG signals) (see Fig. 1-3, [0030-0033]);
receive at least one of sensor data and other context data, wherein the other context data includes data that is un-sensed (i.e. as seen in figure 11, the system of Aguilar uses the user stimuli 608 and a context a prior knowledge of stimuli 606 to combine together to form correlate CUI and Stimuli 612) (see Fig. 11, [0065-0068]);
transform at least a portion of the requested stimuli data, into modified stimuli, based at least in part on at least one of the sensor data and the other context data (i.e. as seen in figure 10 and 11 the stimuli detected is further modified to create useable data for process temporal sequence 600 for the actual user interaction such as command and control application in a combat environment) (see Fig. 10-11, [0066-0068]);
present the modified stimuli and the environmental stimuli to the user with the rendering device configured to mix the modified stimuli and the environmental stimuli, thereby resulting in rendered stimuli (i.e. the selection on the display screen as seen in figure 10A) (see Fig. 10A, [0061-0062]);
receive biosignals from the user representing the user's focus, the biosignals generated in response to the rendered stimuli, on the wearable biosignal sensing device, wherein, the wearable biosignal sensing device is configured to perform gaze tracking to detect the user's focus based in part on the user's fixation on at least one of the rendered stimuli (i.e. as seen in figure 10A the system of figure 10a shows a training system that uses gaze tracking to control the display system of figure 10A) (see Fig. 10A, [0061-0062]);
classify the received biosignals using a classifier based on the modified stimuli, resulting in a classified selection (i.e. the selection on the display screen as seen in figure 10A) (see Fig. 10A, [0061-0062]); and
return the classified selection to the user application (i.e. the display monitor 510 shows the updated display 508 after detection the user 502’s gaze command) (see Fig. 10A, [0061-0062]).
However, the Figures 1-2 and Figures 10-11 embodiments of Aguilar uses different display system to illustrate the design of the input output system, one of head mounted displays system used in a battle field condition as seen in figures 1-2, and the other seen as training system of figure 10a which shows an actual feedback mapping application in training exercises.
Since both display are disclosed as possible implementation of the same use gaze detection system, it would have been obvious for an artisan at the time of the filing date of the current application to have used the actual software implementation of figure 10A in the illustrated device of figure 1-2 to enable a combat application implementation of Aguilar’s display system as disclosed in figure 10 and 11 (see Aguilar Fig. 1-11, [0061-0067]).
As to claim 2, Aguilar teaches the method of claim 1, wherein the gaze tracking uses at least in part, data from cameras inside the wearable biosignal sensing device (i.e. as seen in figure 10A Aguilar explicitly defines the camera 504 which track the user’s gaze which is used to sense the user biosignal, this when view together with the helmet mounted system of figure 11 which shows an eye tracking system which is inside the element 12 HMD system as outlined in element 20 shows that the said camera is indeed in the wearable biosignal sensing device of figure 10 and 11) (see Fig. 10-11, [0061-0068]).
As to claim 3, Aguilar teaches the method of claim 1, further comprising, after receiving the biosignals from the user: determining whether to send the received biosignals to the classifier based at least in part by using gaze based focus detection (i.e. as seen in figure 10A the GUI system of figure 10A uses the user gaze based focus detection to activate a on screen icon based on the classification of the user gaze to change the on screen data outputs) (see Fig. 10A, [0061-0062]).
As to claim 5, Aguilar teaches the method of claim 1, wherein the modified stimuli include steady-state motion visually evoked potential stimuli, and presenting the modified stimuli and environmental stimuli to the user includes rendering the modified stimuli and environmental stimuli on at least one of: an augmented reality optical see-through (AR-OST) device associated with the smart device; and a video see-through (VST) based head-mounted display (HMD) associated with the smart device (i.e. as seen in figure 10A the GUI system of figure 10A uses the user gaze based focus detection to activate a on screen icon based on the classification of the user gaze to change the on screen data outputs which is presented in figure 11 which shows a HMD 12 which uses an augmented reality helmet system for a military type of applications) (see Fig. 10A, 11, [0061-0068]).
As to claim 6, Aguilar teaches the method of claim 1, further comprising, after receiving the biosignals from the user: determining whether to send the received biosignals to the classifier by using at least one of: the existence of an intentional control signal, wherein determination of the existence of the intentional control signal includes at least one of: detecting a manual intention override signal from the smart device; and determining, at least in part, from the received biosignals that the user is intending to fixate on at least one of the rendered stimuli; and the absence of the intentional control signal; on condition the intentional control signal exists: sending the received biosignals, to the classifier; and on condition the intentional control signal is absent: continue receiving the received biosignals from the user (i.e. as seen in figure 10A Aguilar explicitly defines the camera 504 which track the user’s gaze which is used to sense the user biosignal for detecting user’s intention control signal to control the GUI system on screen, this when view together with the helmet mounted system of figure 11 which shows an eye tracking system which is inside the element 12 HMD system as outlined in element 20 shows that the said camera is indeed in the wearable biosignal sensing device of figure 10 and 11, which continues to operate as the system is a battle field communication system that continues to operate during operations) (see Fig. 10-11, [0061-0068]).
As to claim 12, Aguilar teaches the system of claim 11, wherein the wearable biosignal sensing device includes cameras; and the instructions further comprising: use, at least in part by the gaze tracking, data from the cameras inside the wearable biosignal sensing device (i.e. as seen in figure 10A Aguilar explicitly defines the camera 504 which track the user’s gaze which is used to sense the user biosignal, this when view together with the helmet mounted system of figure 11 which shows an eye tracking system which is inside the element 12 HMD system as outlined in element 20 shows that the said camera is indeed in the wearable biosignal sensing device of figure 10 and 11) (see Fig. 10-11, [0061-0068]).
As to claim 13, Aguilar teaches the system of claim 11, the instructions further comprising: determine, after receiving the biosignals from the user, whether to send the received biosignals to the classifier based at least in part by using gaze based focus detection (i.e. as seen in figure 10A the GUI system of figure 10A uses the user gaze based focus detection to activate a on screen icon based on the classification of the user gaze to change the on screen data outputs) (see Fig. 10A, [0061-0062]).
As to claim 15, Aguilar teaches the system of claim 11, wherein the modified stimuli include steady-state motion visually evoked potential stimuli; the instructions further including: present the modified stimuli and environmental stimuli to the user including rendering the modified stimuli and environmental stimuli on at least one of: an augmented reality optical see-through (AR-OST) device associated with the smart device; and a video see-through (VST) based head-mounted display (HMD) associated with the smart device (i.e. as seen in figure 10A the GUI system of figure 10A uses the user gaze based focus detection to activate a on screen icon based on the classification of the user gaze to change the on screen data outputs which is presented in figure 11 which shows a HMD 12 which uses an augmented reality helmet system for a military type of applications) (see Fig. 10A, 11, [0061-0068]).
As to claim 16, Aguilar teaches the system of claim 11, wherein the instructions further configure the system to, after receiving the biosignals from the user: determine whether to send the received biosignals to the classifier by using at least one of: the existence of an intentional control signal, wherein determination of the existence of the intentional control signal includes at least one of: detect a manual intention override signal from the smart device; and determine, at least in part, from received biosignals that the user is intending to fixate on at least one of the rendered stimuli; and the absence of the intentional control signal; on condition the intentional control signal exists: send the received biosignals, to the classifier; and on condition the intentional control signal is absent: continue to receive the received biosignals from the user (i.e. as seen in figure 10A Aguilar explicitly defines the camera 504 which track the user’s gaze which is used to sense the user biosignal for detecting user’s intention control signal to control the GUI system on screen, this when view together with the helmet mounted system of figure 11 which shows an eye tracking system which is inside the element 12 HMD system as outlined in element 20 shows that the said camera is indeed in the wearable biosignal sensing device of figure 10 and 11, which continues to operate as the system is a battle field communication system that continues to operate during operations) (see Fig. 10-11, [0061-0068]).
As to claim 17, Aguilar teaches the system of claim 11, wherein the modified stimuli is based in part on determining a device context state using at least one of the sensor data and the other context data (i.e. as seen in figure 11 the system of Aguilar uses context to identify user action with respect to location and other context data) (see Fig. 11, [0061-0068]).
As to claim 18, Aguilar teaches the system of claim 11, wherein presenting the modified stimuli and environmental stimuli to the user includes rendering the modified stimuli and environmental stimuli using at least one of a visual device, a haptic device, and an auditory device sensed by the user (i.e. as seen in figure 10A the GUI of the display unit is a visual system that shows the user the location of the application simulations) (see Fig. 10A, [0061-0062]).
As to claim 19, Aguilar teaches the system of claim 11, wherein the at least one of the sensor data and the other context data includes at least one of: environmental data, body-mounted sensor data, connected ambulatory device data, location specific connected device data, and network connected device data (i.e. the system of Aguilar figure 10-11 shows a combat network based command and control based application system which is able to detect the body mounted sensor of the soldier and display the information based on environmental data such as location and battlefield conditions) (see Fig. 10-11, [0061-0068]).
Claims 4, 7-10, 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Aguilar as applied to claims 1 and 11 above, and further in view of Mettler May (US Pub: 2017/0061817 A1).
As to claim 4, Aguilar teaches the method of claim 1, further comprising, receiving (EEG) biosignals in conjunction with gaze tracking (i.e. Aguilar only implicitly teaches EEG and do not mention EMG systems) (see Fig. 1-2, [0030-0036]).
But do not teaches using EMG technology.
Mettler May teaches detection comprising, receiving electromyography (EMG) biosignals for further enhance for the system with vision based tracking algorithm (i.e. as seen in figure 5-7, Mettler May teaches the application of EMG signals to track user body signals in addition to equipment motion information from a vision-based tracking systems) (see Fig. 5-7, [0157-0158]).
Therefore, it would have been obvious for one of ordinary skill in the art at the time the filing date of the current application to have used the EMG biosignal detection of Mettler May in place of the EEG system of Aguilar, in order to further enhance detection quality) (see Mettler May, Fig. 5, [0157-0158]).
As to claim 14, Aguilar and Mettler May teaches the system of claim 11, the instructions further comprising, receive electromyography (EMG) biosignals in conjunction with gaze tracking (i.e. as seen in figure 5-7, Mettler May teaches the application of EMG signals to track user body signals in addition to equipment motion information from a vision-based tracking systems) (see Fig. 5-7, [0157-0158]).
As to claim 7, Aguilar teaches the method of claim 1, but do not explicitly teach further comprising: receiving, by a cloud server, the classified selection from the classifier, the cloud server including: a context manager; a machine learning model, used by the smart device to facilitate classification of the received biosignals by the classifier; and at least one model modification process for modifying the machine learning model; receiving, by the context manager, at least one of current context state data and requests for other state data; receiving, by the at least one model modification process, at least one of new state data and updated state data from the context manager; and updating the machine learning model using the at least one model modification process and at least one of the classified selection, the new state data, and the updated state data.
Mettler May teaches further comprising: receiving, by a cloud server, the classified selection from the classifier, the cloud server including: a context manager; a machine learning model, used by the smart device to facilitate classification of the received biosignals by the classifier; and at least one model modification process for modifying the machine learning model; receiving, by the context manager, at least one of current context state data and requests for other state data; receiving, by the at least one model modification process, at least one of new state data and updated state data from the context manager; and updating the machine learning model using the at least one model modification process and at least one of the classified selection, the new state data, and the updated state data (i.e. as seen in figure 9-13 embodiment of Mettler May, the cloud computing system of figure 12 allows the user to take advantage of a machine learning algorithm to enhance system diagnosis and ensure real-time, low latency and high bandwidth operations) (see Figure 9-13, [0180-0209]).
Since both Aguilar and Mettler May uses dynamic networking interconnection to create real time detection and operations, they are analogous as having a similar field of endeavor. Therefore, it would have been obvious to apply the machine learning cloud based serving computing system of Mettler May in the overall military battlefield display system of Aguilar in order to improve the communication system by creating a real-time low latency 10ms high bandwidth system with modern communication applications (see Mettler May Fig. 12, [0180-0185]).
As to claim 8, Aguilar and Mettler May teaches the method of claim 7, further comprising: sending an updated machine learning model, from the cloud server to the smart device; and transmitting the updated machine learning model to the classifier using a machine learning model transmission controller on the smart device (i.e. Aguilar figure 11 and Mettler May figures 9-12 shows that the machine learning model can be used to create a cloud computing communication system that further enhance operational capacity) (see Aguilar Fig. 11, [0061-0068], Mettler May Fig. 9-13, [0180]).
As to claim 9, Aguilar and Mettler May teaches the method of claim 8, further comprising: requesting a new machine learning model from the cloud server, by a context module on the smart device using the machine learning model transmission controller; receiving, by the smart device, the new machine learning model from the cloud server; and transmitting the new machine learning model to the classifier (i.e. Aguilar figure 11 and Mettler May figures 9-12 shows that the machine learning model can be used to create a cloud computing communication system that further enhance operational capacity) (see Aguilar Fig. 11, [0061-0068], Mettler May Fig. 9-13, [0180]).
As to claim 10, Aguilar and Mettler May teaches the method of claim 1, further comprising a context manager on a cloud server, wherein the context manager provides additional context information to the smart device (i.e. Aguilar figure 11 and Mettler May figures 9-12 shows that the machine learning model can be used to create a cloud computing communication system that further enhance operational capacity) (see Aguilar Fig. 11, [0061-0068], Mettler May Fig. 9-13, [0180]).
As to claim 20, Aguilar and Mettler May teaches the system of claim 11, wherein the instructions further configure the system to: receive, by a cloud server, the classified selection from the classifier, the cloud server including: a context manager; a machine learning model, used by the smart device to facilitate classification of the received biosignals by the classifier; and at least one model modification process for modifying the machine learning model (i.e. as seen in figure 9-13 embodiment of Mettler May, the cloud computing system of figure 12 allows the user to take advantage of a machine learning algorithm to enhance system diagnosis and ensure real-time, low latency and high bandwidth operations) (see Figure 9-13, [0180-0209]); receive, by the context manager, at least one of current context state data, and requests for other state data; receive, by the at least one model modification process, at least one of new state data, and updated state data from the context manager; and update the machine learning model using the at least one model modification process and at least one of the classified selection, the new state data, and the updated state data (i.e. Aguilar figure 11 and Mettler May figures 9-12 shows that the machine learning model can be used to create a cloud computing communication system that further enhance operational capacity) (see Aguilar Fig. 11, [0061-0068], Mettler May Fig. 9-13, [0180]).
Response to Arguments
Applicant's arguments filed 01/01/2026 have been fully considered but they are not persuasive. The applicant argues in pages 2-5 of the reply that “Independent claim 1 includes "transforming at least a portion of the requested stimuli data, into modified stimuli, based at least in part on at least one of the sensor data and the other context data" (emphasis added) and subsequently "presenting the modified stimuli." Independent claim 11 contains similar limitations. The Office Action cites Aguilar's Figures 10 and 11 and paragraphs [0066-0068] as teaching this feature, stating that "the stimuli detected is further modified to create useable data for process temporal sequence 600." (See Office Action, page 3). Applicant respectfully submits that the Examiner has conflated the processing of sensor data (input) with the transformation of stimuli data (output) presented to the user. In Aguilar, the system performs "fixation-locked measurement." As described in Aguilar's Abstract and paragraphs [0010]-[0011], the system monitors a user's eye movements in a "free- viewing environment" to detect fixation events. When a fixation is detected, the system time windows the EEG data to determine if a "significant cognitive response" (e.g., recognition of a threat) occurred. The "modification" cited by the Examiner in Aguilar (e.g., removing artifacts, correlating cues, see Fig. 11 step 612 and [0066]-[0068]) relates to the post-processing of the collected EEG and eye-tracking signals. The stimuli themselves are not modified based on the environment in any way; they are simply overlaid on a background. Thus the "modification" of Aguilar does not relate to transforming an application's requested visual content into a modified visual stimulus for presentation to the user.” The examiner respectfully disagrees. As seen in figure 10a the system of Aguilar explicitly teaches a continuous display system that the user interact with which updates the stimuli as the system detect user interaction event with respect to the environment such as nearby layout maps which is updated in real time. This means that the display stimuli of the battle field conditions such as the mapping application shows a constantly changing stimuli viewed directly by the visual tracking which indicates change in the environment which is actively displayed. Specifically Aguilar teaches in paragraph [0068] that “External knowledge of the stimuli, either a priori knowledge of the stimuli (step 606) such as might occur in the controlled presentation of language learning materials or external detection of stimuli (step 608) such as might be provided by acoustic or imaging sensors, can be used to provide a stimuli time-code (step 610). This time-code can be correlated to the fixation-event time code (step 612). This allows external knowledge of the stimulus to be correlated to and processed with the cues. For example, in language learning the system may be able to correlate a specific query to a specific cue. In an urban combat environment, command and control may be able to correlate a sensed condition(imaging, acoustic or other) to a specific cue.” Here, the urban combat environment based stimuli is seen by the user of the display in a dynamic capacity which shows interactions such as selection as seen in figure 10a embodiment. Therefore, Aguilar’s dynamic visual stimuli display system with active fixation lock detect still read on the independent claim 1 and 11 in its broadest reasonable interpretation.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The prior art Huggins et al. (US Pub: 2021/0330242 A1) is cited to teach another design for use visual sensing display system figure 1-3 embodiment. The prior art Wang et al. (US Pat: 11,093,033 B2) is cited to teach another type of uses gaze tracking method with EEG detection method in figures 1-6 embodiments. The prior art Schiff et al. (US Pub: 2020/0012346) is cited to teach another type of user visual tracking sensing device having brain signal detection means in figures 1-3 embodiments. The prior art el Kaliouby et al. (US Pub: 2020/0074154 A1) is cited to teaches figure 1-5 embodiments which demonstrate a cloud based server system for storing and deploying data for a machine vision application system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALVIN C. MA whose telephone number is (571)270-1713. The examiner can normally be reached on 8:00AM-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benjamin C. Lee can be reached on 571-272-2963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALVIN C MA/Primary Examiner, Art Unit 2693 April 18, 2026
20140331791
Prosecution Insights